LSI MegaRAID Full Logs Issue

Some Wavestore systems are installed with LSI MegaRAID cards. These RAID cards write a log file of any RAID-related events to the operating system's main partition. Unfortunately, due to bugs in the Linux drivers for this device, this logging is not configurable and occurs to an unsuitable location, and continues to grow forever.

After a long period of operation, normally in the region of years, these logs can reach a size where they have completely filled the operating system main partition. When this happens, upon reboot, a screen such as the following may be seen:

IMG 2445.jpeg

These errors occur because the Wavestore graphical interface is trying to start up and needs to write some temporary files to do so. Due to the lack of disk space, the temporary files cannot be written and the GUI cannot start. In most circumstances the Wavestore server will have started successfully, and so recording will likely be working as normal, and remote access should be possible. The one server-side feature which is likely to not work is detection of RAID events, such as disk failures, since the Wavestore system monitors the RAID logs and these are no longer being written.

Solutions:

  • If the server is running version v6.10 or later and remote access is possible

From Wavestore version 6.10 there is a command which can be run remotely to clear the log files. To do so...

  1. Connect to the Wavestore using WaveView
  2. Go to "Tools > Execute Command..."
  3. Enter the command clear-tmp and click Run
  • If the server is running v6.8 or earlier or remote access is not possible

You will need physical access to the server and a USB stick with the Wavestore bootable installer on it. It can be any version but if in doubt, use the latest available.

  1. Boot the Wavestore from the USB stick.
  2. At the menu select "Advanced Options"
  3. Select "Rescue installed system"
  4. After a few seconds of start-up you should have a menu with 4 options. Choose option "1) Continue".
  5. The next part should say "Your system has been mounted under /mnt/sysimage..." followed by some more words, then "Please press to get a shell.", so press the Enter key.
  6. Now run this command: rm /mnt/sysimage/MegaSAS.log then press Enter. If it succeeded, you should just get a new line with no errors.
  7. The log has now been cleared. You can run exit or reboot to restart the system.
  • Prevention if running v6.8 or earlier and the system doesn't have the issue yet

If you have a server running v6.8 or earlier and you do not wish to upgrade to a later release, yet you want to avoid this issue occurring, it is possible to install just the software package which contains the new "clear-tmp" command. It can be downloaded here:

https://files.wavestore.com/public/misc-rpms/venusd-exec-6.10.7-1.el7.noarch.rpm

Then it can be installed using the WaveView upgrade screen.

Then you can follow the steps above, that is:

  1. Connect to the Wavestore using WaveView
  2. Go to "Tools > Execute Command..."
  3. Enter the command clear-tmp and click Run