Safe Mode (Improper Installation) After Reboot

[royalcow]

This morning I rebooted our RT Linux target and was rewarded with a safe mode (improper installation) status in MAX. I had originally rebooted since I was getting file I/O errors when trying to deploy a shared variable library. Now I am unable to get it out of safe mode or install software due to an error in MAX "Error while reading a file from the local disk. The file may be corrupt or not present."

I was able to look at some of the trace log files and the errorlog.txt file had the following:

03/28/2016	13:51:55	appweb: Error: Cannot open log file /var/local/natinst/log/SystemWebServer.log
03/28/2016	13:51:55	appweb: Error: Cannot write to ErrorLog: /var/local/natinst/log/SystemWebServer.log
03/28/2016	13:51:55	appweb: Error: Error with directive "ErrorLog"
03/28/2016	13:51:55	appweb: Warning: Server "default" cannot be unregistered from the Service Locator: unrecognized name

Thinking file permissions, I verified that that file and the parent directories had the correct permissions, which they do. I have compared the permissions on this target to a similar one and did not find anything out of the ordinary.

I would rather find and fix this error and not format the target, if possible, since there are non-NI services installed and lots of environment configuration on the target that I don't want to have to backup and restore.

Any ideas? Thanks!

[gratian.crisan · Accepted solution]

Can you check how much free space is available on the runmode partition?

You can do this from safemode via a serial or ssh console by running the 'df' utility. The runmode partition is mounted (while in safemode) at: /mnt/userfs.

Also it would be useful for debug purposes to disable the 'watchdog' that reboots the target into safemode if it detects a software error. You can do this by setting the following token in /etc/natinst/share/ni-rt.ini under the [Startup] section:

     [Startup]

     YouOnlyLiveTwice=false

If you can get to a console after setting that token and rebooting in runmode can you post a list of running processes i.e. the output from running 'ps'. That will help us narrow down what failed to start and caused the target to go into safemode.

[royalcow]

Well! It turns out that the free space was on the runmode partition was the problem. After cleaning up a little space I was able to reboot the controller and get it back up and running. Looks like I need to do a bit of cleanup on that partition. Other than some of the log locations and whatever else we have added to the target, are there some other places we should look to cleanup?

Thank you so much for the point in the right direction! Also thanks for the INI key. I am sure that will come in handy next time something like this happens.

[gratian.crisan]

Glad that helped.

The way I usually go about finding where the space is used is by going to the root '/' folder and using the 'du' utility to find out where the space is used and then drill into subfolders from there.

Something like:

     cd /

     du -hs *

You can ignore the following temporary/volatile folders: /dev, /sys, /proc, /run, and /var/volatile.