Hobbyist Toolkit
Improving deployed executable startup/reboot/longevity reliability
Solved!
Hi fellow wireworkers,
I have success running my deployed LV startup executable on the RPi Zero 2 W continuously for many days (at least 9 to be exact). I now want to solve two problems:
- Startup after a cold start or reboot of RPi is not consistent in that about 30% of the time the executable fails to run, sometimes failing to launch 3-4 reboots in a row . My reboot method is either via RPi GUI (shutdown/reboot) or via a forced watchdog reboot or sometimes removing/replugging USB cable to powercycle. (my app is not doing any SD Card writes at the time). Restarting the LabVIEW service via the LV IDE \Utilities\Restart menu I *think* gives more reliable restarts (I still need to quantify how reliable though)
- Automatically recovering from a lock up of the startup executable after many days of stable running.
On point 1. , is that a common situation that other developers here see ? Any thoughts on how I could improve the reliability because I need it to recover automatically, i.e. if a reboot fails to launch the startup EXE I must keep auto-rebooting until it launches OK.
for 2. I am pursuing two possible ideas
a) Use the RPi internal watchdog. I have succeeded to install and enable the watchdog per many other RPi centric articles e.g. this. My next step is for my LV exe to to disable the wd_keepalive daemon service (using the SSH trick and sudo commands etc). and then have the exe pat/tickle the watchdog every 14 seconds. OR
b) Write my own watchdog (BASH script or even a second LV exe running in parallel at startup if possible) that just restarts the LabVIEW service if the main app doesn't tickle it. It would use the linux command that achieves the same as the \Utilities\Restart I mentioned above) :
sudo systemctl restart labview.service
any thoughts on a) or b) ?
thanks
HI Andy, thanks for replying ! My other reply with the log file attached , I sent before I realised you had replied.
I'm assuming the failure at times to run after restart behaviour I see is not expected or at least unusual.
to answer your questions:
>Is your application using the network interface?
no it isn't. The main things it does prior to the RPi having fully booted is to
- creating and writing to text files at /srv/chroot/labview/usr/local/*.txt
- set some GPIO lines
- poll the USB serial port for data
>Check that all the inputs to your main vi have default values assigned.
ok, that is a suggestion out of left field ! The Main vi only has one input to it per below called Module Admin . Don't default value exist for all controls and indicators ?
I would definitely try to wait for full startup before doing much serious work. Especially the GPIO looks suspicious. Before the host OS is fully started, the mapping of the device entries in the chroot may actually point into non existing device entries on the host. File system could also be affected but I would expect that to be a bit more robust and not just cause a fatal SIGSEGV if not yet fully available.
Since it works much of the time the needed delay may be relatively short and could be just hardwired but you could also try the new VI to check for network availability maybe, that is present since 2019 or so.
I found the main culprit causing BSOD equivalant crashes during execution ! (It was NOT solved by disabling the Remote GPIO setting in the RPi) The culprit - I was calling the following code (Open,Write GPIO,Close) twice in succession without any delay between each successive call.
Inserting a delay of at least 15-20ms solved the "BSOD" equivalent I was getting - a segmentation fault LabVIEW service crashing bug.
Now you might be asking why I do an open and close for my GPIO writes (and I'm inclined to change it). It was because I was doing multiple asynchronous write in parallel. So I initially solved the problem by having the above atomic operation - as I found I could not have multiple instance of the Open Local.vi in my code, I could only have one instance so I would have then had to share the LINX resource reference globally amongst all my asynchronous modules - something I put aside at the time. I suspect/hope that if I was to do that I could then call the Digital Write.vi much more often than 10-15ms, in fact, perhaps I should be able to call it with no delay and it should n ot cause this RPi BSOD ! I am still experiencing this a little at startup per my original post so I will soon go through my init routines to insert delays to confirm I can now solve both problems and the need for a watchdog will drop to a very low priority.
BTW as an aside, it takes only 3 seconds to restart the LabVIEW service and have the startup.rtexe re-running - that is nice and fast. It does take much longer to restart the RPi (like 20-ish seconds before the startup.rtexe is running, so if I ever need to do a watchdog I will do it at two levels, one to restart the LabVIEW service if it or my app stops and then another watchdog to restart the RPi if the first method somehow fails.
My next and hopefully last report back here will be to confirm I solved the startup BSOD that currently happens about 30% of the time and I suspect has the same root cause.
