Target update via SSH (LV 2019+)

[makotanko]

I just wanted to share a basic how-to for remotely updating a RIO using only SSH and bash commands. These have been put together from various snippets here in the discussions, and from my own experimenting. 

These may be used to develop a bash script or an Ansible playbook to update and manage your targets. If you use Ansible's raw module then you may even avoid having to install Python on the targets! 

Creating an image of current system:

Spoiler

1. Reboot target into safe or install mode via SSH: 

   /etc/init.d/nisetbootmode force-install "just gettin' an image" && reboot

2. Login via SSH again once target has rebooted.
3. Create the image file:

   nisystemimage get -x tgz -f some_image_name.tar.gz

   This will take a few minutes to complete, and will complete silently. After it is done your image file should show up in your current folder (I usually just use the home folder).

4. Copy image file to your host PC via SCP. From a terminal on your host PC run:

   scp admin@192.168.0.5:~/some_image_name.tar.gz desired/save/path/

   There are multiple ways to copy a file from a target to your PC, this is just my own preference!
5. Reboot the target. It should automatically boot back into run-mode. NOTE: This will delete the image file that you created since the safe-mode OS is volatile! If it does not automatically go back into run-mode, then SSH in and run: 

   /etc/init.d/nisetbootmode force-runtime && reboot

Applying a system image:

NOTE: Everything in the runtime OS will be overwritten, including the system password!

Spoiler

1. Reboot target into safe or install mode via SSH: 

   /etc/init.d/nisetbootmode force-install "just setting an image" && reboot

2. Copy image file to your target via SCP. From a terminal on your host PC run:

   scp path/to/image/some_image_name.tar.gz admin@192.168.0.5:~

   There are multiple ways to copy a file to your target, this is just my own preference!
3. Login via SSH again and apply the image:
   

   nisystemimage set -x tgz -f some_image_name.tar.gz -p preserve -s preserve

   This usually finishes within a minute or two. I usually get errors like "/dev/sda4 is not encrypted (not a luks device)" or "nilrtdiskcrypt_canopen: No such file or directory". Everything seems to work just fine despite these messages. Both the primary and secondary network adapter settings are preserved here.
4. [Optional] Make sure SSH is still enabled in the new configuration. Run:

   nirtcfg --set section=SYSTEMSETTINGS,token=sshd.enabled,value=True

5. Reboot the target. It should automatically boot back into run-mode. If it does not automatically go back into run-mode, then SSH in and run: 

   /etc/init.d/nisetbootmode force-runtime && reboot

Updating the firmware/safe-mode OS:

Spoiler

1. Locate the firmware *.cfg file for your target on your PC (this should come with your installation of NI-RIO). These are usually located in C:\Program Files (x86)\National Instruments\Shared\Firmware\cRIO\Device_code\ where "Device_code" is "76D6" for a cRIO-9068 and "78E9" for a cRIO-9047.
2. Copy the firmware file to your target via SCP. From a terminal on your host PC run:

   scp path/to/firmware/some_firmware.cfg admin@192.168.0.5:~

   There are multiple ways to copy a file to your target, this is just my own preference!
3. SSH into the target and run:
   

   niinstallsafemode some_firmware.cfg

   This may take a minute or two and usually gives "not a luks device" errors, but this is OK (maybe this only applies to those with encrypted disks?).
4. Verify the installed firmware version by running:
   

   nisafemodeversion

Some notes:

• The niinstallsafemode and nisystemimage commands seem to be new with LV 2019, so you may not have these tools installed on your target. 
• Testing was done using cRIO-9086 and 9047 targets running NI Linux Real-Time 4.14.87-rt49-cg-7.0.0f0. 
• These and other command line tools are located in /usr/local/natinst/bin/. These tools also have little-to-no documentation. This combined with little mention of them in the white papers tells me that they may be unsupported! Use caution!
• An image that was created with RAD can be applied using nisystemimage. You just have to unzip the *.lvappimg file and extract the *.tar.gz file within it. 

[StephenB]

This is about the best thing I've ever seen posted on this forum. Thank you for finding this and documenting it!

We use Linux as our development environment and thus dealing with Windows and MAX to set up the NI Linux RT targets is awful. We have much of it done in ansible but didn't know how to do firmware or images so that remained in MAX and RAD. This fills that gap! We can probably get rid of windows and MAX! 

For PXI I'm also going to cache all the opkg files into our ansible repo so we can just install those without ever using MAX. I havent looked at if opkg based drivers are available for crio yet tho. (we have crios on 2018 and recently started testing PXI on 2019).

[rtollert · Accepted solution]

Several of us internal NI developers are absolutely chuffed at this thread. Thank you.

Also thanks for demonstrating that some customers are using ansible. I do feel obliged to ask how tied you are to ansible, because SystemLink is all salt-based, and you are generally going to have a smoother supportability experience if you can hitch your sails to that instead.

---

Applying a system image:

NOTE: Everything in the runtime OS will be overwritten, including the system password!

Two caveats:

1. User-installed FPGA bitfiles (i.e. which are configured to run at startup) are not saved/restored. IIRC, the higher-level system imaging primitives exposed by NI System Configuration (and RAD) take care of that separately.
2. If you have any private keys installed (used for the webserver and a couple of other things), they are not going to be encrypted; you might consider using `openssl enc` or `gpg --symmetric` or something to protect the image at rest, if you have sensitive information. IIRC, NI System Configuration/RAD solves this by leaving sensitive content out of the tarball and encrypting it separately in the lvappimg.

---

Spoiler

1. Login via SSH again once target has rebooted.
2. Create the image file:

   nisystemimage get -x tgz -f some_image_name.tar.gz

3. Copy image file to your host PC via SCP. From a terminal on your host PC run:

   scp admin@192.168.0.5:~/some_image_name.tar.gz desired/save/path/

---

This will fail if the image is too large.

• One workaround is to save the image to the runmode filesystem (/mnt/userfs). This is what NI System Configuration does. This brings up the unfortunate complication that you need to prevent nisystemimage from saving off its own output image. You can accomplish this by using the `-b <path>` option to add a blacklist entry; any path equal to <path> or which is a subdirectory of <path> will not be saved into images or restored from images.
• But since you're already on a Unix host, you could instead just pipe it:
  

  ssh admin@target "nisystemimage get -x tgz" > image.tar.gz

  And do the likewise thing on restore. The main drawback with this approach is that if the file transfer fails, your runmode is hosed; but since you're already in safemode, you should be able to recover from this.

---

The niinstallsafemode and nisystemimage commands seem to be new with LV 2019, so you may not have these tools installed on your target. 

---

AFAIK, nisystemimage and niinstallsafemode have existed since LV 2014. So I'm not sure how you came to this conclusion.

---

Testing was done using cRIO-9086 and 9047 targets running NI Linux Real-Time 4.14.87-rt49-cg-7.0.0f0.

---

The general nisystemimage process should continue to work, IIRC, for the vast majority of NI Linux RT controllers which use the "old" (non-opkg-based) MAX install process. For opkg-based installs, which includes PXI and SystemLink, it might work, but is not tested and is discouraged — configuration management is meant to be done through SystemLink and/or salt. It definitely won't work at all for NXG.

Offhand, I am aware of no proscriptions against niinstallsafemode, except the usual ones (it's completely unsupported when used directly; if it fails you will need to call NI support to iron things out; etc.)

---

These and other command line tools are located in /usr/local/natinst/bin/. These tools also have little-to-no documentation. This combined with little mention of them in the white papers tells me that they may be unsupported! Use caution!

---

You can say that again!

---

An image that was created with RAD can be applied using nisystemimage. You just have to unzip the *.lvappimg file and extract the *.tar.gz file within it.

---

Yes — with the caveats mentioned above about how NI System Configuration does its own thing about sensitive content. IIRC, the principal side effect of applying a system image directly is that system passwords will probably not be updated.

[StephenB]

I think we would be interested in using system link instead of ansible (since system link is supported first class by NI) if it could be run on a linux server

[makotanko]

@rtollert, huge thanks for providing more info and the caveats about these functions.

Also thanks for demonstrating that some customers are using ansible. I do feel obliged to ask how tied you are to ansible, because SystemLink is all salt-based, and you are generally going to have a smoother supportability experience if you can hitch your sails to that instead.

I explored the option of using SystemLink in the past, it looks like a very promising tool and I personally would have preferred to use it. The decision to go with ansible-based management was primarily driven by conformity. The IT department here is familiar with ansible and already had SSH ports open in the remote networks for existing Unix boxes.

AFAIK, nisystemimage and niinstallsafemode have existed since LV 2014. So I'm not sure how you came to this conclusion.

I did not find these tools on some of our 9068 systems running 6.0.0 and 6.5.0 so I assumed it was a 2019 addition. Perhaps they're located in a different directory?

Thanks again for the info.

[AmilyChen]

I just want to change my NI PXIe 8840 QC from windows to Linux RT. And the local sales not support for this application, just give me some documents.

Now,I changed the boot and the get PXI into Linux safemode.

But, when I formdisk from the MAX, the MAX crashed!!

I can connected PXI by putty.

If I can use the nisystemimage command to install the image files??

And I wander if the \Program File(x86)\National Instruments\RT Image\PXI System Image\19.0.0\systemlink-linux-x64-dkms.tar is the image file?

Hope for your apply! Thanks very much!

[rtollert]

This reply is not pertinent to the original post — that was about advanced control of the system imaging process, while the basic functionality is not working right for you — but I might be able to give you some pointers.

---

@AmilyChen wrote:

Now,I changed the boot and the get PXI into Linux safemode.

But, when I formdisk from the MAX, the MAX crashed!!

Two things. First of all, if MAX crashes there, that is a problem (and probably a recent regression), and NI support should be informed about it. Second (and more importantly) — did you simply switch the BIOS boot setting to Safe Mode? If so, that's insufficient, you need to follow the instructions in e.g. http://www.ni.com/tutorial/55153/en/ to boot off of a USB recovery key to refresh the disk layout.

---

I can connected PXI by putty.

---

I'm assuming you mean ssh. It would not overly surprise me if you were able to ssh in without a working Linux filesystem, but it would surprise me a little. Uploading the contents of /var/log/messages here would not be a bad idea for the purposes of investigation.

If I can use the nisystemimage command to install the image files??

And I wander if the \Program File(x86)\National Instruments\RT Image\PXI System Image\19.0.0\systemlink-linux-x64-dkms.tar is the image file?

---

No, none of the above things are going to help you at all. You need to focus on getting MAX to think happy thoughts. 🙂

[AmilyChen]

Thanks very much for your reply and advices!!

I have followed the steps in the instructions http://www.ni.com/tutorial/55153/en/

And I cat /var/log/messages as in the attachment.

Wish you could help me to find something wrong with my system!

[rtollert]

Sorry, I'm out of easy suggestions, I'd call support.

[AmilyChen]

By monitoring the windows log on computer management of the PC installed MAX.

I found a Framework problem "System.InvalidOperationException".

Maybe this is the reason MAX crash.

I will try to repair this problem, and try again.

I'm a newer for Linux, wish me good luck!

Thank you very much for your help!