Tags fail to recover from network interruption

[MichaelBalzer]

I've run into an issue where if a network connection is interrupted between a cRIO client and a SystemLink server, tags being written to from the cRIO take a long time to update again, or seemingly stop working completely. Steps to reproduce:

1. From the SystemLink server interface, ensure the cRIO is being managed by the server and is connected.
2. Build and deploy the attached VI to a cRIO target, and then reboot the target.
3. Wait a minute or two for the cRIO to reboot and begin updating the tag. Verify the tag is being updated in the SystemLink tag viewer.
4. Disconnect the cRIO from the network for a few minutes (I just removed the network cable). Verify the tag has stopped updating in the tag viewer.
5. Reconnect the cRIO to the network and wait a minute or two. Verify the cRIO visible and connected to the SystemLink server through the systems manager.
6. The tag in the tag viewer will take a long time to begin updating again, or is more common, will never begin updating again.

The code attempts to reopen the tag for each write, which I would have expected would force it to recover from any network interruption. Is there something I'm missing, or is this a known issue?

[MichaelBalzer]

I've just noticed that the longer the cRIO is disconnected from the network, the longer it takes to recover. For example I had just disconnected the cRIO for about 3 minutes, and it took close to 30 minutes to recover. If I disconnect it for 15 seconds however, it recovers almost straight away.

Edit: It looks like the tag updates have died again, though the cRIO was connected the whole time since the 3 minute disconnect and 30 minute recovery. The tags were only up again for maybe 5 minutes.

Edit 2: Also worth noting that one of the CPU cores is stuck at 100% usage (on a cRIO-9067) all of the time, regardless of connection or tag state. If the Skyline VIs are removed, CPU usage is almost 0%.

[GreyGrey · Accepted solution]

Can you try using HTTP instead of using AMQP? This can be done by dropping an Open Configuration down before the tag write and connecting it to the tag writing VIs. It should also be available from the Skyline palette under "Configuration". AMQP is used by default.

[JoshuaP]

I think 17.5 will help address some of these issues.  I recreated your application and deployed it to an RT system.  I shutdown my server for 30 minutes and when I powered it back up my target connected in less than 20 seconds.

The high CPU usage is a known issue when using AMQP.  We have a background thread that is constantly trying to check for new messages that is running faster than it should, however it seems to yield if there are other things going on.  In my cases it just looks worse than it actually is, but regardless we are working on a fix.

[MichaelBalzer]

The HTTP config seems to have done the trick, both in terms of tag recovery and low CPU usage.

As a general rule, when would it make sense to use AMQP or HTTP as the config type?

[JoshuaP]

The nice thing about AMQP is it uses the machine's credentials that are automatically created when you add the target to the server.  HTTP requires you to hard code your username and password into your application, which makes it a lot harder to revoke and update in the future.

In general, we recommend AMQP for "managed" systems where you want to control where they data flows by registering the system to the server.  It's great for headless systems like RT.

We recommend HTTP for client applications being used by a user.  Like an application running on your laptop (most likely not managed by the server) that you may want to pull data from multiple servers and use the users credentials.

In addition, we only support asynchronous operations over AMQP, which is exposed in the Message and File APIs.

[MichaelBalzer]

Thanks for the great comparison Joshua.

[JoshuaP]

We have a fix for the high CPU usage when connected over AMQP and it will be included in the 17.5 release next week as well.