Infrequent problem with syncronization to remote time server

[Chris_12345]

We have a number of cRIO controllers that are configured to syncronize to a remote time server upon boot-up and periodically afterwards. This works fine, except for one peculiar behavior: if we use Deploy All to update the software on the controllers, then select Reboot Now?=No, then manually press the Reset button on each of the controllers, we occasionally observe a controller begin its application with an incorrect clock, off by as much as 5 hours. If we don't use the Deploy All option, and update the software by other methods (such as manually FTPing the files over), then the clock sync issue doesn't happen. This seems really strange, and we can't figure what might be going on.  Any thoughts out there?

The clock syncronization is configurated as follows:

Under NI-MAX, the Time Syncronization section is

   STNP (enabled)

   Software 1588 Time Protocol (Ptp) (enabled)

On the controller, there are two files:

/ni-rt.ini

   the [TIME SYNC] section includes the lines

   source.sntp.interval=60

   source_priority=stnp;rtc;

/ni-rt/system/ni-timesync/ni-timesync-sntp.ini

   the [TIME SYNC] section includes the line

   source.sntp.timekeeperSyncronizationThreshold=0.01

[drich]

Hello,

I am not very familiar with Deploy All but my first guess would be that the wrong plugin is coming up as the highest priority. Normally when you install/reinstall a plugin to a target you are not guaranteed of the ordering in the source_priority list. So that line in the ini file will change depending on the somewhat random order things got deployed to the target.

Is it possible that the PTP/RTC plugins are coming up at the highest priority on the system that is seeing the behavior?

To get deterministic behavior you have a couple of options:

1. Only install one TimeSync plugin at a time.

2. FTP the ini file over.

3. Go and manually configure the order in MAX.

As an aside why are you using both 1588 and SNTP? Is one target sync'ing with SNTP and then the group keeps time with 1588? In general there currently isn't good way to use both plugins on the same controller. Only the highest priority plugin will set the time, so if two plugins exist on a system then one of them usually doesn't do anything.

You can use 1588 to distribute time or as a backup for when the SNTP server goes away but in general I wouldn't run both at the same time.

--Daniel Rich

[Chris_12345]

Daniel,

Thank you for your response. All of the cRIOs sync to a remote time server on a Linux machine on our network. We originally had just SNTP, but we were having problems: if the clock on the Linux machine was updated with a correction of more than 1 second, the SNTP on the cRIOs would not make the correction. I understand that this is the way SNTP works, but its a problem for us. So, we added on the TimeSync. We''ll turn off the SNTP and see if the reliability improves.

-Chris

[drich]

The one second limitation is not inherent to SNTP and I didn't think our implementation had that limitation.

If time is further off than 1 second we should still correct the time to the correct time from the server. We will report having lost synchronization for a short time period, but we should regain it after a period of time.(Faster sync rates will shorten this period).

Which SNTP plugin were you using? There is currently a released "old" SNTP plugin that is not quite as full featured and the newer but still beta NI Labs SNTP plugin.

https://decibel.ni.com/content/projects/timesync-sntp-support

I'm am not as clear on the behavior of the older plugin, it might see this behavior but the newer plugin should not have that limitation. If you are comfortable using beta software then perhaps the newer plugin would give you better performance?

As another tidbit SNTP will generally perform better the faster you sync. If you are using the newer sntp plugin and wanted to sync every 5 seconds you would place

   source.sntp.interval=5

in 

/ni-rt/system/ni-timesync/ni-timesync-sntp.ini

The drawback of sync'ing faster is that it will increase the network traffic, whether or not that is an issue is specific to your application.

Let me know if you are still seeing an issue with not synchronizing with offsets of greater than 1 second.

[Chris_12345]

Here is the link to the documentation that refers to the out-of-range correction limitation for SNTP"

http://digital.ni.com/public.nsf/allkb/F2B057C72B537EA2862572D100646D43

"Note: SNTP synchronization does a large correction of the date-time on boot. After boot, it adjusts the processor clock to account for drift.  The algorithm has an error tolerance but if you exceed that tolerance (i.e. if you are too far out of synch) it will stop trying to adjust the processor clock to synchronize. Calling the Set Date & Time VI typically causes the clock to move beyond the algorithm error tolerance."

I have been unable to find out how to adjust the error tolerance. Are you saying that the newer (beta) TimeSync 1.2.d15 no longer has this error tolerance?

[Chris_12345]

I have tried using PTP instead of SNTP on our cRIO controllers (model 9025).

The clock syncronization is configured as follows:

Under NI-MAX, the Time Syncronization section is

   Software 1588 Time Protocol (Ptp) (enabled)

   STNP (disabled)

On the controller, there is this file:

/ni-rt/system/ni-timesync/ni-timesync-ptp.ini

   the [TIME SYNC] section includes the line

   source.ptp.address=<ip address>

On a remote machine with <ip address>, I have installed TimeSync.

Running the cRIO controller, it does not synchronize its clock.  Any suggestions about what am I doing wrong?

[drich]

Hey Chris,

Sorry I must have missed your earlier reply.

The newer version of SNTP does not have that limitation, if you get too far off it will automatically correct for you. There is no longer this concept of error tolerance. If time is changed dramatically it may take a few minutes to bring time back into a synchronized state but it will synchronize. Maybe this will solve your problem?

As a note remember if you are using the Labs version the source.sntp.interval=5 should be placed in /ni-rt/system/ni-timesync/ni-timesync-sntp.ini instead of ni-rt.ini

As far a ptp there are a few things that I want to make sure off.

1. PTP is an auto-configuring peer-to-peer protocol and is not set using the address flag like SNTP.

2. PTP only runs on LV RT targets, when you install to a host machine it puts support down so you can install to a LV RT target.

3. PTP is used internal to a subnet, which usually means all the ptp targets are co-located, with maybe a couple of network switches and/or hubs to connect them.

If all of the cRIOs that you want to synchronize are on the same subnet then PTP(1588) should configure itself and synchronize them to each other. If you want to synchronize to an external remote time server then PTP won't do that by itself. 

From my understanding of your problem I think the Labs version of SNTP would be your best bet. If I misunderstood or that doesn't help resolve your problem let me know.

--Danny Rich

[Chris_12345]

Yes, from what you describe, SNTP would be our best bet.

As an extra layer of safety to be sure that the clocks are synchronized, I put some code in my cRIO application (one of the startup self-checks) to compare a remote clock with the local clock, using Daytime Protocol:

http://en.wikipedia.org/wiki/Daytime_Protocol

[drich]

Hey Chris,

Sounds good, I am glad that things have mostly worked out. Using the daytime protocol as a backup is good although the SNTP plugin also has a property node where

you can query the status.

It will report on whether it is synchronized and how tightly it is sychronized. There is also a boolean flag that will latch to tell you if it ever loses sync even for a moment.

The property node is found in the normal menu under Programming/Structures/Timed Structures/Absolute Time/TimeSync Property Node

--Daniel Rich

[Chris_12345]

I've uncovered an unusual problem. I inserted code at the beginning of the cRIO application to continually compare a remote clock (via the Daytime Protocol) with the local clock (via the "Get Date/Time in Seconds" function) every second for a 60 second period, and write the results to a local file for later examination. It consistently takes 34 seconds for the local clock to become correct, every time that a chassis is booted, regardless of SNTP settings or which cRIO chassis we are using. The amount of time correction is consistent: on one cRIO it is a little over an hour, and on another it is about 4 hrs 45 min; these remain the same every time a chassis is booted.

However, the NI-MAX utility consistently shows a correct clock setting on each cRIO at all times, even during the time that the local "Get Date/Time in Seconds" function is reporting an incorrect value.

Any ideas on why the "Get Date/Time in Seconds" function would be reporting faulty values for the first 34 seconds of a cRIO application? We are using LV2010, no SP1.