Scan engine Slow, throws error -65512

[Etarkleson]

My scan engine occasionally throws error -65512 regardless of the speed I am running it at. The scan engine simply stops and only through DSM can I see the error and reset it while running my program.

I am running a CRIO 9081 + 9411 with 136 Channels on Scan Engine. The scan engine will occasionally throw the error as low as 10hz, but both cores of my CPU are consistently around 20% with no visible spikes. My memory is less than 5%.

I also noticed while troubleshooting this issue that when the scan engine stops working several loops take a hit in execution time. My timed loop actually freezes, meaning it stops executing but doesn't finish, so that even if I properly shut down the program it hangs on the timed loop but nothing ever happens.

I don't have any intermittent processes that would take up enough resources to cause this behavior. I have read that even with my channel count (136) I should be able to have the scan engine run up to at least 100hz, which is my goal for this project. I would happily take 50hz, but 10 is too slow...

Has anyone ever run into an issue like this? Any advice on how to proceed? If the proccessor isnt busy shouldn't I be able to ramp up the scan engine, or are there other bottlenecks to scan engine speed? 

Also what does this do? Could it effect scan engine performance?

Thanks,

Eric Tarkleson

[DECONN]

Hi Eric,

Researching this topic, I noticed that you are already working with an Applications Engineer on this. I will update this post if a solution is found to help others that may come across this issue. In the meantime, perhaps another person on the forums has experienced the same issue and can add to the conversation as well.

[Etarkleson]

Hello,

So we never found a complete solution for this but I will post my current progress.

Sometimes during startup cpu usage will max out momentarily as all VIs are loaded. The scan engine by itself can rail the CPU even with nothing else happening, so together it may not be uncommon to throw this error during the startup phases. Since this is now an expected error it needs to be dealt with dynamically.

However, the scan engine needs to be monitored via the built in vis to trap and deal with errors during startup and afterwards. Without using these VIs and properly trapping the scan engine errors the program will continue running without updating scan engine values.

The RT Execution Trace Toolkit is a good tool to check execution timing and CPU usage in situations like this to see what processes are holding up other processes.

Adding delays during the startup portion of your code can help to control CPU overloading. I even added time shifted delays before startup of my main loops to insure the initialization of one doesn't block resources needed by other loops.

The scan engine will sometimes hiccup if a timed loop doesnt complete on time, even though there is no scan engine control happening inside. This probably happens because the timed loop is taking up proccessing time when the scan engine needs it.

Hopefully this is helpful to someone in the future.

Eric

[Etarkleson]

Ok, So after what I posted yesterday, today the problem came back full force.

I recompiled all the code, wiped out and reinstalled the CRIO and the 9144.

I then commented out all the code in my main rt vi except for my communications (so I can run the code, and shut it down)

My scan engine is set at 25ms.

When I run this code I still get the -65512 error and it happens every minute or so and throws the error about 50 times in quick succession.

I am at a loss for what to look at next. It seems it cant be my code, but the NI support has little advice.

Any ideas at all?

[Jesse_D]

Hi Etarkleson, 

I believe AE is reproducing your system right now. R&D will work with them to reproduce and explain the behavior.

However, as an immediate work-around, you can set the 65512 fault to be a warning in the Distributed System Manager and it won't shut down your entire program if the fault is thrown. Please note the root cause of this fault though. Some other thread is keeping the scan engine from executing every 25ms. If you set the 65512 to be a warning your outputs might not be updated every 25ms. 

[volker_sbk]

Hi,

Has there been any update on this? 

I have seen the same error being thrown in my application and it seems to be related to the time synchronization. I am using a cRIO9076 controller that requires an externally provided time stamp in order to accurately maintain time. Synchronizing with an external SNTP time server every 60 seconds will eventually result in error -65512 to be thrown in quick succession. This leads to some peculiar behavior in my application. Once I remove the time synchronization section from the ni-rt.ini file, this all goes away.

So if there has been any resolution to this, please share.

All the best,

Volker 

[Etarkleson]

I was given a driver update that is now available online. I believe this is the link.

Also I uninstalled any addons that are not absolutely needed, anything dealing with USB can be non deterministic.

To be perfectly honest I think there are some  underlying issues still that hopefully will be resolved by NI soon. I still occasionally get different scan engine errors as well as other strange behavior (VIs becoming corrupted, builds simply not working, etc. I have simply have to deal with this behavior. Its kinda dissapointing considering the price and power of this hardware. That being said I am overall happy with the capabilities.

Can you give me more information about removing the time synchronization from the init file? You said this solves your problem? Is it an acceptable solution or does it lead to less determinism?

Also I kudoed jesses post because that is the quickest workaround to remove errors and avoid your program halting and causing possibly dangerous effects. The side effect is an occasional large jitter, but it has never been anything that has skewed my data too much. Its good enough...

Eric

[volker_sbk]

Eric,

thanks for the update. The cRIO9076 does not have a battery-backed real time clock. As a result, a valid time stamp must be acquired externally. This can be done throgh the cRIO's OS by connecting to a NTP server and getting a time stamp from there. In order to enable the time synchronization, the following paragraph needs to be included into the cRIO's ni-rt.ini file  

[TIME SYNC]

source_priority="sntp"

source.sntp.enable="TRUE"

source.sntp.address="129.6.15.29"

source.sntp.interval=600

source.sntp.port=123

source.sntp.verbose=false

source.sntp.log=true

When I am using this approach, the cRIO scan engine starts to continuously throw fault -65512 after a while. Once this starts, the IO variables tied to my digital output module channels (NI9375) are not updated correctly. I am writing to them and the status changes shortly before turning back to a default or previous state. Once I am disabling the time synchronization through the OS, this goes away and the IO variables are updated properly. 

Unfortunately, this workaround is not an acceptable solution. I need a valid time stamp. I don't think that the lack of synchronization leads to less determinism. But I do wonder whether there is some jitter introduced when you do sync the time.

not sure where to go with this. NI is looking into the issue and I am trying to incorporate NTP time synch through LV code rather than using the OS time synch functionality. NI seems to think that using LV code for this task, especially when executed at a lower priority, will not interfere with the scan engine as much as the high priority OS implementation.

This remains to be seen.

Volker  

[zahnrad]

Hi Volker,

i'm using almost the same setup (cRIO-9076 with NI9213 in Scan Mode and NI 9234 in FPGA Mode = Hybrid Mode).

The Controller gets the actual time via ntp-sync (as you wrote).

My Problem is the following: the I/O Variables of the NI9213 (Scaning Engine) don't serve the variable activated timestamp.

Is this caused by the ntp synchronisation of the cRIO-9076?

Thx

Robert 

[volker_sbk]

Robert, see attached png. We used to get the timestamp through a scanned variable read of an NI9375.

Hope this helps.

Volker