Chassis TimeSync configuration

[99111]

I was directed to post to this forum for support using the PXI-6683H timing and sync card with Veristand.  The intent of using the 6683H is to provide a trigger to initiate DMA data transfer for all FPGA targets and the RT controller.  The final design will have 7 FPGA targets but this test project has just one MXI-9159 chassis.

Timing card:  PXI-6683H  (slot 12)

Controller:  PXIe-8135RT

Chassis:  PIXe-1065

Veristand 2013

PCL running on cpu3

DPL running on cpu2

FPGA Targets:  RIO4(mxi9159)

RIO4 is first in MXI chain

Master chassis sync:  Chassis  TimeSync

Loop rate set to 4000hz

PCL timing source:  Automatic timing

FPGA target using simple bit file compiled with labview2013, 1 dma write packet

Chasiss time settings:  Generate sample clock on PXI_Trig0, PFI0.  Time reference set to free running (internal).

The problem is the HP count is steady increasing about 20 per minute.  The actual loop rate has intermittent spikes and sometimes unusual patterns.  This is true even when using a project that does not contain the timing card.  Just the presence of the card in the chassis is affecting the performance.   If I remove the card and run a test project with just a PXI-7813R the symptoms are gone, HP count is steady.

Do the versitand settings for the timing card listed above look correct?  The time reference was set to fee running as the card is not needed to provide an absolute time reference, just a trigger.

[StephenB]

Hi 99111,

I'm sorry to hear about the troubles. Could you provide your system definition file and your FPGA code for me me to look at?

Thanks,

[StephenB]

Also- what RIO version are you using? RIO 12.1 and 13.0 had a DMA jitter issue that was fixed in RIO 13.0.1

[99111]

Stephen,

     I just received notice that my reply to your email is delayed, perhaps due to attachment size.  In my reply I mentioned that the NI-RIO version is 13.0 although I just ran NI update this morning.  I will apply the 13.0.1 patch and test again.  Other software versions installed are:

NI-Sync 3.4.1

[99111]

I installed the NI-RIO patch 13.0.1 on the development system, reformatted the RT controller and installed the software.  MAX still shows NI-RIO 13.0 installed on the RT controller and 13.0.1 for the development system.  The same symptoms are present regarding the intermittent HP loop count.

[StephenB]

Hello,

Well I hoped that would be the magic bullet. You can verify the install by looking at the next for #1 on this page: http://digital.ni.com/public.nsf/allkb/95CF25A09DB91FEE86257BE8006193B7?OpenDocument

The next questions would be to check these items:

1. In MAX, go to network settings, for each adapter go to advanced and change the mode to polling instead of interrupt
2. In BIOS (hit delete while controller is booting), make sure legacy USB support is off.
3. In BIOS, make sure hyperthreading is off
4. In BIOS, if you can, turn on turbo boost and C states. I don't remember if it allows you to though. if not, thats fine.

Also you should get the best performance by assigning the primary control loop to core 0.

After that if you still get the issue, you can try disabling monitor output by editing the ni-rt.ini on your target. Easiest way to do that is to use a web browser and navigate to the IP address of your target, then click on the file system button on the left, and then double click ni-rt.ini.

1. Set RTCPULoadMonitoringEnabled to FALSE in the [lvrt] section
2. Set RTEthernetLiveDisplayEnabled to FALSE in the [lvrt] section

If that doesn't work, can you screen shot a graph of the "hp loop duration" channel?

You can also try duing some real time execution traces so we can narrow down what is causing the jitter. These don't typically have too long of a buffer, so I always use a NIVS alarm to toggle the trace so I make sure the jitter blip is inside the trace. Make an alarm on your late count channel that fires a procedure that turns off tracing and then turns it back on. That way when a blip happens the trace is stopped and therefore the trace is dumped to disk. You will have to retrieve the log from the target. It is in c:\ni-rt\veristand\execution traces or something like that.

[99111]

Stephen,
Thanks for the suggestions.  I’m not sure why MAX does not list 13.0.1 for NI-RIO on the RT controller but I diffed /ni-rt/system/niriomtk.dll with one located in “NI-RIO Mite\13.0.1\Pharlap” and they are the same.

I changed all the Ethernet adaptor settings from interrupt to polling (1ms).  For the BIOS settings, Legacy USB support was disabled as was hyperthreading.  Turbo boost was enabled and C-states was disabled and cannot be modified.

I moved the PCL to core 0 and DPL to core 1.

For the ni-rt.ini file I set CPU load monitoring to false.  There was no entry for Ethernet Live Display so I added it and set it to false.  The attached plot of HP Loop duration was taken while tracing was restarted after an HP count.  The spike was due to RT tracing being turned off/on as that pattern is not seen when there is no tracing and HP counts are occurring.

With the above changes, I still see the same symptoms and the HP count increasing.  This test was done using the 6683H as the chassis master sync device and one MXI9159 FPGA target.

If I remove the 6683H and use a PXI-7813R as the master sync device, there are no HP counts.  However, just having the 6683H in the chassis will cause spikes in the Actual Loop Rate even if it is not part of the Veristand project.  I discovered that if I move the 6683H to PXI trigger bus segment 1, it is then possible to deploy the same project with just the PXI7813R and there are no HP counts.  The plots for HP loop duration and actual loop rate don’t look as good as when the 6683H is out of the chassis, but at least there are no HP counts in this test configuration.  It appears that he 6683H is having a problem with using PXI_Trig0 over a trigger bus buffer bridge.  When it was in segment 2, I did set the chassis trigger settings to route away from bus 2 and was able to have it be the chassis master sync device with a 7813R in bus 1, but it still had intermittent HP counts.

I left the 6683H in segment 1 and removed the PXI7813R from the project and added just one MXI-9159 using the 6683H as the chassis master sync device.  The intermittent HP counts unfortunately still continued in this configuration.  With just one MXI9159, there were roughly 3 HP counts in a five minute period.  With three MXI-9159’s, there were over 2000 HP counts in a 60 minute period.

Just to explain why this timing card was purchased – the previous configuration used a PXI7813R as the chassis master sync device and a DO channel to send a trigger to the MXI RIO FPGA chassis.  That configuration works in the sense that there are no HP counts and the actual loop rate is relatively good (with VS2013) as long as there are just a few FPGA targets.   But the problem is that every FPGA target costs about 30usec in HP loop duration and for the MIX RIO chassis there is an additional 7 usec for each depth in the MXI chain.  The latencies add up and there is not enough time in a 250usec frame for five MIX RIO chassis and two PXI-7813R’s.   The backup plan was to have just four MXI RIO’s with one RT controller and use a 6683H as the chassis master sync device.  The expectation was that using the 6683H would save 30usec from having to use a PXI-7813R and also would be a more precise timing source.

Brian

[StephenB]

Hi 99111,

Thank for the detail on this. Much appreciated

We had a conference call here aobut your issue yesterday and decided having multiple forms of communication with you is a little confusing for you and for us. Aldo will be taking this over entirely and escalating the severity of this issue. He is first setting up a replica system so we can debug it locally and therefore place less of a burden on you. I will be assisting his diagnosis and have communicated this information to him.

Thank you for your patience.