579x MIMO synchronization problem

[jchedstrom]

Context:
I have 2 chassis. One with four 5793 transmitters and another with four 5792 receivers. For my MIMO project the transmitted and received packets must be synchronized within <100ns. According to the "Simple NI-579X Streaming" example and the "579X Synchronization Library" documentation, synchronization can be achieved within 1/2 sample clock accuracy (2 ns).

Problem Cause:
Synchronization used to work perfectly before the Labivew 2013 SP1 update. After the update the example no longer builds. The error is caused by the example violating its own design rules. The SP1 update just made the build smart enough to notice. When you dig into the synchronization vi's you find that the heart is TDC.vi which expects an oversampling clock 2x the SCTL it's inside. The example violates this by placing TDC.vi inside a 250MHz SCTL and providing a 375MHz oversampling clock (375/250 = 1.5 causing a "none integer multiple" error when trying to build). I opened a service request, NI reproduced the problem, and was told to role back my software as a temporary fix. Case closed... but to me this is unsatisfactory because the design is broken. I'll just be covering up the problem by roling back to pre-SP1. Not to mention re-exposing myself to a bunch of fixed bugs.

Attempt At Fix:
By placing the synchronization subvi inside a 125MHz SCTL and providing a 250MHz oversampling clock everything builds perfectly. There even appears to be a subvi specifically for this called "Create 579X Half Rate" (as apposed to the example's "Full Rate").

New Problem:
The syncronization only succeeds ~30% of the time and is not repeatable.

Questions:
- Am I doing something stupid? I've attached the example FPGA code and my modified FPGA code.

- What does the synchronization data out mean? I'm getting numbers varying 0-50. I know they are calibration delays from the synchronization but there's no documentation on units. Is 50 a small or big delay? (from the synchronization repeatability example, http://www.ni.com/example/31603/en/)

- If using synchronized triggers, do they need to be in the same SCTL as the synchronization alignment subvi?

Thank you for any help in advance!

[NEWKlear]

Hey JchedStrom,

I'm taking a look at this, but I am running into a lot of the same problems you are, it honestly may be best to call into the support hotline so we can work closer with you to better determine a solution.

Best Regards,

[jchedstrom]

Thanks for looking into this. Yes, I plan to call an application engineer tomorrow. I was waiting until I had more time to experiement with the "repeatability" example.

New info:

- In the repeatability example I'm seeing only two extreme values for the data_out, ~0 and ~50. By playing with the CPTR and oversample_factor values I've discovered that the larger number is always within a few samples less than =CPTR*oversample_factor which in my case is 13*4=52. From that I'm deducing that the synchronizer probably adapts by a change of 1 every loop (it does 1000 measurements by default). So it must be thinking an extra delay is needed every time on several specific FPGAs (which FPGAs is random each init). It keeps on incrementing until it hits a maximum of oversample_factor*CPTR_period. Therefore at the end of the sync some of the FPGAs have small numbers (~0) and some have large numbers (~50).

- Changing the CPTR period only effects the extreme value seen in the sync output. Symptoms remain.

- Putting the synchronizer in a loop results in the same value every iteration. Restarting the VI (reinit FPGAs) results in a new random synchronization state which stays consistent.

- I've tried forcing a synchronization state using the "repeatability" example and then measuring a square wave. Results are ~100ns variation which changes every FPGA init.