Best, safest, most efficient way of updating large 2D arrays in while loop

[AndrazS]

I have large application what pulls data from different hardware, let say USB devices very fast. For this implementation I'm using a while loop that has 2 shift registers initialized as empty 2D arrays when loop is started Then the 2D Arrays are updating on every iteration of a while loop. The values that are placed in arrays are read from other while loop and the loops are communicating using queues. Here there is just sketch of the idea:

The sampling rate should be bellow 1ms so I cannot control at what processor the data is processed in bellow loop and I cannot define how fast loops are.

My question is, what would be more "proffesional" way of updating/recording large data arrays? Should I use "In-place-Element" structure and DVRs?Is there any other way of implementing storing (recording) data in Labview in more memory, processing effitient way of doing that?

My idea was also to use DVRs. For every array I would have "Setter", "Getter", "Creator" and "Destructor". Creator and destructor would only create and destroy DVRs references:

When it commes to updating array values there I would create subVIs like this:

So the updating loop will update array value using subVI above instead of shift register+build array.

Is the second implementation using DVRs good implementation?

Is the "Empty Array" constant ok for initializing DVR at the beggining of the program or should I initialize array that has already its size defined at the beggining when I call Initialize DVR?

[Worle]

As always the answer depends upon exactly what you are doing.

How long is this code running for.

Are you going to be adding elements to your shift register indefinitely or for a fixed known number of samples.

Do you want to/are you going to need to limit the size of the shift registers, e.g. are they actually a circular buffer?

Does the code in the bottom loop take a long time to run or is it just putting data into the shift registers.

If the code in the bottom loop is super quick I would put it in a subVI and call that subVI in your top loop. This would block the execution of your top loop so it is probably no good if the code takes > your acquisition period to run.

If the code does take a while to run I would stick with using Queues. I would put your shift registers in a sub VI though with a case structure to allow you to either write an element or just read the entire array. Example attached

I would however try and initialise your shift registers with arrays containnig the number of samples you are expecting to deal with (maybe an array of NaN's so you know which elements are your initialisation elements). This would preallocate the memory space for your array once at the beginning. At the moment LV is having to reallocate the entire array every iteration. If this array starts getting very large this will start taking longer and longer.

[JÞB]

I'll answer the easy questions first.

• The size of the array doesn't matter to the DVR since the DVR is simply a reference to the arrays buffer.  The memory manager WILL take care of everything if the array size grows and needs to increase the memory size or move it to a new location because arrays MUST occupy contiguous memory blocks.
• Resizing an array is almost always going to hit performance.  This is especially true when arrays grow larger (see above)  and COPIES multiply the hit in performance. 

Now, the rest of the answers depend on what you are doing with the data you collect.  For concurrent processing (e.g. filtering) a channel wire might be best

For Logging to file a "chunking" write solution is desirable.  Both of those require "LOSSLESS" data transfer.  Some uses are OK with "LOSSY" data transfer. Anything displayed to the User is likely to fall into that category since User's eyeballs can only see limited data and have slow reaction times.

How will you use the collected data?

[AndrazS]

Thanks @Niatross for opiniont. Additional information for you: 

"How long is this code running for." -> it is running for about 2 minutes

"Are you going to be adding elements to your shift register indefinitely or for a fixed known number of samples." -> For a fixed number of elements

"Do you want to/are you going to need to limit the size of the shift registers, e.g. are they actually a circular buffer?" -> I'm very interesting in this "circular buffers".. So if I set FIXED number of element for this buffer, what will happen if array buffer will fill up to be full?

"Does the code in the bottom loop take a long time to run or is it just putting data into the shift registers." -> The bottom loop is much slower than upper loop. Let say it can last up to 1 ms.

"If the code does take a while to run I would stick with using Queues. I would put your shift registers in a sub VI though with a case structure to allow you to either write an element or just read the entire array. Example attached" -> This is actually quite a good idea!

[Kevin_Price]

Another approach I've used before is to throttle down my consumer and retrieve data less frequently in bigger chunks.   For example: add a wait primitive for, say, 200 msec and then flush the queue to retrieve a big chunk of data all at once.  The "no data" condition might then be your signal to terminate the consumer loop.

-Kevin P

[AndrazS]

Can you please give us an example of the idea? (VI)

[Kevin_Price]

To illustrate my idea, I modified the shipping example "Simple Queue.vi".  I commented my changes in cyan.   Compare to the original version to understand the differences.

-Kevin P