DVR based Ring / Circular Buffer Library

[User002]

I have been searcing with a light and candle for a decent ring buffer implementation.

However, I did not have much luck googling around, so I decided to write my own buffering.

This example sources also exist as a continously evolving example in the LabQT LGPL Open Source Library.

Currently it handles 1d double array, 2d double array and 1d waveforms array.

Feel free to extend it with more data types and contribute the changes.

Regards,

/Roger

Edit 1: Updated to Verison 2 of the ring buffer.

Edit 2: Removed the old version of the Ring Buffer

Edit 3: Optimized memory usage in Version 3 of the library, thanks iikkamat!

[Christiana]

Very slick code.  I am going to try to use it on my real time system and stream data to a network hard drive.

[User002]

You are welcome, and good luck with the NAS DAQ application!

[Christiana]

Anyone trying to use it on a real time system will need to change the build specification properties to not include unused polymorphic instances and unused vis within the library.  Thanks for the code.

[iikkamat]

Very nice code!

I think, however, that we can improve the memory footprint by modifying the Init VI a little. After reading

this document http://digital.ni.com/public.nsf/allkb/A8BA755EB2A699FA86256FDC00691FFD

and this discussion http://forums.ni.com/t5/LabVIEW/Array-functional-global-uses-2x-the-initialized-space/td-p/923292

I was concerned about whether LabVIEW allocates the memory 1x or 2x when we call the Init VI.

I uploaded a little test project (https://decibel.ni.com/content/docs/DOC-48358), and indeed it seems that with the Reshape Array function the memory is allocated once, whereas with the Initialize Array function it's allocated twice. This is understandable as explained in the discussions above.

I added "RingBuffer.lvlib:Init-3.vi" and "RingBufferTest.vi" in the project and I modified the "RingBuffer.lvlib:Double2DTest.vi" a little (hard coded to allocate 100MB buffer and stop after 30 iterations).

I tested the behaviour on Windows 7 (LabVIEW 2015, 15.0.1f1 32-bit) and on a PXI-8108 (Pharlap ETS, LabVIEW Real-Time 2015). I built an executable for each platform and observed the memory consumption in Windows Task Manager and NI Distributed System Manager.

RESULT: With the original "RingBuffer.lvlib:Double2DInit.vi" we end up allocating 2x memory and with my added "RingBuffer.lvlib:Init-3.vi" we get the nice 1x allocation.

Do you see similar results?

[User002]

Thanks for the patch/update/optimization suggestion.

Though, I didn't have LV2015 on this computer, anyway I went on and made a few assumptions on how init should be reworked. Correct me if I'm wrong.

Attached a picture of init below:

Btw, I think I fixed a bug in the init as well.

[iikkamat]

Yes, that's basically what I also did.

[Mads]

Testing the memory-saving effect of using reshape instead of init I first could not find a difference - until I figured out that we had added a "flush" function to our copy of the library - which used a for-loop with an in-place structure to reset all the values to 0. Having learned about the magic of the reshape function (not very inuitive I must say, they should make the init primitive smart enough to do the same behind the scenes) I replaced that function with a reshape to 0-size, then back to original size to see if that would do the reset to 0 without using extra memory. Voila...back to just one copy. Strange stuff

On a side note one feature I always miss in the different circular buffer implementations I find is the ability to extract a section from anywhere in the buffer, not just all of n-last values. We have our own VI-clone based version that covers that functionality, but we do use this DVR solution in some applications where speed and efficiency is cruical and such functionality is not. Thanks for the great work.

[User002]

You are welcome guys, it's always nice to know this stuff is being used and improved upon.

High-level languages can indeed be a bit "obscure" and if we want to squeeze out the last bit of performance and memory, it certainly takes some forum reading and experimentation. Code readability and maintainability might take a hit as well.

Personally, I'd like NI to put memory allocation and performance data in another layer/dimension of the BD UI. That way it would be much easier to visually identify and correlate memory allocation buffers, memory leaks, reference leaks and performance hot-spots.

Also, I'm looking forward for some .NET/C++ style libraries (e.g with referenced ring buffers, state machines, dictionaries, etc..) to be included natively in LV. I think LV in its current shape is a far too rudimentary general puropse programming language.

I just finished watching a Youtube clip of a C# / Unity 3D game engine coder slap together a simple tower defense game in about 1h. Powerful 3D engines, physics simulators and VR are hitting mainstream developers and it is amazing to follow the speedy advancement of contemporary programming technology.

And here we are with LV2015 lumbering along with clunky, expensive, yet fairly simplistic programming tools/data structures/algorithms and user interfaces.

Anyway, I digress.

Roger