Fastest way to log IMAQ images to Disk

[manig]

Hi,

I am trying to log images to Disk in a lossless format (without missing any frames and pixel value).

Image Acquisition rate - 480MB /S (4096*4096 16-bit Grayscale 15fps). 

I tried a simple producer-consumer to prototype the image log time:

1. IMAQdx AVI - 0.5 second to log a 4096*4096 32 bit image, so this will not be suitable and will shoot up LabVIEW memory and crash it.

2. TDMS streaming (NI Example) - better than AVI but not sufficient to log 480MB/S.

I realized the disk stream rate is 150MB/S for HDD and 200 to 550MB/S for SSD drives.

Please let me know if there are any other options or if this is not feasible.

Thanks,

Mani.

[Bob_Schor]

Your math seems to show that you are generating just too much data to stream to disk.  Try the following experiment:

1. Generate a 4096x4096 I16 Array of random data.
2. Open a Binary disk file (because binary writes are the fastest).
3. Read the Clock and save as Start Time.
4. Write 100 records.
5. Read the Clock and save as Stop Time.
6. Close the file.  Compute Write Time/Record.  Compare with 67 msec (1/15 second).

Data Compression probably won't help you (unless you have a speedy co-processor to handle it).  Reducing the size of your array to 2048x2048 or 1024x1024 will give you speed gains of 4x or 16x.

Bob Schor

[Frozen]

There are speedier SSD's if that helps.

[manig]

Thanks Bob_Schor for your suggestions,

I tried profiling TDMS and Binary streaming and noticed TDMS is bit better. But couldn't achieve even 50% of SSD spec disk write speed consistently for a minute, so with High-speed SSD's also there is a memory shoot up. I have raised a request with NI to confirm if there are any limitations with the TDMS write API itself.

[manig]

Thanks Frozen, I am already using NVMe SSD to profile the write speed.

[Bob_Schor]

So I decided to take my own suggestion about I/O timing and found a very sobering thing.  My first step says to generate a 4096x4096 image of random pixels (I16).  Well, generating 4096x4096 I16 values takes less than 0.5 seconds.  If you also use this to update a 4096x4096 pixel Image (created with IMAQ Create and sized to 4096 x 4096), it takes >270 seconds!  So the Camera does some fancy 2D addressing to speed things up a lot.

I think the only way we can make meaningful suggestions is for you to post your code (all of it that are relevant to acquiring Images and writing AVIs).  If you have a sample 5-frame AVI that we can use as "typical data", that would also help, though that might make it too big for the Forum ...

Bob Schor

[GloriousLion]

---

@Bob_Schor wrote:

So I decided to take my own suggestion about I/O timing and found a very sobering thing.  My first step says to generate a 4096x4096 image of random pixels (I16).  Well, generating 4096x4096 I16 values takes less than 0.5 seconds.  If you also use this to update a 4096x4096 pixel Image (created with IMAQ Create and sized to 4096 x 4096), it takes >270 seconds!  So the Camera does some fancy 2D addressing to speed things up a lot. DQFanSurvey

I think the only way we can make meaningful suggestions is for you to post your code (all of it that are relevant to acquiring Images and writing AVIs).  If you have a sample 5-frame AVI that we can use as "typical data", that would also help, though that might make it too big for the Forum ...

Bob Schor

---

Your information is very interesting. Thank you for sharing

[manig]

Hi Bob_Schor,

Please find attached the zip file containing the time profile code. In the Zip file refer to TDMS Pipeline Log.vi. Zip file also includes similar time profile VI for Binary and AVI.

Thanks,

Mani.

[wiebe@CARYA]

How long do you need to store images? If it's a short period (or have a lot of memory) you can try to keep it all in memory and store in parallel or afterwards.

[Bob_Schor]

Thank you for attaching your code.  I haven't tried to run it, but I see a potential problem.  You are using a Producer/Consumer Design (good).  Your Producer is "clocked" by the Camera, and basically runs at the Frame Rate of 15 Hz.  However, your Consumer has (at least) two time-consuming operations competing for CPU cycles -- displaying the Images and logging them to disk.  

Actually, there are other "time-wasters" in both loops.  Instead of creating a fresh Image Buffer for every Image in the Producer (which takes time) and destroying it in the Consumer (which also takes time), why don't you create an Array of Buffers before entering the Producer loop, and simply reuse them (say you create 10 buffers -- use 0, 1, 2, 3, ..., 8, 9, 0, 1, 2 ..., a technique known as "Circular Buffering") in the Producer, and simply "abandoning" them (i.e. ignoring them) in the Consumer.  You should be doing something like this when you set up the Camera for a Grab, after all ...  Hmm, of course, you should already have allocated the buffers when you configured the Camera, just use the Camera's already-allocated Buffers ...

     Does this make sense to you?  I've got code where I was saving brief (!) AVIs from 24 WebCams, 640x480 RGB running at 30 fps (requires Gigabit Ethernet, but 9-10 cameras work fine with 100-base-T), and I always used the Camera's buffers, which are configured by IMAQdx Image Acquisition.  When I want to save an AVI, I pass the (always-incrementing-with-each-Grab) Grab's Image Buffer (which is, after all, not the Image, but an Image Reference, a.k.a. a "pointer" to the Image currently sitting in memory) to the AVI routine.

     This is rarely explained, but the "trick" to getting IMAQ to work for you is to understand Buffers.  You don't want to move them, create them, release them, you want to reuse them (create more than you need and "pass them around as you need them".

     Try that.  Create a "pool" of 10 Buffers, and recycle them.  If you find you "lose" AVIs, you might need to create a larger pool.  Note that if you have a Buffer pool of 10 buffers, the Buffer number you'll get from the Grab will not be in the range 0 .. 9, but will keep climbing as the contents get "recycled" by having the Camera overwrite them.  To check for this, take 5 buffers-worth of frames of something moving (like a stopwatch) and check if the Images suggest you are seeing all the frames.  If not, a bigger buffer pool.

     Sorry this has been a bit of "Stream of consciousness", but I think it might be the solution to your dilemma (though you will have the problem of trying to cram all those bits onto the disk ...).

Bob Schor