Possible bug in AVI2 Create/Write functions

[Bob_Schor]

We've uncovered a possible bug in the AVI2 Create File and AVI2 Write Frame functions.  We are running a "monitoring" program, written with LabVIEW 2012 SP1 (32-bit) and running on a Windows 7 (64-bit) PC.  The camera acquires images at 30 frames/second.  We have a parallel process that runs at 10 Hz (or 10 frames/second) -- each time this "ticks", we save the current camera buffer in a 25-element lossy queue (this gives us access to the preceding 2.5 seconds of images).  When we notice an "event of interest", we do the following:

1. Create a new AVI file with a unique name.
2. Grab the 25 images in the pre-event Queue and write them to the file.
3. Start saving current images (also buffered in a Queue, so we don't miss any), until we get an "Event End" signal.
4. Close the AVI file and wait for the next Event.

We have 24 (Ethernet) cameras, and have written the code to enable us to acquire data from all 24 cameras simultaneously.  Each camera "sees" Events every minute or so (sometimes longer), and the acquired videos are typically 5 seconds in length.

Recently, we've been recording using 15 cameras, and have noticed that over a two hour recording session, 3 or 4 stations will throw errors (which we trap and use to turn off that station).  The two errors that were most common were -1074395969, Error creating AVI file and -1074396005, Invalid AVI session.  We put some additional code in to figure out which function caused the errors, and were able to show that a common place was between Steps 1 and 2 above, i.e. after the AVI2 Create File had been called (without generating an error), with the error generated after the subsequent call to AVI2 Write Frame.

This made no sense to us, and sounded like a "bug".  We called NI Support, spoke to a very helpful AE, who reported only two cases of the 5969 error, one of which occurred when the user used the Greek letter "mu" as part of the filename.

That got me thinking.  What if the error is due to a "mis-communication" between NI and Windows, specifically what if, on return from Create File, the file really isn't "ready for writing", and there's something of a race condition between Windows "getting ready to write" and the arrival of the first Write Frame function.

Our code, besides writing AVI files from 15 cameras asynchronously, but "simultaneously", was also writing 15 text files associated with each recording session.  Furthermore, each of the 15 cameras and 15 text files lives in a separately-created folder, which also needs to be created when the first file is opened.  In addition to this, while we don't use Greek letters in our file names (though we do use underscores), the file and folder names can get quite long.

On a hunch, I recoded the file creation/naming code to simplify file creation.  In particular:

• All of the files for a single recording session go into the same folder, which is created before any of the files are written.
• In particular, the text files and AVI files "share" this folder (the extension tells them apart).
• The folder name (including the Drive letter and all the back-slashes) is less than 30 characters.
• The file names (not counting the three-letter extensions) is 7 to 11 characters.

I've tested this new code under "dummy" conditions (we didn't have "real events" -- they were being created by "noise" conditions in field).  However, we would typically see the "old" code fail at least one station within a minute, and with the new code, we ran several 3 minute tests without failure.  We plan a two-hour test, under real conditions, tomorrow morning, but I'm optimistic that we've "programmed around" this bug.

[Eric-J]

Hi Bob,

Thank you for bringing this to our attention. It looks like you are currently have a service request with Jason on this issue. I just spoke with him and he would like you to send in the smallest portion of your code that can reproduce the issue.

[NacNuds]

Error -1074395969 occurred at IMAQ AVI2 Write Frame  Possible reason(s):  IMAQ Vision:  (Hex 0xBFF604BF) Error creating AVI file.

I am using also multiple cameras and saving multiple avi's and experience the same thing. I already placed the AVI2 Write Frame in sub vi but as a workaround I changed the properties of this sub vi.

The error occurs when the funcion has the properties:

File – VI Properties – Execution – Reentrancy – Preallocated clone reentrant execution – On

The error does not occur when the function has the properties:

File – VI Properties – Execution – Reentrancy – Non-reentrant execution – On

The function “IMAQ AVI2 Write Frame” creates this error when it is called simultaneous.

The “AVI Refnum” acts normal and remains the same.

I am wondering of there is a CAR# or another solution.

NacNuds

[Bob_Schor]

---

@NacNuds wrote:

Error -1074395969 occurred at IMAQ AVI2 Write Frame  Possible reason(s):  IMAQ Vision:  (Hex 0xBFF604BF) Error creating AVI file.

I am using also multiple cameras and saving multiple avi's and experience the same thing. I already placed the AVI2 Write Frame in sub vi but as a workaround I changed the properties of this sub vi.

The error occurs when the funcion has the properties:

File – VI Properties – Execution – Reentrancy – Preallocated clone reentrant execution – On

The error does not occur when the function has the properties:

File – VI Properties – Execution – Reentrancy – Non-reentrant execution – On

The function “IMAQ AVI2 Write Frame” creates this error when it is called simultaneous.

The “AVI Refnum” acts normal and remains the same.

I am wondering of there is a CAR# or another solution.

NacNuds

---

Very interesting.  If you look at the properties of IMAQ AVI2 Write Frame itself, it is "Shared clone reentrant execution".  To me, this means that (a) there is a single copy of this VI in memory, (b) the VI doesn't keep data internal to itself (so it can be called by many other VIs independently), and (c) the VI can, in principle, call itself (not sure if this last is relevant here).

I was going to point out that changing your sub-VI properties from Preallocated clone reentrant to Non-reentrant would mean that all 15 cameras had to "share" the call to Write Frame, which could potentially slow down the transfer of data (because only one call to Write Frame could take place at a time).  However, if Write Frame is using a Shared Clone reentrancy, then there's still only one of these in memory, and the same "enforced" one-at-a-time execution would take place.

We're currently using LabVIEW 2012 for this project.  It would be interesting to see how LabVIEW 2013 (and 2014) handle this.  Maybe I should read up a bit more on the Shared vs Preallocated Clone Reentrancy issue ...

Bob Schor

[NacNuds]

Hi Bob,

I totally gree with your arguments.

Indead I wanted the AVI2 Write Frame to be executed parallel. Therefore the "Preallocated clone reentrant execution" options checked. But I got problems one 1 computer at one moment. Other times it worked file. Other computers without problems.

Then I checked "Non-reentrant execution" in my sub vi and it solved my problem. I don't know if you still have the problem and have a sub vi also. OR else enable this option or enable it in the vi "AVI2 Write Frame" to solve it.

So my guess it is a problem with the dll in AVI2 Write Frame.

I am using LV2013 SP1 64bit. IMAQdx 4.3.5 (2014.02). My code is way to complex to isolate the problem easily. So not going to do that, but it would be nice to see if the problem occurs with a simple example.

You're also mentioning that you don't want to slow things down by using multiple AVI2 Write Frame.

But my guess that the storing is not going to be faster if you are writing it parallel. Try to copy two large files in Windows parallel. My exeperience is that this takes longer than one by one after each other. Maybe this is different with a SSD harddisk. Maybe you have testet this. I haven't. Maybee in the future, if so I will send the results.

This can be a nice tip for you:

I have changed the number of buffers with the function "IMAQdx Configure Acquisition" from 3 to 100 (increasing memory allocation, see taskmanager in Windows). This way I have less missed frames. LabVIEW 32-bit can address a 2 or 3 GB, 64-bit can use more RAM. Because of this I also started using Windows 7 instead of XP.

NacNuds

[Bob_Schor]

Here's what we do, and how we've set it up (and, so far, it seems to be working for us):

We are doing behavioral experiments, and need to capture on video the behaviors, which occur maybe every few minutes and last for less than 10 seconds.  We want the video to start 2.5 seconds before the event we want to capture.

For each station, we monitor (at 10 times/second) for events.  Our camera (an Axis M1031, communicating via TCP/IP) is taking images at either 10 or 30 frames/seconds (I've forgotten the details).  We allocate a buffer sufficient to capture all of the pre-Event images (and then some), saving the buffer numbers in a Lossy Queue that serves as our "pre-Event memory".  When an Event occurs, we start saving buffers to a new AVI file, saving the pre-Event buffers first, then the "per-Event" buffers until we determine the Event has finished (maybe an additional 30-70 buffers), and then close the AVI file.

The IMAQ Write AVI function is running in its "default" mode, Shared Clone Reentrant Execution.  My VIs, in which this is embedded (several "layers" down from the upper-most level VI) are mostly running as "Pre-allocated Clone Reentrant Execution".  The highest level VI, which I call CLONE Camera, is running as a detached "Clone" VI using Start Asynchronous Call.

When we run an experiment, we typically run 24 stations at once -- that means 24 cameras, 24 Event Detectors, observing 24 subjects all working independently, with the experiment lasting about 2 hours.  We run on a reasonably powerful PC, an at-least-5-years-old Dell workstation with dual Xeon processors running Windows 7 x64.  We run LabVIEW 2012, 32 bit (because 64-bit has the reputation of being a bit flakey).  As near as I can determine, we are successfully able to capture all of our events.  Note that for one experiment, we get 24 sets of videos (each in its own folder), with each set consisting of a few tens of short videos, each showing the Subject starting 2.5 seconds before the Event (also visible on the video) and ending a few seconds later.

So at least in our hands, the Execution Mode aspects of the AVI Write function doesn't seem to be a problem.

Bob Schor

[Bob_Schor]

I forgot to mention an important detail.  We developed (and tested) the code in LabVIEW Development Mode.  We then build an Executable, and that also runs just fine (and is the "usual" mode we use to collect data).

BS