FIFO overflow error using the NI 1429 frame grabber

[ralfseidelXXX]

I am using a NI1429 frame grabber with the new Mikrtron Eosens Full camera link camera (MC1362). There is no camera file yet released, but as suggested by the manufacturer the camera file for the previous version of the camera MC1310 works with it. However, after starting aquisition I get random "FIFO overflow" errors sometimes after few frames somtimes after thousends of frames. I get this error non-depending how I aquire my images in Labview and also in MAX. The system speed is not really limiting since the error occurs also at 60 Hz aquisition frequency, however less frequent than at maximum speed of 500 Hz. I can force to get the error be quickly changing between different windows on the screen. Also the error does not seem to simply correlate with image size there are more and less stable image sizes independent of there total number of pixels. I looked up the other answers to FIFO overflow errors, but none of them was too helpful in my case. Maybe someone has a good suggestion for me?

Thanks,

Ralf

[Elmar_NI]

Hello Ralf,

here you can find more information about the overflow error.

Why Do I Get A FIFO Error From My PCIe-1429 When Doing A Grab?

Problem 

When I do an grab, ring, or sequence with my PCIe-1429, I get Error -1074397140: FIFO overflow caused acquisition to halt. Why is this error occurring, and how can I avoid it?

Solution 

This error can have two causes. The first is due to the system RAM being insufficiently slow or scarce, the second is due to the northbridge on the motherboard.

RAM Cause

It is possible for a grab to run out of memory, in which case the framegrabber is trying to push data through the bus, and the RAM just can't fill fast enough to keep up. If this is the case, the FIFO error is being caused by the system RAM, and can typically be solved by upgrading the RAM speed or adding more RAM to the system.

One symptom of this is that the error occurs more frequently for large image grabs or fast framerates. Another symptom is that the error occurs more frequently when more DMA accesses are being processed during the acquisition.

Northbridge Cause

Theoretically, PCI express is supposed to have dedicated bandwidth. This means that if you have two PCI express cards in your computer, the bus traffic on one will not affect the bandwidth on the other. However, some motherboards (and more specifically the northbridge on the motherboard) do not handle PCI express protocol as well as others. The result is that during times of intense loading of the northbridge, the available bandwidth of the PCI express bus will decrease. If it decreases below the transfer rate of the framegrabber (the data inflow from the camera is greater than the outflow through the PCI express bus), the FIFO memory on the framegrabber will start to fill up. Because the onboard memory on the PCIe-1429 is very small, it quickly fills up and the error occurs. If this is the cause, a new motherboard will have to be procured or a different computer used. Alternatively, decreasing the framerate or image size also reduces the likelihood of the northbridge reducing the bandwidth, thus avoiding the error.

Another possibility is that the PCI express slot is not a full slot. For example, some video card slots have 16 channels going to the device (from the northbridge to the PCIe card) but 1 channel coming from the device. The result is a x16 slot that the PCIe card fits into, but can only capture images at a x1 speed.

One symptom of the motherboard being the cause is that the error occurs on one computer but not another. Another symptom is that the error occurs more frequently with larger images and higher framerates.

Note: These causes could exist in any PCI or PCI express card. The likelihood of it happening is low however because most other framegrabbers don't transmit enough data to overload the northbridge.

Kind regards,

Elmar

[ralfseidelXXX]

Hello Elmar,

thanks for your answer. I couldn' acces the links in your answer, but I guess these would give me just the text I could read anyway?
I am now convinced that your answer points into the right direction although I am still puzzeled what the potential bottle neck in our system could be. The computer we are using is only six month old. It's a Dell Precision 670 workstation, which is in principle designed to handle large and fast data streams. System specs are 1333 MHz FSB, two Quad Core Xeon E5345 2.33GHz processors, 4096 MB 667 QuadCh. DDR2-SDRAM (4x1024MB). From my naive point of view it should be fast enough to handle the incoming data. It has three 4x PCIe slots (wired as 8x). We tried all of them since the IRQ might manage the three slots differently, but still got the error.
Is there somehow a possibility to check whether RAM or Northbridge is limiting? Is there a possibility to monitore the 1429 onboard memory (how large is it actually?).
Just a few symptomes how I can force to get the FIFO error: (i) Displaying all recorded frames (1280x1024 pixels at 500 Hz) (PCI bus problem?). (ii)  Switching between windows on the screen or dragging the the aquisition window quickly forth and back over the screen (PCI bus problem?). (iii)  Doing additional frame analysis, e.g. ImaqImageToArray causes the error after a short while even without displaying frames (RAM problem?).
Genarally my 4x PCIe slots are fast enough since the computer can aquire the full frames at 500 Hz for a long while without skipping frames if you take care of (i) to (iii). I also do not have the other two 4xPCIe slots equipped. So there should not be any fast PCI traffic accept the graphics card in a 16x PCIe slot.
It also looks that running the camera with 60 Hz only on 8 taps x 8 bit is more stable than 10 taps x 8 bit (why can the 1429 run at 10 taps at all?). At this aquisition frequency the total amount of data is rather low. Thus, the error must occur when the actual frame is transfered which speed should be given by the number of taps used?
It's all a bit odd.

Thanks,

Ralf

[Elmar_NI]

Hello Ralf,

could you post your MAX protocol and a print screen of the error message.
Max protocol: Datei/Protokoll erstellen and always weiter (use the defaulft settings).
Because i will contact the developer in Austin for that issue.
I also found a forums post with the fifo error and the solution was a simple driver update.

http://forums.ni.com/ni/board/message?board.id=200&message.id=13026&requireLogin=False

You can find the latest dirver under this link:

http://joule.ni.com/nidu/cds/view/p/id/960/lang/de

Kind regards,

Elmar

[ralfseidelXXX]

Hi Elmar,

we had a similar idea and lready updated this morning to the latest version of NI-IMAQ recommended by you. However, the error still persists.
I attached you the MAX protocoll and the screenshot. Aquisition was done at 432 Hz and 10 taps at which the error came pretty much instant. Recording at the same speed with 8 taps is much more stable.

Thanks!

Ralf

[Elmar_NI]

Hello Ralf,

thanks for the fast feedback.
I think i know why aou get the error message.
FIFO Overflow error indicates that the data from the frame grabber card was transfered to slow to the RAM memory of the pc.
This happends if the PCIExpress Interface of the motherboard use a less number of lanes.
A customer get the same error with the 1430 device.

http://forums.ni.com/ni/board/message?board.id=200&message.id=11024&requireLogin=False

Choosing the Right PC for the NI PCIe-1429

http://digital.ni.com/public.nsf/websearch/1FD6E5B5A08C1E0786256F9B0079D449?OpenDocument

Are you able to test it on another PC with a different motherboard?

Kind regards,

Elmar

[ralfseidelXXX]

Hi Elmar,

thanks for your suggestions and the link to "How to choose the right PC for the 1429".
However, I don't think that the problem is as simple as this. According to the site the recommended mother boards have just been tested using MAX, if they recognize the NI PCIe 1429 frame grabber and if the frame grabber can negotiate the 4 lanes it needs (MAX: My System » Devices and Interfaces » NI-IMAQ Devices » NI PCIe-1429 under Negotiated PCIe Lane Width) but not if the desired bandwidth is achieved. If I do the same with my Dell precision 690 workstation (Intel 5000X chip set, which is probably to new to be in the list), MAX tells me that it succesfully negociated the required 4 lanes. It is not the overall bandwidth which causes my problem since I can aquire at high frame rates with about 500 MB/s. It is that only occasionally I get the FIFO overflow. I will try a testImage simulation, to see whether this problem still persists and also try it on a different computer.

Best,

Ralf

[ralfseidelXXX]

Just a quick update.
I was running test image aquisitions for 1280x1024 frames for which I can get the FIFO overflow too. This is dependent on the testimagelineblanking. Setting it to 100 allows stable aquisition at 330 Hz. Smaller line blankings give higher frame rates but provide occasionally the FIFO error. This is different to a real aquisition with the camera. Here I can even get the FIFO error at 60 Hz although after many frames. This looks to me that there is somewhere a bottle neck even in handling a single frame. The line blanking decreases the transmission rate for a single frame whereas the camera is probably running at its own fixed speed.
This problem has been previously discussed (http://forums.ni.com/ni/board/message?board.id=200&thread.id=8622&view=by_date_ascending&page=3) and the solution was an increased line blanking, which does not work for a real camera. However, it remains unclear what the bottle neck consists of.

Best,

Ralf

[ralfseidelXXX]

Hello Elmar,

we have now tested two othe computers all from Dell.
We do not get FIFO errors on one of them, which is a Precision T5400 workstation with the Intel 5400 chipset.
It runs on the 64 bit version of Windows XP and has two Xeon E5410 quad core processors and 8x 512 MB RAM. The 1429 card is plugged into a 16x PCIe slot, of which 4 lanes are used by the card.
The second system where we however also get the FIFO error is a Precision 390 system with Intel 975X chipset (also found on the page: Choosing the Right PC for the NI PCIe-1429) equipped with 2x 1GB RAM.
I am not sure if the 64 bit XP makes the difference. The Intel 5000X chipset of our original computer is very similar to the 5400 chipset. The general system specs should be much faster then actually needed for the 1429 frame grabber. For all the systems tested, 4 PCIe lanes are negociated from the frame grabber. We also updated all the chip set drivers, which did not make a difference. Do you have any additional suggestion? In my eyes it is very unpredictable, which system works and which doesn't.

Thanks,

Ralf

[Elmar_NI]

Hello Ralf,

i am already in contact with the developer in US about the issue. But you see that the error produce the pc system and not the framegrabber card. Therefore exist the KB: Which PC work with the framegrabber 1429. I think we should wait a littel bit from the US feedback. So that you know why your Dell 640 workstation dosent support the hardware.

Kind regards,

Elmar