GigE actual maximum data rate / Windows hangs

[MichaelWestwood]

Hi,

I am building a system using Labview 2009 and IMAQdx. It has two Basler Ace 1280x960 GigE monochrome cameras running at 10fps and a FLIR A series 640x480 GigE camera running at 50fps. I have encountered a problem where windows hangs completely, ie, the mouse won't move, ctrl-alt-del doesn't work, the only thing I can do is to reboot the PC. By my calculations the data rate is in the order of 550M bits per second which I think, given my understanding of ethernet considering collisions (and I am by no means an expert), is a high data rate for GigE. This calculation excludes possible re-sent packets. I suspect that this is what causes the problem.

I am using a Netgear PoE switch which was sourced from Basler and is thus presumably certified by them. I find with this switch I cannot use 8000 byte packets but must reduce them to 1500. Unfortunately this system uses a laptop so I cannot use the recommended Intel Pro1000 NIC.

My questions are:

1) Has anyone else experienced something similar?

2) What is a realistic maximum data rate for GigE? I could then limit this in software to avoid windows hanging.

3) Is there a property in IMAQdx which would give me an indication that things are going wrong so I could stop the acquisition and give a warning before windows hangs?   

3) Would using 8000 byte packets make a significant  difference?

Thanks,

Mike

[BlueCheese]

Hi Mike,

Generally this type of thing shows up when there is lots of packet loss, lots of resend requests, and you are using a NIC that is not supported by our high-performance driver and so there are lots of requests to the vendor NIC drivers. We do have an open issue to try to detect this "overload" condition better and try to back off from the resend requests more gracefully before the system appears to hang. However, this will not solve your ultimate problem, which is that you are likely just losing data.

One thing to consider here is that most likely you are losing data because all the devices are connected to the same switch and you are exceeding the bandwidth available back to your PC. One item to note is that while your average data rate over several seconds might be the 550Mbit you calculated, the "peak" data rate while the cameras are transmissing their images might actually be 3000Mbit (3 x 1000Mbit). This is because the cameras by default will send the data after each frame is acquired as fast as possible to minimize latency.

What you should do is configure the cameras to ensure they share the bandwidth properly. This can be done by adjusting the "Peak Bandwidth Desired" attribute in the Camera Attributes tab (enable "All Attributes" view) in MAX. The default for this is 1000Mbit. You would want to divide that 1000Mbit in some fashion between your three cameras so that the total does not exceed 1000. This should ensure that the burst rate never exceeds what your link can sustain.

Additionally, you might want to try seeing if you can somehow enable jumbo frames (>1500 bytes) as this reduces the overhead considerably, especially when using non-Intel-based NICs. My suspicion is that the switch you are using can handle them just fine, you likely justg need to go to the settings for your network card and configure it for jumbo frames if supported (most do).

Eric

[AnthonV]

Eric,

I am having a very similar problem - images are captured fine for about 1 minute (this time is variable and dependent on something unknown) and then the images become corrupted (black lines across the images correlating with 'lost packet count' increasing and then *boom* Windows freezes so badly that the mouse and keyboard becomes completely unresponsive.

Michael, I found it is possible to recover by: unplugging the ethernet cable and waiting about 30mins. All this time the PC's fans will be wirring hard indicating that the processor must be very busy and then at some point Windows will recover and the error "Camera removed from the bus" will appear as I unplugged the camera previously. Possibly unplugging the camera is not a necessary step.

I first suspected my code but the same happens in MAX using its Grab button.

What is strange is that:
- I am running well below the GigE bandwidth (only 1x 1280x960 8-bit grayscale Point Grey Flea3 at 3 frames per second).
- The peak bandwidth desired is already set to 100.
- Sometimes it will run well.
- I have a slower laptop running in the other room (WinXP) with 6x of these same cameras running at 7 frames per second without hanging problems, although I get a lot of lost packets.

Eric, I read that setting the packet delay might help. Is this referring to the Attribute: CameraAttributes::TransportLayerControl::GevSCPD attribute?

My system is a Dell i5 laptop running Win7 64bit, no Pro1000 ethernet chipset, no jumbo packets, crossover cable directly to a Flea3 camera.

Any help would be appreciated.

[AnthonV]

Good news - I have alleviated my problem (well until the next glitch that is).  I took the advice from Point Grey given in http://www.ptgrey.com/support/downloads/documents/TAN2010003_Image_tearing_causes_and%20_solutions.p... which basically comes down to disabling all the fancy power management features offered by the modern CPU's.  

I am seeing a huge reduction in lost packets (meaning fewer broken images) and I have so far had no windows hangs (I have been running for the last 6 hours).

Of course my laptop now only lasts about 30mins on the battery but that is a minor issue.

[chris_1243]

Mike,

Did you resolve this problem? I am experiencing something similar except I am not so sure it is related to lost packets. I have "Jumbo Frames" supported on the Card and my images never appear to be missing data. I am streaming 5MB at 15fps, so 75MB/sec range.

The problem is that after 40min to 2 days of my software running, Windows "freezes" and everything (mouse/keyboard/monitor) are unresponsive. I can see the exact time that the PC froze and I also had Task Manager running when it froze and could see that it is not a memory issue. I have my software running on several PCs without any problems but 2 or 3 PCs are giving me this kind of behavior. The PC never freezes when my labview exe is not running.

The PC never freezes in the Camera's native software (I'm using a Prosilica camera). I tried it in MAX and MAX "Stopped Working" precisely after 3hrs and 20 minutes in "Grab" mode. This happened 3 times at the same duration... so I suspect that is more of a MAX issue (not intended to be in Grab mode for so long).

I am using LV2011SP1.

I tried updating the Vision Acquisition Software from 2010 to 2011 and am testing right now to see if this makes any difference. If it does not, I may reinstall Windows completely and/or update my labview exe to LV2012 version.

Best Regards,

-Chris

[Cameron-T]

Did you already try the suggestions of Eric (BlueCheese) above?

[chris_1243]

Cameron,

As far as I can tell I am not losing data or missing packets, excessively. At least, the camera images appear normal. Jumbo Frames is enabled. I do not have any problem running the camera's native software (from AVT) at full frame rate.

Reinstalling VAS (Vision Acquisition Software) to a newer version did not help as the PC still freezes when my Labview exe runs. I just uninstalled all labview components/software and am re-installing to see if that makes a difference. If it does not, I plan to wipe out windows and reinstall the OS, drivers, software and all. The odd thing is that the PC hanging up is only happening on about 20-30% of the PCs and happens within 48 hours usually. These PCs do not hang if I run just the Native software; only when I run the LV exe.

-Chris

[Kilometers]

One thing you might try is making sure the Chunk Data settings on your camera are not in use. These have been known to sometimes cause problems. If you go into your camera's configuration in MAX you'll want to make sure "Chunk Data Decoding Enabled" and "Chunk Mode Active" are both unchecked. 

[chris_1243]

Miles - Thanks for the info. "ChunkModeActive" is unchecked. I cannot find "Chunck Data Decoding Enabled" as it does not appear in the list of camera attributes.

-Chris

[Cameron-T]

Make sure to look through all of the camera attributes list. The two selections are probably not grouped together. The exact location of the attribute depends on the camera manufacturer.