Increase UDP received packets per second

[jjjcjj]

I have a very high packet rate device (100000+ packets per second) which I am trying to capture in LabView.  I have configured a single loop to act as the 'UDP Receive' loop which simply takes the data off the wire and puts that data into a queue to be processed by another loop.

The problem is that whether I use the builtin UDP receive functions, or one of the available pcap library implementations, I will always drop packets.  When I time the execution of the receive loop I find that it is only running in the milisecond range instead of the required <10us.  If I remove the 'enqueue' call, and do nothing except receive the data and increment a packet counter, it is still no where near meeting timing.  This leads me to believe that the problem is with the amount of time it takes to call the vi to perform the read.

Before I write a dll to perform multiple packet reads, and transfer large buffers to labview, is there anything that can be done in labview to increase the packet rate?  Note that I have captured this data in wireshark on the same machine, and it is more the sufficient to capture all of the data, so it is possible.

[billko]

---

@jjjcjj wrote:

I have a very high packet rate device (100000+ packets per second) which I am trying to capture in LabView.  I have configured a single loop to act as the 'UDP Receive' loop which simply takes the data off the wire and puts that data into a queue to be processed by another loop.

The problem is that whether I use the builtin UDP receive functions, or one of the available pcap library implementations, I will always drop packets.  When I time the execution of the receive loop I find that it is only running in the milisecond range instead of the required <10us.  If I remove the 'enqueue' call, and do nothing except receive the data and increment a packet counter, it is still no where near meeting timing.  This leads me to believe that the problem is with the amount of time it takes to call the vi to perform the read.

Before I write a dll to perform multiple packet reads, and transfer large buffers to labview, is there anything that can be done in labview to increase the packet rate?  Note that I have captured this data in wireshark on the same machine, and it is more the sufficient to capture all of the data, so it is possible.

---

If you don't want packet loss, forget UDP.  What is it that requires such a high data rate and can't be lossy?  I guess Wireshark can do this because it is a dedicated packet sniffer.

[Ben]

Wireshark takes advantage of "Permiscuous Mode" that was built into ethernet adapters from the begining of time. In that mode the all of the packets are avaialbe regardless of protocol.

UDP is allowed to drop packets and if it is convienient to do so, the UDP layer will toss packets.

See this thread for a possilbe way to harness WinPCap and get the packets the same way that WireShark would do it.

Using the search term "WinPCap" will give some other threads that were related to sniffers.

Ben

[jjjcjj]

Thanks for your inputs.  Let me clarify a couple things.

- This is a direct connection between a piece of hardware, and a PC.  There are no switches or routers which could drop or reorder packets during times of congestion.  The only thing that might drop the packets is the windows network stack.

- Gaurenteed delivery is not required, nor is 0% packet loss, however 1/10 packet receiption isn't enough.

- I have already imlemented a winpcap based mechansim using the exact library you suggested, and I have even done so in promiscusous mode, however there is no way to collect packets which arrive <10us apart with a loop that can only execute at 1ms.  The packets arrive at the machine just fine, they aren't being dropped.

[Ben]

So you are telling us that the function calls to retireve the packets are limited to about 1000 packet fetches per second?

Can you let us see the code you are using to see if we may be better able to help out?

Ben

[jjjcjj]

Its actually getting 20,000-30,000 packets, the last number I provided was a rough estiamte based on averaging the time-per-loop iteration by eye, but yea that is what the problem appears to be.

It doesn't really take much to demonstrate this.  My computer is connected, by gigabit ethernet, to this special piece of hardware which is throwing about 200,000 packets per second at it.  If you put a UDP Receive in a while loop and increment a counter every time the loop executes then you'll notice about 20,0000 packets per second.  If you close labview and open wireshark isntead then you will notice 200,000 packets per second.

Like I mentioned earlier, I also did this using pcap (with the library found here http://forums.ni.com/t5/LabVIEW/WinPCAP-for-LabView-v-0-1-alpha/td-p/343479/page/2), and the results are the same.

[natasftw]

jj.  Let's take a look at this interaction so far.

jj: Help me figure this out
others: Take a look at this
jj: That still doesn't work

others: can you show us what you're working with so we can try to help troubleshoot

jj: It doesn't take much effort to set this up.  I trust that you can all do this by yourselves and set it up identically to what I'm seeing.  I shouldn't need to show you what I'm working with to get help.

It may be trivial to setup.  Someone else might get the same code you've got.  But, if it's trivial, just share the example you've got that exhibits this behavior.  It makes the help you get more focused to what you're doing and means others are more likely to pick up your request and dig in. 

You're picking up data at roughly 20kHz.  That's actually pretty solid for software timed applications.  What other overhead is in your loop?  The loop itself is overhead, don't forget.  What priority is your VI running at?

[Ben]

---

@jjjcjj wrote:

...

It doesn't really take much to demonstrate this.  ...

---

Hi jjjcjj,

Most of the contributors here do not work for NI and are just volunteers. In my case I only can take quik looks and post thoughts between coffee breaks.

It would be very helpful for those of us that are curious about what you are doing and how we can help you if you shared the code you have so we can take a quick look.

So I ask that you plese help us help you by posting something that we can look at.

Still curious and hoping I can learn by following your work.

Ben

[jjjcjj · Accepted solution]

Attached are two examples which don't work becuase the loop execution rate is too low.  Exactly why the loop executes too slowly is a mystery to me.  Note that the 'pcap' version is using the library from the thread mentioned earlier.

The problem is now solved.  I updated the dll from that thread to include a 'multi-packet-gather' function which delivers bulk packets to labview.  Even after doing this I noticed that there was still a bottleneck when LabView went to assemble the packets (collecting data from multiple packets to make larger data sets).  This was solved by moving some of the project specific data manipulation into the dll as well.  When I get some free time I also plan on posting the new library to that same pcap thread.

[Ben]

---

@jjjcjj wrote:

...

The problem is now solved....  When I get some free time I also plan on posting the new library to that same pcap thread.

---

Thank you in advance for the improved version!

Ben