TCP Communications taking up 50% of CPU on Windows

[thisisnotadream]

Good day!

I am working on one of several real-time applications in our lab.  We are using the typical architecture of PXI controller with RT running models and performing DAQ, etc. all while the Windows "host" PC is running another application that displays and accepts user input and output, etc.  The connection between the two applications we are using is TCP communications, as speed and low overhead was important.  What we are seeing on this (and other) applications is that everything runs great for the first few minutes, then the PC will freeze for upwards of 30 seconds to 2 minutes (meanwhile causing Media Player to skip furiously), show LabVIEW with a huge utilization on the CPU...and then...suddenly return to running smoothly for the next few minutes. 

This is pretty frustrating and I am having difficulty figuring it out.  I profiled the VIs and found that the only commonality between these "moments" was that my TCP communication VI was showing a relatively long running time (of like 15ms) corresponding to each of these "moments".  I have attached examples of the VIs that are interacting on these TCP links.  I've run them by people here at work and been running and probing and staring and scratching for a few days now to no avail.  Can anyone see any potential problem areas?  Even a shrug and a "I don't know.  Looks good to me."  would be reassuring at this point.

Dan

PS--Running LabVIEW 8.2.1

[MattBradley]

Hi Dan,

I can't look at your code right now, but can you check the execution priority of the VI where it seems to hold up on the "host" PC?  Also, can you ping the host computer from the remote PC (or vice versa) during one of these slowdowns?

[thisisnotadream]

The communication modules are set to run in the normal priority (the calling host VI is set to "above normal").  I was thinking about priority earlier, but it's pretty clear to me that LabVIEW is the process that is taking up all the CPU.  Would moving the "connection module" VIs to another Execution System allow the GUI redraw thread to continue to operate?

Haven't played around with the Real-Time utilities.  I'll put together another VI on the target that periodically pings the controller and print the results to the monitor.  Maybe I'll see something when we go into a "moment".  What will that tell me if the ping fails?

Dan

[thisisnotadream]

PS--Didn't realize I had made a key combination for a smiley there.  Please don't try to understand why I would be winking at you at that point in my sentence.

[MattBradley]

I was wondering why you would wink... thanks for the explanation.

What I meant by pinging was to do it from a DOS window.  If it doesn't work, it will timeout and tell you.  It might just go very slow.  Should be less than 1 msec.

[thisisnotadream]

After finally getting WireShark on my host machine I did a little more digging.

Firstly, I ran a ping or 4 off the target while in a "moment".  About half went unanswered, the rest were on the order of 10ms responses (some more).

Next, took a look at the traffic between the two computers.  If WireShark is to be trusted, then the pattern is such:
Periodic traffic between the two computers.

TCP ACKs from the host to the target for the most recent messages

Silence for a period of at least 10 seconds.

Deluge of traffic from the target. 

This deluge of what I'm assuming to be pent-up messages while the target is doing something mysterious is likely to be the cause of the hangups.  The TCP Read VI and the underlying Windows network system calls are likely trying really hard to keep up.  Agreement?  Any other ideas?

Since these are simple GUI updates to the user, I will set the loops that are in charge of periodically updating the host with recent data to "Discard missed loops".  This should cut down on the deluge, but it is likely that I will have to go and figure out what is causing the silence in the first place.

Groan.  Any ideas?

[Jon_S.]

Hellow thisisnotadream,

I took a quick glance at your code.  Try putting a wait in the while loop of your program that runs on the windows machine.

[thisisnotadream]

The Windows application is running using periodic timed loops.

I am suspicious that a few modules in the target application are starving the other modules.  So, I have backed off their periodicity and have also set the TCP read VIs to have a 0ms wait time.  As these VIs are called every 200ms or so, I don't think this should be a problem.  I think the 1ms waits x9 modules may have been giving the other threads fits as some of them operate at 200Hz.

We'll see tomorrow.

Dan

[Jon_S.]

Keep us posted on the results

[thisisnotadream]

Now I find myself "rebooting due to exception C0000005".  Not good.  Can't find any information on RT exceptions.