real-time crashes when reading from DVM

[AntonLodder]

I am running a data acquisition program on a custom-built real-time target running Labview 8.5. I am taking several measurements independantly from different instruments, including an HP3458a Digital Voltage Meter. I have found that if I enable the DVM measurements, the real-time target freezes after a minute of recording data. I have also observed that if I measurements on the DVM within a timed loop structure, in a separate program, this also causes the real-time to freeze.

The most peculiar thing about this problem is that if I run my data acquisition program and then stop it, my data-logging file has contents (I am writing a header and then logging measurements to a text file in parallel). However if I allow it to run until the real-time target freezes, the log file is empty.

Any ideas?

[ColdenR]

Hello Anton,

Are you using the instrument driver for the 3458A (http://sine.ni.com/apps/utf8/niid_web_display.model_page?p_model_id=158)? 

Did this freeze always happen, or has it started happening recently?

Is your real-time target a PC running a real-time OS? What hardware/software are you using for your real-time target?

Does the program freeze if you just run the DVM measurements (and none of your other measurements in parallel)?

[AntonLodder]

Hi Colden,

I am using an instrument driver for the 3458A but it is published by HP, not Agilent. The driver is completely different fromt the one you linked. I see that the HP driver is no longer available - unfortunately I can't say where the driver came from because I am not the original developer, however the readme attached with the driver says that it was NI-supported at one time.

The freeze has always occured on the specific real-time that I am working on, however the program has also been used on two other systems with the same setup, and on those it runs.

The real-time target is a custom PC built to NI specs and running LabVIEW RT 8.5. I am working on getting a full list of the hardware, but all I currently know is that it is running on 2GB of ram and that the hard-drive is brand new (we first thought the program was crashing while waiting on the hard-drive, so we replaced it).

I have built a test vi that just measures and reports voltages, which crashes after about 2 minutes. I have tracked the CPU and Memory usage on the real-time target while the program runs and have not observed anything that I would think caused a crash. I have the measure/read functions inside a timed loop structure. If I use a while loop I don't observe a crash in the test-VI but in my measurement application it will also crash in a while loop.

I have been able to get my measurement application to run without crashing by removing the timed loops that were running some of my concurrant measurements and replacing them with for loops.

Anything else you need to know?

I have attached my test-file and the HP drivers that I am using. I will try to find time later to test this VI on one of our other systems to see if it crashes there, but I don't think it will.

[AntonLodder]

*sorry, when I say that I replaced the timed loops with for loops, I meant that I replaced them with while loops.

[Texas_Diaz]

When your system crashes, is there a file that gets created here:

• c:\ni-rt\system\rtlog.txt

This file may contain a crash log that we can use to back-trace possible reasons for the crash.  LabVIEW 8.5 is fairly old, outside our normal range for tech support, but if we can easily find a reason for the problem then we'll "get 'er done."

-Danny

[AntonLodder]

There is no rtlog.txt generated when the real-time crashes.

[Texas_Diaz]

I re-read your post, and I noticed a few inconsistencies.  Let's be specific here - is it a FREEZE or a CRASH?  I think you're using these two terms interchangeably, and they are not.  A FREEZE is when your application stops running, or the system appears to "HALT".  A CRASH is when the system exits and reboots, or a series of messages are displayed on the console showing error status.  Which issue are you having?

-Danny

[AntonLodder]

Ok, I understand the difference between crash and freeze, but I am not really sure how to go about testing whether or not it actually crashes or just freezes. When the crash/freeze event occurs, the only indication I get is that the connection with the real-time is broken. After this I can't deploy any code on the real-time until I close LabVIEW and then reboot the real-time. 

One indication that the real-time might be crashing is that there is no error log generated. Also, my measurement software is constantly logging data to a text file, but if the crash/freeze event occurs the text file is empty, even though there would certainly have been contents if I had stopped it before the crash/freeze event occured.

If I manually reset the real-time target, the power lights cycle on and off, but if I run my measurement and it crashes I don't observe this.

I am reasonable sure that it is not just a network connection issue because after I loose connection with the real-time target, the code that is running on the real-time no longer interacts with any of the instruments - this assumes that the program doesn't stop on its own when the connection is broken. Furthermore, the crash only occurs in my test-VI when the timed loop is present, and it does not happen at a determinable point in the program.

[Texas_Diaz]

In LabVIEW 8.5, we implemented the CPU load meter that prints to the VGA console on PXI controllers and Desktop systems.  It shows real-time CPU activity and even segments the activity into several "buckets" (ISR, Timed Structuress, Other Threads).  If you have access to the VGA output (by plugging a monitor into the VGA output) you can see the load on the screen.  

It would be useful to look at that information.  If the target is having problems communicating during the "freeze", it could be that the system is locking up because of an interrupt storm (if the drivers, for some reason, are not correctly servicing hardware interrupts).  We cannot see what hardware is having problems, but we could see that problems are being had.

Can you look at this information, and maybe take a picture of what you see during a lockup?

-Danny

[AntonLodder]

Interesting. This sounds like a good idea but do you know how to set it up? My monitor does not detect any input fromt the real-time, is it possible that I have to configure the real-time to generate this output?