Instrument Control (GPIB, Serial, VISA, IVI)
VISA Serial Read does not timeout correctly
I believe that the VISA serial read gives "false" timeouts when used in higly parallel circumstances. This problem was fixed in much older versions of VISA but it is a tricky error and has come back in the more recent versions.
I have a highly parallelized system with MANY concurrent loops executing. Upon upgrading from VISA 2.? to VISA 3.5 I started getting serial port timeouts that had not occurred before. If I run NI Spy the timeouts dissappear!! This can't be true, either the hardware replied in time or not. The hardware is fairly fast and the timeout is many times the expected reply time. These rare and intermittent errors have been driving me and the operators nuts for quite a while now.
This happens if the reads and writes are asynchronous or synchronous!!
So, I have a test VI that I created that allows one to spawn CPU sucking tasks while writing to a loop back connector. In this case the hardware cannot be timing out since the cable length / speed of light is a LOT less than the timeout period. I merely write one character to the serial port and read 1 character back, with a compare. Last night after 3208923 iterations the read back timed out with a BFFF0015 error.
What I believe is happening is a subtle timing error in the VISA read timeout logic. The following scenario describes it. I have indented a different thread that preempts the VISA thread.
VISA starts a timer
VISA starts its read
VISA tests to see if the data is available
----ANOTHER THREAD INTERRUPTS AND STEALS THE CPU CYCLES
----Lots of time passes
data arrives at serial port
----Execution returns to the VISA read thread sometime after the timout passes
VISA checks the time
The time is more than the timeout
VISA reports a timeout error.
What needs to be done is to check the VISA serial port AFTER you check the time. In this case you never report a false timeout, though sometimes on a heavily loaded system you will report that a communication was ok when it MAY have exceeded the timeout.
In this case CPU load prevents determining if the timeout occurred. So unless the system is certain that a timeout occurred it should not report an error. Basically when in doubt, don't blame external hardware. 🙂
System Details:
Mac OS X
VISA 4.0.1
LabVIEW 8.20
Keyspan 28X twin serial USB-Serial adaptor
Test program set to 18 parallel tasks. LabVIEW changed to use 8 threads for all execution engines all priorities using vi.lib/Utility/sysinfo.llb/threadconfig.vi. Changing the threads, the execution priorities, etc. does not seem to affect this problem much.
Open the Serial Port test VI
put a loop back connector on one of your serial ports
Enter that serial port VISA name into the serial port tester control
run the Serial port test VI
Tell it to think "harder" a few times (I had 18 in my test) until your CPU gets loaded up.
Wait about 24 hours and see if it quits with a timeout error.
Here is the front panel of the test VI when the error occurred.
I have attached a LV 8.20 project that I used for this test. This is a very simple test and the most complicated thing is to spawn all the CPU sucking sub tasks. Remember to bring the number of sub tasks back to zero before stopping the VI or it may cause LV to unexpectedly quit.
-Scott
I have a highly parallelized system with MANY concurrent loops executing. Upon upgrading from VISA 2.? to VISA 3.5 I started getting serial port timeouts that had not occurred before. If I run NI Spy the timeouts dissappear!! This can't be true, either the hardware replied in time or not. The hardware is fairly fast and the timeout is many times the expected reply time. These rare and intermittent errors have been driving me and the operators nuts for quite a while now.
This happens if the reads and writes are asynchronous or synchronous!!
So, I have a test VI that I created that allows one to spawn CPU sucking tasks while writing to a loop back connector. In this case the hardware cannot be timing out since the cable length / speed of light is a LOT less than the timeout period. I merely write one character to the serial port and read 1 character back, with a compare. Last night after 3208923 iterations the read back timed out with a BFFF0015 error.
What I believe is happening is a subtle timing error in the VISA read timeout logic. The following scenario describes it. I have indented a different thread that preempts the VISA thread.
VISA starts a timer
VISA starts its read
VISA tests to see if the data is available
----ANOTHER THREAD INTERRUPTS AND STEALS THE CPU CYCLES
----Lots of time passes
data arrives at serial port
----Execution returns to the VISA read thread sometime after the timout passes
VISA checks the time
The time is more than the timeout
VISA reports a timeout error.
What needs to be done is to check the VISA serial port AFTER you check the time. In this case you never report a false timeout, though sometimes on a heavily loaded system you will report that a communication was ok when it MAY have exceeded the timeout.
In this case CPU load prevents determining if the timeout occurred. So unless the system is certain that a timeout occurred it should not report an error. Basically when in doubt, don't blame external hardware. 🙂
System Details:
Mac OS X
VISA 4.0.1
LabVIEW 8.20
Keyspan 28X twin serial USB-Serial adaptor
Test program set to 18 parallel tasks. LabVIEW changed to use 8 threads for all execution engines all priorities using vi.lib/Utility/sysinfo.llb/threadconfig.vi. Changing the threads, the execution priorities, etc. does not seem to affect this problem much.
Open the Serial Port test VI
put a loop back connector on one of your serial ports
Enter that serial port VISA name into the serial port tester control
run the Serial port test VI
Tell it to think "harder" a few times (I had 18 in my test) until your CPU gets loaded up.
Wait about 24 hours and see if it quits with a timeout error.
Here is the front panel of the test VI when the error occurred.
I have attached a LV 8.20 project that I used for this test. This is a very simple test and the most complicated thing is to spawn all the CPU sucking sub tasks. Remember to bring the number of sub tasks back to zero before stopping the VI or it may cause LV to unexpectedly quit.
-Scott
Shoneill,
Sure thing!
This topic was originally misposted (my fault) into the field point forum. I thought I reposted my test VI, but it seems not to be here. Attached is the test VI project in LV 8.2 format.
NOTE: It is important to reduce the number of spawned CPU hogging tasks back to zero before stopping or quitting the test VI or LabVIEW will crash later on. The spawned tasks do not exit and become invisible background tasks. But this is another bug fix thread.
For additional VISA serial port testing, I wrote an interesting script to check the effects of packetizing on serial port transmission optimization. It was sort of interesting. But it did reveal some problems with VISA back in LV 5.1, 6.0, 6.1 (something like that). My test VIs became part of the VISA test validation suite I believe after that. Here are the originals of that test suite. It is labeled "VISA Serial Test".
-Scott
Sure thing!
This topic was originally misposted (my fault) into the field point forum. I thought I reposted my test VI, but it seems not to be here. Attached is the test VI project in LV 8.2 format.
NOTE: It is important to reduce the number of spawned CPU hogging tasks back to zero before stopping or quitting the test VI or LabVIEW will crash later on. The spawned tasks do not exit and become invisible background tasks. But this is another bug fix thread.
For additional VISA serial port testing, I wrote an interesting script to check the effects of packetizing on serial port transmission optimization. It was sort of interesting. But it did reveal some problems with VISA back in LV 5.1, 6.0, 6.1 (something like that). My test VIs became part of the VISA test validation suite I believe after that. Here are the originals of that test suite. It is labeled "VISA Serial Test".
-Scott
Hello Scott,
I've spent some time testing your code and I have a solution for you to try. Add a Wait function between the VISA Write and the VISA Read. When you run NI-Spy or highlight execution, you slow down the speed the code is running at, therefore giving your serial device enough time to perform the write and the read. The way you are running your code right now is not giving your device enough time to perform those actions, especially when you're tying up the processor.
Try this and let me know if you're still having trouble!
Meghan
Applications Engineer
I've spent some time testing your code and I have a solution for you to try. Add a Wait function between the VISA Write and the VISA Read. When you run NI-Spy or highlight execution, you slow down the speed the code is running at, therefore giving your serial device enough time to perform the write and the read. The way you are running your code right now is not giving your device enough time to perform those actions, especially when you're tying up the processor.
Try this and let me know if you're still having trouble!
Meghan
Applications Engineer
Message Edited by Meghan R on 01-23-2007 12:51 PM
Message Edited by Meghan R on 01-23-2007 12:51 PM
Meghan,
Thanks for taking a look at this. However I am not sure that what you propose is either a solution or a work around. Let me see if I can clarify. The program should not wait for data to be available, since it is merely a loopback connector the data is available as soon as it is sent. It does not matter if the VISA calls are synchronous or asynchronous. If the calls are synchronous, then the VISA should not have even returned from the write until the data has been sent. This may be impossible with some devices. But, as I undestand it, VISA writes and reads on a single device block for each other. Only reads and writes on a different VISA device can execute in parallel.
At 9600 bps the writing of one character should take about 1 ms. The 50 msec wait that you propose will slow down the program by about a factor of 50. Thus instead of the error happening 1/day it will happen 1/50 days making it impossible to test in a reasonable amount of time. Merely crippling the test VI is not eliminating the error.
For my actual application this is completely intolerable. The communication is at 57600 bps and even at that rate I was considering raising it 115.2 kpbs to increase performance. To add such a huge delay would make the progam unworkable. The program ran without timeouts for many years using LV 7.0 and Mac OS 9. I moved it to faster hardware and OS X and updated the versions of LV and VISA which started this problem.
Yes, NI spy is changing the timing, you should see the load that it puts on the processor. BUT, VISA should be capable of doing sequential reads and writes without timeouts. It may be that NI Spy is merely making the error less frequent and not eliminating it.
The best workaround that I can think of (and this is untested) is after every read, to test for a timeout and if so then do an additional read with a timeout of zero. This effectively implements the fix I outlined by forcing the last read to be after the timeout is calculated. I will have to put this into my serial port library structure. I haven't time today, but I will try this on the test VI and see if it fixes that case.
-Scott
Thanks for taking a look at this. However I am not sure that what you propose is either a solution or a work around. Let me see if I can clarify. The program should not wait for data to be available, since it is merely a loopback connector the data is available as soon as it is sent. It does not matter if the VISA calls are synchronous or asynchronous. If the calls are synchronous, then the VISA should not have even returned from the write until the data has been sent. This may be impossible with some devices. But, as I undestand it, VISA writes and reads on a single device block for each other. Only reads and writes on a different VISA device can execute in parallel.
At 9600 bps the writing of one character should take about 1 ms. The 50 msec wait that you propose will slow down the program by about a factor of 50. Thus instead of the error happening 1/day it will happen 1/50 days making it impossible to test in a reasonable amount of time. Merely crippling the test VI is not eliminating the error.
For my actual application this is completely intolerable. The communication is at 57600 bps and even at that rate I was considering raising it 115.2 kpbs to increase performance. To add such a huge delay would make the progam unworkable. The program ran without timeouts for many years using LV 7.0 and Mac OS 9. I moved it to faster hardware and OS X and updated the versions of LV and VISA which started this problem.
Yes, NI spy is changing the timing, you should see the load that it puts on the processor. BUT, VISA should be capable of doing sequential reads and writes without timeouts. It may be that NI Spy is merely making the error less frequent and not eliminating it.
The best workaround that I can think of (and this is untested) is after every read, to test for a timeout and if so then do an additional read with a timeout of zero. This effectively implements the fix I outlined by forcing the last read to be after the timeout is calculated. I will have to put this into my serial port library structure. I haven't time today, but I will try this on the test VI and see if it fixes that case.
-Scott
Shawn,
You are most likely right. I tried adding a "flush transmit buffer" into the VI after the VISA write. This should wait until all the bytes are flushed to the hardware. (It may not be true)
This did not fix the problem on the actual hardware. So to test my theory I added a second VISA read with a zero timeout. I still got timeouts on the VISA read.
On my test system it decreases the number of errors but they still happen. That would mean that hardware is being dishonest and either not actually sending the bytes out the serial port when it claims, or not reporting that the bytes are present when they actually are.
This is bad. I don't want timeouts to be set large for performance problems, but I really don't want to see timeouts every 1 in a million transactions. If the USB driver does not get any CPU for a long time, its information is stale but has no way to report that back to VISA. Why I hate serial ports, and another good reason to believe that USB-GPIB or ENET-GPIB will suck as well!!!
If you can't reliably determine an error or a timeout then the device is not as useful as one would think. Merely increasing the timeout only reduces the probability of the error occurring and does not eliminate it. In real life, the application should have filled the serial port buffer so it may just be the CPU doesn't give the USB hander enough cycles to transfer the data to the O/S.
I appologize for blaming VISA! In this case it is the hardware is just being pushed too far and USB can't handle high data rate serial streams reliably. This is going to take some major restructuring of the program to detect and work around this problem.
Thanks for looking into this.
Scott
You are most likely right. I tried adding a "flush transmit buffer" into the VI after the VISA write. This should wait until all the bytes are flushed to the hardware. (It may not be true)
This did not fix the problem on the actual hardware. So to test my theory I added a second VISA read with a zero timeout. I still got timeouts on the VISA read.
On my test system it decreases the number of errors but they still happen. That would mean that hardware is being dishonest and either not actually sending the bytes out the serial port when it claims, or not reporting that the bytes are present when they actually are.
This is bad. I don't want timeouts to be set large for performance problems, but I really don't want to see timeouts every 1 in a million transactions. If the USB driver does not get any CPU for a long time, its information is stale but has no way to report that back to VISA. Why I hate serial ports, and another good reason to believe that USB-GPIB or ENET-GPIB will suck as well!!!
If you can't reliably determine an error or a timeout then the device is not as useful as one would think. Merely increasing the timeout only reduces the probability of the error occurring and does not eliminate it. In real life, the application should have filled the serial port buffer so it may just be the CPU doesn't give the USB hander enough cycles to transfer the data to the O/S.
I appologize for blaming VISA! In this case it is the hardware is just being pushed too far and USB can't handle high data rate serial streams reliably. This is going to take some major restructuring of the program to detect and work around this problem.
Thanks for looking into this.
Scott
