LabVIEW
cRIO-9014 Non Volatile Memory Throughput
Hi All,
I'm wondering how fast I can log data on the non volatile memory of my cRIO-9014 controller before there's any chance of overwhelming the file system. (How many KB/sec, etc?)
Here are the details of the scenario:
The reason I'm asking this question is that I have an application where I'm being asked to log data from a 9205 and two 9211 modules at a rate of four times per second. I put together an application that works in every way except for one problem: My logger thread (loop) locks up after a while. I've traced it down to my ASCII logger VI, which is really doing nothing special. The only conclusion I can come to is that I might be gradually overwhelming the file system. Another curious symptom is that the NI FTP server stops responding. The rest of the application, however, including my TCP server run as if nothing is amiss. I asked my internal "customer" which channels he's actually using, and was able to defer the lockup by removing unnecessary channels from the log and reducing precision.
Before I resort to a datalog or other non-ASCII format, I'm curious to know how fast I should be able to log. Also, I'm running LabVIEW 2009 and NI-RIO 3.2.0 on the controller. (upgrading to 3.2.1 as soon as it's finished downloading)
Thanks as usual,
Jim
Alright, after reading continual posts in one of Darren's threads yesterday, it dawned on me that maybe I need to flush the write buffer before terminating the file reference. ("Flush File" function) The thought process behind this is that I've got another loop that checks for completed logs every five minutes and sends them off via FTP. I thought that maybe the FTP thread was trying to delete the file before it was completely written by the logger thread. Perhaps there is some sort of deadlock occurring and that that might be killing my logger thread.
Well, it ran for longer than it ever has after I added the "flush file" function, but alas, when I came in this morning it had still locked up. I had the console connected this time and the only thing that printed was "-> BREAK". It broke. As usual, all my other loops are still running without any issue and the NI FTP server doesn't respond.
Something odd is going on here. C'mon. Someone must know... Bueller? Bueller? Anyone?
Hi Nick,
Thanks for getting back to me. I've been keeping a close eye on the CPU and memory via DSM and they're both fine all the way until the problem occurs. I have some very thorough error handling in place and I haven't seen a single error at all. Regarding precision, I cut down the number of channels I was logging to twelve: four analog inputs and eight thermocouples. For the analog inputs (from a 9205) I cut it down to a single digit of precision and for the thermocouples I'm merely recording integers. Though I'm queueing up samples of each at 4 Hz, it still doesn't seem like an abnormally taxing scenario. (My original thought)
As far as posting my code, seeing that nobody on the forums had spoken up as of Friday, I decided to call in for support. That said, my code is on the "incoming" FTP server and the support request is being handled by one of the AE's named Chris. (I know-- that narrows it down. 
) The code is in a zip file named with the support request number.
Another bit of information that may or may not be useful:
When I query the running processes via the console ("i" command), I notice that, after my logger thread stops responding, one of the VxWorks processes labeled "LV TL[1]:" has a status of "SUSPEND+I" as opposed to the usual "SUSPEND" or "PEND" status that I normally notice. I might be mistaken in that this may not be an anomaly.
This is a new one for me. I've had plenty of lockups, but it's pretty rare for me that the rest of the threads keep humming along and that the system resources appear to be okay otherwise.
Thanks again,
Jim
Nick,
I have some new information: This weekend the logger thread locked up again, but I was able to determine which part of the code it halted on:
I'm not sure which function caused it, but it's definitely something in this code. (Also, it definitely occurred without the "Flush File" function, since I added that later in an attempt to remedy the issue.) I added a delay, too, thinking that perhaps the file system needs more time to close properly. I'm now leaning toward the "Move" function, since closing the reference seems rather benign.
Hmm...
Jim
Thanks, Nick. Hopefully the simple loop test will reproduce the issue, but if not, it could be a combination of things. (FTP VI, etc.) I appreciate it.
Jim
Alright, I've confirmed that it's definitely the "Move" function. I'm going to experiment further...
Another thought... I wonder if my regular expression might periodically yield unexpected results. Wouldn't that throw an error as opposed to stopping the thread, though?
I removed the "overwrite" Boolean on the "Move" function, thinking that perhaps it's the source of my woes. I also started monitoring the output of the match pattern function with a published shared variable, and that's not the issue. The controller took longer than ever to stop responding, but this morning at about 1 AM that thread locked up again after running for most of yesterday. This time, what's really strange is that it stopped on the "Flush File" function.
Given the apparent lack of consistency in failures, I wonder if the non volatile memory is on its last legs, intermittently failing to respond? Have either of you observed this phenomenon, or has it been more of a sudden failure? I only just recently upgraded the controller to NI-RIO 3.2.1, so it's possible that the damage is already done.
I'm going to see if we have a spare controller around here, and if not, maybe our local NI rep can lend us one just so we can figure out if it's really an equipment failure.
Any thoughts?
Thanks as always,
Jim
