Increasing file-system performance when logging data

[BSeegmiller]

The Challenge

When logging data to a disk, as it streams out to the disk, the operating system has the responsibility of ensuring the data gets there.  One of the challenges we've faced when logging data during our test runs is that the file-system performance can have an effect on the operating system performance.  The challenge here is file-system fragmentation. 

The process is similar whether you're running on Unix, Linux, Windows, or MacOS.  Your process has a file open for writing.  Depending on the operating system, various kinds of guesstimates are made by the operating system as to the size of the initial allocation of disk space for that data.  If you're streaming data to the drive, it's an easy guess that the final amount of data written to the disk is far in excess of the initial amount allocated.  If your process is the only process running, then you can be largely guaranteed that your file will be mostly contiguous.  However, in most modern operating systems, there can be many processes that are running that will be writing to the disk.  When this happens, fragmentation can occur. 

A review of file systems and physical disks is at http://en.wikipedia.org/wiki/File_system and http://en.wikipedia.org/wiki/Disk_storage.

As files are added and removed from the file system, a phenomenon called fragmentation occurs, where the space to allocate files gets broken up into small blocks.  This can also occur when two or more files are being continuously written to.  As space on the disk is allocated for each file, the data in a particular file can end up being scattered all over the disk.  When these fragmented files are accessed, the performance is reduced as the file-system locates the successive blocks of data on the disk.  As fragmentation continues, even the process of allocating space to running processes gets more complicated and starts to take more time.  When the filesystem performance degrades in this manner, the time required can start to affect the performance of the running processes and their ability to keep up with the data acquisition and processing. 

A Solution

By pre-allocating the data file, the burden on the operating system and file-system can be reduced.  The file-system will no longer have to scavenge free blocks on the disk, as the file is already allocated.  Under Windows 2000 and XP, the file is contiguous.  This reduces the burden to simply scheduling the data to be written to the disk and advancing the file pointer.

Using LabVIEW file-handling VIs, create a the log file normally, then use the EOF VI to set the end-of-file marker to the something larger than maximum size you expect to write.  After this is done, use the Seek VI to set the file pointer back to the beginning of the file (if the data is homogeneous: text, arrays of numbers, or sequential clusters).  Write the data normally.  Before closing the file, use the EOF VI to set the end-of-file marker to the current file position, then close the file normally.  A simplified diagram is below.  Yes, there is no error-handling.

Platforms Used

This has been used on Windows platforms Windows 2000 (Win2K) and Windows XP with good success.  The maximum seekable file using LabVIEW is 2^31 bytes (about 2 gigabytes) using version 7 and under.  LabVIEW 8 introduced the 64-bit integer and file pointers, which will allow larger seekable files. 

Observations and Caveats

Detailed performance monitoring of the test systems has not been done, beyond observing that there has been a reduction of hiccups processing data. In our experience running test systems under Windows XP, there is "good enough" and "uh-oh" (which is bad -- anything that drops out of keeping up with data is bad).  We load up the machines with as much RAM as possible, use a very good graphics card to reduce that overhead, and reduce the number of background processes.  The machines are kept off the network.  Unfortunately, instrumenting Windows to monitor performance is done with reluctance (the SysInternals Suite at Microsoft looks very good) because of the potential impact to testing.  I would love to see an article on dealing with Windows file-system performance with LabVIEW.  I know the lucky get to use LabVIEW RT.