Intelligent Data Compression using LabVIEW FPGA

[SteveDwyer]

Introduction

This is an intelligent data compression algorithm targeted for LabVIEW FPGA that aims to reduce the amount of raw data acquired using a NI device with re-programmable IO. The algorithm ensures that the minimum number of data points are transferred, whilst ensuring a specified level of accuracy. This means that the algorithm compresses the data on a defined-loss basis.

The algorithm works in the following way:

As each new point is acquired, the gradient between that point, plus and minus the specified accuracy, and the last stored point will be calculated. This will provide a gradient range.	As new points arrive, the union set of these gradient ranges will be calculated. This is the region within which a straight line from the last stored point will never pass more than the specified accuracy from any of the acquired data points.	If a new acquired point means that the union will no longer exist, a point will be stored for the time of the previous data point at the mid-point of the union. This ensures that when the resulting filtered data is interpolated, it will always be within the specified accuracy of the original points.

Once a point is stored, the algorithm begins again with a new gradient range.

To minimise any data loss in an event such as a power failure, you can define the maximum amount of time allowed between two points. If this time has been exceeded then the algorithm will store the current point regardless of whether the union of gradients still exists.

Decompressing Data

To view the data in its uncompressed form you can simply plot it on an XY Graph and the waveform plotted will be within the specified accuracy of the original data. If you need to access individual points then you can use linear interpolation on the compressed set of data points.

Usage

We have provided both FPGA Top-level and Host example VIs. You can use these together to read 1 second's worth of data from the FPGA and plot it as illustrated in screenshot below. Follow the instructions on the 'Host Example.vi' to run the example.

Limitations

The current implementation of the algorithm has the following limitations/trade-offs:

• The algorithm can only operate on a maximum of 8 channels.
• The Host Example is limited to acquiring and displaying 1 second's worth of data.
• The algorithm has been optimised for linear signals, therefore it may not provide the most efficient compression for non-linear signals.
• The algorithm has been designed for low overhead decoding; more complex schemes may provide higher compression.

Please contact ITDev for integration support, or to discuss solutions optimised for higher compression, different signal characteristics, or higher channel counts.

Files

Inside the attached Data Compression.zip are the following files:

• Data Compression.lvproj - Project containing all required files. This currently targets a PXI-7831R device.
• Host Example.vi - Example Host VI containing communications with the FPGA VI and a graph for visualising the algorithm results.
• FPGA Top Level Example.vi - Example top level FPGA VI containing the main FPGA loop for demonstrating the algorithm.
• Averager.vi - VI to perform 2 point averaging to help filter out noise.
• Split Data.vi - VI to use with the averager to split the data to help pipelining into the data compression algorithm.
• Data Compression/Data Compression.vi - Main VI for the Data Compression algorithm.
• Data Compression/Calculate Gradient Range.vi - Part of the data compression algorithm - Calculates a gradient range from two points.
• Data Compression/Calculate New Point.vi - Part of the data compression algorithm - Calculates a new point to store based on a gradient range and the previous point.
• Data Compression/Check Union of Gradients.vi - Part of the data compression algorithm - Determines if the union of gradients still exists.
• Data Compression/Divider LUT Initialisation.vi - Part of the data compression algorithm - Used to initialise the look-up table used in 'Calculate Gradient Range.vi'. Note: This is not run on the FPGA.
• Documentation - Contains the documentation in HTML format.

Notes

The algorithm has been developed on a PXI-7831R board which uses the Xilinx Virtex II 1M Gate FPGA. The provided example almost uses up the full size of this FPGA due to its small size but should comfortably fit on all the other FPGAs in National Instrument's range.