store a large data base in programe and compare with a new test data

[mjaugustin]

Hi,

In one of my application i need to compare  new test data with prviosly tested test data and which one matches closely with the present one.This poeration need to be done during the test itself with less time and would be implementing in a PXIe(windows based).Waht will be the best strategy for the memory optimised and fast comparison.Will storing of the data base will be a good option ( normally test data in .tdms files will come in terms of GBs)one test data will be 12 colums and mostly 3000 rows.In matlab when Stored as.mat file as a array of structures it is of the size of few MB's.Is there any equivalent data format in LabVIEW, sop that it occupies a small data space.The comparison can be done with built in modules and easily find the likely hood of the data by comparing.But the storing of the data in a smallest possible size format and its easy retrivel looks challengong to me.Can any body suggest most appropriate method for this

Thanks in Advance

[Sam_Sharp]

SQLite database? It's a file (rather than database engine) database engine - allows you run queries on the data for sorting/comparison etc.

[mjaugustin]

Thanks Sam for the suggestion,

I wiil explore the option.Can  we create and read thissql lite file from LabVIEW enviornment?

[Dennis_Knutson]

I don't understand your calculation that 12 columns and 3000 rows will require GBs to store unless you save to a text file. For the least amount of overhead (though tdms does not have that much), you can use the binary file functions.

[mjaugustin]

The tests to be compared will be close to 100 and the row number minimum value is 30000(sorry the value i gave was a mistake ).100 test data with these many rows and colums is coming  close to 1 to 1.5 GB

[Sam_Sharp]

There is an SQLite library available from the LabVIEW Tools Network / VI Package Manager: http://sine.ni.com/nips/cds/view/p/lang/en/nid/212894

The advantage of SQLite over binary files is that you can easily sort/retrieve and run queries on individual items in the SQLite database - for 'fast comparison'. Of course, if you don't need that then binary files would be fine and have a lower overhead.