Split up Large TDMS file, Allow larger-than-memory file sizes

[ColdenR]

Overview
This example shows how to split up a large TDMS file into smaller TDMS files.

Description
This example shows how to split up a large TDMS file into smaller TDMS files. It includes functionality to allow these smaller TDMS files sizes to exceed the computer's memory limitations by splitting up the writes into smaller parts programmatically. So, for example, you could split up a 2 GB file into 250 MB chunks without running out of memory, even if you only have 2 GB of RAM and you're running Windows 7. If you're looking for a simpler example and are not manipulating extremely large files, see this example: Programmatically Create Smaller TDMS Files. Here is a screenshot of the program block diagram:

Requirements

• LabVIEW 2012 (or compatible)

Steps to Implement or Execute Code

1. Download the attached folder to your computer
2. Open the VI "Split TDMS File Memory Independent 2012 NIVerified.vi"
3. Set the path and the parameters
4. Run this VI

Additional Information or References
VI Snippet

 

 **This document has been updated to meet the current required format for the NI Code Exchange.**

[joshe]

Thanks for creating this.  I am getting an error (4) when trying to split a file.  The file is 500MB about 9hrs at 10Hz.  I tried to use a sample size of 72000 (2 hrs @ 10Hz).  If the EOF is happening, why isn’t the EOF indicator on the TDMS Read.vi showing TRUE?  Is there some other reason for the error?  This seems to happen on the 2^nd loop.

Error 4 occurred at TDMS Read in Split TDMS File Memory Independent LV2009.vi

Possible reason(s):

LabVIEW:  End of file encountered.

=========================

NI-488:  Invalid argument or arguments to function call.

Thanks!

[ColdenR]

Hello Joshe,

On which VI in the block diagram is the error occurring? Also, what controls are you using for the front panel (file paths, file sizes)?

[joshe]

It looks to come from the TDMS Read.vi

[ColdenR]

What value did you wire for "Group name in"? It's set to "test" in the example, but you have to change that group name to match the group name of the data you're reading from the TDMS file.

[Titri_SES]

Thanks for you VI.

I am using it in order to split TDMS files which have a size superior to 200 Mo.

However, I would like understand how to configure the field "File size (# of samples)" for getting every time a  TDMS file with a size less than or  equal to 200Mo, regardless the size of the original file.

Can you explain me how choose the number to fill this field, please?

Thank you in advance for you answer.

[ColdenR]

The example doesn't split by size of file, it splits by number of samples. You could either use a number of samples would yield a 200MB file, or you could write a different program to do the splitting by file size.

[RubbahSlippah]

Hi ColdenR,

Thanks for the VI. Unfrotunately, I am having trouble. I have a large TDMS file (~15 GB) that contains voltage data continuously and simultaneously recorded from 16 channels at a sample rate of 400 kHz. When I try to run this large TMDS file through your VI, no matter what I set the "Files size (# of samples)" value to, the first split file always has a size of 46.76 MB, and all the subsequent split files only have a size of 1 KB. Can you help me identify where the problem might be coming from?

[Iron_Bars]

This is perfect! Works as advertised. Thank you ColdenR!

For those of you having problems, make sure you:

• Use the full path and file in the "Large file" field.
  •  C:\Users\ACV\Desktop\ACB\big_file.tdms
• Populate the "smaller file" field. You don't have to include the extension.
  • C:\Users\ACV\Desktop\ACB\big_file_split_
• On the block diagram:
  • See attachment for my customizationVI with minor modifications..