Reading Large Text file

[Munna232]

Hello all,

I need some help. Please help me.

Currently I’m reading Large CAN ASCII (Text format) file (>2GB) with LabVIEW. Here, I’m converting CAN Signals to Physical values and saving into CSV file.

Now, I’m reading data by chunks (10000 lines each) due to memory issues. As of now everything is OK. But I need to do some modifications in this.

While Data conversion, if Signal is not present then taking previous values for each signal.

Now, what I want to do, if signal is stopped after some time and after that Signal is not coming then I need to display NaN.

As per below snap,

I have 3 signals. I need to change old CSV file to New approach.

 

As I’m thinking, I need to check where each signal is Ending (get that Index). After that Index I need to change values to NaN. But, here I’m reading by chunks so, I need to read each file 2times. One time for Signal Ending & other one for Signal conversion.

I think 2 times reading is not best way.

Please let me know if anyone have better idea for better reading.

[GerdW]

Hi Munna,

when you create a new row in your CSV data table you should initialize this with NaN. When you find new values for your signals you overwrite/replace the NaN with the new value. If you don't find new values you have the expected NaN. No need for two-pass analysis…

[Munna232]

Hi GerdW,

How can I replace/rewrite? for that I need to read CSV file again.

Because, I'm not creating complete Data table & at once I'm not writing into CSV.

Reading every 10000 lines , converting & writing into CSV.  

If Data is not present then I need to take previous Index value. at End If same is not coming then I need to display NaN.

[GerdW]

Hi Munna,

the process of "converting & writing into CSV" involves creating a 2D array in memory. Here you can apply the scheme I sketched above…

[Munna232]

Hi GerdW,

Here, two concepts have,

Suppose Signal-A:

At 51th line signal is not coming then I need to take previous value. Same like 101,201, up to 900. After that from Line 901 I need to display NaN. Because Signal-A Stopped at line no.900

[GerdW]

Hi Munna,

how do you know it stopped at row 901? What is the difference to row 51?

Please provide ALL details needed to solve your problem!

[Munna232]

Hi GerdW,

Generally In CAN Network, Some CAN IDs will stops (Sleep) after certain time but other Signals will still continue.

So, when signal is stopped after certain time then I need to display NaN as output in CSV. Before that if signal is not present then I need to take previous value.

In my Snippet, Signal A is coming at every 50 lines. It means line no. 50,100,150,200 etc. but If I want to see value at line no. 51 then I need to take 50^th line value (because no value so taking previous value). But at 901 I need to display NaN because Signal A is stopped at line no.900.

I hope this is clear now.

[GerdW]

Hi Munna,

so you know the send interval of your CAN messages. Keep the last value for the time of this interval, after the time is expired you replace the last value with NaN.

Still your description is flawed: How do you know the signal stops at row 900 when the message could send a new value at row 950? Again: How do you know for sure it stops at row 900 while you are analysing row 901?

[Munna232]

Hi GerdW,

As per Snippet, I’m reading the Last Index of each Signal from Text file. For certain file its 900. It will change file to file.

After last Index is found (an example 900), for remaining lines I need to display NaN.

I have only CAN Text Files. From that I need to check where each signal got stopped.

Note:  900 is not fixed Line. It will change file to file.

[GerdW]

Hi Munna,

so you know start and end of signal data before loading your text files and also before starting to create CSV files.

What's the problem in keeping a counter of the rows created in the CSV file and to select either NaN or last value for a certain signal depending on all that knowledge you have from your input files?