a bug with XML?

[jumpordie]

I've made a TXT-file and a XML-file of both 4 1D arrays and each array has about 60000 elements.
So something like this:
1                          1                           1                     1

2                          2                           2                      2

...                          ...                          ...                    ...

60000                60000                    60000           60000

When I read out the 4 arrays out of the txt-file it takes only some ms (or max a second)

but when I read it out from the XML files it takes some minutes!!!!
Is this a bug?

Of course XML will be not that fast because the file is bigger but the difference 1second to minutes is quite impossible I think.

Finaly XML is also just text...

Message Edited by jumpordie on 03-07-2006 02:59 AM

Message Edited by jumpordie on 03-07-2006 02:59 AM

[jumpordie]

Have a look at the bug.jpg... It's just a simple read-out, isn't it?

Message Edited by jumpordie on 03-07-2006 03:01 AM

Message Edited by jumpordie on 03-07-2006 03:02 AM

[becktho]

Hi

I just tested this similar to your description. The reading of the data (using the read xml-file function) did not take a very long time - I got problems when displaying the read data. It did not matter how I wanted to display it, I always had to terminate LV as it did not react anymore.

Of course, using xml involves parsing, which slows down - but I can't imagine that this slowdown is so high (reading and displaying a xml-file with 240000values, as described earlier, took such a long time, that I interrupted it after about 20 minutes).

I wonder what the guys from NI could tell about this.

[falkpl]

I dont think XML is intended for storage of large amounts of ASCII data since you will need to do huge amounts of parsing on the data and this will slow down and baloon up the memory required to hold such data.  It would be more efficient to store the data in a custom tag and keep the data in a more compressed format like:

<data rows = 4 columns = 60000>lots of binary data</data>

I am still making the transformation to XML and dont understand the full scope of how to implement it but I have found similar issues when dealing with large data sets such as the one you described.

Paul

[becktho]

I agree with you Paul, that XML is not intended to store that amount.

But still it is strange that it takes more than 20 minutes to process "just" a 11MB file.

Thomas

[falkpl]

remember that you are parsing the data and then building up a tree structure in memory this structure will have 4 branches and will be 60000 deep, not very efficient method of storing data, a good way to make your browser cry is to open such a file. 

Paul

[jumpordie]

Thx everyone for you reply... It's clear now that it's best not to use XML for huge data storage.
But you say that with XML there is built up a tree structure with branches etc...
How this is solved with TXT? Will there be no tree structure used with TXT?

[becktho]

Basically there is no tree-structure in either the XML or the TXT file (relly basically a xml-file is just a txt-file).

XML becomes what it is when opened with appropriate tools - for example a browser. These tools parse the data in the file and identify it as xml. The tree structure is built subsequently, accoring to the structure given in the file. If you open the xml-file with notepad or something similar, it's just a text-file.

Thomas

[falkpl]

Thomas,

Thanks for the clarification, I am still teaching myself XML since it is become the de facto standard.  I was thinking of the DOM which looks like the data is stored and manipulated in a tree structure, is root with child nodes... but yes it is really just a text file.  when it is parsed I thought is is translated into a logical tree structure in memory so that transversal is relatively easy.  I assume that when opening the files in a parser can be a slow translation process since it would require much wore processing than a typical serial file.  Either way I hope lo learn much more about XML which plays such a small part in the LABVIEW's developers life but has become the backbone of many other new technologies.  I must admit that I am still using .ini files and tab delimited data storage but am trying to change this habit towards XML and database storage to conform to what other industries use.

Paul