Fast serial readout of channel group properties

[Phex]

Hi, could someone please advice what would be the fastest way to extract channel group properties of around 400.000 files? I currently use DataFileLoadRed with only the channel group I am interested in and the channels stripped down to a single value by setting the no. of intervals to 1 and SectionBegin/End to 1/2. Any improvement suggestions? Phex

[RamonaL]

HI Phex,

it depends what you want to do with the properties.

If you are really only interessted in the properties of that many channels you can search for them and then just load the property. So you get one Channel with for example all the names of your channelgroups.

To load these propertys you can use this commands:

Call Navigator.Display.CurrDataFinder.QueryForm.Search()
Call Navigator.LoadProperty ("Test_Operator", Navigator.Display.CurrDataProvider.ResultsList.Elements)

 First you enter your search criteria (here done manually, but you can do that by script as well), then you search, and then you use the Navigator to Load the properties. 

This will work best when the properties you are interessted in are optimized (Einstellungen>Datafinder>Zusatzeigenschaften optimieren /should something like Settings>Datafinder> optimize extra properties in english).

I hope this helps,

RMathews

[Phex]

Thank you RMathews, this sounds like a pretty fast solution, although I am really struggling to get it working mainly on two aspects right now:

1. the number of search results seems to be limited to 32.000

2. even if I ask for 32.000 results the search will interrrupt with a time out error (setting the time out value higher does not seem to have any influence)

Could you please give me some guidance here? I have not worked with the Navigator object before. Say, e.g. I want to extract all group names with the correspondant description for my 400.000 files. How would I have to script that?

[RamonaL]

Hi Phex,

this is the max. Number of results you can get.

So what you must do is use several search criterias for example:

1. groupproperty

2. Path of the file

Of course you have to make sure, that you will get all the data you need.

Then you will get several channels and can connect them again together. This should still be much faster then loading the data.

Cheers,

[Phex]

I was able to find a search criteria that allows me to get each query search under 32.000 results and to reunite the values afterwards per ChnConcat, but there is still the time out problem. Although I set the query timeout to a rather high value, the error message pops up after about 20 seconds. This happens either when running the script as well as when running the query manually. It looks as if the timeout setting does not take effect.

 

[RamonaL]

When does the timeout happen? Anytime you are searching?

Are you still getting the results? -> its seems so.Which version of DIAdem are you using?

When I was searching, it was working fine, so this cant be a general problem. Is there a way, that I could reproduce this? Like searching the example data (which is not a lot data of course)?

[Phex]

The timeout does not happen all the time and I was not able to find out why it happens. Strange enough in one of ten cases it works, without having changed any settings. If the error occurs, I do not get any results. I am running DIAdem 2012.

A workaround I just applied is to narrow the individual queries even further down, so that I do not get more than a thousand results each time. But of course this implies a significant loss of speed.

[RamonaL]

Is it always the same search that times out or can it run and later it times out again?

Are your properties optimized and also all your files added to the Datafinder?

[Phex]

Yes it is one particular search that times out in 9 of 10 runs.

Yes my properties are optimized.

Yes my files are added to the DataFinder.

[AndreasK · Accepted solution]

An alternative would be to open all files with the folowing code.

Option Explicit

dim groupindex : groupIndex = 1 ' you can also use a groupname instead
dim groupProperty : groupProperty = "description"

' include this in a loop over all your files
dim filePath : filePath = "C:\temp\example.tdm"
dim store : set store = navigator.ConnectDataStoreByParameter("TDM", "<filename readonly=""yes"">"& replace(filePath, "&", "&amp;") & "</filename>")
dim propVal : propVal = store.RootElements(1).Children(groupindex).Properties(groupProperty).Value
MsgBox propVal

 The files are opened and your properties are extracted. Potentailly the datafinder approach should be faster but if you do not get it work this may be a way out.