Auto indexing for loop memory allocation

[madvillain97]

Hello all, new to the forum and Labview. Been lurking for a few weeks and have seen some great advice!

I built a simple state machine program for experiment logging. In the "Acquire" state, I run 2 nested for loops (N_x := times the stage moves in X, same for Y) and am using a simple Auto indexing tunnel and an indicator outside the loops to get the final array.

My problem arises when doing long experiments, requiring from 20x20x255x1024 dimensions (i.e. moving the stage 20 times in each axis). At this point, when looking at the memory use with task manager, as the program logs the data and I can see that the last iteration reaches 800MB. However, the program stops and prompts a "memory is full" dialog just after the final loop is exited; the data is not saved in the array.

Is it caused by:

• The Auto indexing? I thought that it preallocated the memory and initialised the necessary array on the first iteration. Should I initialise the array first and then use Replace Array Subset? Is an In Place Element Structure better for this case?

• The fact that I'm using local variables of the indicator and should instead use a cluster to send data through the states? Should I change to consumer producer completely? Do I have to redesign the whole code as queued message handler or something similar?

• It being a 32 bit application on a 64 bit machine? I can install the 64 bit version but shouldn't it be enough to handle 800MB of memory?

• A memory leak somewhere in the code? Is the memory usage too large for arrays of that dimension? After the acquire state, it goes to the "Save" state, but I guess it never reaches it because the data is not shown in the indicator.

• All of the above

If the answer is the latter, I would appreciate a bit more information on which issues are causing the most trouble, for further reference.

Any insight is greatly appreciated, as I'm trying to learn as much as I can.

[altenbach]

Autoindexing is safe, because the full size can (and will!) be allocated before the loop starts.

Arrays require contiguous memory, so you will run out of useful memory way before you run out of free memory amount.

Having a 4D array (it seems you are saying that?) is insane. Do you really need an indicator for that crazy big array? That would immediately triple the memory and would probably be completely useless for the operator. Maybe you could add controls to e.g. interactively display a chosen plane in an intensity graph. Orders of magnitude less data on the front panel!.

As a first step, attach a simplified version of your code so we can look for problem. We are graphical programmers and no amount of words can describe your code without ambiguities. What is the datatype of the array? (U8? U16, DBL, etc.?)? How are the loops nested and where are the autoindexing tunnels?

You mention local variables and clusters and I have no idea how that relates to the problem. they don't have the same function and you cannot really substitute one for the other. Of course if you use a local variable for the gigantic array you should definitely not do that.

On a 64 bit OS, your LabVIEW application has access to 4GB of ram. Going to 64 bit LabVIEW will help, but done right, it should run fine on 32bit LabVIEW. Just blindly throwing more resources at a poor implementation is not the way to go.

[madvillain97]

Hi altenbach,

Thanks a lot for your reply.

It is indeed a 4D array. I meant to say that I want to save an array A ∈ ℝ^(n × m × 255 × 1024) (U32) where n and m represent the number of measurements in Y and X, respectively.

The software captures an image with a spectrograph each time the stage moves (hence the 255 pixels x 1024 wavelength dimensions for each image). 

Apologies for my 'clusters' comment, it was a complete misnomer. I meant instead to ask if it is better to use the shift registers with a .ctl as seen in the State machine template, to store the data and access it when I want to update the surface graph by using unbundle by name.

Thanks to your reply, I now understand I shouldn't use local variables for this. However, I was using an indicator to be able to index the 4D array into a 2D array, depending on the chosen pixel and wavelength that the user selects on the FP, and then display it with a surface graph.

It may seem like an incredibly easy question, but when you say I shouldn't use an indicator for the array, do you mean should I instead write the data to disk as a file and then open it when I want to index the 2D array? Or is it possible to use the shift registers and .ctl instead to access the array in every state?

I will attach a simplified version of the code as soon as possible. As a student, I only have access to the lab on Fridays. In the meantime, I hope I expressed my questions clearly enough.

"Just blindly throwing more resources at a poor implementation is not the way to go." I agree wholeheartedly.

Thanks again for your time.