NI-DAQ continuous acquisition race condition

[junkhead]

I use NI-DAQmx with PCI-6132 card to acquire data from 3 chaneels simoteneously at 1MHz sampling rate continuously. I used the "unscaledReader" to read 2 bytes for each sample. The code like this

        private int samplesAcquired = 0;

        private void OnDataChunkAcquired(IAsyncResult iRes)

        {

            try

            {

                if (iRes.AsyncState != null )

                {

                    short[,] data = mSLOData.EndReadInt16(iRes);

                    int length = data.GetLength(1);

                    lock (data)

                    {

                        Buffer.BlockCopy(data, 0, SLObuffer, BufferIndex * BytesOfShort, length * BytesOfShort);

                        Buffer.BlockCopy(data, length * BytesOfShort, Xbuffer, BufferIndex * BytesOfShort, length * BytesOfShort);

                        Buffer.BlockCopy(data, 2 * length * BytesOfShort, Ybuffer, BufferIndex * BytesOfShort, length * BytesOfShort);

                        BufferIndex += length;

                    }

                    if (BufferIndex >= SLObuffer.Length) BufferIndex = 0; //circular buffer

                    if (DatapointsPerFrame != -1)

                    {

                        samplesAcquired += length;

                        if (samplesAcquired >= DatapointsPerFrame)

                        {

                            samplesAcquired -= DatapointsPerFrame;

                            if (SloFrameReady != null)

                            {

                                FrameDataReady.Set();

                            }

                        }

                    }

                    mSLOData.BeginReadInt16(datachunk, analogCallback, mSLOTask);

                }

            }

            catch (DaqException e)

            {

                Logger.Error(this, e.Message);

                throw new HardwareException(e.Message);

            }

        }

I use the BeginRead and EndRead to get data. If I just copy the data to a buffer, it is running fine. But if I try to process the data in real-time in another thread. I got the exception:

Measurements: Attempted to read samples that are no longer available. The requested sample was previously available, but has since been overwritten.

Increasing the buffer size, reading the data more frequently, or specifying a fixed number of samples to read instead of reading all available samples might correct the problem.

Property: NationalInstruments.DAQmx.DaqStream.ReadRelativeTo

Corresponding Value: NationalInstruments.DAQmx.ReadRelativeTo.CurrentReadPosition

Property: NationalInstruments.DAQmx.DaqStream.ReadOffset

Corresponding Value: 0

Task Name: SLO Task

Status Code: -200279

--------------------------------------

Any suggestions?

Thanks

Jeff

[Frank-L]

The reason you are getting this error is because you are not pulling data off the buffer fast enough and the data gets overwritten on the buffer. Why Do I Get Error -200279 from my DAQmx Read VI or Property Node? this has some information on basic steps to prevent this error. This has some useful steps for troubleshooting in labview, but since they use the same driver they should have similar commands. When you are doing processing, likely you are taking too long for processing and data is being lost.  Increasing the buffer size would be a great first step as well as reading the data more frequently.

[pelesl]

I am having the same issue. In my case it is almost surely caused by the fact that I'm taking too long to process data between reads. I suspect this because as I've increased the amount of stuff going on in the processing I've started getting this error. However, the way the buffers work make no sense to me. I read the document Frank linked but it's going to take some time to convert to .NET (VB). Also some of the suggestions like "do your processing in a separate loop" imply by its very nature that you have to build your own FIFO. If you have to have two FIFOs because your first FIFO is getting overflows then that means in my opinion the first FIFO is not configured correctly.

I started my code from the example at 

NI-DAQ\Examples\DotNET4.0\Analog In\Measure Voltage\ContAcqVoltageSamples_IntClk

In my case I'm doing acquisition from several analog channels. I create my channels and otherwise configure and verify the task. Then I do (almost directly from the example):

AnalogInReader.BeginReadWaveform(CardConfig.NumberOfSamples, AnalogCallback, AnalogTask)

Then in my call back I do this (again, from the example):

        Try
            If RunningAnalogTask Is ar.AsyncState Then

                AnalogData = AnalogInReader.EndReadWaveform(ar)

                OutputData(AnalogData)

                AnalogInReader.BeginMemoryOptimizedReadWaveform(CardConfig.NumberOfSamples, AnalogCallback, AnalogTask, AnalogData)
            End If
        Catch ex As DaqException
            MessageBox.Show(ex.Message)
        End Try

 OutputData is the function that's slowly growing as I develop the code.

My sample rate right now is 100 Hz and CardConfig.NumberOfSamples = 10. According to http://digital.ni.com/public.nsf/allkb/E1E67695E76BA75B86256DB1004E9B07?OpenDocument (linked from Frank's link), the driver is creating a buffer for 1000 samples.

I have timed the execution time of OutputData and it's definitely slower than it takes for my callback to get called. So my getting the error must be due to some hiccup somewhere. I don't want to create a second FIFO just so I can call OutputData in another thread.

I tried to figure out why the example code uses BeginReadWaveform to start and subsequently  BeginMemoryOptimizedReadWaveform. Is it just because this is a convenient way to allocate AnalogData?

The real question is---can I manually create the buffer that the driver will use. I have RAM coming out of my ears; I want to buffer for 1 million samples. As I said the code in OutputData is running fast enough for this sample rate so I think if I just have a bigger buffer everything will work out. 

[pelesl]

Here's what I see:

OutputData puts on the screen, among other things, the timestamp of the data point (from the NI libraries).

Right above it I put a label which gets updated with the computer time right when the timestamp is read from NI.

You can see often the timestamp lags half a second or so behind the computer clock. But often it will "sync up", that is, my program catches up to real time. So it seems that if my sample rate is 100, and I'm reading 10 samples at a time, I have 10 seconds of buffered data = 100 data reads. That is a *lot* of delay for something which stays within half a second of real time.

I don't understand how I could be getting this error. 

[pelesl]

Here's another thing I don't get. With a sample rate of 100 and requesting to read 50 samples at a time my callback should be called pretty evenly twice a second. Except I can clearly see it's "syncopated".

I put some Trace.Writelines in my code and you can see the timing. Lines that read "AnalogInCallback calling OutputData" are right when my callback gets executed. The following line is right after OutputData. I know the timing is not super accurate but it's something.

2011-07-15-11:07:05.384: calling BeginReadWaveform.
2011-07-15-11:07:05.914: AnalogInCallback calling OutputData.
2011-07-15-11:07:06.014: AnalogInCallback done.
2011-07-15-11:07:06.414: AnalogInCallback calling OutputData.
2011-07-15-11:07:06.824: AnalogInCallback done.
2011-07-15-11:07:06.914: AnalogInCallback calling OutputData.
2011-07-15-11:07:06.984: AnalogInCallback done.
2011-07-15-11:07:07.414: AnalogInCallback calling OutputData.
2011-07-15-11:07:07.804: AnalogInCallback done.
2011-07-15-11:07:07.914: AnalogInCallback calling OutputData.
2011-07-15-11:07:07.974: AnalogInCallback done.

The callback is definitely being called exactly on schedule; every 500 msec. My OutputData routine seems to alternate between taking some 100 msec to some 400. That explains the syncopation, and may have to do with the on-screen data display. I can easily chuck that into a separate timer if needed.

If I change the code to read 10 samples at a time (screen updates at 10 Hz), then the timing looks like this:

2011-07-15-11:12:02.644: calling BeginReadWaveform.
2011-07-15-11:12:02.754: AnalogInCallback calling OutputData.
2011-07-15-11:12:02.804: AnalogInCallback done.
2011-07-15-11:12:02.854: AnalogInCallback calling OutputData.
2011-07-15-11:12:02.904: AnalogInCallback done.
2011-07-15-11:12:02.954: AnalogInCallback calling OutputData.
2011-07-15-11:12:03.004: AnalogInCallback done.
2011-07-15-11:12:03.054: AnalogInCallback calling OutputData.
2011-07-15-11:12:03.174: AnalogInCallback done.
2011-07-15-11:12:03.174: AnalogInCallback calling OutputData.
2011-07-15-11:12:03.224: AnalogInCallback done.
2011-07-15-11:12:03.254: AnalogInCallback calling OutputData.
2011-07-15-11:12:03.304: AnalogInCallback done.

The time to process OutputData is lower now since there are less points each time. However you can see at 3 seconds 054 milliseconds OutpuData actually held up the process; the callback was executed 20 msec late. But the program immediately catches up because the next OutputData only took 50 msec so the last callback listed was called back on time at 304 milliseconds.

None of this explains why I'm getting the error sometimes after acquiring only 40 seconds of data. With this sample rate/number of samples I must have significant delays without the opportunity to catch up 100 times to get a buffer overrun! So either the buffer is not as big as I think it is, or I don't understand something. 

[pelesl]

Looks like you can manually control the buffer size with Timing.ConfigureSampleClock. I put a huge number in for the last parameter and it told me there wasn't enough memory, so I know it is doing something. I'll run a test with a significantly larger number than I had here to see what happens.