async timer stall behavior

[menchar]

In service to my unending quest to understand async timers ...

I have an app with two async timers, each with its own callback, and I understand both run on the same thread.

So if one of the callbacks stalls, will the other ever get scheduled?  If they're truly on the same, single thread, then if you lock up one of the two callbacks (e.g. put it into an endless loop), then the other callback should never get called.

Thanks.

[menchar]

And to follow up, what if the first async callback doesn't go into an endless loop, but rather goes to sleep?  Will the second timer callback get invoked ?

I have two async timers, both run at 30 sec period, but one gets invoked regularly and the other doesn't.  While I might believe I have the first timer callback hung up somewhere, I don't understand how the second timer callback is getting invoked (which it is).

[menchar]

I am seeing erratic async callback scheduling.

I lied about the timer periods - one is 30 sec, the other 15.

In the app, all async timer callbacks are good until the 30 sec timer spends more than 30 seconds in the callback.  After it comes out of that >30 sec callback, the 15 sec async timer callbacks resume as expected, but I never see another 30 sec callback until I kill the async timers! Then a bunch of apparently postponed 30sec callbacks come squirting out.

Async timers will be the death of me ...

[D_Biel]

Hey Menchar,

One thing to note is that the order in which you create the Async Timers does matter. The events for each timer has it's own queue. So if you have two timers then it will always try to handle the events from the timer that was created first before handling the second timer's events. So if timer1 was declared before timer2, and timer1's callback takes longer to execute than it's interval, then timer2 events will never be handled until timer1's queue is emptied. Ideally, you should avoid taking more time in your callbacks than your interval (this also means your combined time between all timers). This is the classic producer/consumer problem. But if this is unavoidable to take a long time in your callbacks, you may need to create your timers in such a way that the timers' who's callback will not exceed the interval are created first. That way the faster timer's callbacks will have an opportunity to execute.

The following code can help illustrate how this works:

#include <utility.h>
#include "asynctmr.h"
#include <ansi_c.h>
static int time2;
static int time1;


int CVICALLBACK timer1 (int reserved, int timerId, int event, void *callbackData, int eventData1, int eventData2)
{
	 printf("Timer1 start\n");
	 Delay(6);
	 printf("Timer1 end\n");
	 return 1;
}

int CVICALLBACK timer2 (int reserved, int timerId, int event, void *callbackData, int eventData1, int eventData2)
{
	 printf("Timer2\n");
	 return 1;
}




int main (int argc, char *argv[])
{
	time1 = NewAsyncTimer (5, -1, 1, timer1, 0);
	time2 = NewAsyncTimer (2, -1, 1, timer2, 0);
	
	while(1)
		Delay(1);
    
    return 0;
}

[Rohama_K]

Hi,

Edit: didn't see the post above...

[menchar]

Thanks for the info!

Yes, I had seen behavior over long runs (application runs for weeks at a time, constantly using two async callbacks) that caused me to believe there was some unobvious semantic to the async timer scheduling, which you've just explained.  If this is NI implmentation detail, it's never been explained anywhere that I can find.

I do need to prevent the 30 second callback from taking longer than the timer period - it normally doesn't, and in that case everything works as expected.

I create the 15 second timer first, and that's the one that continues to get callbacks after the 30 sec timer stops getting its.

What I still don't have a grip on, is that if it's the 15 second timer that gets priority, and it has enqueued a bunch of callbacks missed because the 30 second timer was stuck in it's callback, why doesn't the 15 second timer flush out all of it's enqueued callbacks once it's able to?  Otherwise, queueing them up doesn't make sense.  I would have thought that the 15 sec timer would flush it's enqueud callbacks out as fast as they were serviced, and then both timers would resume getting their callbacks per their schedules.

Here's my code:

#define POLL_INTERVAL 15
#define CONTROL_INTERVAL 30

// Create polling and control async timers, start callbacks
            
   iPollTimer    = NewAsyncTimer (POLL_INTERVAL, -1, 1, MeasurementPoll, 0);
   iControlTimer = NewAsyncTimer (CONTROL_INTERVAL, -1, 1, Control, 0);

Another thought I have is that this is a fast, 8 core PC.  I wonder if there's some weirdness getting in due to the true parallel execution that can take place, though the async timers are on the same thread so shouldn't be a race there.

When the 30 sec timer stops getting callbacks, the 15 sec timer callback interval gets stretched out to about 25 seconds - up until then, I see the expected, regular 15 and 30 sec callbacks.

Thanks again.

[menchar · Accepted solution]

Got a handle on this finally.

The problem was that that the 15 sec timer takes 16 sec to complete the callback

Since it was created first, and since it can't keep up with it's own schedule, it runs all the time and the 30 sec timer gets starved

What made this hard to analyze is that the time inside the callback for the 15 sec timer only goes to 16 sec after the 30 sec timer makes a state change to the system.  So the problem correlated with the behavior of the 30 sec timer callback but the root cause was the behavior of the 15 sec timer callback.  Correlation isn't causation as they say.

I set up a an LED on the GUI  for each timer that gets set on when the timer callback is entered and off when it leaves, and then I was able to see what was happening.

[menchar]

And here's another question on ASYNC timers - we all know that "async timer callbacks are all made on the same thread", BUT I am wondering is it the same thread from run to run?  I.e., is the sense of this statement meant to imply that there's a reserved thread dedicated to calling all of the async timer callbacks within an application, every time, or is the meaning only that if more than one async timer callback is made at one time, it will be on the same thread?

If the intention is that there's always mutual exclusion between async timer callbacks, if only one is being scheduled, and if the async timer handler is synchronous, then this requirement could be satisfied simply by making sure that if at the time of return from an async callback, if another callback is scheduled, reuse the same thread.

Why do I care?

If I'm locking a mutex in an async timer callback, it's locked by the async timer thread.  If the lock isn't released, and the exact same thread makes the next callback, it won't deadlock since a thread can't deadlock itself on a mutex.  On the other hand, if it's a different thread, it will.

Watching the app with task manager, it looks like there's a new thread being created / destroyed that corresponds to the async timer callbacks firing.

[D_Biel]

The async timer will run in it's own unique thread. So yes, it will always use the same thread. This thread is separate from the thread pool.

You can verify this yourself by filling the thread pool with threads (the default pool has 2+2*numProcessors or check ATTR_TP_MAX_NUM_THREADS). I filled my pool with 18 threads and found in the Threads window that I had 20 threads (18 + main thread + async thread).

[menchar]

Thanks for checking this out.  I see six threads when I run my three - thread debug build from CVI.  I'm guessing some of the extra threads are from the IDE.  And I do see a thread get created and destroyed every few seconds - thought that might be associated with the async timers but I guess not.

I could have captured the thread ID from the async callback and watched to see if it chages, I suppose but I asked here on the forum on the chance that someone might know off the top of their head.

Thanks again.