Message stops sending after short time

[wrkcrw00]

I have an actor that continually (10 Hz) sends updates to it's UI and a proxy actor when it's "on-point". It's for a thermal chamber and the Boolean indicates when the process variable has reached the desired temperature. The UI updates just fine. The abstract message that I send to the proxy updates ok for about 30 seconds. Then it apparently just stops sending the message. On the other side of the proxy is a synchronous interface class. 

In the included snapshot, I show the proxy method that consumes the message and the update method that sends the abstract message. Note the probes showing the last updated time. For the first 30 seconds or so of operation, everything updates normally. Then the proxy method stops executing, as denoted by the last update time being 2 minutes behind in this snapshot. 

The update method is still executing, and completes execution so I would assume the abstract message is being sent at 10 Hz. Any ideas why it would just stop receiving the message?

[p4keal]

Have You checked, is the target actor still alive and the enqueuer still valid?

[wrkcrw00]

It's still alive. Sometimes it will pass the data again but only very briefly. 

[p4keal]

Could it be possibly, Your target actor has some time consuming actions?
Does it have special error handling?

[justACS]

What do you see in DETT?  I would expect to see that the sender is still sending the correct child of the abstract message (recall that you can see the name of the message class in the user trace event) and that the proxy is receiving same.

Given what you have described, I would expect that you won't see any receive traces from the proxy.

Also, when you do the DETT trace, turn on error events.  Your sender is discarding errors from the Send Message VI, so if the enqueuer has gone stale, you will never know.

Assuming no errors, and you seen send traces but not receive traces, then I would look very closely at whatever *else* the proxy is doing.  My guess is that the proxy is  being blocked somehow, either because another method of the proxy is taking forever, or because one of the shared resources is being locked by the synchronous class (i.e. a DVR or some other resource is being held open)?

Are you testing this actor hierarchy in its own test harness?

[wrkcrw00]

In DETT I can see the app sending the correct message to the proxy at the expected frequency (verified it's the correct child message). It's consistent. For a short time, now on the order of seconds, DETT shows that the proxy actor receives the message. At some point, it's obvious that the app is sending the message to the proxy but the proxy is not receiving it. 

Finally, after I quit the program, DETT shows two oddities:

1) The proxy receives the Stop.msg and then shows several lines that say it dropped the update message.

2) After the proxy actor has stopped (after is shows in DETT to have sent last ack and to have stopped), a new actor (with the actor name as a number as if the name had been left blank at launch) shows that is has dropped the update messages sent by the app. 

It's very odd that DETT has this new Actor with the default name as its reference number. DETT does not show that actor as having been launched or ever to have received a message until it shows the dropped messages after the program is ended. 

There are no errors (well, one at the very end after I shut down the program - but thats a different problem) and I can probe the proxy actor and get no errors. 

The proxy only does one thing, this update message. It uses a queue to send the boolean back to the synchronous vi. I'm running the app and proxy from a test harness specifically for testing. 

[justACS]

On your two oddities:

1)  I see this behavior when message processing takes longer than the rate at which messages are received, and the stop message has higher priority than the other messages.  The stop jumps ahead of the waiting messages, the actor stops, and then it drops the rest of the messages.  Are you sending an e-stop?  Does the shared queue have a finite size?  If it is full, then you'll have to wait to add an element to it until the synchronous API dequeues an element.  You're only sending this message at 10 Hz, right?

2)  That looks like a race condition between the stopping actor and the DETT support code.  If you look in actor.vi, you can see that the actor unregisters itself in parallel to dropping messages; in your case, the unreg is winning.  Since it's unregistered, you get the actor's handle, not its debug name.  That would be an easy fix, and might be worth a CAR.  I can't file those any more, so I will write to those who still can.  Point is:  don't go looking for a mystery actor; it's a red herring.

[wrkcrw00]

Using the same proxy I refactored slightly how it used and slowed down the rate to 1 Hz (thermal data, so no big deal). That worked. I don't think 10 Hz should be a problem but perhaps it's a timing issue with how other parts of the system works. Anyway, it functions just fine at a slower rate.