Use of Stop Core - Separate Shutdown method?

[MartinMcD]

Hello there,

Am I correct in thinking that my actors should have a separate shutdown message/method in addition to Stop Core? This would close down any instrument comms etc before itself calling Stop Core.

I initially thought that I should just do this in the override of Stop Core but there is no error handling so I thought perhaps it is not intended for that.

Thanks,

Martin

[LVB]

IMO, Stop Core is intended for shutdown.  A well designed actor should attempt to release all resources in the Stop Core method AND shutdown.  If the resources are invalid or cannot be released, the error message can be collected in the Last Ack message OR you can trap the error and handle accordingly in Stop Core (try "n" times).

[AristosQueue (NI) · Accepted solution]

Stop mean that your actor is stopping. Do whatever you have to do to stop.  There is no error handling built in because it is so common for the whole system to be stopping at the same time and often the caller has already shut down, so there would be no one to receive your error message. Also, Stop Core will be called even if the actor never got off the ground in the first place, so many of your errors may be coming from things that are already reported to your caller.

If you know your caller will still be running, you can send them messages in your Stop Core.vi without problems. That works just fine.

There's no reporting built in because

a) shutdown is often triggered by another error in the system that is already being reported

b) the caller is often already shut down (because it told its nested actors to shut down in response to receiving its own Stop message) and it seemed poor to build in a reporting system that people might think was available always.

You can override Handle Error.vi. The Actor:Handle Error.vi says "on any error, stop. Clear the error if it is error code 43 (the "cancel" error, which is generally used for backing out an operation cleanly). If you wanted, you could put your resource deallocation code in that function and then return a different error to the caller than the one that is causing the shutdown (i.e., you might clear error code 43, but then go ahead and return a different error for the shutdown). The error would be returned to the caller in the LastAck... of course, you still have to think about whether your caller still exists or not to get the message.

Having said that... think carefully about errors that you would even attempt to report. Unable to close a communications channel? Why? If the error is because it is already closed, you don't want to bother reporting that. Whatever may have gotten you into that unexpected state earlier, you're now in the expected state, and you generally want to keep going with that state. Remember what I said in the first paragraph about errors that were already reported to the caller? An example of that is if you tried to acquire multiple resources in your Actor Core and failed, so you passed an error into the Call Parent Node, that is already going to be reported in the LastAck, and you wouldn't want to double report the error or report the errors for all the secondary resources you never bothered to acquire.

[Daklu]

There's no reporting built in because...

b) the caller is often already shut down...

A bit of a tangent here not related to built-in error handling...

Stopping actors from the top down seems a bit dangerous to me.  There's always the chance a low level actor might not stop correctly and, assuming the higher level actors did stop, the developer could easily miss the problem with the low level actor.  I'll propogate stop messages from the top down, but I typically don't allow an actor to actually exit its message handling loop until all its sub actors have sent Stopped messages.  (Roughly analagous to the AF LastAck message.)  During development that lets me know immediately if an actor is misbehaving.  It also keeps all "leaf" actors connected to the UI, so I can easily float all the Stopped messages up to a debug window on the UI.  Finding the faulty actor is simply a matter of finding the one that didn't send a Stopped message.

I'm still not familiar enough with AF to know what the best way to achieve the same "bottom up" shut down behavior is.  Would you monitor the sub actor's status as part of the calling actor's Stop Core message, LastAck message, or do something else?

[AristosQueue (NI)]

Daklu wrote:

I'm still not familiar enough with AF to know what the best way to achieve the same "bottom up" shut down behavior is.  Would you monitor the sub actor's status as part of the calling actor's Stop Core message, LastAck message, or do something else?

There's a couple variations. I'll give you the most paranoid version, the one I think I would use if I was really concerned that my nested actor might not stop or might have trouble stopping and I had to do something more creative to bring it to a close.

I would send a message to my nested actor that says "Need To Stop" (not the standard Stop message), set a status flag in myself that says "I have sent this message", and then I would fire a time-delayed message to myself for "Handle Nested Actor Still Not Stopped Msg". I would then go back to my regular message handling loop. If you haven't heard of time-delayed messages, check out the Advanced palette of the Actor Framework. It's in there.
One of three things would happen (in the paragraphs below, the pronoun "I" refers to the caller actor):

1. My nested actor successfully stops and I get the LastAck message before I receive my time-delay message. I would set a flag in myself that says "don't worry about this any more." When my time delay message finally does arrive, it would see the "don't worry" flag and would do nothing. I would *not* try to cancel the time-delay message. It may have already sent and be sitting in my queue, so I would have to have the "don't worry" flag anyway, so I might as well just use that for all cases (and it saves me having to store the notifier anywhere).
2. My nested actor has problems stopping and sends me a message describing the problem before I receive my time-delay message. In this case, I would again set a flag for the time-delay message, but this one would say "resend yourself". Then I would do whatever I could to stop the nested actor (that will vary based on my particular application... I may just need to wait longer for it to clean itself up). Then go back to message waiting. When the time-delay message does arrive, it sees the "resend yourself" message and sends itself again with another time delay. Repeat this until there is nothing left that the caller can do to help the nested actor stop. Eventually I would send an Emergency Stop message to the child to tell it "shut down no matter what state you're going to leave hanging" and then take over its hardware myself, if that is necessary. During this whole process, I may have to send a message to my own caller that says, "This is taking longer than expected, don't worry about me", so my caller also executes case #2.
3. My nested actor is silent. Eventually I receive my time delay message. In this case, something has gone wrong with the child actor and I got nothing from it. I implement whatever emergency routines I have to help it shut down (I think of this as the "shake the pinball table" manuver for when the ball gets stuck but you can't see where it is stuck). If that doesn't get a signal from the child in reasonable time, send it one final Emergency Stop message and then take over its hardware shutdown myself. The Emergency Stop message will be received with a priority ahead of any other message in the system, so if the nested actor ever does shake free, it won't do anything further.

After doing all this, if I now need to shut myself down as part of a bottom up shutdown, I would send a Stop message to myself, close my own bits and pieces, which will eventually send a LastAck to my caller (so that I trigger case 1).

[Daklu]

Thanks for the detailed response.  Since I rarely use dynamic actors I'm mainly concerned with detecting shutdown errors during development, so I take the simple route of just keeping track of which sub actors have sent Stopped messages.  Still, your logic is much more robust than mine.  In situations where the risk of released code not shutting down correctly is unacceptable I think your process is mostly sound. 

The one sticky part I'm seeing is in section 3, where you "take over its hardware shutdown myself."  Maybe I'm missing something, but at first glance having the calling actor take over the sub actor's hardware shutdown breaks encapsulation.  It requires the calling actor to know details about the hardware the sub actor is using.  Any ideas for how to resolve that without overly complicating the code?

[AristosQueue (NI)]

Daklu wrote:

The one sticky part I'm seeing is in section 3, where you "take over its hardware shutdown myself."  Maybe I'm missing something, but at first glance having the calling actor take over the sub actor's hardware shutdown breaks encapsulation.  It requires the calling actor to know details about the hardware the sub actor is using.  Any ideas for how to resolve that without overly complicating the code?

The only solution I can think of is when a nested actor spins up, it sends a message to its caller that says, "In the event of an emergency, run this." Whatever is in that message's payload has instructions for emergency stopping the hardware that the nested actor would normally shut down. This would allow for a case where a nested actor is (accidentally, I hope) stuck in some long operation where it is ignoring the instructions to shut down and the caller could do the emergency stop itself without the caller knowing exactly what it was doing.

Having said that, for these sorts of highly paranoid systems, I am actually more likely to encourage tight coupling of the caller and the actor. Code reuse is great for reusability, but for paranoid systems, I want the software checked out for this specific use case, reusability be damned, and introducing code into the caller specific for that nested actor would be a much smaller concern for me. That's my personal software development philosophy, but I've only worked directly on one truly critical system in my life (indirectly by LV being included in such systems doesn't count).