I'm thinking of formalizing verification of the Stop message in next rev of AF

[Brainstorms]

Which additional messages? 

Should a tree of actors collapse from the ground up, or from the head down?  What is the "more stop work" that needs to be done?  Can this be abstracted to result in framework mods? 

Or is it too application-specific?

In some sense, your goal of "A Stop message means 'do what you need to actually stop'" coupled with "the nested actor is treated as [imminently] dead" is already covered by the existing framework: Send a Stop Message.

If that holds true at the top level, then, given the fractal nature of an actor tree, it holds true at all levels, and therefore each actor only needs to attend to its own internal shutdown state activities.

And therefore the existing framework already addresses this...  And is still unsatisfactory.  Does our "itch" actually have a possibility of being "scratched" a priori through a clever framework implementation?  I'm not convinced...

[Jeannius]

Instead of trying to cover all cases and abstract away, why not just do the minimum (at least for now).

I propose we just set up a flag in the caller actor and an option "Don't shut down until the child actor send last ack or set the flag to true".  This way we still have the option of orderly shut down without the headache (leaving error handling to the designer based on his or her special case)

[kegghead]

AristosQueue wrote:

Important note: After a caller sends Stop to a nested actor, the caller should treat the nested actor as dead.

I agree, though to be clear, you're not implying the caller should maintain this information are you? This sounds to me like the responsibility of the nested actor where it should ignore/error/whatever if it gets an out of context message after a Stop.

[AristosQueue (NI)]

kegghead wrote:

I agree, though to be clear, you're not implying the caller should maintain this information are you?

Honestly, I'm not sure. The caller has to maintain that information for the duration between when caller sends the Stop and when the nested starts handling Stop (which might be an indefinite amount of time), so saying that the caller has to keep tracking it isn't necessarily wrong. Obviously it would be better if after sending the Stop if further messages errored out, but that's only reliable if it works all the way back to the *send* of the stop, not to the *handling* of the stop.

[Brainstorms]

Here's one idea:  Define a message to a nested actor that notifies it of the need to prepare for shutting down.  This message type can be part of the framework.

On receipt, the actor does what is necessary to place itself in a state that is compatible with "pulling the power plug".  This would include shutting down subordinate actors, messaging anyone who needs messaging, putting hardware into safe modes, closing files, etc.  Whatever is needed for a graceful shutdown...

While in that state, the actor could accept messages as it normaly does -- and choose to refuse to handle them, return a response that it's unable to respond, triage them and handle only what applies, etc.  Consider putting this "alternative message handling" logic in the framework as well.

Once imminent shutdown (the "quit" variety of meaning) is ready, the nested actor can send a message to its caller to notify it that "all the hatches have been battened down" and it (and its subordinates) are ready to die.

At that point, sending a Stop Message would proceed to collapse this nested actor sub-tree, with no additional logic or handling necessary, and with a safe ability to "cut off all communications".  The only remaining processing, which would not require communications, would be the processing of the Last Ack Message received from below (likely retrieving & saving relevant state data) -- implying that the tree will, in general, need to deflate from the leaves inward, so that this Last Processing can occur in an orderly manner.

When the caller receives a Last Ack from its nested actor, it knows at that point that the nested actor & its tree are gone.  Any remaining relevant state information from the subtree would be contained in the Last Ack object.  (Being fractal in nature, this occurs at every level as any actor tree/subtree is shut down.)

[drjdpowell]

AristosQueue wrote:

Consider this case: You send Stop to an actor representing some complex bit of hardware like a digging machine. To bring itself to a stop, it has to move itself to some sort of storage position. Doing this move involves messaging many of its subactors, waiting for the subactor to reply with "ok, I did that" and then sending the next instruction to a different subactor. In other words, the digging machine actor is in its "I am stopping" state but still has to be able to receive and respond to a wide range of messages while it finishes its work. While in this state, all messages coming in from callers or other actors (if you've shared the queue around) should be rejected as "too late, I'm already in the process of stopping." Rejection means either ignoring them or sending some sort of error message reply.

That means you have to somehow know which message types are ok during the shutdown process and which are not.

The most obvious solution involves a VI that does type testing on the incoming message to check its type is acceptable. That's both slow and a nasty maintenance burden.

Another way that I've heard suggested to achieve this is for an actor to send a message to its nested actors saying, "Here's a new queue... use this to talk to me instead of the old one." Then close the old queue. Now the only actors who can message you is your own nested actors, so you assume that they'll only send you messages that are valid during shutdown because you've told them "you're now shutting down, too." Obviously, this would have to be baked into the Actor Framework since the caller queue is private data that no actor is allowed to modify.

There's other solutions as well that would need to be explored.

This conversation has lead me to a greater appreciation of the differences between the Actor Framework and the actors of my "Messenger Library".  My actors have an incoming message queue, but for shutdown (and initialization) they aren't handling messages on the queue.  Instead, they are using synchronous methods, "send message and wait for reply", to communicate with subactors.  But I can only do that because of two differences with the AF.  One, my subactors communicate with calling code only via "Reply" and "Notify" API calls, neither of which provide any feedback to the subactor (who has no idea where its messages are going).   Thus, no subactor depends on its caller handling its messages.  Two, though calling-code sets up any communication between subactors (using Registration messages), the actual message-passing is direct and the Caller does not act as an actual Message Router, as in the AF.  Thus, the Caller can switch to working synchronously with subactors without interrupting communication between those subactors.

The result is I have "initialization" and "shutdown" actions, rather than "inializing" and "shuting-down" states.  I can see why the later can be needed in the AF, but this seems to me to be potentially quite complicated.

[kegghead]

drjdpowell wrote:

AristosQueue wrote:

The most obvious solution involves a VI that does type testing on the incoming message to check its type is acceptable. That's both slow and a nasty maintenance burden.

Another way that I've heard suggested to achieve this is for an actor to send a message to its nested actors saying, "Here's a new queue... use this to talk to me instead of the old one." Then close the old queue. Now the only actors who can message you is your own nested actors, so you assume that they'll only send you messages that are valid during shutdown because you've told them "you're now shutting down, too." Obviously, this would have to be baked into the Actor Framework since the caller queue is private data that no actor is allowed to modify.

There's other solutions as well that would need to be explored.

My actors have an incoming message queue, but for shutdown (and initialization) they aren't handling messages on the queue.  Instead, they are using synchronous methods, "send message and wait for reply", to communicate with subactors.

I've traditionally handled shutdown must as JDP has, that is to say synchronous. How you bake things in though is really a matter of preference.

As far as keeping things Actor based, remember anything communicating with the Actor really knows nothing about the Actor proper, they only have an interface to a communication mechanism. As such I'd handle things more like a state machine: have the shutdown message swap out the actor in the loop for a new one whose sole job is to handle shutdown.

[AristosQueue (NI)]

jdpowell: I can pretty much guarantee that for an actor that abstracts some sufficiently complex system from its caller, you're going to need a shutdown state. Just because you've told some subsystem to shutdown doesn't mean your caller gets to stand there and wait for that shutdown to complete... it's got other things to be doing (after all, it might not be the whole application that is shutting down, just one little piece).

kegghead:

> have the shutdown message swap out the actor in the loop for

> a new one whose sole job is to handle shutdown.

That doesn't solve anything about the communications problem. The new state still has to have live communication to the previous state's nested actors, still needs to eventually give a last ack to the caller, and still needs to be able to handle messages while shiutting down. Changing a private implementation detail doesn't change anything about the response of the interface. Ideally, the interface itself needs to change.

[kegghead]

AristosQueue wrote:

That doesn't solve anything about the communications problem. The new state still has to have live communication to the previous state's nested actors, still needs to eventually give a last ack to the caller, and still needs to be able to handle messages while shiutting down.

I'm not seeing "the communication problem", though I agree completely with the requirements you've laid out.

AristosQueue wrote:

Changing a private implementation detail doesn't change anything about the response of the interface. Ideally, the interface itself needs to change.

Just to be clear, is this a reference to your earlier message about to swapping out the queues?

AristosQueue wrote:

"Here's a new queue... use this to talk to me instead of the old one." Then close the old queue. Now the only actors who can message you is your own nested actors, so you assume that they'll only send you messages that are valid during shutdown because you've told them "you're now shutting down, too."

To me this presents the problem of having to manage keeping track of which messages are valid or invalid. I still need to be constantly checking state when it comes time to act on anything that has the potential to step on toes depending on run/shutdown.

if shutting_down() then do_something() else do something_else()

Repetitive logic like that in an object oriented framework gives me nervous ticks. Hopefully I'm not coming off as agressive or overly critical, my intention here is definitely to be constructive. I'm a firm believer in having more eyes on the project is a good thing and I really want to see the framework mature.

Hence my suggestion for actor swapping. Treat the actor as a state and a whole swathe of messages that were intended for the previous actor may no longer operate on the new one. That's one way to solve that "slow and nasty maintenance burden" on the receiving end-- if the message only acts on ActorX but not ShuttingDownActorX then so be it, that message will just error out. Alternatively look at the problem from the sender side of the pipe-- if I have an actor dedicated to shutting down the question never needs to be asked at all. The sole responsibility of this actor is to manage the shutdown so it doesn't even need to be aware of the responsibilties/messages of the previous actor.

[AristosQueue (NI)]

kegghead wrote:

To me this presents the problem of having to manage keeping track of which messages are valid or invalid.

Nope (or, at least, I hope not). If you are an actor and you get a Stop messsage then ALL messages your caller tries to send are invalid. ALL messages from your nested actors try to send are valid until you actually halt yourself. That's the only valid interpretation of "Stop", and it's the situation I'm trying to get us into.