ArT_Actors: A Step beyond Actor Framework 3.0?

[AristosQueue (NI)]

Your objection that the Batch Message cannot be pre-empted by higher prirority messages is valid. There is no way to construct a message within the current AF that allows a batch to guarantee "no one at the same priority can interrupt me but higher priorities can interrupt me".

However, this is absolutely not any different from a method of a class that takes a long time to run in response to a non-batch message. And the solution to both problems is similar.

If I wanted to design a system where a batch message could be sent (thus guaranteeing it is contiguous) but still be pre-empted by higher priorities, I would do the following:

1. Create 1 message at normal priority that says "these are the tasks to perform".
2. When that message is received, pass that list of tasks over to *another loop* to perform the tasks.
3. Have my main message-receive loop continue waiting for more messages. If the next message is another batch, add it to the other loop's queue.
4. If the next message is a high priority message, signal the other loop to pause (pick your favorite technique for doing this... the easiest might be using the Enqueue At Opposite End primitive, since all we need is a simple "pause" signal). Handle the message, then signal the other loop to proceed.

Objections?

PS: Point #5 in the white paper, regarding pausing, is exactly the same as this. The handling loop is not the thing that Runs. The secondary loop in Actor Core does the running. This leaves the handler loop free to listen for an Emergency Stop message, or any other message, should it arrive. The 2012 shipping example does this for the Fan system, which has a separate control loop that is toggled on/off with a standard priority message, but an emergency stop will still get heard and will shut down the loop.

[Dmitry]

drjdpowell wrote:

I must admit that I have mostly used waiting on a reply in adding actors into older non-actor code.  Plus in initialization of subactors (where an actor can't begin full operation until it has properly initialized its subactors).  Staying asynchronous is the general rule.  Note, also, that the way I implement "actors" already has a strong communication asymmetry: the launched actor never has direct access to its Caller's queue, and can't actually explicitly "send" a message to it.  All communication up the hierarchy is by replying to received messages or publishing information that the Caller can subscribe to.  Neither of those two modes supports replys, synchronous or not.  That's why I'm interested in Dmitry's "Topic" design. 

-- James

An old conversation on thisfor Dmitry's interest if he hasn't read it.   Dmitry: do actors in your framework have a "Caller Queue" to send messages to or do you use the "Topic" for communication back to the Caller?

James,

thanks for pointing me to that old "Get Rid of the "Actor-to-Caller" Queue thread. It is quite relevant to what we have been discussing here. Also looks like I have stepped on the same rake as you did - ending up completely unaware of Actor Tree Pattern and it's central role in AF - after reading the White Paper 2 or 3 times. Moreover, I grossly misused the AF term 'caller' thinking of it as 'just any caller' in possession of Actor Queue reference...

To answer your question: No, I do not have the "Actor-to-Caller" queue passed by the "creator" to the Actor. Just as you do, I see significant value in keeping a SubActor completely decoupled from "caller" context. I think having "Actor-to-Caller" built into AF may send a wrong message (sorry for the pun) to Actor developer - encouraging him/her using this communication channel on a regular basis, thus increasing coupling ...

In ArT_Actors you have two options to communicate back to "caller":

1. Publish a message to a Topic (Actor creates the Topic and "caller" has to subscribe to it upon creating the Actor). I see Topics as a integral part of Actor's API
2. Caller can send commands in "Asynchronous Request" or a "Blocking Request" mode. This results in a one-time private reply defined by Caller and sent back by Actor. I think these are called "Self Addressed Message and "Reply Message" in AF)

Note: "Caller" on the diagram means any caller - not exclusively the Actor creator

I think AQ is right - using Observer (pub/sub) Pattern does not provide you control over the execution order of a published message. When I need to have a specific order enforced I use something similar to a Batch Message. Example - shutting down a composite actor controlling a complex instrument. You can even use a Blocking Message inside the Batch to provide rudimentary control of Subactor shutdown completion (I would prefer being able to have the good old block diagram calling Subactors - but, unfortunately, with asynchronous nature of any actor-centric framework we do not have that option )

I recall having something similar to "Actor-to-Caller" Queue in one of my old frameworks - but never developed a taste for using it ...

[Dmitry]

AristosQueue wrote:

Dmitry wrote:

daklu nicely phrased a major Command Design Pattern feature - "I'll execute whatever you send me" . Thus my AF Actor will execute whatever custom payload somebodysends it. ... There is no single location I can review to check everything related to my AF Actor behavior. I think having an option to enforce Message Encapsulation is quite important ...

There is only one entity outside of the actor that is capable of sending messages to the actor: the caller. (For the purposes of this discussion, the nested actors are part of this actor since this actor knows what it is launching and, presumably, is not just launching random nested actors that it finds laying around in the system.) That means that the caller is what needs to be checked for correct messaging of the actor. And that is the only place that needs to be checked.

If the caller chooses -- emphasis on the word chooses -- to share the actor's queue with a third party, that third party now needs to be checked, but there is nothing compelling the caller to share that queue and no way for third-parties to get access to the queue otherwise.

From an earlier post: I'm not grasping why you're concerned about the transmission of arbitrary messages by callers. In a non-message passing environment, any method that is public may be invoked by any caller.

AQ, I think I figured out the source of disconnect we had on message scoping. In all my prior posts, when using "caller" I did [naively] mean an arbitrary entity (likely another Actor) having access to Actor's Queue reference - not the AF "caller" (i.e. "The Caller") - a composite actor that creates and/or owns and/or uses the "Actor" ...

I looked back at the 3.0.7 White Paper - you, guys, use caller/callee throughout the document, but define it as "THE Caller" and the Actor only on Page 7 (out of 10). I respect your right as AF developers to choose whatever terms you like, but find the choice of "caller" to designate "the one who Created and/or Owns" an Actor quite confusing. I think this term is overloaded and will initially be interpret as 'just a caller' by most future AF'ers. Whoever is reading this post - HONK if you've been confused with "caller" choice at any time

The other term is "Synchronous Reply Message". I used to call mine Synchronous Messages too, but found out that people are often not clear what it means. I changed it to Blocking Message or Blocking Mode (that's what it really is about) and found that it works better. I've seen some posts here using a shortened form - "Reply Message" and must admit it just drives me nuts ...

Another one is "Self-Addressed Message". I would be better to call it "Asynchronous Reply Message" in line with the one above, but it's your choice anyway ... I call my messaging modes - Blocking Reply Mode and Asynchronous Reply Mode. I guess, some people will have a problem with these names as well ...

Now on a more important (well, really really important) matter - the "Actor Tree Pattern". There is only a vague referral to the Actor Tree in the White Paper. I concur with James (he voiced this back in November 2011) - this has to be clearly described in the White Paper. I missed it (surprise surprise) and became aware of this AF Central Dogma only after reading "Suggestion: Get Rid of the 'Actor-to-Caller' Queue" discussion (thanks to James) and your most recent replies in this thread. That was the second cause of our disconnect. Guys - HONK again if you had this issue too ...

Now back to Message Scoping/Encapsulation - in AF, "The Caller" is responsible for defining it's Subactor Scope. It can expand such scope by passing Caller-to-Actor Queue reference to other Actors/Entities. AF Central Dogma discourages such sharing without having a solid use case. Correct ?

In ArT_Actors the Subactor itself defines it's Message Scope by providing public Subactor API methods for sending each Message to Subactor and by exposing only the API's (may have more than one API) it deems necesary to expose. And just as in AF, a 3d party can share API objects with whoever it deems necessary. But it cannot add new methods to a Subactor API.

I think AF approach is more scalable (caller context defines allowed messages), while ArT_Actors are more dilligent (one needs to add methods to Subactor API to add allowed messages). But, in [a rare] case an ArT_Actor needs this ultimate scalability - it may add an accessor for its Message Transport and brace for the "I'll execute whatever you send me" mode of operation.

[AristosQueue (NI)]

I'm still mulling over the rest of your post, Demitry. One line though:

> encouraging him/her using this communication channel on a regular basis, thus increasing coupling ...

When I was looking at all the different user systems when AF was just a vague idea of mine, I noticed something: This coupling almost always exists. Except for the top-level software entity, there is always at least one other module that cares about any other module: the one that launched it and is therefore the one responsible for shutting it down. That spin-up/tear-down relationship was treated like any other relationship, and it is what made most of the other messaging frameworks unstable. When I really looked at those applications, the real pattern of behavior was "there is a higher level software module that manages my lifetime and, more importantly, cares about the job I'm doing or else that module would not have started me up in the first place." The AF formalized that pattern and said, "If this relationship always exists, then let's acknowledge it instead of pretending that modules are these isolated bits of code running in a vaccum." Once you acknowledge that channel as special, it becomes obvious how the rest of the system comes together: as a tree.

The coupling that takes place is more than acceptable -- it is highly desired. Oh, I don't mean coupling on the type of the caller, though that is sometimes useful. I do mean coupling on the fact that the caller *exists*. Someone cares about *me* as an actor. And if I rely on that channel to report status messages, errors, and the fact that I've stopped, that's not a bad coupling.

[AristosQueue (NI)]

I'm tweaking the documentation in a couple places, making sure this information is conveyed much earlier in the template documentation and in the white paper. How does this paragraph read?

The communication lines among actors always form a tree. In Actor Framework applications, there is generally a small stub VI that launches one root actor. That actor becomes the caller actor for one or more nested actors. Each of those nested actors may launch their own nested actors, and so on. These caller-to-actor relationships and the actor tree are fundamental to the design of applications using the Actor Framework. In general, an actor can only communicate with its caller and with its own nested actors. The framework discourages establishing any direct communication across the tree (although you may do so when good reason exists).

[AristosQueue (NI)]

And one other block:

The Actor Tree

Actors form a tree. Most module communications frameworks have more of a listener/observer pattern, where a module runs and publishes information and any other module may subscribe to hear about updates, forming a cross-connected graph of listeners. During the development of the Actor Framework, we looked at many of these communications frameworks. We realized that almost without exception, every module had one other module that was somehow responsible for it. Generally, this other module was responsible for starting the module up and telling it when to shut down. We observed that this key relationship was rarely formalized within these networks, and that lack of a formal relationship was often the source of bugs, particularly bugs where some module failed to shut down when the system was trying to exit. By formalizing the actor tree, it is clear which other actor is responsible for shutting this actor down and, if this actor has an error, which other actor needs to hear about that error.

We also noted that the graph of listeners lead to variations on the echo chamber problem: when one module published a message, the listeners would hear that message in a somewhat random order. Sometimes this was related to the order of registration, but it could just as easily be related to some arbitrary sort order among the modules. The two specific bugs that came from this were:

1. A module would hear a message about some system change and would start acting differently, but a module that is listening to it might not have heard about the system change yet, and until the system change message had echoed through the entire graph, the behavior of the modules was undefined.
2. A module would announce some message. Another module would hear that message and repeat it. And then others would repeat it. Eventually the module would hear its own message repeated, but it had no idea if that was its own message or an original message from someone else, so it would repeat it again.

There are multiple solutions to both of these problems, but they all required extensive coding on the part of users of these communications frameworks. The Actor Framework takes the approach of formalizing the caller-to-actor relationships into a tree and encouraging messages to propagate through the tree. The timing of messages disseminating through the tree becomes deterministic, and message propagation has a general direction to flow, dodging most forms of the echo chamber problem.

When a caller launches an actor, the queue to talk to that actor is returned to the caller. No other code module has any way of requesting or looking that queue up by name. Thus the actor tree is preserved by default. When one actor wants to talk to a sibling actor, it passes a message up to the common caller, and the common caller passes a message down to the other actor. In general, actors are constructed to not even know about siblings -- it is the caller that knows about the siblings and makes decisions about which messages need to be passed along. You can bypass the actor tree by creating messages that pass one actor's queue to another actor. You may choose to do this when those two sibling actors have such a strong interaction that you want them to talk directly rather than passing messages through the common caller actor. The caller actor is still in charge of lifetime issues. Short circuiting the actor is something you should do rarely and consciously. Creating any sort of general repository of names mapping to queue references is ill-advised as it reintroduces the issues described above.

[Dmitry]

AristosQueue wrote:

I'm still mulling over the rest of your post, Demitry. One line though:

> encouraging him/her using this communication channel on a regular basis, thus increasing coupling ...

  ....
The coupling that takes place is more than acceptable -- it is highly desired. Oh, I don't mean coupling on the type of the caller, though that is sometimes useful. I do mean coupling on the fact that the caller *exists*. Someone cares about *me* as an actor. And if I rely on that channel to report status messages, errors, and the fact that I've stopped, that's not a bad coupling.

I am completely with you. This feedback channel does exist (in real life) and Actor owner (i.e. caller) has the right to expect specific feedback data it needs from the Actor. And I do agree that an Actor Tree is the simplest way of representing a system for most small- and mid-size applications and that imposing restrictions on awareness' of a specific Actor to it's immediate owner and subactors allows making strong statements on overall system properties (performance, etc.) ...

But, IMHO, limiting 'physical' access to an Actor to its immediate surrounding does not, by itself, prevent coupling that Actor to specific types of surrounding Actors. AF3.0 (did not look at AF4.0 code yet) does not prevent this type of coupling 'by design' - instead we rely on educating AF developers by creating and promoting best practices and reference designs.

Example:

My team is working on a control system for a biotech sample scanner. It has a bunch of more complex subsystems (Camera, XY Stage, Z,Stage, etc.) and several simple ones like a Shutter. There is a junior developer on the team eager to learn AF and impress the rest of us with how good he is at writing SW. Guess what - he gets the Shutter Actor . This junior guy sits in design meetings and is aware of overall design and Scanner internals. And he thinks of a cool way to implement a Shutter function - which requires sending a custom message to Scanner Actor - implements that message (based on intimate knowledge of Scanner plumbing) and everything is honky dory ...  Well, next month, another team try's reusing this Shutter Actor and runs into a problem because they use a different implementation of Scanner Actor for another line of products.

We did not catch this during development phase (aggressive milestones) and AF3.0 did not prevent a subactor sending 'unapproved' messages to its owner. Another example - that same junior screwed up with Shutter Actor data structures - resulting in Shutter Actor sending a Stop Message to Scanner Actor. This kind of things may take substantial troubleshooting effort on a large system.

Do I see a good solution? Yes I do - instead of coupling Shutter Actor to Scanner Actor via Message Transport (Actor-to-Caller Queue), couple them via a Scanner API specifically designed for getting feedback from an Abstract Shutter.

I think coupling Actors via dedicated specific APIs is good design. And  ArT_Actors support such 'good coupling' (sounds to me like 'good cholesterol' vs. 'bad cholesterol'). In fact, this use case is a major reason for having Actor API class hierarchy in ArT_Actors. What's the difference from AF3.0 ? Small but essential - instead of passing an "Actor-to-Caller" queue to Shutter Actor, Scanner Actor would pass it a Shutter API object - with a list of methods an Abstract Shutter can call on its owner (a Scanner in this particular case). This way Scanner has full control over incoming message types from a Shutter Actor. Now a Shutter cannot (accidentally) send it a Stop Message - it's simply not allowed by the Shutter API. In a same way the team should define APIs for all other Subactors - providing complete control of incoming messages from Scanner subordinates.

And one more thing for folks not familiar with Dependency Inversion Design Principle - it tells that we need to, first, define the interfaces (Abstract Classes in LabVIEW) and then code both, the Top Level Module (Scanner Actor) and the Lower Level Modules (Shutter, Stages, Camera, etc.) to such interfaces. This is a solid way for implementing zero coupling and increasing your code reuse level. From Agile Software Development by R.C. Martin:

"The Dependency-Inversion Principle:

a) High-level modules should not depend on low-level modules. Both should depend on abstractions.

b) Abstractions should not depend on details. Details should depend on abstractions"

Software Engineering sometimes gets really really cool

[AristosQueue (NI)]

Ok. I see where you're coming from. Your objection makes sense.

Now, where were you eight months ago? 🙂

Seriously... at this point, the AF is deployed with LV. We could deprecate the "Read Caller Enqueuer.vi", but we cannot remove it outright at this point. Even if we could, I would worry about the learnability impact. We would need to look at the complexity introduced by the segregation. This teeters dangerously close to the "architecture astronoaut" threshold that turns so many people away from software designs. Under this separation, you still have all the message classes, and you add one new class that is the only class that can actually send them. And you have to introduce an interface class for every pair of actor classes. And you have to downcast the interface object received to the particular interface being used, putting a lot more weight on the developer of the callee actor. Not saying its impossible, but it ain't helping the ease-of-comprehension of the system.

What if we were to start annotating the Message Enqueuer class (formerly the Send Queue class)? Suppose when Actor A launches Actor B, it provided a list of class types as an input to Launch Actor.vi, and that list could be stored into the Message Enqueuer that is given to Actor B. Every time a message gets sent, its type could be checked against the list of approved types and rejected if it isn't on the list. That's a run time guard instead of the compile time guard.

Are there other approaches that give you the division you're seeking?

[AristosQueue (NI)]

One more point:

The prevention of the nested actor from being able to say "Stop" to a caller is a useful limitation created by the separation. But you gave two examples. I have a question about the other case...

In your first example, your junior engineer constructs an "intimate knowledge" message that his nested actor sends to its caller. Doing that requires that the public API of the caller expose the necessary functions to do that call. Knowing public API is not intimate knowledge. Assuming that those functions have a good reason for being public, all of those functions would be on the API of the interface object. What is the intimate knowledge that this engineer is taking advantage of? If that knowledge is the knowledge that "when told to do X, the scanner has a side effect of doing Y", that's going to be a problem no matter what interface you give.

Is the separation architecture only intended to stop the messages that may be sent to any actor? Or is it designed to produce an arbitrary subset. If it is an arbitrary subset, is there a way to prevent the API abuse by more creative scoping on the caller actor?

[Dmitry]

AristosQueue wrote:

In your first example, your junior engineer constructs an "intimate knowledge" message that his nested actor sends to its caller. Doing that requires that the public API of the caller expose the necessary functions to do that call. Knowing public API is not intimate knowledge. Assuming that those functions have a good reason for being public, all of those functions would be on the API of the interface object. What is the intimate knowledge that this engineer is taking advantage of? If that knowledge is the knowledge that "when told to do X, the scanner has a side effect of doing Y", that's going to be a problem no matter what interface you give.

Junior could use side effects, but that's not the main path for getting coupled to a Scanner Actor implementation. An Actor, such as a Scanner will typically have a set of public methods for two separate purposes - first set comprises an interface to Scanner customer (caller). The second comprises an interface for Scanner dependencies feedback such as Shutter, XY Stage, etc. It is usually fragmented into smaller interfaces (one per dependency) to keep them decoupled.

The problem with AF3.0 is that it exposes both sets to its nested actors (Shutter, etc.) via Actor-to-Caller Queue. In a clean design it should only expose the Abstract Shutter API to a concrete implementation of the Shutter. Well, this allows our creative junior developer to wire a custom Message.Do using the customer (caller) set of methods. Does this create a coupling ? Yes it does because he can call any public method in a concrete implementation of Scanner A class.

Now, the second team has a different Scanner implementation. Scanner A and Scanner B are siblings. They have the same parent, but Scanner A does some custom overrides to Parent Scanner methods and adds several model-specific public methods. What happens when Team B plugs Shutter A into their code base ? It depends on how Shutter A/Scanner A/Scanner B code is implemented, but even after updating "To more Specific' functions to convert to Scanner B, Team B does not have Scanner A specific behaviors in Scanner B implementation to run Shutter A Actor ... You may want to do some due diligence on the above as I never had a need to do something like this (port Shutter A to Scanner B).

And, BTW, I did make up Example #1 - as at the time of working on a biotech Scanner A, Actor Framework did not exist.

I encourage everybody to check out my presentation on using Dependency Inversion/Injection in LabVIEW here: https://decibel.ni.com/content/docs/DOC-17940. This was a ~75 min presentation at the Bay Area LabVIEW User Group (back in 2011) tailored for developers between CLAD and CLD levels, but I am not sure if it worked that way. However, several CLA's in the audience definitely got it right

Is the separation architecture only intended to stop the messages that may be sent to any actor? Or is it designed to produce an arbitrary subset. If it is an arbitrary subset, is there a way to prevent the API abuse by more creative scoping on the caller actor?

AQ, I am not sure what exactly do you mean by 'separation architecture'  and '...produce an arbitrary subset'. If you are talking about my take on Actor API Class  Hierarchy - see below:

Actor APIs serve several purposes:

·         Actor APIs provide the only wayto communicate with an Actor Component by calling Actor API public methods (hiding an Actor Component Message Queue Reference deep inside Actor API protected data);

·         Actor developers get full control over which Messages may be executed on an Actor by including methods for sending supported messages in Actor API;

·         ArT_Actor_API implements all four Message Modes shown in Figure 1

·         Actor API is a unit of reusable code. It may be extended or altered by any of its child classes

·         Actor APIs help organizing individual  Message Classes by providing an intuitive container for all supported Messages

Actor API Classes may be reused in several ways:

• ‘As Is’ across multiple ArT_Actor_Component classes
• Extended by a child Actor API class (via inheritance - by adding new methods)
• Modified by a child Actor API class (via inheritance - by overwriting method Implementations)
• Used to provide access to several (otherwise unrelated) Actor APIs via a single Actor API Object by ‘chaining’ API classes via class inheritance – a so called “Stacked API Design”. See Figure 8 below.

An Actor Component Class may choose to expose multiple Actor APIs (by returning multiple API Objects out of Component Constructor)to support the following Use Cases:

• An Actor has multiple ‘appearances’represented by subsets of the same Actor API (similar to Java Interfaces). This way an Actor may control its use by different entities (Developer, Service Engineer, Customer, etc.)
• An Actor has multiple ‘facets’represented by unrelated APIs (analogous to multiple inheritance)
• A Composite Actor needs to expose one or more of its subsystems to direct interaction with external callers (AKA a ‘pass-through’interface) instead of wrapping subsystem APIs and exposing their methods through its own API)