Broadcasting events in an application

[testingHotAir]

Hi all,

Was hoping to get some feedback on a design that I've adopted in my latest application.

The basic architecture has a central controller that dynamically launches "plugin" actors, which actually do the work.  Each plugin handles its own concern, as they should, but there must be some communication between them to integrate each task.  In my first version, I had a message for each integration task that was sent from the originator to the controller, which then decided what plugins needed to get the message.  This made for a jungle of message classes, some of which were really really basic.  The whole thing seemed grossly overcomplicated.

What I've done since is to create a generic Application Event message in the mutual ancestor for all plugins that takes an Event Name string and an Event Data variant.  The originator plugin publishes the App Event to the controller, which simply broadcasts it to all active plugins indiscriminately.  Each plugin is then responsible for defining what events it cares about and also what data type to cast the variant to (lots of community scoped typedefs involved). 

The obvious advantage to this is a vastly reduced class count, realized by essentially replacing the unique messages with community typedefs.  Can anyone see any pitfalls I should look out for?  Is this a good practice or am I getting myself into something I'll regret later?

Thanks

[AristosQueue (NI)]

testingHotAir wrote:

The whole thing seemed grossly overcomplicated.

I would love to see this full implementation if you can post it. I want to see more real world examples where people have actually done this. The very few that I have seen or built myself have significantly more files in the code, yes, but are massively easier to maintain in the long term -- much easier to make adjustments to behaviors without stuff propagating weirdly.

Can you post your code that used the message-up-and-back-down approach?

Or can you share it with me privately?

The more real-world examples I can see of this, the more we might be able to work out a better strategy or maybe just better tooling.

testingHotAir wrote:

Can anyone see any pitfalls I should look out for?  Is this a good practice or am I getting myself into something I'll regret later?

Oh yes. 🙂 They may be pitfalls that you prefer over the pitfalls of the other architecture. But there are pitfalls. Big ones.

A) Order of recipients. Plugin A broadcasts a message. You have no control over the order in which B and C get that message. In version one of your code, you don't care. But if you ever do start caring -- and you might because these actors are doing some coordinated activity -- you have a big refactoring headache.

B) Out-of-sync States. Plugin A broadcasts a message. You have a chain of actors (B launched C launched D), all of which get the message. You may have some sort of state transition happen as a result of the message, and now your middle layer may be in a different state ahead or behind the other two layers. Again, possibly not a problem in your current architecture, tends to occur over time.

C) Echo Chamber. It is easy to get into a situation with broadcast messages where Plugin A sends message X, which causes B to send message Y, which causes A to send message X, and so on, in what is called an echo chamber. Because the messages are just broadcasts to the world, there's no context information. This happens A LOT with UI actors that are all trying to do layout on the screen -- when A changes size, B needs to change size, and since B changed size, it tells A that A needs to change size. To stop the echo chamber, you have to add a lot of code where A recomputes its size, realizes it is already that size, and then doesn't continue the echo. The alternative is to have different types of messages... the message that tells A to change size in the first place is "User said to change size like this." The message that tells A to change size to match B is "Update to new bounds". The central dispatcher reinterprets messages for each recipient, and prevents the echo chamber WITHOUT the fairly-expensive-to-run "compute and compare" code.

[AristosQueue (NI)]

All of these problems with broadcast messages become more common the more your actors are peers -- i.e., none of them is managing the others.

I almost forgot about the other big one...

D) Who Controls Lifetime? Eventually, one of your broadcast messages will signal a situation in which one or several actors need to shut down. It is just going to happen unless you are vigilent about not letting it happen. And then you're going to have actors popping out of existence, and you'll have other actors that are still broadcasting messages and assuming that they'll be received and acted upon in order to get a clean shutdown sequence. Because, really, whether you admit it or not, eventually someone in these interconnected systems does have priority and does end up being in charge. And then you develop rules, which you write good documentation for all of your devs to read, about what kind of messages get what priority, and you develop some sort of header... and it starts looking an awful lot like an actor tree but without formal definition and no code traceability. 🙂

[jlokanis]

I use a peer-to-peer broadcast system in my applications.  I do not use this for command and control message but rather for status update messages.

• Rather than having any one actor own the subscriptions, I have the top level application own them.
• Any actor can subscribe and unsubscribe at any time.
• The application stores the most recent message for a particular subscription so any actor can request the latest value even if the sender sent it before the actor subscribed.
• Any actor can send a message on any subscription.
• Any actor that subscribes must be capable of processing the type of messages a subscription is intended to broadcast.  Therefore, when a broadcast message executes in an actor, it does not try to cast to that actor's class.  Instead, I have a parent class above all the actors that will use a particular broadcast message.  The message execute using this class but calls a method that the specific actors must override to implement their behavior when this type of message arrives.

It works very well, particularly for managing UIs with multiple levels of detail on system wide data.  IT also works nicely as a global shutdown message, if you want to implement one.

[drjdpowell]

Is the OP really describing a "broadcast" system?  As decribed, it isn't an everyone-direct-to-everyone broadcast going to every actor at every layer, but rather just plugin-to-plugin via the plugins' caller.  The Caller sees the event first, and can act, if necessary, to prevent any of the issues AQ mentions.

[AristosQueue (NI)]

In theory, the caller could see the event first and do something about it, true.

In practice, in my observation, if you design the system under the assumption that this is a blind broadcast then it becomes a blind broadcast. Developers probably do not include enough information in the messages for the caller to really intervene. I agree that it is *easier* to put that code into the Caller in the future, but I'm not sure that it is as straightforward ... but this is one of the reasons that I really want to look at testingHotAir's code. There are so many variations to these patterns, and it may be that the one that testingHotAir is using is sufficiently flexible to avoid the issues.

I'm also unsure, because I don't have a practical example to work from, whether that later code would centralize more knowledge in the Caller and less information in the relevant senders\receivers of messages. I would hate for the Caller to become a complete bottleneck that has to be edited every time the interaction patterns between two peer actors changes, if that's possible to avoid. Again, practical example would make conversation on this point meaningful, at least for me. I don't want to assert either way in a vaccuum on this point.

[testingHotAir]

Regarding points A, B, and C, I still use strict messaging for any behaviors that would cause those problems.  For instance, I have a message that changes the state of the whole system from "idle" to "run" which has well-defined inputs.  Should I need to change the way a future version of the system behaves on that state transition, I can do so by overriding that method.  Basically, the use case for the App Event messages are similar to what jlokanis describes: status messages, things that don't do mission-critical functions.

An important note about my implementation, the App Event message can't propagate into actors nested below the plugin layer.  That is, if Plugin A launches nested actor B, B would not be able to receive the App Event message.  I think about App Events as living in the Application Layer; anything lower level such as device control or front end acquisition uses the normal Actor messaging paradigm internally.  drjdpowell is correct, basically, except that the caller can't intervene in the message but can only forward it to subscribed plugins.

I actually adopted this pattern specifically because my original messaging system required edits to the caller every time I wanted to change the interactions between plugins.  With the App Events, I can just toss a new event out and set up whatever plugins need to respond to it individually, without changing anything about in the caller.

AQ: I can share the code with you privately if you like, just tell me where to send it.

[drjdpowell]

What do your plugins do, testingHotAir?  I'm having a hard time imagining how that need to interact.  "Plugins" to me suggest optional components, so they can't really depend on each other.

[drjdpowell]

jlokanis wrote:

• The application stores the most recent message for a particular subscription so any actor can request the latest value even if the sender sent it before the actor subscribed.

I do this too.   I have two kinds of "notifications"**: Event, for something has happend, and State, for this is the current state of something.  The latter stores the most recent message, so it can be sent immediatly to any new subscriber.  This simplifies things greatly, as one doesn't have to worry about time ordering.

**My notifications aren't broadcast, though.  Each actor (little-a actor, not AF Actor) maintains it's own subscriber list, and only those who can send registration messages to the actor can subscribe.

[testingHotAir]

Yeah they're not really plugins.  Basically my system has to carry out several different tests, all with slightly different interfaces.  I have one class called Supervisors that defines what "plugins" are needed to accomplish the test defined by the Supervisor.  The Supervisor requests "plugins" from the Controller, which launches them if it can.