Unlocking the actor framework for adding functionality

[HenrikDueholm]

Hi everyone I posted the following on the idea exchange, but it's probably you guys who'd be interested in discussing this:

http://forums.ni.com/t5/LabVIEW-Idea-Exchange/Unlock-actor-framework-for-adding-functionality/idi-p/...

The content:

I've made my own copy of the actor framework to implement some changes that I really needed. Because of this I cannot really share code easily and my framework branch is not automatically updated. The changes that I've made could be made without overwriting the framework if only the framework was more open.

(a bit of what you can do with the changes are listed in brackets)

Creating more functionality for actor framework classes should be done through children of the classes in the framework. The way the framework is setup makes it impossible to add functionality to some classes:

- Message Priority Queue

- Message Enqueuer

- Message Dequeuer

- Message Queue

These classes are all obtained with a class constant internally in the framework.

I would really like it if we could have a class input to all these so we can implent additional queue functionality and have this input as part of either an actor dynamic method inside of the Actor.vi that the constants are read from or an input to the launch VIs.

Changing functionality through dynamic dispatch is the way it's meant to be done. But most of the Actor framework VIs are static dispatch. Vi's that I'd like to make dynamic are:

- Launch Actor Core.vi (and probably all the other launchers as well)

- Actor.vi

- Send Message and Wait For Response

- Send Batch (Or change the class constant to a control

Heavy use of access scopes makes some actions impossible (as intended). I'd like to have the following changed to community scope and added as friends to the Message class with an unbundle method for each:

- Read Self Enqueuer

- Read Caller Enqueuer

And lastly but surely the most controversial is the change of pre launch init from non-reentrant to reentrant.

Yes by launching an actor from within the vi you may crash your program (if you choose a child or itself to launch). But doing it as is will result in a deadlock anyway. The protection of making it non-reentrant is extremly weak! I'd rather have some warnings. What you gain is that the actor that you are launching and the nested actors that it will launch from the pre launch init can share their private data during launch! This is extremely useful with actor shared notifiers, DVRs and so on. Note that you can do this halfway by launching from the actor core.vi however the data will only be shared up in the hierarchy not throughout the entire hierarchy... which is bad/sad.

I hope you'll look into at least some of this to enable a more dynamic usage of the framework for advanced users.

Best Regards / Venlig hilsen

Henrik Dueholm Hansen

Laboratory Engineer

Danfoss A/S

Message was edited by: HenrikDueholm - Removing links

Message was edited by: HenrikDueholm - Fixed mistake in the reentrancy part

[BillMe]

Commented on the Idea Exchange.

[HenrikDueholm]

I got replies on the idea exchange board and ArisitosQueue encouraged me to have the discussion here so I'll post replies here and let the Idea Exchange post end.

As I was partly misunderstood I'll try to clarify some of my points here. I've invested quite a lot of work and passion in my actor framework and I tend to get quite long winded. I tried to avoid that in my first post, but I guess it backfired, if you see a lot of text below I hope you won't be discouraged.

The full answer from AristosQueue can be read on the Idea Exchange board.

AristosQueue wrote:

Before everything else:  You created your own branch of the AF. That's great! I hope you copied it off to a new directory (so no other applications link to it and get unexpected behavior). I would also encourage you to rename the "Actor Framework.lvlib" to something more personal so that the namespacing prevents cross-links in the future. It will make your upgrade experience more pleasant. And you'll be better equipped to diff any new AF versions against your changes if you want to pull in changes from the main branch.

I've created my branch in a new directory and renamed the library... Modded Actor Framework, hurray for my creativity. Anyway I'm aware that I have to manually update it and that it will take a lot of work. My purpose with the original post was to bring to light some of the potential that I believe is lost in the current version of the actor framework. Some of my suggestions can be implemented with wrappers but some cannot. Also please don't think of my "idea" as a single item. To me it's fine if we can discuss the item separately.

The points 1+2 go together as I see it so I'll reply to them as one.

AristosQueue wrote:

1.

Adding dynamic dispatching for the queues injected more performance overhead for the send and receive than was acceptable for many applications. So we cut it out. The biggest performance hotspot in AF is the enqueue operation.

This lack was not a barrier to creating alternate types of communication. Note the Proxy Actors that I built that allow the AF to seamlessly communicate across networks. I did not have to replace or modify the Enqueuer class at all. ¨

2.

> I'd like to have the following changed to community

> scope and added as friends to the Message class:

No need to make those community. Pass that information in as parameters to the Send functions. There's nothing stopping you from creating messages that have that information as required information.

Making those community scoped to the Message class wouldn't help you -- community scoping isn't shared with children. You'd also have to add wrappers to the Message class itself to expose that functionality for children. All of which would be a bad idea... Remember, there are at minimum THREE sources of input to the Enqueuer -- the caller, the self, and the nested. If you've shared the refnum to shortcircuit the tree, there are more. No single sender knows enough to know when it is safe to flush the queue.  Enabling that ability would mean no sender could ever be sure of its communication channel.

Creating "Messages that flush the queue" would be bad, ranging from annoying to disasterous. Even the possible existence of such messages would invalidate most of the guarantees of operation on which the AF depends. Many forum posts on the http://ni.com/actorframework forum have discussed message filtering, and the overwhelming conclusion of the community is that filtration, flushing, etc, should be entirely within the receiving actor and as it processes messages. You can inject a proxy actor into the actor tree to provide filtering, but deciding to dump a block of messages is the receiver's choice alone.

So, in short, the change you ask for would support a use case that does not exist, and if you're doing it for your app, it's because it works in your app, but it would never fly in any libraries of actors that were created for general community use because no one would be able to write reusable actors and have any dependency upon functionality.

1. I understand your point with the flush. It doesn't make sense for an actor to flush the caller. I think it makes perfect sense to flush nested actors though. If it's possible to control it so you can only flush "downwards" It'd make sense. This is not all that important though I'm a lot more interested in "enqueue in front" (also... status should be harmless as well right, why remove the option to use it?).
   1. Case: You have a process running and you need to do something NOW on that process. So you enqueue with normal priority. But then you need something done NOW again and the previous messages are still running so you go to high priority... but then what? You cant go higher with priority. Enqueue in front gives a sort of soft interrupt and also you can do what I do in my branch:
   2. Improve the batch message. As it is the batch message blocks messages of higher priority from executing. This is fine in some cases but if you want to send a message array that can be interrupted it must be enqueued in front on the current priority. When you have this you can easily create a self renewing message! Which is perfect for a simple datalogging actor:
      • wait
      • read
      • renew - contains the array of 1+2 and enqueues itself in front and can still be stopped with higher priority... or with a counter!
        (Also uses the read self enqueuer on a message, but that can be worked around as you say.)
   3. Next step is dynamic renew and nesting these unlocked loop messages within each other leading to a full test sequence with changing parameters. What I've described here should be useable to most people setting up data aquisition, or any other process where you have an actor performing the same operation in a loop. Please don't disregard "Enqueue in Front" together with flushing because this is much more useful!

AristosQueue wrote:

> > And lastly but surely the most controversial is the change

> of pre launch init from non-reentrant to reentrant.

I am open to discussing any change in the AF, as others will attest, but you have to make your case on thsi one because my current position is, "No. Not just no but hell no." 🙂

The lack of a central initialization safety point is one of the most common breaks for applications that attempt to do massively parallel subsystems. By having this in AF, we have a secure point that protects a huge number of users. If you do not wish to use that safe harbor, then do not use it -- any child actor class is free to establish its own API that users must call on the actor object before calling Launch Actor; you would then use Pre-Launch Init for nothing more than setting an error that says "those pre-launch functions were never called."

>The protection of making it non-reentrant is extremly weak!

Seems pretty robust to me. You're going to have to make this argument.

1. Okay you do the following in your pre launch init of an actor:
   
   Keep in mind that this is done on Beta and I ask it to launch Alfa and these actors have no relation. We now have to cases:
   1. Non-reentant:
      1. When you launch Beta the following will happen:
         1. The VI calling the launch will be stuck and cannot be resumed/stopped.
         2. The Pre launch init VI will be dead locked and no actors can be launched again unless you restart your program.
   2. Reentrant:
      1. When you launch Beta the following will happen:
         1. Beta is launched and launches Alpha during the pre launch init as intended. Nothing bad happens in this case
      2. Now if Alpha had been a Beta class or a child of Beta the following would have happend instead:
         1. The VI calling the launch will keep on launching actors untill labview runs out of memory stopping your program.
         2. Other processes can still launch actors while this happens... but that doesn't really matter as the program will crash soon anyway.
            
   3. Now to compare... which of the above is better. Honestly they are both quite bad and neither triggers errors. However you can do some really nice things when it's reentrant (this is discussed below in the next point).
      
   4. I can see what you mean by having a secure point but not how I can launch actors from it. I don't have the enqueuer before I start up the actor.
      

This last part is where I can see that I wasn't quite clear enough I'll try to do better here.

AristosQueue wrote:

>> Note that you can do this halfway by launching from the

> actor core.vi however the data will only be shared up in

> the hierarchy not throughout the entire hierarchy... which is bad/sad.

It sounds like you're trying to eliminate the entire goal of the AF. Information should NOT be shared up the heirarchy except by messages sent from the nested actors. Each actor is a black box from the point of view of its caller. That's the entire goal.

I'll use an example to get this across. I have three actors:

1. "Actor.lvclass"
2. "Actor owner Actor.lvclass", child of "Actor.lvclass" contains a nested actor.
3. "Front panel actor.lvclass", child of "Actor owner Actor.lvclass", has a user interface on actor core.vi.

So the question is where does 2 launch the nested actor and add it to the nested actor array. As it's not possible to launch from pre launch init we'll do it at the beginning of the actor core.

Now we launch 3 and the following happens:

1. Pre launch init for 3 is called, this call propagates to 1.
2. Actor core for 3 is called.
   1. The actor wire branches to what ever we do in the actor core on 3 and to the call parent node.
   2. actor core on 2 is called with the call parent node.
   3. the nested actor is launched and added to the nested actor array.
   4. Actor core  on 1 is called with the call parent node.

So what happens when I try to do something to the nested actor from the UI on "Front panel actor.lvclass"? Nothing happens because the actor wasn't added to the instance of the "Front panel actor.lvclass" on the actor core on 3. Only 1+2 have access to this which means that I have one actor with various instances of the "Front panel actor.lvclass" running on the actor cores in use by that single actor.

That's what I mean when I'm talking about only travelling up through the hierarchy.

If "Actor owner Actor.lvclass" launched the actor on the "pre launch init" this would not be an issue. So basically it's a question of branching wires. And how nice it is to be certain that you share the same instance of a class between all actor cores of said class when you launch it.

It's getting really late so I'll have to stop here. I hope this clarified the issue and that you no longer believe that I want to share data up the hierarchy...

Message was edited by: HenrikDueholm
Removed "[After going through this again I can see that I'm wrong]". I wrote that before the last quote when I was making an example with sharing a notifier. This removed the issue. But this is still an issue when launching an actor from your actor core.

[AristosQueue (NI)]

This post deals ONLY with the "enqueue at front" and alternate actor designs that negate the need for it. In particular, please pay attention to the very last boldface sentence at the bottom as it serves as a general brainstorm override for everything else I've written here.

> This is not all that important though I'm a lot more interested in

> "enqueue in front" (also... status should be harmless as well right, why remove the option to use it?).

As you note, we have priority enqueuing. You note problems with that, but I'm going to push back on your proposed solution. A pure "enqueue in front" just leads to race conditions as multiple high priority messages get reversed in handling. The vast majority of programmers cannot even identify why there is a race condition (I tried an experiment on 10 different CLA-level programmers both inside and outside NI... 9 of the 10 coded a race condition when armed solely with enqueue at front).

If you truly need to do something NOW, and you want to stack them, you almost certainly have a deeper problem with your design. Even an emergency priority message (Emergency Stop and Last Ack) will only be handled when the actor gets around to handling its next message. So if you send a high priority message, you have to assume that the actor is already acting on that message by the time you decide to send another high priority message. If you need to pre-empt the nested actor, your design has to change... I can point you to two solutions.

--------------------------------

BETTER SOLUTIONS

1) You need the nested to send a message up to the caller that says, "Send me the next thing to do." The caller maintains the list of things for the nested to do and only sends one message at a time down to the nested actor, so there is no backlog in the queue. That allows the caller to rearrange the queue as it needs.

2) Caller sends an Emergency Stop to the existing actor, wait for it to return its Last Ack, then relaunch it (you have its full final state in its Last Ack message) and send the message that you want as the first message.

---------------------------------

Any mucking about with the queue directly leads to deeper problems that have been notoriously hard to deal with. These are explicitly what the AF set to block. If you want to manipulate the queue, build a different actor system (as you have done). The AF is meant to rule out edge cases in favor of a framework that is stable and provides various guarantees to its users -- such a framework better enables sharing actors among developers. In the long run, my hope is to see things like device drivers implemented as actors that facilitate actor oriented programming in LabVIEW. The AF may not turn out to be the solution that drives this, but I'm setting up the rules of the road by which such an environment would operate. A "wild West" environment doesn't support such developments.

> Improve the batch message. As it is the batch message blocks

> messages of higher priority from executing.

Batch message is designed to be handled atomically. That's its goal. There's nothing to improve. If you don't want it handled atomically, don't send a batch, just send the sequence of messages. Wanting something to execute atomically and wanting something to be interruptable are mutually exclusive goals. The Batch Msg stays. 🙂  BUT...

... slightly tuning your request, what you seem to want is the ability to enqueue N messages at one priority and guarantee that no other messages at that priority get added in between, but still allow higher priority messages to intervene.

That could be done by adding a new Enqueue Multiple.vi to the Enqueuer class. That's something I'd be willing to entertain. But it seems to me that you could also achive that with today's AF if you add a mid-level actor that unpacks the block message. You have an existing caller and a nested actor. Make it so that caller launches mid launches nested. The mid actor overrides Receive Message and can unpack a Batch Message and forward it down to the actual nested actor. Since no one would be enqueuing for the nested actor except for the mid actor, you'd have your atomic bottleneck. [If the nested actor has its own nested actors, those might be interrupting... you have to decide how you want to handle those anyway... either you want them to be able to interrupt because you only want atomicity with respect to the caller OR you don't, in which case you can make those deeper actors use Low priority when talking to their caller (aka the original nested actor).]

Does all of that make sense?

> renew - contains the array of 1+2 and enqueues itself in front and

> can still be stopped with higher priority... or with a counter!

In general, this is functionality that I would move to a parallel While Loop in the Actor Core rather than trying to handle it in the messaging loop. Offloading the ongoing work to the parallel While Loop gives you the ability to start, stop, pause, etc, that work as you need, without risky behavior for the main messaging queue.

There's a LOT you can do using a parallel While Loop to create your own filtering systems and arbitrary actor structure for a particular actor. I really strongly encourage you to consider these possibilities rather than carve holes in the AF overall.

[drjdpowell]

A relevant LAVA conversation:

https://lavag.org/topic/16607-actor-queue-relationship-in-the-actor-framework/

IMO (now as it was then) Message queues should not be treated as job queues.   Messages should be read promptly even if the reqested actions are long-running.  In practice, it is simpler to make most message-handling code do the coresponding action immediately before reading the next message, BUT, that is a simplification one makes when the actions are fast enough that you don't need to care about anything as complicating as message priority, let alone interuptable batch messages. 

[AristosQueue (NI)]

Different actors can have different designs, IMHO. You can treat it as a job queue or a message queue. If you need more "cuttable" operations, move that code to your parallel loop and make it work that way. Actors that still want to do the whole job on receipt of the message are still valid, they may just not be appropriate in all instances.

To put it another way, I agree with Powell's point, I just state it less strongly... he takes more of a "there is a right way to do it" position and I take more of a "there are two options which may or may not work in various situations." His may be the better way to teach on the forums (reference my "teach the general, mentor the specific" mantra that some of you have heard).

[JerseyJayhawk]

Are you sure that Actor's PreLaunchInit.vi isn't already reentrant?  I just did a fresh install of 2014 and requestd a Actor Framework template using all of the defaults.  And then I looked at the Execution property of the Pre Launch Init VIs and they are Reentrant.

[drjdpowell]

AristosQueue wrote:

he takes more of a "there is a right way to do it" position

Yes, but I'm not saying "do it right regardless of how hard it is"; I'm saying "do it right because it is both easier and better".  There are techniques to do complex things without complicating the messaging system (and not all these involve the extra work of an additional loop to do the work).  What's more, keeping complexity encapsulated within an actor, rather than in the communication between actors, makes understanding of the system of asynchronously-interacting components easier (and that's where the most difficult bugs are). 

[AristosQueue (NI)]

JerseyJayhawk wrote:

Are you sure that Actor's PreLaunchInit.vi isn't already reentrant?

PreLaunch is reentrant, but it is invoked from the non-reentrant "Launch Actor Core.vi".  The effect is that it is not reentrant. It itself probably should have been changed to non-reentrant to make it easier to debug, but I never noticed the setting until now. Doesn't really matter because it is the caller that creates the exclusion. (Actor.vi is its direct caller and that is reentrant because it keeps running multiple instances after launching is done, so it is really Launch Actor Core.vi that would have to be made reentrant.)