Parallel programming is hard -- lesson reinforced yet again

[AristosQueue (NI)]

As I have said many times, the Actor Framework guard rails and protections are not there because I think my users are dumb. They are there because I think parallel programming is very very hard for all of us. And I emphasize "us." It is so easy to think, "Oh, I've coded one of those before... it's easy." But you change one little requirement...

Today's lesson? Code that was written jointly by myself and Darren Nattinger. He wrote primarily with me buddying and verifying. And what was this code? The shipping project template for the Queued Message Handler.

Go ahead... in LV 2013 or LV 2014, use the New Project dialog to create a Queued Message Handler project. Open only the top-level VI. Don't even worry about the subVIs. Just the top-level VI. Most people who read this forum have written a QMH. Look at the code that Darren and I wrote. And then ask yourself... how many race conditions can you find in this code where messages would be dropped or handled incorrectly? I had reason to review this code again recently... much to my dismay, I found three.

How did Darren and I get ourselves to write such bad code? We tried to write a generic QMH that handles lots of user situations. We tried to be flexible without adding lots of confusing complexity. We didn't want to have to have a priority queue implementation (like the AF has), just use the regular queues. We tried to handle errors as a posted message rather than force every message handler to handle its own errors or else trigger a shutdown (like the AF does by default). And with those two small changes, the straightforward concept of a QMH unraveled. It didn't help that the Initialize tried to make itself reinitializable within the message structure (as opposed to initializing before the messsage handling starts and using a common subVI if you need reinitialization, the way the AF encourages).

Worst part? These race conditions are SUBTLE. They probably won't bite an application. But they can. I pretty trivially created a "can't handle error message fast enough to put the handler back in a good state to handle more messages" problem. I even successfully got messages from my producer loop to be handled by the consumer *before* the consumer finished processing its own initialization messages just by loading my CPU such that there was lots of thrashing going on.

We are fixing the template in LV 2015. But this is yet another situation that further convinces me that frameworks -- even tiny ones -- are not something you build by yourself.

So -- thank you to all of you in the community who have put your eyes on the Actor Framework over the years. And please, keep making suggestions to make the guard rails and walls more user friendly. They're necessary but, man, what a pain! 🙂

[AristosQueue (NI)]

PS: The other lesson I'm taking from this is this: Never ever let screen size and a desire to not create "potentially confusing subVIs" guide your architecture. If you want a simple architecture, don't try to solve the general use case!

[drjdpowell]

AristosQueue wrote:

...We didn't want to have to have a priority queue implementation (like the AF has), just use the regular queues. We tried to handle errors as a posted message rather than force every message handler to handle its own errors or else trigger a shutdown (like the AF does by default). And with those two small changes, the straightforward concept of a QMH unraveled.

Daklu and I had some long discussions about the potential flaws of Queued Message Handlers on LAVA a while ago.  What you need is not message priority, but the ability of your handler loop (but NOT other loops) to "enqueue in front".   Then "initialize" can enqueue in front its initialization substeps and be sure that those steps execute next without interference.  Similarly, it allows "Error" to call "Exit" next without any other message being handled in between.  Thus, handling of the "initialize" and "Error" messages become serialized; full initialization, including substeps, happens completely before the next message from the producer loop.  And exiting is part of the action of handling an error, not a new message tacked on the end of the queue.

The majority of QMH designs I have seen have this flaw, of not serializing message handling and instead throwing subactions onto the back of the message queue, allowing the handling of different messages to intermix.

[drjdpowell]

AristosQueue wrote:

But this is yet another situation that further convinces me that frameworks -- even tiny ones -- are not something you build by yourself.

I used the JKI QMH template for years and highly recommend it.  I use my own template now, but it shamelessly steals from the JKI template.  It uses an internal by-value (and thus private) string-based queue for internal actions.  And it's simpler in a way, as it does away with the seemingly pointless "Event Handling Loop" (the loop that doesn't actually handle the event, but just forwards the details to the other loop).