Communication with Preallocated Re-entrant Clones

[Saturn233207]

I'm working on mass serial communications.  Somewhere between 10 and 200 devices depending on how many production has ready to configure and calibrate.  The objective is to open asynchronous preallocated re-entrant clones for however many COM ports VISA scans as product (Silicon Labs CP210x USB to UART Bridge) less exclusion list.  This is simple for the configuration process as they automatically close in the order opened and write pass/fail when closing 1 at a time to the main vi digital reference.

However, for the main vi to read continuously from all the clones during calibration so we can plot them is an issue.

Using LabVIEW 2013 SP1

Opening clones with main vi...

The clone code...

I'm using a GFV as a clone counter for the columns and the iteration terminal for the rows.  I would rather not predimension for a huge number of rows but rather add rows as necessary.  I've tried this with digital references and global variables but there is a problem as the first clone has new data before the last clone gets data.  Felt it best to open them at 500 ms intervals...

What I get using a global is...

Everything should be populated but it is not.  With digital references the results were worse.

Is there any way to succinctly collate in the main vi the results of a non-predetermined number of clones?

Running all that VISA write/read COM from the main vi will be a problem as we wish to read at 1 Hz.

[Kevin_Price]

The code raises a number of questions that I'm afraid I don't have time to address.  I need to boil this down to more general thoughts and suggestions:

1. You use a few different control references in your clone code.  I think some are for pushing data up to the main vi, others are for the main vi to stop the clone?  There's probably a better way to do things.  Control references are a slow way to do updates, and you need to be able to average up to 200 Hz worth of updates.  (200 devices, operating independently at 1 Hz each).

2. Choose to have all clones send data back to the main app via the *same* queue ref or dynamic event ref.

3. The data type for this ref should be a typedef'ed cluster.  The cluster should include, at *minimum*, both the serial data string *AND* some means for identifying associating this string with a particular clone/device/etc.

   A very useful generic cluster is illustrated in the Queued Message Handler template.  It consists of string (used as a message identifier) and a variant (which can hold anything, including a large cluster of related info).  Knowledge of the message ID lets you know how to convert the variant to its native type.

4. Your main vi will keep "listening" for data coming in via queue or event.  No matter what order it gets receives data from all these clones, the cluster of data includes enough info to figure out which device it came from. 

5. With up to 200 independent serial devices communicating asynchronously, I would NOT try to collect serial info into a 2D string array.  If you're doing calibration, the data from each is *logically* separate and independent.  Use data structures that will reflect that fact.  Variable timing, global variables, a bunch of async devices -- these factors don't mix well with the way you seem to be trying to populate the 2D array.

   One step forward might involve a typedef'ed cluster including a bunch of info associated with a particular device, including a 1D array of serial strings, one entry per iteration.  But now a particular device's data is in a structure associated with other things specific to that device.  This kind of data encapsulation and association is a general good practice.

6. I'd spend more time looking for ways to optimize the clone code while running a single instance in stand-alone mode.  You're gonna be launching up to 200 instances of it, it pays to make it good on several dimensions including speed and robustness.

This is far from a full solution, hopefully it helps you with some next steps.

-Kevin P

[cbutcher]

I have a few comments to add and brief questions (numbers for reference, not importance):

1. Firstly, I agree with Kevin_Price - your code sounds like it would benefit a lot from trying to avoid things like Global Variables.
2. Is using LabVIEW 2019 an option? The Map datatype would help a lot in simplifying some of this code.
3. In your clone code, you have (based on the caller code's typedef) a lot of unwired (to the connector pane) controls. If these aren't changing (Serial settings, End Read on term, Term char, etc) you should make them constants to clarify the clone BD.
4. You also have a lot of references - where are these set. I don't see them on the caller Async panel either.
5. You're building a string from your VISA read using a While loop and bytes at port, but it looks like you have a valid termination character. If there isn't some reason to avoid it, you should use the term char to make a single read. Even if you can't do that for some reason, you might want to avoid repeatedly updating the same reference to a string (Str Refnum) during the read. Write it once afterwards. This ties in with Kevin's comments about optimizing your clone. I also don't see the refnum on the caller Async node - how is it populated? (You have DataStrArray 2, which I'm guessing is the refnum called DataStrArray on the caller, but no string refnums).
6. Rather than using the global PlotCloneCounter, pass a value from the caller. You could wire this to "i" in the For loop.
7. You're using a "wait ms.vi" from SparrowBTLEtest. If you're able to use LabVIEW 2017, I think "Stall Data Flow.vim" does the same thing, and is built-in. Not a major issue, since you obviously already have a workaround

I'll look some more and try do a bit of refactoring, but information about the input controls is fairly important (refs particularly). Any information you can add will probably help.

Edit: I'm suspicious that some of these references should be block diagram constants to existing controls/indicators on the clone, and that the snippet creation process maybe broke them. Is that true?

[Saturn233207 · Accepted solution]

Thank you Kevin and CButcher.  LabVIEW 2019 is not an option at present though I wish it were.  To make a queued message handler work for an indeterminate number of clones I had to generate the queues in a for loop.

Made some EZ dummy clones to test...

Reading clones here.

Um, the snippet maker in my LabVIEW 2013 SP1 (full) does not seem to make representative pictures.  It adds strange references.

Not sure I need to incorporate user events into the clones as a digital reference seems to handle the stop well so I'm using a simple array of queues.  If it is difficult to send the Zero and Calibrate commands to the serial devices with digital references I'll set up user events with the control cluster.  However, I'm not sure of how to do this with the clones.

Since the clones open at 500 ms intervals and are then read at 1 s intervals, if there is a large number of clones the # in queue for the first ones opened is larger than the later ones.  Can open the clones faster...

The reason I have a while loop in the serial clones is because for some commands the device responds in several frames separated by the termination character...

Attached is a simplified version of test project code I used to figure this out.  Thanks again for your helpful suggestions.

[Kevin_Price]

Your array of queues isn't a good way for your main app to *receive* info back from the clones.  You're trying to dequeue exactly 1 sample of data from each of the clones, implying that they're all pretty well in sync.  But they aren't.  You're going to run into neverending unpredictable troubles with that kind of approach, I'm pretty sure.

A *much better* approach is that all clones enqueue into 1 common queue that the main app dequeues from.  In this approach, the queue ref itself doesn't give you correspondence to a specific clone instance.  To know that correspondence, you'll need to include appropriate info in the data carried by the queue.

   All this stuff is what I was driving at in points #2-4 in msg #2 of the thread.  At first it may feel more complicated to use an approach that requires you to send a more complex datatype over the queue and then decode it.  This is offset by the advantage that you can actually build a reliable working app this way.

More work will be needed, but the important thing is to understand that these 200 clones are operating independently and they won't stay fully in sync with each other.  You *need* to devise use an architecture and data structures that handle *asynchronous* updates from these many clones.

Your array of queues approach essentially wants the main app to "pull" data from the clones on its own fixed and rigid schedule that doesn't match the way the clones are running.  The approach I'm suggesting lets the clones do the "pushing" of data at their own pace and the main app handles the info as it comes in, i.e., according to the way the clones are actually running.

A closer look at your screenshots makes it appear that you may have tried to use the QMH template in your main app loop.  I would recommend letting the clones push their data messages onto this queue with its string/variant pair.  It appears you already have the structure in place to receive arbitrary message/data pairs, so go ahead and use it.

When I do this kind of thing with a QMH, I tend to have a typedef'ed cluster of "state variable information" that passes into all my message cases.  Different cases can then read from it or update it as appropriate.  Among other things, it'd contain info you'd need to dispatch data received from a specific clone to the correct corresponding destination(s).  

-Kevin P

[RavensFan]

---

@Saturn233207 wrote:

Um, the snippet maker in my LabVIEW 2013 SP1 (full) does not seem to make representative pictures.  It adds strange references.

---

The VI snippet maker built into LabVIEW does that.

Use the Code Capture Tool.  It is much better and doesn't do that.  Download using JKI package manager.

[Saturn233207]

Kevin:  Okay, but parsing the data back is more difficult and perhaps not better than an array of queues as data may not be received in time from a lot of clones using 1 queue.

We have a personal toxic gas monitor to calibrate in production.  In UART mode the device outputs a string of device type (gas), SN, ADC, ppb, T, RH and P.  Best not to query more than 1 Hz as it might overheat.  When plugged in 1 at a time, no matter the COM port given by VISA, devices appear in COM array in order plugged in.  This is to help the operator figure which is which.  There are several commands to enter sensor bar-code, device serial # and the LPM91000 op-amp settings.  This is easy.

After sensor burn in at bias (24 hours or so) we wish to place in zero gas, read and plot the ppb calculated by nA/ppm in the sensor bar-code then when stable enter a zero command.  Zero takes about 45 seconds during which time the clones will not be queuing anything but the output string when finished.  Then we go to a concentration of gas, stabilize while plotting and issue a calibrate command.  That returns a string as well.  Finally, we return to the plot.  All sensors should be lined out at the cal gas ppb.  Those not within a certain tolerance the main VI can point out as recal or duds.  Then we go back to zero gas and check if returned to zero.

Here is what I've done per your suggestions.  Oh, we need an elapsed time to plot...

Open clones with 1 queue

Clones, not using all the control inputs yet...

Making an array of the clone data to plot is tricky with 1 queue.  Below I have 3 columns for each clone, clone # from FGV (corresponds to order device plugged in), elapsed time and ppb.

Seems I must use an interior while loop to sort data replacing NaN in an array the size of which can change based on # of devices.

Front panel when running

This, once all clones are open, updates at the clone "read" rate.

In final v I could probably use SGL data type 2 b more concise.  Not sure I need to apply a user interface to the clones as just 1 digital reference with a type defined enum could do it.  Revised code posted.

Thank you again Kevin.

[Saturn233207]

Knight of NI:  My JKI is current but the Code Capture Tool (a LAVA product) is not listed (unpublished).  Has to go to LAVA link in NI tools network and then Source Forge https://sourceforge.net/projects/lv-cct-tool/files/ , download the *.opg, and install that with JKI.  Looking at the CCT it seems more coding is required to use it. Not such a great thing perhaps.  Simpler to just Alt+Prt Scr.  But, thanks for explaining why the snippet maker adds strange references to stuff not selected.

[Ben]

I just finished off a similar type of application that talks to about 200 cRIOs via TCP and tracks their state and provides functionality to control any of them.

When I spin-up the clone, I create a DVR for each clone, pass the ref to the clone and cache the references so that I can select which clone I want to muck with.

All of the connectivity is made possible via the DVR.

It (the DVR) includes a User Event that the Top Level VI can use to dynamically register for whatever thingy it wants to watch.

Just my 2 cents,

Ben

[RavensFan]

---

@Saturn233207 wrote:

Knight of NI:  My JKI is current but the Code Capture Tool (a LAVA product) is not listed (unpublished).  Has to go to LAVA link in NI tools network and then Source Forge https://sourceforge.net/projects/lv-cct-tool/files/ , download the *.opg, and install that with JKI.  Looking at the CCT it seems more coding is required to use it. Not such a great thing perhaps.  Simpler to just Alt+Prt Scr.  But, thanks for explaining why the snippet maker adds strange references to stuff not selected.

---

True.  It probably is a separate download to get it.  I've had it so long on my PC, it carries over from version to version pretty easily.

No you don't have to do any additional coding.  It will show up under the Tools >  LAVA menu as Code Capture Tool....

It does provide a palette of VI's giving you an API if you wanted to do things like programmatically capture diagrams.  But I've never used that.  The menu choice is easy enough.