Callback Delay

[EthanHenry]

Heyo, I had a previous post about this and thought that I had fixed it, but it turns out I fixed a problem no one knew we had and the main problem is still at large.  Here's what we know:  Seemingly randomly, our Callbacks will stop firing.  Logs indicate that SECSGEM requests and responses are coming in, but the callback VI is not starting.  After a seemingly random amount of time, suddenly all of the callbacks will start to be executed, one after then other.  This happens somewhat frequently, but we still can't seem to figure out why it's happening.

Registering Callbacks at the beginning of execution:

The important VI here is the Msg Callback Shell VI, which has this:

The Log Information Msg does exactly what you might expect and log the string in a log file.  The use of queues with preoppenned VI Refs was a solution to the previous problem.  The main problem shows in the logs as the SECSGEM Request being sent, the Reply coming immediately after, after which case it should launch an event.  Most of the time this works perfectly well.  Here's a sample of what a correct Log entry for this flow would look like

2023/04/03  08:59:15    Engineer    Information SECS-GEM-HEARTBEAT  sending msg

2023/04/03  08:59:15    Engineer    Information msgCallback Started

2023/04/03  08:59:15    Engineer    Information SECS-GEM-HEARTBEAT  port: "equip0" replied with "0 {}"

2023/04/03  08:59:15    Engineer    Information msgCallback MSG Ref Opened

2023/04/03  08:59:15    Engineer    Information msgCallback MSG Core Started

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  Macro: Initialize

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  Data: Initialize

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  Initialize Core Data

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  SECSGEM: Get Event Arguments

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  S3F97 L:0

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  Macro: S3F97

2023/04/03  08:59:15    Engineer    Information SecsMsgReceive  S3F97: Parse

2023/04/03  08:59:16    Engineer    Information SecsMsgReceive  S3F97: Reply

2023/04/03  08:59:16    Engineer    Information SecsMsgReceive  Macro: Exit

2023/04/03  08:59:16    Engineer    Information SecsMsgReceive  Record Time

2023/04/03  08:59:16    Engineer    Information SecsMsgReceive  Data: Cleanup

2023/04/03  08:59:16    Engineer    Information SecsMsgReceive  Exit

2023/04/03  08:59:16    Engineer    Information msgCallback MSG Core Finished

2023/04/03  08:59:16    Engineer    Information msgCallback MSG Ref Closed

2023/04/03  08:59:16    Engineer    Information msgCallback Finished

However, at a seemingly random time, we will get a behavior where the SECS-GEM-HEARTBEAT is logged with the sending msg and the reply multiple times without a single msgCallback logs occuring.  After a while of this, multiple msgCallback events will be logged, each one waiting for the one before to finish.  Anybody seen any behavior like this?  The Customer also reports that it seems to occur at the same time as the UI freezing up (unsure if this is a cause or effect or unrelated).  any help would be greatly appreciated, thanks!

[EthanHenry]

I feel like I should also mention that, while this behavior is apparent in the logs, these are logs from another computer.  I have not yet been able to replicate this behavior on my own computer, so any guesses as to replicate it are also appreciated.  The events are generated by a .NET dll as well, in case that has anything to do with it.

[Bob_Schor]

It is so frustrating to "guess" what's going on when Forum users post pictures of parts of a LabVIEW VI.  It does have the virtue that we don't need to have LabVIEW installed on our PC (or phone) to answer, but we also don't have a full picture of what's going on.  Best is a compressed (zipped) copy of the entire Project Folder, preferably saved as LabVIEW 2019 or 2021 (which many of the "experts" seem to be running).

Your Callback Shell VI uses a Sequence Structure, which is almost always a bad sign of poor Data Flow.  Indeed, all of the sub-VIs that might produce an Error in their Error Out are not connected to the Error Line that Log Err is monitoring, so they are kind of useless.  Delete the entire Frame sequence and "serialize" the various VIs in  portion of the code using the Error In and Error Out terminals of the functions you are using -- that's why they are conveniently placed on the lower left and right corners of the Connector Pane.

More code (files with the extension ".vi") please.

Bob Schor

[EthanHenry]

Yeah sorry about that, the VIs involved are super tied down with dependencies so even sharing a small portion of the code is difficult (just for the Register callback VI LabVIEW loaded ~4000 vis/libs into memory).  I can give some  information and more detailed pictures, and I understand that this isn't the best way to do things (This is all legacy code that primarily used GOOP and done in LabVIEW 2016).  As for the sequence structure, I completely agree, and it is definitely not going to be a part of the final version.  Honestly, the entire msg callback shell vi was created for the sole purpose of increasing logs to try and solve this error.  Originally the callback vi that was registered was the SE MSG Receive Cback VI that is called in the shell vi.  I understand if there's no way to help with the code itself, as it is pretty complicated, but I am still curious if anyone's encountered the delay of events from a .NET dll.

[Bob_Schor]

Thank you for clarifying.  I am completely sympathetic with your situation, as it characterizes the very first LabVIEW Project I encountered.  Not quite as many VIs (well north of 1000, none with any documentation) which I spent multiple months just trying to fix an occasional bug.  I eventually "started over", using almost none of the original code, but keeping the basic data structure that characterized the way the users interacted (and controlled) the Software.  It was based on an Excel Workbook, but (a) the Workbook was very poorly organized, (b) the way the Workbook was used was "Original-Developer Friendly", "User-Hostile", and the main DAQ VI took about 20-30 pages to print out (it was written in "Traditional DAQ").

In consultation with the graduate students who were trying to do their thesis research with this program, we started by redesigning the SpreadSheet, organizing it to match the flow of information from the columns of the "Trials" Worksheet to the sequence of events in a Trial.  I also wrote a suite of VI's based on the Excel functions in the Report Generation Toolkit to extract parameters from an Excel row corresponding to the current Trial.

Did I mention this was a LabVIEW-Real Time project (little "p" -- the LabVIEW Project wasn't yet "invented")?  By this time, DAQmx was released, so the DAQ code "collapsed" to about 3-5 sub-VIs all of which fit on a single 1024 x 768 screen (this was around 2010, remember).  I "built my own State Machine" on both the Host and the PXI Remote Target, documented all VIs, wrote copious Documentation using Word, and tested, tested, tested.  Ended up with a much "slimmer", documented, maintainable, faster, less liable to crashing VI with data files organized in a much more sensible manner, making subsequent data analysis routines (in LabVIEW or Matlab) much simpler to write and to run.

So if you are faced with a LabVIEW routine with several thousands of VIs, especially if it is as disorganized as the routine I inherited (characterized by undocumented sub-VIs, lack of a sensible programming model (such as a QMH, State Machine,  or something similar), VI's of reasonable size (including having all Block Diagrams fit on a single display screen, perhaps up to 1920 x 1200), with every sub-VI and TypeDef documented carefully.

Good luck.  If you can gather a small team (I used some of the Grad students who had some LabVIEW knowledge to let me "walk through" my code and help with testing and design), it could really help.  Document, test, use Version Control Systems (GIT, Subversion, others), and think about designs like State Machines, QMH, etc.

Bob Schor

[wiebe@CARYA]

The main problem is this is all too complex. Complexity is the only reason bugs can't be found.

Why not use the "User parameter" to pass things to the callback VI references? A global (the named queue) doesn't seem needed here (and (arguably) never is).

What does Msg Callback Shell VI actually do? If anything it's doing is handled by an event structure, you will get a deadlock. While LabVIEW is handling a callback in a callback VI, all event structures are blocked. So if Msg Callback Shell VI triggers an event, and waits for the result, you get a deadlock. The callback waits for the result of the event, the event waits for the callback to finish.

Also note that some references in callbacks need to be closed. If you don't you're leaking memory.

Others references are closed automatically when the callback finishes. This means you often can't pass the reference to code outside the callback, although this could work as long as the callback is alive. You'd have a nasty timing issue on your hand. You'd need to (deep) clone the reference before passing it to outside the callback.