What does "Thread consumption suspected" mean?

[JKSH]

Hello,

My RT error logs show things like this:

LabVIEW RT Error Report generated 5/05/2016 6:12:14 AM
Target code: cRIO-9030
Firmware version: 2.5.0f1
******************** LabVIEW Error Log ******************
####
#Date: Wed, May 04, 2016 11:05:17 AM
#OSName: Linux
#OSVers: 3.10.31-ltsi-rt29-2.10.0f0
#OSBuild: 199199
#AppName: lvrt
#Version: 14.0.1
#AppKind: AppLib
#AppModDate: 
InitExecSystem() call to GetCurrProcessNumProcessors() reports: 2 processors
InitExecSystem() call to GetNumProcessors()            reports: 2 processors
InitExecSystem()                                      will use: 2 processors
starting LV_ESys2_Thr0 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)]
starting LV_ESys2_Thr1 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)]
starting LV_ESys2_Thr2 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)]
starting LV_ESys2_Thr3 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)]
starting LV_ESys1248001a_Thr0 , capacity: 1 at [3545175945.01412582, (11:05:45.014126000 2016:05:04)]
Thread consumption suspected: 2 Try starting 1 threads
starting LV_ESys2_Thr4 , capacity: 24 at [3545175980.54276419, (11:06:20.542764000 2016:05:04)]
Thread consumption suspected: 3 Try starting 1 threads
starting LV_ESys2_Thr5 , capacity: 24 at [3545175984.56444788, (11:06:24.564448000 2016:05:04)]
Thread consumption suspected: 4 Try starting 1 threads
starting LV_ESys2_Thr6 , capacity: 24 at [3545176207.70616293, (11:10:07.706163000 2016:05:04)]
Thread consumption suspected: 2 Try starting 1 threads
starting LV_ESys2_Thr7 , capacity: 24 at [3545176256.36498308, (11:10:56.364983000 2016:05:04)]
Thread consumption suspected: 7 Try starting 4 threads
starting LV_ESys2_Thr8 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)]
starting LV_ESys2_Thr9 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)]
starting LV_ESys2_Thr10 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)]
starting LV_ESys2_Thr11 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)]
Thread consumption suspected: 12 Try starting 2 threads
starting LV_ESys2_Thr12 , capacity: 24 at [3545193551.72287989, (15:59:11.722880000 2016:05:04)]
starting LV_ESys2_Thr13 , capacity: 24 at [3545193551.72287989, (15:59:11.722880000 2016:05:04)]

What does "Thread consumption suspected" mean? Should I be concerned about these messages, and try to eliminate them?

FWIW, in the system I observed this in, the Distributed System Manager reports a CPU usage of ~10% when running my VIs from the development environment, and ~1% (yes, that low!) when built + deployed. Memory use is low too, at about 35%.

[ScotSalmon · Accepted solution]

That diagnostic indicates that all the threads in the given execution system have been blocked for "a while" (many milliseconds). Prior to LV 2015, that would mean the execution system couldn't do anything else until one of them unblocked. In 2015, we'll allocate an extra thread or two to see if that un-sticks things. It's not necessarily wrong and can be normal, but it's a bit of a bad sign, because if whatever is consuming all those threads keeps consuming the extras too, eventually the execution system will stop making more threads and will block, which could lead to an unresponsive application.

Often there's a Call Library Function node involved, and the external code being called is taking a long time. If this applies to you, consider whether you can avoid it (perhaps by splitting the long operation into parts, like one call that starts the operation and returns quickly, and another that polls for its completion).

[JKSH]

Thanks for the clarification, Scot. So basically, LabVIEW will complain "thread consumption suspected" when all threads within an execution system are being hogged for an extended amount of time.

I'm not calling any CLFNs directly, but I'm using some official NI API that does. I also have 7 parallel Modbus TCP loops, each with a 1000 ms loop period and talking to a different device. Perhaps they're all waking up at the same time and fighting for TCP resources simultaneously…?

Some questions:

1. If I've understood correctly, an "execution system" (ES) is like a thread manager or a thread pool; each ES has a quota of threads that it can use to run the tasks that are currently queued under this ES. Is this an accurate description?
   
   
2. In general, if I can figure out which LabVIEW loops have lengthy iterations, and then move those loops to different execution systems from each other (e.g. by putting them in different subVIs and changing the subVIs' ES), would that reduce the likelihood of "thread consumption"?
3. Currently, all my Modbus loops are under the one ES. Are there any runtime issues if distribute them across different ES'es? (My gut feeling says no, because a thread is a thread no matter which ES it's under… but I want to be sure)
   
   
4. Just a question about terminology: Is the concept of "consuming a thread" a well-known one? I haven't seen this terminology before, and Googling hasn't gotten me very far (here's a sample search query). I've heard of CPU time being "consumed", but not of threads being "consumed".

[rolfk]

1. Sounds ok. LabVIEW knows of certain execution systems that it preallocates specific (for startup configurable) number of threads for and a VI can be assigned to run in any of those execution systems or use whatever its caller is using.

2. Would seem correct.

3. LabVIEW functions itself should not generally block the threads as they have been all designed to actually run asynchronously if they can block execution in any way. That is true for all of the timer and wait function nodes as well as for the TCP nodes and many other nodes which use some timeout and do not call into an exernal shared library through a Call Library node. There is one specific exception for VISA nodes which you can right click select if they should run synchronously or asynchronously. When set to run synchronously they will block the calling thread until that function returns.

One potential situation where you could be actually consuming threads is if you use some parallel for loop in which you execute a function which takes some time and is set to run synchronously. The parallel for loop will attempt to parallelize the loop execution and launch multiple executions of the synchronous call in multiple threads until it runs out of threads for the current execution system.

4. consuming a thread is a term as well known as anything else in computer languages. It is sort of self explanatory but at the same time also vague. Most programming environments leave the entire thread management to the programmer so terminology in this respect is whatever that programmer chooses. It's not likely that a C programmer will run into such a situation since if he does work with multiple threads he most likely manages them completely and will not easily run into a situation where the underlaying OS will start complaining that it can not create additional threads anymore. LabVIEW is different, it preallocates a pool of threads and manages them all almost fully transparently for you. But that means that you can run into situations where all the threads in a pool (execution system) are in use and while LabVIEW in the beginning simply started to stall new parallel executions of code it added at some point extra intelligence to the internal thread manager to detect that situation and someone in the LabVIEW development team decided that "all threads consumed" is a usefull message in the debug log. Those logs are not really meant for the normal LabVIEW user to be understood, but rather for the LabVIEW developers to give some extra hints about what might have happened, when things go really bad (TM). As such, the terminology used in them is whatever the LabVIEW developers found useful and doesn't necessarely adhere to any "official programming bible" terminology.