TestStand report shows a passing value as a fail

[paulmw]

I ran into a possible bug that I'd like to get some feed back on.  Attached is a sequence and VI that when run in test UUT every other iterations value for the step (i.e. next UUT) the result is maintained from the last UUT and it show up in the report. Iit is a stripped-down-to-nothing version from the acutal test sequence on the floor with some stuff to simulate the error.

The step does have the ignore error on and this situation happens when an error occurs but I would not expect the previous test result value to show up.  Run the sequence and see what I mean.  It generated a report that makes my manufacturing floor question the stability of TS.

PS. LV 8.5 TS 4.0

Message Edited by paulmw on 03-04-2009 12:31 PM

[Scott_Richardson · Accepted solution]

Paul -

The reason are getting failed with a measurement from the last good UUT is because of many compounding issues. The sequence has the default setting of "Optimize non-reentrant calls to this sequence", which basically tells TestStand to cache the runtime copy of the steps in this sequence to improve performance for each subsequent call to the sequence. So the Step.Result.Numeric value will not only be maintained each time you call the step within an instance of the sequence, but now also subsequent sequencial invocations. Next, notice that the report says that you had 10 iterations executed with 0 passed and 0 failed. This means that you got errors on all iterations. Now a looping step reuses the step result as it executes each loop, and the final summary result is a snapshot of the last iteration. Since you are ignoring errors, the numeric value on the step result never gets updated, so the final summary step result is the same as the last UUT. The looping code then sets the loop status to Failed because no iterations passed.

Look at the attached report to illustrate the behavior. The report shows you the results from each iteration as well as the parent summary result.

In general ignoring errors is a bad idea, but assuming that you want to continue to ignore errors, you can do one of the following in order or ease and best behavior:

1) Add a preexpression to set Step.Result.Numeric to a failing value, such as zero or NAN
2) Uncheck the "Optimize non-reentrant calls to this sequence" setting, so that a default value of zero will be used for Step.Result.Numeric
3) Add another step after the one you are running to see if at least one iteration passed or failed.

[paulmw]

Thanks Scott, the "Optimize non-reentrant calls to this sequence" expains to me why this is happening.  The ignore errors was probably a quick cheat that the developer used to get around some daq-mx error from a non triggering task that was happening and then subsequent loops would not error.  Another solution (besides rewriting the whole thing which should probably be done) is to not connect the VI error cluster to TestStand's error cluster.  Especially if it is getting ignored anyway. 

Question: why is the Step.Result.Numeric not getting updated?  Is the status expression not getting executed on an error even if it is ignored?

[RayFarmer]

Hi,

I believe I know whats happening.

The clue is in your statement ....expect the previous test result value to show up.

Your Locals.Vpp[] still contains a set of values from the previous step but looking at the results of this setp there was no actual passes from the 10 iterations and your loop status is defined as

RunState.LoopNumPassed >= 1 ? "Passed" : "Failed"

which gives a result of Failed. there was no failed result either which possible suggests that all the step iteration errored and therefore retained all the values from the previous step.

Your Step Numeric result is based on the Evalute(Step.DataSource) which is Evalute(Max(...all the elements of the array)).

Because you didn't clear the array before each step and because you are getting errors therefore those steps iterations that error no new values will be sent to Locals.Vpp[] therefore they will retain the previous value and give you the wrong indication.

Solution - clear your array in your pre-expression.

Regards

Ray Farmer

[paulmw]

It would appear that my assumption that the status expression (where Step.Result.Numeric gets set for my example) does not get executed durring errors even if the ignore error is set.  Can someone please confirm my assumption?  Also, while I know this is a good example of how not to develop sequences, is this the intended functionality (specifically with ignore error on)?

[RayFarmer]

Hi Paul,

Ignore Errors just means its not going to launch into the Error Callback or jump to Cleanup depending on what settings you have.

The Ignore Error control allows you the freedom of how you wish to deal with that error.

If you really have a situation where you are not interested in the returned error then don't return it and ignore it in your code.

Regards

Ray

[Scott_Richardson]

Paul -

When an error occurs during the execution of a code module for a step or a single loop of a step, a lot is skipped. Yes, the status is set to "Error" by the engine and the status expression is skipped. Any poststeps for the step type are skipped, evaluating the post expression is skipped. Any post-step callbacks are skipped.

[Scott_Richardson]

Ray -

Locals are reset on each invocation of the sequence even when we are caching the run-time copy of the steps. Clearing the locals array in the step pre-expression would not work because TestStand evaluates the expression each time it loops on the step.

[Scott_Richardson]

Paul -

I was incorrect in my posting about what is skipped when you ignore the error. If the step returns the error as you are doing via the Error Out cluster, the status expression is skipped. If an exception occurs while executing the module and you are ignoring errors, then the other stuff like I mentioned are skipped.

Message Edited by Scott Richardson on 03-05-2009 01:26 PM

[RayFarmer]

Hi Scott,

I agree that when a local goes out of scope the data will be loss but while it's in scope the contains will remain unless cleared by the programmer. I wouldn't have expected anything else otherwise whats the point in storing data in variables if the engine is only going to cache it at each step. Remember it's not an array that is part of the structure of the Step.

With the posted example, I setup some data as if from a previous step and depending on what the random number was depended with it passed and stop on the first iteration or not. I was experience an error because of the LabVIEWIOControl (I believe) therefore the remained iterations were skipped and therefore the contains of the array remain unchanged. This greatly affected the final evaluation.

Regards

Ray