Limiting decimal places

[jvh75021]

For some values less than one my data sheets often list a horrendous number of digits for the measurement value. Is there a way to limit the significant digits or fractional part digits displayed on the data sheet? The preference would be two digits right of decimal point. I have a VI that will do this but don't want to filter every reading.

Thanks,

jvh 

[Teds · Accepted solution]

I know exactly what you are talking about, I have had many complaints on this issue from operators asking me why a DMM measurement goes out to some 10 digits of precision.

In TestStand 4.0, click on your Numeric Limit Step.  Click on the Limits Tab, and select the arrow next to the Numeric Format pull-down.  Select Custom, this brings up the Numeric Format window.  Within here you can select various options for the display of the data.  See attachment for a screen shot where I set 2 digits of precision.

Thanks,

Paul Holztrichter

[MALUCORE]

This only apply for the visual presentation. If I set the format to 2 decimals and set the Low limit to <1.00 and measure 0.999999999 this is still Failed even though the result shows 1.00 in the report Failed 1.00 V 1.00 <= x (GE) Or if high limit is <1.00 then 0.99999999999999999999999999999 is Passed Passed 1.00 V Limit: x < 1.00 (LT) This is wrong. If I know my precision of the measurement is only two digits, the comparison should only consider two digits IMO.

[BrinkHombre]

Hey Malucore,

Can you adjust the data values you are sending to the Step.Result.Numeric property? If your sensors are reading measurements with too much precision, you could try to programmatically adjust the measured data before sending it back to TestStand from your DAQ software.

Best regards,

[MALUCORE]

Hi Ryan,

Yes, that's what I have to do. But it means we are spreading out the the control of limits and the precision of those into code.

When I have built my own systems I have always wanted to store the Limits, Unit and the precision of those at one central location, and that those values controls both the mathematics and the presentation of the results, so its always the same.

[MALUCORE]

Update. What I did in my template now is that I send the numeric format setting into the function and decodes it and formats/rounds the results accordingly. That way it is the limit format setting in teststand that is the base for the measurment rounding too.

Works well, except I couldnt access the property programmatically. Had to have a parameter and send it to the .VI.

[RolfO]

---

@MALUCORE wrote:
This only apply for the visual presentation. If I set the format to 2 decimals and set the Low limit to <1.00 and measure 0.999999999 this is still Failed even though the result shows 1.00 in the report Failed 1.00 V 1.00 <= x (GE) Or if high limit is <1.00 then 0.99999999999999999999999999999 is Passed Passed 1.00 V Limit: x < 1.00 (LT) This is wrong. If I know my precision of the measurement is only two digits, the comparison should only consider two digits IMO.

---

I have not tested this in a sequence but it may be possible that you could use the val+str functions or the round-function. 

[ 0.999999<1.0 ] -> True
[ Val(Str(0.999999,"%.2f"))<1.0  ] -> False
[ Round(0.999999*100,4)/100<1.0 ] -> False

Change the second parameter in the round function (in my example 4) to get wanted rounding functionality:
Parameter 2: An optional parameter: how to round.
Pass 0 to round towards zero (the default).
Pass 1 to round away from zero.
Pass 2 to round towards positive infinity.
Pass 3 to round towards negative infinity.
Pass 4 to round to the nearest integer. If the number is exactly between two integers, rounds to the nearest even integer.

Edit the "Data source" expression for you test. It is an error on image, it should be "%.2f", not "0.2f".

[RolfO]

The property Step.Limits.Low.NumericFormat (or Step.Limits.High.NumericFormat) will get the format string set by "Numeric format" field when setting limits.

Resolution of result comparison against limits could then match "Numeric format" by by setting "Data Source Expression" to following:
Val(Str(Step.Result.Numeric, Step.Limits.Low.NumericFormat))

"Numeric Format" could be set to "%.2f":