Tricky Rounding Issuie

[chuckgantz]

I just had a similar problem and it sort of surprised me. I had two 3 digit numbers which were subtracted, and I got the same 16 digits as a result. I work in c#, and in the .NET world there is a function Math.Round which will round a number to however many significant digits you want. I'm not sure if there is an equivalent method in LV or in C, but the method of multiplying by 10^numberOfSignificantDigits, converting to an integer, then doing a floating point divide by 10^numberOfSignificantDigits sort of works. There is an issue with whether you need to round up or down when the next digit is a 5.

The method of converting to a string, taking the number of digits you need, then converting back to a number doesn't always work. It works for the example that was given, 0.123000000045, but won't work for 0.122999999 which would round to 0.122 instead of 0.123. I can't remember how Range and Coerce works so I'm not sure if it has the same problem.

Using the setting on display controlsmay or may not work, depending on what you are using the number for. If all you need is a neat display it will work, as long as LV does a true round instead of just cutting off the digits. I'm not at work so I can't try out what the display does. If it doesn't round it will have the same problem as above. If you don't really care about the display but need 0.123 and not 0.123000000045 then using the display to only display 3 digits won't work since the underlying data store will still have 0.12300000045. In that case you will need to round the number.

Chuck

[James@Home]

I apologize smercurio_fc, I didn't check out the wiki page... 

I just incorrectly assumed the thread was still active and in need of a solution since no responses were marked as a "Solution".

I'll try to pay closer attention in the future.

[RavensFan]

---

chuckgantz wrote:

I just had a similar problem and it sort of surprised me. I had two 3 digit numbers which were subtracted, and I got the same 16 digits as a result. I work in c#, and in the .NET world there is a function Math.Round which will round a number to however many significant digits you want. I'm not sure if there is an equivalent method in LV or in C, but the method of multiplying by 10^numberOfSignificantDigits, converting to an integer, then doing a floating point divide by 10^numberOfSignificantDigits sort of works. There is an issue with whether you need to round up or down when the next digit is a 5.

The method of converting to a string, taking the number of digits you need, then converting back to a number doesn't always work. It works for the example that was given, 0.123000000045, but won't work for 0.122999999 which would round to 0.122 instead of 0.123. I can't remember how Range and Coerce works so I'm not sure if it has the same problem.

Using the setting on display controls may or may not work, depending on what you are using the number for. If all you need is a neat display it will work, as long as LV does a true round instead of just cutting off the digits. I'm not at work so I can't try out what the display does. If it doesn't round it will have the same problem as above. If you don't really care about the display but need 0.123 and not 0.123000000045 then using the display to only display 3 digits won't work since the underlying data store will still have 0.12300000045. In that case you will need to round the number.

Chuck

---

No, you still aren't getting it.

You have to pay attention to the real world significance of your numbers.  It is highly unlikely that you will ever deal with a math calculation where you need 15 or 16 significant figures.  If you do, then you shouldn't be using a double precision datatype, but should be moving to extended precision.

It's not a matter of display.  You are saying that if you have 0.12300000045 and you need it to be .123(00000000)  (I added the zeroes in parenthesis because It sounds like that is what you really mean when you say .123), rounding to that number WILL NOT WORK.  The extra zeroes and 45 are there because that is what a double datatype can represent.  It can't represent a solid .123.  If you do go through the math to round to 3 significant digits, in the end you will still get the zeroes45 because the double datatype can't represent it any other way.

Actually, I don't know where the .123zeroes45 is coming from because when I enter .123, it shows up as .122nines8.  And rounding to 3 sig figs, still leaves it looking the same.

And yes, Labview does a true round and doesn't just cut off digits.

  I entered .123 into Numeric.  Then changed the display format to show 20 significant digits.

Message Edited by Ravens Fan on 03-14-2010 03:21 AM

[Steve_G]

Hi Everyone,

I really appreciate all of the replies and feedback.

Steve 

[Myke]

Just a quick question regarding this thread. Is this anything like the doing a calculation in labview vs a calculator? If it is I am having a similar problem. When i chase the numers on a calculator i get a different number in labview. But i think i know what the problem is, however don't know how to fix it. When labview multiplies the numbers i am using ie. .989, the program actually multiplies .98886.....etc. giving a different answer than one would find on a calculator. I work in the water heater industry and while we don't need to be incredibly precise, it would be helpful if labview could match a calculator. In excel we had to use the round feature (function) and keep it at 3 decimal places. While you can tell your indicators to do this, your answers still won't match.

example

the problem to figure out what BTU your heater is running

Heating value of nat. gas = 1020.29

barometric pressure  = 29.95

Gas inlet pressure = 7inches in water column

gas temp = 74

inches in water column to inches in mercury = multiply by .0735 to give you .5145

calculate number of seconds in 2 hours = 7200

ok now: add Barometric pressure to .5145 to give total pressure then multiply by 520 (constant)

divide that number by 30 (constant), then divide that number by 460 (constant) + gas temp. correction factor should be .989

multiply 1020.29 by 7200 and mutiply that by the .989 (correction factor)

Labview will not agree with your calculator. the answer that you have on a calculator is 7,265,281 vs labview 7,264,281.

once you have the "big" number you clock the time it takes for the heater to use 2 cu. ft. of gas  and divide that into the "big" number

ie 7,265,281 / 95.55 (seconds) and the answer will be 76,036.431 btu's and in labview it will be 76025.682.

[nitad54448]

hi

i think LV is waaaaaaaaaaaay better than a calculator (if a calculator is a 10 dollars object used to make calculations ...)

can you give a simpler example ? (I'll do it by hand just to check 🙂

N

Message Edited by nitad54448 on 03-17-2010 01:52 PM

Message Edited by nitad54448 on 03-17-2010 01:52 PM

[Myke]

agreed. lv is the stuff (to put it nicely 🙂 )

but it is a rounding issue i think that i am experiencing. see the attached vi (previous post of mine) and plug in the numbers to see what I mean.

[nitad54448]

i am kinda lazy now to make all that btu stuff but i think that rounding issue is natural way of representation, see the posts before..

[Dennis_Knutson]

It sounds to me that you are rounding in Excel and apparently the calculator is rounding as well. LabVIEW is calculating with more digits of precision. I'm not sure I would use a cheap calculator as a standard.

[Matt_W1]

Myke,

The exact value of your calculation is 24866099764 / 327075 (approx. 76025.68146143851). I checked this on a program that supports rational numbers. So LabVIEW is much closer to the exact value than your calculator. It's mainly because you round .98886 to .989 the later is only accurate to about three digits, So after multiplying and dividing stuff against it you can't expect more than 3 digits of accuracy (in this case the matching 760). If you want to LabVIEW to match your calculator, do the calculations more accurately, don't expect LabVIEW to do them less accurately.

Where I got the exact value