Divide bug in Labview?

[Kier]

In labview, if you divide fractional numbers, you end up with a rounding error.

For example if you use quotient/remainder and divide 0.6/0.2 the quotient should be 3, but it is 2 instead. This acts exactly the same as doing that divide and then using "round toward -infinity".

[Luis_AM3C]

Hi,

That is not a LabVIEW error, is a computational error from representing fractional numbers at binary (IEEE-754)

0.6 = 5.9999996e-1

0.2 = 2.0000000e-1

Best regards,

[smercurio_fc]

You owe me a nickel for my retirement fund.

[Kier]

OK, thanks.

My solution is to add 0.000001 to the dividend, that seems to work, at least for my application.

[altenbach]

---

@Kier wrote:

My solution is to add 0.000001 to the dividend, that seems to work, at least for my application.

---

That does not sound like a general and robust solution. Try it with 0.0000000000000006/0.0000000000000002 😮

[rolfk]

---

@Kier wrote:

OK, thanks.

My solution is to add 0.000001 to the dividend, that seems to work, at least for my application.

---

This value only makes sense if you use single precision floating point number and the whole part of your number only has one digit. It is meaninful because single precision floating point values have about 7 significant digits in decimal representation. Double precision have about 16 significant digits and you would have to change your number again.

Your value would be to small if you want to avoid rounding errors for numbers having 2 or more digits in the whole number part and the operator to use on this correction would also depend on the operator you want to use for the calculation.