Controlling the precision of a double precision number

[GDthomas]

OK that title doesn't make much sense but I couldn't figure a better way to put it. Let me explain. I'm saving data to a TDMS file. One double precision number saves to 8 places to the right of the decimal point. This takes up a lot of room and isn't necessary. Can I programmaticly control the precision of this, to say 2 places, before it's saved to a file? 

[altenbach]

---

@rtpumps wrote:

Can I programmaticly control the precision of this, to say 2 places, before it's saved to a file? 

---

Yes, of course. Can you show us some simplified code so we can see what you are currently doing?

[Ben]

---

@rtpumps wrote:

OK that title doesn't make much sense but I couldn't figure a better way to put it. Let me explain. I'm saving data to a TDMS file. One double precision number saves to 8 places to the right of the decimal point. This takes up a lot of room and isn't necessary. Can I programmaticly control the precision of this, to say 2 places, before it's saved to a file? 

---

Hmmm....

TDMS saves data a floats and the precision is a function of what size float, SGL DBL, etc.

When displayed they are converted to a text representation that does have an option as to how many places after the decimal point.

Please clarify your desire and objective.

Ben

[altenbach]

The "data" input of tdms write accepts arrays of "alphanumeric strings", so just format your data with two decimal places.

Still, I don't quite see the reason. What supports your statement that it saves to 8 decimal places? Most likely the 8 decimal digits are just a format property of your indicator.

[GDthomas]

OK here's the TDMS data imported into Excel.

As you can see the Time column is the culprit. This is the subvi code,apologies for the mess it's seen a few changes in the last few days. The control "run time" in the bottom left corner is where the time data is brought into the subvi. I like Altenbach's idea of turning it into an alphanumeric string but since I'm using arrays it would take some redoing of the code and I thought maybe there was an easier way.

[altenbach]

This is just a blurry picture. How about changing the format of the excel cell instead?

(You know that "index array" is resizeable and that you don't need to wire the indices if getting all elements in order, right? What's the purpose of the while loop?)

[bharathp10]

this may help

[GerdW]

Hi bharat,

how should this (still kind of blurry) image help?

All you need to do is to set the numeric format for that Excel column as needed…

[bharathp10]

I thought by setting "precision type" to digits of precision and setting the no of "digits" as required will output fixed decimal points and will be reflected on excel. and is shown in image.

[GerdW]

Hi bharat,

I thought by setting "precision type" to digits of precision and setting the no of "digits" as required will output fixed decimal points and will be reflected on excel.

Why do you think so?

All you do is to set the formatting for this special indicator on your front panel.

Changing a formatting option in an indicator will NEVER change the underlying data in the wire - THINK DATAFLOW!

Even in Excel it works the same way: when you change the numeric format of the column you don't change the underlying data - just the applied formatting is changed!