LabVIEW
Data handling/sharing across an application
Hi!
I am trying to improve my application for better handling, sharing and saving my data across the modules, and I would need an advice
Currently, my application controls 1 device via a USB to CAN adapter, by sending command and receiving up to 30 Telemetry messages (which I refer to as "Data" in this post 🙂 ) made of various type. The telemetry is defined at runtime by the user as well as the frequency of acquisition and storage (even though these frequencies will be fixed in the future.)
I can see three solutions and I have already implemented the first one, but I am not convinced about it as it is rather difficult to implement any new changes to it.
- Create 1 sized queue for every telemetry (e.g. 528 elements max) and enqueue a lossy element whenever 1 telemetry data is available. To retrieve any data (either to display in a graph or to save it on disk) a simple access to all the elements in the queue (via Queue status.vi) allows me to have access to all the data in a queue (displayed then as an array). Once the queue full, the oldest data disappears and is replaced newer one, creating a "sliding" effect (like the strip chart update mode).
- Create 1 queue for all data sent as cluster of the actual data + info about it (what telemetry it is). This queue is retrieved in one loop, and a 2D sized array is created (1col for every telemetry, and X rows -e.g. 528- for every value of this telemetry.) This is a tiny bit more complex to implement than N.1, but it is better for future changes where adding/removing telemetry is just a matter of indexes. However it is more complex for sharing the data across the application (displaying vs storing on demand) and potentially creates wires with large amount of data.
- My last idea would be to use TDMS as a temporary file: Whenever telemetry is available, it is stored in a central TDMS file. Displaying its data would be easier than managing arrays in two (simply load the last X element in the file) and sharing across the platform for other use would also be possible (but require to manage properly the reference.) And when a permanent saving of the data is requested, either a "live" read of this file can be done, or a search by timestamp and retrieving all data at once can be done at the end of the recording period.. Once the application closes, the TDMS can be deleted/wiped as no longer necessary.
I believe that Number 3 would be the most scalable option, but it would also be the most complex to integrate and I have no idea about performances (when it could be important given that our computers are not the fastest...). I don't have much experience in the good practices and am mostly learning by doing. So having more experienced point of view would be greatly appreciated.
Thanks in advance,
Vinny.
PS:
A quick illustration of my method currently implemented. In reality, having different data types and wanting to manage the XY graph, this solution isn't optimal as one small change creates a lot of issues across the application.
Hi Mark, thanks for your time.
@Mark_Yedinak wrote:
How are you collecting your data?
I have various modules in my app (all communicating via queues), one of them is the communication module where I write requests, read answers and analyse the data.
Every telemetry has to be requested independently, meaning that if I want to receive 5 specific TM, I have to send first 5 commands (e.g. "get_rate" "get_torque" etc.).
Once my data is analysed, it is sent to the according sized queue.
The answers I get are hex strings containing my device status and internal time; the TM ID (e.g. 40, Hexadecimal number) and then the actual telemetry which can be a number, boolean whatever.
So based on the TM ID I am processing the data differently in a case structure.
@Mark_Yedinak wrote:
Are all of your telemetry points being collected at the same time in the same place? Are you only interested in the most recent data?
So if I am requesting a data set, say 10 TM (rate torque etc.), there will be 1 to 100ms between them (this is arbitrary defined at runtime) and I will request this dataset every 1 to 10Hz (same, this is arbitrary). The data sets will then be spaced equally over time (with the precision that windows open,) but every values within have a delay.
I am interested in both the most recent and an array of X values (ideally, also defined at runtime). I need arrays for the "graph displayable data" (like rate, torque etc.), and I need latest data for other types (Like booleans or device status etc.) and also for logging data (although I'm still unsure for this one if the best is latest data point or array)
@Mark_Yedinak wrote:
I would not recommend your current method with a queue for each piece of telemetry data. This will not scale well.
Yep exactly, I am paying the prize now.
@Mark_Yedinak wrote:
I assume that as you are collecting the data the same set of telemetry data will be collected at the same frequency. If this is the case create a cluster for your data point. You can add a data ID to your current cluster so that you will know what data element this value is for. The as you collect your data place it in an array and queue the array. Create a typedef ENUM which defines all the possible telemetry points you have and use that as the data ID. If you only need the latest data point then use a lossy queue of size 1. If you may need access in multiple places in the code you could also use a notifier instead since it can have multiple readers.
Using this type of approach you could dynamically chose which telemetry points you would collect. I'm guessing you have some finite set of data that you are able to collect. The typedef ENUM would specify all possible values and at runtime you can chose which pieces of data to collect. Using an array of clusters where you cluster contains the timestamp, dataID and data value gives you the flexibility and extensibility that you are looking for.
I'm not sure I understood what you are saying:
A cluster made of 2 element, 1 is the DataID and the other one the Value, and then create one array of this cluster? Do you mean a 1D or 2D array?
I didn't know that the notifier could have multiple readers, but it doesn't bufferize like queue does right?
I do need to access my data in multiple point (display and log typically), and this has always been a struggle for me.
For now, my ugly method is to send the data analysed from the Coms module directly to my data display loop in the Device Control Module. This loop aggregates the data and displays it at a periodic time (different from the acq. frequency for performances issues). When I display it, I also send it to the Logging loop located in the same Device Control Module, which based on user request, will actually save it on disc or not (via a simple boolean and case structure).
It is not good, I know, this is exactly what I am trying to improve, but the multiple access point gives me a headache.
