LabVIEW
Setting up a call library node for a DLL
Solved!
wiebe@CARYA wrote:
The option could have three values:
+ Pass LabVIEW array (as it is now, and the default)
+ Pass as pointer (always points to the data, IIRC, like passing an array)
+ Pass data (like converting to a cluster, pass the bytes)
You have that already when you pass a string or an array to the Call Library Node. Inside a cluster is an entirely different story. There are many clusters in C APIs that contain both inlined arrays/strings as well as pointers to arrays. How the f*ck would you allow to configure that? Also once you pass data pointers there is a whole range of things that simply will go wrong if those data pointers are not only meant as incoming data but also as something the function can write into. You need to either make sure that they are properly preallocated with the correct size, and each of them could have a different size. It can also be that the API you call will allocate them and assign them to that pointer and some other variable in the cluster will contain the actual size of that memory block either in bytes or array elements. LabVIEW needs to be able to know the size of such a pointer in order to be able to turn it back into a proper LabVIEW data handle that can be used in the rest of the diagram.
Last but not least if the DLL allocated those pointers you also need to know how it did that. If it used the Windows GlobalAlloc() or LocalAlloc() function you need to explicitly call the according GlobalFree() or LocalFree() function. It could also have used the COM Allocator or some .Net Runtime allocator just for the fun of it. If it used the C Runtime malloc() function you are totally hosed unless the DLL also exports a specific function to free its own memory buffers. That is because the C runtime of the DLL you call could be a different version than what you call from your own application since it was compiled and linked in a different C compiler or even just a different Visual Studio version. Each C runtime implements its own malloc(), calloc(), and free() routines and friends and manages the according memory heap itself by requesting larger memory blocks from the OS through its kernel APIs and then taking junks of those blocks to satisfy m/calloc() calls and when it receives a free() call it looks up the block in its allocation table and marks that area as again available. Knowing this it is totally obvious that when your DLL allocated memory with malloc() from msvcrt140.dll since it was compiled with Visual Studio 2015, your free() call in LabVIEW unless you happen to explicitly use a Call Library Node to call the free() function in msvcrt140.dll will simply crash as the memory block was not allocated by this version of the C Runtime. And to make matters even worse there is something called SxS (side by side shared library loading). The DLL you call can define in its manifest that it absolutely wants to be linked to msvcrt140.dll (Version 14.0.1.2345) and Windows will happily load exactly that version of the DLL, while LabVIEW might choose to only simply link to msvcrt140.dll in which case Windows will usually load the latest and greatest version it can find (or another explicitly preferred SxS version). In either case Windows will load that dll too and link the LabVIEW link requests to that DLL instead. -> Instant crash when you call free() to deallocate the DLL created buffers in LabVIEW. If LabVIEW was compiled with a different Visual Studio version things will obviously be just as bad although here SxS doesn't come into play at all.
And the funniest thing is of course you might make this work by linking to the right msvcrt140.dll version to call free() from there and then you upgrade either LabVIEW or your DLL which was compiled in a different C compiler version and everything starts to crash. I wish you funny bug hunting with this!
I doubt there will be a lot of cases where (nested) clusters contain a mixture of array pointers, LabVIEW arrays and array data.
LabVIEW arrays and strings and the other forms at the same time: No
Fixed size inlined arrays and strings and array and string pointers in the same struct? Absolutely! Maybe not in trivial DLL interfaces but any sufficiently complex API has such combined structs for sure.
IIRC, LabVIEW does change the data. The LabVIEW cluster contain pointers to arrays. So, when passing the data to the CLFN, the data is converted to array data including the size..
LabVIEW doesn't change the data it passes to a shared library at all. It allows you to configure to pass a pointer to the array or string, a LabVIEW handle or a pointer to a LabVIEW handle for array and string parameters but that is not modifying the data at all, just passing a different pointer into the data to the shared library. There is only one modification LabVIEW does when passing parameters to a shared library and that is for string parameters configured to be passed as C pointers. Before the call LabVIEW makes sure that the buffer is properly terminated by a NULL character and after the call it scans the buffer for a NULL character and resizes its string handle to the size up to and not including this position.
@rolfk wrote:
wiebe@CARYA wrote:
Why isn't there an option to tell the "adapt to type" configuration how to treat arrays?
It seems to me that in a cluster with arrays, arrays should either be a pointer to the array, or the raw data (not including the size)?
Wouldn't that make the CLFN a lot more compatible with C\C++?
The problem is that there is basically no standard in C at all how to "manage" memory. C has been originally defined purposefully to be so close to the bare metal CPU implementation that it left most things entirely to the implementer of the particular compiler and the programmer writing the code. Who allocates memory when and from where is totally up to the implementer of any particular function. And the C (and even the C++ unless you limit yourself more or less fully to only use standard template class features) syntax has simply not one single feature to indicate that directly when declaring a function interface.
There are conventions that have been found to work better than others and are fairly common nowadays and their intention can often be inferred from the naming syntax of the parameter names, but those names have absolutely no meaning in terms of the C syntax and need to be interpreted by the user of those functions to hopefully mean what she/he thinks it does, by reading the according API description anyhow.
Creating a Call Library Function configuration that would allow to configure all these features would be making the dialog several times more complex than it is now, and for 95% of the users it is already way to complex to use.
I agree that the configuration is already complex.
Maybe an option is easier that no option though...
Search LabVIEW like a graph!
wiebe@CARYA wrote:
I agree that the configuration is already complex.
Maybe an option is easier that no option though...
Not really. You have two choices here:
1) Add this option: In its simplest way it will work for a few APIs but still fail many others. But the user will need to understand a lot about C compilers to even be able to make the right choice.
2) Don't add it: The user will then have two other choices\
2a) Do the whole pointer hocus-pocus on the diagram with calling DSNewPtr() and DSDisposePtr() and MoveBlock(). Convinient? In most cases not at all! You end up with a lot of conditional compile code if you want to support both 32-bit and 64-bit and need to understand C compilers to a level that is not funny anymore.
2b) Write a wrapper DLL that allows to translate between LabVIEW datatypes and your API. Complicated? Not really. Sure you need to write some C code but unless you are able to do that already you wouldn't be able to decide which option to use in choice 1) either. And in comparison to choice 2a) the level of expertise about C compiler low level details is several magnitudes lower.
@rolfk wrote:
wiebe@CARYA wrote:
The option could have three values:
+ Pass LabVIEW array (as it is now, and the default)
+ Pass as pointer (always points to the data, IIRC, like passing an array)
+ Pass data (like converting to a cluster, pass the bytes)
You have that already when you pass a string or an array to the Call Library Node. Inside a cluster is an entirely different story. There are many clusters in C APIs that contain both inlined arrays/strings as well as pointers to arrays.
Really? Many?
So the alternative to a difficult solution is no solution? Or what we have now? Because that seems to be quite difficult for a lot of users.
@rolfk wrote:Last but not least if the DLL allocated those pointers you also need to know how it did that. If it used the Windows GlobalAlloc() or LocalAlloc() function you need to explicitly call the according GlobalFree() or LocalFree() function. It could also have used the COM Allocator or some .Net Runtime allocator just for the fun of it. If it used the C Runtime malloc() function you are totally hosed unless the DLL also exports a specific function to free its own memory buffers. That is because the C runtime of the DLL you call could be a different version than what you call from your own application since it was compiled and linked in a different C compiler or even just a different Visual Studio version. Each C runtime implements its own malloc(), calloc(), and free() routines and friends and manages the according memory heap itself by requesting larger memory blocks from the OS through its kernel APIs and then taking junks of those blocks to satisfy m/calloc() calls and when it receives a free() call it looks up the block in its allocation table and marks that area as again available. Knowing this it is totally obvious that when your DLL allocated memory with malloc() from msvcrt140.dll since it was compiled with Visual Studio 2015, your free() call in LabVIEW unless you happen to explicitly use a Call Library Node to call the free() function in msvcrt140.dll will simply crash as the memory block was not allocated by this version of the C Runtime. And to make matters even worse there is something called SxS (side by side shared library loading). The DLL you call can define in its manifest that it absolutely wants to be linked to msvcrt140.dll (Version 14.0.1.2345) and Windows will happily load exactly that version of the DLL, while LabVIEW might choose to only simply link to msvcrt140.dll in which case Windows will usually load the latest and greatest version it can find (or another explicitly preferred SxS version). In either case Windows will load that dll too and link the LabVIEW link requests to that DLL instead. -> Instant crash when you call free() to deallocate the DLL created buffers in LabVIEW. If LabVIEW was compiled with a different Visual Studio version things will obviously be just as bad although here SxS doesn't come into play at all.
And the funniest thing is of course you might make this work by linking to the right msvcrt140.dll version to call free() from there and then you upgrade either LabVIEW or your DLL which was compiled in a different C compiler version and everything starts to crash. I wish you funny bug hunting with this!
Why bring pointers created by the dll into this?
If the dll does allocate memory it would not make anything more difficult as it is now.
@rolfk wrote:LabVIEW arrays and strings and the other forms at the same time: No
Fixed size inlined arrays and strings and array and string pointers in the same struct? Absolutely! Maybe not in trivial DLL interfaces but any sufficiently complex API has such combined structs for sure.
By definition, a sufficiently complex API is sufficiently complex...
I still don't see why a solution the fixes the problem for 95% of the problems is a bad thing...
Anyway, it's not my problem to begin with.
@rolfk wrote:
wiebe@CARYA wrote:
I agree that the configuration is already complex.
Maybe an option is easier that no option though...
Not really. You have two choices here:
1) Add this option: In its simplest way it will work for a few APIs but still fail many others. But the user will need to understand a lot about C compilers to even be able to make the right choice.
2) Don't add it: The user will then have two other choices\
2a) Do the whole pointer hocus-pocus on the diagram with calling DSNewPtr() and DSDisposePtr() and MoveBlock(). Convinient? In most cases not at all! You end up with a lot of conditional compile code if you want to support both 32-bit and 64-bit and need to understand C compilers to a level that is not funny anymore.
2b) Write a wrapper DLL that allows to translate between LabVIEW datatypes and your API. Complicated? Not really. Sure you need to write some C code but unless you are able to do that already you wouldn't be able to decide which option to use in choice 1) either. And in comparison to choice 2a) the level of expertise about C compiler low level details is several magnitudes lower.
So option 1 is bad because you need to understand C?
I'd say 2a and 2b both require even more knowledge of C...
I think there is huge gab between
+ the complex APIs you mention.
+ the difficulty of creating wrapper (maintaining even more software and source code)
and how the common user experiences these... In other words, you're probably dealing with way more complex situations, and have solved them a lot more often before, compared to the average user.
Anyway, back to my own problems (and weekend as well)...
@nathand wrote:
It would be difficult if not impossible to set up a configuration where LabVIEW could map an array inside a cluster to the "correct" C structure -
I disagree. There are not that many options.
@nathand wrote:
- it might seem straightforward for a simple 1D array as in this example, but once you get into 2D arrays and nested structures it would be an enormous challenge.
You can add complexity to any problem until it won't be solvable.
If we'd fix it for the majority of problems, it would be a win for me. But obviously not for you.
@nathand wrote:
Although it sounds appealing to try to make this configuration easier, any attempt to hide the complexities of C memory management and layout is likely to lead to more obscure problems.
I'm not trying to hide complexities, just to provide options.
And it's not about C memory management, the dll doesn't manage any memory. And the complexities are hidden as it is. And it already is obscure.
It just puzzles me that while there are just a few ways to communicate arrays in clusters with external code, LabVIEW only supports the one that is used only by LabVIEW.
wiebe@CARYA wrote:
So option 1 is bad because you need to understand C?
It doesn't just require you to know C programming. It also requires you to know what a C compiler does when putting data structures and pointers to buffers into memory in order to be able to decide which option you need to use.
I'd say 2a and 2b both require even more knowledge of C...
I agree that option 2a) is the worst in terms of low level knowledge you need to have. But I do not agree that option 2b) is worse. There are way to many potential problems with option 1) that an average user will never be able to decide properly.
Let's say you add the option to allow arrays (and strings) to be inlined (fixed sized) into the cluster. What fixed size should LabVIEW use? Configurable would be best except if you have more than one such array they often contain different fixed sizes. So the configuration dialog would need to be able to load the entire data structure (recursively since you can have arrays of clusters containing arrays of clusters etc. into the dialog and display this in a meaningful way and let you configure all the possible options for each element.
Or lets say LabVIEW simply assumes that the fixed size in the data structure needs to be whatever the incoming string or array has:
It's way to easy for someone to tinker with the actual data structure and inadvertently change the default data size of that string or array and suddenly LabVIEW passes an entirely different structure to the DLL which will almost certainly crash the software. Debugging possibilities for 99% of the potential users: None!
Now lets add the option to allow to pass an array data pointer instead of the handle.
Assume this simple API:
typedef struct
{
uInt32 opcode;
uInt32 *pData;
uInt32 length;
} DataStruct;
uInt32 RequestData(Handle handle, DataStruct *data)
{
// Do something
if (ReadFile(handle, data->pData, data->length, &data->length, NULL))
return GetLastError();
return NO_ERROR;
}
This will crash unless the user has made sure to:
& Initialize the byte array in the cluster to some size
& Make sure to pass this size in the "length" variable explicitly
And after return: to use the variable in length to resize the array to the indicated length if different.
That's a very simple example and already poses many problems to crash the code. Things get entirely unmanageable if the DLL can pass back pointers it allocated or has gotten from somewhere else.
I think there is huge gab between
+ the complex APIs you mention.
+ the difficulty of creating wrapper (maintaining even more software and source code)
Maintaining an extra wrapper is indeed more source code (and another tool as you need to have a C compiler). But maintaining a complex DLL configuration is in the long run at least as much work and if you ever intend to go multiplatform even a total pita.
and how the common user experiences these... In other words, you're probably dealing with way more complex situations, and have solved them a lot more often before, compared to the average user.
The common user uses LabVIEW in order to not have to be bothered about text programming. Except that the Call Library Node is not just about text programming but about the lowest common denominator of text programming with the exception of assembly code. Making the Call Library Node configuration more complex and adding more options to crash your code is not going to solve anything for 99% of the LabVIEW users. The number of forum posts that when something crashes randomly suggest to change calling conventions or similar things without the slightest clue about the subject is already big enough. Adding more options to change randomly won't help this at all.
Besides the whole discussion is anyhow useless. LabVIEW Classic is not going to get major feature upgrades anymore. And changing the Call Library Node to have additional options for passing structures is definitely a fairly huge feature upgrade with an accordingly challenging development effort both in the amount of C++ code to write as well as in redesigning the Call Library Node configuration dialog. Either one of the two is already to much to be expected to be added in LabVIEW Classic nowadays and both together just make the chance for it from almost nihil to 1/infinity.
Maybe it will come to NXG one day, who knows. After they caught up with LabVIEW Classic in many other areas first. One complication of such an exercise is the fact that part of this touches multiplatform code and that is always complicated to implement in all cases correctly and the testing is maybe the biggest time effort in such cases. Despite claims that they keep multiplatform options open for NXG I have quite some doubts that it will ever really materialize. That will make adding such a feature sometimes in a not so near future somewhat easier though as you only need to test it on one platform (well three maybe if they intend to support these features for cross compilation to the two NI Linux Realtime platforms).
@rolfk wrote:
wiebe@CARYA wrote:
So option 1 is bad because you need to understand C?
It doesn't just require you to know C programming. It also requires you to know what a C compiler does when putting data structures and pointers to buffers into memory in order to be able to decide which option you need to use.
But you need to know that anyway, because you want to communicate with it!
For now we have to flatten, or convert to cluster, or do whatever magic is needed. With an option we just don't need the boiler plate code, the option will do that. Obviously you need to pick the right option, at the moment we need to make the right solution.
The moral of the entire story is you need to do it right. The option doesn't change that. It will make it easier (less code) to implement the right solution.
The example will also crash in all other scenario's (even the wrapper) if you don't do it right.
If you make a wrapper, it will also crash if the DLL can pass back pointers it allocated or has gotten from somewhere else.
I guess we have to agree to disagree on this one.
@rolfk wrote:Besides the whole discussion is anyhow useless. LabVIEW Classic is not going to get major feature upgrades anymore. And changing the Call Library Node to have additional options for passing structures is definitely a fairly huge feature upgrade with an accordingly challenging development effort both in the amount of C++ code to write as well as in redesigning the Call Library Node configuration dialog. Either one of the two is already to much to be expected to be added in LabVIEW Classic nowadays and both together just make the chance for it from almost nihil to 1/infinity.
Well, it also applies to NXG...
wiebe@CARYA wrote:
@rolfk wrote:
It doesn't just require you to know C programming. It also requires you to know what a C compiler does when putting data structures and pointers to buffers into memory in order to be able to decide which option you need to use.
But you need to know that anyway, because you want to communicate with it!
Well if you write C code some of that complexity is not really important to know! For instance:
typedef struct
{
char myInlineString[25];
char *myStringPointer;
} DataRec;
DataRec dataRec;
dataRec.myStringPointer = malloc(25);
sprintf(dataRec.myInlineString, "Hallo World");
sprintf(dataRec.myStringPointer, "Hallo World");
is syntactically correct. The C compiler takes care about creating the right code to simply take the pointer for the myStringPointer variable and create code to create a reference to the offset in the dataRec structure to the first byte of the myInlineString. If you want to access these variables from an environment like LabVIEW these two things are totally different!
For now we have to flatten, or convert to cluster, or do whatever magic is needed. With an option we just don't need the boiler plate code, the option will do that. Obviously you need to pick the right option, at the moment we need to make the right solution.
For the inlined (fixed size) array, no Flatten is needed at all as shown in this thread. I would say that the Flatten solution only makes the whole thing more complex. You feel like the convert to cluster is a bad thing, but logically it is the right thing. C only does it different since it is so much easier to write:
struct
{
int32 data[256];
};
instead of:
struct
{
int32 data1;
int32 data2;
int32 data3;
int32 data4;
........
int32 data255;
int32 data256;
};
Logically this would be more correct!! But I bet you that every C programmer would have been throwing his computer out of the window after having writting such a definition for the second time.
The moral of the entire story is you need to do it right. The option doesn't change that. It will make it easier (less code) to implement the right solution.
Yes but I conquer that for the less than 1% of LabVIEW users who really are in the know about this, such an option makes no difference whatsover. For an additional 1% who are sort of in the know, but not quite, it makes it easier as they just have to switch a configuration setting, et voila. For the remainder of more than 98% of LabVIEW users it makes it extra complicated since the Call Library Configuration just got even more complex and more incomprehendable for them. Which group do you think will NI select to target if the implementation also costs a lot of effort, both for programming and testing it?
The example will also crash in all other scenario's (even the wrapper) if you don't do it right.
If you make a wrapper, it will also crash if the DLL can pass back pointers it allocated or has gotten from somewhere else.
Nope! If the wrapper crashes you wrote it wrong! If the DLL returns pointers you can deal with that in the wrapper in the way the DLL documents it. In many cases that could mean to pass in a LabVIEW array handle into the wrapper, have the wrapper call the API, and on return resizing the LabVIEW handle to copy all the content into the array handle and after that properly dispose of the pointer in the way the DLL programmer designed it. If the DLL designer thought using malloc() was a good idea and providing an explicit DLL function to deallocate that pointer is only for douche bags, then the DLL is simply broken and you can't even call it from a C command line program reliably. But that is not the callers problem. Such a DLL is simply and utterly broken!
Just as another example: Look at this thread. The OP has a problem of calling a Matlab DLL. He absolutely and definitely wants to use unbounded arrays as parameters for some Matlab Coder specific reasons. This is the VI only solution. It requires a lot of knowledge about how a C compiler calls different functions and allocates parameters, in addition to how to use this particular software API.
This is the equivalent C wrapper when you would create one:
#include "extcode.h"
#include "voigt_fadf_2.h"
#include "lv_prolog.h"
typedef struct
{
int32 cnt;
double elm[1];
} DoubleArrayHdl, DoubleArrayHdl, DoubleArrayHdl;
#include "lv_epilog.h"
MgErr CallVoigtFunc(DoubleArrayHdl in, double sigma, double gamma, DoubleArrayHdl *out)
{
MgErr err = memFullErr;
emxArray_real_T *in_arr;
emxArray_real_T *out_arr;
voigt_fadf_2_initialize();
in_arr = emxCreate_real_T(1, (*in)->cnt);
if (in_arr)
{
MoveBlock((*in)->elm, in_arr->data, (*in)->cnt) * sizeof(double));
out_arr = emxCreate_real_T(1, (*in)->cnt);
if (out_arr)
{
voigt_fadf_2(in_arr, sigma, gamma, out_arr);
err = NumericArrayResize(fD, 1, (UHandle*)out, (*in)->cnt));
if (!err)
{
MoveBlock(out_arr->data, (**out)->elm, (*in)->cnt) * sizeof(double));
(**out)->cnt = (*in)->cnt;
}
emxDestroyArray_real_T(out_arr);
}
emxDestroyArray_real_T(in_arr);
}
voigt_fadf_2_terminate();
return err;
}
While most LabVIEW programmers would claim that the LabVIEW only solution is easier to create, reality is that unless you have the knowledge to do the C wrapper variant you are utterly unable to create the LabVIEW only variant. And once you can do the C wrapper variant AND you have more than one or two of such APIs, then the scale tips extremely quickly to go with the C wrapper as being the faster and much, much easier to maintain solution in the long run.
Also note that the C wrapper is actually more prudent about error handling as it explicitedly checks that the Matlab Coder functions to create the unbounded arrays actually succeeded. The LabVIEW only VI lacks that error checking and could therefore theoretically still crash. Sure it would be ONLY the addition of two more case structure to that diagram, but I don't consider that diagram easy to read already.
Another salient detail is that in order to access the data pointer inside the unbounded array structure you have to do an extra MoveBlock() call in the LabVIEW only VI. In C this is simply solved by:
MoveBlock((*in)->elm, in_arr->data, (*in)->cnt) * sizeof(double));
^^^^^^^^^^^^
The C compiler takes care of such details for you. In LabVIEW you have to do it yourself. Sure you could write a whole palette of LabVIEW nodes to deal with such nitty gritty C details. But nobody would be able to use them since it is still more complex than simply knowing C and writing it in C. Or following your approach you can request NI to add a fourth option to the Configuration for pointer sized integers: to pass them as a reference to a reference to the actual pointer. Problem solved! But at the cost of one more complex option, and that while most people don't even understand the difference between a pointer passed by value and one passed by reference.
