Point cloud library (PCL) and Open Computer Vision (OpenCV) toolkit for labview

[Klemen]

Hello,

this post will be extremely brief. I am putting together some  PCL (http://pointclouds.org/) and OpenCV (http://opencv.org/) functions that can be used in Labview directly via a .dll function call. So far, the following functionalities are implemented in Labview (there is a .vi in the toolkit that returns the current version of the toolkit):

PCL:

- downsample filter

- moving least squares filter

- statistical outliers removal filter

- passthrough filter

- kinect OpenNI based grabber (QVGA and VGA resolution)

- range image from point cloud

- sample consensus rough alignment and icp registration

- point cloud visualizer

- point cloud visualizer with real-tme redraw

OpenCV:

- bilateral filter

- face detection (Haar classifier) and eye detection (Haar classifier)

- grab cut segmentation

- histogram image matching

- histogram of oriented gradients human (pedestrian) detection

- Hough circles detection

- SURF real-time tracking

- template matching

- Shi-Tomasi corner detector

- Harris corner detector

- subpixel corner refinement

- Kalman filter/tracking (constant velocity and constant acceleration model)

- Hough lines detection

- Hough lines detection (probabilistic)

- histogram back projection (hue-saturation histogram backprojection)

- box filter

- Gaussian filter

- median filter

- Canny edge detection

- Sobel edge detection

- Scharr edge detection

- meanshift tracking (hue-saturation histogram backprojection)

- camshift tracking (hue-saturation histogram backprojection)

- face/object tracking (goodFeaturesToTrack->opticalFlow->similarity transform)

- stereo calibration and measurement

THERE ARE ALSO LABVIEW EXAMPLES FOR ALL ABOVE IMPLEMENTED FUNCTIONALITIES. The attached file named "PCL_OpenCV_LabviewToolkit_x64.zip" contains the .dll which is compiled agains x64 libraries. This .dll will not be updated as frequently as the toolkit (which is compiled against x86). In order to get it working, you need to relink the .dll paths.

Both PCL and OpenCV libraries are really extensive, so I plan to add some additional functions as the time goes by (no timeframes, since this is a hobby project).

To make things more simple and clear, all functions listed above have a corresponding example in Labview. The Labview code is in some cases a bit messy, but I do not have the energy nor the time to also make it look pretty. The most important thing is that all the examples have been tested and are 100% working. There are some issues with the PCL visualizers, so always close the visualizer window first and only then stop the Labview program. If you close a program that uses the visualizer functionality, close Labview also and restart the program/project.

The .dll's  were built using PCL 1.6.0 and OpenCV 2.4.9, so you need to have both. Also, please report in the comments box if the examples are working with other versions of PCL and/or OpenCV. If anybody cleans up the examples further, please let me know, so I can make modifications that will make things even more clear for potential users.

I have packed the toolkit with the JKI VI package manager, so you need to download it from:

http://jki.net/vipm

to be able to install the toolkit. This adds the toolkit to the block diagram pallete and simplifies things. There is also an addon that is not packed, since I was having some issues. I am talking ablout the 3D visualizer with real-time redraw, so in order to use it, you would need to manually add the function to your project via the "Select a VI" option on the block diagram pallete.

Also, if you are having any other issues, please report them in the comment box below.

P.S.: Some paths of the addon might be broken, but just point to them manually.

I HAVE REBUILT THE TOOLKIT USING OpenCV 2.4.9.

Also, you need to add the "bin" paths to the environment variables in order for the toolkit to work. In my case:

C:\Program Files (x86)\PCL 1.6.0\bin

C:\Program Files (x86)\OpenCV 2.4.9\opencv\build\x86\vc10\bin

Best regards,

K

P.s: There was some problem with the SURF tracking example (and coding), which was fixed 20.4.2015. If anyone experiences errors, download the toolkit again. All examples were tested.

[simons_84]

Hi Klemen,

thank you for the toolkit!!

I have a problem with the data format.

My PCL data is an array of points with x,y, z coordinates, but I’ve seen that the inputs of your “cloud Visualizer” function (and others) are 3 bidimensional arrays (cloud x, cloudy, cloud z)

Could you help me with the conversion?

Thank you,

Simone

[Klemen]

Hello,

since the data is passed as a pointer it does not matter if your point cloud is organized or not.

The "bidimensional" array is organized, but yours is not. Just modify the library function "visualizer3D_PCL". Change the dimensions of data source form 2 to 1.You still need to seperate the X,Y,Z array into three seperate arrays. Use the lenght of the array as a columns/rows input and value 1 as a rows/columns input. See the attached image.

IMPORTANT - FOR UNORGANIZED POINT CLOUDS SET "ROWS" EQUAL TO 1 (NOT "COLS").

The image above shows value of 1 connected to cols, which is incorrect!!!

Best regards,

K

[angloadi]

Can you please write down the LabView versions it supports?

Thanks.

[lizhuo_lin]

This is a great work! Hope you keep it going.

[Klemen]

Hello,

it supports Labview >= 2013.

Regards,

K

[_JordanHall_]

Will this work with LabVIEW 64-bit? It seems that the compiled DLLs are only for the 32-bit version.

[Klemen]

No,  x86 currently only.  Sorry.  It does work in x64 windows, but you need Labview x86 (and PCL,  OpenCV).

But, I have no tested the toolkit in x64 Labview,  so I am just taking a guess. I would be suprised if it would work.

Regards,

K

[_JordanHall_]

What is required to make it work with 64-bit LabVIEW? Is just compiling the DLL with a 64-bit compiler? Relinking the Call Library Nodes would be easy if you could provide a DLL for x64 targets.

[Klemen]

Yes, I would need to rebuild the .dll for 64-bit (linked to 64 bit OpenCV and PCL libraries). Then you would probably just need to replace the .dll in the Labview folder.

I can try this, but it will take some time - I can get to my computer in a couple of days.

Regards,

K 

[mishti93]

hi...actually i have labview 2012...vl this toolkit not work for it???

[Klemen]

Hello,

no, Labview 2013 at least required.

Regards,

K

[_JordanHall_]

Have you gotten a chance to recompile for x64? I would like to try it out.

[Klemen]

Hello,

I am getting to it today. Sorry, I had absolutely no time and also needed to find a computer to install all x64 dependencies. I don't want to mess up my setup on my computer, since I still need x86 functionality.

I will let you know and will also kindly ask you to test this with x64 Labview.

Best regards,

K

[Klemen]

_JordanHall_ wrote:

                       

Have you gotten a chance to recompile for x64? I would like to try it out.

                   

Hello, I have now attached the .dll compiled with x64 dependencies.

Please try it out and report back.

Best regards,

K

[_JordanHall_]

Thank you! I was able to get it working this afternoon with LabVIEW 2014 SPI 64-bit. I did have to relink the paths in the wrapper call, but after that it worked.

[Klemen]

I am glad it worked.  Thank you for the feedback.

Regards,

K

[Ignacio88]

Holaa,

Podrias compartir el archivo dll para un sistema operativo de 64 bits ??

Saludos Cordiales

[Klemen]

Hello,

the x64. dll is also attached.  Just replace and relink. I am not sure if all functionalities are implemented.  The list in the post refers to x86. dll.

Best regards,

K

[aldus12]

Hello,

In the "Labview PCLOpenCV Toolkit.zip" there is only one dll: "PCL_OpenCV_LabviewToolkit_visualize_redraw.dll"

Is it supposed to be only one?

Where do I find all the other libraries that are used in the examples?

Thank you,

Lorenzo

EDIT: I've found them inside the vi.lib folder!

[Klemen]

Hello,

I know I am not a good writer, but you should be able to decode from the post how to install the toolkit You need to install the vi package manager (see the link in the post) and run the "klemzip@gmail_com_lib_labview_pclopencv_toolkit-1.0.2.1.vip" that is in the .zip.

After doing this, you will find additional functions on the functions pallete.

Let me know if you got it working.

Regards,

K

[aldus12]

Hello,

I had to relink all the dlls to the path in my pc and initially I wasn't able to find them... Then I found them here: ...\National Instruments\LabVIEW 2014\vi.lib\klemzip@gmail.com\Labview PCLOpenCV Toolkit\dlls\

Concerning the OpenCV dlls, should I use the VC10, VC11 or VC12 version?

On a first try I get a call library node exception error with the VC12 dlls. Next week I'll try more thoroughly and I'll report to you.

Thanks,

L

[Klemen]

Hello,

use VC10 version.

Strange about the re-linking. The toolkit installation should take care of that. Can i ask you if you have x86 or x64 Windows? Do you have Program Files (x86) folder?

Thanks.

Best regards,

K

[aldus12]

Hello,

I got it working by using the VC10 version.

Thanks alot.

I have Win7 x64 with LV2014 x86 and x64 installed (but usually I work with LV x86).

Is there any rough documentation of the stereo vision dlls (or source code available to study)?

Thank you.

Best regards,

L

[Klemen]

Hello,

good that you got it working.

You can take a look at the OpenCV's web page for explanation of different parameters for stereo vision.

Best regards,

K

[Steckman]

I have tried installing all the required libraries etc, but when I drop in one of the example PCL VI's I get this error message. I am assuming I must have missed a step during the install but not sure what. I am using Labview 2015. Any help would be appreciated.

[Klemen]

According to the error, you are either missing OpenCV 2.4.9 or you did not add the bin folder to the system path.

Regards,

K

[Steckman]

Thanks for the response. I found that I had overloaded my path statement with too many characters. After deleting a number of unused path information, the library and example seem to load fine.

Appreciate your efforts in supporting these libraries.

Best Regards

Scott

[Steckman]

I have continued to work to utilize the PCL libraries you provided. I have a set of data collected with a laser scanner, and have individual X,Y,Z arrays rather than ordered grids. I noted your post from earlier indicating to change the CLN function to look for 1D arrays rather than bidimensional arrays,which I have done. The visualizer call works fine with my data, but when I try and run it through the outlier filter, labview keeps crashing on me. In order to run the data thru the filters, does it need to be in an ordered fashion? Unfortunately I am not sure how get my data into that form?

Any help would be greatly appreciated. In the end my goal is to scan a part that has some noise etc, filter it using either the outlier algorithm and/or the pass thru filters and then interpolate a Z value based on discrete input X&Y values.

[Klemen]

Hello,

I have updated the toolkit. There is an option to keep the cloud organized (check passthrough and outliers removal examples). Try setting the keepOrganized value to false. I did not test this though, so I will appreciate it if you report back.

Best regards,

K

[Steckman]

I installed your updated code and re-ran my test data. The keepOrganized off value seemed to do the trick and I was able to process the data set thru the outlier algorithm. Thanks for your help.

Scott

[Steckman]

Sorry to keep bothering you with questions. I have been doing more work trying to use the PCL librairies. Now that I have the function calls operating without crashing my system I have been runnng some sample data thru them and unforutnately I am not getting what I beleive to be the right type of results back. For instance on the pass thru filter it appeares that it was taking the minimum value and calculating correctly but not using the max limit value. When I checked the output arrays I was expecting a reduced number of points in each array, yet the array size remained the same as the input array. I have since created my own pass thru algorithm as this is pretty straight forward, but I would like to try and use the outlier or MLS filters to smooth out the laser scan data, and when I run these functions they run without error, but the data out doesn't really appear as I would expect. For instance the MLS filter outputs a plane of point on a 45 degree angle relative to the incoming points? I could send sample data\code if you were interested in looking at it.

Thanks

Scott

[Klemen]

Hello,

I appreciate every feedback.

I don't see any problems with using organized point cloud. All my test were performed using organized point clouds. But it is true if I break organized structure, some strange results are seen.

Could you please send your sample data and code (with modified .vi's for 1D arrays - unorganized clouds) so I can test this and check where the problem is?

Best regards,

K

[Klemen]

Hello,

to continue the discussion on the comments section (so that others can see this too):

First, I am almost 100% I know where the problem lies - basically the (my wrapped) functions are created to deal with organized point clouds. That means after filtering, the data size stays the same, eventhough some points are removed (actually they are replaced with NaN). It is different when using unorganized point clouds. Therefore, I will need to make another function to test this in order to handle unorganized point clouds. I will try this and get back to you.

Also, for unorganized point clouds use cloudRows = 1 (and not cloudCols = 1)! Check here:

http://pointclouds.org/documentation/tutorials/basic_structures.php

P.s.: Statistical outlier removal seems to work on the scan you've provided.

MLS filter also seems to work.

Best regards,

K

[Klemen]

Hello Scott,

now the passthrough filter should work on unorganized point clouds (tested on the data you've provided). I have uploaded the new version (it should be up shortly, the post is currently moderated - version is 1.0.2.4). There is a new control to specify the filter field (x, y or z).

Best regards,

K

[Steckman]

I been testing the new uploads on the unorganized clouds and things seem to be working well now. I haven't tried the pass thru yet, but have been working with the MLS and Outlier filters. On my data set the MLS seems to take much longer to run compared to the Outlier filter but I am assuming this is due to the type of calculations being performed and the settings used as inputs for each. Thanks again for your assistance and efforts.

Scott

[Klemen]

Thank you for your feedback!

Yes, the MLS filter is more computationally expensive.

Best regards,

K

[kovacsellert]

Hello,

Thank you for your work on this toolkit!

I am trying to test the face detection example but I am getting a Library not found or failed to load error. I manually relinked the the .dll file in all three subVIs, as the initial path was not correct, but it still does not execute. (When I try to configure the Call Node VI I select the correct path to the dll and all the functions appear in the dropdown menu, so LabVIEW can see that the dll is there.)

I also set the environment variable to OPENCV_DIR to the v10\bin folder.

What am I missing?

Thank you for your answer.

[Klemen]

Hello,

(under Win 7) open control panel-> system -> advanced system settings -> environment variables -> look under system variables for path -> edit -> add the path to the \vc10\bin folder.

Best regards,

K

[kovacsellert]

My opencv folder is here: D:\opencv

I added the following environment variable: D:\opencv\build\x86\vc10 with the name OPENCV_DIR

Then I edited the Path variable and added to the end: ;%OPENCV_DIR%\bin

I also relinked the dll inside LabVIEW to the following path:

D:\National Instruments\LabVIEW 2014\vi.lib\klemzip@gmail.com\Labview PCLOpenCV Toolkit\dlls\PCL_OpenCV_LabviewToolkit.dll

Finally I copied all the necessary xml files to:

C:\Program Files (x86)\OpenCV 2.4.6\opencv\data\haarcascades

I still get the same error when trying to call the dll.

I am using LabVIEW 2014 32bit on Windows 10.

I don't have PCL installed, only OpenCV. could that be the problem?

UPDATE: I installed PCL 1.6.0 and added the \bin directory to the Path variable. Still no success...

[RaptorUK]

Hi Klemen,

thank you for sharing. 

I have an application where I need to do some image matching so your toolkit seemed like it would be exactly what I needed without the expense of buying the LabVIEW Vision Development Module.

I had some issues getting my system to see the DLLs but I solved that issue.   So now I have your template matching example running, albeit without the LabVIEW Vision stuff which I have removed.  What I am seeing that confuses me is that the template image is always found even if the main image doesn't contain the template image . . .  I am using JPGs for these images.

For example:

Main image:

Template image:

Am I doing something wrong ? or am I misunderstanding what should be happening ?

Hope you can help a little,  many thanks.

Simon

[kovacsellert]

Hello,

Regarding the issues I am having when calling the dll associated to the library:

I found out that the problem is related to my OS, as I am using Windows 10. (The toolkit run perfectly on a Windows 7 virtual machine installed on the same laptop.)

If you have some time please help me with some tips how I could get the toolkit and the dll call working under Windows 10 with LabVIEW 2014. Thank you.

[TRIKM]

Hello,

I have win8.1 x64 with LV2015 and using toolkit x64 dll.

(PCL:1.6.0 msvc2010 win64, opencv:2.4.9,  openNI:1.5.4 win64, Sensor:1.5.4 win64)

I'd like to try PCL filter, but, Outliers removal  and Downsample error has occurred.

(MLS is no problem)

and  "filterPassthrough_set(get)_PCL" is not found in toolkit x64 dll.

there is  "filterPassthrough_PCL" only.

Could you check the toolkit x64 dll？

Your help would be appreciated.

[caleyjag]

I have added directions to the PCL and OpenCV bin folders as you suggested in my system path (Windows 7, running LabVIEW 2015 64-bit) but I keep getting this error.

As far as I can see Open CV and PCL have installed correctly. Has anyone else had this issue?

[abhishek312]

[abhishek312]

Im getting this error when i try to execute visualizeCloud_pcl.vi. I followed all the above instructions. Please Help!!

[VandanaR]

I have a Windows 10 x64 operating system with LabVIEW 2015 32-bit installled along with PCL 1.6.0 and OpenCV 2.4.9 in Program files (x86). Installed the PCL Opencv Labview toolkit throught JKI VI Manager. When I drop the visualize_exmaple.vi from Labview PCLOpencv Toolkit in the funcations palette get an error message - "Library not found or failed to load.". I relinked the dlls from C:\Program Files(x86)\National Instruments\LabVIEW 2015\vi.lib\klemzip@gmail.com\Labview PCLOpenCV Toolkit\dlls\ .. and the error is still there. Any suggestions ?

[bseguin]

Same thing here, it's seems that in the distribution package it's missing one dll

plc_common_release.dll

Is it possible to tell where to get this dll or to add it in the package?

Thanks

Benoit

[Yan1983]

Hi,

I also meet the same problem in LabVIEW 2017 (32bit and 64bit). Does anyone know how to solve this error?

Thanks a lot.

Asher

[madison7]

Greetings! I have followed the installation through VIPM and the forum instructions in exhausting detail and keep getting error 13 (see attached). I am using Windows 10, LabVIEW 32bit 2017. Seems others have had similar issues - wondering if there has been a solution found?? We are really eager to utilize the toolkit.

Many thanks in advance!

Anne

[madison7]

FYI, we found that the problem was an unsuccessful OpenNI installation. Install from here: 

http://sourceforge.net/projects/pointclouds/files/dependencies/OpenNI-Win32-1.5.4-Dev.msi/download

Once you install, you should find that the paths no longer need to be redirected. Hope that helps anyone receiving dll errors!

Anne