LV memory leak - How to use windows API SetProcessWorkingSetSize (from Kernel32.dll)

[Navne]

Hi fellow LV'ers

Okay - this is a bit tricky, but i'll try and explain the problem, then ask for the solution, because it may be that someone knows a better way to deal with this.. might get a bit long, sorry - if a solution comes up this will enable all of us to make more memory efficient LV code so please read on..

Here is the deal:

When building even a very simple LV executable, looking at the windows task manager will yield a rather large amount of memory allocated for such a small program - and the only way to free this up is by physically clicking the windows minimize button, then suddenly the amount drops to only a few MB and upon maximizing the window again the memory consumption will increase somewhat again, but for a simple VI build to an exe this move may change the consumption from +70MB to less than 15 MB.. This is irregardless of the code you put in the VI, so no coding example in this post as it is how LV works - you can even test it with the development environment - look at the task mgr and check LabVIEW's memory consumption, minimze ALL open NI windows incl project explorer etc, and you will see a significant decrease in memory usage even after maximizing again.. This has annoyed me since day one, but since RAM is a near zero cost these days it is not something I stay awake at night to think about.. However - I have moved into the "publish to web" tools now, wanting to do a remote monitoring part for my application for my customers to experience increased usability from the software i sell them..

All is well, publishing is really easy (i use the monitor function, NOT the embeded, as customers need not have Labview RunTimeEngine installed, because they might look at it from a non RTE supported platform such as a mobilephone web browser)

Everything is working fine also for the build application. However - I have noticed that once users start to remote monitor the running application - memory consumption of the running LV application starts to increase - and it keeps doing so - to such an extend that you can drain the computer complete and run off the cliff with a windows error... This is off course not very productive for me, being specialized in measurement applications that usually runs for a long period of time - I initially thought that I had done some poor programming in the VI used to display on the webpage - but it turns out that I can reproduce this behaviour with a simple boolean on an empty front panel..

NI support has been informed, and they admit there is a problem, but so far solutions from them has been a bit too exotic for my taste, and thus I'm seeking the help of fellow LV programmers...

You see - The method to solve the increasing memory consumption, is the exact same as mentioned above..minimize the application running with the "minimize" button and all memory will be freed, as soon as you maximize the application and users are viewing it remotely, the memory usage raises again, and history repeats... As previously mentioned, minimizing the window via normal LV calls to property nodes does not yield the same result, nor does a request deallocation of a VI(When you profile a project, there are no VI's increasing in memory, it is the LV process it self doing it) 

After many many hours googling I stumbled upon this:

http://support.microsoft.com/?kbid=293215

I believe trimming the process with SetProcessWorkingSetSize would solve this problem, and now I would really like to be able to do this in my program, so that users are not forced to minimize the program every X hour depending on their system size...

However - I have absolutely NO experience in calling windows API from LV, i need someone with that knowledge to provide an example of how to call this.. I've looked at examples on how to do calls to windows API - there is an example in this forum with some llb's in it, and I have gained a fair understanding of how parameters are passed between the calls, but none of those include the "hProcess" handle that is apparently needed for this specific winAPI call to work - Anyone in this forum with the knowledge on how to obtain this handle from a VI, if at all possible, and could provide an example VI for me to use - or even better , someone with the knowledge of how to do this within LV it self??

Your help is much appreciated

Best Regards

Jacob

LV8.6.1 patch something

Win XP 

[tst]

I can't comment on the issue you're seeing (I have no practical experience with RFPs), nor can I comment on the use of that API function (other than to say that I would highly recommend AGAINST playing with the memory allocation of a production app).

One practical suggestion I can make is for you to work around the need for the RFP. Since you're only using the monitoring functionality, implementing most of that functionality on your own should be relatively simple - the VI class has a method which allows you to get the FP as an image, which you can then save to a PNG (preferable) or JPEG file using the picture VIs. You can then build a small HTML page which will display that image and point users to that page.

You can easily add a refresh tag in the HTML so that the page reloads every X seconds, thus always displaying the current image .

[Navne]

Hi TST

I'm aware of the solution with simply dumping the front panel as an image, and I guess creating a HTML document that displays the image and autoupdate is doable, but I would like to be able for users on the LAN / WAN network to access this  html doc, simply by typing the .html link after the local IP of the computer.. (i.e. 192.168.0.20/whatever.html)

As I'm good with LabVIEW and not network / html I might have missed something, but I believe that I would need to install a small webserver to actively listen to port 80 for incoming connections, and then display the page asked for by the remote webbrowser - this means finding a open source webserver and learn how to set that up plus adding complexity to the installation build and setup at the end user, who I must consider to have less experience in computers than I have to ensure that our support division is kept unbusy:-) But is there a simple way to broadcast a html document across the network that I have missed?

[tst]

If you enable sharing on the folder where the HTML file resides, I believe the users should be able to access it directly as a file without using HTTP, so no need to install a server (e.g. you tell them to input the address "\\10.10.10.10/c/publish/main.html", where the double backslash is used to refer to a machine).

The auto update is not only doable, but fairly simply (search for "meta tag refresh" and you should probably find something).

[cosmin · Accepted solution]

Hi,

check the document http://decibel.ni.com/content/docs/DOC-9292

it's a sort of task manager with the ability of trimming the memory for any process.

It calls SetProcessWorkingSetSize.

maybe it wil help you

cosmin

www.controlsag.ro

[Navne]

COSMIN - 

You just did what 700 USD a year in NI SSP and two weeks of NI support telling me what I all ready knew couldn't do:-)

1e6 Thanks!

best regards

Jacob

Just for the ease of google / LV forum searching strings:

Solution Ni web server bug memory leak reduce memory size

[Enrico_Segre]

Jacob,

I'd like to highjack your thread, as I think that I'm experiencing the same problem. I have a moderately complex supervisory plant monitor, which I deploy as compiled labview executable, and which users see as webserved panel images. Recently, after migrating to LV9/winXP and after minor changes in the code, I started to have elusive memory leaks, which ultimately lead to process crash as the VM size of the process reaches ~2Gb. After some vane attempts of isolating the part of the code causing the leak, I have some confidence in that LV webserver is to blame, when too many users keep watching for enough time. Your one is the first relevant thread I stumble into, after some unrelated searches.

I too am not considering to undertake the hassle of finding an alternative webserving mechanism, rather than using the tool coming with LV. Therefore, what I'm going to do next, is to try to add to my application a periodical call to emptyProcess, from the processHandles.dll provided by cosmin on http://decibel.ni.com/content/docs/DOC-9292. However, that looks to me little more than just a workaround.

My questions are:

• you mention the LV support reply. Was there an official position? Is the problem known to be LV9 only?
• As far as I can understand, this emptyProcess trick only trims the ram size of the process, not the VM size of it. I'll see if in practice doing the trick frequentle enough also prevents the VM size to grow, but ultimately I'm concerned about the memory leak not being really solved
• Can anyone suggest a quick way of getting in LV the pid of the process to be trimmed (notably for me -of the parent LV executable)? I expected some kind of Application property node or Connectivity/System exec VI providing this number, but can't find any. I'll resort to some construct based on identifying the process name, looked up by get_proc_andIDs.vi from the same contribution of cosmin, but wouldn't there be a more direct way?

TIA, Enrico

Message Edited by Enrico Segre on 03-21-2010 08:33 AM

[Navne]

Hi Enrico

Finally I can give something to the community that has given me so much  🙂

The "official" statement is that "yes we know it is a problem".. Not sure what that will do to the future.. 

I have the problem on 8.6.1 as well  - and in fact it is a general LV problem, that I first time reported to NI with LV8.2 as I was pissed by the fact that even the smallest exe file would consume + 50MB of memory until you manually minimized the window. Well - thanks to the feedback from Cosmin I seem to have solved the problem.

I most warn that having started to "empty process" once in a while has led to occasional program crashes in the lethal "app.exe performed an illegal action and is closed" windows dialog - however what I did was to move the webserver to a seperate exe file and then communicate the data that I want to use via datasocket in a cluster.. It works like a charm and I simply stall the single thread that the webserver is running when ever the empty process is called and I have not seen a crash since then.. (the initial implementation was done in the main app with 4 parallel loops running, and I guess that was a disaster waiting to happen)

Either way - what I have done is made a VI that at a user defined interval calls the empty process, simply by getting the .exe name from the task manager of the calling program - it is simple and very effective. I call it every 5 minutes - needless to say that flushing too often will most likely kill performance of the system. I have not noticed problems with VM - are you sure you are not storing large arrays or moving around copies of data not used frequently?

For future reference to this forum, it is attached here including the .dll required to call - it is a LV8.6.1 file as I have not had the time to yet again test every single function of my program for new problems that could occur with upgrading to LV2009 

I hope this solves your problem..

best regards

Jacob

www.10EaZy.com 

[Enrico_Segre]

Hi Jacob,

Interesting to know your experience. For some reason, though, I became aware of the unbounded growth of my application only after the migration to LV9. Maybe the problem worsens in LV9, maybe that was associated with a more frequent users web view, maybe not.

I have now inserted in my code a periodical cleanup along these lines:

 

This btw is my dirty solution so far to find out the pid of the process - I rely on that it has been called by a *.exe entry point.

I just did the change, so I have still to evaluate over a distance the stability of the solution - effectiveness in the typical usage scenario, errors, performance vs frequency of trimming.

Now my code indeed does a lot of graphics and moves around arrays in memory, so has a non nil potential for memory leaks caused by my own programming; however, I hold my observation about the VM -- if I point multiple times a browser to the webserved VI, I can grow very quickly both "Mem Usage" and "VM size", without cleanup; with cleanup, I can keep "Mem Usag" within very reasonable limits, but I think that "VM size" can still grow. I have still to check it extensively, though.

Enrico

[Enrico_Segre]

Hmm, interesting. Searching better ("memory leak webserver") I ran into other very similar threads -this one, this one and this one (last posts here of Jacob and cosmin)- all very recent. Everybody is complaining about leaks when refreshing monitored pages, and is just recommended to use "embedded" (but embedded needs RTE!). Then somebody realizes that minimizing windows circumvents the problem (but I think only to a limited extent). I now learned that this is Known Issue #156479. Out since almost one year, on a component existing since a decade, and yet unsolved. Too bad! Some of the workaround proposed look too cumbersome to me, I might resolve to downgrade to LV8.5, which in my experience was not suffering of the bug.

Enrico