Unit Testing Group

cancel
Showing results for 
Search instead for 
Did you mean: 

Unit Test Framework and Continuous Integration

Jarobit started a discussion regarding a command line interface for the JKI VI Tester so it could be used in Continous Integration. (see: https://decibel.ni.com/content/message/104670#104670)

One of the replies described doing something similar with Unit Test Framework (kubes_ch https://decibel.ni.com/content/message/104700#104700)

I thought it would be beneficial to separate both discussions.

Anyone else has experience with integrating Unit Test Framework and Jenkins or other CI servers?

Here is a copy of Matthias approach (aka kubes_ch):


kubes_ch wrote:

To start and control UnitTest I wrote a small LV-Application who will called by Jenkins. This Application is called as a cmdline tool like "LabVIEW.exe -- args" and this starts the Unit Tests dependend on the given args in the command line. In a similar procedure like JKI does. I think the presentation and demo code is still available on the JKI site.

The only problem still exists, is to bring the result in a supported xunit format. But it should be possible to convert the results from the UnitTest format in the junit xml format in the LV application so then Jenkins can collect it.

ATML seems to be a better result format but unfortunately Jenkins xUnit Plugin doesn't support ATML result files but there is a possibility to transform it directly in Jenkins Unit Test Plugin by a custom xsl (https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin)

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 1 of 12
(12,640 Views)

We do something similiar for VI Tester, UTF and our own home-brewed framework. We're using TeamCity which can parse the output stream to retrieve results for each test and maintain those as part of the build.The builder application works similary to described except we store the bulk of the configuration in an ini file that is read by the builder - the command line args contains the build version to build to.

The testing is done via dyanmic dispatch to allow us to swap in the appropriate framework wrapper at run-time depending on which framework (or multiple) the config file specifies the builder should run. This helps when we have multiple frameworks in use in a project but does mean that it drags in all the dependencies too.

The fact that TeamCity uses the output stream has made parsing the results in the build server easier than say storing the results seperately as a file since the build server can then track the history of each test etc. across builds.

Here are some screenshots of the last build of a component from our TeamCity server. The first shows, for a single build (2012.5.30.3524) the tests that were executed and the second shows the history of one of those tests (time, pass/fail etc.). This component is built for 2012 (hence the majpr version), component API v5 (hence minor), build 30 (assigned by build server) and SVN source version 3524 of the Repo.

TC All Tests.png

TC Test History.png

Message 2 of 12
(9,302 Views)

Thanks for sharing!

I like the versioning scheme you use, you can tell just from that where the code tested came from.

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 3 of 12
(9,302 Views)

I forgot to add this screenshot of the change that triggered the build server

TC Build History.png

0 Kudos
Message 4 of 12
(9,302 Views)

Does Teamcity use the return code to determine success?

If so how have you wrapped it and output it?

I'm keen to build a standard wrapper for this so we don't have to all create our own. My idea is to really have a command line API that can be called in LabVIEW and hides the details of the wrapping.


Then hopefully we can get to one single CI integration too which can plugin different frameworks, or at least greatly reduce the barrier to entry

James Mc
========
CLA and cRIO Fanatic
My writings on LabVIEW Development are at devs.wiresmithtech.com
Message 5 of 12
(9,302 Views)

As I mentioned, we are not able at this time to parse the unit test results because of the result format from UTF. So actually there is no possibility to do some statistic statement or check the history of tests. We just give the users the link to the stored html report.

After a new checkin from a developer to svn, the CI Server will be triggered to check out a fresh copy and start the UTF on the buildnode. The simplest way to do this is this:

UTF start.png

after the UTF is finished, the CI Server checks for a report and publish it, if found.

For VI Analyzer we use the same approach like tyk007, there we parse the output stream in Jenkins delivered by VI Analyzer. After this, CI is able to track the history of each warning or error and gives feedback to the user if a warning limit is reached. The VI Analyzer takes a lot of time, so our CI triggers this job once in a night.

VIA Overview.png

overview for a single project

VIA details.png

overview a specific build

Matthias

_______________
Automate LabVIEW builds easy with Jenkins Plugin www.kubes.ch/Jenkins
Message 6 of 12
(9,298 Views)

In fact that you have to use the development environment from LabVIEW to build or using UTF and VIA i'ts not possible to get the last errorcode nor a meaningful return value on the cmd line. LabVIEW always sets the return value to "no Error" if you exit the development environment.

The best and easiest way is, to send errors and return values from your "wrapper" to CI Server via Console.Write or use a wrapper who is always running.

Matthias

_______________
Automate LabVIEW builds easy with Jenkins Plugin www.kubes.ch/Jenkins
Message 7 of 12
(9,302 Views)

Yes, we set an envinromnent variable which the TeamCity build script looks for to determine success / failure. As such it is very Windows OS specific.

0 Kudos
Message 8 of 12
(9,302 Views)

An easy solution is to let the script that calls LabVIEW to create an empty file. Then let the LabVIEW application (the wrapper application that runs the UTF) write any errors to this file. The script then examines this file and reports status back to Jenkins together with the error information.

0 Kudos
Message 9 of 12
(9,302 Views)

This is the solution I have seen used before. I think it would be useful though if we could provide real-time updates. At least Jenkins allows you to watch the build progress as it works.

My current thought is a small C++/# console app which opens a socket and LabVIEW can relay it's output to the socket, including a return code.

James Mc
========
CLA and cRIO Fanatic
My writings on LabVIEW Development are at devs.wiresmithtech.com
0 Kudos
Message 10 of 12
(9,302 Views)