Unit Testing Group

Upon public request I just posted an introductory video to learn how to get started with JKI's latest unit test framework Caraya

This is a pretty good tool you has created, it has potentials

I think it would be even better if it was possible to right click on in the project explorer on the VI you has created the unit test for and run the associated unit test. In this way it is easy for others to see if there is a unit test created. This is an issue we are facing in our project because it is a large project with multiple developers.

An other issue. You make a test all VI for running all the unit tests. Here it would be cool if there is way for searching a project for unit tests and run them, like a button at the to of the project explorer. This would be be more fail safe when automating the test from Jenkins or TeamCity, maybe byhaving an API for the tool. The developer is not always adding the single unit test to the all VI

Is it possible to get a report like NUnit or JUnit xml from the test?

I have a question for your video. Why are you applying the the assert test in paralle and not in line?

kljeld

Thank you for your great feedback. Your questions get directly into the heart of the philosophy why we created Caraya and what makes Caraya different. Unfortunately Caraya is designed to be something very different that what you may be looking for. But you may still fall in love with the simplicity of it even though it does not and maybe even cannot have some of the functionality you are requesting.

JKI's VI Tester, another unit test framework from JKI, is designed to work the more like the way you describe. It allows you to run all the tests for a project as you suggest. I recommend using VI Tester, it is a great and proven unit test framework and is a really good fit for the things you're requesting.

With Caraya we intentionally took a very different approach: a test case is a VI, running the VI runs the test case, you can compose test hierarchies by using test cases as subVIs. These are the fundamental design principles of Caraya that makes Caraya special. But also creates some limitations on the aspects you were requesting.

I think it would be even better if it was possible to right click on in the project explorer on the VI you has created the unit test for and run the associated unit test. In this way it is easy for others to see if there is a unit test created. This is an issue we are facing in our project because it is a large project with multiple developers.

As there is absolutely nothing special about Caraya test VIs there cannot be anything special about running them from the project. They are just VIs and we want to keep them this way to keep Caraya simple. LabVIEW already provides a very good support for running VIs. If someone would create a separate tool for running VIs directly from a project explorer, that would be great but there is nothing special about Caraya test VIs in this perspective and hence such a functionality really shouldn't be part of Caraya.

An other issue. You make a test all VI for running all the unit tests. Here it would be cool if there is way for searching a project for unit tests and run them, like a button at the to of the project explorer. This would be be more fail safe when automating the test from Jenkins or TeamCity, maybe byhaving an API for the tool. The developer is not always adding the single unit test to the all VI

For the same reason there cannot be a run my application button in the project explorer, there cannot be run my test button in the project either. There is simply no way of knowing what to run. As LabVIEW doesn't know what is your application and how to run it, LabVIEW doesn't also know what is your top level test VI and how to run it. This is a design choice we made with Caraya that makes Caraya very easy and very powerful tool for unit testing. But because everything is code it makes it more difficult to automatically discover all the tests, reconstruct the test hierarchies properly and execute them. I don't see this functionality be applicable to Caraya. Of course someone could write a heuristic code analysis tool that traverses all VIs in your project to determine which ones are tests and which ones are top level tests that need to be called and then calling all the top level VIs but then you kind of loose everything the speed that makes Caraya special.

What could be added, in principle, is a build spec for running the unit tests similar to a build spec for building and EXE, where you define your top level test case. But again your tests are code and your composition of tests into hierarchies is code. It is up to the developer to specify in code how the tests would need to be run. This provides great flexibility for the user but does not work well for the use case of searching through the project on what to run.

Is it possible to get a report like NUnit or JUnit xml from the test?

Right now we only support text reports but we are planning on adding a report provider interface allowing users to customize reports. Caraya allows infinitely deep and complex test hierarchies that may be something that is not supported by NUnit or JUnit xml report formats. If that were the case we would need to extend the report format to work with Caraya making it potentially incompatible with report viewing tools when Caraya would be used to run complex test hierarchies.

I have a question for your video. Why are you applying the the assert test in paralle and not in line?

This is a great question. As the tests are code, we can choose in code weather to run the tests in parallel or sequentially. When ran in parallel, all the tests are independent from one another. When ran sequentially, the previous tests can impact the tests following them. Especially if a previous test fails and returns an error then the following tests are not executed following normal LabVIEW execution pattern (however Caraya still reports that the tests received an error as the input). The reason I had the tests in parallel is because I wanted all the tests run independently of one another and the outcome of one test not impact the other test. This may not always be the case and you as a developer need to choose what would make most sence for what you are testing. For example if you are testing database operations and you cannot connect to the database you may not want to run all the database tests so it's okay to have the database test VIs follow the set-up VI connecting to the database.

As a rule of thumb you most likely want your tests to run in parallel unless there is a dependency between two tests (such as "write key-value pair" test executing before "read value by key" test).

Tomi

I released version 0.3 of Caraya today with some usability improvements. It's now possible to open the failed tests by double clicking on the results. Several users have requested this feature.

Tomi

Hi,

After a long time working with the VI Tester, I rencently decided to try Caraya and see which were my feelings, after a little time playing with this tool these are my thoughts.

• I like Caraya as much if not better than the VI Tester.
  • It is simple.
  • The way to create tests seems more natural than having an initialization an shut down proces and everything embbedded within a class.
  • I like the fact that you provide with multiple comparissions instead of a single VI to introduce the test result,
  • It is very simple to create a suite and this way to do so seems more natural than the VI tester.
• Though there are some little things that were improvable.
  • Error code 1 when the test fails: I personally hate this error code, actually this is the one that you receive when you lose a reference to a queue or an event, etc. I would personally assign a custom error code that we can differenciate better, and does not match possible error codes that we might be interested to find out, or I would simple fire a warning notifiying the failure to pass the test.
  • Reporting.
    • On the one hand I miss a bit more information about the test than a pass or fail, I'd like to also see the result of the test.
    • As someone else mentioned before I miss a more universal unit test report format, let's say JUnit. Although I understand what Tomy mentions about the possibilities that Caraya allows, I bet there is a way to create a file of this format up to a certain level.

In any way, thank you guys for the good work and the tool, and also for making it open sourced, which means that we can try to improve the little cons on our own.

I hope you find the feedback useful.

Jpina escribió: Though there are some little things that were improvable. Error code 1 when the test fails: I personally hate this error code, actually this is the one that you receive when you lose a reference to a queue or an event, etc. I would personally assign a custom error code that we can differenciate better, and does not match possible error codes that we might be interested to find out, or I would simple fire a warning notifiying the failure to pass the test.

I've just found out that we can define an specific error code in case of failure (or at least I understand we can), thus the comment above makes no sense.