Maintaining Unit Tests

[ASTDan]

Hello,

I am currently using NI's unit test framework, however maintaining all my unit tests can get to be a drag i.e. when I make changes in the code my tests break.  This seems to take a lot of my time.  Is this a common complaint?  Am I just whining, and should suck it up?

[FabiolaDelaCueva]

The thing to keep in mind about Unit Tests is that they are a thermometer for the health of your code. If you end up doing modifications to your code due to a requirement change, you want your tests to break if the API changed or fail if the unit does not behave as before.

In the presentation I did at NI Week, I mentioned that my unit tests have become a good measure to use when a customer requests a "small change". I can do the small change and then report what percentage of tests fail/break. This is a better measure of how small the change really is. After a while, the customer starts asking for the percentage of tests that fail/break due to a request and decide if it is worth it or not to make the change.

Regards,

Fab

[crelf]

ASTDan wrote:

Hello,

when I make changes in the code my tests break.

Well that's a good thing - it means your unit tests are working

So the more helpful answer is that "it depends". If you change something that makes a unit test go from passing to failing, then it depends on what the change was. It sounds like you're not interested in the change, so maybe it's time to look at the unit test and determine if it's testing what's actually important. Can you maybe share an example and talk us through what kind of change breaks it? Like, show us a simple VI with test (upload the files), then show us a new version of the VI where the original unit test no longer does what you want?

[ASTDan]

You make a good point of maybe it's not testing what's important.  That is interesting.

[tyk007]

When I look at creating unit tests I normally use the following as guidance (yes I have a .NET background):

http://artofunittesting.com/definition-of-a-unit-test/

The units of work definition might shake up the previously accepted notion of what a "unit" is but it can often reduce the burden of changing unit tests just because you refactor.

Unfortunately this is not practical in the UTF unless you use custom test VIs for everything.

[FabiolaDelaCueva]

Excellent link. I used to define a unit as a VI, but the more I have read of what is done in other languages, I agree that a unit could be a class, the importance is to test a single logical purpose that can be verified.

I agree that if the UTF is being used, then a wrapper test VI needs to be created for this purpose. The JKI VI Tester is more flexible in that regard.

[crelf]

tyk007 wrote:

The units of work definition might shake up the previously accepted notion of what a "unit" is but it can often reduce the burden of changing unit tests just because you refactor.

Exactly! A "unit" isn't necessarily a VI, a method, a class, a ... A unit is, well, a unit Defining where the unit boundary is is the most fun part, IMO.

tyk007 wrote:

Unfortunately this is not practical in the UTF unless you use custom test VIs for everything.

I don't entirely agree. Sure, there are going to be a lot of cases where that's true, but certainly not every case. And that's what makes the NI UTF attractive: there are times were you don't have to have custom test VIs, whereas all of the other test frameworks AFAIK will require them 100% of the time.

To be clear: I'm not necessarily on any tool side here - I've tried them all, and I like things about each of them individually. I'm with Fab in that I hope that this community can help grow the tools to be more full featured, so I can pick one and be completely in love with it.

[James_McN]

I don't entirely agree. Sure, there are going to be a lot of cases where that's true, but certainly not every case. And that's what makes the NI UTF attractive: there are times were you don't have to have custom test VIs, whereas all of the other test frameworks AFAIK will require them 100% of the time.

That's something I find interesting about UTF, It should be capable of everything and more because of that option, and yet for some reason I often shy away.

Minor plug, I did once make some VIs to try and make it easier though: https://bitbucket.org/JamesMcN/utf-custom-test-helper-library. Ironically consider it quite untested, it was a quiet spots work which I just put up once I ran out of time.

[tyk007]

James_McN wrote:

Minor plug, I did once make some VIs to try and make it easier though: https://bitbucket.org/JamesMcN/utf-custom-test-helper-library. Ironically consider it quite untested, it was a quiet spots work which I just put up once I ran out of time.

Any tools that help make UTF more feasible from a usability perspective are always appreciated. I still struggle with the maintainability of the lvtest files (source control / change control, lack of visibility over what tests are conducted, inability to easily segregate test etc.). I obviously still don't "get it" yet.