Test Evaluation - What types do you often use?

[tyk007]

Depending on the project and developer test style there are often different types of test "assertions" that are used (independent of unit test framework). Common ones are that a test is Pass if a boolean input is true (generic) or that two instances are Equal. Many other types of assertions can be developed from these basic primitive tests on a case-by-case basis. However there are often common patterns for test assertions that are used and repeated and are thus worth-while capturing for re-use. The NI Unit Test Framework, for example, contains many options to provide the developer flexibility in generating these comparisons.

Since the projects I work on a rather insular, I am interested in what types of "common" assertions people find themselves repeating in their projects. We are looking at expanding an in-house tool we use for our unit testing and value any information we can gather from the community.

As an example, here are a few that I commonly use:

• Is the boolean input true (generic, can represent anything with extra test code)
• Are all boolean inputs (array) true? (super-set of the one above)
• Are two floating point numbers approximately equal? I typically only use absolute delta or signficant digits
• Does a string contain a search string (including case-insensitve option)
• Is an Error produced (cluster with appropriate error code).

Thanks for any and all feedback.

[Thoric]

Related link to Assertions in LV: https://decibel.ni.com/content/message/90246#90246 (You need to be in the CLA Community to access this - if you can't follow it, this points to a presentation by Peter Horn of NI UK who has recently developed an Assertions Toolkit for LabVIEW)

[tyk007]

This sounds great but unfortunately I can't access it. Is there any way to post the link to the presentation?

[FabiolaDelaCueva]

tyk007 wrote:

This sounds great but unfortunately I can't access it. Is there any way to post the link to the presentation?

The presentation will happen during the CLA Summit in Europe in April. I will try to remember to post about it after we attend.

[Thoric]

Yes, apologies, my wording was incorrect. As Fabiola says, the presentation is yet to happen, the link points to Peter Horn's proposal to present at the European CLA Summit. I've pointed Peter here, he will no doubt comment in good time

[PNHorn]

Hi, As has been previously noted I'm planning to present on Assertions at the CLA Summit in April. I should also be presenting at some of the UK based LabVIEW User Groups throughout the next few months (tky007 - are you based in the UK? if so I can let you know where and when I'll be presenting so you can arrange to be there). At some point it will make it on to ni.com together with some documentation - this won't happen until around May though.

In general I'd agree with most of what you have noted about assertions and when they should be used. The only one I disagree with is the error cluster being true - as asserts should be used to debug issues during development which should then be solved before the application is released to whoever the customer is. Assuming this has been done correctly the asserts can then be compiled out for deployment.

Error Handling should be in the main application to capture things such as the user selecting an invalid configuration file, or pressing Cancel on a dialog box. These should continue to be handled in the deployed application.

One other area I've been looking at is combining multiple asserts to give a complex algorithm based decision, such as:

(Variable1 = 10) OR (Variable2 > 20)

I hope that helps, more information should be available at some point soon, but feel free to ask any questions and I'll do my best to answer them.

[FabiolaDelaCueva]

Continuing the conversation we start at: https://decibel.ni.com/content/message/90359#90359

Where you said:

"

tyk007 wrote:

No API is perfect right?

When we do bulk floating-point comparisons (such as for unit tests) we do them via a For loop and an "Almost Equals" method such as in the addon and then pass the resulting array to our "Assert Pass If All True" test method. We added this assertion to our own kit when we noticed this functionality wasn't "out of the box" with VI Tester, although you could easy add your own wrapper for that.

How often do you find yourself rolling your own type of "assertions"? What sort of tests do they perform? Our testing tends not to vary much from the primitives and I'd like to hear more about other people;s experiences.

EDIT: Scratch that, I'll start a fresh topic.

"

I went back and looked at some of my unit tests and I found that the times that I had to do custom "equal" had always included a floating point. I had already given the example of arrays of floating points, in which case, you are correct a for loop and the "almost equals" method is used and then pass the resulting array to "assert pass if all true".

Other situations have been for example a cluster that includes a floating point, or an object that has a floating point. In those situations I have to create the class method to serialize/deserialize and then compare the individual items to ensure they are equal.

The other one that creates issues is comparing timestamps. Specially if you have a team that works in different , we had one occasion where the tests will all pass in one test station but would fail in another one. We finally realized that it was because the local time was different.  And this is another discussion, that is why using UTC is better for applications, but there will still be sections of your code where you have to present the local data to your customer and the unit tests for those times where local time is used have to be designed carefully.

[tyk007]

Hi Peter.

Unfortunately I am based in NZ so will be unable to be there in the UK for your presentation! I'd still like to hear about it afterwards though.

I think my .NET background has come through a lot of my wording. When I mean "assertion" I am thinking assertions that exist in test code only in to generate a test result, not code that exists in the raw production code-base in the form of run-time checks. In a lot of .NET test frameworks, the word "Assert" is over-loaded in this sense. From the point of view of external test code, it makes sense to me that I should test that an appropriate error is generated from a public API under an error condition as part of unit testing.

But I think I understand what you mean in your context and I completely agree. I'd definitely be interested in discussing this more. Error handling is often the most poorly devleoped sub-system (looking at my own code in particular) and more dicussions around the mangement of this are always welcome.

[crelf]

tyk007 wrote:

Unfortunately I am based in NZ

Oh come on - NZ isn't that bad!

[crossrulz]

Christopher Relf wrote:

tyk007 wrote:

Unfortunately I am based in NZ

Oh come on - NZ isn't that bad!

Yeah, it's not like you are in Michigan...