Best Practice for testing a VI?

[MikeeB]

I am testing a very simple VI for password validation to make sure that it complies with a standard for a variety of different cases.

e.g.

• No three characters the same in a row i.e. AAA or 111
  
• Must contain at least one Uppercase character.
  

There are more tests but this gives you the flavour.

I want the test results to be readable and meaningful, so I used the comments to describe what each test was doing and had a series of test cases with the same comment but different values. When the report(either HTML or XML) is produced, the comments are missing, so I have no reference to what the test is.

My next step was to produce separate lvtest files for each test and have one report, this works but if I look at any one test I don't have 100% test coverage, that has to go with project test coverage. To create each of the tests was a pain as I cannot rename a test, they are all "untitled test n.lvtest" and going out to Windows Explorer to rename them and then removing them from the project and adding them does not seem optimal.

What am I missing?

Thanks

Mike

P.S. I no longer work for NI, so please ignore the symbol that says I do.

Here is a zip of a sample project. I have included comments in the VI showing the three rules I am working towards and an example lvtest file in the project. It will show the HTML report after it has been created and you have run the lvtest. Th actual project gets more complex, but this will give an idea.

Message was edited by: MikeeB

[crelf]

Without seeing your tests and code (well, inputs & outputs), it's a little difficult to visualize the best way to support this (can you upload them?), but here's an idea: have you thought about putting combinations in test vectors? I don't think you really need separate tests to check the combinations, maybe just vectors.

[MikeeB]

I have attached a sample project to the original post with 3 rules that need to be tested.

Mike

[crelf]

So It's not obvious, but rather than relying on the comment (that's really just for developers I guess, a more apt field to go into a report would be labelled "Description"), you can rename the Test Case itself in the drop down (it looks like it's a numeric dropdown, but it's actually a string with editable fields). I've editied your lvtest to demonstrate, and attached an example report that shows the test cases' text shown.

[MikeeB]

Thanks for the suggestion. I am going use it/something based heavily on it as I go in to the actual password rules that I need to test. It's a shame the reporting options are slightly more full featured. My password rules are going to need about 1000 test cases so the report becomes very important for validation. Thanks Again  Mike

[crelf]

Well, in that case, you don't want to have 1000 test cases: you need one test case for each *thing* you're looking at testing, and test vectors to define the permutations in each thing. See attached (you can do this several different ways - the attaced example is very simple, but you can really ramp up into the vector world and get very sophisticated solutions).

Also, remember that both the test and vector files are ASCII, so you can read/write them with other applications (eg: if you've got a decent requirements traceability suite, you might actually be able to export to something that is easily manipulated into a lvtest and/or lvvect), or you can just open them up in Excel if you need a way to show others the test and vector definitions.

[FabiolaDelaCueva]

Christopher Relf wrote:

Well, in that case, you don't want to have 1000 test cases: you need one test case for each *thing* you're looking at testing, and test vectors to define the permutations in each thing. See attached (you can do this several different ways - the attaced example is very simple, but you can really ramp up into the vector world and get very sophisticated solutions).

Agreed with Chris Relf that the best approach here is a vector instead of defining one test at a time.

Another suggestion to ensure your comments show for each combination of inputs is to add another test vector with the "comments" you want to show for each case and have that be one of your inputs in your Unit under Test VI. That way when the report gets printed you will have the comment next to each test. It is not a very clean way to do it, because it affects the composition of your Unit under test and now you have to worry about indexes in your test case definition. But it might work for you.

I am attaching the modified project from what Chris had posted earlier showing a vector with made up comments.

[crelf]

FabiolaDelaCueva wrote:

Another suggestion to ensure your comments show for each combination of inputs is to add another test vector with the "comments" you want to show for each case and have that be one of your inputs in your Unit under Test VI. That way when the report gets printed you will have the comment next to each test. It is not a very clean way to do it, because it affects the composition of your Unit under test and now you have to worry about indexes in your test case definition. But it might work for you.

That's a neat trick that I hadn't considered before - although I'm struggling a little to get my head around the implementation: can you tlak through what you're doing with vector indexes there?

[FabiolaDelaCueva]

Christopher Relf wrote:

That's a neat trick that I hadn't considered before - although I'm struggling a little to get my head around the implementation: can you tlak through what you're doing with vector indexes there?

If no indexes are defined, then the Unit Test Framework tries to pair up all of your vectors.

So if you have two inputs and one output and you set 2 different values for the first input and 3 different values for the second output, then UTF will end up needing 2*3=6 values for the output, because it will test every combination possible of both input test vectors. That is why we end up with a longer output vector, in this case it would have 6 values.

The comments field that I added is an extra input. If I don't assign an index, then UTF will try to do every possible combination with the two inputs already there and the extra input I added. I don't want that, I want to ensure that one comment aligns with every value of my input, I give them the same index and UTF knows not to try to do every combination possible.

Of course now that I explained this, I realized that we would probably need to do something else, because I paired up UserName and Comments, so I should probably have the same number of elements for the comments as I do for UserName and we go back to square 1.

So, I am attaching a new test that instead of relying on the UTF to do all possible combinations, it expects for us to give a one to one matching. In order to indicate to UTF that I don't want every possible combination, I assign to all my input vectors the same index. I named this new test case 1a and I put in meaningful comments.

Let me know if this makes sense and sorry for the unnecessary confusion. I should have spent more time in my first example

Regards,

Fab

[MikeeB]

Thanks for the thought Fab. I am going to stick with Crelf's idea. I am creating a small VI to create each of the vectors that I require for the test. The first simple test in the actual test is "Password length must be between 8 &10 characters long(inclusive)" This is a short simple VI to create this one. One of the tests is that the password must contain one of ~20 special characters. I can then create a VI which takes each one in turn and puts it into each position in a 9 character password and outputs all of them in an array. This is a 200 element vector, but is created with a simple VI. If I need to recreate/alter the vector it is only one VI to change. Because of all the options and other checks the VI does I am struggling for space on the connector pane(legacy code, before anyone starts)and having the extra input that does nothing other than pass a name through doesn't look good on the test report. Vectors are definitely the way forward, just making the report readable keeps the challenge going 🙂  Mike