Proposal - Ignoring Individual Results in LabVIEW 2016

[Darren]

I have preliminary approval to add some infrastructure to the VI Analyzer in LabVIEW 2016 to allow us to add 'ignore individual result' functionality to VI Analyzer Tests. I'm excited because this is one of the longest-running feature requests for the VI Analyzer. It has been discussed at length here, and that discussion provided the inspiration for the low-risk solution I'm proposing for LabVIEW 2016.

Specifically, I'm planning on adding the 'Check for Ignored Objects.vi' attached below to the LabVIEW distribution. This VI should be relatively easy to drop in to an existing VI Analyzer test to add 'ignore individual results' functionality to the test. I hope to be able to add this to all the VI Analyzer Toolkit test for 2016, and I also hope to add 2016 versions of all the community tests as well that have this functionality.

Here is the description of the VI attached below, just to give everybody an idea about what I'm envisioning:

Once a test calls this VI and uses it appropriately, all you need to do is modify the VI being tested in one of the following ways:

1. Add the text "#VIA_Ignore" to the VI Description of the VI, along with the name of the test(s) you want to ignore on that VI.

2. Add the text "#VIA_Ignore" to a front panel free label on the VI, along with the name of the test(s) you want to ignore on that VI.

3. Add the text "#VIA_Ignore" to a block diagram free label on the VI, along with the name of the test(s) you want to ignore on that VI. If the free label is not attached to anything, then the test(s) will be ignored on that entire VI. If the free label is attached to any object, then that object will be ignored by any VI Analyzer test name(s) found in that free label.

Let me know what y'all think about the utility VI attached below (saved in LabVIEW 2015), and let me know if y'all have any feedback on the plan. There is certainly a lot more other functionality I can envision adding, but I don't know how much additional 2016 time I'll have to work on this, and this seemed like the base level of functionality we would need.

[FabiolaDelaCueva]

Darren,

This is amazing, standing ovation, cannot wait to try it out.

Thanks,

Fab

[Hooovahh]

Neat implementation Darren, I think this covers all the uses I would have for this.  And with the right click framework in 2015+ someone could make a plugin where when you right click an object it can have a VI Analyzer section, and Ignore, then showing all the tests that can be found.  There could even be an option to ignore that test on the VI, or on the object and insert the text on the VI, or make a floating comment that gets attached to the object.  I'm just trying to again make is simple when reviewing code to basically acknowledge a failed test of the VIA.

[warren_scott]

Comments/Thoughts:

Overall, I like it.  Please very much consider suggestions below as constructive criticism from developing and using a similar implementation for the last 2 years.

1) nitpick item: I would probably rename the output "Ignore Entire VI" to "Ignore Entire File" or something else to not ignore that we are also checking custom controls.  Someday hopefully lvlibs and projects and other items too.

2) I would be inclined to also have a "Ignored Panel Items" output (array of references) for items on front panel that are to be ignored.  I would base this on putting "#VIA_Ignore Overlapping Controls" in the description of that item.

3) There is a potential issue where you have a long comment and #VIA_Ignore somewhere in it.  With your current implementation it allows the "Error Style" (or whatever the test name is) to be anywhere in the comment.  I would consider making it a requirement that "Error Style" comes AFTER the #VIA_Ignore tag (which costs no more processing as it would just be that the found index of the name is larger than the found index of the tag, instead of just both being >0).  This would help reduce false positive hits if you put your VIA Ignores mixed in other documentation for other information that is unlucky enough to have the same text as the name of a VIAnalyzer test.  I don't see this as much of an issue for BD comments (as I would expect/encourage dedicated text comments for the VI Analyzer ignores), but would be more of a concern for the description of the VI as a whole (in which hopefully people are writing long descriptive documentation about the VI ... to which they may append a short list of VI Analyzer ignores).

4) There is a potential area of conflict/confusion if one test name is the subset/superset of another.  Example: One test called "Test For Bad Stuff", and another called "Test For Bad Stuff Detailed".  (Not saying this is a good idea, but we know not everyone always has good ideas, or even if they have good ideas whether they implement them).  With the current implementation you can never add an exception for "Test For Bad Stuff" without also inadvertently adding an exception for "Test For Bad Stuff Detailed".  I don't know whether this is worth trying to actually solve, or just deal with it by adding documentation somewhere that it won't work out right if you name your tests like that.

5) One area that I would like to make easier to deal with is items that are really paired together and you may add an ignore to one but you really want the ignore applied to all of them (without adding too many additional text boxes).  Things like Inner and Outer Tunnels -- generally if I get an ignore on one I want it to apply to both.  And likewise with shift registers -- If I put an ignore on the left shift register I would like it to apply to the right one as well (or more technically correct I would like to assign an ignore comment arbitrarily to one of the sides of the shift register and have it apply to the whole set of them).  There's some cases that you would want to be very specific for just one of them, and if your VIAnalyzer test really cared it could put smarts in the test itself to check for left and right shift registers both..... It would be cool if there was a input boolean "include item pairs" or something like that where if the boolean were true and an ignore was found for a left shift register it would also add the paired right shift register to the ignored items list.

6) Another thing that I haven't totally thought through (so take this with a few more grains of salt....).  FrontPanel tests sometimes become annoying to deal with.  Take "Overlapping Controls" for example.  If I create a custom typedef that has overlapping controls in it, any VI that uses that control on the FP automatically fails the "Overlapping Controls" test.  In reality, I probably don't want to fail every VI that uses that custom control (especially if it is a strict typedef) -- only fail on the test of the ctl itself.  To some extent this is a problem with how the "Overlapping Controls" test is written (that it should understand nested controls and give an option to ignore overlaps within typedefs unless you are specifically testing that typedef).  That being said, I would argue it would be more powerful to allow the check for ignored objects to possibly understand to recurse into other files/objects (such as the .ctl for the strict typedef in question) and see if it has any appropriate ignores in it and then take those into consideration when creating the ignored objects list / performing the tests.  Figuring out exactly the rules for when/how to recurse deeper could get tricky fast.  Something to think about for the future, but I would NOT hold up getting this implemented as a whole because of this specific concern/idea.

That's my 1 1/2 cents worth.  Overall I like it and I think it will work out well.  Thank you for your continued dedication to VI Analyzer!!!

[Darren]

Darren wrote:

I hope to be able to add this to all the VI Analyzer Toolkit tests for 2016...

Bad news, I was unable to include the 'ignore this failure' functionality in the VI Analyzer Toolkit tests for 2016. The primary blocker was that the utility VI that facilitates this functionality, in its current form, causes the VI Analyzer Toolkit test suite to run about 30% slower, on average. I haven't had time to optimize the utility VI in time for the 2016 release. The VI itself will still ship with LabVIEW 2016 (in vi.lib\addons\analyzer\_analyzerutils.llb) so we can add it to community tests individually as needed.

Hopefully I will have time after the 2016 release to optimize the performance of the utility VI, or perhaps investigate a different approach, like making changes to the VI Analyzer Engine itself, to provide this functionality.

[Hooovahh]

So I started using this useful feature on a few tests to try to reduce the number of failures found, and I found what an issue with spell check failures.

So I have several comments on the BD that explain what the VI is doing, but it copies and pastes some of the header text file from a C file which has plenty of abbreviations and things that the Spell Check test finds and causes to fail.  So I made a free label:

"#VIA_Ignore NI_VIAnalyzerTKTests.lvlib:Spell Check"

And have it attached to the comment with the known spelling mistakes.  I then modified the spell check test to use the VI Darren attached and it filters objects to not include them in the test.

When I run a VIA test it correctly ignores the spelling mistake, but finds a new spelling mistake

This free label contains the misspelled word "VIAnalyzerTKTests".

In this case I think we could modify the VI attached to not include the library name so the comment dropped won't have a spelling mistake.  Ignoring the whole VI works but I'd like to find other spelling mistakes so I wanted to have the VIA ignore just the one comment.  Another option might be to include the #VIA_Ignore label in the list of Ignored Diagram Objects so the comment used to ignore tests, doesn't create new failures.

Oh and another improvement for performance.  If this VI is intended on being in each test, and scanning each VI, wouldn't it make more sense to cache the scan results for each VI somehow?  I mean the way it is implemented right now I have a VI which returns an array of objects to ignore for a single test.  Wouldn't it make more sense to scan the VI and return the list of objects to ignore for all tests, and then each test can just read this information instead of having to scan the same VI for every test.

[Hooovahh]

Also I noticed there is an invoke node method on a VI class type to return all the Bookmarks in a VI.  It might be faster to read this on a VI and then if the #VIA_Ignore is not found, then we know we don't need to scan the block diagram comments for tests to ignore.

[patric.j]

For those of you reading this post, this feature has been implemented in LabVIEW 2018. See this post: New VI Analyzer Features in LabVIEW 2018

Patric