Format for displaying disabled test names

[Jonny Cooper]

From my experience, when a test gets temporarily disabled it all too often gets forgotten about and remains disabled almost indefinitely - the prompt of how many tests are disabled isn't enough to spur on the effort to `grep "DISABLED"` to find the disabled tests and consider why they are disabled and either fix or remove them - having the disabled test names appear in the output should just be enough to at least keep those tests in mind; hence issue 452.

The patch itself is pretty simple, what I can't decide is the best format of output. Assuming everyone agrees that the disabled tests shouldn't appear in the main body of the output where each test is run, rather they should only appear as a list at the end of the output in the summary, the question is how/where should the list of disabled tests appear?

For sake of clarity, the current summary at the end output, with some failing tests, disabled tests and passing tests, is formatted as (colour omitted):

[----------] Global test environment tear-down
[==========] 100 tests from 10 test cases ran. (496 ms total)
[  PASSED  ] 99 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] FailingTestCase.OneFailingTest
[  FAILED  ] FailingTestCase.AnotherFailingTest

 2 FAILED TESTS
  YOU HAVE 2 DISABLED TESTS

In my opinion, there are two options. Firstly, the list of disabled tests could remain completely separate from the rest of the list of tests, appearing below the line which warns of the disabled tests (using the same format as for passed/failed tests), i.e.

[----------] Global test environment tear-down
[==========] 100 tests from 10 test cases ran. (496 ms total)
[  PASSED  ] 99 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] FailingTestCase.OneFailingTest
[  FAILED  ] FailingTestCase.AnotherFailingTest

 2 FAILED TESTS
  YOU HAVE 2 DISABLED TESTS, listed below:
[ DISABLED ] DisabledTestCase.DISABLED_oneDisabledTest
[ DISABLED ] DisabledTestCase.DISABLED_anotherDisabledTest

Personally, this is my least preferred of the two suggested options. I think it distracts the users eye from the most important part of the report - the failing tests - forcing him/her to 'pick out' the lines containing the failing test namess as opposed to being naturally drawn to them at the bottom of the output.

The second option would be to place the names of the disabled tests in amongst the passed/failed list at the bottom of the report, i.e.

[----------] Global test environment tear-down
[==========] 100 tests from 10 test cases ran. (496 ms total)
[  PASSED  ] 99 tests.
[ DISABLED ] 1 test, listed below:
[ DISABLED ] DisabledTestCase.DISABLED_oneDisabledTest
[ DISABLED ] DisabledTestCase.DISABLED_anotherDisabledTest
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] FailingTestCase.OneFailingTest
[  FAILED  ] FailingTestCase.AnotherFailingTest

 2 FAILED TESTS
  YOU HAVE 2 DISABLED TESTS

In a similar vain to my comments on the previous options, I feel the list of disabled tests should go in between the number of passed tests and list of failing tests as it means the users eye is still naturally drawn to the list of failing tests at the bottom of the list and any users interested in disabled tests can 'pick out' their names from within the list. Furthermore, if there are no failing tests but there are disabled tests, the user will naturally be drawn to the list of the disabled tests at the bottom of the list forcing them to at least consider the disabled tests a bit more than they do currently.

Side note: One could also argue that the number of failing tests should go at the very bottom of the output, and maybe the phrase "you have 2 disabled tests" should be changed to "2 disabled tests" for consistency, something like:
 2 DISABLED TESTS
 2 FAILED TESTS
Although I don't want to rock the boat too much...

All thoughts, comments and suggestions welcome.

[Matthew Gidden]

Hi Johnny,

Long time user of google test, first time poster. I really like this patch idea.

Agreed. Furthermore, this fits the color scheme (pass - green, disabled - (proposed) yellow, fail - red).   

Side note: One could also argue that the number of failing tests should go at the very bottom of the output, and maybe the phrase "you have 2 disabled tests" should be changed to "2 disabled tests" for consistency, something like:
 2 DISABLED TESTS
 2 FAILED TESTS
Although I don't want to rock the boat too much...

All thoughts, comments and suggestions welcome.

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matthew Gidden
Ph.D. Candidate, Nuclear Engineering

The University of Wisconsin -- Madison
Ph. 225.892.3192

[Jonny Cooper]

Hi Matthew,

Good to know someone else actually wants this feature!

On Thursday, 7 August 2014 16:08:43 UTC+1, Matthew Gidden wrote:

Hi Johnny,

Long time user of google test, first time poster. I really like this patch idea.

Agreed. Furthermore, this fits the color scheme (pass - green, disabled - (proposed) yellow, fail - red).   

 
With regards to colours, as you rightly point out (and I forgot to mention!) the '[ DISABLED ]' marker should be yellow to fit in with the colour scheme - the same colour as the 'YOU HAVE 2 DISABLED TESTS' warning.

Jonny

[Alex Valavanis]

Sounds good to me too... the second option seems like a better idea.

AV

[BarbaraJoy Jones]

I would find this pretty useful, too. We frequently have tests that are disabled for weeks or months due to long-standing bugs.

It might be nice to have a command-line switch to disable it, though. I think that disabling a value-parameterized test would make the list get very long (depending on how that test is instantiated).

--

[Jonny Cooper]

Agreed - a command line option would be useful. I think it should default to listing the disabled tests since the whole point is to nag you into doing something about them. Then you can forcibly stop gtest from nagging/listing the tests if you wish.

Any thoughts on a name for the option? The best I could come up with was a boolean flag called '--gtest_dont_list_disabled_tests'.

Jonny

To unsubscribe from this group and stop receiving emails from it, send an email to googletestframework+unsub...@googlegroups.com.

[Ken Turkowski]

I like this idea too.