Best practices for TODO test management?

[Mark Stosberg]

We have a large test suite with 20,000+ tests in it, and a number of
passing TODO tests in it.

Our problem is that once a test is marked as TODO, it falls off our
radar, and we are unlikely to notice it again.

What are some suggestions to make sure that passing TODO tests get
regular attention?

I was hoping that 'prove' might have a flag to treat passing-TODOs as
failures, but Id didn't see one.

I would prefer not to write a custom harness... this seems like a
feature that would be of general interest.

In case this context matters-- our tests are regularly through Jenkins,
so our TAP is getting converted to JUnit. We also run tests from the
command line with prove regularly as well.

Thanks!

Mark

[Pedro Melo]

Hi,

On Wed, Feb 27, 2013 at 2:06 PM, Mark Stosberg <ma...@summersault.com> wrote:

>
> We have a large test suite with 20,000+ tests in it, and a number of
> passing TODO tests in it.
>
> Our problem is that once a test is marked as TODO, it falls off our
> radar, and we are unlikely to notice it again.
>
> What are some suggestions to make sure that passing TODO tests get
> regular attention?
>

Make a TODO that passes all subtests a failure? Not sure how to go about
it, but it sure would get your attention :)…

Bye,
--
Pedro Melo
@pedromelo
http://www.simplicidade.org/
http://about.me/melo
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org

[Ovid]

Hi Mark,

I know this isn't exactly what you want, but a long time ago I added a --directives switch to 'prove' to help with this. If you do this:

    prove -lr --directives t/

'prove' will *not* run in verbose mode, except for any tests with directives (TODO and SKIP).
 
It's not perfect, but at least it will make the TODO tests stand out (they're also colored differently if you're using colors).

Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/

>________________________________
> From: Mark Stosberg <ma...@summersault.com>
>To: perl-qa <per...@perl.org>
>Sent: Wednesday, 27 February 2013, 15:06
>Subject: Best practices for TODO test management?

[Graham TerMarsch]

On February 27, 2013 09:06:23 AM Mark Stosberg wrote:
> What are some suggestions to make sure that passing TODO tests get
> regular attention?
>

[.....snip.....]

>
> In case this context matters-- our tests are regularly through Jenkins,
> so our TAP is getting converted to JUnit. We also run tests from the
> command line with prove regularly as well.

Mark, what are you using to convert your TAP to JUnit?

I ask as TAP::Formatter::JUnit automatically treats "passing TODOs" as
"failures" and reports them as such in the JUnit output.

(disclaimer: I'm the author of TAP::Formatter::JUnit)

--
Graham TerMarsch

[Steffen Schwigon]

Mark Stosberg <ma...@summersault.com> writes:
> We have a large test suite with 20,000+ tests in it, and a number of
> passing TODO tests in it.
>
> Our problem is that once a test is marked as TODO, it falls off our
> radar, and we are unlikely to notice it again.

Maybe this helps:

http://www.nntp.perl.org/group/perl.qa/2009/07/msg12376.html

It's applying my personal hammer "DPath" to my personal nail "TAP::DOM".

Kind regards,
Steffen
--
Steffen Schwigon <s...@renormalist.net>
Perl benchmarks <http://perlformance.net>
Dresden Perl Mongers <http://dresden-pm.org/>

[Mark Stosberg]

On 02/27/2013 09:44 AM, Ovid wrote:
> Hi Mark,
>
> I know this isn't exactly what you want, but a long time ago I added a
> --directives switch to 'prove' to help with this. If you do this:
>
> prove -lr --directives t/
>
> 'prove' will *not* run in verbose mode, except for any tests with
> directives (TODO and SKIP).
>
> It's not perfect, but at least it will make the TODO tests stand out
> (they're also colored differently if you're using colors).

Thanks for that tip, Ovid!

I had overlooked this feature. I did find it helpful, but found it it
doesn't work in combination with the "-j" feature to run tests in
parallel. I'm not sure if that's a bug, or an inherent limitation.

Still, it's nice to know about this option, even if I have to have a
slower test run now-and-then to use it.

Mark

[Mark Stosberg]

On 02/27/2013 02:48 PM, Graham TerMarsch wrote:
> On February 27, 2013 09:06:23 AM Mark Stosberg wrote:
>> What are some suggestions to make sure that passing TODO tests get
>> regular attention?
>>
> [.....snip.....]
>>
>> In case this context matters-- our tests are regularly through Jenkins,
>> so our TAP is getting converted to JUnit. We also run tests from the
>> command line with prove regularly as well.
>
> Mark, what are you using to convert your TAP to JUnit?
>
> I ask as TAP::Formatter::JUnit automatically treats "passing TODOs" as
> "failures" and reports them as such in the JUnit output.

I use "TAP::Harness::JUnit", only because I got it working, and it works
well enough.

Thanks for highlighting this feature of TAP::Formatter::JUnit. I'll have
to give it another look.

When I looked at one one of my passing TODO tests, I found that it
marked as a TODO, because sometimes the test data it was working with
would be in the desired state, and sometimes it wouldn't be. So, it
wasn't a simply matter of of un-TODO'ing the test in that case, as it
would still fail sometimes.

In a case like that, I have to re-work the test so that it has
sufficient control over the test data to give consistent results.
Perhaps that test would be better written to "SKIP" if the data isn't
working with isn't in a state for the test to be meaningful.

Mark