Feedback on the undocumented --rules option for 'prove'

[Mark Stosberg]

In 2008 Alex Vandiver contributed a patch to "prove" that allowed you to
specify that you wanted some tests to run in parallel and others in
serial. This is a great feature for those of us with large test suites
that would like to test advantage of parallism, but have suites that
aren't 100% parallel-ready. I'm grateful that work was done.

The feature was considered experimental, and was not documented in
`prove`. From what I can tell it has remained largely undiscovered, and
is still remains in the same state about 4 years later.

In the last few days I started working on the same problem myself, not
realizing that the undocumented feature exist.

I asked on StackOverflow [1], and proceeded to code up my own solution
[2] before I ended up here to report my results and ask for help.

1.
http://stackoverflow.com/questions/11977015/how-to-run-some-but-not-all-tests-in-a-perl-test-suite-in-parallel/11977495#11977495
2. https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/3

After I fixed in a bug in calculating the tests to run in my own
solution, it appeared to work, but then at the end of the run, no
summary would be reported. It would just stop and it wasn't clear what
the problem was. So, I tried to see if I could get the "--rules" option
to work for me.

First I tried putting this syntax in my my .proverc:

--rules seq=t/first_serial_test.t
--rules seq=t/second_serial_test.t

It appeared that it was starting to run my serial tests first... but it
appeared to keep going with running everything in serial, despite "-j 4"
on the command line.

So, I tried this:

--rules seq=t/first_serial_test.t
--rules seq=t/second_serial_test.t
--rules par=**

This appeared to have the opposite result. No *everything* appeared to
be run in parallel.

Finally, I tried this variation:

--rules par=**
--rules seq=t/first_serial_test.t
--rules seq=t/second_serial_test.t

This appeared that it was doing the right thing.... until it failed the
same way own patch did.... the test suite run just ended, with no
summary.

Perhaps is this simply a documentation issue, and I haven't gotten the
syntax just right. How do I specify that there are some tests that I
always want to be run in serial?

My goal is to be able to specify this list of exceptions once in a file
and forget about it. I don't want the existence of a rule to imply that
I want to that test. I only want rules applied if tests are actually
selected to run. So far, I'm not sure if specify a rule also implies
that I'm selecting a test to run, which I wouldn't want.

Thanks for considering this feature again with me. Let's get it
finished, documented and published!

*UPDATE*

After I drafted this message, I found Test::Steering, which also
advertises the feature of mixing parallel and serial runs.

However, I found it didn't work. First, I had to patch it just to get
basic functionality going [3], and then when I got a real test to run,
it produced two "summary" reports... the one for the parallel runs was
printed in the middle of the run, while a second summary just for the
serial runs appeared at the end.

https://rt.cpan.org/Ticket/Display.html?id=62681
https://metacpan.org/module/Test::Steering

Perhaps everyone in this problem space using the "Roll your own"
approach, but it sure seems like there's potential for a generally
useful, re-usable tool for this.

Thanks!

Mark

[James E Keenan]

On 8/17/12 3:53 PM, Mark Stosberg wrote:
>
> In 2008 Alex Vandiver contributed a patch to "prove" that allowed you to
> specify that you wanted some tests to run in parallel and others in
> serial. This is a great feature for those of us with large test suites
> that would like to test advantage of parallism, but have suites that
> aren't 100% parallel-ready. I'm grateful that work was done.
>
> The feature was considered experimental, and was not documented in
> `prove`. From what I can tell it has remained largely undiscovered, and
> is still remains in the same state about 4 years later.
>

And it's not alone in being undocumented. Go to
http://search.cpan.org/~ovid/Test-Harness-3.25/lib/App/Prove.pm#Attributes
and note that 'rules' is one of over 40 attributes which are described
as existing but are otherwise undocumented.

However, I see that 'rules' is the subject of testing in t/scheduler.t.
Do the individual tests in that file give you any clue as to how to
proceed?

Thank you very much.
Jim Keenan

[James E Keenan]

On 8/19/12 1:07 PM, James E Keenan wrote:

>
> And it's not alone in being undocumented. Go to
> http://search.cpan.org/~ovid/Test-Harness-3.25/lib/App/Prove.pm#Attributes
> and note that 'rules' is one of over 40 attributes which are described
> as existing but are otherwise undocumented.
>

Slightly OT: Not only are parts of App::Prove lacking in documentation;
they also lack test coverage, particularly with respect to other parts
of the Test-Harness distribution. See:

http://thenceforward.net/Test-Harness/coverage/coverage.html

[Mark Stosberg]

> However, I see that 'rules' is the subject of testing in t/scheduler.t.
> Do the individual tests in that file give you any clue as to how to
> proceed?

Thanks for the feedback, Jim.

I believe I found what I needed over the weekend.

First, I made a list of what needed to be done here as some raw notes:
https://github.com/markstos/Test-Harness/wiki/Missing-documentation-for-'rules'-and-'scheduler'-in-Test::Harness---App::Prove

This morning I've submitted a "pull request" with proposed docs here.
There is more feedback in this Github pull request:

https://github.com/Perl-Toolchain-Gang/Test-Harness/pull/5

I would be interested in a peer-review of my work.

Thanks!

Mark

[Mark Stosberg]

When testing this on my large test suite, I believe I've found bug.
Here's my understanding of what appears to be happening:

A correct "schedule" is being created, with most tests set to run in a
parallel. The few exceptions I've sent are clearly being put in sequence
at the end.

Yet, a number of the exceptions still fail in these runs, but pass if
run by themselves.

Using a well-timed capture of activity with "ps", I was able to confirm
that my "exceptions" are running at the same time as some of the other
parallel tests.

I suspect there's a bug that works as follows, but I haven't isolated it
yet. Here's my suspected trigger:

- First start some tests that are parallel ready, but some of them are slow.
- Next in the schedule have some tests which much be run in sequence.

I think the "sequence" tests are in fact being run in sequence, but some
of the "slow" parallel tests are still running.

I'll try to mock-up this situation and see what I find.

Mark

[Eric Wilhelm]

# from Mark Stosberg on Thursday 30 August 2012:

>I suspect there's a bug that works as follows, but I haven't isolated
>it yet. Here's my suspected trigger:
>
>- First start some tests that are parallel ready, but some of them are
>slow. - Next in the schedule have some tests which much be run in
>sequence.
>
>I think the "sequence" tests are in fact being run in sequence, but
>some of the "slow" parallel tests are still running.

Is that a bug? I thought sequenced tests were only excluded from
running in parallel with each other -- and that parallel tests were
compatible running alongside all others.

--Eric
--
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------

[Mark Stosberg]

On 08/30/2012 06:56 PM, Eric Wilhelm wrote:
> # from Mark Stosberg on Thursday 30 August 2012:
>> I suspect there's a bug that works as follows, but I haven't isolated
>> it yet. Here's my suspected trigger:
>>
>> - First start some tests that are parallel ready, but some of them are
>> slow. - Next in the schedule have some tests which much be run in
>> sequence.
>>
>> I think the "sequence" tests are in fact being run in sequence, but
>> some of the "slow" parallel tests are still running.
>
> Is that a bug? I thought sequenced tests were only excluded from
> running in parallel with each other -- and that parallel tests were
> compatible running alongside all others.

Eric,

Thanks for the follow-up. I think you are right... I think that answer
means there is no way to use the "--rules" flags to 'prove' to
accomplish what I want, as it is written

I looked into the App/Prove.pm source code and immediately spotted the
issue. It hardcodes that all "rules" should run in parallel. Thus, there
would be no way to specify that something should never be run in
parallel with anything else.

I patched prove using the below patch, and my tests ran almost as I
wanted. Using the "--rules" I specified, the exceptions were all run
first, one at a time, and then the rest of the tests were run in parallel.

However, this triggered a related issue that I had seen sometimes before
in my testing-- No summary output is produced! I just see an "ok" result
line printed for the last test, and that's it. I'm not sure what's going
on there.

What could use TAP::Harness to fail to produce the summary report?

Mark


# Alter the logic of the --rules processing for prove, so that each rule
is considered in sequence.
# This is what you want if you want to specify some exceptions with 'seq'
# that can't run in parallel.
--- old-trunk/perllib/App/Prove.pm 2012-08-31 15:01:57.000000000 -0400
+++ new-trunk/perllib/App/Prove.pm 2012-08-31 15:01:58.000000000 -0400
@@ -373,13 +373,13 @@
my @rules;
for ( @{ $self->rules } ) {
if (/^par=(.*)/) {
- push @rules, $1;
+ push @rules, { par => $1};
}
elsif (/^seq=(.*)/) {
push @rules, { seq => $1 };
}
}
- $args{rules} = { par => [@rules] };
+ $args{rules} = { seq => [@rules] };
}

return ( \%args, $self->{harness_class} );

[Eric Wilhelm]

# from Mark Stosberg on Friday 31 August 2012:

>I looked into the App/Prove.pm source code and immediately spotted the
>issue. It hardcodes that all "rules" should run in parallel. Thus,
>there would be no way to specify that something should never be run
>in parallel with anything else.

Hi Mark,

See also: 'rules' in the TAP::Harness pod, the source/comment on
TAP::Parser::Scheduler (_set_rules() &c.), and t/scheduler.t.

There is a disconnect between command-line flags to prove and the data
structure there. I haven't had time to totally grok the scheduler code,
but I think you need something different than what the --rules option
was written to do (though we would have to ask someone who is using it
whether your patch breaks that use case -- I think the usage was to
prevent just a few tests from running at the same time as each other,
not from all others.)

In your case, it might be better to be able to just pass a schedule of
nested arrays (maybe as json or something?)

[Mark Stosberg]

On 09/03/2012 01:52 PM, Eric Wilhelm wrote:
> # from Mark Stosberg on Friday 31 August 2012:
>> I looked into the App/Prove.pm source code and immediately spotted the
>> issue. It hardcodes that all "rules" should run in parallel. Thus,
>> there would be no way to specify that something should never be run
>> in parallel with anything else.
>
> Hi Mark,
>
> See also: 'rules' in the TAP::Harness pod, the source/comment on
> TAP::Parser::Scheduler (_set_rules() &c.), and t/scheduler.t.

Thanks for the reply, Eric. I did look in some detail at the related
scheduling code as part of preparing the documentation patch I submitted.

> There is a disconnect between command-line flags to prove and the data
> structure there. I haven't had time to totally grok the scheduler code,
> but I think you need something different than what the --rules option
> was written to do (though we would have to ask someone who is using it
> whether your patch breaks that use case -- I think the usage was to
> prevent just a few tests from running at the same time as each other,
> not from all others.)
>
> In your case, it might be better to be able to just pass a schedule of
> nested arrays (maybe as json or something?)

At this point, I have confirmed that the right schedule is being
generated and used. However, it is triggering a bug elsewhere, perhaps
in the aggregation or multiplexing code. At the end of the run, there
is simply no summary report generated.

However, I may not pursue this further soon at this point because of a
course change.

It's been so slow to get proper support for exceptions to parallel test
runs, that I decided to dig into the problematic tests to see if I could
make them parallel friendly. It turns out that was faster just to fix
the tests. :)

Mark