The "new" partest

[Eugene Burmako]

So I've been sitting in this cave for the last few months, and now I'm
finally catching up with master and rebasing paradise off M3.

I've got two things that I want to discuss about the rehashed partest.
I could code up both and submit a pull request, but before doing that
I'd like to figure out stuff that I might be missing.

1) The colors are not very friendly to black-on-white terminals. Is
there any pre-existing way to deal with that?

2) Unlike with the old partest, with the new one it's much harder to
figure out the list of tests that have failed. Compare [1] (an old
log) and [2] (a new log). The former clearly lists all the tests
failed in an easy for eyes format, the latter requires machine
processing. Is there any flag that makes this sort of information
better consumable by a human?

Don't get me wrong - I like the fact that we now have inline diffs
printed out, so that they don't break the flow of the testing when
it's going on. It's just that I'd want to have a way to quickly figure
out the list of failed tests.

[1] https://scala-webapps.epfl.ch/jenkins/job/scala-checkin-manual/870/console
[2] https://scala-webapps.epfl.ch/jenkins/job/scala-checkin-manual/878/console

[Jason Zaugg]

On Sun, May 19, 2013 at 9:48 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

So I've been sitting in this cave for the last few months, and now I'm
finally catching up with master and rebasing paradise off M3.

I've got two things that I want to discuss about the rehashed partest.
I could code up both and submit a pull request, but before doing that
I'd like to figure out stuff that I might be missing.

1) The colors are not very friendly to black-on-white terminals. Is
there any pre-existing way to deal with that?

The color-terminal detection code is in the partest runner:

term_colors=$(tput colors 2>/dev/null)

if [[ $? == 0 ]] && [[ $term_colors -gt 2 ]]; then

  git_diff_options="--color=always --word-diff"

  color_opts="-Dpartest.colors=$term_colors"

else

  unset color_opts

  git_diff_options="--nocolor"

fi

Perhaps you can add a --nocolor command line option.

2) Unlike with the old partest, with the new one it's much harder to
figure out the list of tests that have failed. Compare [1] (an old
log) and [2] (a new log). The former clearly lists all the tests
failed in an easy for eyes format, the latter requires machine
processing. Is there any flag that makes this sort of information
better consumable by a human?

`./tools/partest --failed` probably needs a little publicity.

It appears to run any test that has left a log file behind:

  def failedTests: List[Path]            = standardTests filter (p => logOf(p).isFile)

-jason

[Eugene Burmako]

2) Well, I have my own test infrastructure built atop of partest (it stores test suites in dedicated files that can be auto-populated while partest is running, can be backed up, can be created from jenkins reports, etc), and partest --failed covers a bit less cases than I'm accustomed to have covered.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paul Phillips]

On Sun, May 19, 2013 at 12:48 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

2) Unlike with the old partest, with the new one it's much harder to
figure out the list of tests that have failed.

It's horrible. It makes me cry to think that anyone is out there imagining this code approximates my aesthetics. It's just that the goddamn tests were passing and given that I have rewritten partest half a dozen times I was going to take any chance to close the deal. Unfortunately it was not in a real polished state when it comes to appearance. (At least the functional parts appear to be holding up.)

[Som Snytt]

It's true, I think Paul was actually sobbing.

He notes somewhere that he wants to do another clean-up, but that's kind of suboptimal use of time.  Do you want Paulp-Test or will Par-Test suffice?  Keeping in mind that one paulp-hour is 1.5 ordinary man-days.

I've got a couple of Par-Test PRs, and I'm willing to tweak more if someone creates an issue.

I didn't think I was up for it after almost burning out on the Testicolor merge; what got me through that was looking at paulp code.  For every brain cell that died, a new one sprouted forth on a different lobe.

I just now needed it to handle crashes with nuance, since sometimes the compiler throws FatalError when it's not so much crashing as falling on its sword.

Also, +1 to --failed.  A couple of weeks ago, I made a change to put stderr in the logs and had to update a gazillion tests, finding a few minor issues.  --failed worked charmingly.  I also benefited from Jason's feature to call out `updated' status.

[Jason Zaugg]

On Sun, May 19, 2013 at 5:17 PM, Som Snytt <som....@gmail.com> wrote:

It's true, I think Paul was actually sobbing.

He notes somewhere that he wants to do another clean-up, but that's kind of suboptimal use of time.  Do you want Paulp-Test or will Par-Test suffice?  Keeping in mind that one paulp-hour is 1.5 ordinary man-days.

I've got a couple of Par-Test PRs, and I'm willing to tweak more if someone creates an issue.

I didn't think I was up for it after almost burning out on the Testicolor merge; what got me through that was looking at paulp code.  For every brain cell that died, a new one sprouted forth on a different lobe.

I just now needed it to handle crashes with nuance, since sometimes the compiler throws FatalError when it's not so much crashing as falling on its sword.

Also, +1 to --failed.  A couple of weeks ago, I made a change to put stderr in the logs and had to update a gazillion tests, finding a few minor issues.  --failed worked charmingly.  I also benefited from Jason's feature to call out `updated' status.

My #1 feature request would be to avoid littering the output with successful tests. It's nice to see things are moving, but you could delete the line once tests succeed. SBT does something similar as it resolves dependencies.

-jason 

[Eugene Burmako]

And I'd be delighted to be able to opt out of post-test diagnostics, reducing it to a plain list of failed tests.

--

[Paul Phillips]

Okay, how about this:

#### Generate a little test noise ####

% for f in test/files/run/t41*check; do echo "BIPPY\nBOO" >> $f ; done

% test/partest test/files/run/t41*

Discarding 8 invalid test paths

Selected 11 tests drawn from specified tests

# starting 11 tests in run

!!  1 - run/t4110-old.scala                       [output differs]

!!  2 - run/t4148.scala                           [output differs]

ok  3 - run/t4119                               

ok  4 - run/t4147.scala                         

!!  5 - run/t4138.scala                           [output differs]

ok  6 - run/t4122.scala                         

!!  7 - run/t4124.scala                           [output differs]

!!  8 - run/t4110-new.scala                       [output differs]

!!  9 - run/t4171.scala                           [output differs]

!! 10 - run/t4190.scala                           [output differs]

!! 11 - run/t4172.scala                           [output differs]

# 3/11 passed, 8 failed in run

Failed test paths (this command can be run to update checkfiles)

test/partest --update-check \

  test/files/run/t4110-new.scala \

  test/files/run/t4110-old.scala \

  test/files/run/t4124.scala \

  test/files/run/t4138.scala \

  test/files/run/t4148.scala \

  test/files/run/t4171.scala \

  test/files/run/t4172.scala \

  test/files/run/t4190.scala

3/11 passed, 8 failed (elapsed time: 00:00:18)

Test Run FAILED

#### paulp pastes the above suggestion into the terminal ####

% test/partest --update-check \

>   test/files/run/t4110-new.scala \

>   test/files/run/t4110-old.scala \

>   test/files/run/t4124.scala \

>   test/files/run/t4138.scala \

>   test/files/run/t4148.scala \

>   test/files/run/t4171.scala \

>   test/files/run/t4172.scala \

>   test/files/run/t4190.scala

Selected 8 tests drawn from specified tests

# starting 8 tests in run

++ 1 - run/t4171.scala                           [updated check file]

++ 2 - run/t4110-old.scala                       [updated check file]

++ 3 - run/t4148.scala                           [updated check file]

++ 4 - run/t4138.scala                           [updated check file]

++ 5 - run/t4124.scala                           [updated check file]

++ 6 - run/t4190.scala                           [updated check file]

++ 7 - run/t4110-new.scala                       [updated check file]

++ 8 - run/t4172.scala                           [updated check file]

8/8 passed (elapsed time: 00:00:15)

Test Run PASSED

[Paul Phillips]

Oh, and I completely forgot about it myself, but there has all along been

  test/partest --terse

Which outputs each success with a dot, like this:

% test/partest --terse

Selected 3395 tests drawn from 12 named test categories

# starting 1087 tests in pos

........................................................................

........................................................................

............................

But I'm making that the default and pushing per-test success messages into --verbose.

[Som Snytt]

Failed test paths (this command can be run to update checkfiles)

test/partest --update-check \

  test/files/run/t4110-new.scala \

  test/files/run/t4110-old.scala \

  test/files/run/t4124.scala \

  test/files/run/t4138.scala \

  test/files/run/t4148.scala \

  test/files/run/t4171.scala \

  test/files/run/t4172.scala \

  test/files/run/t4190.scala

3/11 passed, 8 failed (elapsed time: 00:00:18)

Test Run FAILED

Update checkfiles now? (y/n/i) i
  test/files/run/t4110-new.scala ? (y/n) n
  test/files/run/t4110-old.scala ? (y/n) y

[Paul Phillips]

I've been saying to adriaan for a while that jenkins should run with --update-check on all the time. When it's all done, we have a bunch of updated checkfiles which the bot can commit and push to github, then send a link to someone who takes all this seriously, if we can find such a person. That human pushes a thumbs-up button and we all go home.

[Paul Phillips]

Given that test runs always take place in a clean checkout, one can think of this approach as simply leveraging git for its particular talents in managing differences.

[Simon Ochsenreither]

Is there a way in the new partest to recognize an executed test as skipped? E. g. by throwing some TestSkippedThrowable (as suggested on GitHub) which is catched by partest?

I'm asking this because it just became interesting for Avian: t4294 is failing, because the GC is currently not "smart enough to determine the exact lifetime of each local/stack reference, and therefore considers the "this" reference to be reachable throughout the body of Stream.collect" (Joel).

I think this is a case where we just want to skip the test for now.