Intermittently Failing Benchmarks

[Joshua Gatcomb]

All:
In collecting the historical data for the benchmark
statistics and graphs, I discovered that there were a
few days where I had to play the CVS time game to get
a working parrot for that day. I expected this.

What I have found interesting though is when
individual benchmarks don't work. For instance, from
10/20 to 10/22, gc_generations and gc_header_reuse
would just hange (still running after 10 minutes).
Last night (11/3 at 23:59) addit2.imc is doing the
same thing. I checked a up to the minute checkout and
it is finishing now - but there is no printed output.

So I have 2 questions:
1. Would people prefer missing data for benchmarks
where they won't work or a manually entered high
number to draw attention to them?
2. Should we be checking that the output of the
benchmarks (right or wrong) is consistent?

Cheers
Joshua Gatcomb
a.k.a. Limbic~Region

__________________________________
Do you Yahoo!?
Check out the new Yahoo! Front Page.
www.yahoo.com

[Matt Diephouse]

On Thu, 4 Nov 2004 08:57:28 -0800 (PST), Joshua Gatcomb
<limbic_re...@yahoo.com> wrote:
> What I have found interesting though is when
> individual benchmarks don't work. For instance, from
> 10/20 to 10/22, gc_generations and gc_header_reuse
> would just hange (still running after 10 minutes).
> Last night (11/3 at 23:59) addit2.imc is doing the
> same thing. I checked a up to the minute checkout and
> it is finishing now - but there is no printed output.

Maybe the benchmarks should be part of the test suite? They're valid
code, so they should work at all times: if they don't, something's
broken. Seems like a good opportunity for testing to me.

--
matt

[Leopold Toetsch]

Matt Diephouse <mdd...@gmail.com> wrote:

> Maybe the benchmarks should be part of the test suite? They're valid
> code, so they should work at all times: if they don't, something's
> broken. Seems like a good opportunity for testing to me.

Yep.

Patches welcome. But please make sure that they don't run too long.
Output of a few (gc_*.pasm comes to my mind) needs some tweaking WRT
result.

leo

[James Mastros]

Joshua Gatcomb wrote:
> 1. Would people prefer missing data for benchmarks
> where they won't work or a manually entered high
> number to draw attention to them?

Make the harness time out at ten minutes, and enter a completion time of
11 minutes for those that don't finish in time? (For many graphs of,
say, IP performance, a router that drops 99 of 100 packets, but delivers
that 1 packet after only 1ns will show up as simply having a 1ns average
response time, which is amazingly misleading.)

> 2. Should we be checking that the output of the
> benchmarks (right or wrong) is consistent?

Possibly. I'd be more interested in running the test suite as well as
the benchmarks, and plotting a line of %success along with the response
time.

-=- James Mastros,
theorbtwo