Solicitation of Ideas for Performance Statistics and Graphs
Joshua Gatcomb
Matt Diephouse and I spent the majority of our time
coming up with a flexible design and gathering
historical statistics. We didn't spend a lot of time
in how to present the data since everybody has their
own opinion (including us).
What we would like to do is determine if what we have
done so far is sufficient or, if not, what
specifically
people would like to see. Some of our unimplemented
ideas so far are:
1. Include the computed goto core
2. Summary of results over N week(s)/month(s)
3. Provide user form for dynamic results
If people would like this, they also need to
indicate what the form should provide:
(benchmark name, date, executable, etc)
4. Provide HTML table of data for some/all of graphs
5. Provide links for people to work locally
A. A db schema/structure dump so people can
collect statistics on other architectures
B. Source code
C. daily db dump
If you would like to see any of these ideas
implemented, or you have some of your own - please
respond to this on the list.
While Matt and I are likely to do some of the work, we
are always open to volunteers as well. In fact, if
this idea generates a lot of buzz, we may cross post
to other Perl forums such as PerlMonks
(http://www.perlmonks.org). This is a perfect way for
non C programmers and those who have no knowledge of
Parrot to still contribute in a meaningful way.
Cheers
Joshua Gatcomb
a.k.a. Limbic~Region
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Matt Fowles
I think it would be really cool if commits that had a significant
increase or descrease in speed would be flagged. Possibly just a
section of the page could be a table with commit dates and the percent
effect they had. This table would not contain all commits dates, but
only the most recent N that caused a greater than 5% change in speed.
The idea being that this would provide an easy way to sync to
particular revision for people playing the optimization game without
having to guess based on the graphs.
Matt
--
"Computer Science is merely the post-Turing Decline of Formal Systems Theory."
-???
Dan Sugalski
>On Wed, 3 Nov 2004 16:04:38 -0500, Matt Fowles <uber...@gmail.com> wrote:
>> I think it would be really cool if commits that had a significant
>> increase or descrease in speed would be flagged. Possibly just a
>> section of the page could be a table with commit dates and the percent
>> effect they had. This table would not contain all commits dates, but
>> only the most recent N that caused a greater than 5% change in speed.
>
>running once per day, which narrows it sufficiently.
It's also tough to tell exactly what a commit is, with CVS. It
doesn't have the concept of multi-file commits, so if I check in
twenty files at once it's viewed as twenty commits. Which is one of
CVS's issues.
--
Dan
--------------------------------------it's like this-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Bill Coffman
> What we would like to do is determine if what we have
> done so far is sufficient or, if not, what specifically
> people would like to see. Some of our unimplemented
> ideas so far are:
> 1. Include the computed goto core
> 2. Summary of results over N week(s)/month(s)
> 3. Provide user form for dynamic results
> If people would like this, they also need to
> indicate what the form should provide:
> (benchmark name, date, executable, etc)
> 4. Provide HTML table of data for some/all of graphs
> 5. Provide links for people to work locally
> A. A db schema/structure dump so people can
> collect statistics on other architectures
> B. Source code
> C. daily db dump
>
> If you would like to see any of these ideas
> implemented, or you have some of your own - please
> respond to this on the list.
I think this is a really great idea. I made some picture graphs for
the register allocation stuff myself, and tried to evangelize the idea
a little, but you guys have introduced a useful tool that can help
developers see the results of their work. Now, one can anxiously
check the results after a checkin and see if a desired speedup has
occurred or not. Great job.
To make this idea, and philosophy, pay off, we need (3&5B)++. I think
what you're suggesting is to provide the software to others, so they
can run their own tests. That's great, but I'd rather create the
test, check it in, and modify a master config file, that says which
tests will run. The administrator, of course would have to approve
the patch. For long tests, they don't have to be in the nightly test
suit, but could be run by name, by the user.
For number 4, I'd say that a comma separated values (CSV) file would
be most useful. Then users can view and manipulate the data
themselves. Various spreadsheets have nice capabilities for this, and
of course certain scripting languages can parse this kind of data
well. So I hear, at least :).
Once again, great idea, great job!
~Bill
Leopold Toetsch
> If you would like to see any of these ideas
> implemented, or you have some of your own - please
> respond to this on the list.
I've amother one. Parrot has some internal settings and tweakable magic
constants, mainly all inside the garbage collector. It would be great,
if some of these settings could be compared.
E.g.
# bench.config
[ "Default" ] # label for config section / graph
[ "IMS GC" ] # incremental M&S GC
# file # subst to turn config on
include/parrot/settings.h s/(PARROT_GC_MS)\s+1/$1 0/
# next setting, either based on previous or on default
Now instead of an overlay graph of unoptimized/optimized/non-JIT/JIT
(which isn't utterly useful) I'd like to see a comparison of the given
configurations, optimized build only.
WRT parrot run-options: there is a "-Ot" switch, which should enable the
fastest runtime options available on that machine.
> Cheers
> Joshua Gatcomb
leo