New (possibly) codespeed user: GHC

[Joachim Breitner]

Hi,

I’d just like to say hello! I have evaluated a few benchmark dashboards,
and codespeed seemed to be the most accessible, i.e. it has a
project-agnosic home, instructions on how to set it up and so on. So I
picked it for a not-yet-official performance monitoring solution for
GHC, the Haskell compiler.

Currently it runs on
http://ghcspeed-nomeata.rhcloud.com/ on OpenShift. If it works well it
might be moved to GHC’s own infrastructure eventually.

I keep my code in
https://github.com/nomeata/codespeed (branch ghc). I didn’t separate my
bugfix commits from customization commit yet, but will do that
eventually. I also filed a few bugs and feature requests myself :-)

Greetings,
Joachim


--
Joachim “nomeata” Breitner
ma...@joachim-breitner.de • http://www.joachim-breitner.de/
Jabber: nom...@joachim-breitner.de • GPG-Key: 0xF0FBF51F
Debian Developer: nom...@debian.org

[Miquel Torres]

Hi Joachim,

that's pretty cool!

I already looked at some of the issues you posted and what you are
improving in your fork. Makes a lot of sense. I also appreciate how
you went through older bugs and triaged them. Thanks a bunch!

I certainly need to cleanup the issues and pull requests a bit. Plus
polish the README and a couple of other things and then doing a minor
release. Your feedback is very useful there.

Speaking of feedback, it would be great if you could list the biggest
pain points you encountered while settings up the GHC codespeed
instance, and from your perspective what the weakest points of the app
itself are. Being the last project to adopt codespeed, I assume from
the latest release or master, your input would be very useful in
ironing out rough edges.

Cheers,
Miquel

[Joachim Breitner]

Dear Miguel,

Am Donnerstag, den 17.07.2014, 21:06 +0200 schrieb Miquel Torres:
> Speaking of feedback, it would be great if you could list the biggest
> pain points you encountered while settings up the GHC codespeed
> instance, and from your perspective what the weakest points of the app
> itself are. Being the last project to adopt codespeed, I assume from
> the latest release or master, your input would be very useful in
> ironing out rough edges.

I’m working off master.

One important feature that we’ll need before fully deploying this for
GHC are authenticated result submissions. I think a very simple solution
(shared secreted attached to the environment object that needs to be
part of the commit) should be sufficient for us.

I also found that the reports are not very reliable: If I upload the
results from revisions in the wrong order, they are WRT to the wrong
revision. (This is https://github.com/tobami/codespeed/issues/52).

Maybe related, maybe not: Sometimes the reports (or the revisions, not
sure) get a wrong date, and hence are sorted in the wrong spot, compared
to the wrong revision etc. Recalculating the report sometimes helps. I
report results as follows
{
"units_title" : "Tests",
"project" : "GHC",
"lessisbetter" : "False",
"branch" : "default",
"environment" : "nomeata's buildbot",
"result_value" : "3634",
"commitid" : "55e7ab1210975e6276f3cab3ac0e1f35bcd772f0",
"executable" : "ghc",
"benchmark" : "testsuite/expected passes",
"units" : "tests"
},
and rely on the GitHub integration to pull the revisions’s meta data.

It might help if the report would indicate against what revision the
„Change“ is calculated.


Generally, I would wish for a better git integration. Often the “Commit
logs” list more commits than they should. And I would expect codespeed,
which orders commits by date, to get confused if commits are added to
the repo in a different order than what their Dates indicate. I would
find it great if it could completely ignore the date here and use the
parent-relationship of commits directly. But I imagine that this is
hard, given that codespeed wants to support other VCSs as well.


That’s the most critical stuff so far. I’ll keep you up to date on more
problem that I find, either here or or on the bug tracker.

Looking forward to your commits!

[Miquel Torres]

OK. Before I or anyone else implements anything, it would be good to
know what you are already tackling. If you don't want to start sending
small pull requests, yet, i can also integrate individual patches that
seem meaningful. For example the admin action. Or just list the issues
you are definitely *not* solving or implementing yourself.

What do you think?

[Joachim Breitner]

Hi,

I’m currently not working on anything, so don’t let me stop you. If I do
want to work on anything, I’ll make sure to announce it on a relevant
ticket.

I’ll also put all my patches that I think might go into the mainline on
separate branches and offer them as pull requests.

Greetings,
Joachim

[Miquel Torres]

Great, thanks