Benchmarks with Jenkins

jukka...@gmail.com

unread,

Jun 20, 2013, 4:44:49 AM6/20/13

to junit-be...@googlegroups.com

Hi!

We are looking for a benchmarking solution that can be integrated with Jenkins in such a way that:

1. Benchmark result trends can be analyzed from Jenkins

2. Performance degradation above certain threshold would cause an alert

We have looked at Caliper but it does not work with Windows development and does not seem to be in active development.

If we were to implement the features to JUnitBencmarks I mentioned above, what would be the simplest and most effective way to go forward?

Regards,

Jukka Aho

Dawid Weiss

unread,

Jun 20, 2013, 5:12:01 AM6/20/13

to junit-be...@googlegroups.com

Hi Jukka,

> We have looked at Caliper but it does not work with Windows development and
> does not seem to be in active development.

Caliper is actively developed. Some development takes place behind the
scenes at Google and they occasionally bulk-merge the changes. The
latest release does not work on Windows because they've moved to
pipe-based communication channels but the previous release did work on
Windows and has similar capability.

> If we were to implement the features to JUnitBencmarks I mentioned above,
> what would be the simplest and most effective way to go forward?

I'd say JUnitBenchmarks could be run from Jenkins but then another
run-round should pull the benchmark results from a database and raise
an alert if something deviates from normal. It shouldn't be the test
itself.

There are also other projects aimed at performance testing -- openjdk
has one, for example:
http://openjdk.java.net/projects/code-tools/jmh/

JUnitBenchmarks is currently in "sleep" mode. I don't know where to
take it further or if I even should. If you'd still like to fork the
project go ahead -- improvements are always welcome and I'll gladly
review the patches.

Dawid

jukka...@gmail.com

unread,

Jun 20, 2013, 6:13:47 AM6/20/13

to junit-be...@googlegroups.com

Thanks for the info!

For the time being we will use JUnitBenchmarks but perhaps for the CI-integration side we might be well off with manual inspection of benchmark results.

And then we'll wait and see how Caliper goes forward.

-Jukka

Peter Veentjer

unread,

Jun 20, 2013, 6:24:04 AM6/20/13

to junit-be...@googlegroups.com

I have tried setting up Caliper and I found it a nightmare.

With Junit-Benchmarks it was very simple and I had pretty graphs in now time. We also created a small script that generates and overview page based on all the generated pages and integrated that in Jenkins. By no means perfect, but something is better than nothing.

After the mvn has run the benchmarks from jenkins, we also configured the 'execute shell' section:

cd /usr/share/tomcat7/webapps/benchmarks

./generateindex.sh > index.html

And this is the script that is in the same directory as the generated html pages. After this script is run, you have a single index.html that points to all generated benchmark pages.

#!/bin/bash

echo '<html>'

echo "<head>"

echo "<title>Hazelcast Benchmarks</title>"

echo "<style type="text/css">

#content {

width: 800px ;

margin-left: auto;

margin-right: auto;

}

#chart {

width: 600px;

height: 400px;

margin-left: auto;

margin-right: auto;

margin-bottom: 10px;

display: block;

}

ul.expander {

display: inline;

margin-left: 0;

padding-left: 0;

font-size: .8em;

}

ul.expander li {

margin-left: 0;

list-style: none;

display: inline;

}

ul.expander li:before {

content: "\25BA\0020"; // 25B8

}

ul.expander li.expanded:before {

content: "\25BC\0020"; // 25BE

}

table.fancy {

margin: 1em 1em 1em 0;

background: whitesmoke;

border-collapse: collapse;

}

table.fancy th, table.fancy td {

border: 1px silver gainsboro;

padding: 0.2em;

}

table.fancy th {

background: gainsboro;

text-align: left;

}

table.fancy tr:hover td {

background: mintcream !important;

}

table.fancy caption {

margin-left: inherit;

margin-right: inherit;

}

</style>"

echo '</head>'

echo '<body style="font-family: Arial; margin: 30px;">'

echo "<h1>Hazelcast Benchmark Results</h1>"

echo "<table class=\"fancy\">"

echo "<tr>"

echo "<th>Performance history</th>"

echo "<th>Performance latest benchmark</th>"

echo "</tr>"

for f in *

do

if [[ "$f" == *.html ]]

then

if [[ ("$f" != *-history.html) && ("$f" != index.html) ]]

then

b=$(echo $f | sed -e 's/$.html$*$//g')

echo '<tr>'

echo '<td>'

echo "<a href=\"${b}-history.html\">${b}-history</a>"

echo '</td>'

echo '<td>'

echo "<a href=\"$f\">$b</a>"

echo '</td>'

echo '</tr>'

fi

done

echo "</table>"

echo '</body>'

echo '</html>'

--
You received this message because you are subscribed to the Google Groups "JUnitBenchmarks: Performance Benchmarking for JUnit4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to junit-benchmar...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dawid Weiss

unread,

Jun 20, 2013, 7:16:41 AM6/20/13

to junit-be...@googlegroups.com

Ha! Thanks for sharing, Peter :)
Dawid

Reply all

Reply to author

Forward