Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

[sharmaa...@gmail.com]

Hi everyone,

As a part of this year's Google Summer of Code, I have been working on a
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
run with Jenkins. The basic framework was implemented in the Role Strategy
Plugin through this pull request:
https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
benchmark, a temporary Jenkins instance is created, similar to how
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
through a unit test which allows them to be integrated with the normal pull
request builds (if required). Apart from configuring the Jenkins instance
for test using Java code, I've also added support for easily configuring the
instance by YAML files using Jenkins Configuration as Code. The benchmark
reports are generated as JSON files which are compatible with the jmh-report
plugin. I have implemented a couple of benchmarks for the Role Strategy
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
have attached screenshots of these results as visualized by the jmh-report
plugin. To make the framework easier to use, I'm also working on running the
benchmarks through a Maven profile.

To make it available to the rest of the community, I would like to propose
to make it either a part of Jenkins Test Harness or a part of new library so
it can be used by other plugins and perhaps by Jenkins core itself. I would
to love to hear your feedback, comments and suggestions. It would be even
better if you could join the project's weekly meetings (Tuesdays and Fridays
at 7:00 AM UTC) and help us improve this framework.

Thanks
Abhyudaya
GitHub: AbhyudayaSharma

[Oleg Nenashev]

Hi all,

Just to facilitate this thread, I am +1 regarding having micro-benchmarking support in Jenkins Test Harness. There are many stories related to performance degradation in Jenkins, and having a framework could help us to have some checks for the critical functionality like permission checks. The end goal of the project is to improve Role Strategy performance (which is far from perfect now), but it would be great if we could reuse the framework in other components.

Tomorrow at 7am UTC we will be doing a recorded demo of the current framework state, and everybody is welcome to join.

Best regards,

Oleg

[Matt Sicker]

I'm +1 in using JMH and exploring benchmarks of various functionality.
Seems like a good idea, especially with all the reinvented database
functionality implicit in Jenkins and plugins.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Matt Sicker
Senior Software Engineer, CloudBees

[Oleg Nenashev]

Hi all,

You can find the video recording of the demo here: https://www.youtube.com/watch?v=sr28UADG1AE

There is also an ongoing pull request to make it a part of Jenkins Test Harness: https://github.com/jenkinsci/jenkins-test-harness/pull/135

Best regards,

Oleg

On Thursday, May 30, 2019 at 6:44:38 PM UTC+2, Matt Sicker wrote:

I'm +1 in using JMH and exploring benchmarks of various functionality.
Seems like a good idea, especially with all the reinvented database
functionality implicit in Jenkins and plugins.

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[sharmaa...@gmail.com]

Hi everyone,

 

Just to update you all, the framework for running JMH benchmarks is almost ready to be merged into Jenkins Test Harness. Please see this pull request. There is also a pull request to the plugin-pom for running benchmarks through a Maven profile. For ease of use in plugins hosted on ci.jenkins.io, there is also a work-in-progress pull request to add support for running them as a pipeline step to Jenkins’ pipeline library.

 

I would love to hear your feedback and comments.

 

Thanks

Abhyudaya Sharma

GitHub: AbhyudayaSharma

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/W1jrwTEFnz0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/676323d7-7d32-43be-9003-d0d8eab9e02e%40googlegroups.com.

[Abhyudaya Sharma]

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a blog post summarizing it on jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin here. Looking forward to your feedback and comments.

Thanks

Abhyudaya Sharma

[Ullrich Hafner]

Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 

E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:

- parse warnings

- compute fingerprints

- collect blames from Git

etc. 

Is this supported somehow yet?

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.

[Mark Waite]

Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

[Rishabh Budhouliya]

From Role Strategy Plugin or Configuration as a Code Plugin, the JMH examples show us that the intention is to provide a JMH environment which is fairly similar to a JUnit test environment. For example, as Mark mentioned, a JenkinsState was created to provide a JenkinsRule in the JMH environment to benchmark associated functionalities.

JMH provides a static state class where you can initialize almost everything you need to benchmark a step or a particular operation. I used that approach to benchmark git operations without using a JenkinsRule by just creating a custom class which would provide me fresh local repos for the lifetime of the benchmark. (the idea was simply copied from JUnit @Rule TemporaryFolder class)

So it is fair to assume that anything we can test in a JUnit test, we can do that in a JMH benchmark too.

On Sunday, June 7, 2020 at 10:26:31 PM UTC+5:30, Mark Waite wrote:

Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[Ullrich Hafner]

I see, that makes sense. Seems that I was just following the wrong path…  

Do you have a link to those Git performance tests? Are they in the git or in the git-client plugin?

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com.

[Rishabh Budhouliya]

The benchmark module in the git-client plugin: https://github.com/jenkinsci/git-client-plugin/tree/master/src/test/java/jmh/benchmark
A JMH benchmark on git fetch: https://github.com/jenkinsci/git-client-plugin/blob/master/src/test/java/jmh/benchmark/GitClientFetchBenchmark.java

On Monday, June 8, 2020 at 4:57:51 PM UTC+5:30, Ullrich Hafner wrote:

I see, that makes sense. Seems that I was just following the wrong path…  

Do you have a link to those Git performance tests? Are they in the git or in the git-client plugin?

Am 07.06.2020 um 18:56 schrieb Mark Waite <mark.e...@gmail.com>:

Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.