Migrating a maven-based job to pipeline style

[Victor Noël]

Hi,

I'm trying to wrap wy head around moving to pipeline jobs.

Currently I use maven jobs (but my question applies to freestyle jobs that execute maven) by executing one single build command that will validate constraints (checkstyle and/or other stuffs like that), generate code, compile, run test, run it test and deploy artefacts.

What is the best way to convert to pipeline with multiple stages?

My main concern is that if I simply run each of the step with maven in each stage, then I feel like some of steps from maven may be executed multiple times (such as test being executed again for it tests or for deploy).

Is there some success stories around here on how to handle such a migration?

Thanks!

[wo...@anomalizer.net]

Hello Victor,

I am faced with a similar problem and gave up on the idea rather early on. Here are some of the characteristics of multiple stages of a pipeline

• Ability of each stage to be explictly invoked by some command
• Ability of each stage to be re-run in such a way that it can start off from where the previous stage finished

The maven lifecycle reactor does not play well with these concepts. For example, there is no elegant way to tell maven to compile the code assuming that the generation phase has happened. Likewise, one cannot ask maven to simply execute the tests assuming that compilation has already taken place. The short answer that the benefits of breaking something into various pipeline stages is lost when maven.

The situation with gradle is not better as far as the relationship with the build tool and jenkins is concerned. However, the fact that gradle is much better at quickly checking if earlier steps have been completed makes it only marginally better.

[Victor Noël]

Thanks for your feedbacks!

So in the end, what is the solution you adopted? more coarse grained stages?

[khmarbaise]

Hi,

To run checkstyle alone you can simply call mvn checkstyle:check ? If you really like to run that into a separate pipeline step...

The question is what kind of advantage you would expect by running generating code alone and afterwards in a different pipeline step to compile the generated code and another pipeline step to run the unit tests etc. instead of simply running: mvn deploy ? in a single step? What is the real advantage of this approach?

Kind regards

Karl Heinz Marbaise

[jieryn]

My team faces similar challenges, and I agree with almost everything
said so far. I definitely echo the sentiment that it feels like there
is a lot of wasted repetition when invoking discrete phases
separately. We currently do this via separate jobs now, which often
run at different frequencies. However, we have been able to configure
Maven such that we really only have two repetitious phases, across
these multiple jobs, which luckily are not costly for us. Maven
generate-resources and compile phases, as we can skip over tests and
deployments which are not part of the logical unit being executed. In
the end, even though we know we are wasting some execution time, it
turns out not to be all that much at all.. and isn't yet worthy of any
kind of optimization or special attention.

So right now our (~2min) compile+unit test runs every commit and only
runs Junit tests. Our (~8min) compile+integration test runs almost
every commit+jenkins build window to group multiple commits and only
runs Arquillian tests. Our (~15min) compile+user acceptance test runs
every day and only runs Selenium tests. Then we have lots of on the
side jobs which ensure legal compliance, code coverage, file encoding
fixes, code style fixes, static analysis, etc etc, which run hourly,
daily, or weekly, and are mostly fast (<5min).

We are exploring Jenkins Pipeline in order to chain, both sequential
where necessary and then parallel where it makes sense, these discrete
jobs in such a way that we will have a continuous delivery available
at the end which has run the full gauntlet of all our testing and best
practices verification for every commit without doing unnecessary and
expensive steps, failing the pipeline as soon as possible. Each commit
triggers a build which triggers a new git branch which is passed to
subsequent phases, if any stage fails, the branch is deleted and makes
no further progress. Additionally, we are trying to leverage separate
Jenkins jobs for each discrete job so that we can trigger it on demand
in order to do some quick checking, we also find that we can get
easier cross-project reuse using that technique. We're using Git,
Apache Maven, Jenkins CI, Jacoco, Arquillian, SonarQube, Checkstyle,
PMD, FindBugs, and many others.

There are only two critiques we can find with this style: 1) the
aforementioned modest amount of wasted generate-resources and compile
phases across multiple jobs in the same pipeline, and 2) each phase is
producing artifacts to be tested and not reusing the ones built and
verified at previous stages, thus there may be a hypothetical
difference between what different stages are actually testing, though
we think the risk of this is not consequential. This technique
otherwise seems like a holy grail to me and the team. I welcome ideas
about it.

> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/e869c7c4-2ea5-4d8b-8268-7a083348b2b7%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[khmarbaise]

Hi,

so have you really measured the time you are "wasting" ? Furthermore which Maven version and plugin versions do you use? Are you using freestyle or Maven job type...

On Saturday, July 30, 2016 at 8:13:14 PM UTC+2, Jesse Farinacci wrote:

My team faces similar challenges, and I agree with almost everything
said so far. I definitely echo the sentiment that it feels like there
is a lot of wasted repetition when invoking discrete phases
separately. We currently do this via separate jobs now, which often
run at different frequencies. However, we have been able to configure
Maven such that we really only have two repetitious phases, across
these multiple jobs, which luckily are not costly for us. Maven
generate-resources and compile phases, as we can skip over tests and
deployments which are not part of the logical unit being executed. In
the end, even though we know we are wasting some execution time, it
turns out not to be all that much at all.. and isn't yet worthy of any
kind of optimization or special attention.

So right now our (~2min) compile+unit test runs every commit and only
runs Junit tests.

2 Minutes..Ok how many time is "wasted"  (timestamper plugin and the mojo execution in Jenkins if you you are using maven job type can help here) 

BTW: How many modules does you project contains of?

 

Our (~8min) compile+integration test runs almost
every commit+jenkins build window to group multiple commits and only
runs Arquillian tests.

You might run jobs in parallel ?

You could create in Jenkins Maven repositories to provided the artifacts to other steps...Or can package them by Jenkins an give them to the next step...

So it sounds you don't trust your version control ? If you checkout the same sha1 you will get always the same state? And if you build from that you will always get the same result didn't you ? Or I could say you don't trust the build system to create the same result from the same source state ?

 

This technique 

otherwise seems like a holy grail to me and the team. I welcome ideas
about it.

If it is already the hole grail I can't say anything about it...

[jieryn]

Hi,

On Sat, Jul 30, 2016 at 2:39 PM, khmarbaise <goo...@soebes.de> wrote:
> so have you really measured the time you are "wasting" ? Furthermore which
> Maven version and plugin versions do you use? Are you using freestyle or
> Maven job type...

Yes, we have measured. Both with the Tesla Maven Profiler, but also
even just crudely. The profiler led us to remove a lot of stuff that
we had attached to the build lifecycle and have now pushed into
dedicated jobs (e.g. pathological but necessary m-enforcer-p
enforcement, other stuff). So for example, running a clean install and
skipping all tests on a typical project versus clean install yields
about 10-20% reduction in wall clock execution time. That, to me,
means we are repeating about 80-90% of "stuff" when we execute it
again but just to run the integration tests and again for acceptance
tests. Those timings were found on my development machine.

We do not run the Maven job type anymore, I used to be a big fan of
it, but in order to get supremely fast flow, we abandoned it. A lot of
the nice stuff where Maven integrates with Jenkins just isn't all
useful enough for the cost. I realize that is a sensitive topic, and I
don't want to get into it more, and all the research that I performed
is at least a year old, I'm not looking to start a discussion on this,
and I'm certainly not willing to go through another round of research.
We use Freestyle, and that is unlikely to change unless someone else
can demonstrate very sizable performance wins.

We always run the latest Maven version, 3.3.9 currently, all
dependencies and plugins are up to date (thank you
versions:display-{dependency,plugin}-updates mojo which we run weekly
and then perform updates). Finally, most developers in our area use
--threads 2.0C but on Jenkins we do not because we strictly control
the number of slaves per real machine, and do not like to get
surprised about how much usage a slave is actually going to cost.
Though, since we did move to RHEL 7 with systemd and cgroups, this
probably isn't an actual issue anymore and we should revisit it. But
for simplicity, -T1 is used for all slaves.

> On Saturday, July 30, 2016 at 8:13:14 PM UTC+2, Jesse Farinacci wrote:
>>
>> My team faces similar challenges, and I agree with almost everything
>> said so far. I definitely echo the sentiment that it feels like there
>> is a lot of wasted repetition when invoking discrete phases
>> separately. We currently do this via separate jobs now, which often
>> run at different frequencies. However, we have been able to configure
>> Maven such that we really only have two repetitious phases, across
>> these multiple jobs, which luckily are not costly for us. Maven
>> generate-resources and compile phases, as we can skip over tests and
>> deployments which are not part of the logical unit being executed. In
>> the end, even though we know we are wasting some execution time, it
>> turns out not to be all that much at all.. and isn't yet worthy of any
>> kind of optimization or special attention.
>>
>> So right now our (~2min) compile+unit test runs every commit and only
>> runs Junit tests.
>
> 2 Minutes..Ok how many time is "wasted" (timestamper plugin and the mojo
> execution in Jenkins if you you are using maven job type can help here)
> BTW: How many modules does you project contains of?

Please see above for timing calculation process and results. A typical
project for us has 25 Maven modules.

>> Our (~8min) compile+integration test runs almost
>

> You might run jobs in parallel ?

It would be kind of crummy if we burned 8 minutes on integration test
if a 2 minute unit test would have aborted the build pipeline
previously. Our integration tests do not run the unit tests. Our user
acceptance tests do not run either the integration or unit tests. We
basically check out the project, update the version to be
1.0.$BUILD_NUMBER, run the unit tests, and if they pass we push the
branch to GitLab. That kicks off a ton of compliance jobs, all in
parallel (yaay!) And then after those all pass, we go to integration
test stage. This stage is just running our Arquillian tests, and given
how costly this phase and the phases after it is in time and
resources, we go back down to blocking pipeline on this job.

I like the idea, I would also consider using the archive and unarchive
options of Jenkins Pipeline. The problem to me seems to be how to run
unit and integration tests on previously built .jar files. Like, skip
the compile phase for src/main/java and just run src/test/java and
src/it/java against some Jenkins' archived version of the previously
compiled .jar. It didn't seem possible to me. We could definitely skip
packaging the web applications which are used for the user acceptance
tests using Selenium, but we don't use Docker as it isn't actually
available on our target platform.

>> This technique
>> otherwise seems like a holy grail to me and the team. I welcome ideas
>> about it.
>
> If it is already the hole grail I can't say anything about it...

:-D

[khmarbaise]

On Sunday, July 31, 2016 at 1:26:58 AM UTC+2, Jesse Farinacci wrote:

Hi,

On Sat, Jul 30, 2016 at 2:39 PM, khmarbaise <goo...@soebes.de> wrote:
> so have you really measured the time you are "wasting" ? Furthermore which
> Maven version and plugin versions do you use? Are you using freestyle or
> Maven job type...


Yes, we have measured. Both with the Tesla Maven Profiler, but also
even just crudely. The profiler led us to remove a lot of stuff that
we had attached to the build lifecycle and have now pushed into
dedicated jobs (e.g. pathological but necessary m-enforcer-p
enforcement, other stuff). So for example, running a clean install and
skipping all tests on a typical project versus clean install yields
about 10-20% reduction in wall clock execution time.

Skipping Test will of course reduce the time (Do you know how long the tests alone take?) 

but I wanted to know the exact "wasted" time you have mentioned by compile/resources? 

So running a linear (single thread) mvn clean install vs. mvn install (in both cases with skipping tests at all)...

If you always do a clean than Maven (plugins) can't identify things which have already been done cause they are not there...so yes in this case you are wasting time...

 

That, to me,
means we are repeating about 80-90% of "stuff" when we execute it
again

Based on the usage of "clean" ...

 

but just to run the integration tests and again for acceptance
tests. Those timings were found on my development machine.

We do not run the Maven job type anymore, I used to be a big fan of
it, but in order to get supremely fast flow, we abandoned it. A lot of
the nice stuff where Maven integrates with Jenkins just isn't all
useful enough for the cost. I realize that is a sensitive topic, and I
don't want to get into it more,

As others and me too say Maven Job Type is evil ...cause it influences a Maven Build in many ways...

 

and all the research that I performed
is at least a year old, I'm not looking to start a discussion on this,
and I'm certainly not willing to go through another round of research.
We use Freestyle, and that is unlikely to change unless someone else
can demonstrate very sizable performance wins.

We always run the latest Maven version, 3.3.9 currently, all
dependencies and plugins are up to date (thank you
versions:display-{dependency,plugin}-updates mojo which we run weekly
and then perform updates).

very good...

[jieryn]

Hi,

On Sun, Jul 31, 2016 at 3:09 AM, khmarbaise <goo...@soebes.de> wrote:
>
>> enforcement, other stuff). So for example, running a clean install and
>> skipping all tests on a typical project versus clean install yields
>> about 10-20% reduction in wall clock execution time.
>
> Skipping Test will of course reduce the time (Do you know how long the tests
> alone take?)
> but I wanted to know the exact "wasted" time you have mentioned by
> compile/resources?
> So running a linear (single thread) mvn clean install vs. mvn install (in
> both cases with skipping tests at all)...
>
> If you always do a clean than Maven (plugins) can't identify things which
> have already been done cause they are not there...so yes in this case you
> are wasting time...

Maybe I am wrong, but I think the unit tests account for 10-20% of the
time of the build in my example. When I run the exact same build with
tests and then without tests, without tests is faster by 10-20% of the
total time. Therefore I blame 10-20% of the time on tests and 80-90%
of the time for the remainder of the build that will be in common for
just about every run of the same project.

Running clean every time only accounts for a small amount of the
waste. I quickly did a test just now, on the same project I ran clean
install and then install by itself, there was only a savings of 3%
total time for omitting the clean phase. By 80-90% waste, I also meant
that before discrete separate jobs/pipeline stages I could just stack
all my goals into a single Maven invocation. The build would be very
long, but I wouldn't pay the Maven POM initialization and other costs
for general running multiple times.

Finally, since our pipeline / CD strategy uses a per-build version
with updated pom.xml for each release candidate branch, won't the
updated pom.xml timestamp force a rebuild of all objects anyway? I
think it ought to, even if Abc.java and Abc.class appear to be up to
date and in sync, there's a more new pom.xml timestamp which should
invalidate all previously built classes; assuming we are even lucky
enough to run on the same node as previous stage, which seems
increasingly unlikely given how many more steps we'll be running now,
occupying more and more executors.