Where to place build/package/publish logic

[Adam Rosien]

I'm using Jenkins for CI and I'm debating where to place various logic (as scripts, etc.) for building, packaging, versioning and publishing. 

For example, in a Java/Scala maven/sbt world, you can perform all of these tasks with maven/sbt itself using the project descriptors (.pom, build.sbt, etc.) and various plugins (assembly to make a "fat" jar, etc.). An important feature of this style is that the entire pipeline of build/package/version/publish is self-contained in the source repo itself, and Jenkins really only needs to run a single command to perform all these steps.

For other kinds of projects there aren't necessarily "one-stop-shop" solutions for this pipeline. One constant debate I have with folks is that "source code shouldn't know where the artifact repository is", or "a project shouldn't manage its versioning mechanism", etc. 

One could imagine giving Jenkins the responsibility for any of these steps. For example, the project itself is responsible for supplying the build and test machinery, but Jenkins builds the artifact and uploads it to some artifact repository. The downside to this approach is that now there are a bunch of dependencies between the project and Jenkins in order to complete the whole pipeline, and the extra dependencies magnify the downside when there are problems, i.e., somebody breaks the script that figures out the new version number of a project and all projects can't get a new version.

Can you all help me find good principles and structures for assigning the responsibilities of build/package/version/publish?

Thanks,

.. Adam

 

[Anthony Shortland]

Hi Adam,

The design pattern we promote is the loosely-coupled toolchain approach to Integrating DevOps tools into a Service Delivery Platform.

This is exemplified by integrating tools around Jenkins (build console) and Rundeck (deploy console).

As for modular automation ... you should pick your poison for an implementation language (Python, Ruby, Chef, Puppet, etc, etc) but if you like good old Bash scripts, we're putting a lot of effort into Rerun just now ;-)

Anthony.

[Adam Rosien]

So are you implying that the source repos shouldn't know how to build/test/package/publish themselves?

[mithun]

I guess the correct answer to your questions is "It depends ..." (pun unintended).

Typically, the source repo _should_ know how to build, test and package - just because every source repo can have different requirements to do so. However, the source repo should also rely on the "process" (Jenkins, wrapper scripts, etc ...) for assistance on the environment. For e.g, Is this a development vs. production build? What is the test-passing threshold? Which database is the application talking to? etc .

The source repo should also be responsible for it's own versioning scheme. In the Java/maven world, this is automatically taken care of (via snapshot/release builds). From my experience, it is difficult to dictate a consistent versioning scheme across different source repos.

The source repo should not care about it's "publishing". The "process" should decide where an artifact gets published. The process should also help the source repo with the artifact repository so that dependencies can be resolved during the build/test/package stage.

> The downside to this approach is that now there are a bunch of dependencies between the project and Jenkins in order to complete the whole pipeline

Whether this is a downside or not is debatable. Are you trying to establish a "standard" pipeline that multiple source repos can use? Or trying to setup a Jenkins job for one source repo? If it is the later, then yes, its a downside with maintenance overhead. If it is the former, then I believe this is an advantage.

-- Mithun

[Adam Rosien]

Thanks for the reply. Having CI/CD/Jenkins/"process" inform the source repo of the environment is a great way to think about the whole system.

For my particular use case I have a deployment-manager service which has a maven repo upstream, but I'm trying to wedge non-maven project artifacts into the process, which means adopting the maven world of versioning and repositories (the two are closely coupled, versions map to URL paths) to project types that aren't traditionally "maven-ish", that is, language X packaged as a tarball, etc. This may or may not be a good idea.

Perhaps it may be better to adapt the deployment-manager to other versioning/repo schemes instead of requiring things in the maven way (which I find mostly sane).

.. Adam

[Anthony Shortland]

We've done a lot of work pushing system packages (like RPMs) into Maven repositories and have found it possible to reconcile Maven versioning conventions with those of the system. e.g. The RPM Maven plug-in is a thing of beauty and a joy forever allowing you to adopt or override conventional versioning strategies (the nexus-yum-plugin is another key point of integration, by the way).

The key is to differentiate between planning dependency resolution for build artifacts vs deployable artifacts, in my opinion. 

That said, does anyone have an opinion about reconciling the Maven snapshot/release repository convention with the needs of continuous deployment? I'm facing that challenge right now.

Anthony.

[Jez Humble]

The answer is, essentially, don't use snapshots. Maven snapshots and continuous delivery are fundamentally antithetical. I had this out with the Maven community a while ago. You can read a summary here: http://www.lucasward.net/2010/11/maven-and-continuous-delivery.html TLDR: one of the fundamental premises of continuous delivery is that every build is a potential release candidate.

There are a couple of articles describing how to do it right here:

http://blog.orfjackal.net/2012/08/continuous-delivery-with-maven-and-go.html

http://jamesbetteley.wordpress.com/2012/02/21/continuous-delivery-using-maven/

Thanks,

Jez.

--
Jez Humble

Co-author, Continuous Delivery

http://continuousdelivery.com/

http://jezhumble.net/

[Adam Rosien]

Didn't know about the RPM Maven plugin. But I'm in Scala/sbt land :(.  Reminds me of FPM but with a less-cool name.

[Adam Jacob]

Not that you need anyone but Jez to answer, but i came to the exact conclusion on a recent project with Maven. No snapshots, embed a universally unique build id into the version (or replace the version entirely), and call it a day. Basically skip the maven release plugin.

Adam

Sent from my iPhone