How to set artifact version in a Go pipeline from a task/script?

[Pat Podenski]

I am establishing an artifact version scheme for a Maven project pipeline wherein the artifact_version is composed of the Maven 3-part version number plus a git commit label as follows:

artifact_version = [main_version]-[git_hash]     (e.g., 1.2.3-cab5734)

WHERE:

main_version == [major].[minor].[incremental] (per Maven conventions)

git_hash    == first 7 characters in corresponding git SHA-1 commit label

Some details:

• main_version (1.2.3) is being parsed during the build from the Maven pom.xml project/version element via a task/script that I have written
  
• git_hash is extracted from the full SHA-1 label (e.g., bash sub-string expression: ${GO_REVISION:0:7}
  

It's straightforward to compose the artifact_version as described above, but how do I set this artifact_version in the Go pipeline?

So far I looked at using a property which can be set via a POST with the REST API for properties:

For example, curl -u user:passwd -d "value=1.2.3-cab5734" http://localhost:8153/go/properties/my-pipeline/46/my-stage/1/my-job/artifact_version

This will set the property 'artifact_version', however this is only being set for the job in the specified stage instance.

How would I then be able to use the 'my-version' property throughout a Go pipeline instance? Won't the property only be visible in the job?

Is there another way to do this so that the artifact_version becomes available throughout the Go pipeline instance when set from my task/script?

Or is there some way to use environment variables to do this from a task/script?

[pshi...@thoughtworks.com]

Hi,

You can retrieve property set for a job using API http://[server]:8153/go/properties/[pipeline]/[pipelinecounter]/[stage]/[stagecounter]/[job]/[propertyname]. Please refer http://www.thoughtworks.com/products/docs/go/current/help/Properties_API.html for more details. Tricky part here is getting pipeline counter and stage counter right. You can use 'GO_PIPELINE_COUNTER' and 'GO_STAGE_COUNTER' environment variables set by Go, but this approach might not work very well with stage re-run. 

Our suggested approach would to publish artifact_version into file. The job which generates artifact_version can generate property file with artifact_version and publish it as artifact. Later same can be fetched from other jobs using fetch artifact task.

Hope that helps.

Thanks

Praveen S

[Rustin Daniels]

hi Pat,

Here's my 2 cents..

Forgive me, my main development platform is windows and .net and have come accustomed to using Go Artifacts repository to manage upstream and downstream dependencies and as you might know this allows the for a dependency tree to be built and published all the way down even a large complex graph and always be in sync.

I would give a scenario but it would end up being as long as a blog post.

My question is why even use maven on the server? In my mind it feels like a bit of a hack.

Instead my suggestion would be to rethink your design of your build pipeline and sandbox taxonomy to fit in with Go's powerful dependency management features.

Having built 100's of components myself and having Go orchestrate the entire build and dependency management among them is one of its best features.

Cheers

Rustin

[Pat Podenski]

Thanks for the tip.  :-)

I published an artifact_version.txt file with the version (e.g., 1.2.3-cab5734) as its contents.

Once the artifact has been published by Go, I can then access each instance of the artifact_version.txt file via the REST API for artifacts. When I want to retrieve the artifact_version for the current pipeline instance, I can use the REST API which will return the contents of the artifact_version.txt file:

curl http://localhost:8153/go/files/my-pipeline/${GO_PIPELINE_COUNTER}/my-stage/latest/my-job/artifact_version.txt

While this certainly does work, it seems to me that Go should simply have an environment variable or parameter (e.g., PIPELINE_VERSION) that can be set either statically in the Go configuration UI, or dynamically during the initial commit build via scripting. The concept of artifact_version is quite important to the whole business of continuous delivery. Go does take care of the revision using GO_REVISION, but has no support for the version that's found in the pom.xml. 

[Pat Podenski]

I'm sure that Go could handle the whole build including dependencies. However the Maven world possesses a very rich set of capabilities around dependency management and when one's projects have all been built using Maven it's not going away.

Now that Go is 'free' to compete with OSS tools such as Jenkins we'll hopefully see it getting used a great deal more. While Go was designed to comprehensively deal with both CI and Release/Deployment, in some ways Jenkins has been much more vetted, extended and exercised simply due to its OSS nature.

Jenkins definitely seems like it's being pushed beyond its limits to do more than CI, however the Jenkins creator just published an article pointing out how to do exactly that:

  http://www.infoq.com/articles/orch-pipelines-jenkins

[Nathan Fisher]

Hello Pat,

Unfortunately there is no way to specify the revision within the build that I'm aware of but, I think that's generally a good thing**.  Below is an outline of how I would approach doing a Continuous Delivery (CD) pipeline using maven. It assumes you subscribe to "build once, reuse everywhere" and are open to ignoring mavens "best practises" regarding revisions tracked in SCM.

1. Keep the pom.xml as 0.0.0.0-SNAPSHOT in source control (quick way to identify dev from release candidates).

2. Prefer clean checkouts/workspaces (assuming it doesn't impose too much of an impact on build time).

3. For the Go label use a manual identity for major and minor, Go's count for revision, SCM revision for a sub-revision (unfortunately I don't think Go supports git short revs yet).

4. First step in the build run;

mvn --batch-mode release:update-versions -DautoVersionSubmodules=true -DdevelopmentVersion=${GO_REVISION}-SNAPSHOT

5. Run the maven package target.

6. Consume package as an artefact.

There's likely a better way to generate an artefact that *isn't* a snapshot but, the above should be acceptable for a service in my opinion. Keep in mind that if everything is a release candidate the idea of "tagging" a release becomes outmoded. For libraries you'd likely want to do an actual release as the first step which is slightly more tedious due to SCM integration.  The primary benefit for the above is reduced SCM churn/noise in your pom files related to version bumping, less effort related to releases (its done automatically as part of the commit) and greater visibility of the release process/progress via Go.

The following are some of the tenants of CD which Go either depends on for the greatest benefit or aides with. 

a) your trunk is (almost) always in a "releasable" state,

b) everything reasonable is automated (inc./esp. this component of the release process) and

c) the process is ideally repeatable and reliable (aka idempotent).

The following problems come to mind with the "out-of-band increment" your suggesting whereby the version is extracted from a POM file or by other means beyond Go's management;

1) it requires that someone ensure the revision is updated with some regularity (sometimes tedious/error-prone w/ multi-module maven projects and is also a *minor* communication overhead) and

2) it imposes a need to ensure a unique identity is generated for every release candidate.

To support the above statements consider;

i) What happens if the revision in the pom.xml or the script that generates the artefacts UID fails (hash collision or some other unexpected edge case)?

ii) Where does the code for UID exist and who maintains ownership of it? Is it a project of its own, packaged with each project, exist as a sub-module, etc?

iii) How does a label collision impact an operators understanding (e.g. QA or Operations) of available deployments for environments they own and deploy to? If r1.3.4 is created twice there is an exceptional case where a developer needs to convey not just the build label but, also the date/time it was generated. Each additional variable increases complexity and potential for error during hand-offs.

iv) What happens when the same source is resubmitted for a build with a different tool-chain, flags, etc?  The label ceases to be a unique identifier for a single binary but, could represent multiple. Ideally any changes to the compilation and resulting binary would be captured in it's label.

v) what happens if you're releasing n times a day where n could be as often as every 15-20mins? How would you co-ordinate and communicate release versions?

In conclusion by allowing Go to manage the "identity" of an artefact it encourages a uniqueness that is unlikely to have a collision through neither human error or malicious intent. Furthermore, communication and traceability becomes simplified as the build identity is basically synonymous with a single binary rather than a manually carved "release" where multiple *attempts* maybe required.

** I admit there are some organisational processes where it would be desirable to consume artefacts and/or revisions outside of Go's management. However, in my experience the organisations with such processes in place are often a long way off from automated CD.

Kind Regards,

Nathan

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pat Podenski]

Obtaining the SCM revision is quite simple. Go publishes an environment variable that contains the git commit label (SHA-1). Picking off the short form is quite simple as well. The bash script shown below is used to form a version number consisting of the three numerical parts and the short revision.

[1] The first line uses mvn evaluation to obtain project.version from the pom.xml and extracts the numerical part.

[2] The second line combines the main_version (e.g., 1.2.3) with the short git revision label (e.g., cab5734)

artifact_version (e.g., 1.2.3-cab5734) is the version used to identify a unique build in the deployment pipeline.

The bash script:

#!/bin/bash

# sets the artifact_version

main_version=`mvn help:evaluate -Dexpression=project.version | grep -v '\[' | grep -o '[0-9]*\.[0-9]*\.[0-9]*'`

artifact_version="$main_version-${GO_REVISION:0:7}"

I set the version in the Maven project using the versions plugin:

mvn -Psit versions:set -DnewVersion=#{version} -DgenerateBackupPoms=true

At the end of the commit build, the changes to the pom.xml are rolled back so that the version is SCM is of the form 1.2.3-LOCAL.

I completely avoid accidental rebuilds by using Nexus which guards against a second build of an artifact using the same version.

I have blogged about a similar approach previously for Jenkins and Subversion:  https://techinations.wordpress.com/author/stug23/ 

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.

[Nathan Fisher]

Hi Pat,

Apologies I feel like I’m trolling but, I think I might be missing the appropriate context. Regarding Nexus as a guard if I understand correctly is step 4 below the best opportunity to fail if we want it fast?

1) build

2) test

3) package

4) fail at publish

To me it seems that either the build was either unnecessary (so why was it run) or was desired and will leave someone frustrated that it failed after the time required to complete steps 1-3. Sure you could add a step before 1 to see if the revision exists in Nexus but, it’s just another element to break/maintain when there’s likely so many other things that could be focused on.

Kind Regards,

-- 

Nathan Fisher

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

[Pat Podenski]

Nathan,

In the development context that I operate in, we have a separate automation in place to bump up the main_version when that's required. The likelihood of failing due to an artifact having been already deployed into Nexus is very slight because we don't provide a manual trigger (this is currently is implemented in a Jenkins pipeline) to run the commit build - this build is triggered by changes to the project source. In that event, the SCM revision will have changed which forms a different version of the artifact. So there aren't any conflicts in that scenario.

[Pat Podenski]

I have slightly modified the script mentioned earlier in order to accommodate a short git commit hash which can sometimes be ambiguous with only 7 characters:

#!/bin/bash

# sets the artifact_version

main_version=`mvn help:evaluate -Dexpression=project.version | grep -v '\[' | grep -o '[0-9]*\.[0-9]*\.[0-9]*'`

echo "main_version from pom.xml: $main_version"

# Obtain a short git commit hash: http://www.quora.com/How-do-you-get-the-short-commit-hash

short_hash=`git rev-parse --short $GO_REVISION`

echo "short git commit hash: $short_hash"

artifact_version="$main_version-$short_hash"

[Ronald Pomeroy]

Hi Pat,

I'm just starting with Go also.  Did you ever figure out an elegant solution - one you would promote (has to work on Windows also unfortunately) ?

Regards,

Ron

On Tuesday, April 1, 2014 11:00:21 AM UTC-7, Pat Podenski wrote:

[Matt Urbanski]

You've probably figured this out already, but substrings in windows are accomplished using the following syntax %GO_REVISION:~0,8%.zip. That gets the first 8 characters of the GO_REVISION env var.