Pass SVN revision number to a job dynamically

[Jason Voegele]

Hi,

We have a fairly complicated Jenkins set up, but after having attended the Jenkins User Conference I know that there are others out there with Jenkins implementations even more complex than ours and I'm hoping someone on this list can help out with a problem we are having.

We have dozens of separate projects, each of which has many different Jenkins jobs associated with it. The basic structure is, for each project we (typically) have:

* One "publisher" job that deploys Maven artifacts to our Maven snapshots repository. This job is a Maven project job in Jenkins, and in addition to deploying the Maven artifacts it also runs unit tests and records the Subversion revision number that was used in a properties file on a shared drive.
* Dozens of "test" jobs that execute long running integration tests (up to six hours per job). These test jobs are Freestyle projects (not Maven projects) and are therefore not considered to be downstream from the publisher jobs, even though conceptually that's exactly what they are.

We are trying to make sure that the test jobs checkout from Subversion using the "blessed" revision number from the publisher job. In effect, we want the Subversion URL for these test jobs to be of the form http://path.to.svn/project/trunk@${BLESSED_REVISION} , where BLESSED_REVISION is the property generated by the publisher job. We've tried several approaches to doing this, all of which (so far) have had their shortcomings.

1. Initially, we were using the Setenv Plugin to read the BLESSED_REVISION from the properties file on the shared drive. This has the benefit that the properties file is read from the master node (not the slave), which means that we don't need visibility to the shared drive from each slave. However, the main problem is that this plugin runs after Subversion checkout, and is too late to effect the revision used for checkout. We had it doing an svn update after the fact instead, but this means that the revision numbers reported on the build status page are not accurate.

2. Next, we tried using the EnvInject plugin which does have the ability to run before svn checkout. This *almost* works for us, but there were two problems: (1) the EnvInject plugin runs on the slave instead of the master, and accessibility to the shared drive is not available from every slave, and (2) it only works with JDK 1.6 and therefore our test jobs that use JDK 1.5 cannot use this plugin (fatal error).

3. Finally, we tried the pre-scm-buildstep plugin. Actually, we tried using this plugin in two separate ways. First, I tried to create a pre-scm build step that uses the EnvInject plugin described above, but ran into the same limitations as using EnvInject directly. Second, I tried to create a pre-scm build step that uses the Groovy plugin in the hopes that I could use Groovy to read the properties file and manipulate the Subversion URL and/or revision number from a Groovy script. Alas, this did not work because the pre-scm-buildstep plugin apparently cannot execute Groovy build steps. When I even tried to configure it for a job, it would crash with a fatal error.

Since all of the above approaches have failed us for one reason or another, can anyone offer any other ideas or approaches that we could try? This seems like it might be a common use case so I'm hoping someone else has already figured out a good solution.

Thanks!

--
Jason Voegele
If all the world's economists were laid end to end, we wouldn't reach a
conclusion.
-- William Baumol

[Nord, James]

I am sure you have your reasons - but why are you using svn to check out the sources that have already been built by the "publisher" job"?

Why not just pull the built code from Jenkins (copy artifacts plugin or some other script) and trigger the job on a successful build of the Maven job?

/James

Hi,

Thanks!

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

[Jason Voegele]

On Nov 2, 2011, at 1:20 PM, Nord, James wrote:
> I am sure you have your reasons - but why are you using svn to check out the sources that have already been built by the "publisher" job"?
>
> Why not just pull the built code from Jenkins (copy artifacts plugin or some other script) and trigger the job on a successful build of the Maven job?

Hi James, thanks for the feedback. What you mention is an interesting and even preferred approach, but I'm not sure it would work in our circumstances. The reason is that our publisher jobs and test jobs, even though they are separate jobs in Jenkins, are all based on a single Maven project. For example, we might have something like this:

project
├── moduleA
├── moduleB
├── moduleC
└── system-test

Each of moduleA, moduleB, moduleC might have its own set of integration tests, and the system-tests module might have some additional system tests. (It's not always this regular. We have dozens of projects, some of which follow this pattern, some use more modules, some fewer.) Right now, our Jenkins publisher jobs just execute "mvn deploy" to put the artifacts into our snapshots repository, then the test jobs typically do something like "mvn -Psystem-tests verify". To do as you are suggesting, we would have to find some way to run "mvn -Psystem-tests verify" without recompiling all the modules, effectively skipping the compile phase and forcing Maven to pull artifacts from the repository instead. I don't know of a way to do that with Maven, or whether it's even possible.

Thoughts?

--
Jason Voegele
Language is a virus from another planet.
-- William Burroughs

[Neil Bird]

Around about 02/11/11 18:32, Jason Voegele typed ...

> To do as you are suggesting, we would
> have to find some way to run "mvn -Psystem-tests verify" without
> recompiling all the modules, effectively skipping the compile phase and
> forcing Maven to pull artifacts from the repository instead. I don't know
> of a way to do that with Maven, or whether it's even possible.

We actually run our test jobs in the build job's workspace (manually
setting the workspace location for the test jobs). So when the build
succeeds, the tests are run on that very build.

The downside to this is that both build and test jobs have to be tied to
a particular box (to ensure that the directory's available). But we have to
have similar ties to allow for on-commit incremental build jobs to work
properly (otherwise any given chose box will have an arbitrary number of
revisions to update to in order to start the build).

--
[neil@fnx ~]# rm -f .signature
[neil@fnx ~]# ls -l .signature
ls: .signature: No such file or directory
[neil@fnx ~]# exit

[Jason Voegele]

On Nov 4, 2011, at 9:27 AM, Neil Bird wrote:
> We actually run our test jobs in the build job's workspace (manually setting the workspace location for the test jobs). So when the build succeeds, the tests are run on that very build.
>
> The downside to this is that both build and test jobs have to be tied to a particular box (to ensure that the directory's available). But we have to have similar ties to allow for on-commit incremental build jobs to work properly (otherwise any given chose box will have an arbitrary number of revisions to update to in order to start the build).

Yeah, unfortunately the downside you mention makes it unusable in our circumstances. We have dozens of different test jobs per project, and some of them are tied to specific operating systems, etc. so we can't run them all on the same box.

Thanks anyway.

--
Jason Voegele
Sauron is alive in Argentina!

[danny staple]

Could you start the test jobs as remote builds using a url and passing
the build number as a parameter that way?
We do something like this, but we also have intermediate "kicker" or
"scheduling" jobs os the builds only need to poke those and they then
schedule/kick all the downstream jobs.

--
Danny Staple

Director, ODM Solutions Ltd
w: http://www.odmsolutions.co.uk
Blog: http://orionrobots.co.uk/blog1-Danny-Staple

[Sami Tikka]

Could you use Parameterized trigger plugin to pass the subversion revision to test jobs?

-- Sami

[Jason Voegele]

Thanks for the suggestions, everyone. I eventually resolved this problem by contacting the author of the EnvInject plugin, and he graciously made some fixes to the plugin to allow us to use it for this purpose in our environment.

--
Jason Voegele
If you lose your temper at a newspaper columnist, he'll get rich,
or famous or both.

[dimasla]

Hi Jason.

Please share more details.

Are fixes in plugin already released? In what version of plugin?
What is a brief of made fixes?
And how these changes helped to achieve your needs?

Thanks for advance.