Multi-branch pipeline project with Subversion

[Pieter-Jan Busschaert]

Hello,


We have a debian Jenkins server which runs on the latest jenkins PPA (currently 2.6). I'm interested to test the Multi-branch pipeline functionality, in combination with our SVN server. However, all documentation I find about the topic talks about git. The concept of branches etc in SVN is significantly different than in GIT. Can someone here please answer these questions, based on our typical SVN folder structure (see below):

1. Where should I store the Jenkinsfile? Is there 1 file for all branches or should there be 1 file in each branch?
2. Which URL should I configure in the job page as the "project repository base"?
3. What patterns should I put in the include/exclude branches boxes?
4. Currently we create 2 types of builds for each project : 1 incremental build which runs after each commit, 1 clean build which runs at night. How would we do something like that ? Would that need 2 Jenkinsfiles per project ?
   

Background info: our typical SVN layout:

repo_url

-- trunk

---- projectA

---- projectB

---- shared

-- branches

---- branch_name_1

------ projectA

------ shared

---- branch_name_2

------ projectB

------ shared

The projectA, projectB folders in the above structure contain some svn:externals to the shared folder, so I have to define additional credentials (which are actually identical as the ones for the main URL) in the job config SVN section (don't know if this is relevant for my above questions).

Based on the limited information available, I would currently guess the answers to my question above to be something like this:

1. put the jenkins file as repo_url/trunk/projectA/Jenkinsfile (with automatic copies each time I make a branch like: repo_url/branches/branch_name_1/projectA/Jenkinsfile)
2. configure the repository base as repo_url
3. include pattern : trunk/projectA, branches/*/projectA
4. No idea how to do this

Am I close with these guesses? Is there any place where this is documented / explained better that I didn't find?

Kind regards,

Pieter-Jan

[Pieter-Jan Busschaert]

Hello,

I have made some progress on this (much slower than I would like) and would like to document some answers to my earlier questions + add some new questions / problems I've run into. Hopefully somebody here can help me out?


On Friday, 27 May 2016 17:17:21 UTC+2, Pieter-Jan Busschaert wrote:

Can someone here please answer these questions:

• Where should I store the Jenkinsfile? Is there 1 file for all branches or should there be 1 file in each branch?

Putting Jenkinsfile into trunk/projectA seems to work OK.

• Which URL should I configure in the job page as the "project repository base"?

Using the repository root seems to work OK. Also, the credentials for the svn:externals also don't need to be repeated as "extra credentials" with pipeline, which I like.

• What patterns should I put in the include/exclude branches boxes?

trunk/projectA,branches/*/projectA seems to work OK.

• Currently we create 2 types of builds for each project : 1 incremental build which runs after each commit (build + test), 1 clean build which runs at night (clean + build + test + create installers). How would we do something like that ? Would that need 2 Jenkinsfiles per project ?
  

No solution for this yet. As there is clearly only 1 Jenkinsfile per branch (I didn't find any config option to specify a different filename), probably I'd have to make my Jenkinsfile parameterized and then create multiple multi-branch projects, one with the "incremental" parameter set to true, one with that parameter set to false. Is this a reasonable approach or is there a better one? I have seen a few similar questions to this list the last few days, but the replies are mostly from people who want clarifications on why you would want/need multiple Jenkins jobs for a single source folder. I hope my usecase is clear?


Now, some documentation about what I tried initially after reading the tutorial and had to change because it didn't work. This might be useful as new-user input for any update to documentation / tutorial.

I started by just putting the current "execute windows bat script" contents in actual .bat files and created a Jenkinsfile based on the original build steps (just put them in the same order):

node {
  stage 'Checkout'
  checkout scm
  stage 'Build'
  bat 'call jenkins_build.bat'
  stage 'Test'
  bat 'call jenkins_test.bat'
  stage 'Collect testreports'
  step ([ $class: 'XUnitPublisher', ... ])  // generated by "Pipeline syntax" utility
  stage 'Email notifications'
  step ([ $class: 'Mailer', ...]) // generated by "Pipeline syntax" utility
}

Now, what didn't work in the above and I was able to solve:

1. %WORKSPACE% is not available for pipeline plugins, so I had to use the workaround described in https://issues.jenkins-ci.org/browse/JENKINS-33511
2. However, the resulting %WORKSPACE% variable doesn't really work as-is, probably because there is a newline character at the end. Doing something like "set WORKSPACE2=%WORKSPACE%" fixes (probably removing the newline).
   
3. No timestamps in the console log, so wrap everything in a big timestamps{}  block.
4. I was not receiving any email notifications for failures, because the pipeline just stops.
   
1. Putting everything in a try{} block and the email notification stage in a finally{} block did not solve it.
2. I had to put a catch{} block in between where I explicitly set currentBuild.result = "FAILURE"

So, my current Jenkinsfile is something along these lines:

node {
  timestamps {
    try {
      stage 'Checkout'
      checkout scm

      // %WORKSPACE% workaround, see https://issues.jenkins-ci.org/browse/JENKINS-33511
      bat 'cd > workspace.txt'
      env.WORKSPACE = readFile('workspace.txt')
      bat 'del workspace.txt'

      stage 'Build'
      bat 'call jenkins_build.bat'
      stage 'Test'
      bat 'call jenkins_test.bat'
      stage 'Collect testreports'
      step ([ $class: 'XUnitPublisher', ... ])  // generated by "Pipeline syntax" utility
    } catch (caughtError) {
      err = caughtError               // not sure if this line is really needed ?
      currentBuild.result = "FAILURE"
    } finally {
      stage 'Email notifications'
      step ([ $class: 'Mailer', ...]) // generated by "Pipeline syntax" utility
    }
  }
}

However, my project doesn't build yet. It appears Microsoft Visual Studio 2012 has an issue with running inside folders with a "%" in the name. The workspace folder is named according to a sanitized branch location. In my case this is "trunk%2FprojectA". This is a valid foldername on windows, but certainly not common and apparently not supported by VS2012 command line tools. When specifying an include directory with /I"libraryX\include", I get these errors while building:

cl : Command line error D8038: invalid argument 'C:\jenkins\workspace\pipeline_test\trunk%2FprojectA\libraryX\include'

After some debugging it seems to be caused by the % sign. If I rename the folder on my own build pc to have a % in the name, I get the same error. I did not find a way to specify or alter the way a sanitized workspace location is generated. However, I did find a similar issue/request reported here: https://issues.jenkins-ci.org/browse/JENKINS-34564  and while looking that one back up, I found https://issues.jenkins-ci.org/browse/JENKINS-30744 which actually contains a workaround. I'll try this workaround right now.


Anyway, in summary, I must say it seems to me the relevant documentation on how to convert normal jobs to pipeline jobs is quite scattered around (tutorial.md, blogs, mailinglist, issue tracker, ...). It seems flexible enough, but I don't like putting in all the workarounds, because the Jenkinsfile is supposed to be simple, not containing a lot of logic.