Getting started with Pipeline
834 views
Skip to first unread message
Jerry Steele
Jun 10, 2016, 11:40:12 AM6/10/16
to Jenkins Users
Hello,
I'm looking into getting Jenkins to build feature branches for our github projects, but I'm not entirely sure where to start. Pipeline looks like it might fit the bill, but I'm having trouble getting my head round the Jenkinsfile. I've found the online docs and the "Groovy" generator but am not really sure how to tie it all together. If anyone has a bit oftime to help me, that would be great :)
We currently use our own build tool to test code as deployed to github, then build the artifacts into a debian package which is uploaded to Amazon S3 and deployed by hand later.
We currently have separate jobs for each of the major branches of our project:
subproject_a-qa
subproject_a-staging
subproject_a-production
subproject_b-qa
subproject_b-staging
subproject_b-production
subproject_c-qa
subproject_c-staging
subproject_c-production
The jobs are very simple - they poll github, looking at a specific branch, then if that has changed, they will execute a shell script which looks like this (generic):
sudo -Hs build_tool arg1 $GIT_URL <subproject_a> <environment(qa/staging/prod)>So, what I'd need is something that builds the following jobs when a feature branch is pushed to look something like:
sudo -Hs build_tool arg1 $GIT_URL <subproject_a> <feature_branch_name>
sudo -Hs build_tool arg1 $GIT_URL <subproject_b> <feature_branch_name>
sudo -Hs build_tool arg1 $GIT_URL <subproject_c> <feature_branch_name>Or else, know how to build those.
Is this possible with Pipeline? Or am I looking at the wrong tool here? I've started a multibranch test project, but am basically stuck at the Jenkinsfile stage, and most tutorials appear to refer to using mvn, which I'm not familiar with. the build tool is written in Python and is testing building for Ruby on Rails :)
Any help very much appreciated. Any more info needed, please let me know...
Thanks
Jerry
Thomas Zoratto
Jun 10, 2016, 12:34:38 PM6/10/16
to jenkins...@googlegroups.com
Hello,
I’m not sure to understand correctly but your needs seems quite straightforward.
To achieve that, you want to create a multibranch pipeline job (as you have already done I see) which will scan all your project’s branches looking for Jenkinsfiles.
In those Jenkinsfiles, you’ll have to describe your pipeline. Basically, you have to checkout the project and then execute your step. I suppose you can run your build tool with a "shell step" like sh 'build_tool <args>' if your build_tool is in the PATH.
This way you’ll have a Jenkinsfile per branch.
Then, if you wanna define a real pipeline (tests then deploy to some environment and then deploy again somewhere else for example) the documentation/tutorials will help you I think
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1414775a-307d-49fe-9119-2f77995385ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Craig Rodrigues
Jun 10, 2016, 3:31:18 PM6/10/16
to Jenkins Users
Hi,
You can try something like this to get started:def gitUrl = "https://github.com/twisted/twisted.git"
def gitBranch = "trunk"
node {
stage "Check out from Git"
git branch: "$gitBranch", url: "$gitUrl"
stage "Build code"
sh "sudo -Hs build_tool arg1 $gitUrl subproject_a $gitBranch"
}
def gitBranch = "trunk"
node {
stage "Check out from Git"
git branch: "$gitBranch", url: "$gitUrl"
stage "Build code"
sh "sudo -Hs build_tool arg1 $gitUrl subproject_a $gitBranch"
}
I would recommend going further. Make your Pipeline job parameterized. Add a parameter GIT_BRANCH,
and set the default value of that to the branch you want to build in that specific job.
def gitUrl = "https://github.com/twisted/twisted.git"
def gitBranch
if (getBinding().hasVariable("GIT_BRANCH")) {
gitBranch = GIT_BRANCH
}
node {
stage "Check out from Git"
git branch: "$gitBranch", url: "$gitUrl"
stage "Build code"
sh "sudo -Hs build_tool arg1 $gitUrl subproject_a $gitBranch"
}
You can add more build parameters as you need.
--
Craig
Jerry Steele
Jun 13, 2016, 6:32:14 AM6/13/16
to Jenkins Users, rod...@freebsd.org
Thanks for getting me started on this. If you don't mind helping me troubleshoot, I'll carry on:
I think the parameterized version is the one I need, as I would like the build tool to run with the $gitBranch argument of the new branch that has just been created. However, when I attempt to run this job, I get the following error:
This seems to suggest that the if statement is not possible for security reasons. Is the branch name exposed as an environment variable that I can grab in the Jenkinsfile? Or alternatively, can I change the security settings to allow the script to run?
Thanks!
Sverre Moe
Jun 13, 2016, 6:40:12 AM6/13/16
to Jenkins Users, rod...@freebsd.org
Pipelines with Jenkinsfile runs in a sandbox and thus you need to approve certain functions.
Manage Jenkins -> In-process Script Approval
Mark Waite
Jun 13, 2016, 7:00:22 AM6/13/16
to Jenkins Users, rod...@freebsd.org
For the multi-branch development work I've been doing, it has been better to avoid placing branch information inside the Jenkinsfile. The problem I had was that most of my branches are short-lived. They exist long enough to allow me to validate a pull request, but then they are merged to the master branch. If I place branch information inside the Jenkinsfile on that short-lived branch, then the branch information from the short-lived evaluation branch would be merged into the master branch when the proposed change is merged into the master branch.
Instead of placing branch information inside the Jenkinsfile, I think you want to use the "checkout scm" pipeline step like Liam Newman did in https://github.com/jenkinsci/git-plugin/blob/master/Jenkinsfile . That same Jenkinsfile exists on other branches in that repository, and each branch that has a Jenkinsfile is now evaluated from a multi-branch pipeline project (as in https://github.com/jenkinsci/git-plugin/blob/2.5.0-beta2/Jenkinsfile).
The same technique is being used in for many more branches in my evaluation fork of that plugin repository. Some examples include https://github.com/MarkEWaite/git-plugin/blob/3.0.0-beta/Jenkinsfile , https://github.com/MarkEWaite/git-plugin/blob/master-PR350-extendGitSCMSource/Jenkinsfile , https://github.com/MarkEWaite/git-plugin/blob/master-PR398-ioexception-from-submodule/Jenkinsfile and https://github.com/MarkEWaite/git-plugin/blob/master-PR411-parallel-tests/Jenkinsfile .
Thanks,
Mark Waite
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/649bffd5-727d-49a5-991b-67ee068dcb4a%40googlegroups.com.
Jerry Steele
Jun 13, 2016, 7:37:11 AM6/13/16
to Jenkins Users, rod...@freebsd.org
Thanks for the info re: In-process script approval, that worked :)
...which seems to be an issue with git itself.
@Mark, I'm pretty sure that I need to glean info about the branch from the current build, because this has to be passed as an argument to my build tool. I was hoping that this could be done via an environment variable, but this doesn't seem to be possible.
I tried the method here, but confusingly, that threw an error like this:
[multibranch test] Running shell script + git rev-parse --abbrev-ref HEAD fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]'
...
...which seems to be an issue with git itself.
Is there a tried and tested way of getting environment variables like this into the build process?
Thanks
Jerry
Mark Waite
Jun 13, 2016, 7:42:58 AM6/13/16
to Jenkins Users, rod...@freebsd.org
By default, the git plugin checks out a "detached head" rather than checking out to a named local branch. If you add the "Additional Behaviours" to do a "Checkout to specific local branch" and assign it the value "**", then the plugin will attempt to checkout to a local branch which matches the branch name from the source repository.
That capability was added in git plugin 2.4.3 as the implementation of https://issues.jenkins-ci.org/browse/JENKINS-33202 .
Mark Waite
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e9f89c01-4539-43d5-be3a-1fc2d6923707%40googlegroups.com.
Jerry Steele
Jun 13, 2016, 9:32:06 AM6/13/16
to jenkins...@googlegroups.com, rod...@freebsd.org
That makes absolute sense, but I just tried that and got the same error..
--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/ZrkPJGo4x18/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFwLz0ji%2BEwzuTPE0K%2BiaydzMNB3X%3DrpAMhp3cpFZof-A%40mail.gmail.com.
Jerry Steele
Jun 14, 2016, 10:52:36 AM6/14/16
to Jenkins Users, rod...@freebsd.org
Hello again. So, my Jenkinsfile now looks like this, and I'm having some success:
Obviously, the variables are not being substituted. I've also tried "$git_url", "git_url" and "${git_url}" with similar results. Does anyone see where I might be going wrong?
def git_url = "g...@github.com:myorg/my_project.git"
def git_branch
node {
stage "Check out from git"
checkout scm
stage "Make sure our git branch var is accessible"
sh 'git rev-parse --abbrev-ref HEAD > GIT_BRANCH'
git_branch = readFile('GIT_BRANCH').trim()
echo git_branch
stage 'Use build_tool to build the branch'
sh 'sudo -Hs build_tool dist '${env.git_url}' hark_web '{env.git_branch}''
}However, the variable substitution does not appear to work, and the job fails at the build_tool stage with:
+ sudo -Hs build_tool dist usage: build_tool usage....
Obviously, the variables are not being substituted. I've also tried "$git_url", "git_url" and "${git_url}" with similar results. Does anyone see where I might be going wrong?
Thanks
Slide
Jun 14, 2016, 11:00:23 AM6/14/16
to jenkins...@googlegroups.com
In groovy, variable substitution is only done in double quoted strings, not single quoted strings. Try replacing the single quotes with double quotes.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ae4572c3-1aa5-47c6-91b7-c29fe7ee5ee0%40googlegroups.com.
Jerry Steele
Jun 14, 2016, 11:17:08 AM6/14/16
to Jenkins Users
Brilliant! That worked. I'm onto the next error now, but the pipeline script appears to be doing what I want it to :) For reference, the finished JEnkinsfile looks like this:
def git_url = "g...@github.com:myorg/my_project.git""
def git_branch
node {
stage "Check out from git"
checkout scm
stage "Make sure our git branch var is accessible"
sh 'git rev-parse --abbrev-ref HEAD > GIT_BRANCH'
git_branch = readFile('GIT_BRANCH').trim()
echo git_branch
stage 'Use build_tool to build the branch'
sh "sudo -Hs build_tool dist ${git_url} my_project ${git_branch}"
}Thanks very much :)
Reply all
Reply to author
Forward
0 new messages