Fastest way to git tag in a Jenkinsfile?

[Idan Adar]

Checkouts in Jenkins, AFAIK, are headless, meaning that I cannot do git operations on them.

If I want to do an operation, like git tag and then git push... I need to first do git clone inside an sshagent block or similar.

For large repositories this can be a problem, as the process becomes lengthy...

checkout scm...

do stuff (tests, build, etc)...

git clone...

git tag...

git push...

This can be made a little better by git cloning using --depth 1, but this still will git clone the whole repo, even if without history.

So I'm looking for a way to be able to git tag/push, but not also git clone the repo again... is this at all possible, to maybe check scm with head (or, not headless) so git operations would be possible?

[Stephen Connolly]

You will be wanting my changes that should make a beta early next week

Then you add the local branch trait and that does exactly what you want

HTH 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/52f875bc-020c-422f-8508-2277a628f608%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Idan Adar]

I'm sorry, but could you please elaborate? :)

What changes to which plug-in and an example implementation...

Right now instead of "checkout scm" in the checkout stage and "git clone" in the stage where I need to do git tag/push, I do only the following in the checkout stage:

sshagent (credentials: ['...']) {
    sh "git clone --depth 1 -b develop g...@myrepo.git"
}

And then I should be able to do the operations...

Essentially, Jenkinsfile now does the "checkout" not-headlessly, but it does meant it's not declarative anymore...

[Idan Adar]

I take it back, not good workaround for me after all...

[Mark Waite]

.You're correct that the git push needs to be wrapped in some form of credentials.

Git checkouts in Jenkins can be done to the branch name of your choice, including "match the branch name".  Refer to the "Additional Behaviour" called "Checkout to specific local branch".  The online help for that option says that you can use "**" as the value and that will be interpreted to use the branch name which was selected for checkout.  An example is available in https://github.com/MarkEWaite/jenkins-bugs/blob/JENKINS-33827/Jenkinsfile .  The "pipeline syntax" for checkout will add that behavior for you as well (though as Stephen notes, the UI is not pretty and shows you many things as options which you should not use).

Sorry that is not the default, but preserving historical behavior has been important for the many users of the plugin.

Mark Waite

On Sat, Jun 17, 2017 at 1:09 AM Idan Adar <id...@adar.me> wrote:

I take it back, not good workaround for me after all...

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/954f8f8f-9cc3-4085-ba95-d3a241dc8e8a%40googlegroups.com.

[Idan Adar]

Thanks Mark. Since I'm using declarative pipeline in a Jenkinsfile in a multi-branch job, UI option is kinda scarce and less relevant for me. So anything that can be added to the declarative way of doing things is great. The example you linked to is in scripted pipeline form, which I'm trying to get away from where ever I can.

That said, I'm not versed enough in Git to understand these concepts mentioned... "checkout to specific local branch" doesn't mean much to me... I understand "checkout" but "_to_ a specific branch"? Not sure what it means.

What I understand is that the "checkout scm" command does a headless checking out of a given branch of a repository, which takes place thanks to a github webhook with "pull request" and "push", pointing to the Jenkins instance. So basically what I want, and hopefully is what you and Stephen are referring to soon be available, is that the "checkout scm" command will be such that won't be headless?

I'd be great if I could specify for it "--depth 1" and be able to do git operations on the checked out repo/branch. :)

[Idan Adar]

Oh, hm, interesting...

In my multibranch job I selected "GitHub" as the "Branch source"... for "GitHub" there aren't many options... but for "Git" there are... and I guess these play fine with Jenkinsfile...

Question.

Lets say I setup "Git" as the branch source and add some "additional options", but in my Jenkinsfile I set:

options {
      skipDefaultCheckout()
}


stages {
      stage ("Checkout SCM") {
          checkout scm
      }
}

Does this contradict one another?

[Idan Adar]

Another question, if I do customize "checkout", do I need to include everything like in your example, or only what I want specifically, like depth, and if nothing else is mentioned that the defaults are taken...?

[Mark Waite]

Defaults are taken when values are not provided.

On Sat, Jun 17, 2017 at 11:19 AM Idan Adar <id...@adar.me> wrote:

Another question, if I do customize "checkout", do I need to include everything like in your example, or only what I want specifically, like depth, and if nothing else is mentioned that the defaults are taken...?

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ed87e710-1906-46b3-9486-6440b92da5ba%40googlegroups.com.

[Kevin Burnett]

it is possible to skipDefaultCheckout and then "checkout scm". this isn't typically necessary or desirable if you want to operate on the same repo where your Jenkinsfile lives, but it sounds like you might want to override some checkout options, which seems cool. we also do this for some jobs that "trash the workspace" by writing some files as root, so the default checkout can't work on subsequent builds. in a Jenkinsfile stage we first "fix the workspace" by chown'ing it to the jenkins user, which allows `scm checkout` to work.

[Idan Adar]

Okay, so looks like what I should try in order to:

1. make checkouts faster

2. be able to git tag & push

Is the following:

1. Try with:

checkout(
 extensions: [
     [$class: 'CloneOption',
          depth: 1,
          shallow: true
     ]
 ]
)

2. Wait for Stephen...

BTW, I thought the shallow cloning is basically to specify depth, so why shallow:true is needed in additition to depth:1 ?

[Mark Waite]

On Saturday, June 17, 2017 at 11:48:19 AM UTC-6, Idan Adar wrote:

Okay, so looks like what I should try in order to:

1. make checkouts faster

2. be able to git tag & push

Is the following:

1. Try with:

checkout(
 extensions: [
     [$class: 'CloneOption',
          depth: 1,
          shallow: true
     ]
 ]
)

I look forward to hearing that it helped with your use case.

There are several ways to reduce the work done by git fetch or git clone, each with its own set of reasons it helps and reasons it hinders.  For example:

• An intentionally narrow refspec (like +refs/heads/master:refs/remotes/origin/master) reduces the objects copied in the clone to only those in that refspec (typically the desired branch), while still providing the full history of the working branch (but not the history of other branches in the repository)
• A reference repository updated on the agent (by whatever means you wish) can avoid network copies of objects by allowing git fetch and git clone  to point to the references instead of retrieving them.  This can be a major savings with large repositories, since multiple jobs can point to a single copy of the reference repository and save disc space and network transfer
• Large file support (LFS) can reduce network transfers by only transferring the current copy of large files which are being tracked, rather than transferring the entire history of large files.  LFS requires a newer version of git, changes in the repository, and some extra setup for support from the git server
• A sparse checkout can reduce the disc usage of the working directory by only performing a checkout of the specified subdirectories
• Avoiding fetch of tags can reduce network transfer by not copying the objects associated with tags
• A shallow clone can reduce network data transfer by only copying a subset of history into the working directory.  Depth beyond a single digit value has not seemed to be any different than not having shallow clone in the cases I've seen.  Shallow clone sometimes fails on older git versions (like the git included with CentOS 6 and earlier)

Slides from Jenkins World 2016 that discuss several of those options are available at https://jenkins.io/files/2016/jenkins-world/large-git-repos.pdf .

If you prefer video, there is a roughly 5 minute segment on git, Jenkins, and large repositories in  https://www.youtube.com/watch?v=TsWkZLLU-s4 .

Mark Waite

[Idan Adar]

• An intentionally narrow refspec (like +refs/heads/master:refs/remotes/origin/master) reduces the objects copied in the clone to only those in that refspec (typically the desired branch), while still providing the full history of the working branch (but not the history of other branches in the repository)
  

Is this relevant in a multibranch job? I thought that in this job that specifies "master develop" branches, a checkout of the develop branch checks out just the develop branch and a checkout of the master branch checks out just the master branch. Seems narrow enough?

If Stephen's changes will really simply allow me to git push as part of the default "checkout scm" command, I could just do that without trying to customize it... as I would be fine with the single checkout and then operate it on (instead of the forced git cloning I need to do now in order to git push).

BTW, trying that checkout customization in my previous post failed with an exception...

java.lang.NullPointerException
	at org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState.get(MultiSCMRevisionState.java:56)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:106)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:260)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[Idan Adar]

BTW, this whole exercise is essentially only so that I could tag builds of the master branch with 0.0.$BUILD_NUMBER so that I could record these as "GitHub releases".

Naturally, if there is a simpler approach to create these releases/tags instead of clone-tag-push, it'd simplify the whole process...

[Idan Adar]

I eventually used the following to accomplish my goal: https://stackoverflow.com/questions/44612496/bash-variable-escaping-in-a-jenkinsfile#44612844

Still looking forward to changes in "checkout"!

[Lee Meador]

Is this relevant to your tagging use-case?

https://stackoverflow.com/questions/38769976/is-it-possible-to-git-merge-push-using-jenkins-pipeline

On Sun, Jun 18, 2017 at 4:13 AM, Idan Adar <id...@adar.me> wrote:

I eventually used the following to accomplish my goal: https://stackoverflow.com/questions/44612496/bash-variable-escaping-in-a-jenkinsfile#44612844

Still looking forward to changes in "checkout"!

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9801a313-e975-4266-9f82-bb830d9b1825%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- Lee Meador
Sent from gmail. My real email address is lee AT leemeador.com