[JIRA] (JENKINS-43894) Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

9 views
Skip to first unread message

patrick@tario.org (JIRA)

unread,
Mar 15, 2018, 5:13:02 AM3/15/18
to jenkinsc...@googlegroups.com
Patrick Ruckstuhl commented on Improvement JENKINS-43894
 
Re: Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

As none of those variables provide us with something that can be configure on a slave level, this is what we did as an (ugly) workaround. We define an environment variable called REFERENCE_REPO on each slave and the master.

Inside our jenkins pipeline library we create a step called checkoutCurrent

import java.util.regex.Pattern
import hudson.plugins.git.extensions.impl.CloneOption

// checkout current scm and resolve the REFERENCE_REPO environment variable, workaround for JENKINS-43894
def call(originalScm) {
	if (originalScm.hasProperty('extensions') && originalScm.extensions) {
		print('has extensions')
		def extensions = originalScm.extensions
		def updatedExtension = null
		for (int i = 0; i < extensions.size(); i++) {
			def extension = extensions[i]
			if (extension instanceof CloneOption && extension.hasProperty('reference') && extension.reference) {
				def reference = extension.reference
				print('replacing reference: ' + reference)
				reference = reference.replaceAll(Pattern.quote('${REFERENCE_REPO}'), env.REFERENCE_REPO)
				print('with: ' + reference)

				updatedExtension = new CloneOption(extension.shallow, extension.noTags, reference, extension.timeout)
			}
		}

		if (updatedExtension) {
			extensions.replace(updatedExtension)
		}
	}
	checkout changelog: false, poll: false, scm: originalScm
}

and then in the pipeline instead of

checkout changelog: false, poll: false, scm: scm

we do

checkoutCurrent(scm)
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

atikhonova@parallels.com (JIRA)

unread,
Mar 15, 2018, 5:36:02 AM3/15/18
to jenkinsc...@googlegroups.com

Patrick Ruckstuhl So 'scm' extensions can be overridden actually... awesome tip, thanks! I have submitted a PR to git-plugin some time ago: https://github.com/jenkinsci/git-plugin/pull/575. No one who can merge seems to have time / be interested in it. I have just built the plugin with my patch and installed it in my instance.

patrick@tario.org (JIRA)

unread,
Mar 15, 2018, 9:07:02 AM3/15/18
to jenkinsc...@googlegroups.com
Patrick Ruckstuhl edited a comment on Improvement JENKINS-43894
As none of those variables provide us with something that can be configure on a slave level, this is what we did as an (ugly) workaround. We define an environment variable called REFERENCE_REPO on each slave and the master.

Inside our jenkins pipeline library we create a step called checkoutCurrent

{code}
import java.util.regex.Pattern
import hudson.plugins.git.extensions.impl.CloneOption

// checkout current scm and resolve the REFERENCE_REPO environment variable, workaround for JENKINS-43894
def call(originalScm) {
def originalExtension = null
def originalMetaClass = null

try {
if (originalScm.hasProperty('extensions') && originalScm.extensions) {
  print('has extensions')
  def extensions = originalScm.extensions
  def updatedExtension = null
  for (int i = 0; i < extensions.size(); i++) {
   def extension = extensions[i]
   if (extension instanceof CloneOption && extension .hasProperty('reference') && extension.reference instanceof String ) {

    def reference = extension.reference
    print('replacing reference: ' + reference)
    reference = reference.replaceAll(Pattern.quote('${REFERENCE_REPO}'), env.REFERENCE_REPO)
    print('with: ' + reference)

    updatedExtension      originalExtension = new CloneOption( extension .shallow,
     originalMetaClass =
extension. noTags, reference, metaClass
extension. timeout) metaClass.invokeMethod = { String name, args ->
   }
  }

  if ( updatedExtension 'getReference' == name ) {
   extensions        return reference
      } else {
       return this
. replace invokeMethod ( updatedExtension name, args )
  }
}
    }
   }
  }

checkout changelog: false, poll: false, scm: originalScm
} finally {
  if (originalExtension) {
   originalExtension.metaClass = originalMetaClass
  }
}


}
{
code}


and then in the pipeline instead of
{code}

checkout changelog: false, poll: false, scm: scm
{code}

we do

{code}
checkoutCurrent(scm)
{code}

patrick@tario.org (JIRA)

unread,
Mar 15, 2018, 9:10:02 AM3/15/18
to jenkinsc...@googlegroups.com

Updated my code a little bit, now it will revert the changes after doing the checkout (as otherwise this modified the general config I had in the job).

Would be really nice to get your PR merged.

patrick@tario.org (JIRA)

unread,
Mar 15, 2018, 9:46:02 AM3/15/18
to jenkinsc...@googlegroups.com
Patrick Ruckstuhl edited a comment on Improvement JENKINS-43894
As none of those variables provide us with something that can be configure on a slave level, this is what we did as an (ugly) workaround. We define an environment variable called REFERENCE_REPO on each slave and the master.

Inside our jenkins pipeline library we create a step called checkoutCurrent

{code}
import java.util.regex.Pattern
import java.lang.reflect.Field
import java.lang.reflect.Modifier

// checkout current scm and resolve the REFERENCE_REPO environment variable, workaround for JENKINS-43894
def call(originalScm) {
Field field = null
def originalExtension updatedExtension = null
def
originalMetaClass originalReference = null ;

try {
  if (originalScm.hasProperty('extensions') && originalScm.extensions) {
   print('has extensions')
   def extensions = originalScm.extensions
   for (int i = 0; i < extensions.size(); i++) {
    def extension = extensions[i]
    if (extension.hasProperty('reference') && extension.reference instanceof String) {
     def reference      updatedExtension = extension
     originalReference = extension
.reference
     print('replacing reference: ' +
reference originalReference )
     def      reference = reference originalReference .replaceAll(Pattern.quote('${REFERENCE_REPO}'), env.REFERENCE_REPO)
     print('with: ' + reference)

     originalExtension      // https://gist.github.com/pditommaso/263721865d84dee6ebaf
     field
= extension .class.getDeclaredField("reference")
     originalMetaClass      Field modifiersField = extension Field . metaClass class.getDeclaredField("modifiers")
     extension      modifiersField . metaClass setAccessible(true)
     modifiersField
. invokeMethod = { String name setInt(field , args -> field.getModifiers() & ~Modifier.FINAL)
      if      field.setAccessible ( 'getReference' == name true ) {
       return reference
      } else {
       return this
     field . invokeMethod set ( name extension , args reference )
      }
     }
    }
   }
  }

  checkout changelog: false, poll: false, scm: originalScm
} finally {
  if ( originalExtension field ) {
   originalExtension    print('reset reference to: ' + originalReference)
   field
. metaClass = originalMetaClass set(updatedExtension, originalReference)
  }
}


}

{code}


and then in the pipeline instead of
{code}
checkout changelog: false, poll: false, scm: scm
{code}

we do

{code}
checkoutCurrent(scm)
{code}

stevengfoster@gmail.com (JIRA)

unread,
Apr 10, 2018, 12:22:02 PM4/10/18
to jenkinsc...@googlegroups.com

Does this only work on environment variables configured on the node in jenkins? and so wouldn't work for non-static agents i.e. launched from cloud provider?

 

Or does it work on environment variables set on the agent itself?

atikhonova@parallels.com (JIRA)

unread,
Apr 11, 2018, 5:29:02 AM4/11/18
to jenkinsc...@googlegroups.com

Steven Foster For cloud slaves you will need to support setting environment variables via Node Properties in Jenkins. That should be implemented in a plugin that provides those slaves.

The patch above should work for environment variables set on an agent.

patrick@tario.org (JIRA)

unread,
Oct 24, 2018, 8:41:05 AM10/24/18
to jenkinsc...@googlegroups.com

As far as I can tell the fix for this https://github.com/jenkinsci/git-plugin/pull/575 was merged to master, so should be in the next 4.0 based release.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

patrick@tario.org (JIRA)

unread,
Oct 24, 2018, 8:41:08 AM10/24/18
to jenkinsc...@googlegroups.com
Patrick Ruckstuhl started work on Improvement JENKINS-43894
 
Change By: Patrick Ruckstuhl
Status: Open In Progress

patrick@tario.org (JIRA)

unread,
Oct 24, 2018, 8:42:02 AM10/24/18
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Apr 3, 2019, 11:30:03 AM4/3/19
to jenkinsc...@googlegroups.com
Jesse Glick commented on Improvement JENKINS-43894
 
Re: Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

By the way the use case described in the issue description seems like a bad idea to me. More straightforward to have the script determine the Git commit being built, and then rather than running checkout scm on each node, explicitly construct a GitSCM configured exactly the way you want, according to whatever Groovy logic you need. The kind of plugin-based environment variable expansion seen in this PR should only be necessary in freestyle projects.

patrick@tario.org (JIRA)

unread,
Apr 3, 2019, 12:44:02 PM4/3/19
to jenkinsc...@googlegroups.com

Jesse Glick maybe I'm missing something but to me this looks totally normal to use in pipeline jobs.
We have some linux slaves and some windows slaves and they each have a different location of the reference repo. I can even think of cases where you have e.g. linux nodes which for whatever reason have different paths to the reference repo. Having this configurable on a slave level and then just use it in the multibranch config is perfect, that way the pipeline does not care about how the checkout happens.

mathias.hasselmann@meiller.com (JIRA)

unread,
Apr 4, 2019, 4:31:03 AM4/4/19
to jenkinsc...@googlegroups.com

Exactly that. Reference repo paths, library paths, tool paths: This all are node specific settings which have no place in the actual pipeline description. As an extreme example imagine that your project is published on Github or the like. Do you really want to share with the world the gritty details of your local Jenkins configuration?

jglick@cloudbees.com (JIRA)

unread,
Apr 4, 2019, 8:30:08 AM4/4/19
to jenkinsc...@googlegroups.com

library paths, tool paths

These are of course better handled via Docker containers or the like if you can manage it.

node specific settings which have no place in the actual pipeline description

I did mean to imply that the Jenkinsfile must list actual filesystem paths on each possible node, merely that the program has some way of determining this information on the fly, preferably not involving Jenkins global configuration. That could be an environment variable set on the agent’s account, etc. The point is that the selection and configuration of a reference repo is something under the control of the job maintainer or Jenkins admin, rather than being baked into a plugin.

imagine that your project is published on Github

I do not have to imagine that, it is a daily reality.

Do you really want to share with the world the gritty details of your local Jenkins configuration?

In the case of the Jenkins project, in fact we do, but in case you do not, there are various alternatives.

Daniel.Pasto@gmail.com (JIRA)

unread,
Jan 13, 2020, 12:09:03 PM1/13/20
to jenkinsc...@googlegroups.com
D Pasto commented on Improvement JENKINS-43894

Any updates on releasing this?  I verified it does not work on my v4.0 plugin (can resolve $JENKINS_HOME but not $HOME so is useless) and the changelog does not list it for that or v4.1 beta

We have pipelines across many nodes, including different platforms, so we can't take full advantage of reference repos (the workaround of manually re-defining the checkout for every stage would be maintenance disaster) — we need a way of dynamically building the reference repo path per node per pipeline.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 13, 2020, 12:37:03 PM1/13/20
to jenkinsc...@googlegroups.com

I've proposed automatic repository caching as a Google Summer of Code idea and have offered the pull request as a starting point for consideration.

Have you tried the alternative that is described by Jesse Glick? It seems like it would work in your case if you perform the environment variable expansion in the Jenkins checkout statement.

patrick@tario.org (JIRA)

unread,
Jan 13, 2020, 12:50:03 PM1/13/20
to jenkinsc...@googlegroups.com

D Pasto did you actually specify HOME as an explicit variable on the node or are you just depending on an existing environment variable. If I remember correctly we're using this with some 4.* version for a while, but always with explicitly specified node variables.

Daniel.Pasto@gmail.com (JIRA)

unread,
Jan 13, 2020, 2:01:04 PM1/13/20
to jenkinsc...@googlegroups.com
D Pasto commented on Improvement JENKINS-43894

Good call Patrick Ruckstuhl Unable to render embedded object: File (  It works) not found.  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs

Daniel.Pasto@gmail.com (JIRA)

unread,
Jan 13, 2020, 2:09:03 PM1/13/20
to jenkinsc...@googlegroups.com
D Pasto edited a comment on Improvement JENKINS-43894
Good call [~tario] !     It works!  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs

Edit: forgot the most important part - this lets me define a single path (using forward slashes) that works on both Linux and Windows nodes.

patrick@tario.org (JIRA)

unread,
Jan 13, 2020, 2:27:03 PM1/13/20
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages