String interpolation in sh command with credentials and (shared lib) function call

[ST]

Since pretty recently, Jenkins prints out warnings when one inserts e.g. credentials into the 'script' arg of the sh command, and the warnings referring to this help page:

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation

Me being curious I am trying to get rid of those warnings but failing no matter what I try.

Here is the relevant section of our scripted pipeline file:

withCredentials([usernamePassword(credentialsId: 'credentialsGenerateFromDb',
                           usernameVariable: 'GEN_USR',
                           passwordVariable: 'GEN_PASSWD')]) {
   String m2RepoIdentifier = "${env.BRANCH_NAME}_${env.BUILD_NUMBER}"

   withEnv(javaAndMavenEnvArray()) {
      sh '${JAVA_HOME}/bin/java -version'

      sh "${mvnCommand(m2RepoIdentifier)} -f project-reactor/pom.xml" +
            " -P GenerateProfile -DdeployAtEnd=true" +
            " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'" +
            " '-Dproj.build.generatefromdb.user=${env.GEN_USR}'" +
            " '-Dproj.build.generatefromdb.password=${env.GEN_PASSWD}'" +
            " clean ${mvnBuildGoal} pmd:pmd pmd:cpd"
   }
}

Where both mvnCommand(String) and jdbcUrlNoData() are functions defined in a shared library.

I have tried to use single quotes (which obviously works in some cases, see the first sh command in my example above), and multi-line strings using single quotes and double quotes.

But either the variables do not get interpolated, or I dont get rid of the warnings.

So what is the recommended way of building up an sh command that contains both dynamic values from functions and credentials that should not be left to groovy to be interpolated? The more I google around the more complicated this gets (and the more different "solutions" I find, including e.g. wrapping the ${SOME_VAR} with double quotes inside a single-quoted string).

Would really appreciate if someone could shed some light on this problem, thanks!

 stefan.

[Dirk Heinrichs]

Am Dienstag, den 17.11.2020, 13:45 +0100 schrieb ST:

" '-Dproj.build.generatefromdb.user=${env.GEN_USR}'"

Try with ${GEN_USR} inside shell scripts, not ${env.GEN_USR}.

HTH...

Dirk

-- 

Dirk Heinrichs

Senior Systems Engineer, Delivery Pipeline

OpenText ™ Discovery | Recommind

Phone: +49 2226 15966 18

Email: dhei...@opentext.com

Website: www.recommind.de

Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach

Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

[ST]

Thanks Dirk, but that does not solve my problem, I'm still stuck.

I think that my question boils down to why the multi-line sh command below does not work?

According to https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation

single-quoted strings should be interpolated by Jenkins (instead of using Groovy string interpolation). I would assume this applies

to both variables (e.g. GEN_USR below) and method calls returning a string (e.g. jdbcUrlNoData() below)?

withCredentials([usernamePassword(credentialsId: 'credentialsGenerateFromDb',
                           usernameVariable: 'GEN_USR',
                           passwordVariable: 'GEN_PASSWD')]) {
   String m2RepoIdentifier = "${env.BRANCH_NAME}_${env.BUILD_NUMBER}"

   withEnv(javaAndMavenEnvArray()) {

      sh '${JAVA_HOME}/bin/java -version' // single quotes - this works fine!

      sh script: '''
            ${mvnCommand(m2RepoIdentifier)}\
             -f proj-reactor/pom.xml\
             -P GenerateStorables -DdeployAtEnd=true\
             -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\
             -Dproj.build.generatefromdb.user=${GEN_USR}\
             -Dproj.build.generatefromdb.password=${GEN_PASSWD}\
             clean ${mvnBuildGoal} pmd:pmd pmd:cpd\
      '''
   }
}

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4dd32ea372d43c9dde0800bed9a16253d45cb2e5.camel%40opentext.com.

[JonathanRRogers]

I don't think you've read the Jenkins pipeline documentation correctly. String interpolation is performed by Groovy. Jenkins doesn't add any interpolation behavior. Jenkins will always treat single-quoted strings as simple literals. The "sh" step calls a shell which can do its own variable expansion, which is the approach recommended for passing credentials to external commands.

Your multi-line sh step includes calls to "mvnCommand()" and "jdbcUrlNoData()". Unless those are shell functions, you're unlikely to get the results you want.

[Gianluca]

Somethings said are correctly … somethings said are wrong.

I’ll try to summarise pointing to the lines in your pipeline and explain what’s wrong and how to fix:

Single quoted string are literals. They never get interpolated and passed as they are.

So, in your case:

sh '${JAVA_HOME}/bin/java -version' // single quotes - this works fine!

Work … not because ${JAVA_HOME} is interpolated by Jenkins … but because for an unfortunate coincidence … ${JAVA_HOME} is a correct bash syntax for reference an environment variable in bash and get expanded correctly.

DO NOT confuse Jenkins pipeline environment variable that exists only in the pipeline context with the BASH environment variable that exists in the shell when “sh” is executed.

Now, also take into account that Jenkins also creates a corresponding BASH environment variable for each pipeline environment variable.

So, what in Groovy double-quoted string like “${env.GEN_USR}” get expanded using the value of the Jenkins environment variable GEN_USR, but if you want to let bash expand the environment variable then you have to pass '${GEN_USR}’ … in single-quoted string … or … you need to escape “$” in double-quoted string to avoid that Groovy does the interpolation instead of bash.

Your multi-line command suffer from the fact somethings need to be expanded by Groovy and something else need to be expanded by BASH, so you need to split in different string and concatenate or escape $:

 sh script: '''
            ${mvnCommand(m2RepoIdentifier)}\

This is a Jenkins function … so need to be expanded by Groovy and it’s in the double-quoted string… so far so good

             -f proj-reactor/pom.xml\
             -P GenerateStorables -DdeployAtEnd=true\
             -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\

That is the same … so it’s good

             -Dproj.build.generatefromdb.user=${GEN_USR}\

That’s the problem. You are using “withCredentials” and that command only creates BASH environment variable … and you are letting Groovy do the expansion but GEN_USR doesn’t exist into Jenkins pipeline … and neither env.GEN_USR … the only one that exist is the BASH environment variable and you have to let bash expand … so you need to escape $ to pass as it is to BASH:

-Dproj.build.generatefromdb.user=\${GEN_USR}\

             -Dproj.build.generatefromdb.password=${GEN_PASSWD}\

Same as above… it’s a BASH environment variable

             clean ${mvnBuildGoal} pmd:pmd pmd:cpd\

That is probably a Groovy variable but I’m not entire sure.

      '''

I hope that helps.

As suggestion, try to split and use a mix of double-quoted and single-quoted so you avoid escaping:

      sh script: 
            "${mvnCommand(m2RepoIdentifier)} “ +
             '-f proj-reactor/pom.xml ‘ +
             '-P GenerateStorables -DdeployAtEnd=true ‘ +
             "-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}” +
             '-Dproj.build.generatefromdb.user=${GEN_USR} ‘ +
             '-Dproj.build.generatefromdb.password=${GEN_PASSWD} ‘ +
             'clean ${mvnBuildGoal} pmd:pmd pmd:cpd'

Cheers,

Gianluca.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bed31997-28f8-4e9b-b205-43f02d4348a5n%40googlegroups.com.

[JonathanRRogers]

Unless "mvnBuildGoal" is the name of a Jenkins environment variable, which would be added to the shell process environment, it won't work inside a single-quoted string.

[ST]

Thanks Jonathan and Gianluca for your explanations. I now got most of my confusion cleared up, most importantly the fact that String interpolation is done either by Groovy or by the bash but never by Jenkins.

I think the first sentence here got me on the wrong path: "Jenkins Pipeline uses rules identical to Groovy for string interpolation"

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation

To me that suggests that Jenkins also does some kind of string interpolation.

@Gianluca: Thanks for your attempts to solve my problem. I've tried again, and I am pretty sure now that mixing simple strings (single quotes) and GStrings (double quotes) does not help because I suspect that there is some automatic type conversion between simple strings and GStrings involved here that destroys the simple strings - see below.

Also keep in mind that I need to quote all of jdbcUrl/username/password in the bash command because they can contain special characters (& ! etc).

So trying your approach like this:

sh "${mvnCommand(m2RepoIdentifier)}" +

    ' -f proj-reactor/pom.xml ' +
    ' -P GenerateStorables -DdeployAtEnd=true' +
    " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'" +
    ' -Dproj.build.generatefromdb.user=${GEN_USR}' +
    ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' +
    " clean ${mvnBuildGoal} pmd:pmd pmd:cpd"

-> Jenkins will still print the warning about insecure interpolation of sensitive variables

-> The build works fine though, e.g. all variables get passed to the maven build as expected

BUT what actually worked in the end is the following:

sh "${mvnCommand(m2RepoIdentifier)}".toString() +

    ' -f proj-reactor/pom.xml ' +
    ' -P GenerateStorables -DdeployAtEnd=true' +

    " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'".toString() +

    ' -Dproj.build.generatefromdb.user=${GEN_USR}' +
    ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' +

    " clean ${mvnBuildGoal} pmd:pmd pmd:cpd".toString()

-> No warning about insecure interpolation of sensitive variables

-> The build works fine, e.g. all variables get passed to the maven build as expected

-> Also nice for readability to not need to escape any quotes/characters in any part of the command above

Hope this helps other folks to clear up any string interpolation confusion :-)

 stefan.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b5086225-d400-4500-a936-0413c025edf6n%40googlegroups.com.

[JonathanRRogers]

I'm glad you figured it out. It sure looks like a mess.