What is the freestyle "inject environment variables" equivalent inside blue ocean pipelines?

[jeremy....@wonderful.fr]

Hi.

I'm trying to migrate from a freestyle job to a pipeline that has three stages (build, test and deploy)

In the deploy stage, I'd like to rsync files to a remote server.

In my freestyle job, I used to inject environment variables to the build process, by specifying them in the properties content textarea like so:

SSHUSERNAME=foo

SSHSERVER=178178.178.178

REMOTEPATH='/var/virtual_www/htdocs';

That meant that I could then always use the same deploy script, that would use those variables, no matter the project:

rsync -uvr --delete --exclude-from $WORKSPACE/CI/exclude-file.txt $WORKSPACE/* $SSHUSERNAME@$SSHSERVER:$REMOTEPATH;

Now regarding pipeline scripts.

How could I implement something equivalent in pipelines per branch? 

As this is sensitive data, I do not see myself declaring variables inside a script that is committed to a repo, so writing them inside the Jenkins File seems out of order.

And that would also mean changing the deploy script for each client, which I'd like to avoid.

I've been told to look into credentials, but from what I understood (I'm really not a Jenkins expert, I'm trying to learn), credentials seem to be global key / value pairs ?

Is there a way to create credentials that are only within the project scope? Or are they only global?

If they are global? Does that mean that I need to create one credential for each line I've just shown as example? And that I also need to change the name for each branch, and for each branch per project? Like project1_master_sshuser, project1_develop_sshuser?

Doesn't that seem a bit cumbersome?

Or maybe I need to use a secret file to put all those information in?

Well as you can see this part is still a bit obscure for me :)

Any help on implementing this would be hugely appreciated.

I've arleady planned to write a blog post about the entire setup to spread the help.

TL;DR

- I'm trying to migrate from a freestyle job to a pipeline

- In my freestyle job, I used to inject environment variables to the build process, by specifying them in the properties content textarea

- How could I inject environment variables in pipelines per branch?  

Thanking you.

Regards.

[Bill Dennis]

If you put the pipeline / branch jobs inside a folder, you can scope the credentials to just that folder. Pretty sure that is available in Jenkins OSS and not just Enterprise - you need the CloudBees Folders plugin. Have a look on here, it might have some clues: https://support.cloudbees.com/hc/en-us/articles/204264974-How-inject-your-Maven-settings-xml-at-folder-level-with-the-Credentials-plugin

I am not sure if this helps in your branch scenario. I put all my credentials globally then realised I could scope them to the folder level - I missed it due to some nuances in the credentials UI.

Bill

[Bill Dennis]

Just some other things I thought of -

If you use the credentials file feature you can put all those sensitive properties in a properties file stored as 'jenkins credentials'. 

Then pull that props file into your workspace using 'withCredentials' in the pipeline.

Next thing is to grab the pipeline utility steps plugin which has a readProperties step (it is not one of the standard pipe plugins - you will need to add it).

https://plugins.jenkins.io/pipeline-utility-steps

Then you have the file properties loaded as Java properties and you can use them as before.

I did this move from Freestyle too and there is a lot to learn but it is worth it. Another recommendation is to look at the declarative pipeline not just scripted pipeline. Declarative has post build handling in the pipeline which you may miss from FreeStyle jobs. In scripted pipeline you have to do a lot of try-catch handling for build errors.

Bill

[jeremy....@wonderful.fr]

Hi Bill.

Thanks so much for your reply.

I like this credential file option. That would mean I can create a file with all the environment variables I need for my branches inside (one per branch I guess). And if I could scope it inside my project folder even better.

I've tried to google information about how to use credential files, but without much success. Would you have an example of how you'd write one?

Is it a key / value format? bash variables declarations? JSON? XML?

Thank you for your time and your help.

Regards.

Jeremy.

[Bill Dennis]

It can be any format file you like XML, properties, txt whatever you need for some sort of configuration (except large binary files I guess). 

There is a CloudBees article here that should help:

https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs

The article shows creating these globally but you can create them scoped on a folder.

Then to use in the pipelines I suggest to drop into the pipeline syntax link on  a pipeline job that drops into the snipper generator in the Jenkins pipeline UI and go through the 'withCredentials' snippet generator. It found it best to experiment around a bit to figure it out.

--Bill

[jeremy....@wonderful.fr]

Hi Bill.

I've tried many things with no luck.

So I've got this JSON credential file whose content looks like this:

{
  "sshUserKey":"sshuserval",
  "sshHostKey":"sshhostval"
}

I've successfully managed to open this file in my pipeline:

withCredentials([file(credentialsId: 'secrettest', variable: 'testMasterCred')]) {
    sh "cat ${testMasterCred}";
}

The cat command shows effectively the content of the JSON file.

Then, how would you parse this JSON?

I've tried readJSON file: $testMasterCred; but this doesn't work, and throws the following message: No such property: $testMasterCred for class: groovy.lang.Binding

I've got the feeling that I'm not very far from the truth.

There's not much help on the pipeline-utility-steps-plugin readme regarding this.

Would you have an example about how I could get the parsing right?

Thanking you.

Regards.

[Bill Dennis]

Hi -

Yes, I have done this JSON parse. Example on my Github:

https://github.com/macg33zr/jenkins-experimental-pipelines/blob/master/json-parse-pipeline.groovy

It is a bit ugly as it needs script approvals (or turn off pipeline sandbox or put the code in a trusted global pipeline library).

You need a NonCPS method like this:

import groovy.json.JsonSlurperClassic

@NonCPS
def parseJsonToMap(String json) {
    final slurper = new JsonSlurperClassic()
    return new HashMap<>(slurper.parseText(json))
}

Another way to go would be a command line tool to parse JSON like this one:

https://stedolan.github.io/jq/

Then just run a 'sh' command to get the data out. I haven't used that myself.

Have fun!

--Bill

[jeremy....@wonderful.fr]

Hi Bill.

After many workarounds, I've finally hacked together a script that works, using your parsJsonToMap function.

My part looks a bit dirty to me, but I've lost so many hours on this that I'm glad I can finally carry on.

Here's what's working:

steps {
    echo "Deploying Develop Branch"
    withCredentials([file(credentialsId: 'develop_credentials', variable: 'MY_SECRET_FILE')]) {
        script {
            MY_SECRET_FILE_CONTENT = sh (
                script: "cat ${MY_SECRET_FILE}",
                returnStdout: true
            ).trim()
            def creds = parseJsonToMap(MY_SECRET_FILE_CONTENT)
            echo "Sending files to remote server"
            sh "rsync -uvr --delete --exclude-from ${WORKSPACE}/CI/exclude-file.txt ${WORKSPACE}/* ${creds.sshUser}@${creds.sshServer}:${creds.sshRemotePath};"
        }
    }
}

Thank you very much for your help and your time.

Regards.

[Bill Dennis]

Hey -

You don't have to 'cat' the file, there is a readFile pipeline step you can use:

MY_SECRET_FILE_CONTENT = readFile MY_SECRET_FILE