Is there any way to !include a shell script? (JJB is eating whitespace)

[Aleksey Tsalolikhin]

Hello,

I would like to take some multi-line shell code that my team has embedded in JJB YAML files and split it off into .sh files so we can get the benefit of syntax highlighters, linters, and language-specific pre-commit checks (e.g. shellcheck).

Here is my first attempt (using version 3.10.0, latest stable), and it looks like Jenkins Job Builder eats the whitespace when it includes the file:

[~/git/learn-jjb/jjb1] [my-venv] $ cat *yml

- job:

    name: job-name

    builders:

      - shell: !include shell.sh

[~/git/learn-jjb/jjb1] [my-venv] $ cat *sh

#!/bin/bash -e

          echo hello world
          echo hello world

[~/git/learn-jjb/jjb1] [my-venv] $ jenkins-jobs test . 2>/dev/null | grep 'command>'

      <command>echo hello world echo hello world</command>

[~/git/learn-jjb/jjb1] [my-venv] $

Second attempt:

[~/git/learn-jjb/jjb1] [my-venv] $ cat *yml

- job:

    name: job-name

    builders:

      - shell: |

          !include shell.sh

[~/git/learn-jjb/jjb1] [my-venv] $ jenkins-jobs test . 2>/dev/null | grep -A1 '<command>'

      <command>!include shell.sh

</command>

[~/git/learn-jjb/jjb1] [my-venv] $

Is this a bug or am I doing it wrong?  :)

I know I can create a repo for shell scripts and use the SCM Git plugin to check out the repo and then have a shell builder that runs a script from that repo. I suppose that's my workaround if I can't include the shell script with JJB.

So, is there any way to !include a shell script with JJB?  :)

Best,

Aleksey

 

[Andrew Grimberg]

On 9/8/21 1:23 PM, 'Aleksey Tsalolikhin' via jenkins-job-builder wrote:
> Hello,
>
> I would like to take some multi-line shell code that my team has
> embedded in JJB YAML files and split it off into .sh files so we can get
> the benefit of syntax highlighters, linters, and language-specific
> pre-commit checks (e.g. shellcheck).

Yes, it's possible. There are several ways to use include. See the docs [0]

What you probably want is either !include-raw or !include-raw-escape

The first will read a data file and injected in raw which will then get
JJB interpolated. The second will inject the file in with no
interpolation or more precisely, it auto-escapes ${} expressions for you
to ${{}} expressions.

-Andy-

[0]
https://docs.openstack.org/infra/jenkins-job-builder/definition.html#inclusion-tags

> --
> You received this message because you are subscribed to the Google
> Groups "jenkins-job-builder" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkins-job-bui...@googlegroups.com
> <mailto:jenkins-job-bui...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkins-job-builder/CAGG6SKSuY1dchqu94NHMUPYvuNQ%3DsG1i%3DjLYEgG49UcjWp%2B2cQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkins-job-builder/CAGG6SKSuY1dchqu94NHMUPYvuNQ%3DsG1i%3DjLYEgG49UcjWp%2B2cQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
Andrew J Grimberg
Manager Release Engineering
The Linux Foundation

[Aleksey Tsalolikhin]

Thanks, Andy!!  That's exactly what I needed.  :)  

Best,

Aleksey

--

Aleksey Tsalolikhin DevOps - Sr. Systems Engineer | Golf - Scrum Master / Developer | Data Nexus Integration Team - Scrum Master  LinkedIn  •   Instagram  •   Youtube

 

[Aleksey Tsalolikhin]

Hi,

I finally got this change socialized and merged.  I'm a little confused as I thought that if I used "!include-raw", I would have to escape my Make-style shell variables to keep JJB away from trying to expand the contents of the braces -- for example, I tested and confirmed that if I had

     echo {fruit}

in my included shell script, and I had a variable "fruit" in the JJB project set to "banana", I would see "banana" in the shell script output in the Jenkins build log.

However I am getting an error:

/tmp/jenkins7912277661751964409.sh: line 28: ${{GIT_REPOSITORIES_LIST//\'/\"}}: bad substitution

That's from the following included shell script:

    GIT_REPOSITORIES_LIST=${{GIT_REPOSITORIES_LIST//\'/\"}}

I fear I misunderstood something.

How should I treat braces in the shell script I want JJB to include in the job definition?

Thanks,

Aleksey

--

Aleksey Tsalolikhin Sr. Systems Engineer

[Antoine Musso]

On 22/11/2021 19:56, 'Aleksey Tsalolikhin' via jenkins-job-builder wrote:

I finally got this change socialized and merged.  I'm a little confused as I thought that if I used "!include-raw", I would have to escape my Make-style shell variables to keep JJB away from trying to expand the contents of the braces -- for example, I tested and confirmed that if I had

     echo {fruit}

in my included shell script, and I had a variable "fruit" in the JJB project set to "banana", I would see "banana" in the shell script output in the Jenkins build log

Hello,

!include-raw reads the text from the file and injects it as yaml, it is then treated exactly as is without any specific escaping. Since JJB uses curly braces to delimit variables, {fruit} is expanded when in a job template.  If you wanted the job to contain literally {fruit} without any expansion by jjb, you need the !include-raw-escape which would escape the curly brace from the source file by doubling them.

Assuming a script echo.sh having:

repos=${GIT_REPO:-}
echo '{fruit}'

I ) In a job

!include-raw-escape: echo.sh gives:

<command>repos=${{GIT_REPO:-}}
echo '{{fruit}}'
</command>

The curly braces have been escaped by doubling them.

On the other hand, !include-raw: echo.sh gives the expected output:

<command>repos=${GIT_REPO:-}

echo '{fruit}'
</command>

That is because a job does not do any interpolation of the curly braces.

II ) in a job-template

!include-raw-escape: echo.sh gives what you are looking for:

<command>repos=${GIT_REPO:-}

echo '{fruit}'
</command>

The curly braces have been escaped and they are not expanded to variable by the template.

!include-raw: echo.sh yields an error. The template is unable to be formatted, it is missing the GIT_REPO and fruit variables.

In conclusion:

In a JOB use include-raw

In a TEMPLATE use include-raw-escape

Which also mean that builders macro can not be reused between jobs and templates.  I got almost everything to use job templates for consistency and avoiding surprises.

The documentation might require some improvement here and there. I will be more than happy to approve any such change ;)

Antoine "hashar" Musso
Release Engineering
Wikimedia Foundation

[Aleksey Tsalolikhin]

Thanks, Antoine!

This is a job definition (not a job-template).

I am using "!include-raw".

My shell code (in the included file) has double braces to escape expansion:

    GIT_REPOSITORIES_LIST=${{GIT_REPOSITORIES_LIST//\'/\"}}

My shell code (in the Jenkins job) ALSO has double braces!  This is not valid shell syntax and it breaks:

/tmp/jenkins7912277661751964409.sh: line 28: ${{GIT_REPOSITORIES_LIST//\'/\"}}: bad substitution

 Jenkins Job Builder version: 3.11.0

I just re-read the Macro Notes in the JJB documentation (https://docs.openstack.org/infra/jenkins-job-builder/definition.html#macro-notes) -- my builder is directly in the job definition, it is not a macro.  So I shouldn't be escaping the curly braces, since that only applies to builders in macros.  But if or when I want to move my builder in a macro, I would need to escape the curly braces in the included shell script.

How does that sound?

Thanks,

Aleksey

[Darragh Bailey]

Hi Aleksey,

The idea of the various yaml tags was to remove the need to escape shell brackets in scripts, so they could be debugged outside of JJB easily as well.

Unless you have a usec ase where you need to perform substitution on your script contents, look to remove any escaping so that the script could be run directly outside of JJB, and use '!include-raw: <script>' when including directly in a job definition or a non expanding macro, and 'include-raw-escape: <script>' when including via templates or macros that are themselves expanded.

Some macros are useful without being passed parameters and are thus not expanded, hence the subtly in behaviour.

Darragh Bailey

--

You received this message because you are subscribed to the Google Groups "jenkins-job-builder" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-job-bui...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkins-job-builder/CAGG6SKTLyLRRYyJcbk04k476Zmkn7MnQUxOa-Zv1XYtr4CEx9g%40mail.gmail.com.

[Aleksey Tsalolikhin]

Nice! Thank you for explaining that subtlety, Darragh.

Aleksey