Realizing a job-template using different lists from the same project

[Paride Legovini]

Hi,

I'd like to realize a given job-template more than once from the same project,
but using different lists. I tried with (also in pastebin [1]):



- project:

name: project-name

list1:

- foo

- bar

list2:

- moreFOO

- moreBAR

list3:

- foofoo

- barbar

jobs:

- somejob-{x}-{y}:

x: {list1}

y: {list2}

- somejob-{x}-{y}:

x: {list1}

y: {list3}



- job-template:

name: somejob-{x}-{y}




but this generates weird jobs like:

Job name: somejob-OrderedDict([('list1', None)])-OrderedDict([('list2', None)])
Job name: somejob-OrderedDict([('list1', None)])-OrderedDict([('list3', None)])

This with: Jenkins Job Builder version: 3.9.0.

I could get the result I'm looking for by using two separate projects, at the
cost of duplicating list1, which I'd like to avoid (the actual case I'm
working at is more complex than this).

Does JJB support this in some way, perhaps with job groups?

Thanks!

Paride

[1] http://ix.io/3m3q

[Adam Romanek]

Hi Paride,

jobs:
- somejob-{x}-{y}:
  x: {list1}
  y: {list2}

JJB reads "x" and "y" as dictionaries (OrderedDict) with "listX" as keys and no values (None). That's why you see OrderedDict([('listX', None)]) in the names of the jobs.

Could you please elaborate on what you're trying to achieve? Are you expecting all combinations of x:list1 and y:list2 and then also all combinations of x:list1 and y:list3?

BR,

Adam Romanek

[Paride Legovini]

Hi Adam,

Exactly as you said: I'd expect http://ix.io/3m3q to generate all the

combinations of x:list1 and y:list2 and then also all combinations of

x:list1 and y:list3.

This specific case could be solved by merging list2 and list3, but this
is only because I simplified the problem as much as I could. The case
where I have job-template

somejob-{x}-{y}-{z}

and I want to realize it once as

somejob-{list1}-{list2}-{list3}

and also as

somejob-{list1}-{list4}-{list5}

can't be solved by merging lists. Splitting it into separate projects
works but requires duplicating list1 and all project-level variables.

Thanks!

Paride

Adam Romanek wrote on 10/05/2021:
> Hi Paride,
>
> jobs:
> - somejob-{x}-{y}:
>   x: {list1}
>   y: {list2}
>
> JJB reads "x" and "y" as dictionaries (OrderedDict) with "listX" as keys
> and no values (None). That's why you see OrderedDict([('listX', None)])
> in the names of the jobs.
>
> Could you please elaborate on what you're trying to achieve? Are you
> expecting all combinations of x:list1 and y:list2 and then also all
> combinations of x:list1 and y:list3?
>
> BR,
> Adam Romanek
> On Friday, May 7, 2021 at 12:16:39 PM UTC+2 Paride Legovini wrote:
>
> Hi,
>
> I'd like to realize a given job-template more than once from the
> same project,
> but using different lists. I tried with (also in pastebin [1]):
>

> [removed mangled yaml]

>
> but this generates weird jobs like:
>
> Job name: somejob-OrderedDict([('list1',
> None)])-OrderedDict([('list2', None)])
> Job name: somejob-OrderedDict([('list1',
> None)])-OrderedDict([('list3', None)])
>
> This with: Jenkins Job Builder version: 3.9.0.
>
> I could get the result I'm looking for by using two separate
> projects, at the
> cost of duplicating list1, which I'd like to avoid (the actual case I'm
> working at is more complex than this).
>
> Does JJB support this in some way, perhaps with job groups?
>
> Thanks!
>
> Paride
>

> [1] http://ix.io/3m3q <http://ix.io/3m3q>

[Adam Romanek]

Hi Paride,

One project produces all combinations of all lists defined on its level. In the very simplest form like:

- project:

    name: project-name

    list1:

      - foo

      - bar

    list2:

      - moreFOO

      - moreBAR

    list3:

      - foofoo

      - barbar

    jobs:

      - somejob-{list1}-{list2}-{list3}:

- job-template:

    name: somejob-{list1}-{list2}-{list3}

It will produce 8 jobs - all combinations of list1, list2 and list3. I know that's not what you want but it's just to illustrate how it works.

If you want to have more control then you need to use separate projects. And in order to avoid duplication you should use anchors and aliases [1].

So in the end you should have something like:

- common: &common

    name: common

    list1:

      - foo

      - bar

- project:

    name: project-name-A

    <<: *common

    list2:

      - foofoo

      - barbar

    jobs:

      - somejob-{list1}-{list2}:

- project:

    name: project-name-B

    <<: *common

    list2:

      - moreFOO

      - moreBAR

    jobs:

      - somejob-{list1}-{list2}:

- job-template:

    name: somejob-{list1}-{list2}

HTH

Adam

[Paride Legovini]

[Repost from the Ggroups web interface, as my email replies apparently aren't reaching the group. I hope the formatting remains sane.]

Hi Adam, yaml anchors/aliases are indeed a good way to avoid
duplication, that's an excellent suggestion, thanks!

I still think that what I was looking for still would allow for more terse definitions in some cases, and in the meantime I noticed more things. Interestingly this works:

- project:
    name: project-name
    mylist:
      - foo
      - bar
    jobs:
      - somejob-{mylist}-{x}:
          x: '{mylist}'

- job-template:
    name: somejob-{mylist}-{x}

and generates the two jobs:

  Job name:  somejob-bar-bar
  Job name:  somejob-foo-foo

but **only works when quoting** the string given to 'x':

      - somejob-{mylist}-{x}:
          x: '{mylist}'

while doing

      - somejob-{mylist}-{x}:
          x: {mylist}

produces the "empty OrderedDict" jobs we already encountered:

  Job name:  somejob-bar-OrderedDict([('mylist', None)])
  Job name:  somejob-foo-OrderedDict([('mylist', None)])

The documentation suggests to always use quotes ("it is good practice to
wrap the entire string in quotes, even if the rules of YAML syntax don’t
require it"), but I've been definitely been bitten here: it took a while
to figure out what was happening. Lesson learned.

This also means that what I'm looking for *almost* works already. This:

- project:
    name: project-name
    mylist:
      - foo
      - bar
    jobs:
      - somejob-{x}:
          x: '{mylist}'

- job-template:
    name: somejob-{x}

builds the job:

  Job name:  somejob-['foo', 'bar']

Obviously not what I want, but I think this is only because the JJB
templating engine iterates over lists only when a list name appears in
the template name. It should iterate even when a list is passed to a
variable that is in the template name ('x' in the example above). I'll
have a look at the code :-)

Paride