Need help with Matrix-build

[Roland Asmann]

Hi guys,

I need some help with a matrix-build. Until now, I had the build configured to run on all nodes that have a label 'deb' and 'rpm' as axis 'distro' combined with axis 'arch' which checks on 'i386' and 'x64'.
This is because we are building some binaries that are specific for i386 and x64 and packaging the builds depending in either RPM or DEB. So in all, we had 4 artifacts resulting.

Now, I need to add another distro-type, namely a Windows EXE. The thing is that this doesn't have to be built on both architectures, since the project will only create a 32-bit binary.
I know I can set the matrix to invalidate the combination 'exe' and 'x64', but that is not exactly doing what I want. If I would do this, it would never build my EXE on a 64-bit node, but I would like to have these nodes included in the build-pool as well! The nodes will need to have the label 'x64' set for some other builds I have, so changing them to 'i386' is also not an option.

So, the question is, how can I setup the build to run the following 'combinations':
- deb x i386
- deb x x64
- rpm x i386
- rpm x x64
- exe

and have the 'exe' run on any available machine that has the label 'exe', no matter if it has the label 'i386' or 'x64'?

Any help is greatly appreciated!

Roland

[Stephen Connolly]

Of course with literate builds you get a much easier way:

Here's the markdown you'd have

# Environments

* `debian`

    * `i386`

    * `x64`

* `redhat`

    * `i386`

    * `x64`

* `windows`

# Build

* `debian`

        whatever command you use to make the deb

* `redhat`

        rpmbuild blah blah blah

* `windows`

        whatever command you use to make the exe

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Les Mikesell]

On Thu, Oct 3, 2013 at 11:27 AM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
> Of course with literate builds you get a much easier way:
>
> Here's the markdown you'd have
>
> # Environments
>
> * `debian`
> * `i386`
> * `x64`
> * `redhat`
> * `i386`
> * `x64`
> * `windows`
>
> # Build
>
> * `debian`
>
> whatever command you use to make the deb
>
> * `redhat`
>
> rpmbuild blah blah blah
>
> * `windows`
>
> whatever command you use to make the exe
>

Sooo... Why didn't Jenkins do it that way from the start (at least
allowing the job definition to be stored/versioned as part of the
project)?

--
Les Mikesell
lesmi...@gmail.com

[Roland Asmann]

I'm not quite sure what you mean with 'literate builds'... Do you mean splitting them in multiple jobs?

[Stephen Connolly]

Because you probably don't want to couple your source code too strongly to your SCM.

Of course the breakthrough from my PoV is that you always want a README in SCM that says what the build command(s) is/are (hopefully it's a single command, but sometimes you need a couple of them)

Hijacking the README to infer the build command thus is much better IMHO than reading an 80 page spec and deciphering a pile of different "magic" defaults based on what values you put in a config file.

The .jenkins directory is thus more about giving Jenkins some metadata about the build output, it doesn't interfere if you want to change CI system, and it is not critical for a build+test verification build. If you want to try another CI system the .jenkins directory should not get in your way.

Finally the marker file is all about identifying the branches to build without having to visit the UI... And since we have that file we can use it (if non empty) as a replacement for the README if you want to tweak the CI build

From my PoV this is all obvious *now* but that's the thing about good ideas, they are always obvious when you look back after discovering them.

There are a lot more subtleties to literate but I don't think people are quite ready for the brain melt if I start explaining them now ;-)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Sent from my phone

[Stephen Connolly]

http://jenkins-ci.org/content/literate-builds-wtf

[Curtis Kline]

Roland, have you looked at the Build Flow plugin? I am using that instead of the matrix builds in some circumstances. 

Curtis

[Les Mikesell]

On Thu, Oct 3, 2013 at 12:53 PM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
> Because you probably don't want to couple your source code too strongly to
> your SCM.

How so? Requiring different build commands/options/targets seems very
normal across the same code's version history and branches.

> Of course the breakthrough from my PoV is that you always want a README in
> SCM that says what the build command(s) is/are (hopefully it's a single
> command, but sometimes you need a couple of them)
>
> Hijacking the README to infer the build command thus is much better IMHO
> than reading an 80 page spec and deciphering a pile of different "magic"
> defaults based on what values you put in a config file.

Putting it in README is a nice touch and invokes the 'literate
programming' concept, but the functionality of controlling the build
and targets within the versioned project is the part that intrigues
me.

> The .jenkins directory is thus more about giving Jenkins some metadata about
> the build output, it doesn't interfere if you want to change CI system, and
> it is not critical for a build+test verification build. If you want to try
> another CI system the .jenkins directory should not get in your way.

I haven't gotten that far yet. And maybe I can't on the stable builds
- I don't see it as an available plugin.

> Finally the marker file is all about identifying the branches to build
> without having to visit the UI... And since we have that file we can use it
> (if non empty) as a replacement for the README if you want to tweak the CI
> build

How do you find 'branches' in subversion which doesn't really have
branches - or more accurately, treats any place you copy something as
a branch?

> From my PoV this is all obvious *now* but that's the thing about good ideas,
> they are always obvious when you look back after discovering them.
>
> There are a lot more subtleties to literate but I don't think people are
> quite ready for the brain melt if I start explaining them now ;-)

This thread probably isn't the right place, but I've always preferred
the 'think first, then act' approach and would like to see that
explanation. In particular, how direct/complete is the relationship
between what you can pass in the controlling file and the full jenkins
api? If you want to do anything unusual, will you end up doing it in
groovy anyway?

--
Les Mikesell
lesmi...@gmail.com

[Stephen Connolly]

On Thursday, 3 October 2013, Les Mikesell wrote:

On Thu, Oct 3, 2013 at 12:53 PM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
> Because you probably don't want to couple your source code too strongly to
> your SCM.

How so?  Requiring different build commands/options/targets seems very
normal across the same code's version history and branches.

Typo I meant coupling your source code to your CI too tightly is not a good thing... I think Jenkins is best, but I want users to reach that conclusion themselves by trying Jenkins and other options.

 

> Of course the breakthrough from my PoV is that you always want a README in
> SCM that says what the build command(s) is/are (hopefully it's a single
> command, but sometimes you need a couple of them)
>
> Hijacking the README to infer the build command thus is much better IMHO
> than reading an 80 page spec and deciphering a pile of different "magic"
> defaults based on what values you put in a config file.

Putting it in README is a nice touch and invokes the 'literate
programming' concept, but the functionality of controlling the build
and targets within the versioned project is the part that intrigues
me.

> The .jenkins directory is thus more about giving Jenkins some metadata about
> the build output, it doesn't interfere if you want to change CI system, and
> it is not critical for a build+test verification build. If you want to try
> another CI system the .jenkins directory should not get in your way.

I haven't gotten that far yet.  And maybe I can't on the stable builds
- I don't see it as an available plugin.

Any plugin with alpha or beta is hidden from the regular update centres... You need to configure the experimental update centre to see them

 

> Finally the marker file is all about identifying the branches to build
> without having to visit the UI... And since we have that file we can use it
> (if non empty) as a replacement for the README if you want to tweak the CI
> build

How do you find 'branches' in subversion which doesn't really have
branches - or more accurately, treats any place you copy something as
a branch?

You declare a root and a set of patterns to search, if the marker file is found => it's a branch

Works quite well and ends up being faster than you think due to how I prune the path search.

Warning the beta version of the subversion plugin does not migrate credentials fully yet, so not advised to upgrade your production server to the 2.0 version of subversion plugin *yet* 

> From my PoV this is all obvious *now* but that's the thing about good ideas,
> they are always obvious when you look back after discovering them.
>
> There are a lot more subtleties to literate but I don't think people are
> quite ready for the brain melt if I start explaining them now ;-)

This thread probably isn't the right place, but I've always preferred
the 'think first, then act' approach and would like to see that
explanation.    In particular, how direct/complete is the relationship
between what you can pass in the controlling file and the full jenkins
api?   If you want to do anything unusual, will you end up doing it in
groovy anyway?

Publishers are completely 1:1 mapped

Build steps turn into shell commands (or batch on windows)

Heavy intra-build coordination... I have a different idea for that... Likely my different idea will not get approval for being OSSed (hey I'm happy to have set this one free)

Maven stuff... I have a strong opinion on the right way to solve the maven problem... It will be quite elegant and work well with literate... Just have a few small tweaks first

After that most of the interactions will need branch properties provided by the plugins so things like build wrappers and job properties are not defined in the literate build... I may find ways to pull them in but it needs thinking or the solution will uglify

[Roland Asmann]

Thanks Curtis, I'll check it out!

> <mailto:jenkinsci-users%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>

> --
> You received this message because you are subscribed to a topic in the

> Google Groups "Jenkins Users" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/0ujfEFyJ5Xs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Les Mikesell]

On Thu, Oct 3, 2013 at 2:24 PM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
>
>> How so? Requiring different build commands/options/targets seems very
>> normal across the same code's version history and branches.
>
>
> Typo I meant coupling your source code to your CI too tightly is not a good
> thing... I think Jenkins is best, but I want users to reach that conclusion
> themselves by trying Jenkins and other options.

Sure, but having the files there doesn't force you to use jenkins -
anything else would just ignore them.

>> > From my PoV this is all obvious *now* but that's the thing about good
>> > ideas,
>> > they are always obvious when you look back after discovering them.
>> >
>> > There are a lot more subtleties to literate but I don't think people are
>> > quite ready for the brain melt if I start explaining them now ;-)
>>
>> This thread probably isn't the right place, but I've always preferred
>> the 'think first, then act' approach and would like to see that
>> explanation. In particular, how direct/complete is the relationship
>> between what you can pass in the controlling file and the full jenkins
>> api? If you want to do anything unusual, will you end up doing it in
>> groovy anyway?
>
>
> Publishers are completely 1:1 mapped
>
> Build steps turn into shell commands (or batch on windows)

That brain melt thing you mentioned is starting to happen. Is there,
or can there be, a groovy build step option?

> Heavy intra-build coordination... I have a different idea for that... Likely
> my different idea will not get approval for being OSSed (hey I'm happy to
> have set this one free)

Likewise, having an option to run a system groovy steps at the top
level would give you fine-grained control when you need it.

> After that most of the interactions will need branch properties provided by
> the plugins so things like build wrappers and job properties are not defined
> in the literate build... I may find ways to pull them in but it needs
> thinking or the solution will uglify

Abstractions are nice as long as the person who wrote them anticipated
exactly what you need to describe. But somehow that isn't always the
case, so having a way to work around missing parts would be a good
thing,

--
Les Mikesell
lesmi...@gmail.com

[Roland Asmann]

Quite a nice read! This might be a step in the right direction!

One question though: currently I control the whole build with a single
Perl script. It doesn't matter if I'm on Windows or Linux (eg it reads
env-vars I set on the nodes)... So how would I write that with literate
builds? Add another environment above the three you mentioned? Or is
there another way to describe the build without having to reference the
environments?

> from it, send an email to jenkinsci-use...@__googlegroups.com.
> For more options, visit
> https://groups.google.com/__groups/opt_out
> <https://groups.google.com/groups/opt_out>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-use...@googlegroups.com

> <javascript:_e({}, 'cvml',
> 'jenkinsci-users%2Bunsu...@googlegroups.com');>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> --
> Sent from my phone
>

> --
> You received this message because you are subscribed to a topic in the

> Google Groups "Jenkins Users" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/0ujfEFyJ5Xs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Stephen Connolly]

On Thursday, 3 October 2013, Les Mikesell wrote:

This is not the solution you want.

 

> After that most of the interactions will need branch properties provided by
> the plugins so things like build wrappers and job properties are not defined
> in the literate build... I may find ways to pull them in but it needs
> thinking or the solution will uglify

Abstractions are nice as long as the person who wrote them anticipated
exactly what you need to describe.  But somehow that isn't always the
case, so having a way to work around missing parts would be a good
thing,

I think you just haven't got the change fully absorbed

[Les Mikesell]

On Thu, Oct 3, 2013 at 3:21 PM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
>
>> Likewise, having an option to run a system groovy steps at the top
>> level would give you fine-grained control when you need it.
>
> This is not the solution you want.

Right, but that means something else has to provide all the options I need.

>> Abstractions are nice as long as the person who wrote them anticipated
>> exactly what you need to describe. But somehow that isn't always the
>> case, so having a way to work around missing parts would be a good
>> thing,
>
>
> I think you just haven't got the change fully absorbed

I'm sure I don't. But, suppose I want to collect a bunch of libraries
built on different platform/nodes into one tree in the master job for
an atomic commit into a subversion tag (to be subsequently used via
svn externals in other projects). Can I describe the layout of that
tree such that it might be different in a branch build than in other
versions? That may not be a reasonable thing to want to do, but
that's not quite the point here.

--
Les Mikesell
lesmi...@gmail.com

[Stephen Connolly]

On Thursday, 3 October 2013, Roland Asmann wrote:

Quite a nice read! This might be a step in the right direction!

One question though: currently I control the whole build with a single Perl script. It doesn't matter if I'm on Windows or Linux (eg it reads env-vars I set on the nodes)... So how would I write that with literate builds?

If the exact same command works across all platforms the build section is even simpler:

# Build

    perl blah blah blah

 You only specify bullets in the build section if you need different commands on different environments

The best literate builds will just have one command only

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Roland Asmann]

On 04.10.2013 0:10, Stephen Connolly wrote:
>
>
> On Thursday, 3 October 2013, Roland Asmann wrote:
>
> Quite a nice read! This might be a step in the right direction!
>
> One question though: currently I control the whole build with a
> single Perl script. It doesn't matter if I'm on Windows or Linux (eg
> it reads env-vars I set on the nodes)... So how would I write that
> with literate builds?
>
>
> If the exact same command works across all platforms the build section
> is even simpler:
>
> # Build
>
> perl blah blah blah
>
> You only specify bullets in the build section if you need different
> commands on different environments
>
> The best literate builds will just have one command only

Nice! I think I have to set up a test-instance of Jenkins and give this
plugin a testdrive! Thanks!