Perform a single svn checkout in a multi-configuration project

[Fernando Cerezal]

Hello,
I am configuring a Jenkins server and I dont know, if possible, how to
configure correctly svn checkout.

I have a multiexecutable project. Building steps are:
Checkout entire repository
Build per project
Test per project.

So, I configured a multi-configuration project with an user-axis named
"project" for iterating. However, instead of a single entire svn
checkout, Jenkins performs a checkout in workspace, it creates a
"project" directory, it creates a directory for each project and,
inside each project directory, checkout again entire directory.
I can't perform checkout iteratively because are interrelated
projects, I need the code of two or three projects at the same time
before I could build any of them.

Is there any way to perform an unique checkout to workspace?

Thank you in advance.

[Les Mikesell]

Multi-configuration jobs are generally used to build in parallel
across a different set of platforms or environments. If your job is
just one set of serialized steps why can't you just use a job with
several build steps or your own build script that does them all -
perhaps using svn externals to get all the code components? Or
organize things so the artifacts from one build are available to the
next.

--
Les Mikesell
lesmi...@gmail.com

[Fernando Cerezal]

2013/2/5 Les Mikesell <lesmi...@gmail.com>:

Thank you for answering.

Really, there are several projects, with their own tests each one, but
all of them are folders of the same svn repository. For a multipart
project, jenkins generate reports automatically for every subproject.
The problem is it downloads n+1 times the entire repository for n
projects.

Solution can be so simple as don't use axis on checkout, if it is possible.

Regards.

>
> --
> Les Mikesell
> lesmi...@gmail.com
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[nicolas de loof]

2013/2/6 Fernando Cerezal <kryp...@gmail.com>

2013/2/5 Les Mikesell <lesmi...@gmail.com>:
> On Tue, Feb 5, 2013 at 6:33 AM, Fernando Cerezal <kryp...@gmail.com> wrote:
>> Is there any way to perform an unique checkout to workspace?
>
> Multi-configuration jobs are generally used to build in parallel
> across a different set of platforms or environments.  If your job is
> just one set of serialized steps why can't you just use a job with
> several build steps or your own build script that does them all -
> perhaps using svn externals to get all the code components?  Or
> organize things so the artifacts from one build are available to the
> next.


Thank you for answering.

Really, there are several projects, with their own tests each one, but
all of them are folders of the same svn repository. For a multipart
project, jenkins generate reports automatically for every subproject.
The problem is it downloads n+1 times the entire repository for n
projects.

Solution can be so simple as don't use axis on checkout, if it is possible.

You mean, cloning "head" workspace for axe builds ? Would this really be more efficient than a fresh checkout ?

[Les Mikesell]

On Wed, Feb 6, 2013 at 6:44 AM, Fernando Cerezal <kryp...@gmail.com> wrote:
>
>>> Is there any way to perform an unique checkout to workspace?
>>
>> Multi-configuration jobs are generally used to build in parallel
>> across a different set of platforms or environments. If your job is
>> just one set of serialized steps why can't you just use a job with
>> several build steps or your own build script that does them all -
>> perhaps using svn externals to get all the code components? Or
>> organize things so the artifacts from one build are available to the
>> next.
>
>
> Thank you for answering.
>
> Really, there are several projects, with their own tests each one, but
> all of them are folders of the same svn repository.

That doesn't quite mesh with the description of each one needing
access to the others contents.

> For a multipart
> project, jenkins generate reports automatically for every subproject.
> The problem is it downloads n+1 times the entire repository for n
> projects.
>
> Solution can be so simple as don't use axis on checkout, if it is possible.

Assuming you have enough disk space, if your job check-out strategy is
'use svn update', after the first run it will just download the
changes. Perhaps not ideal, but it shouldn't take a lot of bandwidth
between runs unless you are making huge changes or checking in
binaries.

--
Les Mikesell
lesmi...@gmail.com

[Fernando Cerezal]

2013/2/6 Les Mikesell <lesmi...@gmail.com>:

> On Wed, Feb 6, 2013 at 6:44 AM, Fernando Cerezal <kryp...@gmail.com> wrote:
>>
>>>> Is there any way to perform an unique checkout to workspace?
>>>
>>> Multi-configuration jobs are generally used to build in parallel
>>> across a different set of platforms or environments. If your job is
>>> just one set of serialized steps why can't you just use a job with
>>> several build steps or your own build script that does them all -
>>> perhaps using svn externals to get all the code components? Or
>>> organize things so the artifacts from one build are available to the
>>> next.
>>
>>
>> Thank you for answering.
>>
>> Really, there are several projects, with their own tests each one, but
>> all of them are folders of the same svn repository.
>
> That doesn't quite mesh with the description of each one needing
> access to the others contents.

All of them are libraries, with executables for testing. Several of
them uses each other.

>
>> For a multipart
>> project, jenkins generate reports automatically for every subproject.
>> The problem is it downloads n+1 times the entire repository for n
>> projects.
>>
>> Solution can be so simple as don't use axis on checkout, if it is possible.
>
> Assuming you have enough disk space, if your job check-out strategy is
> 'use svn update', after the first run it will just download the
> changes. Perhaps not ideal, but it shouldn't take a lot of bandwidth
> between runs unless you are making huge changes or checking in
> binaries.

This is not present behaviour. It creates a directory in workspace
named like user-axis, inside ir creates a directory for every axis
value and inside each of them it performs a svn checkout. I only want
a checkout in workspace.


Thanks for your answer.

[Fernando Cerezal]

2013/2/6 nicolas de loof <nicolas...@gmail.com>:

>
> 2013/2/6 Fernando Cerezal <kryp...@gmail.com>
>>
>> 2013/2/5 Les Mikesell <lesmi...@gmail.com>:
>> > On Tue, Feb 5, 2013 at 6:33 AM, Fernando Cerezal <kryp...@gmail.com>
>> > wrote:
>> >> Is there any way to perform an unique checkout to workspace?
>> >
>> > Multi-configuration jobs are generally used to build in parallel
>> > across a different set of platforms or environments. If your job is
>> > just one set of serialized steps why can't you just use a job with
>> > several build steps or your own build script that does them all -
>> > perhaps using svn externals to get all the code components? Or
>> > organize things so the artifacts from one build are available to the
>> > next.
>>
>>
>> Thank you for answering.
>>
>> Really, there are several projects, with their own tests each one, but
>> all of them are folders of the same svn repository. For a multipart
>> project, jenkins generate reports automatically for every subproject.
>> The problem is it downloads n+1 times the entire repository for n
>> projects.
>>
>> Solution can be so simple as don't use axis on checkout, if it is
>> possible.
>
>
> You mean, cloning "head" workspace for axe builds ? Would this really be
> more efficient than a fresh checkout ?

I mean:

- svn checkout in workspace
- Use axis for build
- execute postbuild (without axis)

Currently it does:

- Use axis for svn checkout
- Use axis for build
- execute postbuild (without axis)

Thanks for your answer.

[Les Mikesell]

On Wed, Feb 6, 2013 at 8:46 AM, Fernando Cerezal <kryp...@gmail.com> wrote:
>>>
>>> Really, there are several projects, with their own tests each one, but
>>> all of them are folders of the same svn repository.
>>
>> That doesn't quite mesh with the description of each one needing
>> access to the others contents.
>
> All of them are libraries, with executables for testing. Several of
> them uses each other.

I think the typical way to handle this in subversion is to make each
project that builds separately have its own trunk/branches/tags tree
and every part that needs to include components from a separate area
would use svn externals to pull them into a checkout. That (a) lets
you either float the external references on the trunk for coordinated
development or pin them to tested/released versions as components
diverge and the project using them nears a release - and (b) minimizes
each checkout to the parts that are actually needed, whether by
jenkins or a developer.

>>
>> Assuming you have enough disk space, if your job check-out strategy is
>> 'use svn update', after the first run it will just download the
>> changes. Perhaps not ideal, but it shouldn't take a lot of bandwidth
>> between runs unless you are making huge changes or checking in
>> binaries.
>
> This is not present behaviour. It creates a directory in workspace
> named like user-axis, inside ir creates a directory for every axis
> value and inside each of them it performs a svn checkout. I only want
> a checkout in workspace.

Yes, that is the layout your job requests. But what I meant was that
on the 2nd and subsequent runs of this job, jenkins should be doing
an 'svn update' into those existing workspaces which should not be an
expensive operation even if you have to repeat it a few times. And,
if you ever want to version your components independently you really
will want separate copies for each build - so there is a reason for
jenkins' behavior.

--
Les Mikesell
lesmi...@gmail.com

[roeh...@googlemail.com]

I have what I take to be problem related to Fernando's. And I think the solution should be the same. So here is my explanation.

I have essentially one svn repository. Its checkout is used for a multi configuration job. The user axes provide variants of my Ant build. The build is part of the checkout. It knows how to handle these variants itself, if provided with the current axes configuration as Ant properties, which the multi configuration job promised to do.

By default, Jenkins checks out the svn repository N times. N being the number of all combinations, e.g. axis1 contains 2 items, axis2 contains 3 items, yielding 6 combinations.

This is a pain. Jenkins produces N checkouts, where I require only one. This results in overhead for the initial checkout as well as subsequent updates. How to fix that? I have found no solution so far elsewhere.

I think the typical way to handle this in subversion is to make each
project that builds separately have its own trunk/branches/tags tree
and every part that needs to include components from a separate area
would use svn externals to pull them into a checkout.

This seems too complicated for my use case. Do you agree?

Yes, that is the layout your job requests.  But what I meant was that
on the 2nd and subsequent runs of this job,  jenkins should be doing
an 'svn update' into those existing workspaces which should not be an
expensive operation even if you have to repeat it a few times.

Alright. So there just is no way to have a multi configuration job reuse the same checkout?

Regards,
Alexander

[Les Mikesell]

On Wed, Apr 24, 2013 at 8:14 AM, <roeh...@googlemail.com> wrote:

>>> I think the typical way to handle this in subversion is to make each
>> project that builds separately have its own trunk/branches/tags tree
>> and every part that needs to include components from a separate area
>> would use svn externals to pull them into a checkout.
>
>
> This seems too complicated for my use case. Do you agree?

No, any time you are re-using components, svn externals make a lot of
sense. And you should almost never have to check out the whole
trunk/branches/tags tree at once.

>> Yes, that is the layout your job requests. But what I meant was that
>> on the 2nd and subsequent runs of this job, jenkins should be doing
>> an 'svn update' into those existing workspaces which should not be an
>> expensive operation even if you have to repeat it a few times.
>
>
> Alright. So there just is no way to have a multi configuration job reuse the
> same checkout?

No, in the general case, the jobs will run on different slaves and may
be checked out with different native line endings. And they may run
in parallel so they can't use the same workspace. If you are really
just doing everything in one build on one host, why not write your own
script wrapper to handle the variations?

--
Les Mikesell
lesmi...@gmail.com

[Les Mikesell]

On Wed, Apr 24, 2013 at 10:50 AM, Les Mikesell <lesmi...@gmail.com> wrote:
>>>
>> Alright. So there just is no way to have a multi configuration job reuse the
>> same checkout?
>
> No, in the general case, the jobs will run on different slaves and may
> be checked out with different native line endings. And they may run
> in parallel so they can't use the same workspace. If you are really
> just doing everything in one build on one host, why not write your own
> script wrapper to handle the variations?
>

On a second thought - maybe you could make it work with "Use custom
workspace" and "." for the "directory for sub-builds" (see
https://issues.jenkins-ci.org/browse/JENKINS-8317). But you may need
to think about the checkout strategy (probably revert before update)
and use the 'run each configuration sequentially' option.

--
Les Mikesell
lesmi...@gmail.com

[Alexander Röhnsch]

Thanks for your suggestions Les. I'm trying them out.