Where to place build/package/publish logic - devops-toolchain

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Where to place build/package/publish logic
Received-SPF: neutral (google.com: 209.85.219.53 is neither permitted nor denied by best guess record for domain of jhum...@jezhumble.net) client-ip=209.85.219.53;
MIME-Version: 1.0
Sender: jhum...@jezhumble.net
In-Reply-To: <3ECBD237-F3EC-4FF3-863B-5BE0AFFA0...@dtosolutions.com>
References: <CADMB+XbqeHKtU4D6yK4XxF0wo+y5DECb_3tCAa7rUusiwH2...@mail.gmail.com>
 <5f863eec-1a5a-403f-8da7-22daee42e28e@googlegroups.com> <CADMB+XbxtkuDc9+iruM=sYQPruGrp-eSeb=MXLV8fwiQp2-...@mail.gmail.com>
 <3ECBD237-F3EC-4FF3-863B-5BE0AFFA0...@dtosolutions.com>
From: Jez Humble <j...@jezhumble.net>
Date: Sat, 15 Sep 2012 21:23:06 -0700
Message-ID: <CAP3ToFvVDMB_RCgC7QbR930m=S7-5QR17f9=ARaA7Ca0kGn...@mail.gmail.com>
Subject: Re: Where to place build/package/publish logic
To: devops-toolchain@googlegroups.com
Content-Type: multipart/alternative; boundary=e89a8f6436f8b9b02d04c9ca049d

--e89a8f6436f8b9b02d04c9ca049d
Content-Type: text/plain; charset=ISO-8859-1

The answer is, essentially, don't use snapshots. Maven snapshots and
continuous delivery are fundamentally antithetical. I had this out with the
Maven community a while ago. You can read a summary here:
http://www.lucasward.net/2010/11/maven-and-continuous-delivery.html TLDR:
one of the fundamental premises of continuous delivery is that every build
is a potential release candidate.

There are a couple of articles describing how to do it right here:

http://blog.orfjackal.net/2012/08/continuous-delivery-with-maven-and-go.html
http://jamesbetteley.wordpress.com/2012/02/21/continuous-delivery-using-maven/

Thanks,

Jez.

On 15 September 2012 21:13, Anthony Shortland <anth...@dtosolutions.com>wrote:

> We've done a lot of work pushing system packages (like RPMs) into Maven
> repositories and have found it possible to reconcile Maven versioning
> conventions with those of the system. e.g. The RPM Maven plug-in<http://mojo.codehaus.org/rpm-maven-plugin/> is
> a thing of beauty and a joy forever allowing you to adopt or override
> conventional versioning strategies (the nexus-yum-plugin<http://code.google.com/p/nexus-yum-plugin/> is
> another key point of integration, by the way).
>
> The key is to differentiate between planning dependency resolution for
> build artifacts vs deployable artifacts, in my opinion.
>
> That said, does anyone have an opinion about reconciling the Maven
> snapshot/release repository convention with the needs of continuous
> deployment? I'm facing that challenge right now.
>
> Anthony.
>
> On Sep 15, 2012, at 10:17 AM, Adam Rosien <a...@rosien.net> wrote:
>
> Thanks for the reply. Having CI/CD/Jenkins/"process" inform the source
> repo of the environment is a great way to think about the whole system.
>
> For my particular use case I have a deployment-manager service which has a
> maven repo upstream, but I'm trying to wedge non-maven project artifacts
> into the process, which means adopting the maven world of versioning and
> repositories (the two are closely coupled, versions map to URL paths) to
> project types that aren't traditionally "maven-ish", that is, language X
> packaged as a tarball, etc. This may or may not be a good idea.
>
> Perhaps it may be better to adapt the deployment-manager to other
> versioning/repo schemes instead of requiring things in the maven way (which
> I find mostly sane).
>
> .. Adam
>
> On Sat, Sep 15, 2012 at 6:58 AM, mithun <m0t0rbr...@gmail.com> wrote:
>
>> I guess the correct answer to your questions is "It depends ..." (pun
>> unintended).
>>
>> Typically, the source repo _should_ know how to build, test and package -
>> just because every source repo can have different requirements to do so.
>> However, the source repo should also rely on the "process" (Jenkins,
>> wrapper scripts, etc ...) for assistance on the environment. For e.g, Is
>> this a development vs. production build? What is the test-passing
>> threshold? Which database is the application talking to? etc .
>>
>> The source repo should also be responsible for it's own versioning
>> scheme. In the Java/maven world, this is automatically taken care of (via
>> snapshot/release builds). From my experience, it is difficult to dictate a
>> consistent versioning scheme across different source repos.
>>
>> The source repo should not care about it's "publishing". The "process"
>> should decide where an artifact gets published. The process should also
>> help the source repo with the artifact repository so that dependencies can
>> be resolved during the build/test/package stage.
>>
>> > The downside to this approach is that now there are a bunch of
>> dependencies between the project and Jenkins in order to complete the whole
>> pipeline
>>
>> Whether this is a downside or not is debatable. Are you trying to
>> establish a "standard" pipeline that multiple source repos can use? Or
>> trying to setup a Jenkins job for one source repo? If it is the later, then
>> yes, its a downside with maintenance overhead. If it is the former, then I
>> believe this is an advantage.
>>
>> -- Mithun
>>
>> On Friday, September 14, 2012 12:51:48 PM UTC-5, Adam Rosien wrote:
>>>
>>> I'm using Jenkins for CI and I'm debating where to place various logic
>>> (as scripts, etc.) for building, packaging, versioning and publishing.
>>>
>>> For example, in a Java/Scala maven/sbt world, you can perform all of
>>> these tasks with maven/sbt itself using the project descriptors (.pom,
>>> build.sbt, etc.) and various plugins (assembly to make a "fat" jar, etc.).
>>> An important feature of this style is that the entire pipeline of
>>> build/package/version/publish is self-contained in the source repo itself,
>>> and Jenkins really only needs to run a single command to perform all these
>>> steps.
>>>
>>> For other kinds of projects there aren't necessarily "one-stop-shop"
>>> solutions for this pipeline. One constant debate I have with folks is that
>>> "source code shouldn't know where the artifact repository is", or "a
>>> project shouldn't manage its versioning mechanism", etc.
>>>
>>> One could imagine giving Jenkins the responsibility for any of these
>>> steps. For example, the project itself is responsible for supplying the
>>> build and test machinery, but Jenkins builds the artifact and uploads it to
>>> some artifact repository. The downside to this approach is that now there
>>> are a bunch of dependencies between the project and Jenkins in order to
>>> complete the whole pipeline, and the extra dependencies magnify the
>>> downside when there are problems, i.e., somebody breaks the script that
>>> figures out the new version number of a project and all projects can't get
>>> a new version.
>>>
>>> Can you all help me find good principles and structures for assigning
>>> the responsibilities of build/package/version/publish?
>>>
>>> Thanks,
>>>
>>> .. Adam
>>>
>>>
>>>
>>
>
>


-- 
Jez Humble
Co-author, *Continuous Delivery <http://continuousdelivery.com/>*
http://continuousdelivery.com/
http://jezhumble.net/

--e89a8f6436f8b9b02d04c9ca049d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The answer is, essentially, don&#39;t use snapshots. Maven snapshots and co=
ntinuous delivery are fundamentally antithetical.=A0I had this out with the=
 Maven community a while ago. You can read a summary here:=A0<a href=3D"htt=
p://www.lucasward.net/2010/11/maven-and-continuous-delivery.html">http://ww=
w.lucasward.net/2010/11/maven-and-continuous-delivery.html</a> TLDR: one of=
 the fundamental premises of continuous delivery is that every build is a p=
otential release candidate.<div>

<br></div><div>There are a couple of articles describing how to do it right=
 here:</div><div><br></div><div><a href=3D"http://blog.orfjackal.net/2012/0=
8/continuous-delivery-with-maven-and-go.html">http://blog.orfjackal.net/201=
2/08/continuous-delivery-with-maven-and-go.html</a></div>

<div><a href=3D"http://jamesbetteley.wordpress.com/2012/02/21/continuous-de=
livery-using-maven/">http://jamesbetteley.wordpress.com/2012/02/21/continuo=
us-delivery-using-maven/</a></div><div><br></div><div>Thanks,</div><div>
<br>
</div><div>Jez.<br><br><div class=3D"gmail_quote">On 15 September 2012 21:1=
3, Anthony Shortland <span dir=3D"ltr">&lt;<a href=3D"mailto:anthony@dtosol=
utions.com" target=3D"_blank">anth...@dtosolutions.com</a>&gt;</span> wrote=
:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word">We&#39;v=
e done a lot of work pushing system packages (like RPMs) into Maven reposit=
ories and have found it possible to reconcile Maven versioning conventions =
with those of the system. e.g. The=A0<a href=3D"http://mojo.codehaus.org/rp=
m-maven-plugin/" target=3D"_blank">RPM Maven plug-in</a>=A0is a thing of be=
auty and a joy forever allowing you to adopt or override conventional versi=
oning strategies (the=A0<a href=3D"http://code.google.com/p/nexus-yum-plugi=
n/" target=3D"_blank">nexus-yum-plugin</a>=A0is another key point of integr=
ation, by the way).<div>

<br></div><div>The key is to differentiate between planning dependency reso=
lution for build artifacts vs deployable artifacts, in my opinion.=A0</div>=
<div><br></div><div>That said, does anyone have an opinion about reconcilin=
g the Maven snapshot/release repository convention with the needs of contin=
uous deployment? I&#39;m facing that challenge right now.</div>

<span class=3D"HOEnZb"><font color=3D"#888888"><div><br></div><div>Anthony.=
</div></font></span><div><div class=3D"h5"><div><br><div><div>On Sep 15, 20=
12, at 10:17 AM, Adam Rosien &lt;<a href=3D"mailto:a...@rosien.net" target=
=3D"_blank">a...@rosien.net</a>&gt; wrote:</div>

<br><blockquote type=3D"cite">Thanks for the reply. Having CI/CD/Jenkins/&q=
uot;process&quot; inform the source repo of the environment is a great way =
to think about the whole system.<div><br></div><div>For my particular use c=
ase I have a deployment-manager service which has a maven repo upstream, bu=
t I&#39;m trying to wedge non-maven project artifacts into the process, whi=
ch means adopting the maven world of versioning and repositories (the two a=
re closely coupled, versions map to URL paths) to project types that aren&#=
39;t traditionally &quot;maven-ish&quot;, that is, language X packaged as a=
 tarball, etc. This may or may not be a good idea.</div>


<div><br></div><div>Perhaps it may be better to adapt the deployment-manage=
r to other versioning/repo schemes instead of requiring things in the maven=
 way (which I find mostly sane).</div><div><br></div><div>.. Adam<br><br>


<div class=3D"gmail_quote">On Sat, Sep 15, 2012 at 6:58 AM, mithun <span di=
r=3D"ltr">&lt;<a href=3D"mailto:m0t0rbr...@gmail.com" target=3D"_blank">m0t=
0rbr...@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


I guess the correct answer to your questions is &quot;It depends ...&quot; =
(pun unintended).<div><br></div><div>Typically, the source repo _should_ kn=
ow how to build, test and package - just because every source repo can have=
 different requirements to do so. However, the source repo should also rely=
 on the &quot;process&quot; (Jenkins, wrapper scripts, etc ...) for assista=
nce on the environment. For e.g, Is this a development vs. production build=
? What is the test-passing threshold? Which database is the application tal=
king to? etc .</div>


<div><br></div><div>The source repo should also be responsible for it&#39;s=
 own versioning scheme. In the Java/maven world, this is automatically take=
n care of (via snapshot/release builds). From my experience, it is difficul=
t to dictate a consistent versioning scheme across different source repos.<=
/div>


<div><br></div><div>The source repo should not care about it&#39;s &quot;pu=
blishing&quot;. The &quot;process&quot; should decide where an artifact get=
s published. The process should also help the source repo with the artifact=
 repository so that dependencies can be resolved during the build/test/pack=
age stage.</div>


<div><div><br></div><div>&gt;=A0The downside to this approach is that now t=
here are a bunch of dependencies between the project and Jenkins in order t=
o complete the whole pipeline</div><div><br></div></div><div>Whether this i=
s a downside or not is=A0debatable. Are you trying to establish a &quot;sta=
ndard&quot; pipeline that multiple source repos can use? Or trying to setup=
 a Jenkins job for one source repo? If it is the later, then yes, its a dow=
nside with maintenance overhead. If it is the former, then I believe this i=
s an advantage.</div>


<span><font color=3D"#888888"><div><br></div><div>-- Mithun</div></font></s=
pan><div><div><br>On Friday, September 14, 2012 12:51:48 PM UTC-5, Adam Ros=
ien wrote:<blockquote class=3D"gmail_quote" style=3D"margin:0;margin-left:0=
.8ex;border-left:1px #ccc solid;padding-left:1ex">


I&#39;m using Jenkins for CI and I&#39;m debating where to place various lo=
gic (as scripts, etc.) for building, packaging, versioning and publishing.=
=A0<div><br></div><div>For example, in a Java/Scala maven/sbt world, you ca=
n perform all of these tasks with maven/sbt itself using the project descri=
ptors (.pom, build.sbt, etc.) and various plugins (assembly to make a &quot=
;fat&quot; jar, etc.). An important feature of this style is that the entir=
e pipeline of build/package/version/publish is self-contained in the source=
 repo itself, and Jenkins really only needs to run a single command to perf=
orm all these steps.</div>



<div><br></div><div>For other kinds of projects there aren&#39;t necessaril=
y &quot;one-stop-shop&quot; solutions for this pipeline. One constant debat=
e I have with folks is that &quot;source code shouldn&#39;t know where the =
artifact repository is&quot;, or &quot;a project shouldn&#39;t manage its v=
ersioning mechanism&quot;, etc.=A0</div>



<div><br></div><div>One could imagine giving Jenkins the responsibility for=
 any of these steps. For example, the project itself is responsible for sup=
plying the build and test machinery, but Jenkins builds the artifact and up=
loads it to some artifact repository. The downside to this approach is that=
 now there are a bunch of dependencies between the project and Jenkins in o=
rder to complete the whole pipeline, and the extra dependencies magnify the=
 downside when there are problems, i.e., somebody breaks the script that fi=
gures out the new version number of a project and all projects can&#39;t ge=
t a new version.</div>



<div><br></div><div>Can you all help me find good principles and structures=
 for assigning the responsibilities of build/package/version/publish?</div>=
<div><br></div><div>Thanks,</div><div><br></div><div>.. Adam</div><div>



<br></div><div>=A0</div>
</blockquote></div></div></blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br c=
lear=3D"all"><div><br></div>-- <br>Jez Humble<div>Co-author, <i><a href=3D"=
http://continuousdelivery.com/" target=3D"_blank">Continuous Delivery</a></=
i></div>

<div><a href=3D"http://continuousdelivery.com/" target=3D"_blank">http://co=
ntinuousdelivery.com/</a></div><div><a href=3D"http://jezhumble.net/" targe=
t=3D"_blank">http://jezhumble.net/</a></div><div><br></div><br>
</div>

--e89a8f6436f8b9b02d04c9ca049d--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
