modernizing the integrated build process

[AdrianCole]

The build process detailed in Integrated_build_instructions is
thorough and probably works fine. That said, such a complicated
manual process doesn't encourage community participation.

Being unpaid volunteers, the community generally want to get coding
fast and without the time sink of a complicated build process. A lot
of us are spoiled by "mvn install" simplicity. When a project takes a
lot of steps, copying, and configuration to build, some will lose
interest and not bother helping. We don't want that.

I'm sure the current build process was great a few years back when
there was no maven2. Nowadays, I'm sure we can do better.

Is there an existing plan to modernize the sh + maven + manual steps
process into maven2 or some other integrated solution? If not, what
do you think?

I noticed increased community participation is one of the 2009 goals.
This seems like a great place to start.

my 2p,
-Adrian Cole

[Alex-SF]

Hi Adrian,

Yes, the build system is pretty old, complicated and needs updating.
We've only lived with it so long because all the individual steps are
wrapped by a configuration of "Builder" objects that coordinate the
steps. On our build server we can run a single command to kick off the
whole process (ie, an InstallerBuilder "BuildAll" command kicks off
all the subordinate build steps). The only problem is this set of
Builder objects is currently not usable outside of our build server
and the Sourceforge web accounts where the intermediate artifacts are
staged.

The build process needs reengineering and we've only postponed it due
to lack of time. I agree having a clean, portable build process is
essential to wider developer participation.

Perhaps, we can use this thread to hash out approaches to getting the
build process cleaned up.

Thanks

[Anthony Shortland]

This is an excellent idea and well overdue!

One guideline I'd propose is that the build(s) should be structured to work equally well without ControlTier as with it. In other words, that even if the release builds to Sourceforge are automated, it should be possible for a developer to conveniently build without using ControlTier.

Experience shows that, paradoxically, this not only yields a more flexible setup, but also a cleaner ControlTier integration!

Additionally, since CI has become an integral part of our best-practice implementations, I'd propose we use this opportunity to introduce it for the ControlTier builds.

How might these suggestions drive the restructuring? 

Anthony.

[Adrian Cole]

Interesting point, Anthony.  I suppose vanilla modularization may be one way to attack the problem.  If we forget that ControlTier is "in the business" and think of it as a composite suite of applications and modules, standard practices could follow.   We could keep the current ControlTier process and put up a new one in parallel and in the same source tree.  This may well be maven2/hudson or the like.  When everything is finished and with normal CI, the old process can be decommissioned, or ported to use the maven2 infrastructure.

What do you think?

-Adrian

[Alex-SF]

I think there would need to be several CI projects. This would at
least be three, one to monitor the checkins for each of the
subprojects (controltier, ctl-dispatch, webad, moduleforge). In fact,
there are more than these three build projects that potentially could
be managed as CI projects. Within the "controltier" Sourceforge
project there are actually four:

* common
* commander
* itnav
* reportcenter.

Integrated builds exist at different levels too where there are build
dependencies. For example, the CTL distribution includes ctl-dispatch,
commander, common, itnav and moduleforge. A CI project could be
maintained for each subcomponent, yielding one of the artifacts later
consumed by an installer/bundle build.

I think it's worth making the clarification that ControlTier is more
like a software distribution than it is a single project and hence
isn't a simple single source repo + build procedure.

[Adrian Cole]

Thanks for the clarification, Alex.  Sounds like the ControlTier part may be implementable as a maven2 assembly in that case. 

From a CI perspective, It may make sense to have a separate CI monitor for each module regardless of whether it is a top-level project or a sub-module.  These can be configured to go off based on version control updates and updates to their dependencies. 

For starters, there could be a single tier of CI instances.  Engineering further, there could be a second tier that performs integration tests on modules whose artifacts passed unit tests.

[Anthony Shortland]

The driver for the number of CI projects (at least with my experience with CruiseControl) is the structure of the source base and build scripts: a 1:1 relationship.

Perhaps the ControlTier source base is too fragmented? Starting with the fact that it's spread across more than one Sourceforge project, not just the ControlTier project.

Since the system ultimately works as the sum of its parts, should we consolidate the Soureforge projects as a starting point?

Anthony.

[Adrian Cole]

One project tree could certainly help, if nothing else keep issues coherent.

Unless the module structure doesn't make sense, it's probably fine to keep it modularized.  I've been working in Cargo, which has a very well organized maven2 module structure of about a dozen small modules.  It actually doesn't complicate things to have more modules when they are carefully put together.  In fact, modularization helps, as unit testing your changes happens much faster, and you are less likely to break other stuff.

my 2p.

[Alex-SF]

Would be nice to have a long term plan to upgrade it that would allow
us to gradually improve the process. For example, we could upgrade to
maven2 on a per-project basis.
Then once the builds are upgraded we could then take advantage of the
new underlying tooling.

On Feb 23, 10:14 am, Adrian Cole <fernc...@gmail.com> wrote:
> One project tree could certainly help, if nothing else keep issues coherent.
>
> Unless the module structure doesn't make sense, it's probably fine to keep
> it modularized.  I've been working in Cargo, which has a very well organized
> maven2 module structure of about a dozen small modules.  It actually doesn't
> complicate things to have more modules when they are carefully put
> together.  In fact, modularization helps, as unit testing your changes
> happens much faster, and you are less likely to break other stuff.
>
> my 2p.
>
> On Mon, Feb 23, 2009 at 6:01 PM, Anthony Shortland

> <anth...@controltier.com>wrote:

[Adrian Cole]

I second the notion, Alex. 

Why don't we raise issues to convert each of the existing project.xml to pom.xml.  If we have one issue per project.xml, it will help us to not step on each-others' toes. 

Note that some project.xml files may be better split into two poms, if they are dual purpose.  For example, if one pom creates libraries, then creates a distribution tree, it may be better to split that into two: one for the code, and one for the distribution.  We can raise those sorts of things to the list as we encounter them.

When the poms are done, there will be a second round of work to connect these projects to eachother via dependencies so that there is no copying of jars from here to there.

Does this seem like a good way to kick-start the effort?

Cheers,
-Adrian

[Greg Schueler]

I think this is the progression from easiest to hardest to convert:

1. common libs ($CTIERSVN/common)

   - generates 2 jar files, so should be split into 2 pom.xml files.

2. CTL 

   - we need to generate 1 simple .jar for classes, and:

   - distribution tree of CTL, in both .tgz, and .zip.  Actually perhaps we can agree to just go with .zip?  we include the expanded CTL content in the Installer, so no real need for 2 forms of it.

   - (p.s. the "/bundle" build-step can I think be totally deprecated as of 3.4+ since we now expand controltier-seed/coreutils at install time.)

3. controltier-seed ($MODULEFORGESVN/controltier) 

   - we actually way back when did create a maven2 plugin to build seed.jar files.  Would have to resurrect this.  I think it's in the common source. (Depends on ctl/common)

4. commander-extension ($CTIERSVN/commander)

   - 1 .jar of classes

   - 1 extension.jar.  I think this can be a pretty simple jar of a distribution tree.

5. coreutils-extension.  similar to commander-extension, but also includes a module build step like the controltier-seed

6. Workbench ($CTIERSVN/workbench)

   - builds a .war, I don't think there's any special build steps that we need to keep

7. Installer ($CTIERSVN/installer)

   - builds two forms: the .zip distribution, and the IZPack .jar installer.  Haven't looked into whether IZPack supports a maven2 build process.

8. Both of the 2 Grails based applications: Jobcenter, and Reportcenter.

   - I think Grails 1.1 supports maven2 based builds, but not the 1.0.x versions that we use, so we would have to upgrade these to the newest grails.  Not difficult, but requires testing to be sure things don't break.

--
Greg Schueler

ControlTier Software, Inc
gr...@controltier.com
650-292-9660x709

http://www.controltier.com

[Adrian Cole]

Great analysis, Greg.  It seems there are a lot of independent tasks to divy up.  From the list below, can you highlight the artifacts (results of pom.xml) which are inputs to other poms? (dependencies). 

Cheers,
-Adrian

[Greg Schueler]

inline comments:

On Fri, Feb 27, 2009 at 10:54 AM, Adrian Cole <fern...@gmail.com> wrote:

Great analysis, Greg.  It seems there are a lot of independent tasks to divy up.  From the list below, can you highlight the artifacts (results of pom.xml) which are inputs to other poms? (dependencies). 

Cheers,
-Adrian

On Fri, Feb 27, 2009 at 6:08 PM, Greg Schueler <gr...@controltier.com> wrote:

I think this is the progression from easiest to hardest to convert:

1. common libs ($CTIERSVN/common)

   - generates 2 jar files, so should be split into 2 pom.xml files.

 

generates: ctier-common.jar, ctier-common-vocabulary.jar 

2. CTL 

   - we need to generate 1 simple .jar for classes, and:

   - distribution tree of CTL, in both .tgz, and .zip.  Actually perhaps we can agree to just go with .zip?  we include the expanded CTL content in the Installer, so no real need for 2 forms of it.

   - (p.s. the "/bundle" build-step can I think be totally deprecated as of 3.4+ since we now expand controltier-seed/coreutils at install time.)

 

generates: ctl.jar, ctl.tgz/.zip

3. controltier-seed ($MODULEFORGESVN/controltier) 

   - we actually way back when did create a maven2 plugin to build seed.jar files.  Would have to resurrect this.  I think it's in the common source. (Depends on ctl/common)

 

depends-on: ctl.jar, ctier-common.jar, ctier-common-vocabulary.jar 

generates: controltier-seed.jar

4. commander-extension ($CTIERSVN/commander)

   - 1 .jar of classes

   - 1 extension.jar.  I think this can be a pretty simple jar of a distribution tree.

 

depends-on: ctl.jar, ctier-common.jar, ctier-common-vocabulary.jar, controltier-seed.jar

generates: commander.jar, commander-extension.jar 

5. coreutils-extension.  similar to commander-extension, but also includes a module build step like the controltier-seed

 

depends-on: ctl.jar, ctier-common.jar, ctier-common-vocabulary.jar

generates: coreutils-extension.jar 

6. Workbench ($CTIERSVN/workbench)

   - builds a .war, I don't think there's any special build steps that we need to keep

 

depends-on: ctl.jar, ctier-common.jar, ctier-common-vocabulary.jar, commander.jar

generates: itnav.war 

7. Installer ($CTIERSVN/installer)

   - builds two forms: the .zip distribution, and the IZPack .jar installer.  Haven't looked into whether IZPack supports a maven2 build process.

 

generates: ControlTier-Installer.jar, ControlTier.zip

depends-on: itnav.war, coreutils-extension.jar, commander-extension.jar, ctl.zip, reportcenter.zip, jobcenter.zip 

8. Both of the 2 Grails based applications: Jobcenter, and Reportcenter.

   - I think Grails 1.1 supports maven2 based builds, but not the 1.0.x versions that we use, so we would have to upgrade these to the newest grails.  Not difficult, but requires testing to be sure things don't break.

 

Jobcenter:

depends-on: ctl.jar, ctier-common.jar, ctier-common-vocabulary.jar, commander.jar

generates: jobcenter.zip (&.war)

Reportcenter:

depends-on: ctier-common.jar

generates: reportcenter.zip (&.war) 

[Adrian Cole]

Looks good, Greg.

For common libs, does the following structure make sense?

$CTIERSVN/common/pom.xml <- creates ctier-common.jar
$CTIERSVN/common/vocabulary/pom.xml <- creates ctier-common-vocabulary.jar

Cheers,
-Adrian