Trigger build on multiple paths

[sahendrickson]

I'd really like for bitten to trigger builds based on multiple paths,
as described in the following ticket:
http://bitten.edgewall.org/ticket/244

If someone were willing to work with me a bit, I'd be willing to work
on a patch. But, I'm a Python novice :(

I think a good solution would be to leave the current build
configurations as-is, i.e., with a single path that acts as a value
for the ${path} variable in a build configuration -- this would be
backwards compatible and hopefully minimize the code changes that are
necessary. However, to support multiple paths, add a separate list of
paths to watch at the bottom of the Repository Mapping section.
Depending on the state of the bitten code, I would envision a single
table with a Build ID foreign key and a corresponding path for each
additional path. If one of those paths triggered a build (by some
additional checking after a commit), then the original code could be
called, which uses the currently defined path just like before.

Sound reasonable? Anyone willing to work with me to show me where I
should be looking in the code?

Thanks,
-- Scott

[Erik Bray]

I'm not entirely clear on the purpose of that ticket. Why not just
copy the build configurations and use separate configurations for each
path? It's no different, for example, from having a configuration for
a development branch and a stabilization branch.

I /do/ think it should be easier to copy a build configuration for
purposes like this.

[sahendrickson]

Hi Erik,

The main problem is that bitten assumes that projects will always have
a single, common, root folder. We're using multiple project roots [1].
Our development server has 4 different library subprojects and a
single main application project. We want the main application to be re-
released every time it or any of the libraries are updated. (We're
developing eclipse plugins. Having a multiple project roots layout is
very common for that domain.)

Currently we are doing what you suggest. We have five different,
duplicate build configurations. However, this is less than ideal for a
few reasons:

- First, it's a maintenance nightmare. I recently had to redirect the
logging of three of the tasks in my build. This meant making 15 total
edits (3x5 configurations). This is error prone. Copying and
configuring build configurations would help, but not much. It would
still be necessary to test all five build configurations each time I
make a small change. So, while I'm in support of that, it only
indirectly affects this issue.

- Second, having multiple build configurations eats up cpu and
bandwidth, even if I take advantage of "waiting for stabilization" and
not building "all revisions" on all 5 builds. Making commits to
multiple projects or across projects will always trigger all builds
that are touched -- because they don't share the "all revision" or
"wait for stabilization" information. Instead, Bitten treats them as
completely independent builds. We're sharing our server and need to
play nice.

- Third, there is other information in these projects that shouldn't
trigger builds (such as documentation) and other projects being
developed by others on the same subversion server, which are not part
of our application. So, while we could reorganize the repository it
would make it harder to share these libraries with others, and while
we could trigger a build on a higher, common path, it would make an
excess of builds that are a result of changes to other, unrelated
projects. Each build takes 30 minutes and is quite CPU intensive, so
that would be bad :-(

Your comment on the development and stable branches is an interesting
point. Clearly, I see them as a different case. But, the 5
configurations are only for the development build. When we start to
build a stable branch as well this will be *another* 5 configurations.
I think this is part of the reasons why I haven't set that up yet:
It's not something I'm looking forward to doing ;-)

I hope that helps to explain the situation.

-- Scott



[1] http://svnbook.red-bean.com/en/1.5/svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout






On Dec 17, 8:38 am, "Erik Bray" <hyugaricd...@gmail.com> wrote:

[Erik Bray]

Okay, I understand your use case now, and agree that having multiple
configurations would be unmanageable. I'm tempted to say something
like "well maybe it would make sense to just keep all your subprojects
under a single root", but I'm in no position to judge your development
process. I'm sure it there reasons it is the way it is.

I'm not sure exactly what the best approach here would be, but
allowing multiple checkout paths would certainly be a start, since all
those paths would have to monitored for triggering a build, as you've
said.

For a simple short-term solution I'm tempted to just suggest allowing
${path} to be a comma-separated list of paths. Of course that would
get you into trouble if any of your paths have commas in them, but I
hope to $deity that no one's doing that to begin with. And then the
SVN checkout command would have to be able to recognize that it's a
list of different paths and expand it somehow.

A better long term solution might, as you suggested, involve creating
a separate table mapping multiple paths to a configuration. This
would also require extending the UI to allow adding multiple paths the
configuration, which isn't a problem. Then the interesting question
is how to reference multiple paths in the build recipe.

One thought I had, which some might object to, is to go ahead treat
the build recipe as a Genshi template. If the configuration just uses
a single path, ${path} could still just be a string and be treated as
it is currently. But if there are multiple paths, ${path} could be a
list. And with the power of Genshi, it would be easy to reference
individual indices (${path[0]}, ${path[1]}, etc). Making the recipe a
Genshi template also allows all sorts of other power that's been
requested before, such as looping, conditionals, and other magic. I
think the idea's come up before and has been shot down though. It's
certainly a much larger change by itself than some other
implementation of multiple paths that doesn't depend on Genshi.

[sahendrickson]

On Dec 18, 7:23 am, "Erik Bray" <hyugaricd...@gmail.com> wrote:
> Okay, I understand your use case now, and agree that having multiple
> configurations would be unmanageable.  I'm tempted to say something
> like "well maybe it would make sense to just keep all your subprojects
> under a single root", but I'm in no position to judge your development
> process.  I'm sure it there reasons it is the way it is.
>
> I'm not sure exactly what the best approach here would be, but
> allowing multiple checkout paths would certainly be a start, since all
> those paths would have to monitored for triggering a build, as you've
> said.
>
> For a simple short-term solution I'm tempted to just suggest allowing
> ${path} to be a comma-separated list of paths.  Of course that would
> get you into trouble if any of your paths have commas in them, but I
> hope to $deity that no one's doing that to begin with.  And then the
> SVN checkout command would have to be able to recognize that it's a
> list of different paths and expand it somehow.
>

In my case, the build instructions are in the application project. So,
I don't need to know the other paths in order to build it. I just need
the build to be triggered based on the other paths. A comma-separated
solution (if easier) is good. I think a text box with one path per
line might be easier to see and maintain. So long as I can refer to
the first path somehow, such as having ${paths[0]} similar to what you
suggested, and ${path} be an alias to ${paths[0]} for backward
compatibility.

I don't need anything as complex as the Genshi template solution that
you've mentioned. ( I personally think that the build configuration
duplicates ant functionality -- i.e., the benefit of bitten is in the
triggering and gathering of build information and not in the process
of performing the build, as ant is already virtually ubiquitous, but
that's another topic :) )

-- Scott

> A better long term solution might, as you suggested, involve creating
> a separate table mapping multiple paths to a configuration.  This
> would also require extending the UI to allow adding multiple paths the
> configuration, which isn't a problem.  Then the interesting question
> is how to reference multiple paths in the build recipe.
>
> One thought I had, which some might object to, is to go ahead treat
> the build recipe as a Genshi template.  If the configuration just uses
> a single path, ${path} could still just be a string and be treated as
> it is currently.  But if there are multiple paths, ${path} could be a
> list.  And with the power of Genshi, it would be easy to reference
> individual indices (${path[0]}, ${path[1]}, etc).  Making the recipe a
> Genshi template also allows all sorts of other power that's been
> requested before, such as looping, conditionals, and other magic.  I
> think the idea's come up before and has been shot down though.  It's
> certainly a much larger change by itself than some other
> implementation of multiple paths that doesn't depend on Genshi.
>

> > [1]http://svnbook.red-bean.com/en/1.5/svn.reposadmin.planning.html#svn.r...

[sahendrickson]

I've created and attached a patch to the ticket:
http://bitten.edgewall.org/ticket/244

Erik, if you don't mind, would you try it out and see if it is
satisfactory?

Thank you,
-- Scott

On Dec 18 2008, 9:39 am, sahendrickson <sahendrick...@gmail.com>
wrote:

[sahendrickson]

On Feb 12, 9:21 am, sahendrickson <sahendrick...@gmail.com> wrote:
> I've created and attached a patch to the ticket:http://bitten.edgewall.org/ticket/244
>
> Erik, if you don't mind, would you try it out and see if it is
> satisfactory?
>

Oh, a brief overview and one question regarding the patch:

The patch stores paths within the DB table as a string of individual
paths separated by the '\n' character. However, within the rest of the
code, the paths are treated as an array of strings. I believe that
this is backwards compatible with current configurations since when
joining or splitting the paths for DB operations, the code would just
interpret a previous entry as a single path for the array.

One bug that I can't figure out how to fix is in the webpage. Within
the admin.py, line 191, where it reads:
'active': config.active, 'paths': '<br>'.join(config.paths),
it is trying to format the list of paths in to a list for display on
the main configuraiton page. However, the '<br>' gets interpreted as
literal text rather than as markup and a '\n' doesn't achieve the goal
either. Instead, the list of paths is always squashed down to a single
line of text. I'm guessing this is a feature of Genshi, about which
I'm not that familiar.

Finally, this patch is against r571 of
http://svn.edgewall.org/repos/bitten/branches/experimental/trac-0.11

Thanks again,
-- Scott

[Erik Bray]

On Thu, Feb 12, 2009 at 12:21 PM, sahendrickson <sahend...@gmail.com> wrote:
>
> I've created and attached a patch to the ticket:
> http://bitten.edgewall.org/ticket/244
>
> Erik, if you don't mind, would you try it out and see if it is
> satisfactory?
>
> Thank you,
> -- Scott

I'll have to spend more time looking it over tomorrow, but I can say
now that it does have a number of problems, most of them minor. I'd
be happy to fix it up and post an updated patch, but I'm not sure
about some of this, and don't have an easy way to test it right now.
So as soon as I get a chance.

[sahendrickson]

Yes, I see there are a number of errors. If you haven't had a chance
to do anything with it, let me work on it a little longer. Maybe I
can address a few of them. I'll post an update on the ticket after I
have. Should I comment on the newsgroup when I do?
-- Scott

On Feb 12, 4:02 pm, Erik Bray <hyugaricd...@gmail.com> wrote:

[sahendrickson]

... And, thanks Erik for comments and offering to look it over :)
-- Scott

[Erik Bray]

I've posted an update to the ticket about the patch. Sorry it ended
up taking so long.