builds depending on status of other builds
Jason Miller
I have no idea where to post such a thing. But I have begun work on a
method that should allow (code is the best way to describe it):
<build xmlns:c="http://bitten.edgewall.org/tools/c"
xmlns:sh="http://bitten.edgewall.org/tools/sh"
onerror="continue"
requires="build_recipe01(fission-gnu, fission-intel),
build_recipe_07(*), build_recipe_08">
<step id ... </step>
</build>
In other words, the build you see before you, would require that three
other builds (including specific targets) to be completed first,
before bitten-master would allow it as a potential build to be
available when a bitten-slave makes a POST request (I am learning
about the bitten-master internal queueing system). I will be allowing
wildcards for targets (build_recipe_07(*)). I will also allow _not_
specifying any targets to mean every target within that build
(build_recipe_08).
Reasons being?? Well... if you have seen any of my previous posts, I
have some strange requirements where I work. Strange translating to:
crazy bash/python scripts to manipulate bitten-slave as much as
possible. If I can get the above working, I can simply drop all of
that, and run bitten-slave as a simple no-strings-attached
chronological job... period.
Thoughts?
Unfortunately, I have no idea how to design this as an augmentation
plugin, and am having to modify Bitten code directly. I was reluctant
to post this, knowing I can't modify public bitten code for
distribution, and possibly receiving some harsh comments :)
Thanks!
Jason Miller
Simon Cross
> <build xmlns:c="http://bitten.edgewall.org/tools/c"
> xmlns:sh="http://bitten.edgewall.org/tools/sh"
> onerror="continue"
> requires="build_recipe01(fission-gnu, fission-intel),
> build_recipe_07(*), build_recipe_08">
How does one tell which revisions these other builds need to be for?
Sort of implicit in your example is that these other builds are for
the "same" revision as the build they depend on. However, I'm not sure
how this will work in practice. Even with just a single repository
builds triggered for different paths within the same repository will
usually be built for a disjoint set of revisions.
> Unfortunately, I have no idea how to design this as an augmentation
> plugin, and am having to modify Bitten code directly. I was reluctant
> to post this, knowing I can't modify public bitten code for
> distribution, and possibly receiving some harsh comments :)
I'm not sure why you thought that. Bitten is distributed under a
three-clause BSD license. You are welcome to create modifications and
distribute them. We also welcome patches for inclusion into Bitten
itself.
I would be interested in the details of the approach you're taking. An
HTTP API of some sort that allows the triggering of builds has been
discussed in the past and might be another viable approach to
supporting your use case.
Schiavo
Simon
Jason Miller
On Dec 7, 2011, at 8:01 AM, Simon Cross wrote:
> On Wed, Dec 7, 2011 at 4:04 PM, Jason Miller <m.jason...@gmail.com> wrote:
>> <build xmlns:c="http://bitten.edgewall.org/tools/c"
>> xmlns:sh="http://bitten.edgewall.org/tools/sh"
>> onerror="continue"
>> requires="build_recipe01(fission-gnu, fission-intel),
>> build_recipe_07(*), build_recipe_08">
>
> How does one tell which revisions these other builds need to be for?
> Sort of implicit in your example is that these other builds are for
> the "same" revision as the build they depend on. However, I'm not sure
> how this will work in practice. Even with just a single repository
> builds triggered for different paths within the same repository will
> usually be built for a disjoint set of revisions.
>
In my environment I am only interested in the build status. Not necessarily revision.
But, that is a darn good point. I suppose I could add some kind of ${revision} logic in there. But that still is going to be mismatched to all hell.
This can get ugly, but it is possible to modify the recipe after a build status change. haha, I don't even want to think about that. At the end of the day, I suppose my contribution would only help a basic stable-target dependency.
>> Unfortunately, I have no idea how to design this as an augmentation
>> plugin, and am having to modify Bitten code directly. I was reluctant
>> to post this, knowing I can't modify public bitten code for
>> distribution, and possibly receiving some harsh comments :)
>
> I'm not sure why you thought that. Bitten is distributed under a
> three-clause BSD license. You are welcome to create modifications and
> distribute them. We also welcome patches for inclusion into Bitten
> itself.
>
Because of your first question actually. I knew that my needs are very different then others. And I didn't want to step on any ones toes. But, if I get this going correctly, I will be happy to contribute.
> I would be interested in the details of the approach you're taking. An
> HTTP API of some sort that allows the triggering of builds has been
> discussed in the past and might be another viable approach to
> supporting your use case.
>
> Schiavo
> Simon
>
I'll try to keep you updated!
> --
> You received this message because you are subscribed to the Google Groups "Bitten" group.
> To post to this group, send email to bit...@googlegroups.com.
> To unsubscribe from this group, send email to bitten+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/bitten?hl=en.
>
Jason
Roland Wilczek
> On Dec 7, 2011, at 8:01 AM, Simon Cross wrote:
> > On Wed, Dec 7, 2011 at 4:04 PM, Jason Miller <m.jason...@gmail.com>
wrote:
> >> <build xmlns:c="http://bitten.edgewall.org/tools/c"
> >>
> >> xmlns:sh="http://bitten.edgewall.org/tools/sh"
> >> onerror="continue"
> >> requires="build_recipe01(fission-gnu,
> >> fission-intel),
> >>
> >> build_recipe_07(*), build_recipe_08">
> >
> This can get ugly [...]
I would be glad to see the "dependent build problem" solved at the level of
target-platforms.
Target platforms always share $revision and $path from their build
configuration. So an implementation at this level would be much simpler.
What, if we were able to declare platformA as dependent from platformA and
platformB, so that the build on platformC starts only if/when platformA and
platformB built successfully?
In my eyes this would solve at least the majority of use cases.
--
kind regards
- Roland Wilczek
Simon Cross
> I would be glad to see the "dependent build problem" solved at the level of
> target-platforms.
>
> Target platforms always share $revision and $path from their build
> configuration. So an implementation at this level would be much simpler.
Introducing dependencies between platforms is indeed a much simpler
problem. It does feel a bit odd for platforms (which all use the same
build recipe) to depend on each other though. Do you have a recipe
which checks the platform and does different things for each?
A simple "depends on" field for platforms shouldn't be too difficult
to implement (this would put it outside of the build recipe).
Schiavo
Simon
Jason Miller
> You received this message because you are subscribed to the Google Groups "Bitten" group.
> To post to this group, send email to bit...@googlegroups.com.
> To unsubscribe from this group, send email to bitten+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/bitten?hl=en.
>
Not sure if the email was targeted at me or not, but I, certainly do
have need of it. Even though I can (and do) accomplish the above
through bash/python scripts, however, doing so causing other issues I
run into; Five super computer clusters do not equal the same
environment as the next.
Thus, I end up with 5 (or more if your counting a stable build which
_depends_ on others finishing first) different pre-initialization,
run-before-this-or-that environmental loading scripts... If I can
consolidate that at the proverbial head (bitten recipes) then I am
going to do it. Thus, I have decided to modify the bitten-master queue
system.
Update:
I am very near completion. I have only one dependency issue that I ran
into at EOD; I need to iterate through all build recipes (not just
those being requested) for dependencies and link them to their
platform equivalents. I ran into a scenario that if a recipe needing a
dependency was considered (queued) before a recipe that did _not_ need
a dependency, then my changes would effect the the prior recipe. Easy
fix. But, it was the end of Business day.
The syntax has been adjusted to make it more easy to iterate through
the dependencies:
requires="recipe_001(target_01,target_02,target_03) recipe_002(target_01)"
Alas no wild cards. Not yet. Basically, a space is the delimiter per
recipe dependency and you must specify every build target. I was also
asked by internal customers to make sure that the same recipe being
called can have a dependency to a target within its encapsulated
recipe. In other words:
# recipe_01 contains:
<build requires="recipe_01(target_02)">
In other words, I will code to allow a target within the very recipe
that contains the same target in question to be scrutinized before
continuing.
All in all, I have only needed to modify bitten/queue.py file. Modify
one define and add a new one, which is about 20 lines long. I am
looking forward to the scrutiny of making it better! Especially with
my calls to the SQL backend. I know there are improvements available
there.
Thanks all :)
Jason Miller
Brettschneider Falk
> -----Original Message-----
> From: bit...@googlegroups.com [mailto:bit...@googlegroups.com] On
Behalf
> Of Jason Miller
...
> In other words, the build you see before you, would require that three
> other builds (including specific targets) to be completed first,
Cascaded calls of builds including the waiting of builds until others
have finished first - that's what http://bitten.edgewall.org/ticket/669
also addresses.
Look at
http://bitten.edgewall.org/attachment/ticket/669/skippedExample.JPG.
Platform "myProduct-Tests" is waiting until platform "DSP" and "Windows"
have finished first. Sometimes "myProduct-Tests" is running and busy on
its own steps, and when it returns it may _skip_ several revisions (blue
state "Skipped"!) since meanwhile some builds of "DSP" and "Windows"
happened.
The waiting is solved here by a little program bittenWatch.exe that is
constantly looking for the state of the other platforms by e.g. simple
checks for files created by the build platforms on finishing their
build.
Though this means to think different in the whole approach of organizing
your build system. You need to organize in "columns" than step-oriented
in "rows".
CU, falkb
Geschaftsfuhrer: Dr. Joachim Ihlefeld . Marcel Seeber . Dr. Oliver Vietze
Sitz der Gesellschaft: Radeberg
Amtsgericht Dresden: HRB 15379
Ust. ID: DE 189714583
Jason Miller
Jason Miller
> You received this message because you are subscribed to the Google Groups "Bitten" group.
> To post to this group, send email to bit...@googlegroups.com.
> To unsubscribe from this group, send email to bitten+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/bitten?hl=en.
>
I forgot to mention, yes, this was one of my goals as well. And I have
implemented that in my patch. The dependency can target a platform
from the very same recipe from which it depends on. When the
dependency check realizes the bitten-slave request will satisfy the
same dependency, I allow the build.
I think I should go back however, and see if I can implement something
similar at the step stage.
Thanks,
Jason Miller