Implementing build queues - is this wanted?

13 views
Skip to first unread message

kiwidude

unread,
Oct 20, 2006, 5:53:30 PM10/20/06
to ccnet-devel
One of the ever ongoing hot topics for CC.Net is controlling which
projects build simultaneously to prevent conflicts to shared resources
like the file system or databases. Richard did a nice job recently with
the <sequentialProject> and <sequential> task plugins which is great as
a starting point without requiring CC.Net source code changes.

However that implementation does have a couple of limitations I see...
- no visibility of projects waiting for a lock (would be nice for the
cctray status does to show "Pending" for instance)
- cannot cancel builds which are waiting for a lock.
- project timeouts (you really only want to start the build time once
it gets the lock)

I believe these limitations are best overcome by making changes to
CC.Net itself.

I've started playing around with this tonight as a proof of concept,
and fairly quickly managed to get a build queue arrangement up and
running. Kudos to the developers for a nice object design to easily
slot into.

The basic premise is a new optional queueName attribute for the
<project> node. If not specified it will default to the project name
(i.e. only project on it's own queue). When the ProjectIntegrator
detects it is time to build, it puts a request on the named queue. If
its the only project on the queue it starts straight away - pretty
simple stuff.

I'm thinking of also adding some enhancements to CCTray (and ultimately
I guess the web dashboard as well) to give visibility on the projects
queued for a build, with some right click options to cancel it etc.

Clearly the more I add to this the more "intrusive" the changes are
going to be to CC.Net. Which finally leads to my questions...

1. Given these are changes to the core classes in CC.Net such as
ProjectIntegrator etc, just how open are the project admins to such
changes? I'm not talking major refactoring required, just additions
here and there. I know plugins are happily encouraged but I don't
believe this can be done that way. Obviously I can submit the changes
as patch files to someone for validation and making sure I'm not
heading in a direction you don't like.

2. Is anyone else currently also attempting something similar? In which
case am I wasting my time and/or would you like to share resource on
this.

3. Are people interested enough in this feature for me to crack on with
it?

If the answers are thumbs up to at least give it a go then I will
obviously give updates here about the proposed implementation and would
clearly love feedback for suggestions for features. Equally input from
those who worked on the core of CC.Net as to anticipated problems would
be appreciated.

And if I would be just wasting my time - well that would be nice to
know too ;-)

Regards,
Grant.
http://www.kiwidude.com/blog/

Ruben Willems

unread,
Oct 21, 2006, 12:37:11 AM10/21/06
to ccnet...@googlegroups.com
Hi

I like this approach, a question though :
does it also take into account the triggername ?

Suppose the following happens :
- checkin at 21:59 (CI trigger)
- 22:00 time (Nighly Build)
- checkin at 22:15 (CI trigger)


I would like to see the following :
- do a normal build (change detected)
- do the nightly build of 22:00
- do a normal build

From the current setup of CCNet, I think this behaviour is already there,
just want to check ;-)


with kind regards
Ruben Willems

kiwidude

unread,
Oct 21, 2006, 7:11:29 AM10/21/06
to ccnet-devel
Ruben Willems wrote:
> Hi
>
> I like this approach, a question though :
> does it also take into account the triggername ?
>
> Suppose the following happens :
> - checkin at 21:59 (CI trigger)
> - 22:00 time (Nighly Build)
> - checkin at 22:15 (CI trigger)
>
>
> I would like to see the following :
> - do a normal build (change detected)
> - do the nightly build of 22:00
> - do a normal build
> ...
Hi Ruben,

I hadn't spotted this new feature of named triggers in CC.Net 1.1 -
thanks for bringing my attention to it. My understanding of existing
behaviour from looking at the existing code is that the
ProjectIntegrator effectively "suspends" all it's triggers by blocking
the thread while the integration (build) takes place. When that
integration completes it is then able to "resume" the trigger checking.
So your scenario above should indeed work. The changes I am proposing
in no way change this behaviour I believe as they follow the same
pattern.

There is an interesting twist for force builds. As I understand it if
two developers try to force a build while a build is integrating, the
"latest one" wins (i.e. there can be only one forced build pending).
That's a good thing I believe - queuing two builds would be unnecessary
as the first of them would pickup any changes. If I am to give
visibility to that force build then I need to add the request to the
queue when it occurs. However I then create a problem of ensuring that
only one of those force builds is on the queue. My best guess would be
to mirror the existing behaviour of ensuring that only one "force
build" for a particular project is queued at a time. I guess I will
also replicate existing behaviour of replacing the queued
IntegrationRequest with the latest one.

Regards,
Grant.
http://www.kiwidude.com/blog/

Ruben Willems

unread,
Oct 22, 2006, 3:55:05 AM10/22/06
to ccnet...@googlegroups.com
Hi

<quote>


ProjectIntegrator effectively "suspends" all it's triggers by blocking
the thread while the integration (build) takes place. When that
integration completes it is then able to "resume" the trigger checking.

<quote>

Suppose a build takes +/- 10 minutes, and we have the same scenario :


- checkin at 21:59 (CI trigger)

- 22:00 time (Nightly Build)


- checkin at 22:15 (CI trigger)

I think I loose the nightly build. (If not, skip reading the rest of the mail)

The build starts at 21:59, takes 10 minutes, and the thread is
'unblocked'/un-suspended.
So the time is 22:09, and the schedule trigger of 22:00 hours does not
get fired.
This is a scenario that I do not like that much.

A solution : foresee a 'build manager' (or whatever name is chosen)
This class does the actual launching of the builds itself.

For the moment the project integrator launches the builds, with this
'build manager',
the project integrator en queues a build for a certain project.
To keep it simple for the moment, this could be a hash list consisting of
the project name and it's trigger. Whenever a build is done by this
'build manager',
it will build the next one (if one is present).

Schematic approach :

Suppose a build takes +/- 10 minutes, and we have this scenario :


- checkin at 21:59 (CI trigger)

- 22:00 time (Nightly Build)


- checkin at 22:15 (CI trigger)

ProjectIntegrator detects the CI 21:59 change, and enqueues it for the
build manager.
The build manager sees that there is a change in his queue, and starts
processing it.

ProjectIntegrator detects the Interval Trigger of 22:00 and enqueues
it for the build manager.
The buildmanager sees that there is a change in his queue, and starts
processing it.
(build is busy, so first continue with the current build)

ProjectIntegrator detects the CI 22:15 change, and enqueues it for the
build manager.
The buildmanager sees that there is a change in his queue, and starts
processing it.
(build is busy, so first continue with the current build)

If this 'buildmanager' has the same capabilities as your 'queue' in
your first mail,
than there is no problem for me..

With kind regards
Ruben Willems

kiwidude

unread,
Oct 22, 2006, 5:37:24 AM10/22/06
to ccnet-devel

Ruben Willems wrote:
> Hi
>
> <quote>
> ProjectIntegrator effectively "suspends" all it's triggers by blocking
> the thread while the integration (build) takes place. When that
> integration completes it is then able to "resume" the trigger checking.
> <quote>
>
> Suppose a build takes +/- 10 minutes, and we have the same scenario :
> - checkin at 21:59 (CI trigger)
> - 22:00 time (Nightly Build)
> - checkin at 22:15 (CI trigger)
>
> I think I loose the nightly build. (If not, skip reading the rest of the mail)
>
No you do not lose the nightly build. The schedule trigger gets picked
up on the next "thread cycle" after the CI trigger has completed. So
even though the time is now 22:09, since it uses a "greater than"
comparison rather than an "equals" it sees that 22:00 has indeed been
reached and will kickoff the nightly build. Once that nightly build
completes, it then sets the next integration time to be the following
day at 22:00.

Regards,
Grant.
http://www.kiwidude.com/blog/

kiwidude

unread,
Oct 22, 2006, 7:01:58 AM10/22/06
to ccnet-devel
Sorry I should have clarified this further particularly as it does
bring up a point about the proposed queue implementation.

Lets say you have:
Project A - CI trigger every 1 min, Scheduled trigger @ 22:00
Project B - CI trigger every 1 min
Each project taking 10 mins to build

Under the existing CC.Net implementation (without using the sequential
lock stuff) then potentially at 21:59 both Project A and project B CI
triggers fire. When they complete at 22:09, each project has it's "next
due time" set by adding the 1 min interval to the current time - i.e.
the CI triggers will now be due at 22:10. The ProjectIntegrator A & B
threads do a sleep for 100ms then fire again (still within 22:09).
Project A sees that it now has a scheduled trigger "overdue" (as time >
22:00 today) so kicks off a build. Project B thread will keep sleeping
until 22:10 when it's CI trigger is due and fires again. When Project A
scheduled trigger finishes at 22:19, the scheduled trigger is
rescheduled to 22:00 the next day. At the next Project A thread cycle
it sees the CI trigger is now "overdue" and fires...

The only way the proposed queue mechanism affects this is to serialise
the builds. The point I wanted to flag up however is that with one
exception a single project will only appear once in a queue. The one
exception is if the project is at the front of the queue currently
integrating and someone does a force build - the "force" request will
be added to the queue which means temporarily that project does appear
twice. As I mentioned previously above - if a project is "queued" but
not yet started any additional force build requests are discarded as
they would be redundant.

What this means for Project A is that you will NOT see the scheduled
22:00 integration on the queue until the CI trigger finishes first - as
like current behaviour the triggers are "suspended" until the project
completes it's integration. To change the logic so that schedule
triggers keep getting checked for future builds on the queue is way too
problematic for what I had in mind. For people who use separate
projects for continuous and daily builds you will of course see both
being on the queue - it is only people who use the multiple trigger
scenario you described who will not.

Personally I don't see this as a major problem as I prefer individual
projects over multiple triggers - as in our case we do a lot more
things in the daily build like FxCop, code coverage etc that are not
done in the CI project. So when I drill into a daily project I will
always know that the code coverage ccnet web link will have results for
instance, rather than a bit of a random log clickfest search if we were
to run all under the same project. That's just my opinion and
preference though and if we in fact did the same steps in both builds
then a single project/multiple trigger would be a better choice. If
others think the "lack of visibility of pending triggers within a
project" is a big limitation then let me know. Worst case it just means
the new feature makes no difference to you assuming you have no other
projects sharing the named queue.

Regards,
Grant.
http://www.kiwidude.com/blog/

Carrette, George

unread,
Oct 22, 2006, 9:24:20 AM10/22/06
to ccnet...@googlegroups.com
What about the idea of implementing higher-level features by writing
code that uses the CC.Net client api's to monitor what is going on and
doing useful things? That is, instead of modifying the core CC.Net
service to introduce more complex "job control" style functionality,
build a client program to do it. Then, obviously, such a client program
could be part of some other CC.Net task.

For the work that my group does we use CC.Net as a straightforward build
server based on changes in the source code control system, but we also
use it as a replacement for the crude "Windows Scheduled Tasks"
mechanism, as just a way to be able to run a "job/process" in the
background.

But it has crossed my mind that I could use the same api's that the web
dashboard and/or CCTray are using to program arbitrary capabilities that
way. Seems a lot less risky and a lot cheaper (in time) than getting
into the CCNet service guts.

Just a suggestion of course.


NOTICE:
This message may contain privileged or otherwise confidential information. If you are not the intended recipient, please immediately advise the sender by reply email and delete the message and any attachments without using, copying or disclosing the contents. (FE1)

kiwidude

unread,
Oct 22, 2006, 2:31:34 PM10/22/06
to ccnet-devel
Hi George,

If I thought there was a way of implementing this as some sort of
plugin then definitely I agree it's likely to be less work. However
unless someone far more clever than I am can suggest how I cannot see
that it is possible to do this functionality without impacting the
core. The APIs are not rich enough for this feature - I believe Richard
with his sequential plugins has done as much as is feasible down that
line.

Anyways in the spirit of optimism or madness I decided to have a go at
an implementation. You can find the JIRA and patch file for it here
(yes I accept the CC.Net terms etc):
http://jira.public.thoughtworks.org/browse/CCNET-770

This seems to be working ok for a first stab but clearly would love it
if others can take a look and let me know if I'm on the right track
etc. Feel free to e-mail me if you want to take it offline rather than
me continuing to add noise here if not wanted.

The patch attached implements the named integration queues as
described. There is a new "orange" icon for CCTray with a state of
"Pending" to show builds which are queued. A new right-click menu
option of "Cancel Pending" is available on projects in this state to
remove them from the queue (you can only cancel pending builds, not a
build which has started).

I did end up having to touch more interfaces than I would have hoped
but as this is pretty fundamental to the operation of CC.Net perhaps
that can't be helped. Functionally though I'm pretty happy with it and
nothing further to be done except apply your feedback to it. I've had a
crack at the unit tests although undoubtedly more could be added.

Note this patch also includes fixes for two other minor bugs I found in
CCTray (CCNET-768 and CCNET-769)

Greatly appreciate any feedback or comments.

Regards,
Grant.
http://www.kiwidude.com/blog/

Ruben Willems

unread,
Oct 23, 2006, 2:32:04 AM10/23/06
to ccnet...@googlegroups.com
Hi

Looks ok to me,
I use the named triggers of CCNet 1.1 to distinguish for a CI build,
or a daily build. You can ofcourse use 2 CCNet projects as you do.

Both approches work, it is just a matter of what you prefer.

Waiting on feedback from the maintainers of CCNet ;-)

with kind regards
Ruben Willems

Daniel Piessens

unread,
Oct 26, 2006, 3:12:29 PM10/26/06
to ccnet...@googlegroups.com
Ok I've been trying to load this patch for the last half hour to see what it did and it we could commit it, because at least from reading the patch it looks good. The issue I'm having is that every system I've tried fails on applying this patch, mostly because the diff utilities that run after don't like that fact that new files are being added. So could someone please tell me how that got the patch to apply? I tried the patch.exe and TortiseSVN (apply patch) and both failed. Also you're missing the orange icon :-).
 
Thanks,
 
-Dan Piessens

 

kiwidude

unread,
Oct 26, 2006, 3:19:21 PM10/26/06
to ccnet-devel
Daniel,

I did wonder how well the patch was going to work as TortoiseSVN did
grumble a couple of times but since it produced a file I thought give
it a go - clearly too optimistic of me ;-)

I will send you an e-mail which if you can reply to hopefully we can
get something happening. Perhaps I need to break it into two parts -
one containing the updated files, the other containing all the new
files, and then a bit of jiggery pokery on your part to stitch it back
together again?

Regards,
Grant.

Daniel Piessens

unread,
Oct 26, 2006, 8:07:21 PM10/26/06
to ccnet...@googlegroups.com
Hi Grant,
 
Just copying the group what I sent you. Aside from a few minor test things, I think this is an excellent first stab. If we can eliminate a test quirk or two, I think this will be excellent. As an aside I'm working on a new dashboard, and I think a good plugin (but put into standard code) would be a screen per server that allows users to view projects sequentially in the queue and allow them to cancel it from there. I realize that this involves a significant amount of web coding, so I may be able to help you with that since I've gotten used to the who web dash model. I'll post again when this gets into standard code provided Mike or Owen don't have any objections to this model.
 
-Dan Piessens

 

David Cameron

unread,
Oct 26, 2006, 9:36:36 PM10/26/06
to ccnet...@googlegroups.com
Hi

Is this intended to provide an ordering the same way the
<sequentialProject> tag does? If it is, what's the mechanism for
managing the order in which projects are added to the queue?


Dave

On 10/21/06, kiwidude <grant...@gmail.com> wrote:
>

kiwidude

unread,
Oct 27, 2006, 3:39:59 AM10/27/06
to ccnet-devel
Hi Dan,

Thanks for taking the time to look into this. You have me intrigued
about the "test quirks" you mentioned - all the tests consistently
passed for me so you have me wondering. Can you e-mail me what problems
you found? As I put in a previous post the tests were definitely an
area that could be added to but I was hitting some apparent limitations
of the LatchMock class implementation which hopefully you guys can
steer around...

As for the idea of the web page screen, certainly that sounds a good
idea. As you no doubt spotted in the code I did start putting the
placeholder functions to return the list of queue names and their
contents, but clearly still missing the code to expose this all the way
up through the interfaces to the public API for use by the web pages
and/or CCTray. It was certainly one of my original goals that users
could see an ordered list of the queue contents but I had to "pause"
somewhere to get some feedback in case you guys didn't like the overall
approach.

The appearance of the UI usually evokes debates - would you like to
discuss in the forum or offline? For instance...
- do you want one screen that displays all the queues and their
contents in a tree style approach (in which case I should alter the API
so it returns the current contents of all queues at once rather than a
single named queue).
- or should it just display a list of queues which the user then clicks
on one to just see it's contents?
- how would this page be navigated to?
- I think I need to alter the way the list of queue names is
constructed. Currently it is only as a project gets built for the first
time that new queue names are added to the queue. If you have a screen
listing all queue names I think for consistency you would want all the
queue names up front not incrementing over time? Obviously there are a
couple of ways of doing that - spin through the project integrators
asking each project for it's queue name rather than the queue itself,
or get each project to "register" itself with the queue when the
integrator starts (the latter sounds better?)
- do we offer a similar screen for CCTray? For instance a treeview
control panel that can be shown/hidden using a button next to
"ForceBuild" so people who don't care dont need to see it?

You obviously have some ideas and experiences on the implementing the
web side so look forward to any suggestions you have - I will hold off
any actual further coding on this until you have had a chance for the
others to review etc and work off what gets integrated into the trunk.

Regards,
Grant.
http://www.kiwidude.com/blog/

Daniel Piessens wrote:

> ------=_Part_29677_4094023.1161907641828
> Content-Type: text/html; charset=ISO-8859-1
> X-Google-AttachSize: 1743
>
> <div>Hi Grant,</div>
> <div>&nbsp;</div>
> <div>Just copying the group what I sent you. Aside from a few minor test things, I think this is an excellent first stab. If we can eliminate a test quirk or two, I think this will be excellent. As an aside I'm working on a new dashboard, and I think a good plugin (but put into standard code) would be a screen per server that allows users to view projects sequentially in the queue and allow them to cancel it from there. I realize that this involves&nbsp;a significant amount of web coding, so I may be able to help you with that since I've gotten used to the who web dash model. I'll post again when this gets into standard code provided Mike or&nbsp;Owen don't have any objections to this model.
> </div>
> <div>&nbsp;</div>
> <div>-Dan Piessens<br><br>&nbsp;</div>
> <div><span class="gmail_quote">On 10/26/06, <b class="gmail_sendername">kiwidude</b> &lt;<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:grant...@gmail.com" target="_blank">grant...@gmail.com
> </a>&gt; wrote:</span>
> <blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><br>Daniel,<br><br>I did wonder how well the patch was going to work as TortoiseSVN did<br>grumble a couple of times but since it produced a file I thought give
> <br>it a go - clearly too optimistic of me ;-)<br><br>I will send you an e-mail which if you can reply to hopefully we can<br>get something happening. Perhaps I need to break it into two parts -<br>one containing the updated files, the other containing all the new
> <br>files, and then a bit of jiggery pokery on your part to stitch it back<br>together again?<br><br>Regards,<br>Grant.<br><br></blockquote></div><br>
>
> ------=_Part_29677_4094023.1161907641828--

kiwidude

unread,
Oct 27, 2006, 4:00:56 AM10/27/06
to ccnet-devel
Hi Dave,

Yes this is intended as a direct replacement for <sequentialProject>
and currently the ordering is a FIFO queue. There is currently no
prioritisation capability in there - the queue literally reflects an
ordered timeline of when integration requests were made. As the
<sequentialProject> implementation uses a lock I'm not sure that the
build order is deterministic for it - rather whichever waiting thread
happens to get woken first after the lock is released will kick off
next? I'm not 100% sure about the latter from memory.

In terms of "managing" the queue currently the functionality is a
right-click option in CCTray to cancel pending items, with a similar
option proposed for a web page by Dan.

I assume (apologies if wrong!) you are leading to whether queue
priorities is something we need to be considering? i.e. should there be
the concept of higher priority projects, such that requests for these
jump to be next in line rather than timeline based?

Alternatively rather than just "cancel" a pending integration request,
perhaps there should be an option to force a promotion/demotion in the
queue? That would save people from cancelling and then doing force
build requests to change the order, but doesn't sound as clean as the
above.

Certainly all of the above would be possible - perhaps a
"queuePriority" optional numeric attribute to supplement the
"queueName" one for a <project>, where 1 represents the highest
priority, followed by 2 etc. A default of zero means just add to the
back of the queue like it does now?

Suggestions welcomed - obviously I don't want to overengineer this but
I do know of CC.Net implementations where they only ever want one build
running at a time across a number of projects (i.e. one queue), in
which case the priority idea likely has some interest.

Regards,
Grant.
http://www.kiwidude.com/blog/

Daniel Piessens

unread,
Oct 27, 2006, 10:34:59 AM10/27/06
to ccnet...@googlegroups.com
Hi Grant,
 
I sent you the details, essentially none are failed tests just some output issues that may need to be tweaked. As for the UI, If you wouldn't mind waiting a week or two I have some larger changes going into the dash for build progress and a new way of looking at projects, and once I get a few kinks worked out I'll commit that and I think we can use it as a base for some other things.
 
-Dan

 

David Cameron

unread,
Oct 29, 2006, 2:58:55 AM10/29/06
to ccnet...@googlegroups.com
Hi Grant

inline...

On 10/27/06, kiwidude <grant...@gmail.com> wrote:

> I assume (apologies if wrong!) you are leading to whether queue
> priorities is something we need to be considering? i.e. should there be
> the concept of higher priority projects, such that requests for these
> jump to be next in line rather than timeline based?

I was wondering whether it would solve the build pipelining sort of
requirement that keeps coming up. Basically, builds A, B and C share
the same code base in a subversion repository, but B needs A to be
built and C needs A and B to be built. I'm concerned that if I set
this all up on one queue, and code changes, projects A, B and C will
race to see who gets in the queue first. If anyone beats A to the
queue, then they will fail. Let me know if I've thought it through
wrong.

I think the "Lock" in the sequentialProject implementation has unusual
properties, although I can't remember at the moment.

Maybe your changes are not intended to solve this problem?

Dave

kiwidude

unread,
Oct 29, 2006, 4:14:29 AM10/29/06
to ccnet-devel
Hi Dave,

I hadn't wondered about it overlapping onto the build dependency
problem. FYI I have gone ahead and added the queuePriority attribute
implementation I mentioned previously.

Perhaps you can give me an example scenario? By that I mean - why you
could not use the existing mechanism of the force build publisher to
chain the builds in sequence?

Lets say you have C depends on B depends on A. If you want a change in
the codebase to ensure that all projects are built A->B->C then could
you put your CI trigger in A, and get it to do force builds of B and C.
I would have thought you can do this currently in CC,Net by putting a
forcebuild of B publisher in project A, and a forcebuild of C in
project B?

With the queuePriority mechanism, you can setup priorities A=1,B=2,C=3
and now put in project A to do a forcebuild of projects B and C. In
fact if you ordered the force build publishers you dont even need to
setup the queue priorities to do that.

So it's not really adding huge value for this scenario? I'm sure you
have a more complex one in mind - please describe it for me as if there
is a natural extension of what I am doing that could help then
obviously happy to look at including it.

Regards,
Grant
http://www.kiwidude.com/blog/

On Oct 29, 8:58 am, "David Cameron" <dave...@gmail.com> wrote:
> Hi Grant
>
> inline...
>

> On 10/27/06, kiwidude <grant.dr...@gmail.com> wrote:
>
> > I assume (apologies if wrong!) you are leading to whether queue
> > priorities is something we need to be considering? i.e. should there be
> > the concept of higher priority projects, such that requests for these

> > jump to be next in line rather than timeline based?I was wondering whether it would solve the build pipelining sort of

Gary Feldman

unread,
Oct 29, 2006, 12:35:41 PM10/29/06
to ccnet...@googlegroups.com
kiwidude wrote:
> ...I've started playing around with this tonight as a proof of concept,

> and fairly quickly managed to get a build queue arrangement up and
> running. Kudos to the developers for a nice object design to easily
> slot into.
>
> The basic premise is a new optional queueName attribute for the
> <project> node. If not specified it will default to the project name
> (i.e. only project on it's own queue). When the ProjectIntegrator
> detects it is time to build, it puts a request on the named queue. If
> its the only project on the queue it starts straight away - pretty
> simple stuff.
>
Being a firm believer in user-centered design, I consider the
implementation details to be secondary. Could I ask that you post some
sample ccnet.config files illustrating the use of the feature? And a
first pass at user documentation?

Thanks,

Gary


David Cameron

unread,
Oct 29, 2006, 7:28:03 PM10/29/06
to ccnet...@googlegroups.com
Hi Grant

It turns out there is a message in the archives describing the problem
I'm concerned about here:
http://groups.google.com.ag/group/ccnet-user/browse_frm/thread/8e1015588e7d5790/c509a4c5ea1ae7be?tvc=1&q=more+project+dependencies#c509a4c5ea1ae7be

Basically, you can have a queue develop in the following way. Letters
denote project names, numbers denote the subversion revision that
triggered the build. The first build listed is always in progress.

- A_200
... project A finishes building ...
- B_200
... a dev commits ...
- B_200, A_201
... project B finishes building ...
- A_201, C_200

C_200 is now guranteed to fail if A changes shared state in some
incompatible way. Now, granted, builds shouldn't do that, but we know
that we have users whos builds do do that. Also, the stated goal of
"preventing conflicts on shared resources" suggests to me that shared
state changes are more likely in exactly the situations where people
are likely to use queues. Shared resources potentially introduce
shared state.

I'm concerned that a queue facility, particularly one with priorities,
_looks_ like it's a solution to a build dependency problem, but for
complex setups is just more rope for users to hang themselves up with.
I'm a bit conflicted though, because it sure would be nice to have a
way to limit the number of builds happening at one time, at least for
performance's sake.

Dave

PS Is the following scenario possible? I think you explained this
earlier in the thread, but I'm not sure just based on that email. B is
a very long running build in this scenario.

- A_200
... project A completes ...
- B_200
... dev commits ...
- B_200, A_201
... dev commits again ...
- B_200, A_201, A_202
... dev commits one more time ...
- B_200, A_201, A_202, A_203
... etc ...

In an A triggers B triggers C situation that could get quite ugly...

Owen Rogers

unread,
Oct 29, 2006, 8:36:45 PM10/29/06
to ccnet...@googlegroups.com
ok, i'm a bit slow catching up on my ccnet emails.

grant: thanks for raising this discussion and putting forward a
solution for this long-standing issue.

On 20/10/06, kiwidude <grant...@gmail.com> wrote:
> The basic premise is a new optional queueName attribute for the
> <project> node. If not specified it will default to the project name
> (i.e. only project on it's own queue). When the ProjectIntegrator
> detects it is time to build, it puts a request on the named queue. If
> its the only project on the queue it starts straight away - pretty
> simple stuff.

i think that this is a good approach. this is the direction that i've
been trying to gradually move the server code.

> I'm thinking of also adding some enhancements to CCTray (and ultimately
> I guess the web dashboard as well) to give visibility on the projects
> queued for a build, with some right click options to cancel it etc.

this sounds interesting. i hadn't considered this.

> 1. Given these are changes to the core classes in CC.Net such as
> ProjectIntegrator etc, just how open are the project admins to such
> changes?

i'm open to it.

> I'm not talking major refactoring required, just additions
> here and there. I know plugins are happily encouraged but I don't
> believe this can be done that way. Obviously I can submit the changes
> as patch files to someone for validation and making sure I'm not
> heading in a direction you don't like.

i haven't looked at your patch yet, but will try to do so tonight.

> 2. Is anyone else currently also attempting something similar? In which
> case am I wasting my time and/or would you like to share resource on
> this.

i have made various attempts at this in the past. i've always been a
little concerned of locking issues and about the potential for missing
something that would screw things up for our users. basically i've
been too chicken to commit anything that i've worked on until i had
thoroughly tested it -- and i never managed to find the
time/environment to test it to that level.

> 3. Are people interested enough in this feature for me to crack on with
> it?

please do.

--
Owen Rogers | http://dotnetjunkies.com/weblog/exortech |
CruiseControl.NET - http://ccnet.thoughtworks.com

Owen Rogers

unread,
Oct 29, 2006, 8:48:33 PM10/29/06
to ccnet...@googlegroups.com
On 21/10/06, kiwidude <grant...@gmail.com> wrote:
> There is an interesting twist for force builds. As I understand it if
> two developers try to force a build while a build is integrating, the
> "latest one" wins (i.e. there can be only one forced build pending).
> That's a good thing I believe - queuing two builds would be unnecessary
> as the first of them would pickup any changes. If I am to give
> visibility to that force build then I need to add the request to the
> queue when it occurs. However I then create a problem of ensuring that
> only one of those force builds is on the queue. My best guess would be
> to mirror the existing behaviour of ensuring that only one "force
> build" for a particular project is queued at a time. I guess I will
> also replicate existing behaviour of replacing the queued
> IntegrationRequest with the latest one.

the way that i think of it is that ccnet implements a priority queue
with a depth of 1. force builds are of higher priority than triggered
builds. scheduled triggered builds are of a higher priority than
interval triggers (if necessary we can the triggers to reflect this
priority).

certainly for now, i don't think that it is necessary to hold more
than one request in the queue at a time. i think that it is fine for
the most recent, highest priority request to override the others.

there is also the potential to do some boolean combination of
triggers. ie. only trigger a build if trigger A AND trigger B fire.
(the default operand is OR).

i haven't checked your implementation yet; however, currently, other
than force builds, triggers operate in the same thread as the project
build. in other words, triggers are only checked to see if they
should fire when a project build is not running. this simplification
minimizes the number of concurrent threads able to operate or
influence a single project.

cheers,
owen.

Owen Rogers

unread,
Oct 29, 2006, 9:00:07 PM10/29/06
to ccnet...@googlegroups.com
hi ruben,

On 21/10/06, Ruben Willems <ruben....@gmail.com> wrote:
> Suppose a build takes +/- 10 minutes, and we have the same scenario :
> - checkin at 21:59 (CI trigger)
> - 22:00 time (Nightly Build)
> - checkin at 22:15 (CI trigger)
>
> I think I loose the nightly build. (If not, skip reading the rest of the mail)

under the current implementation, the schedule trigger knows if it
fired or not. that means that if the first build finishes after the
schedule trigger has been checked, the schedule trigger will still
fire (as long as the server's not restarted :-/)

i could be wrong, i'm not sure how this would change under the
proposed build queue implementation.

Owen Rogers

unread,
Oct 29, 2006, 9:05:14 PM10/29/06
to ccnet...@googlegroups.com
On 22/10/06, kiwidude <grant...@gmail.com> wrote:
> The patch attached implements the named integration queues as
> described. There is a new "orange" icon for CCTray with a state of
> "Pending" to show builds which are queued. A new right-click menu
> option of "Cancel Pending" is available on projects in this state to
> remove them from the queue (you can only cancel pending builds, not a
> build which has started).

ok, this is going to be an issue -- because i'm just about to commit a
change to cctray that uses an orange icon to indicate that a build is
broken but is currently building (right now there's no way to tell
that a project is broken and building without checking the dashboard).

anyway, i'm sure that we can work this out :)

i do like the idea of a pending state. my impression is that its goal
is to communicate to the team that a build will happen when the
current build completes. being able to cancel pending requests also
seems useful -- especially if you know that the current changes are
going to break the build.

> Note this patch also includes fixes for two other minor bugs I found in
> CCTray (CCNET-768 and CCNET-769)

thanks for the patch. i'll take a look.

Owen Rogers

unread,
Oct 29, 2006, 9:37:15 PM10/29/06
to ccnet...@googlegroups.com
hi dan and grant,

On 26/10/06, Daniel Piessens <dan.pi...@gmail.com> wrote:
> Just copying the group what I sent you. Aside from a few minor test things,
> I think this is an excellent first stab. If we can eliminate a test quirk or
> two, I think this will be excellent. As an aside I'm working on a new
> dashboard, and I think a good plugin (but put into standard code) would be a
> screen per server that allows users to view projects sequentially in the
> queue and allow them to cancel it from there. I realize that this involves a
> significant amount of web coding, so I may be able to help you with that
> since I've gotten used to the who web dash model. I'll post again when this
> gets into standard code provided Mike or Owen don't have any objections to
> this model.

i'd like to wait a bit before we apply the patch for two reasons:
i) i think that we need to make a point release to 1.1. there were
two issues that i'm aware of that need to be resolved pretty quickly
before they affect more users:
- working directory is not being set for initial builds. this is
causing problems for users setting up new projects in ccnet or for new
ccnet users. pretty critical. i fixed it in a ccnetlive build, but i
think that we should make a release with this change.
- cctray projects using the dashboard is hammering iis. i've set the
default option for adding a ccnet server back to use remoting.
hopefully this will be sufficient to discourage people from using this
feature until the problems with it are resolved.

ii) adding support for build queues has the potential for introducing
instability into the server. we need to be prepared to follow through
in resolving this instability before the next release. as a result, i
think that we should either a) commit these changes on a branch or b)
branch 1.1 and ensure that all fixes are applied on both the branch
and the trunk so that we do retain the option to make another release
before this work is complete.

just my 2c.

kiwidude

unread,
Oct 30, 2006, 3:20:01 AM10/30/06
to ccnet-devel

Owen Rogers wrote:
> On 22/10/06, kiwidude <grant...@gmail.com> wrote:
> > The patch attached implements the named integration queues as
> > described. There is a new "orange" icon for CCTray with a state of
> > "Pending" to show builds which are queued. A new right-click menu
> > option of "Cancel Pending" is available on projects in this state to
> > remove them from the queue (you can only cancel pending builds, not a
> > build which has started).
>
> ok, this is going to be an issue -- because i'm just about to commit a
> change to cctray that uses an orange icon to indicate that a build is
> broken but is currently building (right now there's no way to tell
> that a project is broken and building without checking the dashboard).
>
> anyway, i'm sure that we can work this out :)


lol... I shall see if Stuart has any other colour icons already
prepared in his bag of tricks...


> i do like the idea of a pending state. my impression is that its goal
> is to communicate to the team that a build will happen when the
> current build completes. being able to cancel pending requests also
> seems useful -- especially if you know that the current changes are
> going to break the build.


If you want to see an example of how this looks visually in CCTray I
have put together an animated gif on my blog from the work done this
weekend - the entry is here:

http://www.kiwidude.com/blog/2006/10/cruisecontrolnet-serialised-build_29.html

As per the blog entry comments/suggestions welcomed - and clearly the
icon colour needs to change ;-)

Regards,
Grant.
http://www.kiwidude.com/blog/

kiwidude

unread,
Oct 30, 2006, 4:01:49 AM10/30/06
to ccnet-devel

Owen Rogers wrote:

> - cctray projects using the dashboard is hammering iis. i've set the
> default option for adding a ccnet server back to use remoting.
> hopefully this will be sufficient to discourage people from using this
> feature until the problems with it are resolved.

Hi Owen,

I've had to do a fair bit of work with CCTray to get the display of
queues for a server to work this weekend and one of the things that
surprised me was how the project monitoring works. For each project
being monitored, it is actually calling a per server interface to
retrieve the status for all projects, and then does a for loop to
extract the result for that project (in RemotingCruiseProjectManager).

So if you have 10 projects being monitored from a single CCTray user
thats 10 calls to the server every poll interval returning the same
result. Doesn't sound the most efficient way of doing things? WIth
remoting being so fast I think this hides the issue whereas with Http
it exacerbates it. I am not saying this is the cause of your
performance issues (I havent looked at the web side at all) but I don't
believe it helps!

I had to introduce a new IServerMonitor interface for the changes I
made this weekend (and all the associated classes etc to build a unique
list of servers to poll etc). I think it might be worth considering
changing how that polling of project status works at some point so that
the call is now done via the IServerMonitor interface rather than the
IProjectMonitor one, and it then delegates the results. I'm not
touching it at this point myself as clearly I have enough change
introduced as it is... ;-)

As for your comments on the commit - as I have said all along I am at
the will of yourselves as to whether this even gets added at all, so if
it needs to wait or however you decide to do it just let me know when
you are ready. My main concern was whether the approach and the concept
were of interest - which it sounds like it is. From Dan having seen how
I have written the first part of the code and giving a thumbs up I can
at least continue out on my own for a bit longer. I've probably got one
more weekend worth of work to tidy up the CCTray side of things. Then
that just leaves a web interface for it all which as per Dan's
recommendation I will not start until his changes get committed.

Regards,
Grant.
http://www.kiwidude.com/blog/

kiwidude

unread,
Oct 30, 2006, 10:10:08 AM10/30/06
to ccnet-devel
David Cameron wrote:
> Hi Grant
>
> It turns out there is a message in the archives describing the problem
> I'm concerned about here:
> http://groups.google.com.ag/group/ccnet-user/browse_frm/thread/8e1015588e7d5790/c509a4c5ea1ae7be?tvc=1&q=more+project+dependencies#c509a4c5ea1ae7be


The problem with the <sequentialProject> approach mentioned in that
thread as I indicated right in the very first post is the order of who
picks up the lock when it is released is not deterministic, so you can
indeed get the issue of a second build of A being attempted before the
first C is built. An ordered build queue does solve that particular
problem.


> Basically, you can have a queue develop in the following way. Letters
> denote project names, numbers denote the subversion revision that
> triggered the build. The first build listed is always in progress.
>
> - A_200
> ... project A finishes building ...
> - B_200
> ... a dev commits ...
> - B_200, A_201
> ... project B finishes building ...
> - A_201, C_200
>
> C_200 is now guranteed to fail if A changes shared state in some
> incompatible way. Now, granted, builds shouldn't do that, but we know
> that we have users whos builds do do that. Also, the stated goal of
> "preventing conflicts on shared resources" suggests to me that shared
> state changes are more likely in exactly the situations where people
> are likely to use queues. Shared resources potentially introduce
> shared state.
>
> I'm concerned that a queue facility, particularly one with priorities,
> _looks_ like it's a solution to a build dependency problem, but for
> complex setups is just more rope for users to hang themselves up with.
> I'm a bit conflicted though, because it sure would be nice to have a
> way to limit the number of builds happening at one time, at least for
> performance's sake.


Ok, lets say you give A, B and C the same project queue (which is
effectively the same as the other thread you mentioned achieved by
using the <sequentialProject> approach). We are now saying that only
one build can take place at a time - so hence the scenario of A and C
building simultaneously will not happen. So any "shared state" you
mention (such as a build output folder) will not get erased by parallel
builds.

What will happen is they will get queued. The trick will be to make
sure they get queued in the right order for building. The first answer
I believe is to set the priority for building in reverse order (e.g.
C=1, B=2, A=3). So A will only get added to the queue at a position
after anything else sharing that queue name is completed.

If you choose this approach then what should happen is (picking up from
B_200 in progress):
... developer commits, triggers A added to queue.
B_200, A_201 (since sharing queue, A has not yet started)
... project B reaches publishers section which forces build of C which
will add it to the queue. As C has a higher priority than A it gets
inserted before it:
B_200 (still publishing), C_200, A_201
... project B really completes, releasing next item in queue (C)
C_200, A_201
... A finally reaches front of queue
A_201
... A triggers next force build of B
A_201 (still publishing), B_201
... A finishes
B_201
... B triggers build of C
B_201 (still publishing), C_201
... etc.

So I think this might solve your build dependency problem? To make this
all work you would require that only A has responsibility for "get
latest source", or that the related revision number is passed to B/C.
Otherwise when C_200 reaches it's turn to build, it could pickup
changes from A_201 then you really could have a mess. I guess you have
the same problem already? Of course if you do that then you could never
do a force build of B or C - is that a problem?

I was wondering about a second way of solving it without priorities.
What if project A added B and C to the queue, rather than B adding C?
So your queue will look like this:
A_200 (stilll publishing), B_200, C_200

The problem with this approach is of course what if B fails - you do
not want to build C. I think this would be solvable by adding a new
<cancelPendingBuildOnError> publisher (or similar), which you add to
project B and specify project C as the target. I already have the API
calls in place for cancel pending builds from the queue.


> PS Is the following scenario possible? I think you explained this
> earlier in the thread, but I'm not sure just based on that email. B is
> a very long running build in this scenario.
>
> - A_200
> ... project A completes ...
> - B_200
> ... dev commits ...
> - B_200, A_201
> ... dev commits again ...
> - B_200, A_201, A_202
> ... dev commits one more time ...
> - B_200, A_201, A_202, A_203
> ... etc ...
>
> In an A triggers B triggers C situation that could get quite ugly...


No it is not possible in the existing CC.Net or in my version. In
existing CC.Net, as soon as A_201 starts building (or is held by a
<sequentialProject> lock) it's triggers are suspended. The same applies
to my build queues version - only one pending integration can be queued
at a time for a project. When A gets added to the queue behind B_200,
it's triggers are suspended while it wait's it's turn to build.

When it finally reaches the front of the queue, it then will start
building by sweeping up all changes at that point in time. If revision
A_203 got committed by the time B finishes then it will be that
revision that builds. While A is building, of course A_204 (etc etc)
may get committed. What happens in this scenario is that as soon as
A_203 finishes building it's triggers are enabled again. So 100ms later
they will see further changes are available and add A back to the queue
again.

Hope this makes sense and of course if I've overlooked something let me
know.

kiwidude

unread,
Oct 30, 2006, 10:41:53 AM10/30/06
to ccnet-devel
David Cameron wrote:
> Hi Grant
>
> It turns out there is a message in the archives describing the problem
> I'm concerned about here:
> http://groups.google.com.ag/group/ccnet-user/browse_frm/thread/8e1015588e7d5790/c509a4c5ea1ae7be?tvc=1&q=more+project+dependencies#c509a4c5ea1ae7be

Hi Dave,

The problem with the <sequentialProject> approach mentioned in that
thread as I indicated right in the very first post is the order of who
picks up the lock when it is released is not deterministic, so you can
indeed get the issue of a second build of A being attempted before the
first C is built. An ordered build queue does solve that particular
problem.

> Basically, you can have a queue develop in the following way. Letters
> denote project names, numbers denote the subversion revision that
> triggered the build. The first build listed is always in progress.
>
> - A_200
> ... project A finishes building ...
> - B_200
> ... a dev commits ...
> - B_200, A_201
> ... project B finishes building ...
> - A_201, C_200
>
> C_200 is now guranteed to fail if A changes shared state in some
> incompatible way. Now, granted, builds shouldn't do that, but we know
> that we have users whos builds do do that. Also, the stated goal of
> "preventing conflicts on shared resources" suggests to me that shared
> state changes are more likely in exactly the situations where people
> are likely to use queues. Shared resources potentially introduce
> shared state.
>
> I'm concerned that a queue facility, particularly one with priorities,
> _looks_ like it's a solution to a build dependency problem, but for
> complex setups is just more rope for users to hang themselves up with.
> I'm a bit conflicted though, because it sure would be nice to have a
> way to limit the number of builds happening at one time, at least for
> performance's sake.

> PS Is the following scenario possible? I think you explained this
> earlier in the thread, but I'm not sure just based on that email. B is
> a very long running build in this scenario.
>
> - A_200
> ... project A completes ...
> - B_200
> ... dev commits ...
> - B_200, A_201
> ... dev commits again ...
> - B_200, A_201, A_202
> ... dev commits one more time ...
> - B_200, A_201, A_202, A_203
> ... etc ...
>
> In an A triggers B triggers C situation that could get quite ugly...

No it is not possible in the existing CC.Net or in my version. In
existing CC.Net, as soon as A_201 starts building (or is held by a
<sequentialProject> lock) it's triggers are suspended. The same applies
to my build queues version - only one pending integration can be queued
at a time for a project. When A gets added to the queue behind B_200,
it's triggers are suspended while it wait's it's turn to build.

When it finally reaches the front of the queue, it then will start
building by sweeping up all changes at that point in time. If revision
A_203 got committed by the time B finishes then it will be that
revision that builds. While A is building, of course A_204 (etc etc)
may get committed. What happens in this scenario is that as soon as
A_203 finishes building it's triggers are enabled again. So 100ms later
they will see further changes are available and add A back to the queue
again.

Hope this makes sense and of course if I've overlooked something let me
know.

Regards,

Daniel Piessens

unread,
Oct 30, 2006, 12:22:33 PM10/30/06
to ccnet...@googlegroups.com
Hi Owen,
 
It looks like you 1.1 branched, which is what I would have suggested. I realize it means duplicating/mergin fixes, but I don't have issues with that as we're doing it all the time with work. In fact, there are a number of large changes I'd like to make to the dash that I'd like to discuss. So my questions / suggestions are two fold:
 
   1. Is your branching a go to commit to the trunk? (I can switch up the CCTray Icon)
   2. Can we schedule a time to have a live conversation (I would think at a minimum you, me, Mike, Grant and any active core team members) on Skype or Google Groups and then invite any members interested in joining to discuss the architecture changes (this and a few others).
 
Just throwing it out there...
 
-Dan

 

kiwidude

unread,
Oct 30, 2006, 4:32:41 PM10/30/06
to ccnet-devel

Hi Gary,

I have just posted to the JIRA CCNET-770 an overview document that can
hopefully be used towards help documentation. It has some
comments/questions inside it for further discussion points. I have also
shown some example syntax without actually including some project
config files themselves - if there is anything else you would like to
see please let me know.

http://jira.public.thoughtworks.org/browse/CCNET-770

Regards,
Grant
http://www.kiwidude.com/blog/

David Cameron

unread,
Nov 1, 2006, 2:23:48 AM11/1/06
to ccnet...@googlegroups.com
Hi Grant

I had thought there was another race condition involving the
priorities and sequential builds, but based on the examples you worked
through in your last message I'm not worried about.

Thanks for being patient with my scepticism.

Dave

Gary Feldman

unread,
Nov 1, 2006, 3:42:27 PM11/1/06
to ccnet...@googlegroups.com
kiwidude wrote:
> Gary Feldman wrote:
>
>> ....

>> Being a firm believer in user-centered design, I consider the
>> implementation details to be secondary. Could I ask that you post some
>> sample ccnet.config files illustrating the use of the feature? And a
>> first pass at user documentation?
>>
> I have just posted to the JIRA CCNET-770 an overview document that can
> hopefully be used towards help documentation. It has some
> comments/questions inside it for further discussion points. I have also
> shown some example syntax without actually including some project
> config files themselves - if there is anything else you would like to
> see please let me know.
>
> http://jira.public.thoughtworks.org/browse/CCNET-770
>
I've finally had a chance to read this, and I guess I feel that this is
the proverbial hammer in search of a nail. Exposing queues at the user
level forces the user to learn about an implementation detail, and
worse, to have to be the one to define and build the queues - stuff that
ought to be handled under the covers. I'm not sure it totally solves the
problem, or even that the problem has been thoroughly defined.

For example, suppose I have three projects, A, B, and C. Neither B nor C
can run while A is running, but they can run in parallel with each
other. Putting all three onto one queue would preclude B and C from
running simultaneously.

I wrote an alternative syntax proposal last month, which can be found at
http://sourceforge.net/mailarchive/forum.php?thread_id=30581110&forum_id=32447.
It doesn't preclude using queues for the implementation, but in my
opinion, does a better job of expressing the project constraints in
terms that relate to the problem.

Looking at the example in the documented you posted, consider that:

<cruisecontrol>
<project name=”project1” queueName = “queue1” >

</project>
<project name=”project2” queueName = “queue1” >

</project>
</cruisecontrol>

is semantically identical to

<cruisecontrol>
<project name=”project1” >

</project>
<project name=”project2” queueName = “project1”>

</project>
</cruisecontrol>

since according to the specs, the default queue name is the project
name. The implication of this is that the queue name isn't needed. And,
of course, the queueName attribute could equally well be spelled
"exclusiveWith" or some other term that actually describes what the
build engineer is trying to achieve. Similarly, the project priority can
be expressed without any mention of a queue. The user doesn't much care
about the implementation; it could, in theory, be implemented with no
real queue but by rescanning the entire set of projects at each cycle
and calculating the next project to start.

The queue is simply one particular implementation-specific data
structure by which CC.Net can track projects and select ones to start.
But the build engineer doesn't care how CC.Net implements this decision
process, but rather about expressing the constraints and relationships
between projects, determining the current state of projects, and knowing
what manual operations are available on projects in any given state.

Gary


Owen Rogers

unread,
Nov 6, 2006, 10:29:32 AM11/6/06
to ccnet...@googlegroups.com
On 30/10/06, kiwidude <grant...@gmail.com> wrote:
> lol... I shall see if Stuart has any other colour icons already
> prepared in his bag of tricks...

ah, you know stuart. very cool. do you mind asking him for the .psd
files so that we can check them in and produce new colour icons
ourselves?

> > i do like the idea of a pending state. my impression is that its goal
> > is to communicate to the team that a build will happen when the
> > current build completes. being able to cancel pending requests also
> > seems useful -- especially if you know that the current changes are
> > going to break the build.
>
> If you want to see an example of how this looks visually in CCTray I
> have put together an animated gif on my blog from the work done this
> weekend - the entry is here:
>
> http://www.kiwidude.com/blog/2006/10/cruisecontrolnet-serialised-build_29.html

i think that it looks great. thanks for all your hard work on it.

Owen Rogers

unread,
Nov 6, 2006, 10:33:12 AM11/6/06
to ccnet...@googlegroups.com
hi grant,

On 30/10/06, kiwidude <grant...@gmail.com> wrote:

> I have just posted to the JIRA CCNET-770 an overview document that can
> hopefully be used towards help documentation. It has some
> comments/questions inside it for further discussion points. I have also
> shown some example syntax without actually including some project
> config files themselves - if there is anything else you would like to
> see please let me know.
>
> http://jira.public.thoughtworks.org/browse/CCNET-770

i spent some more time yesterday reviewing the code. i really like
the implementation. it gets around a number of issues that i
encountered when i was trying to do this. what i would like to do is
integrate this work in pieces, starting with the integrationqueue and
ending with the cctray changes. thanks again for your work on
implementing this.

JohnF

unread,
Nov 10, 2006, 3:53:15 PM11/10/06
to ccnet-devel
I modified the CC.NET 1.1 source to provide a solution for this as we
just needed it.

In my ccnet.config file I have an optional element:

<serverinfo>
<concurrentIntegrators>1</concurrentIntegrators>
</serverinfo>

Set it to whatever you want (the above would be a single project at a
time).

I added a "BlockedForResources" to ProjectIntegratorState to solve the
visibility issue and created a fair semaphore object which is shared by
each project integrator to mitigate the contention (the .NET semaphore
was clearly not fair - we found some projects that barely got built at
all and others all the time...).

This is one of several enhancements I've made - I'll post a summary of
the rest here in a second.

What I really came to the group for was how to do I submit my diffs
back to the group?

David Cameron

unread,
Nov 12, 2006, 8:25:49 PM11/12/06
to ccnet...@googlegroups.com
Hi John

The best way is to attach a patch to an outstanding JIRA issue. JIRA
is our issue/ticket database at http://jira.public.thoughtworks.org .
If there isn't a JIRA issue, you can make one.

Second choice is to post .patch files to this mailing list.

Dave

Reply all
Reply to author
Forward
0 new messages