ANNOUNCE: CC.NET Plug-In's
Richard Hensley
I've created three plug in's that I needed to get my builds running smoothly. I've released them on sourceforge for anybody who might find them useful.
The home page is http://ccnetplugins.sourceforge.net
The plug-in's are:
sequential - A task that only allows tasks to run one at a time. This task works across and within a project, and can cooperate with sequentialSource
sequentialSource - A source control provider that allows source control operations to run one at a time. This task works across and within projects, and can cooperate with sequential. This source control is a drop in replacement for the multi source control provider.
pathfilter - A source control provider that filters modifications using strict Apache Ant patterns.
The sequential plug-in's were created so that at any give time, only a single cvs task was pounding on the cvs server, or only a single nant task was pounding on the build server. The sequential task also helps when a global resource is needed by a build, such as an ncover library that is registered and unregistered by the build process.
Richard Hensley
Moshe Hajaj
I have set my new machine of CCNet (v1.1.1) to work with sequential
task, and it just does what is said. putting nant in sequential for all
projects per build machine gives me the singleton solution that has
been discussed here and there (jira, sourceforge) lengthy.
David Keaveny
Hmm, interesting – just last week I wrote a CC.NET plug-in to create version numbers from Subversion revision numbers. Would I be able to add it to your site if I wrote up some HTML copy documenting usage?
David Keaveny
Richard Hensley
Sure. I will send you private e-mail to work out the details.
Subject: [ccnet-user] Re: ANNOUNCE: CC.NET Plug-In's
Date: Mon, 9 Oct 2006 09:19:53 +1000
From: dkea...@fairfaxdigital.com.au
To: ccnet...@googlegroups.com
kiwidude
Congrats on the nice work. There is however one situation that I think
falls through the cracks with this model for the sequential task.
The issue is when you still need to do some work in the <publishers>
section before you feel ready to give up the lock. For instance, the
first thing people will often have in their build is a "clean" task
which will erase the build output folder. It's also likely that they
will produce a series of artifacts during the build, such as unit test
xml or even html reports. Then you will have in the <publishers>
section merge tasks and possibly additional NAnt tasks copying those
artifacts.
In this situation you want to hold the lock open over both the tasks
and the publishers, and not give it up as soon as the task exits.
Otherwise surely it would be possible for a queued follow up build to
kick off, execute its clean task and subsequently erase the results of
the previous build before they had been copied out/merged?
Am I missing something here that you cater for already or can this
situation happen?
I guess one workaround for us is to have a small sleep task prior to
the clean, to give the previous build a chance to do what it needs to
do.
Your thoughts?
Grant.
Richard Hensley
I had to modify my builds to ensure that the disk area they were writing to was private. In other words, no matter when the build kicks off, it has a folder that is specific to the tasks that will execute. I did this after running into the fighting builds problem you describe. I was lead down this road after reading the multiple project CCNET documentation with basically says you must have a private playground for each project. After the main build completes, my nant script copies all the executable and report artifacts to a known place that is not cleaned by a build process.
Basically, I had to eliminate the problem of holding a lock over the pre-build, tasks, and publishers to get my builds to work correctly.
Now, if I had that problem, the only way I could think of to solve it would be at the container level. In otherwords, adding a feature to the project that would cause it to gain a lock when a trigger occurs, and release it after everything is complete. Frankly, I didn't even look into that level of change to the core of ccnet.
> From: grant...@gmail.com
> To: ccnet...@googlegroups.com
kiwidude
Yeah many of us are hitting that same issue I think, and like you I
believe one possible solution would lie with a container level locking
mechanism. Also like you I had gone down a path of a solution which
didn't involve CC.Net changes (mine involved NAnt tasks using lock
files rather than mutexes, with a call as a publisher to another NAnt
task to release the lock file after all work is done). I like the
elegance of your implementation far better - apart from this little
problem to be aware of and workaround as I indicated.
In terms of separating the builds - its not just a case of separate
playgrounds for each CC.Net project, which we do indeed do. There is
also the problem which I really meant above of when multiple builds on
the SAME project are kicked off. This could be by someone doing a force
build while a build of that project is in progress, or indeed if you
have a long build process someone doing a checkin while a build of that
project is in progress. In these situations unless you would need a new
playground for every build number - which would be safest but I think
would be difficult to implement without changes to CC.Net.
We have also had issues with the NAnt <solution> task choking on .resx
files being locked in the temp directory, presumably while two projects
are trying to compile at the same time. That would imply a need for a
global lock for the short period of a compile as well as a project lock
- it all starts getting rather complicated. For now the guys just see
it failed for that reason and force build again which is obviously
simpler!
It does make me wonder how people do get on with scaling with CC.Net on
a single build server for more than just a couple of continuous build
projects...
Regards,
Grant.
Norman Rasmussen
> It does make me wonder how people do get on with scaling with CC.Net on
> a single build server for more than just a couple of continuous build
> projects...
I've got 28 projects on my build server at the moment, and with plain
vss source, and 60 second monitoring the box can hog around 8mb/s of
network, load average ~10 (just checking vss history). With the
fileSystem filter in place, I can keep this to below 100kb/s, la ~1,
assuming that all the filters are working correctly.
At the moment we've just added a few new projects, and I haven't got
the filters from them right yet, so it's using 1.5mb/s, la ~6.
I'm using MRTG [1] running on another machine to graph these details,
using the Load Average service from Arkane Systems [2].
[1] http://oss.oetiker.ch/mrtg/
[2] http://www.arkane-systems.net/products/LoadAverage/index.aspx
--
- Norman Rasmussen
- Email: nor...@rasmussen.co.za
- Home page: http://norman.rasmussen.co.za/
Richard Hensley
One of the reason's that I built the pathfilter plug-in was because I could not get the filtered source plug in to do what I wanted. I investigated the source, but noticed a large amount of regex voodoo. So, I opted for a solution that I knew worked correctly, the Apache Ant source code. The code in the path filter is a direct port of the relavent methods from the Apache Ant project to C# and then integrated into a CCNET plugin. As I recall, the specific use case that I had was getting filters to accept files against patterns like the following:
pattern = **/artifact/**/*.*
file = /work/nantbuild/target/artifact/project/x.dll
If you are having difficulties with the builtin filter source control, you might want to check out the ant path filter.
> Date: Mon, 9 Oct 2006 17:14:12 +0200
> From: norman+c...@rasmussen.co.za
Richard Hensley
I think the main problem I need to address is ensuring that the report artifacts of a build remain intact and available through the publish phase of the build process. So, if triggered and a forced builds start at the same time. The report artifacts of the triggered build need to be merged with the build log for the triggered build, and the report artifacts of the forced build need to be merged with the build log of the forced build.
So, maybe a lock and unlock task? My only heartburn with such an animal is the danger in misusing these two task. If for some reason, the unlock was missed, a fail safe would be needed.
I'm not familiar enough with CCNET to know if the following would work correctly in the success and failure cases. I reviewed the Project.cs source code, it appears that all publishers would be executed, unless one throws an exception. I know this would not work if any of the tasks throws an exception.
<tasks>
<lock lockName="mine" /> <!-- at the top of the tasks -->
.... other stuff ....
</tasks>
<publishers>
.... other stuff ....
<unlock lockName="mine" /> <!-- at the bottom of publishers -->
</publishers>
Another solution would be to create a plug-in that is a new project class.
This class would extend Project
Give it a new NetReflector name like sequentialProject
It would add a lockName property
It would construct the base Project using an IntegrationRunner that locks using the lockName around the getmodifications, check for build, prebuild, getsource, build, labelsource, and publish phases.
Hmmm, that second one has some merit, and is easy to accomplish.
Any other suggestions?
> From: grant...@gmail.com
> a single build server for more than just a couple of continuous build
> projects...
>
> Grant.
>
kiwidude
> Another solution would be to create a plug-in that is a new project class.
> This class would extend ProjectGive it a new NetReflector name like sequentialProject.
> It would add a lockName property.
> It would construct the base Project using an IntegrationRunner that locks using the lockName
> around the getmodifications, check for build, prebuild, getsource, build, labelsource, and publish phases.
> Hmmm, that second one has some merit, and is easy to accomplish. Any other suggestions?
This one sounds great - can't comment on the "technical difficulty"
having not looked at the source code but if flies then it would be a
winner for me as far as my first thoughts are about it. The bonus of
this would be that it should be possible to combine it with your
<sequential> task, catering for that situation where you have a
resource on the build server that needs (for instance) a global lock
for a brief period unlike the project itself...
Great stuff, ready by tomorrow then? ;-)
Grant.
Richard Hensley
I'm still investigating, but I'm leaning towards changing CCNET to allow for injecting the dependencies, and copying the project class just to get running.
Comments?
From: rhens...@msn.com
To: ccnet...@googlegroups.com
Subject: [ccnet-user] Re: ANNOUNCE: CC.NET Plug-In's
Date: Mon, 9 Oct 2006 10:28:45 -0600
Oh, well I haven't run into that case yet. You are right, that would cause genuine havoc with my build system. Hmmm... Gonna have to consider this one more closely.
I think the main problem I need to address is ensuring that the report artifacts of a build remain intact and available through the publish phase of the build process. So, if triggered and a forced builds start at the same time. The report artifacts of the triggered build need to be merged with the build log for the triggered build, and the report artifacts of the forced build need to be merged with the build log of the forced build.
So, maybe a lock and unlock task? My only heartburn with such an animal is the danger in misusing these two task. If for some reason, the unlock was missed, a fail safe would be needed.
I'm not familiar enough with CCNET to know if the following would work correctly in the success and failure cases. I reviewed the Project.cs source code, it appears that all publishers would be executed, unless one throws an exception. I know this would not work if any of the tasks throws an exception.
<tasks>
<lock lockName="mine" /> <!-- at the top of the tasks -->
.... other stuff ....
</tasks>
<publishers>
.... other stuff ....
<unlock lockName="mine" /> <!-- at the bottom of publishers -->
</publishers>
Another solution would be to create a plug-in that is a new project class.
This class would extend Project
Give it a new NetReflector name like sequentialProject
It would add a lockName property
It would construct the base Project using an IntegrationRunner that locks using the lockName around the getmodifications, check for build, prebuild, getsource, build, labelsource, and publish phases.
Hmmm, that second one has some merit, and is easy to accomplish.
Any other suggestions?
Richard
Richard
reflection voodoo.
Specifically, I had to use Type.GetField() to find the private field.
Using the FieldInfo object returned, I retrieved the IIntegratable
object from the Project, wrapped it up in a IIntegratable
implementation that locks and delegates, and then set the value back
into the field.
I'm testing my implementation right now in my ccnet server. When it is
done cooking, I will make a ccnetplugins release that includes a
sequentialProject element.
Owen Rogers
On 09/10/06, Richard Hensley <rhens...@msn.com> wrote:
> Another solution would be to create a plug-in that is a new project class.
>
> This class would extend Project
> Give it a new NetReflector name like sequentialProject
> It would add a lockName property
> It would construct the base Project using an IntegrationRunner that locks
> using the lockName around the getmodifications, check for build, prebuild,
> getsource, build, labelsource, and publish phases.
>
> Hmmm, that second one has some merit, and is easy to accomplish.
this was possible at one time and was a good way to extend the basic
ccnet workflow. however, because of some configuration loading
changes a while back, this is no longer possible directly -- though i
would like to get it back. as it looks like you (and others) are
making more use of the plugin model (yay!), i'd like to provide some
more support in this regard.
cheers,
owen.
--
Owen Rogers | http://dotnetjunkies.com/weblog/exortech |
CruiseControl.NET - http://ccnet.thoughtworks.com
Richard Hensley
Maybe I should join the developers list because I have some refactoring
suggestions that would be helpful in the extension model.
I was able to work around my issue with the following ugly hack.
IIntegratable i = GetIntegratableViaReflection();
_integrationRunner = new SequentialIntegrationRunner(i);
SetIntegratableViaReflection(_integrationRunner);
The methods "...ViaReflection do exactly what the name implies." Dig
around in the Project class for a private field using relfection and
change it. Not very pretty, but the best way I could figure out to
reuse all the hardwork that exists in the Project class.
On Oct 11, 12:16 am, "Owen Rogers" <exort...@gmail.com> wrote:
> still catching up on old emails...
>
> On 09/10/06, Richard Hensley <rhensle...@msn.com> wrote:
>
> > Another solution would be to create a plug-in that is a new project class.
>
> > This class would extend Project
> > Give it a new NetReflector name like sequentialProject
> > It would add a lockName property
> > It would construct the base Project using an IntegrationRunner that locks
> > using the lockName around the getmodifications, check for build, prebuild,
> > getsource, build, labelsource, and publish phases.
>
> > Hmmm, that second one has some merit, and is easy to accomplish.this was possible at one time and was a good way to extend the basic