how to avoid continuous integration build problems blocking agile developers
Andrew
think alot about build problems during continuous integration. I see
three problems:
1. the build fails when any of the tests fail. This is the number ONE
reason for the build going red. IMO the build should only go red when
there is a failure to build. Testing is something else. The reason for
it being done this way is, of course, that when a test fails it
indicates that somewhere there is broken code, i.e. something has
stopped working. However, what often happens is that an area requires
more test data so it is added and this breaks the tests that are in a
completely different area.
2. When the build is red this blocks the developers from checking in
their changes. This means that when the build goes green a flurry of
changes are committed in a very short space of time which often makes
the build go red again. This the project is very much stop-start-stop.
The periods when it is stopped tend to last a long time, due to the
long time it takes to do a build because it takes a long time to run
all the tests.
3. Fixing defects does not take first priority. There may be dozens of
defects even when the build is green. These will be filed in a bug
reporting system but there are no failing tests for them. Hence, when
a
build goes red due to a failing test, by stopping and fixing it a
relatively minor problem is fixed whilst many more problems are left
unsolved.
I welcome suggestions on how to improve on this situation. Here are
some initial thoughts:
First, I would distinguish between a failure to actually build (e.g.
due to a compilation error or cyclic dependency) and failing tests. I
would only make the build go red on the former. Under these conditions
when the build goes red I would programmatically stop all checkins. I
would arrange things so that change sets only get a final commit once
they have passed through the build. The build without tests is quite
quick, taking only a few minutes, so IMO this would not be too
onerous.
Second, I would establish a rule that defects must be fixed before
there is any coding for new functionality. This is supposed to be the
agile way anyway. I would try to find a way whereby a failing test
automatically raises a defect, ensuring that it gets fixed as rapidly
as possible. I think that when such test failures are as a result of
fragile tests that are overly sensitive to the addition of new test
data this rule will quickly make such tests less sensitive.
Can anyone suggest any other ways to avoid the build problems I have
described?
Regards,
Andrew Marlow
http://www.andrewpetermarlow.co.uk
John Roth
The standard XP answer is that you should never have failing
unit tests. Period. End of discussion.
So I'd ask why failing unit tests are happening in the integration
process in the first place? This should be a very unusual occurrence,
not an everyday occurrence.
At a guess, the relevant part of the test suite isn't being run on
the developer's machines. If this is so, then that's the problem
that needs to be fixed.
There are, of course, a dozen other possibilities. Most XP discussion
takes place on yahoo Groups in the extremeprogramming mailing list.
You'll get a much more thorough discussion there.
John Roth
Andrew
> On Sep 6, 11:59 am, Andrew <marlow.and...@googlemail.com> wrote:
> > I welcome suggestions on how to improve on this situation. Here are
> > some initial thoughts:
>
> > First, I would distinguish between a failure to actually build (e.g.
> > due to a compilation error or cyclic dependency) and failing tests. I
> > would only make the build go red on the former.
> The standard XP answer is that you should never have failing
> unit tests. Period. End of discussion.
I should have mentioned that the failing tests are always at
integration or acceptance level. Of course the unit tests should
always run as part of the build and any unit test failure should cause
the build to fail.
> So I'd ask why failing unit tests are happening in the integration
> process in the first place? This should be a very unusual occurrence,
> not an everyday occurrence.
See above.
> At a guess, the relevant part of the test suite isn't being run on
> the developer's machines. If this is so, then that's the problem
> that needs to be fixed.
The test suite IS being run on the developers machine.
John Roth
> I should have mentioned that the failing tests are always at
> integration or acceptance level. Of course the unit tests should
> always run as part of the build and any unit test failure should cause
> the build to fail.
XP doesn't distinguish between unit and integration tests; code
is integrated on the developer's machines. What the integration
system is for is to catch the, hopefully very rare, occasions where
two developers have made incompatible changes and a race
condition occurs.
In the same sense, previously passing automated acceptance tests
should pass on the developer's machines before the code is checked in.
>
>
> > There are, of course, a dozen other possibilities. Most XP discussion
> > takes place on yahoo Groups in the extremeprogramming mailing list.
> > You'll get a much more thorough discussion there.
Let me emphasize that you'll get a much more thorough discussion
on the Yahoo Groups ExtremeProgramming list than you will here.
You're lucky that anyone at all is still looking at it - this usenet
group
is pretty much dead.
>
> > John Roth
Paul Sinnett
> The test suite IS being run on the developers machine.
Can you describe a case where a test passes on the developer's machine
but fails on the server?
What is the cause of the error?
Andrew
> On Sep 8, 7:59 am, Andrew <marlow.and...@googlemail.com> wrote:
> > The test suite IS being run on the developers machine.
> Can you describe a case where a test passes on the developer's machine
> but fails on the server?
Yes. The selenium tests are failing due to timing issues. Sometimes
the test code waits for something asynchronous to happen and does not
wait long enough. These errors are intermittant.
>
> What is the cause of the error?
Problems in the test code. It is very dangerous to have to wait for
asynchronously computed results in tests.
John Roth
Do these timing problems show up in production, or are they an
artifact of the way that tests are being written?
John Roth
Paul Sinnett
> Yes. The selenium tests are failing due to timing issues. Sometimes
> the test code waits for something asynchronous to happen and does not
> wait long enough. These errors are intermittant.
My guess is that a pure XP approach is unsuited to the project you are
describing. I think you'll need to make adjustments for these
problems. You describe the size of the project as "large," whereas XP
was designed for a team of a dozen. You have real-time requirements,
whereas XP was designed on (I think I'm correct in saying) an offline
batch processing program.
I'm sure it's possible to run your in an agile way. But I think trying
to follow XP by the book might not be the best approach in this case.