Continuous Outegration
Phlip
Suppose your project tests frantically, integrates continuously, and
releases easily.
Suppose then you hire a new coder, and need another workstation.
Then you have a party. All the source requires 3rd-party libraries, of
various versions, which must be matched, and these require configuration
options and scripts installed here there and everywhere. The project's
eldest guru must pair with the new guy, run the system over and over again,
diagnose each glitch, and reconfigure it away. This situation resembles
delaying integration long enough to encounter "integration hell". Instead of
smooth saling you get dominos of bugs.
Is this a process smell? Does it indicate someone should add a motherhood
story indicating the codebase is continuously outegratable? Would an
off-the-shelf configuration manager handle it? If so, wouldn't configuring
that tool be a chore each time the team changes a library?
(Cloning a dev machine's hard drive is cheating, and insecure.)
--
Phlip
http://www.xpsd.org/cgi-bin/wiki?TestFirstUserInterfaces
Laurent Bossavit
> codebase is continuously outegratable?
"All release builds are to be done off a virgin workstation."
VMWare provides an economical way of getting a virgin workstation
whenever you need one.
If you release frequently enough (like once per iteration), that rule
would flush out the appropriate solutions to that problem in your
context. (There's a meta-pattern here - "Prescribe the problem".)
"Eldest guru" is an antipattern. The new guy has to be able to make a
product build, provided only a pointer to the source repository and a
page of documentation. The dark lore of what incantations are required
to reproduce a build needs to be extracted from the gurus and stored in
the form of shell scripts and (lightweight) docs.
Third-party version issues: there are different kinds of dependencies
involved, and thus as many different kinds of solutions. "Nice"
components (dependency is on a single file, such as a JAR file) go into
a "binary" directory of the version control tool; the file needs to be
adorned with a version number. Components with a more complex directory
structure can reduce to "nice" components if you zip up the whole thing.
"Nasty" components (such as Oracle or Sybase DBs) are a different
problem. One solution is to install those on your virtual workstation,
off a "clean" OS, and snapshot the workstation's hard disk. Label the
snapshot file with a version number and store it with binaries. As soon
as you have more than one "nasty" component, that stops being as good a
solution - you get closer to "clone a developer's hard disk".
Sean Case
"Phlip" <phli...@yahoo.com> wrote:
> Then you have a party. All the source requires 3rd-party libraries, of
> various versions, which must be matched, and these require configuration
> options and scripts installed here there and everywhere.
This is why we have configuration management. You need to know how to
create a new workstation when needed, just as you need to know how to
replace one of your servers.
> The project's
> eldest guru must pair with the new guy, run the system over and over again,
> diagnose each glitch, and reconfigure it away.
Interesting that you do it by pairing, rather than having a document,
which would be more usual in a non-extreme setting. But surely anyone
should be able to do it... maybe you could have a different person
configure a blank machine each week, to keep the knowledge live. I'd
still recommend writing something down, though.
> This situation resembles
> delaying integration long enough to encounter "integration hell". Instead of
> smooth saling you get dominos of bugs.
You forestall integration problems by integrating frequently; you could
do the same for configuration issues by regularly taking a blank machine
and following the script all the way through to checking out and building
the product.
Sean Case
--
Sean Case g...@zip.com.au
Code is an illusion. Only assertions are real.
John Roth
"Phlip" <phli...@yahoo.com> wrote in message
news:S5e2c.56030$J97....@newssvr33.news.prodigy.com...
> eXtremos:
>
> Suppose your project tests frantically, integrates continuously, and
> releases easily.
>
> Suppose then you hire a new coder, and need another workstation.
>
> Then you have a party. All the source requires 3rd-party libraries, of
> various versions, which must be matched, and these require configuration
> options and scripts installed here there and everywhere. The project's
> eldest guru must pair with the new guy, run the system over and over
again,
> diagnose each glitch, and reconfigure it away. This situation resembles
> delaying integration long enough to encounter "integration hell". Instead
of
> smooth saling you get dominos of bugs.
>
> Is this a process smell? Does it indicate someone should add a motherhood
> story indicating the codebase is continuously outegratable? Would an
> off-the-shelf configuration manager handle it? If so, wouldn't configuring
> that tool be a chore each time the team changes a library?
Yes, it's a process smell of the same order as assuming that a program
is never going to be updated, so it isn't important to write it clearly.
I'm not going to write a motherhood story for that, just like I'm
not going to write a motherhood story for coming to work in
locally acceptable attire. There are certain things you do because
they are "generally accepted good practice", and keeping everyone's
development environment in synch with the integration system is
one of them.
If the customer wants to cut corners on that, then we've got a
real division of responsibility issue.
John Roth
Phlip
> > Does it indicate someone should add a motherhood story indicating the
> > codebase is continuously outegratable?
>
> "All release builds are to be done off a virgin workstation."
>
> VMWare provides an economical way of getting a virgin workstation
> whenever you need one.
>
> If you release frequently enough (like once per iteration), that rule
> would flush out the appropriate solutions to that problem in your
> context. (There's a meta-pattern here - "Prescribe the problem".)
>
> "Eldest guru" is an antipattern. The new guy has to be able to make a
> product build, provided only a pointer to the source repository and a
> page of documentation. The dark lore of what incantations are required
> to reproduce a build needs to be extracted from the gurus and stored in
> the form of shell scripts and (lightweight) docs.
The eldest guru does not assist every danged release.
(The corollary here is that every workstation can simulate building a
release.)
In my parable, the guru must assist installing the build environment onto a
new workstation, the first time, before the new kid can use it. But you are
calling _that_ an antipattern.
So by "all release builds are to be done off a virgin workstation" do you
mean the workstation has no libraries, compilers, script languages,
databases, or anything, and the top-level build script installs everything
that wasn't there?
If so, all projects that release anything significant run under the
Motherhood Story "each time you upgrade, look at, or think about a new tool,
you must upgrade the build scripts to test automatically installing it onto
a virgin release computer, or a new development workstation.
If so, the XP verbiage under-represents this. Newbie teams ain't doing it.
Some components fiercely resist automated installing. Researching how to
squeeze them into a script could turn into a very long job.
Sean Case wrote:
> Interesting that you do it by pairing, rather than having a document,
> which would be more usual in a non-extreme setting. But surely anyone
> should be able to do it... maybe you could have a different person
> configure a blank machine each week, to keep the knowledge live. I'd
> still recommend writing something down, though.
In my antipattern, if they didn't think to script it, they wouldn't think to
document it. Would you like to be the newbie, handed a shabby and obsolete
document, and told "here, install this stuff, and don't look bad or
interrupt us, on your first day?"
Next, manually practicing installing breaks the rule "Three Strikes and you
Automate". Anything a programmer finds themselves doing often enough to spot
the pattern, they have a duty to automate that task.
> You forestall integration problems by integrating frequently; you could
> do the same for configuration issues by regularly taking a blank machine
> and following the script all the way through to checking out and building
> the product.
In our pattern, one need only execute the script, and it does the rest -
installs, configures, checks out source, rebuilds totally, tests totally,
builds an installer, and tests the installer.
(A related question: Would probing the installer with its own diagnostic
tools be enough to "test" it? Or should we install onto our own clean
machine to test it?)
--
Phlip
http://www.xpsd.org/cgi-bin/wiki?TestFirstUserInterfaces
Laurent Bossavit
I've consulted to one firm where the people /thought/ they could do a
build without the Eldest in attendance. Then one day they actually tried
that, at my prompting. Surprise...
> In my parable, the guru must assist installing the build environment onto a
> new workstation, the first time, before the new kid can use it. But you are
> calling _that_ an antipattern.
Indeed I am. "What we need to do a build" is part of the intellectual
capital tied up in the product, just as much as the source code. It must
be inspectable, reviewable, modifiable, just as much as the source code.
> So by "all release builds are to be done off a virgin workstation" do you
> mean the workstation has no libraries, compilers, script languages,
> databases, or anything, and the top-level build script installs everything
> that wasn't there?
As close to that as you can manage. How close you can manage depends
wildly on the environment, though. Installing a Linux environment brings
in any number of nice things, such as CVS, gcc, etc. which only require
checking a few checkboxes after inserting the CD. An equivalent Windows
setup is more involved.
I'd install "big" items such as the compiler and databases manually,
then ghost the disk or snapshot the virtual PC, and call the resulting
image a "virgin workstation". Everything less pervasive, such as third
party libraries, build tools, etc. would go into a repository of some
sort.
> If so, all projects that release anything significant run under the
> Motherhood Story "each time you upgrade, look at, or think about a new tool,
> you must upgrade the build scripts to test automatically installing it onto
> a virgin release computer, or a new development workstation.
You can probably dispense with scripting for "think about", but yes, if
you bring in something crucial to reproducing the product, the decision
has to be under collective ownership - reviewable, testable, etc.
It's also a reason to minimize the frills and keep the build design
simple, just like the code design.
> Some components fiercely resist automated installing. Researching how to
> squeeze them into a script could turn into a very long job.
I'm fiercely hoping that widespread application of the principles I have
outlined turns into a market force, driving out of the market tools
which resist reviewability of *any* part of the product design. :)
> Would you like to be the newbie, handed a shabby and obsolete
> document, and told "here, install this stuff, and don't look bad or
> interrupt us, on your first day?"
Point. :)
"Hi Dan, this here is your brand new workstation. Please take ten
minutes to pull down the sources from the CVS repository 'CoolProduct',
which lives on the server named Gandalf. Run the top-level make script;
when you get to a green bar, raise your hand and I'll scoot over to pair
with you, and introduce you to the story I'm working on. We'll see how
well that twenty-minute presentation of the metaphor from this morning
worked. If we've screwed up and you don't get to a green bar within
twenty minutes, just holler."
> (A related question: Would probing the installer with its own diagnostic
> tools be enough to "test" it? Or should we install onto our own clean
> machine to test it?)
This gets hairy if the installer does things like tweak the Windows
registry. In the simplest case, the installer is just a wrapper atop the
equivalent of unzipping an archive to a directory; I wouldn't test much
beyond "doing a build results in populating a directory, the contents of
which match the following manifest".
In more complex cases, there will be user stories driving what the
installer does, and the installer is a subproject in its own right; my
prejudice is that we'll have to find a way to separate testing the
installer from testing the main product.
Jeff Grigg
> "Eldest guru" is an antipattern. The new guy has to be able to make a
> product build, provided only a pointer to the source repository and a
> page of documentation. The dark lore of what incantations are required
> to reproduce a build needs to be extracted from the gurus and stored in
> the form of shell scripts and (lightweight) docs.
For builds, yes. But for setting up and configuring a new workstation
(which should be just like the existing workstations), I'd say it's a
common anti-pattern to inflict this on the new guy.
I favor having an experienced member of the team help the new guy set
up the development environment and get it running. They should be
able to do private builds, updates, etc. from then on, with no further
outside help and minimal documentation. (But in a pairing
environment, help would be well neigh inescapable, I think. ;-)
I've had success with starting development machines from a clean
master image. It's *much* better for an experienced developer to
create the master image by building on a clean machine than to copy an
existing development machine that's been in use for a number of
months. The problem this solves is that it can often take *DAYS* to
install all the appropriate software needed on a development
workstation from CDs. People forget about this issue.
> Third-party version issues: [...]
All the developers must be on the same version of the libraries (if
they're working on the same release version of the software). To
upgrade, one or two developers can "experiment" with the new version
until they get the bugs worked out, and then everyone else upgrades.
Internet Explorer is the product I've had the most trouble with: It's
somewhere between hard and impossible to get old versions, and they're
changing it all the time -- with automated downloads. On my current
project, which uses extended browser functionality extensively, this
has been a significant thorn in our sides for some time. (Eventually
we dedicated a person to track down and fix the problems in our code,
so it would be more portable across versions.)
> "Nasty" components (such as Oracle or Sybase DBs) [...]
Databases are usually upward compatible. But that doesn't mean that
successful integration test with version 9 will work the same in
production with version 8. It's usually not a big problem, but it
*can* bite you. (I've found that people being in denial are a bigger
problem than the actual technical issues, on this one.)
Bil Kleb
> Phlip wrote:
> > The eldest guru does not assist every danged release.
>
> I've consulted to one firm where the people /thought/ they could do a
> build without the Eldest in attendance. Then one day they actually tried
> that, at my prompting. Surprise...
>
The whole exchange was simply excellent. Very refreshing.
Thanks,
--
Bil Kleb, NASA, Hampton, Virginia
sirgilligan
The way we do it is through the use of an image of the tools for the
current configuration and a wiki site to take you through any steps
that are needed for finishing off the configuration and where the CVS
repository resides and which CVS modules to check out. I have noticed
that there is need for some personal interaction with any new hire.
They feel that they have to prove themselves and if the wiki site has
incorrect directions then the new hire might feel like they have
failed their first task (since they might assume the steps are
correct).
I find it a more concerning problem that early on the new hire is
taught that the senior guru is busy and not to bother him! Does anyone
consider the cost of properly integrating a new hire? If you train a
new hire not to "bother" anyone, then how can one complain when they
do not communicate?
M2CW.
Geoff
Robert Atkins
> involved, and thus as many different kinds of solutions. "Nice"
> components (dependency is on a single file, such as a JAR file) go into
> a "binary" directory of the version control tool; the file needs to be
That bit is easy... we have lib under version control.
> "Nasty" components (such as Oracle or Sybase DBs) are a different
I've been dealing with this all week.
Our project team have just moved from the customer site to our office
(yes yes, onsite customer and all that -- but you don't want to go
there). Recent staff movements have made *me* the resident guru (!) and
I've had to set up four new workstations.
Install ATG Dynamo, patch ATG Dynamo, install IIS, install IIS
connection module, configure IIS connection module by tweaking registry,
install server and client certs for IIS. Install VSS client. Configure
VSS client on Novell network with no NT authentication. This stuff is
*very* hard to automate. Some of these things can only be scripted on
paper (especially under rotten environments like Windows).
For the frequency it is done, I think the most practical solution is the
one mentioned upthread: local guru gives setup doc to newbie and says,
"I'll be over in 30 minutes if (when?) it doesn't work". Note that I
don't like this either.
Cheers, Robert.
Phlip
> Install ATG Dynamo, patch ATG Dynamo, install IIS, install IIS
> connection module, configure IIS connection module by tweaking registry,
> install server and client certs for IIS. Install VSS client. Configure
> VSS client on Novell network with no NT authentication. This stuff is
> *very* hard to automate. Some of these things can only be scripted on
> paper (especially under rotten environments like Windows).
I worked a site with a cool solution for "virgin release machine" - but not
for warming up a newbie's 'puter.
They ran Linux with a dual-boot. The Linux partitions could treat the other
partitions as data, and copy complete images in. So the Linux side had a
menu of install sets to chose from.
Note that the Law of Pairing requires the newbie to pair the first day, on
communal machines. If they all roll over and one falls out, that will be the
guru, who must now configure a new workstation for himself.
(Rob - I have to ask - have you been the target of much ribbing for all the
"health food" these days that tastes like the cardboard tube from the center
of a roll of paper towels?
(Sorry - I couldn't resist;)
--
Phlip
http://www.xpsd.org/cgi-bin/wiki?TestFirstUserInterfaces
Robert Atkins
> partitions as data, and copy complete images in. So the Linux side had a
> menu of install sets to chose from.
This sort of thing is much easier (possible?) when you have a Real
Operating System, but often you don't get that choice.
> (Rob - I have to ask - have you been the target of much ribbing for all the
> "health food" these days that tastes like the cardboard tube from the center
> of a roll of paper towels?
Actually, being in Australia I wasn't aware of my more famous namesake
until someone emailed me thanking me for operating on his wife's kidneys
15 years ago. And you've got no idea how odd it is to wake up to a
newspaper headline telling you that you're dead...
Cheers, Robert.
Sean Case
"Phlip" <phli...@yahoo.com> wrote:
> Note that the Law of Pairing requires the newbie to pair the first day, on
> communal machines. If they all roll over and one falls out, that will be the
> guru, who must now configure a new workstation for himself.
So how do people other than the guru learn how to configure new
workstations?
Phlip
> Phlip wrote:
>
> > Note that the Law of Pairing requires the newbie to pair the first day,
on
> > communal machines. If they all roll over and one falls out, that will be
the
> > guru, who must now configure a new workstation for himself.
>
> So how do people other than the guru learn how to configure new
> workstations?
Note that the Law of Promiscuous Pairing requires the guru to pair when
configuring a new workstation. Note this is also a development effort -
tuning all the scripts.
--
Phlip
http://www.xpsd.org/cgi-bin/wiki?TestFirstUserInterfaces
Robert Atkins
> In article <nb_3c.57802$eG1....@newssvr33.news.prodigy.com>,
> "Phlip" <phli...@yahoo.com> wrote:
>> Note that the Law of Pairing requires the newbie to pair the first day, on
>> communal machines. If they all roll over and one falls out, that will be the
>> guru, who must now configure a new workstation for himself.
>
> So how do people other than the guru learn how to configure new
> workstations?
When the Guru leaves or is incapacitated, the apprentice guru must
puzzle it out from the available docs.
Cheers, Robert.
Phlip
> When the Guru leaves or is incapacitated, the apprentice guru must
> puzzle it out from the available docs.
Important tip to ambitious apprentices: Make sure your guru has updated the
documents before incapacitating them to take their position.
--
Korax
neurophysical-reeducator
> eldest guru must pair with the new guy, run the system over and over again,
> diagnose each glitch, and reconfigure it away. This situation resembles
> delaying integration long enough to encounter "integration hell". Instead of
> smooth saling you get dominos of bugs.
>
> Is this a process smell?
Not repeatable. Not spinach.
Kevin Cline
>
> Suppose your project tests frantically, integrates continuously, and
> releases easily.
>
> Suppose then you hire a new coder, and need another workstation.
>
> Then you have a party. All the source requires 3rd-party libraries, of
> various versions, which must be matched, and these require configuration
> options and scripts installed here there and everywhere. The project's
> eldest guru must pair with the new guy, run the system over and over again,
> diagnose each glitch, and reconfigure it away. This situation resembles
> delaying integration long enough to encounter "integration hell". Instead of
> smooth saling you get dominos of bugs.
>
> Is this a process smell?
Yes. You need to automate this as much as possible, and then
painstakingly document anything that can't be automated. On Linux or
Unix that would be pretty trivial -- you put all the software on a
network disk, and tell the user to add a call to
setup_hairy_project.sh.
> Does it indicate someone should add a motherhood
> story indicating the codebase is continuously outegratable?
> Would an off-the-shelf configuration manager handle it?
On UNIX, yes, practically any configuration manager could handle 99%
of this problem, and you can take care of the remaining 1% by defining
some symbolic links. On Windows, AFAICT, it seems the answer is no,
since every file reference in windows is qualified by some drive
letter that will vary across installations.
> If so, wouldn't configuring
> that tool be a chore each time the team changes a library?
Again, on Unix it's no problem at all. You just change the symbolic
reference to point to the new version, or else just install the new
version in your configuration management system.
John Roth
"neurophysical-reeducator" <n0s...@yahoo.com> wrote in message
news:967085a1.0403...@posting.google.com...
You might, however, like to look at
for one way of handling the configuration issue.
John Roth