Ok, so I mind the toolchain for a project...

There are several components to this, mostly clumped together by there
speed of evolution.

There is the Operating System / Linux distro we develop on. (Windows
users think Windows version)

Well, we think of that as static / getting upgraded once a year or so...

But actually it is getting trickle upgraded continuously (security
fixes / bug fixes / ...)

There is the toolchain, cross compilers, linker, debugger, scripting
language / make tool, ...

There is the RTOS the product runs on (in this case ecos)

There are 3rd party libraries. (Typically held at static versions)
upgraded hardly ever.

There is the main body of code, continuously on the boil, but version
controlled.

The build system / makefiles which is checked in and versioned with the
code.

Now I need to allow the OS to be upgraded periodically.

I need to upgrade the cross compilers (rarely, but does happen)

The product has been released several times.

Now the requirements are....

* You must always be able to pull any version of the source code out
corresponding to a release, and rebuild it, and get exactly what you gave
to the customers so you can debug an obscure issue found only in the
field.

* You need to be able to pull any version of the source code and build
with the latest tool set.

* You need to build with the latest toolset, and the previous version of
the tools to compare. (Debug by difference)

* You need to be able to roll this out rapidly to a team of twenty, which
are not all working on the same release stream. Some of which are working
on more than one release stream simultaneously.

* Some of the tools you deliver are also delivered by the distro, by you
deliver custom ones for uniformity / repeatability and / or your custom
ones are cross targetted at different CPU / OS.

* You want exactly the same output for the same code given the same build
command on each developers box.

To give an example of a standard "make my day miserable" happening...

We rolled forward to new version of the distro. One of the standard OS
distro tools, "tail" has been made more POSIX compliant.

This breaks the RTOS build (but only if you first do a clean build, so it
didn't show up in initial testing).

The quickest workaround is to create a backwards compatible version of
"tail" and check it in with the source. Except none of the old versions
of the product will build then.

The next quickest workaround is to drop the hack version of "tail" onto
the path like /usr/local/bin. But then the build is relying on a kludge.
Something "special" about a build box. Mega Yuck. And what else in the
distro that uses the new version of tail will break?

The true fix would be to hack the RTOS to make it POSIX compliant, and
roll it out to 20 developers. Can do. Mucky. Specially as things were set
up so the RTOS code was read/only to unpriviledged users. So need to get
20 root passwords.

Anyway, that isn't the point of this post, I post not to whinge, but to
talk/think through patterns of toolchain config management. Any
suggestions or pointers to existing literature would be useful.

Pattern 1: never install in a system directory (eg /usr) or partition
(eg. / partition), or a new /different distro has the right to clobber
it. When you install an OS, partition disk into (at least) an "OS" root
partition, and a "home".

Install the toolchain in the "/home" partition.

Pattern 2 : Bundle your DLL's with the tools.

Modern build systems _insist_ on using DLL's. Bugger me, but it's an
uphill struggle spitting into the wind all the way to build a completely
statically linked tool chain these days.

A partial fix is to bundle all the DLL's you use with the tools you
deliver. (Question: How far do you go... into the realm of things that
live in /lib and /usr/lib? If not, you will have less reproducability
between boxes / version/... If you do, you may have mismatch between
kernel and libc...)

Pattern 3: Install in...

/opt/toolname/tool_version

and create a symbolic link
/opt/toolname/project_related_version to exact tool version. This version
relates to semantic changes in the tool set your project cares about, not
what the tool vendor cares about.

So you can roll forward and back and coexist.

Pattern 4: Explicitly code the path to /opt/toolname/
project_related_version version of the tool you use in paths within the
build system. This allows you to coexist new and old versions of the
code. And roll in no change in client require, only a bug fix changes to
the tool.

Pattern 5: Be lavish with your build systems output directory tree
structure. If a single flag changes on the compiler command line, the
output files should go into a different directory tree. Else you're going
to get the dependency/rebuild tracking in your build system wrong.

Pattern 6: Store your tool chain in a decent version control system,
treat each roll out to any developers desk exactly the same way as you'd
treat a release to a customer. Release branch and tag. The tool chain you
roll out is a separate product / module to the sources you use to build
the components of the toolchain. If you use the version control system to
deliver the toolchain onto the guys desk, remember you need to have
several versions of the tool available simultaneously. ie. Don't check
out mainline of toolchain, into /opt/toolname and then use VC to roll
forward and back. You must check out a tagged release of a tool into a /
opt/tool/version directory.

Pattern 7: Discard requirements. It's all too hard anyway. The
requirements are conflicting, the schedules too tight, the developers
won't cant't upgrade when you want them to... So tough, judiciously
discard requirements...