Lessons from Chromium development
Matthew Fernandez
presentation (at the bottom of http://www.chromium.org/Home) and was
pleasantly surprised to find that it has good general advice for
developing in a large team. If you're interested, I recommend flicking
through the presentation yourself, but if you just want the good stuff
here are my takeaways:
The obvious:
- Don't tie a project to any specific IDE or VCS; let developers use
whatever they're comfortable with (I've been barracking for version
control independence for a while at work).
- Make your builds thread safe (e.g. so make -j... doesn't cause chaos).
- Don't commit/push just before you're about to go offline.
- Have a very strict style guide (e.g.
http://www.chromium.org/developers/coding-style,
https://www.kernel.org/doc/Documentation/CodingStyle)
- If you're doing low-level development, use Valgrind (http://valgrind.org/).
- Make sure your regression/CI testing is bulletproof. A test should
only fail because of an actual code error.
- Use code review before checkin.
- Keep individual commits as small as possible.
The less obvious:
- Chromium seems to use a code review tool called Reitveld. I'd never
heard of it, but looks interesting.
- Looks like they have an agent that automatically tests each commit
(possibly continuous integration style) and automatically updates a
tag called LKGR to point at the last known good revision. This is
pretty cool and something that I would have found valuable in the past
on large scale projects.
- They use a staging/try server, which is something we've been
looking at recently. Basically the idea is that you have a
server-class machine that you can push an individual patch to. It
takes that patch, applies it to a fresh LKGR checkout and runs a full
test suite. Useful for validating your changes before pushing to the
mainline.
- They have performance and memory regression tests (i.e. flag a
commit that dramatically decreases performance or increases memory
usage).
- Not really a general point, but apparently Chromium needs >4GB RAM
during linking. WTF... I would say this is a lesson in what not to do.
Andrew Hill
The obvious:
- Don't tie a project to any specific IDE or VCS; let developers use
whatever they're comfortable with (I've been barracking for version
control independence for a while at work).
- Make your builds thread safe (e.g. so make -j... doesn't cause chaos).
- If you're doing low-level development, use Valgrind (http://valgrind.org/).
- Make sure your regression/CI testing is bulletproof. A test should
only fail because of an actual code error.
- Use code review before checkin.
- Keep individual commits as small as possible.
The less obvious:
- Chromium seems to use a code review tool called Reitveld. I'd never
heard of it, but looks interesting.
- Looks like they have an agent that automatically tests each commit
(possibly continuous integration style) and automatically updates a
tag called LKGR to point at the last known good revision. This is
pretty cool and something that I would have found valuable in the past
on large scale projects.
- They use a staging/try server, which is something we've been
looking at recently. Basically the idea is that you have a
server-class machine that you can push an individual patch to. It
takes that patch, applies it to a fresh LKGR checkout and runs a full
test suite. Useful for validating your changes before pushing to the
mainline.
- They have performance and memory regression tests (i.e. flag a
commit that dramatically decreases performance or increases memory
usage).
- Not really a general point, but apparently Chromium needs >4GB RAM
during linking. WTF... I would say this is a lesson in what not to do.
Matthew Fernandez
> On Fri, Nov 25, 2011 at 10:52 AM, Matthew Fernandez
> <matthew....@gmail.com> wrote:
>> - Make your builds thread safe (e.g. so make -j... doesn't cause chaos).
>
> FYI CMake handles these nicely (:
OK, but surely there are some things that are just unavoidable without
good development practice. E.g. random homebrew script noob-codegen.py
that always uses /tmp/noob-build.tmp as scratch space. I think often
you end up with build processes like this because people think that
it's a small project so it's fine to enforce a serial build. Fast
forward six months and devs become aggravated by serial builds taking
half an hour while they have seven idle cores. It's worth making
everything thread safe from the word go.
>> - Not really a general point, but apparently Chromium needs >4GB RAM
>> during linking. WTF... I would say this is a lesson in what not to do.
>
> Our (C++, heavily templated) graph-search library hits ~700 MB per make (-j)
> instance when compiling (on some compilers), but I don't think I've ever
> seen linking cause memory problems.
I would guess the templating is what's making the working set so
large. Do you see the same kind of behaviour for less complex
libraries? 700MB would be pretty rare for a C compile. E.g. make -j 8
of Linux on my desktop barely uses 200MB.
> Any ideas how/why this is happening during linking? What is the lesson we're
> meant to be learning?!
I think the main reason is that they build a monolithic executable; as
in, everything linked into one file. OS code will typically be more
intensive during linking than user-level code because you're often
defining complex section layouts. Why they create a single image
though is confusing as the system surely doesn't load its entire 1.3GB
image into memory on boot. I think the "lesson" is to keep your builds
lean if possible. Faster builds = more productive developers and in
the open source community requiring >4GB RAM (and a 64-bit OS) can be
a barrier to people getting involved.
Andrew Hill
On 25 November 2011 13:10, Andrew Hill <and...@thefrog.net> wrote:
> On Fri, Nov 25, 2011 at 10:52 AM, Matthew Fernandez
> <matthew....@gmail.com> wrote:
>> - Make your builds thread safe (e.g. so make -j... doesn't cause chaos).OK, but surely there are some things that are just unavoidable without
>
> FYI CMake handles these nicely (:
good development practice. E.g. random homebrew script noob-codegen.py
that always uses /tmp/noob-build.tmp as scratch space. I think often
you end up with build processes like this because people think that
it's a small project so it's fine to enforce a serial build. Fast
forward six months and devs become aggravated by serial builds taking
half an hour while they have seven idle cores. It's worth making
everything thread safe from the word go.
I would guess the templating is what's making the working set so
>> - Not really a general point, but apparently Chromium needs >4GB RAM
>> during linking. WTF... I would say this is a lesson in what not to do.
>
> Our (C++, heavily templated) graph-search library hits ~700 MB per make (-j)
> instance when compiling (on some compilers), but I don't think I've ever
> seen linking cause memory problems.
large. Do you see the same kind of behaviour for less complex
libraries? 700MB would be pretty rare for a C compile. E.g. make -j 8
of Linux on my desktop barely uses 200MB.
I think the main reason is that they build a monolithic executable; as
> Any ideas how/why this is happening during linking? What is the lesson we're
> meant to be learning?!
in, everything linked into one file. OS code will typically be more
intensive during linking than user-level code because you're often
defining complex section layouts. Why they create a single image
though is confusing as the system surely doesn't load its entire 1.3GB
image into memory on boot. I think the "lesson" is to keep your builds
lean if possible. Faster builds = more productive developers and in
the open source community requiring >4GB RAM (and a 64-bit OS) can be
a barrier to people getting involved.
Matthew Fernandez
Oh, wow. I thought you meant 700MB total, not each. Yes, OSX measures
memory in odd ways. I think it does sneaky things with virtual memory
like sharing identical read-only memory between processes. So the sum
of all your processes' memory usage may actually be more than your
total RAM.
When you say Chrome are you talking about the browser or parts of your
IDE? (note to Google/others: stop calling things "chrome." This term
is far too overloaded.)
If your compiler is actually paging doesn't everything get unusable?
Do you just go have lunch while compiling?
Very good point. Premature optimisation is the enemy of productivity.
With regards to minimal includes, I would have thought including too
many files wouldn't actually add too much overhead. Unless there are
static inline functions in your headers, surely your compiler is only
caching a reference to where the definition of the function is to be
found. On the other hand, if your compiler is doing some sort of
advanced cross-module inlining then maybe this does come into play.
As an aside, I think companies don't talk enough about their internal
procedures. Inevitably you have a couple of homebrew systems you use
that are really cool (read: other companies/individuals would get
value from them). They're often proprietary, which is fair enough, but
this shouldn't prevent the company showing off how they work or
writing blog posts about them. I regularly hear "hey, at my old work
we had this thing called X that did Y automatically. Why don't we
implement that here?" It's unfortunate that these things only seem to
get passed around by word of mouth and not shared with the general
public.