BK kernel workflow

6 views
Skip to first unread message

Jeff Garzik

unread,
Oct 19, 2004, 12:20:10 PM10/19/04
to
Although tangential to the problem, I thought LKML and BitMover (and
maybe Andrew or Linus as well) might be interested in a general
description of my workflow.


For net drivers in the Linux kernel, there exists two patch queues,
net-drivers-2.6 and netdev-2.6 (and corresponding 2.4 versions).
net-drivers-2.6 could be described as the "upstream immediately" or "for
Linus" queue, and netdev-2.6 could be described as the "testing" queue.

When receive a patch from some random person on the Internet, fixing a
net driver bug in the "8139too" driver, I do

1) create "topic-specific" repo, if it doesn't already exist

Key point: when dealing with a large number of incoming changes, like I
do, sorting the changes locally into logically separate _sets of
changes_ (one set per repo) has a lot of advantages, given that
BitKeeper doesn't allow changeset "cherrypicking".

cd /spare/repo/netdev-2.6 # not a repo, just a subdir
bk clone -ql ../linux-2.6 8139too

2) move the email containing the patch, in my MUA, to 25_Patches mbox

3) merge the patch into the topic-specific repo, using Linus's patch
importing tools,

cd 8139too && dotest < /garz/nsmail/25_Patches

4) pull the topic-specific repo into "collector" repo, and merge conflicts

cd ALL && bk pull ../8139too

5) build and test 'ALL' repo [heh... usually...]

6) push 'ALL' to external netdev-2.6 repos on gkernel.bkbits.net and
kernel.bkbits.net

7) Andrew's workflow includes automatically pulling netdev-2.6 repo into
his more-experimental "-mm" tree for wider review and testing.

[time passes]

8) Linus and Andrew release the latest and greatest 2.6.N stable release.

9) every day or so, I 'bk pull' a few "topic-specific" repos into a
local clone of net-drivers-2.6, test that, and send it off to Linus/Andrew.

Key point: thanks to the daily snapshots, my changes show up broken up
across several daily snapshots, rather than "one big huge lump of
changes that's been waiting to go in".

cd /spare/repo
bk clone -ql linux-2.6 net-drivers-2.6
bk pull ../netdev-2.6/8139too
bk pull ../netdev-2.6/viro-sparse-annotations
bk pull ../netdev-2.6/janitor
bk pull ../netdev-2.6/misc
bk push && bk push kernel.bkbits.net:net-drivers-2.6

# and then email Linus/Andrew the output of
# bk-make-sum + gcapatch (see Documentation/BK-usage)


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Jeff Garzik

unread,
Oct 19, 2004, 6:10:08 PM10/19/04
to
On Tue, Oct 19, 2004 at 11:33:40PM +0200, Paolo Ciarrocchi wrote:

> On Tue, 19 Oct 2004 12:06:49 -0400, Jeff Garzik <jga...@pobox.com> wrote:
> > Although tangential to the problem, I thought LKML and BitMover (and
> > maybe Andrew or Linus as well) might be interested in a general
> > description of my workflow.
> >
> > For net drivers in the Linux kernel, there exists two patch queues,
> > net-drivers-2.6 and netdev-2.6 (and corresponding 2.4 versions).
> > net-drivers-2.6 could be described as the "upstream immediately" or "for
> > Linus" queue, and netdev-2.6 could be described as the "testing" queue.
>
> So you have two bk trees,
> - patches good for mainstream
> - patches good for -mm tree

Close:
- patches ready for mainstream
- patches eventually ready for mainstream

and changes flow "up" from netdev-2.6 to net-drivers-2.6.


> It would be cool if all the maintainers could adopt your working method,
> Andrew is already automatically pulling from a bunch of trees, why not
> having Linusdoing the same too?

That's what Linus does already, when I email him :)

But Linus is essentially "senior editor", so we don't want to automate
the system, otherwise there is no editorial control. He is our
emporer penguin, after all.

Jeff

Paolo Ciarrocchi

unread,
Oct 19, 2004, 6:30:10 PM10/19/04
to
On Tue, 19 Oct 2004 17:38:03 -0400, Jeff Garzik <jga...@pobox.com> wrote:
> On Tue, Oct 19, 2004 at 11:33:40PM +0200, Paolo Ciarrocchi wrote:
> > On Tue, 19 Oct 2004 12:06:49 -0400, Jeff Garzik <jga...@pobox.com> wrote:
> > > Although tangential to the problem, I thought LKML and BitMover (and
> > > maybe Andrew or Linus as well) might be interested in a general
> > > description of my workflow.
> > >
> > > For net drivers in the Linux kernel, there exists two patch queues,
> > > net-drivers-2.6 and netdev-2.6 (and corresponding 2.4 versions).
> > > net-drivers-2.6 could be described as the "upstream immediately" or "for
> > > Linus" queue, and netdev-2.6 could be described as the "testing" queue.
> >
> > So you have two bk trees,
> > - patches good for mainstream
> > - patches good for -mm tree
>
> Close:
> - patches ready for mainstream
> - patches eventually ready for mainstream
>
> and changes flow "up" from netdev-2.6 to net-drivers-2.6.

Yup, I hope that almost all the patches move from "patches eventually
ready for mainstream" to "patches ready for mainstream" ;-)

>
> > It would be cool if all the maintainers could adopt your working method,
> > Andrew is already automatically pulling from a bunch of trees, why not
> > having Linusdoing the same too?
>
> That's what Linus does already, when I email him :)

Good ;) but AFAIK Andrew is not publishing the "patches ready for
mainstream" (bk or collection of patches) tree.


> But Linus is essentially "senior editor", so we don't want to automate
> the system, otherwise there is no editorial control. He is our
> emporer penguin, after all.

I understand,
I know I'm pedantic but can we all see the list of bk trees ("patches
ready for mainstream" and "patches eventually ready for mainstream")
that we'll be used by Linus ?

I'm learning a lot trying to understand the development method used by
this community,
but there are things that I don't fully understand,
is it possible to "formalize" it (somehow) ?

Ciao and thanks!

--
Paolo
Personal home page: www.ciarrocchi.tk

Linus Torvalds

unread,
Oct 19, 2004, 6:40:04 PM10/19/04
to

On Tue, 19 Oct 2004, Paolo Ciarrocchi wrote:
>
> I know I'm pedantic but can we all see the list of bk trees ("patches
> ready for mainstream" and "patches eventually ready for mainstream")
> that we'll be used by Linus ?

Even _I_ don't have that kind of list.

It's on a case-by-case basis (although with certain developers, the cases
tend to be pretty clear-cut), and it literally changes over time. Some
people use throw-away trees that are just used for some particular set,
and I merge them (or not), and they go away.

Linus

Greg KH

unread,
Oct 19, 2004, 8:00:30 PM10/19/04
to
On Tue, Oct 19, 2004 at 11:54:16PM +0200, Paolo Ciarrocchi wrote:
> I know I'm pedantic but can we all see the list of bk trees ("patches
> ready for mainstream" and "patches eventually ready for mainstream")
> that we'll be used by Linus ?

The -mm releases has these as a big patch, starting with bk-*

Those should give you an idea of what is being staged, before it is sent
to Linus.

thanks,

greg k-h

Paolo Ciarrocchi

unread,
Oct 20, 2004, 3:50:11 AM10/20/04
to
On Tue, 19 Oct 2004 15:11:55 -0700 (PDT), Linus Torvalds
<torv...@osdl.org> wrote:
>
>
> On Tue, 19 Oct 2004, Paolo Ciarrocchi wrote:
> >
> > I know I'm pedantic but can we all see the list of bk trees ("patches
> > ready for mainstream" and "patches eventually ready for mainstream")
> > that we'll be used by Linus ?
>
> Even _I_ don't have that kind of list.
>
> It's on a case-by-case basis (although with certain developers, the cases
> tend to be pretty clear-cut), and it literally changes over time. Some
> people use throw-away trees that are just used for some particular set,
> and I merge them (or not), and they go away.

So ATM is not possible to "formalize" the process (and probably even
not necessary)
Thank you.

--
Paolo
Personal home page: www.ciarrocchi.tk

maximilian attems

unread,
Oct 20, 2004, 5:00:12 AM10/20/04
to
> 3) merge the patch into the topic-specific repo, using Linus's patch
> importing tools,
>
> cd 8139too && dotest < /garz/nsmail/25_Patches

where are those tools available?

they are the missing chain in Andrew's patch-scripts.

--
maks
kernel janitor http://janitor.kernelnewbies.org/

Jeff Garzik

unread,
Oct 20, 2004, 5:20:08 AM10/20/04
to
maximilian attems wrote:
>>3) merge the patch into the topic-specific repo, using Linus's patch
>>importing tools,
>>
>> cd 8139too && dotest < /garz/nsmail/25_Patches
>
>
> where are those tools available?
>
> they are the missing chain in Andrew's patch-scripts.


http://bktools.bkbits.net/

Jeff

Larry McVoy

unread,
Oct 23, 2004, 12:30:14 PM10/23/04
to
On Tue, Oct 19, 2004 at 03:11:55PM -0700, Linus Torvalds wrote:
>
>
> On Tue, 19 Oct 2004, Paolo Ciarrocchi wrote:
> >
> > I know I'm pedantic but can we all see the list of bk trees ("patches
> > ready for mainstream" and "patches eventually ready for mainstream")
> > that we'll be used by Linus ?
>
> Even _I_ don't have that kind of list.

I don't know how you could have that sort of list. We have some idea
of the potential size of that list and it's huge. Based on the lease
requests back to us (BK is lease based, it connects back to us once a
month), we estimate that there well over 10,000 clones of the Linux kernel
in BK. If even 1/10th of those are going to have a patch for Linus that's
1,000 potential patches. Pretty hard to keep that all in your head.
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Paolo Ciarrocchi

unread,
Oct 24, 2004, 6:30:15 AM10/24/04
to
On Sat, 23 Oct 2004 09:12:53 -0700, Larry McVoy <l...@bitmover.com> wrote:
> On Tue, Oct 19, 2004 at 03:11:55PM -0700, Linus Torvalds wrote:
> > On Tue, 19 Oct 2004, Paolo Ciarrocchi wrote:
> > > I know I'm pedantic but can we all see the list of bk trees ("patches
> > > ready for mainstream" and "patches eventually ready for mainstream")
> > > that we'll be used by Linus ?
> >
> > Even _I_ don't have that kind of list.
>
> I don't know how you could have that sort of list. We have some idea
> of the potential size of that list and it's huge. Based on the lease
> requests back to us (BK is lease based, it connects back to us once a
> month), we estimate that there well over 10,000 clones of the Linux kernel
> in BK. If even 1/10th of those are going to have a patch for Linus that's
> 1,000 potential patches. Pretty hard to keep that all in your head.

Well, I'm not interested in having the list of all the bk trees used
during the develpoment of a release.
I was looking to the trees used by mantainers.
That number should me really different from "1,000".
Do you agree ?

CIao,


--
Paolo
Personal home page: www.ciarrocchi.tk

Picasa users groups: www.picasa-users.tk
join the blog group: http://groups-beta.google.com/group/blog-users

Larry McVoy

unread,
Oct 24, 2004, 10:50:07 AM10/24/04
to
On Sun, Oct 24, 2004 at 12:24:42PM +0200, Paolo Ciarrocchi wrote:
> On Sat, 23 Oct 2004 09:12:53 -0700, Larry McVoy <l...@bitmover.com> wrote:
> > On Tue, Oct 19, 2004 at 03:11:55PM -0700, Linus Torvalds wrote:
> > > On Tue, 19 Oct 2004, Paolo Ciarrocchi wrote:
> > > > I know I'm pedantic but can we all see the list of bk trees ("patches
> > > > ready for mainstream" and "patches eventually ready for mainstream")
> > > > that we'll be used by Linus ?
> > >
> > > Even _I_ don't have that kind of list.
> >
> > I don't know how you could have that sort of list. We have some idea
> > of the potential size of that list and it's huge. Based on the lease
> > requests back to us (BK is lease based, it connects back to us once a
> > month), we estimate that there well over 10,000 clones of the Linux kernel
> > in BK. If even 1/10th of those are going to have a patch for Linus that's
> > 1,000 potential patches. Pretty hard to keep that all in your head.
>
> Well, I'm not interested in having the list of all the bk trees used
> during the develpoment of a release.
> I was looking to the trees used by mantainers.

And how do you define a maintainer? That's a moving target. Part of the
beauty of the Linux development model is that mini forks are not only
allowed, they are encouraged. So people can go off on their own and do
something interesting. Who knows? One of those people could be the next
Alan.

> That number should me really different from "1,000".

Sure. But even so it's a moving target and labeling a set of people as
maintainers implies that other people aren't. I suspect Linus isn't
that interested in the labels, he'll take good code from anyone.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Paolo Ciarrocchi

unread,
Oct 24, 2004, 1:00:08 PM10/24/04
to
On Sun, 24 Oct 2004 07:44:48 -0700, Larry McVoy <l...@bitmover.com> wrote:
> On Sun, Oct 24, 2004 at 12:24:42PM +0200, Paolo Ciarrocchi wrote:
[...]

> >
> > Well, I'm not interested in having the list of all the bk trees used
> > during the develpoment of a release.
> > I was looking to the trees used by mantainers.
>
> And how do you define a maintainer? That's a moving target. Part of the
> beauty of the Linux development model is that mini forks are not only
> allowed, they are encouraged. So people can go off on their own and do
> something interesting. Who knows? One of those people could be the next
> Alan.

I totally agree on the "beauty of Linux development" part of your statement,
I don't 100% agree on your definition of maintainer.
There a few well defined maintainer in the community and those names
are defined and written in the kernel documentation.
So, I think that is moving target but we have a few stable and well
know references.

> > That number should me really different from "1,000".
>
> Sure. But even so it's a moving target and labeling a set of people as
> maintainers implies that other people aren't. I suspect Linus isn't
> that interested in the labels, he'll take good code from anyone.

Good point,
even though I'm almost sure that both Linus and Andrew prefer getting
patches after they are reviewed by those "maintainers".

Anyway,
I'm not a kernel hacker at all so my comments will sound just silly to
a lot of people involved in the development but as a user I'm
sometimes "alarmed" by the lack of formality in the development
process (see the naming wars),

Maybe we cannot avoid it, maybe that's the reason because linux is
"the best free OS" but maybe we can add a few more rules/indications
to the process.

Larry,
thank you for your answer, I always appreciate having such a kind of
open discussions with people that are really involved in the
development.

Ciao,
--
Paolo

Linus Torvalds

unread,
Oct 24, 2004, 1:50:10 PM10/24/04
to

On Sun, 24 Oct 2004, Paolo Ciarrocchi wrote:
>
> Well, I'm not interested in having the list of all the bk trees used
> during the develpoment of a release.
> I was looking to the trees used by mantainers.

Well, not only is "maintainer" fairly fluid, as Larry said, even if you
accept the fact that things will change, there's the issue that

- I do _not_ want people to "own" subsystems. And that's not so much an
anti-maintainer issue (I _love_ maintainers), as more of a "conceptual"
issue. When people expect one person or one group of person to be the
only way to touch a certain subsystem, we have problems. I really
really want everybody (users, developers _and_ maintainers) to realize
that no code is an island, and work with other people touching their
subsystem.

And part of that is that I do not like codifying maintainership. Even
something as simple as saying "this tree is the xxxx tree" is in my
opinion _bad_. Yes, a lot of core development for subsystems happen in
some specific "subsystem tree", but every time that has turned into
something "exclusive", it's been a _major_ problem.

And yes, we've had that problem several times. People having CVS trees
for networking, sound drivers, and other special development
subprojects invariably ended up breakign and throwing away work that
happened "unofficially". And often the "unofficial" work is nearly as
important as the official one.

So BK helps this model, because the distributed nature of BK means that
you can have several pseudo-official trees _and_ totally unofficial
ones, and merging is automatic and basically impossible to avoid, so
the "official" tree never gets to drown out the unofficial work. But
despite that, I want to make people _aware_ that maintainership does
not imply total ownership, and that we don't have a "hierarchy" of
developers but a *network* of developers.

To make a long story short: I do not ever even WANT "official" trees.
Because it gives the wrong idea to people.

I don't know if you've noticed, but I try to encourage other people to
make their own version of _my_ "official" tree, and unlike pretty much
all other open source projects I'm aware of, the Linux kernel
development model has always encouraged things like the "Alan Cox" tree
or "Andrew Morton" tree or "Andrea Arcangeli" tree. Or all the vendor
trees.

Many other projects try to control "the one true tree". Linux never
really did, and for the last several years it's been a conscious
decision for me to _encourage_ people to do their own trees. Exactly
because I don't want people to think that there are any really official
trees. My tree perhaps comes closest, but even I don't expect to be the
"final word on Linux".

This keeps us all honest.

Second, and less fundamentally:

- Even if we had "official maintainers" (and at any one time, certain
sub-areas certainly tend to have pretty strong maintainership), those
maintainers tend to have pretty fluid trees of their own, and they
change pretty dynamically.

Look at how trees are merged, and you'll notice that several
maintainers did a special "merge these things for 2.6.9" tree that
contained the stuff that they wanted to push out quickly, and that I
merged for just that release. It was basically a "throw-away" tree that
got used once.

This happens all the time, and again, I _like_ it. It means that people
can react a lot more dynamically to what is going on. Again, having a
documented "list of trees" would not make this kind of thing
technically impossible, but it would foster the wrong kind of "mental
landscape".

See what I'm saying? It all boils down to the fact that I really like
having a dynamic development model, and that I want to try to avoid
putting in mental road-blocks to that model. I want Linux development to
be fluid, and I think the best way to reach that goal is to make people
_think_ of it as being fluid.

It's the old "perception changes reality" thing. It's really true. How you
think about something quite heavily influences what you do.

Wow. That was deep. Time to go watch TV again.

Linus

Linus Torvalds

unread,
Oct 24, 2004, 2:00:11 PM10/24/04
to

On Sun, 24 Oct 2004, Linus Torvalds wrote:
>
> So BK helps this model, because the distributed nature of BK means that
> you can have several pseudo-official trees _and_ totally unofficial
> ones, and merging is automatic and basically impossible to avoid, so
> the "official" tree never gets to drown out the unofficial work. But
> despite that, I want to make people _aware_ that maintainership does
> not imply total ownership, and that we don't have a "hierarchy" of
> developers but a *network* of developers.

Btw, I've tried in the past to express why I think the BK model is so
good, and why CVS ans Subversion totally suck, and I think the previous
email perhaps explains it best.

It really is a very important conceptual thing, that "network" vs
"hierarchy" difference. And I know we all love bashing Larry, but give
the guy a pat on the back for really making that difference visible.

[ Larry removed from the cc, because he's got ego enough as it is ;]

Michael Buesch

unread,
Oct 24, 2004, 2:50:08 PM10/24/04
to
Quoting Linus Torvalds <torv...@osdl.org>:
>
> On Sun, 24 Oct 2004, Linus Torvalds wrote:
> >
> > So BK helps this model, because the distributed nature of BK means that
> > you can have several pseudo-official trees _and_ totally unofficial
> > ones, and merging is automatic and basically impossible to avoid, so
> > the "official" tree never gets to drown out the unofficial work. But
> > despite that, I want to make people _aware_ that maintainership does
> > not imply total ownership, and that we don't have a "hierarchy" of
> > developers but a *network* of developers.
>
> Btw, I've tried in the past to express why I think the BK model is so
> good, and why CVS ans Subversion totally suck, and I think the previous
> email perhaps explains it best.

What do kernel developers think about svk?
(Yes, it's not mature, yet.)
I mean the svk concept. Does it also suck for kernel development?

> It really is a very important conceptual thing, that "network" vs
> "hierarchy" difference. And I know we all love bashing Larry, but give
> the guy a pat on the back for really making that difference visible.
>
> [ Larry removed from the cc, because he's got ego enough as it is ;]

;)

> Linus

Michael.

Roman Zippel

unread,
Oct 24, 2004, 6:50:04 PM10/24/04
to
Hi,

On Sun, 24 Oct 2004, Linus Torvalds wrote:

> - I do _not_ want people to "own" subsystems. And that's not so much an
> anti-maintainer issue (I _love_ maintainers), as more of a "conceptual"
> issue. When people expect one person or one group of person to be the
> only way to touch a certain subsystem, we have problems. I really
> really want everybody (users, developers _and_ maintainers) to realize
> that no code is an island, and work with other people touching their
> subsystem.

OTOH people often just send patches directly to you without bothering to
contact the maintainer. There is no system that sends out notifications,
if a patch touches file x/y, so that one has a chance to comment on them.
To be very clear on this, I don't want to control what goes in, I'd just
like to know about changes _before_ they go in.

> So BK helps this model, because the distributed nature of BK means that
> you can have several pseudo-official trees _and_ totally unofficial
> ones, and merging is automatic and basically impossible to avoid, so
> the "official" tree never gets to drown out the unofficial work. But
> despite that, I want to make people _aware_ that maintainership does
> not imply total ownership, and that we don't have a "hierarchy" of
> developers but a *network* of developers.

One should also add that bk is not the answer to everything, e.g. bk
doesn't really help with maintaining separate patches. As one cannot
remove or change information from/in bk, that makes it difficult to
work on a series of patches, e.g. as Andrew does with -mm, a tool like
quilt is more helpful here.
This means patches should only be put into bk, when they are ready
(otherwise the repo accumulates useless info nobody is really interested
in), during their development bk is about as useful as any other SCM
system. bk simplifies mainly your work, but the further one is away from
the core of this developer network, the more bk looses its advantage
(and might only increase the amount of merge information bk has to
manage in relation to patch information).

bye, Roman

Linus Torvalds

unread,
Oct 24, 2004, 7:20:04 PM10/24/04
to

On Mon, 25 Oct 2004, Roman Zippel wrote:
>
> OTOH people often just send patches directly to you without bothering to
> contact the maintainer.

And sometimes that ends up being becaue the maintainer is unresposive.

It happens. I'd much rather make that be the _normal_ flow of events than
have people think that it's something horrible.

> There is no system that sends out notifications,
> if a patch touches file x/y, so that one has a chance to comment on them.

Well, these days there actually _is_. No, it's not per-file, but a
procmail filter on the patch bots will actually do a lot better than some
random "notify me on these files", because it can be personalized any
which way you want..

> One should also add that bk is not the answer to everything, e.g. bk
> doesn't really help with maintaining separate patches.

Yes. I think the -mm tree and Andrew's (and other peoples) patch-
maintenance ends up being another part of the workflow, "outside" the BK
trees, exactly because of that.

Linus

Larry McVoy

unread,
Oct 24, 2004, 7:40:08 PM10/24/04
to
On Sun, Oct 24, 2004 at 06:44:38PM +0200, Paolo Ciarrocchi wrote:
> I totally agree on the "beauty of Linux development" part of your statement,
> I don't 100% agree on your definition of maintainer.

I explicitly didn't define maintainer. That's the beauty of Linux
development. I've been around for more than a decade in the Linux kernel
and people come and people go. People show up and prove themselves and
become more important because they _are_ more important. Not because
they have the maintainer title.

Linus sets the tone on this one, he has always played it fairly harshly
in that he respects the work you did today. What you did (or did not do)
yesterday isn't relevant. One way to sum it up is "There is no tenure
in Linux kernel development". You are useful as long as you are useful
and then you get pushed aside. I don't always agree with this model,
I think it has chased away some useful people who felt like "Linus owed
them" but on the other hand it is extremely open to new useful people.

> There a few well defined maintainer in the community and those names
> are defined and written in the kernel documentation.
> So, I think that is moving target but we have a few stable and well
> know references.

Yeah but focussing on them is the wrong answer. Who knew who Andrew
Morton was 5 years ago? He's clearly one of the most important people
around today but tomorrow someone else may emerge. No offense at all
intended towards Andrew, he's clearly kicking butt, I watch his trees,
the amount of work he does is astounding, if you aren't watching you
should be. But the cool part of the process is that Andrew came out of
left field and became who he is.

I've worked at places where it was all about who you knew, that sort of
clubby old boy feeling, and new people had a very tough time breaking
into the system. The Linus model is crushingly hard on people who
have paid their dues but have slowed down, that's the "no tenure" part,
but the good part is that you don't have to be anybody to be important.
You just need to be good. All the rest is just talk.

> I'm not a kernel hacker at all so my comments will sound just silly to
> a lot of people involved in the development but as a user I'm
> sometimes "alarmed" by the lack of formality in the development
> process (see the naming wars),

I was interviewed by a guy at Business Week a few days ago about this
process and I tried to get the point across that there actually is
a process but it's more of a zen thing than something that managers
would like.

> Larry,
> thank you for your answer, I always appreciate having such a kind of
> open discussions with people that are really involved in the
> development.

Sadly, I'm not at all involved in the development other than as someone
who is trying to help indirectly. BK is clearly helping, I think even
the BK license haters have finally admitted that, but I'm pretty much
out of the process. I'm just like you, sitting on the sidelines admiring
what is going on. As Linus said in some mail to me a while back, I'm like
the guy who provides the lumber for the house raising. No matter how much
I'd like to be a part of the house raising, I'm not, I'm just a vendor
providing some needed parts.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Matthias Urlichs

unread,
Oct 25, 2004, 7:40:08 AM10/25/04
to
Hi, Larry McVoy wrote:

> I was interviewed by a guy at Business Week a few days ago about this
> process and I tried to get the point across that there actually is a
> process but it's more of a zen thing than something that managers would
> like.

So, did you point him at Linus' Documentation/ManagementStyle paper?

--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | sm...@smurf.noris.de

Andrea Arcangeli

unread,
Oct 25, 2004, 7:50:11 AM10/25/04
to
On Sun, Oct 24, 2004 at 04:32:14PM -0700, Larry McVoy wrote:
> the BK license haters have finally admitted that, [..]

dream on

Joe Perches

unread,
Oct 25, 2004, 8:40:13 AM10/25/04
to
On Mon, 2004-10-25 at 13:46 +0200, Andrea Arcangeli wrote:
> On Sun, Oct 24, 2004 at 04:32:14PM -0700, Larry McVoy wrote:

More completely, Larry said:
"BK is clearly helping, I think that even"

> > the BK license haters have finally admitted that, [..]

> dream on

Andrea, I'm confused by your response.

Are you suggesting one or more of:

1. BK has improved LK workflow
2. BK has not improved LK workflow
3. BK has had no impact on LK workflow
4. you hate the license

Any of these or something else?

If your answer does not include 1, why?

I've read http://kerneltrap.org/node/view/3148

>From my view, LK workflow post patch penguin (ie: BK)
is improved simply because fewer patches seem to be lost.

I too do not use BK because of the license restrictions
on developing any other SCM systems. I think that portion
of the license is just wrong, but BK is BitMover's product
and I believe Larry can license it as he chooses.

Matthias Urlichs

unread,
Oct 25, 2004, 9:10:10 AM10/25/04
to
Hi, Greg KH wrote:

> On Tue, Oct 19, 2004 at 11:54:16PM +0200, Paolo Ciarrocchi wrote:
>> I know I'm pedantic but can we all see the list of bk trees ("patches
>> ready for mainstream" and "patches eventually ready for mainstream")
>> that we'll be used by Linus ?
>
> The -mm releases has these as a big patch, starting with bk-*

Umm, yeah, but it's *one* big patch (or, if you use my -mm import tree
from bk://smurf.bkbits.net/linux-2.6.X-rcY-mmZ, one BK pull which has
that as a subtree).

Paolo wants to see two distinct ones.

Andrew also does things like

bk-netdev.patch
e1000-module_param-fix.patch
ne2k-pci-pci-build-fix.patch
r8169-module_param-fix.patch

which my mind translates as "there's something stupid, incomplete or
outdated in the bk-netdev tree", or "that tree's maintainer should apply
these patches. Now." (Ideally, of course, my import script should do the
same thing.)

--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | sm...@smurf.noris.de

-

Andrea Arcangeli

unread,
Oct 25, 2004, 9:50:12 AM10/25/04
to
On Mon, Oct 25, 2004 at 05:29:02AM -0700, Joe Perches wrote:
> 1. BK has improved LK workflow
[..]

> If your answer does not include 1, why?

as a matter of fact nobody can know how the workflow would be if _all_
kernel developers would have been able to take advantage of a
distributed revision control system in the last ~3 years. The simple
fact I try to avoid discussing this topic on public lists to avoid be
targeted as whiner as usual, doesn't mean Larry can speak for myself
saying the few people like me now changed their mind and they agrees BK
generally helps the workflow.

> From my view, LK workflow post patch penguin (ie: BK)
> is improved simply because fewer patches seem to be lost.

IMHO that's largerly thanks to Andrew, and he's not using BK but quilt
to manage -mm, and as far as I can see his merges with Linus are using
patches. Infact I'm only sending my seldom patches to Andrew.

Paolo Ciarrocchi

unread,
Oct 25, 2004, 11:10:09 AM10/25/04
to
On Mon, 25 Oct 2004 15:01:50 +0200, Matthias Urlichs
<sm...@smurf.noris.de> wrote:
[...]

> Paolo wants to see two distinct ones.

Yes,
Paolo thinks that is a good idea.

But Linus already answered to my concerns ;)


--
Paolo
Personal home page: www.ciarrocchi.tk

Linus Torvalds

unread,
Oct 25, 2004, 12:40:14 PM10/25/04
to

On Mon, 25 Oct 2004, Andrea Arcangeli wrote:
>
> arch exists and it's exactly as distributed as BK.

And I looked at it before starting BK. Trust me, it was nowhere _near_
usable, which was my point. Nothing you have described has existed for
three years. Except for BK.

I doubt arch is there today either, but hey, if it displaces CVS, I
certainyl won't complain. How are the gcc people doing with it?

Linus

Larry McVoy

unread,
Oct 25, 2004, 12:40:07 PM10/25/04
to
On Mon, Oct 25, 2004 at 03:39:51PM +0200, Andrea Arcangeli wrote:
> On Mon, Oct 25, 2004 at 05:29:02AM -0700, Joe Perches wrote:
> > 1. BK has improved LK workflow
> [..]
> > If your answer does not include 1, why?
>
> as a matter of fact nobody can know how the workflow would be if _all_
> kernel developers would have been able to take advantage of a
> distributed revision control system in the last ~3 years. The simple
> fact I try to avoid discussing this topic on public lists to avoid be
> targeted as whiner as usual, doesn't mean Larry can speak for myself
> saying the few people like me now changed their mind and they agrees BK
> generally helps the workflow.

That's strange, I wonder why you think BK doesn't help. The prevailing
wisdom is that it has helped. It's well documented by third parties
who have nothing to do with you or me.

Is it possible that you have an axe to grind about BK and that any
positive statement about BK will be challenged by you no matter how
silly that makes you appear?

Maybe you know about some different system that would have worked better.
Perhaps you could share with the rest of us what that system is and how
it works better.

> > From my view, LK workflow post patch penguin (ie: BK)
> > is improved simply because fewer patches seem to be lost.
>
> IMHO that's largerly thanks to Andrew, and he's not using BK but quilt
> to manage -mm, and as far as I can see his merges with Linus are using
> patches. Infact I'm only sending my seldom patches to Andrew.

The implication that Andrew doesn't use BK is incorrect, we see him in
the logs all the time. You're right that he uses other tools to manage
his collection of patches and we agree with his reasons for doing so.
But he also uses BK and has pointed out himself the fact that BK has
helped the kernel development effort.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Jeff Garzik

unread,
Oct 25, 2004, 12:50:07 PM10/25/04
to
Matthias Urlichs wrote:
> Andrew also does things like
>
> bk-netdev.patch
> e1000-module_param-fix.patch
> ne2k-pci-pci-build-fix.patch
> r8169-module_param-fix.patch
>
> which my mind translates as "there's something stupid, incomplete or
> outdated in the bk-netdev tree", or "that tree's maintainer should apply
> these patches. Now." (Ideally, of course, my import script should do the
> same thing.)

Wrong on all counts.

The right answer is, "bk-netdev conflicts with some other BK tree that's
also not yet in upstream"

Jeff

Matthias Urlichs

unread,
Oct 25, 2004, 1:00:09 PM10/25/04
to
Hi,

Jeff Garzik:
> Wrong on all counts.
>
I stand corrected.

> The right answer is, "bk-netdev conflicts with some other BK tree that's
> also not yet in upstream"
>

That sounds somewhat painful.

Personally I'd do the BK tree integration in a somewhat more
Bitkeeper-ish way, but that's (obviously) Andrew's call.

signature.asc

Andrea Arcangeli

unread,
Oct 25, 2004, 1:10:05 PM10/25/04
to
On Mon, Oct 25, 2004 at 09:20:22AM -0700, Larry McVoy wrote:
> The implication that Andrew doesn't use BK is incorrect, we see him in
> the logs all the time. [..]

I assume this is great news for bitmover: one more tainted open source
developer that will not be allowed by law to ever compete with your
business, right? Anyways this is offtopic for l-k, but you can still
celebrate elsewhere.

Larry McVoy

unread,
Oct 25, 2004, 1:30:11 PM10/25/04
to
On Mon, Oct 25, 2004 at 06:47:32PM +0200, Andrea Arcangeli wrote:
> On Mon, Oct 25, 2004 at 09:20:22AM -0700, Larry McVoy wrote:
> > The implication that Andrew doesn't use BK is incorrect, we see him in
> > the logs all the time. [..]
>
> I assume this is great news for bitmover: one more tainted open source
> developer that will not be allowed by law to ever compete with your
> business, right? Anyways this is offtopic for l-k, but you can still
> celebrate elsewhere.

You have made this point hundreds of times, everyone knows your position
that we're screwing over the open source world by preventing them from
copying our work. OK, I can see that you think that, we respectfully
disagree because we feel we have to protect our IP. Rightly or wrongly,
we feel we did some new work that is worth protecting.

The big flaw in your position is that you are not tainted, you are a good
programmer, you claim to care deeply about this issue, so you could go
solve the problem. Either by fixing arch or creating your own system.
You should be the poster child for why the BitKeeper license is a
flawed idea.

We both think we care about helping the kernel. You think that we're
screwing over a pile of people who could write a GPLed SCM system.
We think that that is not true and you yourself prove our point. Kernel
guys like working on the kernel. If they liked working on SCM they would
be doing that, there would be a GPLed answer that was better than BK or at
least good enough, and we wouldn't be arguing, we'd probably be friends.

So who cares more about the kernel development process? The guy who took
crap from you and your friends for years but cared enough to keep going
because what counted were _results_, or the guy who sits around and whines
that we are polluting the well which was already dry?


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Andrea Arcangeli

unread,
Oct 25, 2004, 2:00:16 PM10/25/04
to
On Mon, Oct 25, 2004 at 10:18:27AM -0700, Linus Torvalds wrote:
>
>
> On Mon, 25 Oct 2004, Andrea Arcangeli wrote:
> >
> > I assume this is great news for bitmover: one more tainted open source
> > developer [...]
>
> Andrea, shut up.
>
> It's not _your_ decision to make, or your decision to complain about. It's
> the developers decision. It was mine, Jeff's, David's, Andrew's... Not
> yours.
>
> It's your decision is to not use BK. Fine. But then complaining when
> people decide to use the best tool available is fricking impolite. Not
> just to Larry, but to the people who made the choice.
>
> You whine about BK taking rights away, but the fact is, BK is an _option_
> for people to use. _YOU_ are the one trying to limit what people are
> supposed to do.
>
> In short, BK isn't the problem. You are.

you know, I cannot care less if I'm the problem. And I won't shut up.

Andrea Arcangeli

unread,
Oct 25, 2004, 2:00:20 PM10/25/04
to
On Mon, Oct 25, 2004 at 10:12:23AM -0700, Larry McVoy wrote:
> We think that that is not true and you yourself prove our point. Kernel
> guys like working on the kernel. [..]

This is sure fine with Linus and Andrew (peraphs even with myself), but
not everybody is Andrew and Linus. The reason I don't use it myself is
to avoid tainting _other_ people that might be beginners that might one
day write their own GPL'd SCM.

Andrea Arcangeli

unread,
Oct 25, 2004, 2:10:13 PM10/25/04
to
On Mon, Oct 25, 2004 at 09:10:47AM -0700, Linus Torvalds wrote:
> I doubt arch is there today either, but hey, if it displaces CVS, I
> certainyl won't complain. How are the gcc people doing with it?

gcc people are stuck with CVS AFIK. Apparently CVS is good enough for
them.

arch isn't ready for prime time with the kernel. It would be ready if we
were ok to limit it to say 5000 changesets and to obsolete the older
changesets once in a while. the backend needs a rewrite to handle that.

Thanks to various improvements we did (I only did one that allows
caching with hardlinked trees, Chris and others did more), probably arch
would be already way faster than BK in a daily checkout checkin and
cloning (nobody on the open source side can verify since we cannot use
BK, AFIK Miles tried to buy a copy of BK but Larry refused to sell it,
but I seriously doubt BK has such an advanced hardlinking cache
mechanism like arch), but the very first setup on a new machine would be
very inefficient (if compared to CVS) and the local copy of the
repository would take more space (again if compared to CVS).

The user interface isn't nice either, it'd be nicer at least to avoid
overlaps between commands.

I believe this all can be fixed, it just needs a critical mass of users
and some big initial pain.

Jon Smirl

unread,
Oct 25, 2004, 2:30:19 PM10/25/04
to
On Mon, 25 Oct 2004 09:20:22 -0700, Larry McVoy <l...@bitmover.com> wrote:
> That's strange, I wonder why you think BK doesn't help. The prevailing
> wisdom is that it has helped. It's well documented by third parties
> who have nothing to do with you or me.

From what I see BitKeeper has definitely helped the kernel development
processes. On the other hand BitKeeper has been stable for the last
couple of years. Are we going to see any large changes in BK any time
soon? For example BK could be extended to handle the workflow AndrewM
does. Another extension would be for moving signed patches through the
system to help avoid another SCO problem. Any hints on where the
future is going? Can BK be extended to further automate the kernel
development workflow?

--
Jon Smirl
jons...@gmail.com

Matthias Urlichs

unread,
Oct 25, 2004, 4:10:09 PM10/25/04
to
Hi, Andrea Arcangeli wrote:

> one more tainted open source
> developer that will not be allowed by law to ever compete with your
> business, right?

Wrong. Since when does usage constitute "tainted" knowledge?
You get tainted knowledge by looking at source code, not by usage.

By the same token, users of MS Office, which is even more restrictively
licensed than BK (no free use whatsoever, remember?), couldn't work on
OpenOffice.org.


The license says that, if you work on a competing system, you cannot use
BK. It cannot prevent somebody who has used BK sometime in the past, from
writing an SCM sometime in the future, since the license governs only your
use of BK, but not whatever else you're doing. It's a license for
BitKeeper and not for anybody's brain, after all.

<parrot whom="Linus">Andrea, shut up.</parrot>

--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | sm...@smurf.noris.de

-

Jeff Garzik

unread,
Oct 25, 2004, 4:30:19 PM10/25/04
to
Andrea Arcangeli wrote:
> On Mon, Oct 25, 2004 at 08:14:25AM -0700, Linus Torvalds wrote:
>
>>Your point is pointless. No such distributed revision control system
>>exists. And without BK, the people who have worked on them wouldn't
>
>
> arch exists and it's exactly as distributed as BK.


It doesn't scale or merge as well as BK though.

I've told Larry that, if both BK and <open source tool> were completely
equal in terms of function, I'd use the open source tool. Neither arch
(scalability) nor subversion (scalability + stability) are there yet.

Jeff

Andrew Morton

unread,
Oct 25, 2004, 6:10:13 PM10/25/04
to
Jeff Garzik <jga...@pobox.com> wrote:
>
> Matthias Urlichs wrote:
> > Andrew also does things like
> >
> > bk-netdev.patch
> > e1000-module_param-fix.patch
> > ne2k-pci-pci-build-fix.patch
> > r8169-module_param-fix.patch
> >
> > which my mind translates as "there's something stupid, incomplete or
> > outdated in the bk-netdev tree", or "that tree's maintainer should apply
> > these patches. Now." (Ideally, of course, my import script should do the
> > same thing.)
>
> Wrong on all counts.

"wrong" became "right". Those patches need to be applied now. I'll resend them.

But Matthias has described the algorithm correctly: I try to keep "fixups"
as close as possible to the patches which they fix.

I also try to keep patches in when-to-go-to-linus order, but that's much
harder to achieve, for various reasons.

Larry McVoy

unread,
Oct 25, 2004, 7:40:07 PM10/25/04
to
On Mon, Oct 25, 2004 at 02:18:43PM -0400, Jon Smirl wrote:
> On Mon, 25 Oct 2004 09:20:22 -0700, Larry McVoy <l...@bitmover.com> wrote:
> > That's strange, I wonder why you think BK doesn't help. The prevailing
> > wisdom is that it has helped. It's well documented by third parties
> > who have nothing to do with you or me.
>
> >From what I see BitKeeper has definitely helped the kernel development
> processes. On the other hand BitKeeper has been stable for the last
> couple of years. Are we going to see any large changes in BK any time
> soon? For example BK could be extended to handle the workflow AndrewM
> does. Another extension would be for moving signed patches through the
> system to help avoid another SCO problem. Any hints on where the
> future is going? Can BK be extended to further automate the kernel
> development workflow?

While BK has been stable that doesn't mean it is even remotely close to
being done. We've got more people working on BK now than ever before.
I think you might be used to scm systems sucking so you look at BK and
say it's better therefor it's done. We don't agree.

Things we are working on include performance (Wayne has a hot cache
linux-2.5 tree consistency check down to around 2 seconds, that's
about a 10x improvement over what it is now), we're revamping the GUIs
to be useable by normal humans, we're working on scaling to >500,000
changesets in one tree, we're working on scaling the number of files and
source to millions of files and GB of source (we just had to recompile
with largefile support for a customer this weekend, while it worked,
we thought the performance on a 2GB file was pathetic, it is pathetic,
it should run at the platter speed but as I dug into I found it out
wasn't that easy), we're working bug database integration, we're working
on review queues, we're working on project management, we're working on
large binary management, etc.

The list goes on, if anyone wants to come work on it we are always looking
for good systems people and we pay well. Try userland, you might like it :)

As for digital signatures, we quietly added that a long time ago,
every push by Linus or Marcelo to bkbits.net is digitally signed twice,
once over all the data including BK metadata and once over the checked
out files. The reason we do both is so people not using BK can verify
the signatures. If you want to be on the mailing list which gets these
signatures send me mail, if I get swamped I'll push it over to mailman.
Right now they go to Linus, Ted T'so, AndrewM, and Marcelo. They should
go more places so we have more people who can archive them in case
something goes wrong.

As for handling AndrewM's workflow, we're very interested in that
area because there seems to be a sort of bimodal development model,
changes which are not yet "frozen" (best managed by something like
quilt it seems) and changes which are frozen (best managed by BK).
We do well at part of it and suck at the other part. I've always
excepted arch to come into use for the unfrozen part but for some
reason people seem to like quilt better. I haven't played with
quilt enough to know why yet.

The unfrozen management works for Adrew because he is doing all the work,
the tools help as much as they can but he is really very much in control.
That model doesn't scale to a distributed/replicated development model, at
least it doesn't scale as well as BK scales, it scales as well as diff &
patch & wiggle scale. For Andrew, it's more or less fine, but if Andrew
tried to scale that to a few hundred developers all doing additional
work on the moving target of patches he'd go nuts. As it is I can't
imagine how he doesn't go nuts.

We'd like to handle it better and I've had conversations with various
people over the years and so far no lightbulb has come on. If there is
one, we'll find it, some of the problems we solved in BK felt sort of
similar and had to cook for around 3 years before the lightbulb came on.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Linus Torvalds

unread,
Oct 25, 2004, 9:00:12 PM10/25/04
to

On Tue, 26 Oct 2004, Roman Zippel wrote:


>
> On Mon, 25 Oct 2004, Linus Torvalds wrote:
>
> > In short, BK isn't the problem. You are.
>

> I think you're making it yourself way too easy.
>
> Blaming Andrea for this mess is of course easy, but it doesn't solve
> anything and misses the point

[ Answering these in a different order ]

The only thing I blame Andrea for is _whining_. Don't get me wrong, I
don't blame him for the bad state of CVS or anything like that.

The fact is, Andrea doesn't actually do anything constructive when it
comes to SCM. He just complains every time somebody says something
positive about a product that (a) he didn't do anything for and (b) nobody
forces him to use, and (c) there are no real alternatives for today (much
less the three years ago he was whining about).

> The ability to handle big amounts of patches includes also the
> possibility to merge a lot of crap. What keeps up the general quality?

No SCM is _ever_ going to be a quality manager.

And I also claim that people who think that "processes" are quality
management (see iso9000, or Dilbert) are seriously mistaken too.

The thing that keeps up the general quality is _people_. Good people, who
take pride in the quality. They end up being maintainers, not because they
chose the job, but because people ended up chosing them, for better of for
worse.

And the way to help those people is to make the day-to-day job easy, so
that they can spend as much time as possible on the thing that matters:
upholding good taste (and in the process keeping quality up). And that's
where an SCM comes in - not as a primary source of quality, but as a way
to keep track of the details, so that people can concentrate on what is
important.

And the SCM doesn't have to be anything really fancy. It can be a few
scripts to keep track of patches (that tends to grow and become slightly
more sophisticated over time). I'm not saying that BK is "it". There's a
number of BK users there, but clearly there are other ways to maintain
patches too - and people use them.

But complaining when a maintainer uses a tool that suits him is _stupid_.
It's arrogant to think that you can tell me how to do my work, but it's
really stupid when you can't give any reasonable alternatives that would
help me do it as efficiently.

And that's what Andrea is doing. Sure, BK is commercial, but dammit, so is
that 2GHz dual-G5 too and that Shuttle box in my corner. They happen to be
the tools I use for what I do. If Andrea told me that I should use a
slower machine because that's what most people use, I'd consider him a
total idiot. Similarly, when he complains that people use software tools
that clearly _do_ make them more productive, I consider his complaint
stupid.

There are other tools I use to make myself more productive. Many of them
are open source. Some I wrote myself. But I still use "uemacs" and "pine"
as part of my tool-chest, for example - and last I saw, they weren't open
source either (but I hear that the uemacs author stopped caring, so that
one might have been re-licensed).

Should I (or anybody else) ask Andrea's permission before we start using
non-opensource tools? No. If Andrea were complaining about my "pine"
usage, he'd be laughed off the planet. It may be ass-backwards and old and
text-only, but the fact is, it's really none of his damn business, even
though he can see the effects in every email I write in the headers.

Similarly, Andrea can see some of the effects of me using BK when he looks
at the tar-balls and patches - syntactic markers that show that they have
been generated by a person who uses BK. It's really _no_ different from
the fact that I use pine to communicate. And no, neither BK nor pine are
under an open source license. Deal with it.

Can Andrea point me to open-source tools and ask me politely whether I've
considered them as alternatives? Hell yes. I encourage him to do so when
something appears.

Linus

Andrea Arcangeli

unread,
Oct 25, 2004, 9:10:05 PM10/25/04
to
On Mon, Oct 25, 2004 at 09:51:31PM +0200, Matthias Urlichs wrote:
> Wrong. Since when does usage constitute "tainted" knowledge?
> You get tainted knowledge by looking at source code, not by usage.

come on, as someone stated in other thread the licence doesn't even
appear legal in europe, so what do you expect?

> By the same token, users of MS Office, which is even more restrictively
> licensed than BK (no free use whatsoever, remember?), couldn't work on
> OpenOffice.org.

Note that if I would be buying a BK non-free licence the "restrictions"
would go away AFIK. However Miles (on the arch list) tries to buy one
(exactly so he could test BK) and he failed to get one.

"no free use" may be less restrictive than the "free licence". You pay
to get more freedom, which sounds fair.

> The license says that, if you work on a competing system, you cannot use
> BK. It cannot prevent somebody who has used BK sometime in the past, from
> writing an SCM sometime in the future, since the license governs only your
> use of BK, but not whatever else you're doing. It's a license for
> BitKeeper and not for anybody's brain, after all.

either you're completely wrong or Larry did not protect any IP.

Larry said in an earlier email today: "we have to protect our IP.


Rightly or wrongly, we feel we did some new work that is worth

protecting.".

don't get me wrong, I've nothing against protecting IP, but my point is
how can he protect IP if I can use BK, learn all the features, then the
next day I stop using them and contribute to another SCM after the
knowledge learned? I mean, there must be a delay between the two events
otherwise what a protection is that. I can use BK the next second I stop
using BK and I make a patch to arch, the next second I use BK again.

I would be *very* happy if you were right in the way you interpret the
licence, if Larry could confirm you didn't misread the licence, I could
sure use bitkeeper too (at least to try it once, you know I may be
curious about testing bk speed after all these talks).

But I'm not attempting that unless Larry confirms I can learn using bk
and immediatly after I can contribute to arch or some other SCM to
possibly implement overlapping features (and so far I clearly understood
this was not possible, and this is also why it didn't look legal in
europe but IANAL).

Larry McVoy

unread,
Oct 25, 2004, 9:20:04 PM10/25/04
to
On Tue, Oct 26, 2004 at 02:06:54AM +0200, Roman Zippel wrote:
> bk makes it in first
> place easier to apply and merge a lot of patches, but patches still have
> to be written, reviewed and maintained. The ability to handle big amounts

> of patches includes also the possibility to merge a lot of crap. What
> keeps up the general quality?

http://www.bitkeeper.com/press/newsforgearticle.html - see part 2, the
long answer on development models. It goes into detail about how it
can work.

> There is a certain license problem, which very effectly keeps out a lot of
> those people, who might have other ideas for managing the kernel sources
> and improving the development process.

Well, you don't use BK and Andrea doesn't use BK and the arch developers
don't use BK and you have all had at least 3 years to do something better.

> The "protecting IP" talk is
> complete bullshit, one doesn't need direct access to figure out how it
> works. There are a few free SCM systems out there, that use very similiar
> mechanism, they of course still need to work out the details, but it's no
> magic and just takes times.

There is a lot more magic in there than you believe and the fact that
nobody has replicated BK in the last 5 years gives us some reason to
believe that the license is actually working.

While there is some fairly profound stuff buried inside of BK, it's the
nasty little corner cases which are the hard work. If I had to divide
up the work I think that the corner cases are more than 80% of the work.
You are mistakenly assuming that the way BK stores the data, or does
merges, or synchronizes is what we think is worth protecting, and you
are pretty much wrong. Yes, that stuff is interesting to an engineer
because it is something you can describe, it's like math. But the math
is easy compared to the polishing.

The hard part is all the knowledge about all the little things which go
wrong and how to fix them. It's relatively easy to make a system which
looks like BK, there are several out there. They all fall over as soon
as you try and stress them in any way. The fact that BK doesn't is what
is hard and yes, it *is* worth protecting, we worked very hard to make
BK work for you and we think it is perfectly reasonable that you either
agree to not rip us off or you don't get to use BK.

Our license is not one iota less reasonable than the GPL. What about all
those people who want to use your work and not give back? How about that
guy who wants a BSD licensed Linux, for the obvious reasons? You were all
outraged that that guy would want to rip off your work, why is it so darned
hard for you to see that we are equally outraged when you want to use our
work for free and rip it off? If you really were about just helping the
world it wouldn't bother you in the slightest to offer up a BSD licensed
Linux kernel. Admit it, you want to keep on playing in your playground.
That's fine, we want the same thing. We are both using licensing to
accomplish our goals.

> Being able to import the kernel repository
> into them would be a great way to test them, but unfortunately all the
> free testing is reserved for bk, as usual the winner takes it all and the
> losers eat the dust, isn't that how open source works...?

Huh? We make the CVS exported tree available nightly and have been doing
so for more than a year and in spite of all the dire predictions that it
wouldn't last it's still there.

There are 23,000 commits in the BK2CVS 2.5/2.6 tree. All on kernel.org
for your importing pleasure complete with all the checkin comments and
other history.

You can import to your hearts content and people have, only to find that
their system falls over dead when they do.

> Blaming Andrea for this mess is of course easy, but it doesn't solve

> anything and misses the point, the only thing Andrea is to blame for is
> that he is not very diplomatic, but that's unfortunately a trait seldom
> found under hackers. Ignoring the problems and silencing the critics
> doesn't solve anything and I would be more concerned if nobody would
> object anymore, when Larry is on one of his ego trips.

Nobody is "blaming Andrea for this mess", all that was being asked was
that he treat other people how he would like to be treated. What Andrea
and you are doing is similar to that guy asking for a BSD licensed Linux
kernel and continuing to do so at every opportunity. Wouldn't that get
a bit old?

As for my ego, my friend, don't flatter yourself that I would come to
this forum to inflate my ego in the unlikely event I felt it needed a
tuneup. You'd have to be insane to think that hanging out here and
getting crapped on for 5 years is an ego inflating process. Sheesh.
Who are you kidding?


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Chris Wedgwood

unread,
Oct 25, 2004, 10:10:04 PM10/25/04
to
On Mon, Oct 25, 2004 at 04:01:28PM -0700, Larry McVoy wrote:

> Things we are working on include performance (Wayne has a hot cache
> linux-2.5 tree consistency check down to around 2 seconds, that's
> about a 10x improvement over what it is now),

i'm still at 16s here, i would *love* to see this performance speed
increase in the free version that is made available for kernel people

> we're revamping the GUIs to be useable by normal humans, we're

> working on scaling to >500,000 changesets in one tree

one thing i wondered about, is there some way you can optimize
performance knowing for repositories like the kernel tree where the
access patterns are pretty much (for me anyhow) pull in new stuff,
look back a few weeks, maybe clone back as far as a month but beyond
that i rarely pay any attention?

i guess what i'm doing a bad job of saying is that it seems i'm using
only the most recent weeks/days of changes 99% of the time --- can i
do something knowing this that makes my day to day life easier at
maybe the expensive of cloning something really old?

> As for handling AndrewM's workflow, we're very interested in that
> area because there seems to be a sort of bimodal development model,
> changes which are not yet "frozen" (best managed by something like
> quilt it seems) and changes which are frozen (best managed by BK).

right now i use quilt and bk together, i'm still trying to refine a
nice way of doing that (which i should document) but on the whole they
don't conflict as a rule and it's really not that bad. previously i
was using bk like quilt using a script to make csets from patches and
then i would undo to roll back before pulling and what-not but this
was error-prone at times (sometimes i would pull from the parent which
would do a merge so i couldn't roll back after that)


i really don't know why people bitch so much about bk personally, i
really like it, it's much more reliable than svn for me (bk checksums
whilst i might complain about them have found fs bogons a number of
times, with svn i just have a collection of busted db files), it's
much faster than CVS and the bk development model (well, the model i
have when i use it) works better by far than anything i've used
previously

i like bk, i'll continue to use it where possible, for those that have
strong feelings against bk, well, stick with CVS (or whatever) then.
enjoy your pain.

Miles Bader

unread,
Oct 25, 2004, 10:30:08 PM10/25/04
to
Jeff Garzik <jga...@pobox.com> writes:
>> arch exists and it's exactly as distributed as BK.
>
> It doesn't scale or merge as well as BK though.

Examples?

Scalability I'm not sure about; BK's "you must inform BK before you
change a file" model gives it a potential for being very quick at
"tree-compare" operations -- but makes it more annoying for the user.

Merging also seems a bit hard to judge. From what I understand of BK,
it has a much more limited merging model than arch does; to do a
reasonable comparison, you'd have to see how well arch did if you
limited yourself to that restricted model, and then give arch some more
points for not forcing you to do so.

BK no doubt wins on the rough-edges-sanded-down front though; there are
a _few_ advantages to commercial software...

-Miles
--
97% of everything is grunge

Miles Bader

unread,
Oct 25, 2004, 10:30:09 PM10/25/04
to
Andrea Arcangeli <and...@novell.com> writes:
> Note that if I would be buying a BK non-free licence the "restrictions"
> would go away AFIK. However Miles (on the arch list) tries to buy one
> (exactly so he could test BK) and he failed to get one.

Nope, that wasn't me; I don't have any money... :-(

-Miles
--
97% of everything is grunge

Jon Smirl

unread,
Oct 25, 2004, 10:40:08 PM10/25/04
to
On Mon, 25 Oct 2004 16:01:28 -0700, Larry McVoy <l...@bitmover.com> wrote:
> As for digital signatures, we quietly added that a long time ago,
> every push by Linus or Marcelo to bkbits.net is digitally signed twice,

Would it be useful for BK to provide more detailed tracking of who
made what changes to the kernel?

For example a digital signature trail on change sets that tracks who did
what. Say I do some changes on my local tree, export it and
mail it out on lkml. This change set is picked up by Andrew. Andrew
munges on it some and sends it onto Linus. Linus changes another
couple of lines and puts it in his tree. Now I go to bkbits and look
at the history for the lines changed. Can I see what Linus, Andrew and
I have all changed including digital signatures?

The final change set going into linus' tree would have to embed my
orginal change set intact but hidden since it is signed by me and
Linus doesn't have my key. Andrew's and Linus' changes would be added
as signed diffs against my original change set.

You would also need a scheme to import signed diffs and preserve the
signature for non-bk users.

This would automate the "Signed off by" process. It would also much
more accurately track who did what to the tree.

--
Jon Smirl
jons...@gmail.com

Miles Bader

unread,
Oct 25, 2004, 10:50:04 PM10/25/04
to
Linus Torvalds <torv...@osdl.org> writes:
> The fact is, Andrea doesn't actually do anything constructive when it
> comes to SCM.

This is not true.

Andrea has given some very useful input on the gnu-arch mailing list.
He's definitely doing more than just complaining about BK.

-Miles
--
"Though they may have different meanings, the cries of 'Yeeeee-haw!' and
'Allahu akbar!' are, in spirit, not actually all that different."

Matthias Urlichs

unread,
Oct 26, 2004, 3:00:17 AM10/26/04
to
Hi, Jon Smirl wrote:

> For example a digital signature trail on change sets that tracks who did
> what. Say I do some changes on my local tree, export it and
> mail it out on lkml. This change set is picked up by Andrew. Andrew
> munges on it some and sends it onto Linus. Linus changes another
> couple of lines and puts it in his tree. Now I go to bkbits and look
> at the history for the lines changed. Can I see what Linus, Andrew and
> I have all changed including digital signatures?

Sure you can, if you use the "bksend" script Andrew can import your
changes into a bk tree, change them, and then send the combined changes to
Linus.

The problem is that unless both Andrew's and Linus' tree are at exactly
the same version yours is, there will be some merge changesets in between
which really muddle up the change history.

That's fixable, but "bk clone" plus "bk undo" are too expensive at the
moment -- if that took a few seconds instead of a few minutes, people
would probably do it more readily.

--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | sm...@smurf.noris.de

-

Matthias Urlichs

unread,
Oct 26, 2004, 3:40:08 AM10/26/04
to
Hi, Michael Buesch wrote:

> What do kernel developers think about svk?
> (Yes, it's not mature, yet.)
> I mean the svk concept. Does it also suck for kernel development?

The basic idea seems to be "We need feature X. We have a system that's
mostly nice, but it cannot do X. Thus, graft a system to do X onto it.
Voila, one SCM which can do everything we need". Experience suggests
that if you do that, you end up with an ugly mess.

One reason why svk won't work for Linux is this:

A true peer-to-peer system requires that both branches are treated equal.
Bitkeeper got that one exactly right: if I have two repositories A and B,
then importing A to B is the same thing as importing B to A. SVN can't do
that, thus svk can't do it either.

There are others.

Chuck Ebbert

unread,
Oct 26, 2004, 3:50:14 AM10/26/04
to
Larry McVoy wrote:

> we're working bug database integration, we're working
> on review queues, we're working on project management, we're working on
> large binary management, etc.

How about support for backporting patches, e.g. backport csets
1.2000.4.120 and 1.2000.12.16 to kernel 2.6.9? I could see no way to get
bk to spit out these two changes as patches against 2.6.9. Given all the info
in there this should be trivial...

But first you should write some actual documentation and fix your programs
so they emit error messages when something goes wrong:

[me@d4 linux-2.6]$ bk c2r -...@v2.6.9 mm/vmscan.c
[me@d4 linux-2.6]$ <--- what???
[me@d4 linux-2.6]$ bk tags
<snip>
Chan...@1.1988.97.17, 2004-10-18 11:50:06-07:00, torv...@ppc970.osdl.org
Linux 2.6.9
TAG: v2.6.9
<snip>
[me@d4 linux-2.6]$ bk c2r -r1.1988.97.17 mm/vmscan.c
1.231

'bk help terms' says a tag can be used, but apparently not and there's
no error message.

Then there's this one:

[me@d4 linux-2.6]$ bk difftool -rv2.6.8 -rv2.6.9 mm/vmscan.c <--- forgot the @
child process exited abnormally <--- hee hee!

I like the way the GUI flashes up on the screen then abruptly disappears, leaving
the user wondering what happened.


> The list goes on, if anyone wants to come work on it we are always looking
> for good systems people and we pay well.

But you're located in a large American metro area, a.k.a. "hell on earth."


--Chuck Ebbert 26-Oct-04 03:19:25

Giuseppe Bilotta

unread,
Oct 26, 2004, 9:20:09 AM10/26/04
to
Larry McVoy wrote:
> Things we are working on include performance (Wayne has a hot cache
> linux-2.5 tree consistency check down to around 2 seconds, that's
> about a 10x improvement over what it is now), we're revamping the GUIs
> to be useable by normal humans, we're working on scaling to >500,000
> changesets in one tree, we're working on scaling the number of files and
> source to millions of files and GB of source (we just had to recompile
> with largefile support for a customer this weekend, while it worked,
> we thought the performance on a 2GB file was pathetic, it is pathetic,
> it should run at the platter speed but as I dug into I found it out
> wasn't that easy), we're working bug database integration, we're working
> on review queues, we're working on project management, we're working on
> large binary management, etc.

Ok this is probably a very stupid proposal, but since I just
woke up I feel like saying stupid things, and plus I'm happy
with my 512 extra RAM, so here goes:

have you considered some kind of "open source obsolete
versions" procedure à la Aladdin? They develop GhostScript, and
have some sort of "dual licence" strategy: the latest version
follows the AFPL licence, and older versions are released under
GPL.

--
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
(Roger Waters)

Larry McVoy

unread,
Oct 26, 2004, 10:40:07 AM10/26/04
to
On Tue, Oct 26, 2004 at 03:38:06AM -0400, Chuck Ebbert wrote:
> Larry McVoy wrote:
>
> > we're working bug database integration, we're working
> > on review queues, we're working on project management, we're working on
> > large binary management, etc.
>
> How about support for backporting patches, e.g. backport csets
> 1.2000.4.120 and 1.2000.12.16 to kernel 2.6.9? I could see no way to get
> bk to spit out these two changes as patches against 2.6.9. Given all the info
> in there this should be trivial...

Trivial? Sure, if the patches apply cleanly. Not trivial if the changesets
have other changesets in the way.

But I have a 70 line shell script which does this, I'm not sure why you can't
write the same thing.

> But first you should write some actual documentation and fix your programs
> so they emit error messages when something goes wrong:
>
> [me@d4 linux-2.6]$ bk c2r -...@v2.6.9 mm/vmscan.c
> [me@d4 linux-2.6]$ <--- what???

That's because you didn't read the documentation which you appear to love
so much.

work /tmp/linux-2.5 bk c2r -rv2.6.9 mm/vmscan.c
1.231

> 'bk help terms' says a tag can be used, but apparently not and there's
> no error message.

You passed a legal rev syntax to the command. I can't help it if you don't
understand the tool.

> Then there's this one:
>
> [me@d4 linux-2.6]$ bk difftool -rv2.6.8 -rv2.6.9 mm/vmscan.c <--- forgot the @
> child process exited abnormally <--- hee hee!
>
> I like the way the GUI flashes up on the screen then abruptly disappears, leaving
> the user wondering what happened.

You know, I like the way you politely offer to help out by sending in
patches to the documentation for this tool which you get to use for free.
You are an inspiration to us and you are just the sort of user we love
to help.

> > The list goes on, if anyone wants to come work on it we are always looking
> > for good systems people and we pay well.
>
> But you're located in a large American metro area, a.k.a. "hell on earth."

No worries, Chuck, after careful consideration of your skills we don't see
a match at this time. But we'll keep your resume on file and thanks for
your interest.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Chuck Ebbert

unread,
Oct 26, 2004, 12:10:19 PM10/26/04
to
Larry McVoy wrote:

> But I have a 70 line shell script which does this, I'm not sure why you can't
> write the same thing.

Shirley you jest.


>> [me@d4 linux-2.6]$ bk c2r -...@v2.6.9 mm/vmscan.c
>> [me@d4 linux-2.6]$ <--- what???
>
> That's because you didn't read the documentation which you appear to love
> so much.

So now it's MY fault I didn't get any kind of error message whatsoever?
Silent failures like this are really, really annoying...


>> Then there's this one:
>>
>> [me@d4 linux-2.6]$ bk difftool -rv2.6.8 -rv2.6.9 mm/vmscan.c <--- forgot the @
>> child process exited abnormally <--- hee hee!
>>
>> I like the way the GUI flashes up on the screen then abruptly disappears, leaving
>> the user wondering what happened.
>
> You know, I like the way you politely offer to help out by sending in
> patches to the documentation for this tool which you get to use for free.

Are you deliberately missing my point?

Programs should return some kind of error message when they fail. Is that
so hard to understand?


>> But you're located in a large American metro area, a.k.a. "hell on earth."
>
> No worries, Chuck, after careful consideration of your skills we don't see
> a match at this time. But we'll keep your resume on file and thanks for
> your interest.

Your loss, not mine.


--Chuck Ebbert 26-Oct-04 11:16:10

Ryan Anderson

unread,
Oct 26, 2004, 2:30:12 PM10/26/04
to
On Tue, Oct 26, 2004 at 07:23:21AM -0700, Larry McVoy wrote:
> That's because you didn't read the documentation which you appear to love
> so much.

Well, Larry, to be perfectly honest - the documentation is ... terse, at
the moment.

Yes, I know, documentation isn't fun - it probably needs an editor to
go over it and say, "What assumptions should we make about the knowledge
a user has?" and then try really hard to keep that list *short* and
strictly adhered to in all the pages.

Oh, and I *do* play a paying customer at my day job. Just some days I
wish the documentation was a little more helpful. (Specifically, you
desperately need more examples of triggers.)

--

Ryan Anderson
sometimes Pug Majere

Larry McVoy

unread,
Oct 26, 2004, 3:30:20 PM10/26/04
to
On Tue, Oct 26, 2004 at 02:16:22PM -0400, Ryan Anderson wrote:
> On Tue, Oct 26, 2004 at 07:23:21AM -0700, Larry McVoy wrote:
> > That's because you didn't read the documentation which you appear to love
> > so much.
>
> Well, Larry, to be perfectly honest - the documentation is ... terse, at
> the moment.

Yes, we know. On the other hand, we're probably pushing 60,000 users or
so who have figured it out by tinkering with the tool. There is no
question that there needs to be a BitKeeper book or at least a much more
expanded and updated userguide.

> Yes, I know, documentation isn't fun - it probably needs an editor to
> go over it and say, "What assumptions should we make about the knowledge
> a user has?" and then try really hard to keep that list *short* and
> strictly adhered to in all the pages.

Yup. We have an open req for a docs person, if you know someone who
wants a fulltime job in the Bay Area or an incredibly exceptional remote
person we're very interested (and I'll shut up about that, I'm not trying
to use the list for free job advertising, just don't want to create the
impression that we don't recognize the issues).

> Oh, and I *do* play a paying customer at my day job. Just some days I
> wish the documentation was a little more helpful. (Specifically, you
> desperately need more examples of triggers.)

Well if you are paying your support includes that sort of help. Call
us up and ask for examples or send mail to support and ask for help.
Even if you aren't paying you can always ask, if we can we help people
out promptly.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Larry McVoy

unread,
Oct 26, 2004, 5:00:20 PM10/26/04
to
On Tue, Oct 26, 2004 at 11:54:46AM -0400, Chuck Ebbert wrote:
> Shirley you jest.

Chuck, if you really want to talk about this let's take it off line,
the rest of the kernel list doesn't need to know your opinion of our
docs or tools. Last I heard, they were pretty sick of any sort of BK
discussion that wasn't focussed on solving kernel problems.

For the rest of you, if you want something fixed in BK a few ways
to get that done: file a bug|rfe with "bk sendbug", ask questions on
bitkeeper-users at bitmover.com list, and/or send mail to support at
bitmover.

It's probably obvious but bears repeating: polite people tend to
get better support than impolite people, just something to consider.
It's not our policy to distinguish, we fix bugs regardless of how they
are reported, but if you are more or less asking us to do some work for
you we'll put the nice people at the head of the list, that's just
human nature. FYI.


--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

Roman Zippel

unread,
Oct 26, 2004, 10:20:11 PM10/26/04
to
Hi,

On Mon, 25 Oct 2004, Linus Torvalds wrote:

> The only thing I blame Andrea for is _whining_. Don't get me wrong, I
> don't blame him for the bad state of CVS or anything like that.

You must be seeing something I don't. Andrea was objecting to how Larry
was practically declaring defeat for the "bk haters", how you're defining
this as whining I don't know.

> The fact is, Andrea doesn't actually do anything constructive when it
> comes to SCM. He just complains every time somebody says something
> positive about a product that (a) he didn't do anything for and (b) nobody
> forces him to use, and (c) there are no real alternatives for today (much
> less the three years ago he was whining about).

Miles already mentioned that you're wrong here and I don't really know
what you expect here from him.

> > The ability to handle big amounts of patches includes also the
> > possibility to merge a lot of crap. What keeps up the general quality?
>
> No SCM is _ever_ going to be a quality manager.

No disagreement here, but that wasn't really point I was trying to make.

> And that's what Andrea is doing. Sure, BK is commercial, but dammit, so is
> that 2GHz dual-G5 too and that Shuttle box in my corner. They happen to be
> the tools I use for what I do. If Andrea told me that I should use a
> slower machine because that's what most people use, I'd consider him a
> total idiot. Similarly, when he complains that people use software tools
> that clearly _do_ make them more productive, I consider his complaint
> stupid.

Your analogy is flawed, nobody cares what you are using privately, but
your decisions as kernel maintainer have an effect on other people, may
this be the patches you include in the next release or the tools you
distribute them with.
In the end it's your decision what tools you use, if you think the
advantages outweigh the license which goes contrary to the open
devolopment process, that's fine, but so have other people the right to
disagree with that decision. Maybe you could make some suggestion on how
to articulate this more politically correct?
Linus, what disturbs me here is that I don't see that you don't even try
to acknowledge that the bk license might be part of problem, you don't
mention the bk license with a single word. Nobody hates bk, that's a myth
I'd expect from Lar