Kernel SCM saga..
Linus Torvalds
Ok,
as a number of people are already aware (and in some cases have been
aware over the last several weeks), we've been trying to work out a
conflict over BK usage over the last month or two (and it feels like
longer ;). That hasn't been working out, and as a result, the kernel team
is looking at alternatives.
[ And apparently this just hit slashdot too, so by now _everybody_ knows ]
It's not like my choice of BK has been entirely conflict-free ("No,
really? Do tell! Oh, you mean the gigabytes upon gigabytes of flames we
had?"), so in some sense this was inevitable, but I sure had hoped that it
would have happened only once there was a reasonable open-source
alternative. As it is, we'll have to scramble for a while.
Btw, don't blame BitMover, even if that's probably going to be a very
common reaction. Larry in particular really did try to make things work
out, but it got to the point where I decided that I don't want to be in
the position of trying to hold two pieces together that would need as much
glue as it seemed to require.
We've been using BK for three years, and in fact, the biggest problem
right now is that a number of people have gotten very very picky about
their tools after having used the best. Me included, but in fact the
people that got helped most by BitKeeper usage were often the people
_around_ me who had a much easier time merging with my tree and sending
their trees to me.
Of course, there's also probably a ton of people who just used BK as a
nicer (and much faster) "anonymous CVS" client. We'll get that sorted out,
but the immediate problem is that I'm spending most my time trying to see
what the best way to co-operate is.
NOTE! BitKeeper isn't going away per se. Right now, the only real thing
that has happened is that I've decided to not use BK mainly because I need
to figure out the alternatives, and rather than continuing "things as
normal", I decided to bite the bullet and just see what life without BK
looks like. So far it's a gray and bleak world ;)
So don't take this to mean anything more than it is. I'm going to be
effectively off-line for a week (think of it as a normal "Linus went on a
vacation" event) and I'm just asking that people who continue to maintain
BK trees at least try to also make sure that they can send me the result
as (individual) patches, since I'll eventually have to merge some other
way.
That "individual patches" is one of the keywords, btw. One thing that BK
has been extremely good at, and that a lot of people have come to like
even when they didn't use BK, is how we've been maintaining a much finer-
granularity view of changes. That isn't going to go away.
In fact, one impact BK ha shad is to very fundamentally make us (and me in
particular) change how we do things. That ranges from the fine-grained
changeset tracking to just how I ended up trusting submaintainers with
much bigger things, and not having to work on a patch-by-patch basis any
more. So the three years with BK are definitely not wasted: I'm convinced
it caused us to do things in better ways, and one of the things I'm
looking at is to make sure that those things continue to work.
So I just wanted to say that I'm personally very happy with BK, and with
Larry. It didn't work out, but it sure as hell made a big difference to
kernel development. And we'll work out the temporary problem of having to
figure out a set of tools to allow us to continue to do the things that BK
allowed us to do.
Let the flames begin.
Linus
PS. Don't bother telling me about subversion. If you must, start reading
up on "monotone". That seems to be the most viable alternative, but don't
pester the developers so much that they don't get any work done. They are
already aware of my problems ;)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Greg KH
>
> So I just wanted to say that I'm personally very happy with BK, and with
> Larry. It didn't work out, but it sure as hell made a big difference to
> kernel development. And we'll work out the temporary problem of having to
> figure out a set of tools to allow us to continue to do the things that BK
> allowed us to do.
I'd also like to publicly say that BK has helped out immensely in the
past few years with kernel development, and has been one of the main
reasons we have been able to keep up such a high patch rate over such a
long period of time. Larry, and his team, have been nothing but great
in dealing with all of the crap that we have been flinging at him due to
the very odd demands such a large project as the kernel has caused. And
I definitely owe him a beer the next time I see him.
thanks,
greg k-h
Daniel Phillips
> it got to the point where I decided that I don't want to be in
> the position of trying to hold two pieces together that would need as much
> glue as it seemed to require.
Hi Linus,
Well I'm really pleased to hear that you won't be drinking this koolaid any
more. This is a really uplifting development for me, thanks.
Regards,
Daniel
Jon Smirl
> So I just wanted to say that I'm personally very happy with BK, and with
> Larry. It didn't work out, but it sure as hell made a big difference to
> kernel development. And we'll work out the temporary problem of having to
> figure out a set of tools to allow us to continue to do the things that BK
> allowed us to do.
Larry has stated several time that most of his revenue comes from
Windows. Has ODSL approached Bitmover about simply buying out the
source rights for the Linux version? From my experience in the
industry a fair price would probably be around $2M, but that should be
within ODSL's capabilities. ODSL could then GPL the code and quiet the
critics.
--
Jon Smirl
jons...@gmail.com
Matan Peled
> ODSL could then GPL the code and quiet the
> critics.
And also cause aaid GPL'ed code to be immediatly ported over to Windows. I don't
think BitMover could ever agree to that.
--
[Name ] :: [Matan I. Peled ]
[Location ] :: [Israel ]
[Public Key] :: [0xD6F42CA5 ]
[Keyserver ] :: [keyserver.kjsl.com]
encrypted/signed plain text preferred
Paul P Komkoff Jr
> Ok,
> as a number of people are already aware (and in some cases have been
Actually, I'm very disappointed things gone such counter-productive
way. All along the history, I was against Larry's opponents, but at
the end, they are right. That's pity. To quote vin diesel' character
Riddick, "there's no such word as friend", or something.
Anyway, seems that folks in Canonical was aware about it, and here's
the result of this awareness: http://bazaar-ng.org/
This need some testing though, along with really hard part - transfer
all history, nonlinear ... I don't know how anyone can do this till 1
Jul 2005, sorry :(
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
Monotone is good, but I don't really know limits of sqlite3 wrt kernel
case. And again, what we need to do to retain history ...
--
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
This message represents the official view of the voices in my head
Jon Smirl
> Jon Smirl wrote:
> > ODSL could then GPL the code and quiet the
> > critics.
>
> And also cause aaid GPL'ed code to be immediatly ported over to Windows. I don't
> think BitMover could ever agree to that.
Windows Bitkeeper licenses are not that expensive, wouldn't you rather
keep your source in a licensed supported version? Who is going to do
this backport, then support it and track new releases? Why do people
pay for RHEL when they can get it for free? They want support and a
guarantee that their data won't be lost. Even with a GPL'd Linux
Bitkeeper I'll bet half of the existing Linux paying customers will
continue to use a paid version.
There is a large difference in the behavior of corporations with huge
source bases and college students with no money. The corporations will
pay to have someone responsible for ensuring that the product works.
Hua Zhong
> paying customers will continue to use a paid version.
By what? How much do you plan to put down to pay Larry in case you lose your
bet?
Hua
kfo...@collab.net
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
By the way, the Subversion developers have no argument with the claim
that Subversion would not be the right choice for Linux kernel
development. We've written an open letter entitled "Please Stop
Bugging Linus Torvalds About Subversion" to explain why:
http://subversion.tigris.org/subversion-linus.html
Best,
-Karl Fogel (on behalf of the Subversion team)
Jeff Garzik
> Linus Torvalds wrote:
>
>>PS. Don't bother telling me about subversion. If you must, start reading
>>up on "monotone". That seems to be the most viable alternative, but don't
>>pester the developers so much that they don't get any work done. They are
>>already aware of my problems ;)
>
>
> By the way, the Subversion developers have no argument with the claim
> that Subversion would not be the right choice for Linux kernel
> development. We've written an open letter entitled "Please Stop
> Bugging Linus Torvalds About Subversion" to explain why:
>
> http://subversion.tigris.org/subversion-linus.html
A thoughtful post. Thanks for writing this.
Jeff
Jon Masters
> as a number of people are already aware (and in some
> cases have been aware over the last several weeks), we've
> been trying to work out a conflict over BK usage over the last
> month or two (and it feels like longer ;). That hasn't been
> working out, and as a result, the kernel team is looking at
> alternatives.
What about the 64K changeset limitation in current releases?
Did I miss something (like the fixes promised) or is there going to be
another interim release before the end of support?
Jon.
P.S. Apologies if this already got addressed.
Christoph Lameter
> There is a large difference in the behavior of corporations with huge
> source bases and college students with no money. The corporations will
> pay to have someone responsible for ensuring that the product works.
Or they will merge with some other entity on the whim of some executive
and the corporation then decides to kill the product for good without
releasing the source leaving you out in the cold.
Martin Pool
I'd like bazaar-ng to be considered too. It is not ready for adoption
yet, but I am working (more than) full time on it and hope to have it
be usable in a couple of months.
bazaar-ng is trying to integrate a lot of the work done in other systems
to make something that is simple to use but also fast and powerful enough
to handle large projects.
The operations that are already done are pretty fast: ~60s to import a
kernel tree, ~10s to import a new revision from a patch.
Please check it out and do pester me with any suggestions about whatever
you think it needs to suit your work.
--
Martin
Jeff Garzik
> On Wed, 06 Apr 2005 23:39:11 +0400, Paul P Komkoff Jr wrote:
>
> > http://bazaar-ng.org/
>
> I'd like bazaar-ng to be considered too. It is not ready for adoption
> yet, but I am working (more than) full time on it and hope to have it
> be usable in a couple of months.
>
> bazaar-ng is trying to integrate a lot of the work done in other systems
> to make something that is simple to use but also fast and powerful enough
> to handle large projects.
>
> The operations that are already done are pretty fast: ~60s to import a
> kernel tree, ~10s to import a new revision from a patch.
By "importing", are you saying that importing all 60,000+ changesets of
the current kernel tree took only 60 seconds?
Jeff
Martin Pool
>> The operations that are already done are pretty fast: ~60s to import a
>> kernel tree, ~10s to import a new revision from a patch.
>
> By "importing", are you saying that importing all 60,000+ changesets of
> the current kernel tree took only 60 seconds?
Now that would be impressive.
No, I mean this:
% bzcat ../linux.pkg/patch-2.5.14.bz2| patch -p1
% time bzr add -v .
(find any new non-ignored files; deleted files automatically noticed)
6.06s user 0.41s system 89% cpu 7.248 total
% time bzr commit -v -m 'import 2.5.14'
7.71s user 0.71s system 65% cpu 12.893 total
(OK, a bit slower in this case but it wasn't all in core.)
This is only v0.0.3, but I think the interface simplicity and speed
compares well.
I haven't tested importing all 60,000+ changesets of the current bk tree,
partly because I don't *have* all those changesets. (Larry said
previously that someone (not me) tried to pull all of them using bkclient,
and he considered this abuse and blacklisted them.)
I have been testing pulling in release and rc patches, and it scales to
that level. It probably could not handle 60,000 changesets yet, but there
is a plan to get there. In the interim, although it cannot handle the
whole history forever it can handle large trees with moderate numbers of
commits -- perhaps as many as you might deal with in developing a feature
over a course of a few months.
The most sensible place to try out bzr, if people want to, is as a way to
keep your own revisions before mailing a patch to linus or the subsystem
maintainer.
--
Martin
David Lang
> I haven't tested importing all 60,000+ changesets of the current bk tree,
> partly because I don't *have* all those changesets. (Larry said
> previously that someone (not me) tried to pull all of them using bkclient,
> and he considered this abuse and blacklisted them.)
pull the patches from the BK2CVS server. yes some patches are combined,
but it will get you in the ballpark.
David Lang
> I have been testing pulling in release and rc patches, and it scales to
> that level. It probably could not handle 60,000 changesets yet, but there
> is a plan to get there. In the interim, although it cannot handle the
> whole history forever it can handle large trees with moderate numbers of
> commits -- perhaps as many as you might deal with in developing a feature
> over a course of a few months.
>
> The most sensible place to try out bzr, if people want to, is as a way to
> keep your own revisions before mailing a patch to linus or the subsystem
> maintainer.
>
> --
> Martin
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majo...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
Daniel Phillips
> On Wed, 06 Apr 2005 23:39:11 +0400, Paul P Komkoff Jr wrote:
> > http://bazaar-ng.org/
>
> I'd like bazaar-ng to be considered too. It is not ready for adoption
> yet, but I am working (more than) full time on it and hope to have it
> be usable in a couple of months.
>
> bazaar-ng is trying to integrate a lot of the work done in other systems
> to make something that is simple to use but also fast and powerful enough
> to handle large projects.
>
> The operations that are already done are pretty fast: ~60s to import a
> kernel tree, ~10s to import a new revision from a patch.
Hi Martin,
When I tried it, it took 13 seconds to 'bzr add' the 2.6.11.3 tree on a
relatively slow machine.
Regards,
Daniel
Martin Pool
> On Thu, 7 Apr 2005, Martin Pool wrote:
>
> > I haven't tested importing all 60,000+ changesets of the current bk tree,
> > partly because I don't *have* all those changesets. (Larry said
> > previously that someone (not me) tried to pull all of them using bkclient,
> > and he considered this abuse and blacklisted them.)
>
> pull the patches from the BK2CVS server. yes some patches are combined,
> but it will get you in the ballpark.
OK, I just tried that. I know there are scripts to resynthesize
changesets from the CVS info but I skipped that for now and just pulled
each day's work into a separate bzr revision. It's up to the end of
March and still running.
Importing the first snapshot (2004-01-01) took 41.77s user, 1:23.79
total. Each subsequent day takes about 10s user, 30s elapsed to commit
into bzr. The speeds are comparable to CVS or a bit faster, and may be
faster than other distributed systems. (This on a laptop with a 5400rpm
disk.) Pulling out a complete copy of the tree as it was on a previous
date takes about 14 user, 60s elapsed.
I don't want to get too distracted by benchmarks now because there are
more urgent things to do and anyhow there is still lots of scope for
optimization. I wouldn't be at all surprised if those times could be
more than halved. I just wanted to show it is in (I hope) the right
ballpark.
--
Martin
bert hubert
> Monotone is good, but I don't really know limits of sqlite3 wrt kernel
> case. And again, what we need to do to retain history ...
I would't fret over that :-) the big issue I have with sqlite3 is that it
interacts horribly with ext3, resulting in dysmal journalled write
performance compared to ext2. I do not know if this is a sqlite3 or an ext3
problem though.
--
http://www.PowerDNS.com Open source, database driven DNS Software
http://netherlabs.nl Open and Closed source services
Paul Mackerras
> That "individual patches" is one of the keywords, btw. One thing that BK
> has been extremely good at, and that a lot of people have come to like
> even when they didn't use BK, is how we've been maintaining a much finer-
> granularity view of changes. That isn't going to go away.
Are you happy with processing patches + descriptions, one per mail?
Do you have it automated to the point where processing emailed patches
involves little more overhead than doing a bk pull? If so, then your
mailbox (or patch queue) becomes a natural serialization point for the
changes, and the need for a tool that can handle a complex graph of
changes is much reduced.
> In fact, one impact BK ha shad is to very fundamentally make us (and me in
> particular) change how we do things.
From my point of view, the benefits that flowed from your using BK
were:
* Visibility into what you had accepted and committed to your
repository
* Lower latency of patches going into your repository
* Much reduced rate of patches being dropped
Those things are what have enabled us PPC developers to move away from
having our own trees (with all the synchronization problems that
entailed) and work directly with your tree. I don't see that it is
the distinctive features of BK (such as the ability to do merges
between peer repositories) that are directly responsible for producing
those benefits, so I have hope that things can work just as well with
some other system.
Paul.
David Woodhouse
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
One feature I'd want to see in a replacement version control system is
the ability to _re-order_ patches, and to cherry-pick patches from my
tree to be sent onwards. The lack of that capability is the main reason
I always hated BitKeeper.
--
dwmw2
Jan Hudec
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
I have looked at most systems currently available. I would suggest
following for closer look on:
1) GNU Arch/Bazaar. They use the same archive format, simple, have the
concepts right. It may need some scripts or add ons. When Bazaar-NG
is ready, it will be able to read the GNU Arch/Bazaar archives so
switching should be easy.
2) SVK. True, it is built on subversion, but adds all the distributed
features necessary. It keeps mirror of the repository localy (but can
mirror only some branches), but BitKeeper did that too. It just hit
1.0beta1, but the development is progressing rapidly. There was
a post about ability to track changeset dependencies lately on their
mailing-list.
I have looked at Monotone too, of course, but I did not find any way for
doing cherry-picking (ie. skipping some changes and pulling others) in
it and I feel it will need more rework of the meta-data before it is
possible. As for the sqlite backend, I'd not consider that a problem.
-------------------------------------------------------------------------------
Jan 'Bulb' Hudec <bu...@ucw.cz>
Zwane Mwaikambo
> On Thu, Apr 07, 2005 at 11:40:23AM +1000, Martin Pool wrote:
> > On Wed, 06 Apr 2005 23:39:11 +0400, Paul P Komkoff Jr wrote:
> >
> > > http://bazaar-ng.org/
> >
> > I'd like bazaar-ng to be considered too. It is not ready for adoption
> > yet, but I am working (more than) full time on it and hope to have it
> > be usable in a couple of months.
> >
> > bazaar-ng is trying to integrate a lot of the work done in other systems
> > to make something that is simple to use but also fast and powerful enough
> > to handle large projects.
> >
> > The operations that are already done are pretty fast: ~60s to import a
> > kernel tree, ~10s to import a new revision from a patch.
>
> By "importing", are you saying that importing all 60,000+ changesets of
> the current kernel tree took only 60 seconds?
Probably `cvs import` equivalent.
Arjan van de Ven
> Linus,
>
> > That "individual patches" is one of the keywords, btw. One thing that BK
> > has been extremely good at, and that a lot of people have come to like
> > even when they didn't use BK, is how we've been maintaining a much finer-
> > granularity view of changes. That isn't going to go away.
>
> Are you happy with processing patches + descriptions, one per mail?
> Do you have it automated to the point where processing emailed patches
> involves little more overhead than doing a bk pull? If so, then your
> mailbox (or patch queue) becomes a natural serialization point for the
> changes, and the need for a tool that can handle a complex graph of
> changes is much reduced.
alternatively you could send an mbox with your series in... that has a
natural sequence in it ;)
Magnus Damm
> On Thu, 7 Apr 2005, Martin Pool wrote:
>
> > I haven't tested importing all 60,000+ changesets of the current bk tree,
> > partly because I don't *have* all those changesets. (Larry said
> > previously that someone (not me) tried to pull all of them using bkclient,
> > and he considered this abuse and blacklisted them.)
>
> pull the patches from the BK2CVS server. yes some patches are combined,
> but it will get you in the ballpark.
While at it, is there any ongoing effort to convert/export the kernel
BK repository to some well known format like broken out patches and a
series file? I think keeping the complete repository public in a well
known format is important regardless of SCM taste.
/ magnus
Andrew Morton
>
> One feature I'd want to see in a replacement version control system is
> the ability to _re-order_ patches, and to cherry-pick patches from my
> tree to be sent onwards.
You just described quilt & patch-scripts.
The problem with those is letting other people get access to it. I guess
that could be fixed with a bit of scripting and rsyncing.
(I don't do that for -mm because -mm basically doesn't work for 99% of the
time. Takes 4-5 hours to out a release out assuming that nothing's busted,
and usually something is).
Sergei Organov
> On Wed, 2005-04-06 at 08:42 -0700, Linus Torvalds wrote:
> > PS. Don't bother telling me about subversion. If you must, start reading
> > up on "monotone". That seems to be the most viable alternative, but don't
> > pester the developers so much that they don't get any work done. They are
> > already aware of my problems ;)
>
> One feature I'd want to see in a replacement version control system is
> the ability to _re-order_ patches, and to cherry-pick patches from my
> tree to be sent onwards. The lack of that capability is the main reason
> I always hated BitKeeper.
darcs? <http://www.abridgegame.org/darcs/>
Paul Mackerras
> The problem with those is letting other people get access to it. I guess
> that could be fixed with a bit of scripting and rsyncing.
Yes.
> (I don't do that for -mm because -mm basically doesn't work for 99% of the
> time. Takes 4-5 hours to out a release out assuming that nothing's busted,
> and usually something is).
With -mm we get those nice little automatic emails saying you've put
the patch into -mm, which removes one of the main reasons for wanting
to be able to get an up-to-date image of your tree. The other reason,
of course, is to be able to see if a patch I'm about to send conflicts
with something you have already taken, and rebase it if necessary.
Paul.
David Woodhouse
> (I don't do that for -mm because -mm basically doesn't work for 99% of
> the time. Takes 4-5 hours to out a release out assuming that
> nothing's busted, and usually something is).
On the subject of -mm: are you going to keep doing the BK imports to
that for the time being, or would it be better to leave the BK trees
alone now and send you individual patches.
For that matter, will there be a brief amnesty after 2.6.12 where Linus
will use BK to pull those trees which were waiting for that, or will we
all need to export from BK manually?
--
dwmw2
David Vrabel
> David Woodhouse <dw...@infradead.org> wrote:
>
>> One feature I'd want to see in a replacement version control system is
>> the ability to _re-order_ patches, and to cherry-pick patches from my
>> tree to be sent onwards.
>
> You just described quilt & patch-scripts.
>
> The problem with those is letting other people get access to it. I guess
> that could be fixed with a bit of scripting and rsyncing.
Where I work we've been using quilt for a while now and storing the
patch-set in CVS. To limit the number of potential stuff-ups due to two
people working on the same patch at the same time (the chance that CVS's
merge will get it right is zero) we use CVS's locking feature to ensure
that only one person can edit/update a patch or the series file at any
one time. It seems to work quite well (though admittedly there's only
two developers working on the patch-set and it currently contains a mere
61 patches).
We also have a few scripts to ensure we always due the correct locking.
The main ones are:
qec -- to edit a file either as part of the top 'working' patch or as an
existing patch. It does the quilt push which I always forget to do
otherwise.
qrefr -- like quilt refresh only it locks the patch first.
qimport -- like quilt import only it locks the series file first.
You can grab a tarball of these (and other, less interesting ones) from
http://www.davidvrabel.org.uk/quilt-n-cvs-scripts-1.tar.gz
Note that I'm providing this purely on an as-is basis in case any one is
interested.
And I've just realized I can't remember how exactly to set up the CVS
repository of the patch-set. I think you need to do a cvs watch on when
it's checked-out.
David Vrabel
Andrew Morton
>
> With -mm we get those nice little automatic emails saying you've put
> the patch into -mm, which removes one of the main reasons for wanting
> to be able to get an up-to-date image of your tree.
Should have done that ages ago..
> The other reason,
> of course, is to be able to see if a patch I'm about to send conflicts
> with something you have already taken, and rebase it if necessary.
<hack, hack>
How's this?
This is a note to let you know that I've just added the patch titled
ppc32: Fix AGP and sleep again
to the -mm tree. Its filename is
ppc32-fix-agp-and-sleep-again.patch
Patches currently in -mm which might be from yourself are
add-suspend-method-to-cpufreq-core.patch
ppc32-fix-cpufreq-problems.patch
ppc32-fix-agp-and-sleep-again.patch
ppc32-fix-errata-for-some-g3-cpus.patch
ppc64-fix-semantics-of-__ioremap.patch
ppc64-improve-mapping-of-vdso.patch
ppc64-detect-altivec-via-firmware-on-unknown-cpus.patch
ppc64-remove-bogus-f50-hack-in-promc.patch
Andrew Morton
>
> On Thu, 2005-04-07 at 01:50 -0700, Andrew Morton wrote:
> > (I don't do that for -mm because -mm basically doesn't work for 99% of
> > the time. Takes 4-5 hours to out a release out assuming that
> > nothing's busted, and usually something is).
>
> On the subject of -mm: are you going to keep doing the BK imports to
> that for the time being, or would it be better to leave the BK trees
> alone now and send you individual patches.
I really don't know - I'll continue to pull the bk trees for a while, until
we work out what the new (probably interim) regime looks like.
> For that matter, will there be a brief amnesty after 2.6.12 where Linus
> will use BK to pull those trees which were waiting for that, or will we
> all need to export from BK manually?
>
I think Linus has stopped using bk already.
Russell King
> On Thu, 2005-04-07 at 01:50 -0700, Andrew Morton wrote:
> > (I don't do that for -mm because -mm basically doesn't work for 99% of
> > the time. Takes 4-5 hours to out a release out assuming that
> > nothing's busted, and usually something is).
>
> On the subject of -mm: are you going to keep doing the BK imports to
> that for the time being, or would it be better to leave the BK trees
> alone now and send you individual patches.
>
> For that matter, will there be a brief amnesty after 2.6.12 where Linus
> will use BK to pull those trees which were waiting for that, or will we
> all need to export from BK manually?
Linus indicated (maybe privately) that the end of his BK usage would
be immediately after the -rc2 release. I'm taking that to mean "no
more BK usage from Linus, period."
Thinking about it a bit, if you're asking Linus to pull your tree,
Linus would then have to extract the individual change sets as patches
to put into his new fangled patch management system. Is that a
reasonable expectation?
However, it's ultimately up to Linus to decide. 8)
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
David Woodhouse
> Thinking about it a bit, if you're asking Linus to pull your tree,
> Linus would then have to extract the individual change sets as patches
> to put into his new fangled patch management system. Is that a
> reasonable expectation?
I don't know if it's a reasonable expectation; that's why I'm asking.
I could live with having to export everything to patches; it's not so
hard. It's just that if the export to whatever ends up replacing BK can
be done in a way (or at a time) which allows the existing forest of BK
trees to be pulled from one last time, that may save a fair amount of
work all round, so I thought it was worth mentioning.
--
dwmw2
Matthias Andree
> David Woodhouse <dw...@infradead.org> writes:
>
> > On Wed, 2005-04-06 at 08:42 -0700, Linus Torvalds wrote:
> > > PS. Don't bother telling me about subversion. If you must, start reading
> > > up on "monotone". That seems to be the most viable alternative, but don't
> > > pester the developers so much that they don't get any work done. They are
> > > already aware of my problems ;)
> >
> > One feature I'd want to see in a replacement version control system is
> > the ability to _re-order_ patches, and to cherry-pick patches from my
> > tree to be sent onwards. The lack of that capability is the main reason
> > I always hated BitKeeper.
>
> darcs? <http://www.abridgegame.org/darcs/>
Close. Some things:
1. It's rather slow and quite CPU consuming and certainly I/O consuming
at times - I keep, to try it out, leafnode-2 in a DARCS repo, which
has a mere 20,000 lines in 140 files, with 1,436 changes so far, on a
RAID-1 with two 7200/min disk drives, with an Athlon XP 2500+ with
512 MB RAM. The repo has 1,700 files in 11.5 MB, the source itself
189 files in 1.8 MB.
Example: darcs annotate nntpd.c takes 23 s. (2,660 lines, 60 kByte)
The maintainer himself states that there's still optimization required.
2. It has an impressive set of dependencies around Glasgow Haskell
Compiler. I don't personally have issues with that, but I can already
hear the moaning and bitching.
3. DARCS is written in Haskell. This is not a problem either, but I'd
think there are fewer people who can hack Haskell than people who
can hack C, C++, Java, Python or similar. It is still better than
BitKeeper from the hacking POV as the code is available and under an
acceptable license.
Getting DARCS up to the task would probably require some polishing, and
should probably be discussed with the DARCS maintainer before making
this decision.
Don't get me wrong, DARCS looks promising, but I'm not convinced it's
ready for the linux kernel yet.
--
Matthias Andree
Geert Uytterhoeven
> Andrew Morton writes:
> > The problem with those is letting other people get access to it. I guess
> > that could be fixed with a bit of scripting and rsyncing.
>
> Yes.
Me too ;-)
> > (I don't do that for -mm because -mm basically doesn't work for 99% of the
> > time. Takes 4-5 hours to out a release out assuming that nothing's busted,
> > and usually something is).
>
> With -mm we get those nice little automatic emails saying you've put
> the patch into -mm, which removes one of the main reasons for wanting
> to be able to get an up-to-date image of your tree. The other reason,
FYI, for Linus' BK tree procmail was telling me, if it encountered a patch on
the commits list which was signed-off by me.
> of course, is to be able to see if a patch I'm about to send conflicts
> with something you have already taken, and rebase it if necessary.
And yet another reason: to monitor if files/subsystems I'm interested in are
changed.
Summarized: I'd be happy with a mailing list that would send out all patches
(incl. full comment headers, cfr. bk-commit) that Linus commits.
An added bonus would be that people would really be able to reconstruct the
full tree from the mails, unlike with bk-commits (due to `strange' csets caused
by merges). Just make sure there are strictly monotone sequence numbers in the
individual mails.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Andrew Walrond
mentioned on LKML that he had imported the kernel tree). Very impressed so
far, but as you say,
> 1. It's rather slow and quite CPU consuming and certainly I/O consuming
I expect something as large as the kernel tree would cause problems in this
respect.
> 2. It has an impressive set of dependencies around Glasgow Haskell
> Compiler. I don't personally have issues with that, but I can already
> hear the moaning and bitching.
:) I try to built everthing from the original source, but in this case I
couldn't. The GHC needs the GHC + some GHC addons in order to compile
itself...
>
> 3. DARCS is written in Haskell. This is not a problem either, but I'd
> think there are fewer people who can hack Haskell than people who
> can hack C, C++, Java, Python or similar. It is still better than
True, though as you say, not a show-stopper.
From a functionality standpoint, darcs seem very similar to monotone, with a
couple minor trade-offs in either direction.
I wonder if Linus would mind publishing his feature requests to the monotone
developers, so that other projects, like darcs, would know what needs working
on.
Andrew Walrond
Andrew Walrond
>
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
Care to share your monotone wishlist?
Paul Mackerras
> > The other reason,
> > of course, is to be able to see if a patch I'm about to send conflicts
> > with something you have already taken, and rebase it if necessary.
>
> <hack, hack>
>
> How's this?
Nice; but in fact I meant that I want to be able to see if a patch of
mine conflicts with one from somebody else.
Paul.
Daniel Phillips
> When I tried it, it took 13 seconds to 'bzr add' the 2.6.11.3 tree on a
> relatively slow machine.
Oh, and 135 seconds to commit, so 148 seconds overall. Versus 87 seconds to
to bunzip the tree in the first place. So far, you are in the ballpark.
Linus Torvalds
On Thu, 7 Apr 2005, Paul Mackerras wrote:
>
> Are you happy with processing patches + descriptions, one per mail?
Yes. That's going to be my interim, I was just hoping that with 2.6.12-rc2
out the door, and us in a "calming down" period, I could afford to not
even do that for a while.
The real problem with the email thing is that it ends up piling up: what
BK did in this respect was that anythign that piled up in a BK repository
ended up still being there, and a single "bk pull" got it anyway - so if
somebody got ignored because I was busy with something else, it didn't add
any overhead. The queue didn't get "congested".
And that's a big thing. It comes from the "Linus pulls" model where people
just told me that they were ready, instead of the "everybody pushes to
Linus" model, where the destination gets congested at times.
So I do not want the "send Linus email patches" (whether mboxes or a
single patch per email) to be a very long-term strategy. We can handle it
for a while (in particular, I'm counting on it working up to the real
release of 2.6.12, since we _should_ be in the calm period for the next
month anyway), but it doesn't work in the long run.
> Do you have it automated to the point where processing emailed patches
> involves little more overhead than doing a bk pull?
It's more overhead, but not a lot. Especially nice numbered sequences like
Andrew sends (where I don't have to manually try to get the dependencies
right by trying to figure them out and hope I'm right, but instead just
sort by Subject: line) is not a lot of overhead. I can process a hundred
emails almost as easily as one, as long as I trust the maintainer (which,
when it's used as a BK replacement, I obviously do).
However, the SCM's I've looked at make this hard. One of the things (the
main thing, in fact) I've been working at is to make that process really
_efficient_. If it takes half a minute to apply a patch and remember the
changeset boundary etc (and quite frankly, that's _fast_ for most SCM's
around for a project the size of Linux), then a series of 250 emails
(which is not unheard of at all when I sync with Andrew, for example)
takes two hours. If one of the patches in the middle doesn't apply, things
are bad bad bad.
Now, BK wasn't a speed deamon either (actually, compared to everything
else, BK _is_ a speed deamon, often by one or two orders of magnitude),
and took about 10-15 seconds per email when I merged with Andrew. HOWEVER,
with BK that wasn't as big of an issue, since the BK<->BK merges were so
easy, so I never had the slow email merges with any of the other main
developers. So a patch-application-based SCM "merger" actually would need
to be _faster_ than BK is. Which is really really really hard.
So I'm writing some scripts to try to track things a whole lot faster.
Initial indications are that I should be able to do it almost as quickly
as I can just apply the patch, but quite frankly, I'm at most half done,
and if I hit a snag maybe that's not true at all. Anyway, the reason I can
do it quickly is that my scripts will _not_ be an SCM, they'll be a very
specific "log Linus' state" kind of thing. That will make the linear patch
merge a lot more time-efficient, and thus possible.
(If a patch apply takes three seconds, even a big series of patches is not
a problem: if I get notified within a minute or two that it failed
half-way, that's fine, I can then just fix it up manually. That's why
latency is critical - if I'd have to do things effectively "offline",
I'd by definition not be able to fix it up when problems happen).
> If so, then your mailbox (or patch queue) becomes a natural
> serialization point for the changes, and the need for a tool that can
> handle a complex graph of changes is much reduced.
Yes. In the short term. See above why I think the congestion issue will
really mean that we want to have parallell merging in the not _too_
distant future.
NOTE! I detest the centralized SCM model, but if push comes to shove, and
we just _can't_ get a reasonable parallell merge thing going in the short
timeframe (ie month or two), I'll use something like SVN on a trusted site
with just a few committers, and at least try to distribute the merging out
over a few people rather than making _me_ be the throttle.
The reason I don't really want to do that is once we start doing it that
way, I suspect we'll have a _really_ hard time stopping. I think it's a
broken model. So I'd much rather try to have some pain in the short run
and get a better model running, but I just wanted to let people know that
I'm pragmatic enough that I realize that we may not have much choice.
> * Visibility into what you had accepted and committed to your
> repository
> * Lower latency of patches going into your repository
> * Much reduced rate of patches being dropped
Yes.
Linus
Linus Torvalds
On Thu, 7 Apr 2005, David Woodhouse wrote:
>
> On Wed, 2005-04-06 at 08:42 -0700, Linus Torvalds wrote:
> > PS. Don't bother telling me about subversion. If you must, start reading
> > up on "monotone". That seems to be the most viable alternative, but don't
> > pester the developers so much that they don't get any work done. They are
> > already aware of my problems ;)
>
> One feature I'd want to see in a replacement version control system is
> the ability to _re-order_ patches, and to cherry-pick patches from my
> tree to be sent onwards. The lack of that capability is the main reason
> I always hated BitKeeper.
I really disliked that in BitKeeper too originally. I argued with Larry
about it, but Larry (correctly, I believe) argued that efficient and
reliable distribution really requires the concept of "history is
immutable". It makes replication much easier when you know that the known
subset _never_ shrinks or changes - you only add on top of it.
And that implies no cherry-picking.
Also, there's actually a second reason why I've decided that cherry-
picking is wrong, and it's non-technical.
The thing is, cherry-picking very much implies that the people "up" the
foodchain end up editing the work of the people "below" them. The whole
reason you want cherry-picking is that you want to fix up somebody elses
mistakes, ie something you disagree with.
That sounds like an obviously good thing, right? Yes it does.
The problem is, it actually results in the wrong dynamics and psychology
in the system. First off, it makes the implicit assumption that there is
an "up" and "down" in the food-chain, and I think that's wrong. It's
increasingly a "network" in the kernel. I'm less and less "the top", as
much as a "fairly central" person. And that is how it should be. I used to
think of kernel development as a hierarchy, but I long since switched to
thinking about it as a fairly arbitrary network.
The other thing it does is that it implicitly puts the burden of quality
control at the upper-level maintainer ("I'll pick the good things out of
your tree"), while _not_ being able to cherry-pick means that there is
pressure in both directions to keep the tree clean.
And that is IMPORTANT. I realize that not cherry-picking means that people
who want to merge upstream (or sideways or anything) are now forced to do
extra work in trying to keep their tree free of random crap. And that's a
HUGELY IMPORTANT THING! It means that the pressure to keep the tree clean
flows in all directions, and takes pressure off the "central" point. In
onther words it distributes the pain of maintenance.
In other words, somebody who can't keep their act together, and creates
crappy trees because he has random pieces of crud in it, quite
automatically gets actively shunned by others. AND THAT IS GOOD! I've
pushed back on some BK users to clean up their trees, to the point where
we've had a number of "let's just re-do that" over the years. That's
WONDERFUL. People are irritated at first, but I've seen what the end
result is, and the end result is a much better maintainer.
Some people actually end up doing the cleanup different ways. For example,
Jeff Garzik kept many separate trees, and had a special merge thing.
Others just kept a messy tree for development, and when they are happy,
they throw the messy tree away and re-create a cleaner one. Either is fine
- the point is, different people like to work different ways, and that's
fine, but makign _everybody_ work at being clean means that there is no
train wreck down the line when somebody is forced to try to figure out
what to cherry-pick.
So I've actually changed from "I want to cherry-pick" to "cherry-picking
between maintainers is the wrong workflow". Now, as part of cleaning up,
people may end up exporting the "ugly tree" as patches and re-importing it
into the clean tree as the fixed clean series of patches, and that's
"cherry-picking", but it's not between developers.
NOTE! The "no cherry-picking" model absolutely also requires a model of
"throw-away development trees". The two go together. BK did both, and an
SCM that does one but not the other would be horribly broken.
(This is my only real conceptual gripe with "monotone". I like the model,
but they make it much harder than it should be to have throw-away trees
due to the fact that they seem to be working on the assumption of "one
database per developer" rather than "one database per tree". You don't
have to follow that model, but it seems to be what the setup is geared
for, and together with their "branches" it means that I think a monotone
database easily gets very cruddy. The other problem with monotone is
just performance right now, but that's hopefully not _too_ fundamental).
Linus
Rik van Riel
> the very odd demands such a large project as the kernel has caused. And
> I definitely owe him a beer the next time I see him.
Seconded. Besides, now that the code won't be on bkbits
any more, it's safe to get Larry drunk ;)
Larry, thanks for the help you have given us by making
bitkeeper available for all these years.
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
Daniel Phillips
> On Thu, 7 Apr 2005, David Woodhouse wrote:
> > On Wed, 2005-04-06 at 08:42 -0700, Linus Torvalds wrote:
> > > PS. Don't bother telling me about subversion. If you must, start
> > > reading up on "monotone". That seems to be the most viable alternative,
> > > but don't pester the developers so much that they don't get any work
> > > done. They are already aware of my problems ;)
> >
> > One feature I'd want to see in a replacement version control system is
> > the ability to _re-order_ patches, and to cherry-pick patches from my
> > tree to be sent onwards. The lack of that capability is the main reason
> > I always hated BitKeeper.
>
> I really disliked that in BitKeeper too originally. I argued with Larry
> about it, but Larry (correctly, I believe) argued that efficient and
> reliable distribution really requires the concept of "history is
> immutable". It makes replication much easier when you know that the known
> subset _never_ shrinks or changes - you only add on top of it.
However, it would be easy to allow reordering before "publishing" a revision,
which would preserve immutability for all published revisions while allowing
the patch _author_ the flexibility of reordering/splitting/joining patches
when creating them. In other words, a virtuous marriage of the BK model with
Andrew's Quilt.
Regards,
Daniel
Daniel Phillips
> On Thu, 7 Apr 2005, Paul Mackerras wrote:
> > Do you have it automated to the point where processing emailed patches
> > involves little more overhead than doing a bk pull?
>
> It's more overhead, but not a lot. Especially nice numbered sequences like
> Andrew sends (where I don't have to manually try to get the dependencies
> right by trying to figure them out and hope I'm right, but instead just
Hi Linus,
In that case, a nice refinement is to put the sequence number at the end of
the subject line so patch sequences don't interleave:
Subject: [PATCH] Unbork OOM Killer (1 of 3)
Subject: [PATCH] Unbork OOM Killer (2 of 3)
Subject: [PATCH] Unbork OOM Killer (3 of 3)
Subject: [PATCH] Unbork OOM Killer (v2, 1 of 3)
Subject: [PATCH] Unbork OOM Killer (v2, 2 of 3)
...
Regards,
Daniel
Al Viro
> Also, there's actually a second reason why I've decided that cherry-
> picking is wrong, and it's non-technical.
>
> The thing is, cherry-picking very much implies that the people "up" the
> foodchain end up editing the work of the people "below" them. The whole
> reason you want cherry-picking is that you want to fix up somebody elses
> mistakes, ie something you disagree with.
No. There's another reason - when you are cherry-picking and reordering
*your* *own* *patches*. That's what I had been unable to explain to
Larry and that's what made BK unusable for me.
As for the immutable history... Ever had to read or grade students'
homework?
* the dumbest kind: "here's an answer <expression>, whaddya
mean 'where's the solution'?".
* next one: "here's how I've solved the problem: <pages of text
documenting the attempts, with many 'oops, there had been a mistake,
here's how we fix it'>".
* what you really want to see: series of steps leading to answer,
with clean logical structure that allows to understand what's being
done and verify correctness.
The first corresponds to "here's a half-meg of patch, it fixes everything".
The second is chronological history (aka "this came from our CVS, all bugs
are fixed by now, including those introduced in the middle of it; see
CVS history for details"). The third is a decent patch series.
And to get from "here's how I came up to solution" to "here's a clean way
to reach the solution" you _have_ to reorder. There's also "here are
misc notes from today, here are misc notes from yesterday, etc." and to
get that into sane shape you will need to split, reorder and probably
collapse several into combined delta (possibly getting an empty delta
as the result, if later ones negate the prior).
The point being, both history and well, publishable result can be expressed
as series of small steps, but they are not the same thing. So far all I've
seen in the area (and that includes BK) is heavily biased towards history part
and attempts to use this stuff for manipulating patch series turn into fighting
the tool.
I'd *love* to see something that can handle both - preferably with
history of reordering, etc. being kept. IOW, not just a tree of changesets
but a lattice - with multiple paths leading to the same node. So far
I've seen nothing of that kind ;-/
Linus Torvalds
On Thu, 7 Apr 2005, Daniel Phillips wrote:
>
> In that case, a nice refinement is to put the sequence number at the end of
> the subject line so patch sequences don't interleave:
No. That makes it unsortable, and also much harder to pick put which part
of the subject line is the explanation, and which part is just metadata
for me.
So my prefernce is _overwhelmingly_ for the format that Andrew uses (which
is partly explained by the fact that I am used to it, but also by the fact
that I've asked for Andrew to make trivial changes to match my usage).
That canonical format is:
Subject: [PATCH 001/123] [<area>:] <explanation>
together with the first line of the body being a
From: Original Author <or...@email.com>
followed by an empty line and then the body of the explanation.
After the body of the explanation comes the "Signed-off-by:" lines, and
then a simple "---" line, and below that comes the diffstat of the patch
and then the patch itself.
That's the "canonical email format", and it's that because my normal
scripts (in BK/tools, but now I'm working on making them more generic)
take input that way. It's very easy to sort the emails alphabetically by
subject line - pretty much any email reader will support that - since
because the sequence number is zero-padded, the numerical and alphabetic
sort is the same.
If you send several sequences, you either send a simple explaining email
before the second sequence (hey, it's not like I'm a machine - I can use
my brains too, and in particular if the final number of patches in each
sequence is different, even if the sequences got re-ordered and are
overlapping, I can still just extract one from the other by selecting for
"/123] " in the subject line), or you modify the Subject: line subtly to
still sort uniquely and alphabetically in-order, ie the subject lines for
the second series might be
Subject: [PATCHv2 001/207] x86: fix eflags tracking
...
All very unambiguous, and my scripts already remove everything inside the
brackets and will just replace it with "[PATCH]" in the final version.
Linus
Linus Torvalds
On Thu, 7 Apr 2005, Al Viro wrote:
>
> No. There's another reason - when you are cherry-picking and reordering
> *your* *own* *patches*.
Yes. I agree. There should be some support for cherry-picking in between a
temporary throw-away tree and a "cleaned-up-tree". However, it should be
something you really do need to think about, and in most cases it really
does boil down to "export as patch, re-import from patch". Especially
since you potentially want to edit things in between anyway when you
cherry-pick.
(I do that myself: If I have been a messy boy, and committed mixed-up
things as one commit, I export it as a patch, and then I split the patch
by hand into two or more pieces - sometimes by just editing the patch
directly, but sometimes with a combination of by applying it, and editing
the result, and then re-exporting it as the new version).
And in the cases where this happens, you in fact often have unrelated
changes to the _same_file_, so you really do end up having that middle
step.
In other words, this cherry-picking can generally be scripted and done
"outside" the SCM (you can trivially have a script that takes a revision
from one tree and applies it to the other). I don't believe that the SCM
needs to support it in any fundamentally inherent manner. After all, why
should it, when it really boilds down to
(cd old-tree ; scm export-as-patch-plus-comments) |
(cd new-tree ; scm import-patch-plus-comments)
where the "patch-plus-comments" part is just basically an extended patch
(including rename information etc, not just the comments).
Btw, this method of cherry-picking again requires two _separate_ active
trees at the same time. BK is great at that, and really, that's what
distributed SCM's should be all about anyway. It's not just distributed
between different machines, it's literally distributed even on the same
machine, and it's actively _used_ that way.
Linus
Chris Wedgwood
> So my prefernce is _overwhelmingly_ for the format that Andrew uses
> (which is partly explained by the fact that I am used to it, but
> also by the fact that I've asked for Andrew to make trivial changes
> to match my usage).
>
> That canonical format is:
>
> Subject: [PATCH 001/123] [<area>:] <explanation>
>
> together with the first line of the body being a
>
> From: Original Author <or...@email.com>
>
> followed by an empty line and then the body of the explanation.
Having a script to check people get this right before sending it via
email would be a nice thing to put into scripts/ or probably
Documentation/ perhaps?
Does such a thing already exist?
Daniel Phillips
> The point being, both history and well, publishable result can be expressed
> as series of small steps, but they are not the same thing. So far all I've
> seen in the area (and that includes BK) is heavily biased towards history
> part and attempts to use this stuff for manipulating patch series turn into
> fighting the tool.
>
> I'd *love* to see something that can handle both - preferably with
> history of reordering, etc. being kept. IOW, not just a tree of changesets
> but a lattice - with multiple paths leading to the same node. So far
> I've seen nothing of that kind ;-/
Which is a perfect demonstration of why the scm tool has to be free/open
source. We should never have had to plead with BitMover to extend BK in a
direction like that, but instead, just get the source and make it do it, like
any other open source project.
Three years ago, there was no fully working open source distributed scm code
base to use as a starting point, so extending BK would have been the only
easy alternative. But since then the situation has changed. There are now
several working code bases to provide a good starting point: Monotone, Arch,
SVK, Bazaar-ng and others.
Sure, there are quibbles about all of those, but right now is not the time for
quibbling, because a functional replacement for BK is needed in roughly two
months, capable of losslessly importing the kernel version graph. It only
has to support a subset of BK functionality, e.g., pulling and cloning. It
is ok to be a little slow so long as it is not pathetically slow. The
purpose of the interim solution is just to get the patch flow process back
online.
The key is the _lossless_ part. So long as the interim solution imports the
metadata losslessly, we have the flexibility to switch to a better solution
later, on short notice and without much pain.
So I propose that everybody who is interested, pick one of the above projects
and join it, to help get it to the point of being able to losslessly import
the version graph. Given the importance, I think that _all_ viable
alternatives need to be worked on in parallel, so that two months from now we
have several viable options.
Regards,
Daniel
Bartlomiej Zolnierkiewicz
> On Thu, Apr 07, 2005 at 08:32:04AM -0700, Linus Torvalds wrote:
> > Also, there's actually a second reason why I've decided that cherry-
> > picking is wrong, and it's non-technical.
> >
> > The thing is, cherry-picking very much implies that the people "up" the
> > foodchain end up editing the work of the people "below" them. The whole
> > reason you want cherry-picking is that you want to fix up somebody elses
> > mistakes, ie something you disagree with.
>
> No. There's another reason - when you are cherry-picking and reordering
> *your* *own* *patches*. That's what I had been unable to explain to
> Larry and that's what made BK unusable for me.
Yep, I missed this in BK a lot.
There is another situation in which cherry-picking is very useful:
even if you have a clean tree it still may contain bugfixes mixed with
unrelated cleanups and sometimes you want to only apply bugfixes.
Bartlomiej
Magnus Damm
> So my prefernce is _overwhelmingly_ for the format that Andrew uses (which
> is partly explained by the fact that I am used to it, but also by the fact
> that I've asked for Andrew to make trivial changes to match my usage).
>
> That canonical format is:
>
> Subject: [PATCH 001/123] [<area>:] <explanation>
>
> together with the first line of the body being a
>
> From: Original Author <or...@email.com>
>
> followed by an empty line and then the body of the explanation.
>
> After the body of the explanation comes the "Signed-off-by:" lines, and
> then a simple "---" line, and below that comes the diffstat of the patch
> and then the patch itself.
While specifying things, wouldn't it be useful to have a line
containing tags that specifies if the patch contains new features, a
bug fix or a high-priority security fix? Then that information could
be used to find patches for the sucker-tree.
/ magnus
Jörn Engel
> On Thu, 7 Apr 2005, Al Viro wrote:
> >
> > No. There's another reason - when you are cherry-picking and reordering
> > *your* *own* *patches*.
>
> Yes. I agree. There should be some support for cherry-picking in between a
> temporary throw-away tree and a "cleaned-up-tree". However, it should be
> something you really do need to think about, and in most cases it really
> does boil down to "export as patch, re-import from patch". Especially
> since you potentially want to edit things in between anyway when you
> cherry-pick.
For reordering, using patcher, you can simply edit the sequence file
and move lines around. Nice and simple interface.
There is no checking involved, though. If you mode dependent patches,
you end up with a mess and either throw it all away or seriously
scratch your head. So a serious SCM might do something like this:
$ cp series new_series
$ vi new_series
$ SCM --reorder new_series
# essentially "mv new_series series", if no checks fail
Merging patches isn't that hard either. Splitting them would remain
manual, as you described it.
> Btw, this method of cherry-picking again requires two _separate_ active
> trees at the same time. BK is great at that, and really, that's what
> distributed SCM's should be all about anyway. It's not just distributed
> between different machines, it's literally distributed even on the same
> machine, and it's actively _used_ that way.
Amen!
Jörn
--
He who knows that enough is enough will always have enough.
-- Lao Tsu
Dmitry Yusupov
> Three years ago, there was no fully working open source distributed scm code
> base to use as a starting point, so extending BK would have been the only
> easy alternative. But since then the situation has changed. There are now
> several working code bases to provide a good starting point: Monotone, Arch,
> SVK, Bazaar-ng and others.
Right. For example, SVK is pretty mature project and very close to 1.0
release now. And it supports all kind of merges including Cherry-Picking
Mergeback:
http://svk.elixus.org/?MergeFeatures
Dmitry
Daniel Phillips
> On Thu, 7 April 2005 10:47:18 -0700, Linus Torvalds wrote:
> > a temporary throw-away tree and a "cleaned-up-tree". However, it should
> > be something you really do need to think about, and in most cases it
> > really does boil down to "export as patch, re-import from patch".
> > Especially since you potentially want to edit things in between anyway
> > when you cherry-pick.
>
> For reordering, using patcher, you can simply edit the sequence file
> and move lines around. Nice and simple interface.
>
> There is no checking involved, though. If you mode dependent patches,
> you end up with a mess and either throw it all away or seriously
> scratch your head. So a serious SCM might do something like this:
>
> $ cp series new_series
> $ vi new_series
> $ SCM --reorder new_series
> # essentially "mv new_series series", if no checks fail
>
> Merging patches isn't that hard either. Splitting them would remain
> manual, as you described it.
Well it's clear that adding cherry picking, patch reordering, splitting and
merging (two patches into one) is not even hard, it's just a matter of making
it convenient by _building it into the tool_. Now, can we just pick a tool
and do it, please? :-)
Regards,
Daniel
Daniel Phillips
> On Thu, 2005-04-07 at 13:54 -0400, Daniel Phillips wrote:
> > Three years ago, there was no fully working open source distributed scm
> > code base to use as a starting point, so extending BK would have been the
> > only easy alternative. But since then the situation has changed. There
> > are now several working code bases to provide a good starting point:
> > Monotone, Arch, SVK, Bazaar-ng and others.
>
> Right. For example, SVK is pretty mature project and very close to 1.0
> release now. And it supports all kind of merges including Cherry-Picking
> Mergeback:
>
> http://svk.elixus.org/?MergeFeatures
So for an interim way to get the patch flow back online, SVK is ready to try
_now_, and we only need a way to import the version graph? (true/false)
Regards,
Daniel
Daniel Phillips
> On Thu, 7 Apr 2005, Daniel Phillips wrote:
> > In that case, a nice refinement is to put the sequence number at the end
> > of the subject line so patch sequences don't interleave:
>
> No. That makes it unsortable, and also much harder to pick put which part
> of the subject line is the explanation, and which part is just metadata
> for me.
Well, my list in the parent post _was_ sorted by subject. But that is a
quibble, the important point is that you just officially defined the
canonical format, which everybody should stick to for now:
Regards,
Daniel
Sam Ravnborg
> On Thursday 07 April 2005 11:10, Linus Torvalds wrote:
> > On Thu, 7 Apr 2005, Paul Mackerras wrote:
> > > Do you have it automated to the point where processing emailed patches
> > > involves little more overhead than doing a bk pull?
> >
> > It's more overhead, but not a lot. Especially nice numbered sequences like
> > Andrew sends (where I don't have to manually try to get the dependencies
> > right by trying to figure them out and hope I'm right, but instead just
> > sort by Subject: line)...
>
> Hi Linus,
>
> In that case, a nice refinement is to put the sequence number at the end of
> the subject line so patch sequences don't interleave:
>
> Subject: [PATCH] Unbork OOM Killer (1 of 3)
> Subject: [PATCH] Unbork OOM Killer (2 of 3)
> Subject: [PATCH] Unbork OOM Killer (3 of 3)
> Subject: [PATCH] Unbork OOM Killer (v2, 1 of 3)
> Subject: [PATCH] Unbork OOM Killer (v2, 2 of 3)
This breaks the rule of a descriptive subject for each patch.
Consider 30 subjetcs telling you "Subject: PCI updates [001/030]
That is not good.
Sam
Arjan van de Ven
> On Thu, 7 April 2005 10:47:18 -0700, Linus Torvalds wrote:
> > On Thu, 7 Apr 2005, Al Viro wrote:
> > >
> > > No. There's another reason - when you are cherry-picking and reordering
> > > *your* *own* *patches*.
> >
> > Yes. I agree. There should be some support for cherry-picking in between a
> > temporary throw-away tree and a "cleaned-up-tree". However, it should be
> > something you really do need to think about, and in most cases it really
> > does boil down to "export as patch, re-import from patch". Especially
> > since you potentially want to edit things in between anyway when you
> > cherry-pick.
>
> For reordering, using patcher, you can simply edit the sequence file
> and move lines around. Nice and simple interface.
>
> There is no checking involved, though. If you mode dependent patches,
> you end up with a mess and either throw it all away or seriously
> scratch your head. So a serious SCM might do something like this:
just fyi, patchutils has a tool that can "flip" the order of patches
even if they patch the same line of code in the files.... with it you
can make a "bubble sort" to move stuff about safely...
Linus Torvalds
On Thu, 7 Apr 2005, Martin Pool wrote:
>
> Importing the first snapshot (2004-01-01) took 41.77s user, 1:23.79
> total. Each subsequent day takes about 10s user, 30s elapsed to commit
> into bzr. The speeds are comparable to CVS or a bit faster, and may be
> faster than other distributed systems. (This on a laptop with a 5400rpm
> disk.) Pulling out a complete copy of the tree as it was on a previous
> date takes about 14 user, 60s elapsed.
If you have an exportable tree, can you just make it pseudo-public, tell
me where to get a buildable system that works well enough, point me to
some documentation, and maybe I can get some feel for it?
Linus
Dave Airlie
>
> Yes. That's going to be my interim, I was just hoping that with 2.6.12-rc2
> out the door, and us in a "calming down" period, I could afford to not
> even do that for a while.
>
> The real problem with the email thing is that it ends up piling up: what
> BK did in this respect was that anythign that piled up in a BK repository
> ended up still being there, and a single "bk pull" got it anyway - so if
> somebody got ignored because I was busy with something else, it didn't add
> any overhead. The queue didn't get "congested".
>
> And that's a big thing. It comes from the "Linus pulls" model where people
> just told me that they were ready, instead of the "everybody pushes to
> Linus" model, where the destination gets congested at times.
Something I think we'll miss is bkbits.net in the long run, being able
to just push all patches for Linus to a tree and then forget about
that tree until Linus pulled from it was invaluable.. the fact that
this tree was online the whole time and you didn't queue up huge mails
for Linus's INBOX to be missed, meant a lot to me compared to pre-bk
workings..
Maybe now that kernel.org has been 'pimped out' we could set some sort
of system up where maintainers can drop a big load of patchsets or
even one big patch into some sort of public area and say this is my
diffs for Linus for his next pull and let Linus pull it at his
lesuire... some kinda rsync'y type thing comes to mind ...
so I can mail Linus and say hey Linus please grab
rsync://pimpedout.kernel.org/airlied/drm-linus and you grab everything
in there and I get notified perhaps or just a log like the bkbits
stats page, and Andrew can grab the patchsets the same as he does for
bk-drm now ... and I can have airlied/drm-2.6 where I can queue stuff
for -mm then just re-generate the patches for drm-linus later on..
Dave.
Ian Wienand
> If you must, start reading up on "monotone".
One slightly annoying thing is that monotone doesn't appear to have a
web interface. I used to use the bk one a lot when tracking down
bugs, because it was really fast to have a web browser window open and
click through the revisions of a file reading checkin comments, etc.
Does anyone know if one is being worked on?
bazaar-ng at least mention this is important in their design docs and
arch has one in development too.
Jesse Barnes
> Larry, thanks for the help you have given us by making
> bitkeeper available for all these years.
A big thank you from me too, I've really enjoyed using BK and I think it's
made me much more productive than I would have been otherwise. I don't envy
you having to put up with the frequent flamefests...
Jesse
Jeff Garzik
>
> On Thu, 7 Apr 2005, Daniel Phillips wrote:
>
>>In that case, a nice refinement is to put the sequence number at the end of
>>the subject line so patch sequences don't interleave:
>
>
> No. That makes it unsortable, and also much harder to pick put which part
> of the subject line is the explanation, and which part is just metadata
> for me.
>
> So my prefernce is _overwhelmingly_ for the format that Andrew uses (which
> is partly explained by the fact that I am used to it, but also by the fact
> that I've asked for Andrew to make trivial changes to match my usage).
>
> That canonical format is:
>
> Subject: [PATCH 001/123] [<area>:] <explanation>
>
> together with the first line of the body being a
>
> From: Original Author <or...@email.com>
Nod. For future reference, people can refer to
http://linux.yyz.us/patch-format.html
and/or
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt
Jeff
Jeff Garzik
> In other words, this cherry-picking can generally be scripted and done
> "outside" the SCM (you can trivially have a script that takes a revision
> from one tree and applies it to the other). I don't believe that the SCM
> needs to support it in any fundamentally inherent manner. After all, why
> should it, when it really boilds down to
>
> (cd old-tree ; scm export-as-patch-plus-comments) |
> (cd new-tree ; scm import-patch-plus-comments)
>
> where the "patch-plus-comments" part is just basically an extended patch
> (including rename information etc, not just the comments).
Not that it matters anymore, but that's precisely what the script
Documentation/BK-usage/cpcset
did, for BitKeeper.
Jeff
Chris Wedgwood
> PS. Don't bother telling me about subversion. If you must, start reading
> up on "monotone". That seems to be the most viable alternative, but don't
> pester the developers so much that they don't get any work done. They are
> already aware of my problems ;)
I'm playing with monotone right now. Superficially it looks like it
has tons of gee-whiz neato stuff... however, it's *agonizingly* slow.
I mean glacial. A heavily sedated sloth with no legs is probably
faster.
Using monotone to pull itself too over 2 hours wall-time and 71
minutes of CPU time.
Arguably brand-new CPUs are probably about 2x the speed of what I have
now and there might have been networking funnies --- but that's still
35 monutes to get ~40MB of data.
The kernel is ten times larger, so does that mean to do a clean pull
of the kernel we are looking at (71/2*10) ~ 355 minutes or 6 hours of
CPU time?
Linus Torvalds
On Thu, 7 Apr 2005, Chris Wedgwood wrote:
>
> I'm playing with monotone right now. Superficially it looks like it
> has tons of gee-whiz neato stuff... however, it's *agonizingly* slow.
> I mean glacial. A heavily sedated sloth with no legs is probably
> faster.
Yes. The silly thing is, at least in my local tests it doesn't actually
seem to be _doing_ anything while it's slow (there are no system calls
except for a few memory allocations and de-allocations). It seems to have
some exponential function on the number of pathnames involved etc.
I'm hoping they can fix it, though. The basic notions do not sound wrong.
In the meantime (and because monotone really _is_ that slow), here's a
quick challenge for you, and any crazy hacker out there: if you want to
play with something _really_ nasty (but also very _very_ fast), take a
look at kernel.org:/pub/linux/kernel/people/torvalds/.
First one to send me the changelog tree of sparse-git (and a tool to
commit and push/pull further changes) gets a gold star, and an honorable
mention. I've put a hell of a lot of clues in there (*).
I've worked on it (and little else) for the last two days. Time for
somebody else to tell me I'm crazy.
Linus
(*) It should be easier than it sounds. The database is designed so that
you can do the equivalent of a nonmerging (ie pure superset) push/pull
with just plain rsync, so replication really should be that easy (if
somewhat bandwidth-intensive due to the whole-file format).
Never mind merging. It's not an SCM, it's a distribution and archival
mechanism. I bet you could make a reasonable SCM on top of it, though.
Another way of looking at it is to say that it's really a content-
addressable filesystem, used to track directory trees.
Chris Wedgwood
> Yes. The silly thing is, at least in my local tests it doesn't
> actually seem to be _doing_ anything while it's slow (there are no
> system calls except for a few memory allocations and
> de-allocations). It seems to have some exponential function on the
> number of pathnames involved etc.
I see lots of brk calls changing the heap size, up, down, up, down,
over and over.
This smells a bit like c++ new/delete behavior to me.
H. Peter Anvin
By author: Chris Wedgwood <c...@f00f.org>
In newsgroup: linux.dev.kernel
>
> On Thu, Apr 07, 2005 at 09:42:04PM -0700, Linus Torvalds wrote:
>
> > Yes. The silly thing is, at least in my local tests it doesn't
> > actually seem to be _doing_ anything while it's slow (there are no
> > system calls except for a few memory allocations and
> > de-allocations). It seems to have some exponential function on the
> > number of pathnames involved etc.
>
> I see lots of brk calls changing the heap size, up, down, up, down,
> over and over.
>
> This smells a bit like c++ new/delete behavior to me.
>
Hmmm... can glibc be clued in to do some hysteresis on the memory
allocation?
-hpa
Martin Pool
>
> On Thu, 7 Apr 2005, Martin Pool wrote:
> >
> > Importing the first snapshot (2004-01-01) took 41.77s user, 1:23.79
> > total. Each subsequent day takes about 10s user, 30s elapsed to commit
> > into bzr. The speeds are comparable to CVS or a bit faster, and may be
> > faster than other distributed systems. (This on a laptop with a 5400rpm
> > disk.) Pulling out a complete copy of the tree as it was on a previous
> > date takes about 14 user, 60s elapsed.
>
> If you have an exportable tree, can you just make it pseudo-public, tell
> me where to get a buildable system that works well enough, point me to
> some documentation, and maybe I can get some feel for it?
Hi,
There is a "stable" release here:
http://www.bazaar-ng.org/pkg/bzr-0.0.3.tgz
All you should need to do is unpack that and symlink bzr onto your path.
You can get the current bzr development tree, stored in itself, by
rsync:
rsync -av ozlabs.org::mbp/bzr/dev ~/bzr.dev
Inside that directory you can run 'bzr info', 'bzr status --all', 'bzr
unknowns', 'bzr log', 'bzr ignored'.
Repeated rsyncs will bring you up to date with what I've done -- and
will of course overwrite any local changes.
If someone was going to development on this then the method would
typically be to have two copies of the tree, one tracking my version and
another for your own work -- much as with bk. In your own tree, you can
do 'bzr add', 'bzr remove', 'bzr diff', 'bzr commit'.
At the moment all you can do is diff against the previous revision, or
manually diff the two trees, or use quilt, so it is just an archival
system not a full SCM system. In the near future there will be some
code to extract the differences as changesets to be mailed off.
I have done a rough-as-guts import from bkcvs into this, and I can
advertise that when it's on a server that can handle the load.
At a glance this looks very similar to git -- I can go into the
differences and why I did them the other way if you want.
--
Martin
Matthias Urlichs
> 1) GNU Arch/Bazaar. They use the same archive format, simple, have the
> concepts right. It may need some scripts or add ons. When Bazaar-NG is
> ready, it will be able to read the GNU Arch/Bazaar archives so
> switching should be easy.
Plus Bazaar has multiple implementations (C and Python). Plus arch can
trivially export single patches. Plus ... well, you get the idea. ;-)
Linus: Care to share your SCM feature requirement list?
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | sm...@smurf.noris.de
Linus Torvalds
On Fri, 8 Apr 2005, Martin Pool wrote:
>
> You can get the current bzr development tree, stored in itself, by
> rsync:
I was thinking more of an exportable kernel tree in addition to the tool.
The reason I mention that is just that I know several SCM's bog down under
load horribly, so it actually matters what the size of the tree is.
And I'm absolutely _not_ asking you for the 60,000 changesets that are in
the BK tree, I'd be prfectly happy with a 2.6.12-rc2-based one for
testing.
I know I can import things myself, but the reason I ask is because I've
got several SCM's I should check out _and_ I've been spending the last two
days writing my own fallback system so that I don't get screwed if nothing
out there works right now.
Which is why I'd love to hear from people who have actually used various
SCM's with the kernel. There's bound to be people who have already tried.
I've gotten a lot of email of the kind "I love XYZ, you should try it
out", but so far I've not seen anybody say "I've tracked the kernel with
XYZ, and it does ..."
So, this is definitely not a "Martin Pool should do this" kind of issue:
I'd like many people to test out many alternatives, to get a feel for
where they are especially for a project the size of the kernel..
Rogan Dawes
> Followup to: <2005040805...@taniwha.stupidest.org>
> By author: Chris Wedgwood <c...@f00f.org>
> In newsgroup: linux.dev.kernel
>
>>On Thu, Apr 07, 2005 at 09:42:04PM -0700, Linus Torvalds wrote:
>>
>>
>>>Yes. The silly thing is, at least in my local tests it doesn't
>>>actually seem to be _doing_ anything while it's slow (there are no
>>>system calls except for a few memory allocations and
>>>de-allocations). It seems to have some exponential function on the
>>>number of pathnames involved etc.
>>
>>I see lots of brk calls changing the heap size, up, down, up, down,
>>over and over.
>>
>>This smells a bit like c++ new/delete behavior to me.
>>
>
>
> Hmmm... can glibc be clued in to do some hysteresis on the memory
> allocation?
>
> -hpa
Take a look at
http://www.linuxshowcase.org/2001/full_papers/ezolt/ezolt_html/
Abstract
GNU libc's default setting for malloc can cause a significant
performance penalty for applications that use it extensively, such as
Compaq's high performance extended math library, CXML. The default
malloc tuning can cause a significant number of minor page faults, and
result in application performance of only half of the true potential.
This paper describes how to remove the performance penalty using
environmental variables and the method used to discover the cause of the
malloc performance penalty.
Regards,
Rogan
ro...@lug.udel.edu
> quick challenge for you, and any crazy hacker out there: if you want to
> play with something _really_ nasty (but also very _very_ fast), take a
> look at kernel.org:/pub/linux/kernel/people/torvalds/.
Interesting. I like it, with one modification (see below).
> First one to send me the changelog tree of sparse-git (and a tool to
> commit and push/pull further changes) gets a gold star, and an honorable
> mention. I've put a hell of a lot of clues in there (*).
Here's a partial solution. It does depend on a modified version of
cat-file that behaves like cat. I found it easier to have cat-file
just dump the object indicated on stdout. Trivial patch for that is included.
Two scripts are included:
1) makechlog.sh takes an object and generates a ChangeLog file
consisting of all the parents of the given object. It's probably
breakable, but correctly outputs the sparse-git changes when run on
HEAD. Handles multiple parents and breaks cycles.
This adds a line to each object "me <sha1>". This lets a change
identify itself.
It takes 35 seconds to produce all the change history on my box. It
produces a single file named "ChangeLog".
2) chkchlog.sh uses the "me" entries to verify that #1 didn't miss any
parents. It's mostly to prove my solution reasonably correct ::-)
The patch is below, the scripts are attached, and everything is
available here:
http://lug.udel.edu/~ross/git/
Now to see what I come up with for commit, push, and pull...
--
Ross Vandegrift
ro...@lug.udel.edu
"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
--- cat-file.orig.c 2005-04-08 01:53:54.000000000 -0400
+++ cat-file.c 2005-04-08 01:57:51.000000000 -0400
@@ -11,18 +11,11 @@
char type[20];
void *buf;
unsigned long size;
- char template[] = "temp_git_file_XXXXXX";
- int fd;
if (argc != 2 || get_sha1_hex(argv[1], sha1))
usage("cat-file: cat-file <sha1>");
buf = read_sha1_file(sha1, type, &size);
if (!buf)
exit(1);
- fd = mkstemp(template);
- if (fd < 0)
- usage("unable to create tempfile");
- if (write(fd, buf, size) != size)
- strcpy(type, "bad");
- printf("%s: %s\n", template, type);
+ printf ("%s", buf);
}
Daniel Phillips
> Take a look at
> http://www.linuxshowcase.org/2001/full_papers/ezolt/ezolt_html/
>
> Abstract
>
> GNU libc's default setting for malloc can cause a significant
> performance penalty for applications that use it extensively, such as
> Compaq's high performance extended math library, CXML. The default
> malloc tuning can cause a significant number of minor page faults, and
> result in application performance of only half of the true potential.
This does not smell like an n*2 suckage, more like n^something suckage.
Finding the elephant under the rug should not be hard. Profile?
Regards,
Daniel
Andrea Arcangeli
> look at kernel.org:/pub/linux/kernel/people/torvalds/.
Why not to use sql as backend instead of the tree of directories? That solves
userland journaling too (really one still has to be careful to know the
read-committed semantics of sql, which is not obvious stuff, but 99% of
common cases like this one just works safe automatically since all
inserts/delete/update are always atomic).
You can keep the design of your db exactly the same and even the command line
of your script the same, except you won't have deal with the implementation of
it anymore, and the end result may run even faster with proper btrees and you
won't have scalability issues if the directory of hashes fills up, and it'll
get userland journaling, live backups, runtime analyses of your queries with
genetic algorithms (pgsql 8 seems to have it) etc...
I seem to recall there's a way to do delayed commits too, so you won't
be sychronous, but you'll still have journaling. You clearly don't care
to do synchronous writes, all you care about is that the commit is
either committed completely or not committed at all (i.e. not an half
write of the patch that leaves your db corrupt).
Example:
CREATE TABLE patches (
patch BIGSERIAL PRIMARY KEY,
commiter_name VARCHAR(32) NOT NULL CHECK(commiter_name != ''),
commiter_email VARCHAR(32) NOT NULL CHECK(commiter_email != ''),
md5 CHAR(32) NOT NULL CHECK(md5 != ''),
len INTEGER NOT NULL CHECK(len > 0),
UNIQUE(md5, len),
payload BYTEA NOT NULL,
timestamp TIMESTAMP NOT NULL
);
CREATE INDEX patches_md5_index ON patches (md5);
CREATE INDEX patches_timestamp_index ON patches (timestamp);
s/md5/sha1/, no difference.
This will automatically spawn fatal errors if there are hash collisions and it
enforces a bit of checking.
Then you need a few lines of python to insert/lookup. Example for psycopg2:
import pwd, os, socket
[..]
patch = {'commiter_name': pwd.getpwuid(os.getuid())[4],
'commiter_email': pwd.getpwuid(os.getuid())[0] + '@' + socket.getfqdn(),
'md5' : md5.new(data).hexdigest(), 'len' : len(data),
payload : data, 'timestamp' : 'now'}
curs.execute("""INSERT INTO patches
VALUES (%(committer_name)s, %(commiter_email)s,
%(md5)s, %(len)s, %(payload)s, %(timestamp)s)""", patch)
('now' will be evaluated by the sql server, who knows about the time too)
The speed I don't know for sure, but especially with lots of data the sql way
should at least not be significantly slower, pgsql scales with terabytes
without apparent problems (modulo the annoyance of running vacuum once per day
in cron, to avoid internal sequence number overflows after >4 giga
committs, and once per day the analyser too so it learns about your
usage patterns and can optimize the disk format for it).
For sure the python part isn't going to be noticeable, you can still write it
in C if you prefer (it'll clearly run faster if you want to run tons of
inserts for a benchmark), so then everything will run at bare-hardware
speed and there will be no time wasted interpreting bytecode (only the
sql commands have to be interpreted).
The backup should also be tiny (runtime size is going to be somewhat larger due
the more data structure it has, how much larger I don't know). I know for sure
this kind of setup works like a charm on ppc64 (32bit userland), and x86 (32bit
and 64bit userland).
monotone using sqlite sounds a good idea infact (IMHO they could use a real
dbms too, so that you also get parallelism and you could attach another app to
the backing store at the same time or you could run a live backup and to
get all other high end performance features).
If you feel this is too bloated feel free to ignore this email of course! If
instead you'd like to give this a spin, let me know and I can help to
set it up quick (either today or from Monday).
I also like quick dedicated solutions and I was about to write a backing
store with a tree of dirs + hashes similar to yours for a similar
problem, but I give it up while planning the userland journaling part
and even worse the userland fs locking with live backups, when a DBMS
gets everything right including live backups (and it provides async
interface too via sockets). OTOH for this usage journaling and locking
aren't a big issue since you may have the patch to hash by hand to find
any potentially half-corrupted bit after reboot and you probably run it
serially.
About your compression of the data, I don't think you want to do that.
The size of the live image isn't the issue, the issue is the size of the
_backups_ and you want to compress an huge thing (i.e. the tarball of
the cleartext, or the sql cleartext backup), not many tiny patches.
Comparing the size of the repositories isn't interesting, the
interesting thing is to compare the size of the backups.
BTW, this fixed compliation for my system.
--- ./Makefile.orig 2005-04-08 09:07:17.000000000 +0200
+++ ./Makefile 2005-04-08 08:52:35.000000000 +0200
@@ -8,7 +8,7 @@ all: $(PROG)
install: $(PROG)
install $(PROG) $(HOME)/bin/
-LIBS= -lssl
+LIBS= -lssl -lz
init-db: init-db.o
Thanks.
Marcel Lanz
--- git-0.02/Makefile.orig 2005-04-07 23:06:19.000000000 +0200
+++ git-0.02/Makefile 2005-04-08 09:24:28.472672224 +0200
@@ -8,7 +8,7 @@ all: $(PROG)
install: $(PROG)
install $(PROG) $(HOME)/bin/
-LIBS= -lssl
+LIBS= -lssl -lz
init-db: init-db.o
H. Peter Anvin
> On Friday 08 April 2005 03:05, Rogan Dawes wrote:
>
>>Take a look at
>>http://www.linuxshowcase.org/2001/full_papers/ezolt/ezolt_html/
>>
>>Abstract
>>
>>GNU libc's default setting for malloc can cause a significant
>>performance penalty for applications that use it extensively, such as
>>Compaq's high performance extended math library, CXML. The default
>>malloc tuning can cause a significant number of minor page faults, and
>>result in application performance of only half of the true potential.
>
>
> This does not smell like an n*2 suckage, more like n^something suckage.
> Finding the elephant under the rug should not be hard. Profile?
>
Lack of hysteresis can do that, with large swats of memory constantly
being claimed and returned to the system. One way to implement
hysteresis would be based on a decaying peak-based threshold;
unfortunately for optimal performance that requires the C runtime to
have a notion of time, and in extreme cases even be able to do
asynchronous deallocation, but in reality one can probably assume that
the rate of malloc/free is roughly constant over time.
-hpa
Matt Johnston
>
> >
> > I'm playing with monotone right now. Superficially it looks like it
> > has tons of gee-whiz neato stuff... however, it's *agonizingly* slow.
> > I mean glacial. A heavily sedated sloth with no legs is probably
> > faster.
>
> Yes. The silly thing is, at least in my local tests it doesn't actually
> seem to be _doing_ anything while it's slow (there are no system calls
> except for a few memory allocations and de-allocations). It seems to have
> some exponential function on the number of pathnames involved etc.
>
> I'm hoping they can fix it, though. The basic notions do not sound wrong.
That is indeed correct wrt pathnames. The current head of
monotone is a lot better in this regard (the order of 2-3
minutes for "monotone import" on a 2.6 linux untar). The
basic problem is that in the last release (0.17), a huge
amount of sanity checking code was added to ensure that
inconsistent or generally bad revisions can never be
written/received/transmitted.
The focus is now on speeding that up - there's a _lot_ of
low hanging fruit for us to look at.
Matt
Andrea Arcangeli
> I know I can import things myself, but the reason I ask is because I've
> got several SCM's I should check out _and_ I've been spending the last two
> days writing my own fallback system so that I don't get screwed if nothing
> out there works right now.
I tend to like bzr too (and I tend to like too many things ;), but even
if the export of the data would be available it seems still too early in
development to be able to help you this week, it seems to miss any form
of network export too.
> I'd like many people to test out many alternatives, to get a feel for
> where they are especially for a project the size of the kernel..
The huge number of changesets is the crucial point, there are good
distributed SCM already but they are apparently not efficient enough at
handling 60k changesets.
We'd need a regenerated coherent copy of BKCVS to pipe into those SCM to
evaluate how well they scale.
OTOH if your git project already allows storing the data in there,
that looks nice ;). I don't yet fully understand how the algorithms of
the trees are meant to work (I only understand well the backing store
and I tend to prefer DBMS over tree of dirs with hashes). So I've no
idea how it can plug in well for a SCM replacement or how you want to
use it. It seems a kind of fully lockless thing where you can merge from
one tree to the other without locks and where you can make quick diffs.
It looks similar to a diff -ur of two hardlinked trees, except this one
can save a lot of bandwidth to copy with rsync (i.e. hardlinks becomes
worthless after using rsync in the network, but hashes not). Clearly the
DBMS couldn't use the rsync binary to distribute the objects, but a
network protocol could do the same thing rsync does.
Thanks.
Geert Uytterhoeven
> git on sarge
>
> --- git-0.02/Makefile.orig 2005-04-07 23:06:19.000000000 +0200
> +++ git-0.02/Makefile 2005-04-08 09:24:28.472672224 +0200
> @@ -8,7 +8,7 @@ all: $(PROG)
> install: $(PROG)
> install $(PROG) $(HOME)/bin/
>
> -LIBS= -lssl
> +LIBS= -lssl -lz
>
> init-db: init-db.o
>
I found a few more `issues' after adding `-O3 -Wall'.
Most are cosmetic, but the missing return value in remove_file_from_cache() is
a real bug. Hmm, upon closer look the caller uses its return value in a weird
way, so another bug may be hiding in add_file_to_cache().
Caveat: everything is untested, besides compilation ;-)
diff -purN git-0.02.orig/Makefile git-0.02/Makefile
--- git-0.02.orig/Makefile 2005-04-07 23:06:19.000000000 +0200
+++ git-0.02/Makefile 2005-04-08 11:02:02.000000000 +0200
@@ -1,4 +1,4 @@
-CFLAGS=-g
+CFLAGS=-g -O3 -Wall
CC=gcc
PROG=update-cache show-diff init-db write-tree read-tree commit-tree cat-file
@@ -8,7 +8,7 @@ all: $(PROG)
install: $(PROG)
install $(PROG) $(HOME)/bin/
-LIBS= -lssl
+LIBS= -lssl -lz
init-db: init-db.o
diff -purN git-0.02.orig/cat-file.c git-0.02/cat-file.c
--- git-0.02.orig/cat-file.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/cat-file.c 2005-04-08 11:07:28.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
int main(int argc, char **argv)
{
unsigned char sha1[20];
@@ -25,4 +27,5 @@ int main(int argc, char **argv)
if (write(fd, buf, size) != size)
strcpy(type, "bad");
printf("%s: %s\n", template, type);
+ exit(0);
}
diff -purN git-0.02.orig/commit-tree.c git-0.02/commit-tree.c
--- git-0.02.orig/commit-tree.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/commit-tree.c 2005-04-08 11:06:08.000000000 +0200
@@ -6,6 +6,7 @@
#include "cache.h"
#include <pwd.h>
+#include <string.h>
#include <time.h>
#define BLOCKING (1ul << 14)
diff -purN git-0.02.orig/init-db.c git-0.02/init-db.c
--- git-0.02.orig/init-db.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/init-db.c 2005-04-08 11:07:33.000000000 +0200
@@ -5,10 +5,12 @@
*/
#include "cache.h"
+#include <string.h>
+
int main(int argc, char **argv)
{
char *sha1_dir = getenv(DB_ENVIRONMENT), *path;
- int len, i, fd;
+ int len, i;
if (mkdir(".dircache", 0700) < 0) {
perror("unable to create .dircache");
@@ -25,7 +27,7 @@ int main(int argc, char **argv)
if (sha1_dir) {
struct stat st;
if (!stat(sha1_dir, &st) < 0 && S_ISDIR(st.st_mode))
- return;
+ exit(1);
fprintf(stderr, "DB_ENVIRONMENT set to bad directory %s: ", sha1_dir);
}
diff -purN git-0.02.orig/read-cache.c git-0.02/read-cache.c
--- git-0.02.orig/read-cache.c 2005-04-07 23:23:43.000000000 +0200
+++ git-0.02/read-cache.c 2005-04-08 11:07:37.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
const char *sha1_file_directory = NULL;
struct cache_entry **active_cache = NULL;
unsigned int active_nr = 0, active_alloc = 0;
@@ -89,7 +91,7 @@ void * read_sha1_file(unsigned char *sha
z_stream stream;
char buffer[8192];
struct stat st;
- int i, fd, ret, bytes;
+ int fd, ret, bytes;
void *map, *buf;
char *filename = sha1_file_name(sha1);
@@ -173,7 +175,7 @@ int write_sha1_file(char *buf, unsigned
int write_sha1_buffer(unsigned char *sha1, void *buf, unsigned int size)
{
char *filename = sha1_file_name(sha1);
- int i, fd;
+ int fd;
fd = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
if (fd < 0)
diff -purN git-0.02.orig/read-tree.c git-0.02/read-tree.c
--- git-0.02.orig/read-tree.c 2005-04-08 04:58:44.000000000 +0200
+++ git-0.02/read-tree.c 2005-04-08 11:07:41.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
static void create_directories(const char *path)
{
int len = strlen(path);
@@ -72,7 +74,6 @@ static int unpack(unsigned char *sha1)
int main(int argc, char **argv)
{
- int fd;
unsigned char sha1[20];
if (argc != 2)
diff -purN git-0.02.orig/show-diff.c git-0.02/show-diff.c
--- git-0.02.orig/show-diff.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/show-diff.c 2005-04-08 11:07:44.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
#define MTIME_CHANGED 0x0001
#define CTIME_CHANGED 0x0002
#define OWNER_CHANGED 0x0004
@@ -60,7 +62,6 @@ int main(int argc, char **argv)
struct stat st;
struct cache_entry *ce = active_cache[i];
int n, changed;
- unsigned int mode;
unsigned long size;
char type[20];
void *new;
diff -purN git-0.02.orig/update-cache.c git-0.02/update-cache.c
--- git-0.02.orig/update-cache.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/update-cache.c 2005-04-08 11:08:55.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
static int cache_name_compare(const char *name1, int len1, const char *name2, int len2)
{
int len = len1 < len2 ? len1 : len2;
@@ -50,6 +52,7 @@ static int remove_file_from_cache(char *
if (pos < active_nr)
memmove(active_cache + pos, active_cache + pos + 1, (active_nr - pos - 1) * sizeof(struct cache_entry *));
}
+ return 0;
}
static int add_cache_entry(struct cache_entry *ce)
@@ -250,4 +253,5 @@ int main(int argc, char **argv)
return 0;
out:
unlink(".dircache/index.lock");
+ exit(0);
}
diff -purN git-0.02.orig/write-tree.c git-0.02/write-tree.c
--- git-0.02.orig/write-tree.c 2005-04-07 23:15:17.000000000 +0200
+++ git-0.02/write-tree.c 2005-04-08 11:07:51.000000000 +0200
@@ -5,6 +5,8 @@
*/
#include "cache.h"
+#include <string.h>
+
static int check_valid_sha1(unsigned char *sha1)
{
char *filename = sha1_file_name(sha1);
@@ -31,7 +33,7 @@ static int prepend_integer(char *buffer,
int main(int argc, char **argv)
{
- unsigned long size, offset, val;
+ unsigned long size, offset;
int i, entries = read_cache();
char *buffer;
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Catalin Marinas
> I'm playing with monotone right now. Superficially it looks like it
> has tons of gee-whiz neato stuff... however, it's *agonizingly* slow.
> I mean glacial. A heavily sedated sloth with no legs is probably
> faster.
I tried some time ago to import the BKCVS revisions since Linux 2.6.9
into a monotone-0.16 repository. I later tried to upgrade the database
(repository) to monotone version 0.17. The result - converting ~3500
revisions would have taken more than *one year*, fact confirmed by the
monotone developers. The bottleneck seemed to be the big size of the
manifest (which stores the file names and the corresponding SHA1
values) and all the validation performed when converting. The
solution, unsafe, is to disable the revision checks in monotone but
you can end up with an inconsistent repository (haven't tried this).
--
Catalin
Matthias Andree
The length is too optimistic and insufficient to import the current BK
stuff. I'd vote for 64 or at least 48 for each, although 48 is going to
be a tight fit. It costs a bit but considering the expected payload
size it's irrelevant.
Committer (double t) email is up to 36 characters at the moment and the
name up to 43 characters when analyzing the shortlog script with this
little Perl snippet:
------------------------------------------------------------------------
while (($k, $v) = each %addresses) {
$lk = length $k;
$lv = length $v;
if ($lk > $mk) { $mk = $lk; }
if ($lv > $mv) { $mv = $lv; }
}
print "max key len $mk, max val len $mv\n";
------------------------------------------------------------------------
which prints: (key is the email, val the name)
max key len 43, max val len 36
--
Matthias Andree
Florian Weimer
>> commiter_name VARCHAR(32) NOT NULL CHECK(commiter_name != ''),
>> commiter_email VARCHAR(32) NOT NULL CHECK(commiter_email != ''),
>
> The length is too optimistic and insufficient to import the current BK
> stuff. I'd vote for 64 or at least 48 for each, although 48 is going to
> be a tight fit. It costs a bit but considering the expected payload
> size it's irrelevant.
You should also check your database documentation if VARCHAR(n) is
actually implemented implemented in the same way as TEXT (or what the
unbounded string type is called), plus an additional length check. It
doesn't make much sense to use VARCHAR if there isn't a performance
(or disk space) benefit, IMHO, especially for such data.
Linus Torvalds
On Fri, 8 Apr 2005, Andrea Arcangeli wrote:
>
> Why not to use sql as backend instead of the tree of directories?
Because it sucks?
I can come up with millions of ways to slow things down on my own. Please
come up with ways to speed things up instead.
Linus
Linus Torvalds
On Fri, 8 Apr 2005 ro...@jose.lug.udel.edu wrote:
>
> Here's a partial solution. It does depend on a modified version of
> cat-file that behaves like cat. I found it easier to have cat-file
> just dump the object indicated on stdout. Trivial patch for that is included.
Your trivial patch is trivially incorrect, though. First off, some files
may be binary (and definitely are - the "tree" type object contains
pathnames, and in order to avoid having to worry about special characters
they are NUL-terminated), and your modified "cat-file" breaks that.
Secondly, it doesn't check or print the tag.
That said, I think I agree with your concern, and cat-file should not use
a temp-file. I'll fix it, but I'll also make it verify the tag (so you'd
now have to know the tag in advance if you want to cat the data).
Something like
cat-file -t <sha1> # output the tag
cat-file <tag> <sha1> # output the data
or similar. Easy enough. That way you can do
torvalds@ppc970:~/git> ./cat-file -t `cat .dircache/HEAD `
commit
and
torvalds@ppc970:~/git> ./cat-file commit `cat .dircache/HEAD `
tree ca30cdf8df2f31545cc1f2c1be62619111b6f6aa
parent c2474b336d7a96fb4e03e65d229bcddc62b244fc
author Linus Torvalds <torv...@ppc970.osdl.org> Fri Apr 8 08:16:38 2005
committer Linus Torvalds <torv...@ppc970.osdl.org> Fri Apr 8 08:16:38 2005
Make "cat-file" output the file contents to stdout.
New syntax: "cat-file -t <sha1>" shows the tag, while "cat-file <tag> <sha1>"
outputs the file contents after checking that the supplied tag matches.
I'll rsync the .dircache directory to kernel.org. You'll need to update
your scripts.
> Now to see what I come up with for commit, push, and pull...
A "commit" (*) looks roughly like this:
# check with "show-diff" what has changed, and check if
# you need to add any files..
update-cache <list of files that have been changed/added/deleted>
# check with "show-diff" that it all looks right
oldhead=$(cat .dircache/HEAD)
newhead=$(commit-tree $(write-tree) -p $oldhead < commit-message)
# update the head information
if [ "$newhead" != "" ] ; then echo $newhead > .dircache/HEAD; fi
(*) I call this "commit", but it's really something much simpler. It's
really just a "I now have <this directory state>, I got here from
<collection of previous directory states> and the reason was <reason>".
The "push" I use is
rsync -avz --exclude index .dircache/ <destination-dir>
and you can pull the same way, except when you pull you should save _your_
HEAD file first (and then you're screed. There's no way to merge. If
you've made changes and committed them, your changes are still there, but
they are now on a different HEAD than the new one).
That, btw, is kind of the design. "git" really doesn't care about things
like merges. You can use _any_ SCM to do a merge. What "git" does is track
directory state (and how you got to that state), and nothing else. It
doesn't merge, it doesn't really do a whole lot of _anything_.
So when you "pull" or "push" on a git archive, you get the "union" of all
directory states in the destination. The HEAD thing is _one_ pointer into
the "sea of directory states", but you really have to use something else
to merge two directory states together.
Linus
Matthias-Christian Ott
SQL Databases like SQLite aren't slow.
But maybe a Berkeley Database v.4 is a better solution.
Matthias-Christian Ott
Catalin Marinas
> Which is why I'd love to hear from people who have actually used various
> SCM's with the kernel. There's bound to be people who have already
> tried.
I (successfully) tried GNU Arch with the Linux kernel. I mirrored all
the BKCVS changesets since Linux 2.6.9 (5300+ changesets) using this
script:
http://wiki.gnuarch.org/BKCVS_20to_20Arch_20Script_20for_20Linux_20Kernel
My repository size is 1.1GB but this is because the script I use
creates a snapshot (i.e. a full tarball) of every main and -rc
release. For each individual changeset, an arch repository has a
patch-xxx directory with a compressed tarball containing the patch, a
log file and a checksum file.
GNU Arch may have some annoying things (file naming, long commands,
harder to get started, imposed version naming) and I won't try to
advocate them but, for me, it looked like the best (free) option
available regarding both features and speed. Being changeset oriented
also has some advantages from my point of view. Being distributed
means that you can create a branch on your local repository from a
tree stored on a (read-only) remote repository (hosted on an ftp/http
server).
I can't compare it with BK since I haven't used it.
The way I use it:
- a main repository tracking all the changes to the bk-head,
linux--main--2.6 (for those that never read/heard about arch, a tree
name has the form "name--branch--version")
- my main branch from the mainline tree, linux-arm--main--2.6, that
was integrating my patches and was periodically merging the latest
changes in linux--main--2.6
- different linux-arm--platformX--2.6 or linux-arm--deviceX--2.6 trees
that were eventually merged into the linux-arm--main--2.6 tree
The main merge algorithm is called star-merge and does a three-way
merge between the local tree, the remote one and the common ancestor
of these. Cherry picking is also supported for those that like it (I
found it very useful if, for example, I fix a general bug in a branch
that should be integrated in the main tree but the branch is not yet
ready for inclusion).
All the standard commands like commit, diff, status etc. are supported
by arch. A useful command is "missing" which shows what changes are
present in a tree and not in the current one. It is handy to see a
summary of the remote changes before doing a merge (and faster than a
full diff). It also supports file/directory renaming.
To speed things up, arch uses a revision library with a directory for
every revision, the files being hard-linked between revisions to save
space. You can also hard-link the working tree to the revision library
(which speeds the tree diff operation) but you need to make sure that
your editor renames the original file before saving a copy.
Having snapshots might take space but they are useful for both fast
getting a revision and creating a revision in the library.
A diff command takes usually around 1 min (on a P4 at 2.5GHz with IDE
drives) if the current revision is in the library. The tree diff is
the main time consuming operation when committing small changes. If
the revision is not in the library, it will try to create it by
hard-linking with a previous one and applying the corresponding
patches (later version I think can reverse-apply patches from newer
revisions).
The merge operation might take some time (minutes, even 10-20 minutes
for 1000+ changesets) depending on the number of changesets and
whether the revisions are already in the revision library. You can
specify a three-way merge that places conflict markers in the file
(like diff3 or cvs) or a two-way merge which is equivalent to applying
a patch (if you prefer a two-way merge, the "replay" command is
actually the fastest, it takes ~2 seconds to apply a small changeset
and doesn't need go to the revision library). Once a merge operation
completes, you would need to fix the conflicts and commit the
changes. All the logs are preserved but the newly merged individual
changes are seen as a single commit in the local tree.
In the way I use it (with a linux--main--2.6 tree similar to bk-head)
I think arch would get slow with time as changesets accumulate. The
way its developers advise to be used is to work, for example, on a
linux--main--2.6.12 tree for preparing this release and, once it is
ready, seal it (commit --seal). Further commits need to have a --fix
option and they should mainly be bug fixes. At this point you can
branch the linux--main--2.6.13 and start working on it. This new tree
can easily merge the bug fixes applied to the previous version. Arch
developers also recommend to use a new repository every year,
especially if there are many changesets.
A problem I found, even if the library revisions are hard-linked, they
still take a lot of space and should be cleaned periodically (a cron
script that checks the last access to them is available).
By default, arch also complains (with exit) about unknown files in the
working tree. Its developer(s) believe that the compilation should be
done in a different directory. I didn't find this a problem since I
use the same tree to compile for several platforms. Anyway, it can be
configured to ignore them, based on regexp.
I also tried monotone and darcs (since these two, unlike svn, can do
proper merging and preserve the merge history) but arch was by far the
fastest (CVS/RCS are hard to be bitten on speed).
Unfortunately, I can't make my repository public because of IT desk
issues but let me know if you'd like me to benchmark different
operations (or if you'd like a simple list of commands to create your
own).
Hope you find this useful.
--
Catalin
Linus Torvalds
On Fri, 8 Apr 2005, Matthias-Christian Ott wrote:
>
> SQL Databases like SQLite aren't slow.
After applying a patch, I can do a complete "show-diff" on the kernel tree
to see the effect of it in about 0.15 seconds.
Also, I can use rsync to efficiently replicate my database without having
to re-send the whole crap - it only needs to send the new stuff.
You do that with an sql database, and I'll be impressed.
Chris Wedgwood
> After applying a patch, I can do a complete "show-diff" on the kernel tree
> to see the effect of it in about 0.15 seconds.
How does that work? Can you stat the entire tree in that time? I
measure it as being higher than that.
Matthias-Christian Ott
>On Fri, 8 Apr 2005, Matthias-Christian Ott wrote:
>
>
>>SQL Databases like SQLite aren't slow.
>>
>>
>
>After applying a patch, I can do a complete "show-diff" on the kernel tree
>to see the effect of it in about 0.15 seconds.
>
>Also, I can use rsync to efficiently replicate my database without having
>to re-send the whole crap - it only needs to send the new stuff.
>
>You do that with an sql database, and I'll be impressed.
>
> Linus
>
>
>
Ok, but if you want to search for information in such big text files it
slow, because you do linear search -- most datases use faster search
algorithms like hashing and if you have multiple files (I don't if
you're system uses multiple files (like bitkeeper) or not) which need a
system call to be opened this will be very slow, because system calls
itself are slow. And using rsync is also possible because most databases
store their information as plain text with meta information.
Mattthias-Christian Ott
Jon Masters
> So I propose that everybody who is interested, pick one of the above projects
> and join it, to help get it to the point of being able to losslessly import
> the version graph. Given the importance, I think that _all_ viable
> alternatives need to be worked on in parallel, so that two months from now we
> have several viable options.
What about BitKeeper licensing constraints on such involvement?
Jon.
Jeff Garzik
>
> On Fri, 8 Apr 2005, Matthias-Christian Ott wrote:
>
>>SQL Databases like SQLite aren't slow.
>
>
> After applying a patch, I can do a complete "show-diff" on the kernel tree
> to see the effect of it in about 0.15 seconds.
>
> Also, I can use rsync to efficiently replicate my database without having
> to re-send the whole crap - it only needs to send the new stuff.
>
> You do that with an sql database, and I'll be impressed.
Well... it took me over 30 seconds just to 'rm -rf' the unpacked
tarballs of git and sparse-git, over my LAN's NFS.
Granted that this sort of stuff works well with (a) rsync and (b)
hardlinks, but it's still punishment on the i/dcache.
Jeff
Linus Torvalds
On Fri, 8 Apr 2005, Chris Wedgwood wrote:
> On Fri, Apr 08, 2005 at 10:14:22AM -0700, Linus Torvalds wrote:
>
> > After applying a patch, I can do a complete "show-diff" on the kernel tree
> > to see the effect of it in about 0.15 seconds.
>
> How does that work? Can you stat the entire tree in that time? I
> measure it as being higher than that.
I can indeed stat the entire tree in that time (assuming it's in memory,
of course, but my kernel trees are _always_ in memory ;), but in order to
do so, I have to be good at finding the names to stat.
In particular, you have to be extremely careful. You need to make sure
that you don't stat anything you don't need to. We're not talking just
blindly recursing the tree here, and that's exactly the point. You have to
know what you're doing, but the whole point of keeping track of directory
contents is that dammit, that's your whole job.
Anybody who can't list the files they work on _instantly_ is doing
something damn wrong.
"git" is really trivial, written in four days. Most of that was not
actually spent coding, but thinking about the data structures.
Linus
Chris Wedgwood
> I can indeed stat the entire tree in that time (assuming it's in memory,
> of course, but my kernel trees are _always_ in memory ;), but in order to
> do so, I have to be good at finding the names to stat.
<pause ... tapity tap>
I just tested this (I wanted to be sure you didn't have some 47GHz
LiHe cooled Xeon or something).
On my somewhat slowish machine[1] (by today's standards anyhow) I can
stat a checked out tree (ie. the source files and not SCM files) in
about 0.10s it seems and 0.26s for an entire tree with BK files in it.
> In particular, you have to be extremely careful. You need to make
> sure that you don't stat anything you don't need to.
Actually, I could probably make this *much* still faster with a
caveat. Given that my editor when I write a file will write a
temporary file and rename it, for files in directories where nlink==2
I can check chat first and skip the stat of the individual files.
And I guess if I was bored I could have my editor or some daemon
sitting in the background intelligently using dnotify to have this
information on-hand more or less instantly. For this purpose though
that seems like a lot of effort for no real gain right now.
> Anybody who can't list the files they work on _instantly_ is doing
> something damn wrong.
Well, I do like to do "bk sfiles -x" fairly often. But then again I
can stat dirs and compare against a cache to make that fast too.
[1] Dual AthlonMP 2200
Linus Torvalds
On Fri, 8 Apr 2005, Matthias-Christian Ott wrote:
>
> Ok, but if you want to search for information in such big text files it
> slow, because you do linear search
No I don't. I don't search for _anything_. I have my own
content-addressable filesystem, and I guarantee you that it's faster than
mysql, because it depends on the kernel doing the right thing (which it
does).
I never do a single "readdir". It's all direct data lookup, no "searching"
anywhere.
Databases aren't magical. Quite the reverse. They easily end up being
_slower_ than doing it by hand, simply because they have to solve a much
more generic issue. If you design your data structures and abstractions
right, a database is pretty much guaranteed to only incur overhead.
The advantage of a database is the abstraction and management it gives
you. But I did my own special-case abstraction in git.
Yeah, I bet "git" might suck if your OS sucks. I definitely depend on name
caching at an OS level so that I know that opening a file is fast. In
other words, there _is_ an indexing and caching database in there, and
it's called the Linux VFS layer and the dentry cache.
The proof is in the pudding. git is designed for _one_ thing, and one
thing only: tracking a series of directory states in a way that can be
replicated. It's very very fast at that. A database with a more flexible
abstraction migt be faster at other things, but the fact is, you do take a
hit.
The problem with databases are:
- they are damn hard to just replicate wildly and without control. The
database backing file inherently has a lot of internal state. You may
be able to "just copy it", but you have to copy the whole damn thing.
In "git", the data is all there in immutable blobs that you can just
rsync. In fact, you don't even need rsync: you can just look at the
filenames, and anything new you copy. No need for any fancy "read the
files to see that they match". They _will_ match, or you can tell
immediately that a file is corrupt.
Look at this:
torvalds@ppc970:~/git> sha1sum .dircache/objects/e7/bfaadd5d2331123663a8f14a26604a3cdcb678
e7bfaadd5d2331123663a8f14a26604a3cdcb678 .dircache/objects/e7/bfaadd5d2331123663a8f14a26604a3cdcb678
see a pattern anywhere? Imagine that you know the list of files you
have, and the list of files the other side has (never mind the
contents), and how _easy_ it is to synchronize. Without ever having to
even read the remote files that you know you already have.
How do you replicate your database incrementally? I've given you enough
clues to do it for "git" in probably five lines of perl.
- they tend to take time to set up and prime.
In contrast, the filesystem is always there. Sure, you effectively have
to "prime" that one too, but the thing is, if your OS is doing its job,
you basically only need to prime it once per reboot. No need to prime
it for each process you start or play games with connecting to servers
etc. It's just there. Always.
So if you think of the filesystem as a database, you're all set. If you
design your data structure so that there is just one index, you make that
the name, and the kernel will do all the O(1) hashed lookups etc for you.
You do have to limit yourself in some ways.
Oh, and you have to be willing to waste diskspace. "git" is _not_
space-efficient. The good news is that it is cache-friendly, since it is
also designed to never actually look at any old files that aren't part of
the immediate history, so while it wastes diskspace, it does not waste the
(much more precious) page cache.
IOW big file-sets are always bad for performance if you need to traverse
them to get anywhere, but if you index things so that you only read the
stuff you really really _need_ (which git does), big file-sets are just an
excuse to buy a new disk ;)
Linus
Jon Smirl
> How do you replicate your database incrementally? I've given you enough
> clues to do it for "git" in probably five lines of perl.
Efficient database replication is achieved by copying the transaction
logs and then replaying them. Most mid to high end databases support
this. You only need to copy the parts of the logs that you don't
already have.
--
Jon Smirl
jons...@gmail.com
Linus Torvalds
On Fri, 8 Apr 2005, Jeff Garzik wrote:
>
> Well... it took me over 30 seconds just to 'rm -rf' the unpacked
> tarballs of git and sparse-git, over my LAN's NFS.
Don't use NFS for development. It sucks for BK too.
That said, normal _use_ should actually be pretty efficient even over NFS.
It will "stat" a hell of a lot of files to do the "show-diff", but that
part you really can't avoid unless you depend on all the tools marking
their changes somewhere. Which BK does, actually, but that was pretty
painful, and means that bk needed to re-implement all the normal ops (ie
"bk patch").
What's also nice is that exactly because "git" depends on totally
immutable files, they actually cache very well over NFS. Even if you were
to share a database across machines (which is _not_ what git is meant to
do, but it's certainly possible).
So I actually suspect that if you actually _work_ with a tree in "git",
you will find performance very good indeed. The fact that it creates a
number of files when you pull in a new repository is a different thing.
> Granted that this sort of stuff works well with (a) rsync and (b)
> hardlinks, but it's still punishment on the i/dcache.
Actually, it's not. Not once it is set up. Exactly because "git" doesn't
actually access those files unless it literally needs the data in one
file, and then it's always set up so that it needs either none or _all_ of
the file. There is no data sharing anywhere, so you are never in the
situation where it needs "ten bytes from file X" and "25 bytes from file
Y".
For example, if you don't have any changes in your tree, there is exactly
_one_ file that a "show-diff" will read: the .dircache/index file. That's
it. After that, it will "stat()" exactly the files you are tracking, and
nothing more. It will not touch any internal "git" data AT ALL. That
"stat" will be somewhat expensive unless your client caches stat data too,
but that's it.
And if it turns out that you have changed a file (or even just touched it,
so that the data is the same, but the index file can no longer guarantee
it with just a single "stat()"), then git will open have exactly _one_
file (no searching, no messing around), which contains absolutely nothing
except for the compressed (and SHA1-signed) old contents of the file. It
obviously _has_ to do that, because in order to know whether you've
changed it, it needs to now compare it to the original.
IOW, "git" will literally touch the minimum IO necessary, and absolutely
minimum cache-footprint.
The fact is, when tracking the 17,000 files in the kernel directory, most
of them are never actually changed. They literally are "free". They aren't
brought into the cache by "git" - not the file itself, not the backing
store. You set up the index file once, and you never ever touch them
again. You could put the sha1 files on a tape, for all git cares.
The one exception obviously being when you actually instantiate the git
archive for the first time (or when you throw it away). At that time you
do touch all of the data, but that should be the only time.
THAT is what git is good at. It optimized for the "not a lot of changes"
things, and pretty much all the operations are O(n) in the "size of
change", not in "size of repo".
That includes even things like "give me the diff between the top of tree
and the tree 10 days ago". If you know what your head was 10 days ago,
"git" will open exactly _four_ small files for this operation (the current
"top" commit, the commit file of ten days ago, and the two "tree" files
associated with those). It will then need to open the backing store files
for the files that are different between the two versions, but IT WILL
NEVER EVEN LOOK at the files that it immediately sees are the same.
And that's actually true whether we're talking about the top-of-tree or
not. If I had the kernel history in git format (I don't - I estimate that
it would be about 1.5GB - 2GB in size, and would take me about ten days to
extract from BK ;), I could do a diff between _any_ tagged version (and I
mention "tagged" only as a way to look up the commit ID - it doesn't have
to be tagged if you know it some other way) in O(n) where 'n' is the
number of files that have changed between the revisions.
Number of changesets doesn't matter. Number of files doesn't matter. The
_only_ thing that matters is the size of the change.
Btw, I don't actually have a git command to do this yet. A bit of
scripting required to do it, but it's pretty trivial: you open the two
"commit" files that are the beginning/end of the thing, you look up what
the tree state was at each point, you open up the two tree files involved,
and you ignore all entries that match.
Since the tree files are already sorted, that "ignoring matches" is
basically free (technically that's O(n) in the number of files described,
but we're talking about something that even a slow machine can do so fast
you probably can't even time it with a stop-watch). You now have the
complete list of files that have been changed (removed, added or "exists
in both trees, but different contents"), and you can thus trivially create
the whole tree with opening up _only_ the indexes for those files.
Ergo: O(n) in size of change. Both in work and in disk/cache access (where
the latter is often the more important one). Absolutely _zero_ indirection
anywhere apart from the initial stage to go from "commit" to "tree", so
there's no seeking except to actually read the files once you know what
they are (and since you know them up-front and there are no dependencies
at that point, you could even tell the OS to prefetch them if you really
cared about getting minimal disk seeks).
Linus
Chris Wedgwood
> Don't use NFS for development. It sucks for BK too.
Some times NFS is unavoidable.
In the best case (see previous email wrt to only stat'ing the parent
directories when you can) for a current kernel though you can get away
with 894 stats --- over NFS that would probably be tolerable.
After claiming such an optimization is probably not worth while I'm
now thinking for network filesystems it might be.
Florian Weimer
> On Apr 8, 2005 2:14 PM, Linus Torvalds <torv...@osdl.org> wrote:
>> How do you replicate your database incrementally? I've given you enough
>> clues to do it for "git" in probably five lines of perl.
>
> Efficient database replication is achieved by copying the transaction
> logs and then replaying them.
Works only if the databases are in sync. Even if the transaction logs
are pretty high-level, you risk violating constraints specified by the
application. General multi-master replication is an unsolved problem.