New (now current development process)
Paolo Ciarrocchi
I would like to write a short article about the new development
process discussed during last Linux Kernel Developers' Summit for my
local LUG.
Since I'm not able to find an accurate report of what has been
discussed during that meeting I try to summariza what is my
understanding of the current process:
The are two kind of releases, 2.6.x kerneles and 2.6.x.y
2.6.x.y are maintained by the "stable" team (stable at kernel dot
org), are released amost every week.
Here is the latest ( I hope ) announce I found in the archive
including an explanation of how stable development works:
Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
- It must be obviously correct and tested.
- It can not bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a
problem..." type thing.)
- It must fix a problem that causes a build error (but not for things
marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
security issue, or some "oh, that's not good" issue. In short,
something critical.
- No "theoretical race condition" issues, unless an explanation of how
the race can be exploited.
- It can not contain any "trivial" fixes in it (spelling changes,
whitespace cleanups, etc.)
- It must be accepted by the relevant subsystem maintainer.
- It must follow Documentation/SubmittingPatches rules.
Procedure for submitting patches to the -stable tree:
- Send the patch, after verifying that it follows the above rules, to
sta...@kernel.org.
- The sender will receive an ack when the patch has been accepted into
the queue, or a nak if the patch is rejected. This response might
take a few days, according to the developer's schedules.
- If accepted, the patch will be added to the -stable queue, for review
by other developers.
- Security patches should not be sent to this alias, but instead to the
documented secu...@kernel.org.
Review cycle:
- When the -stable maintainers decide for a review cycle, the patches
will be sent to the review committee, and the maintainer of the
affected area of the patch (unless the submitter is the maintainer of
the area) and CC: to the linux-kernel mailing list.
- The review committee has 48 hours in which to ack or nak the patch.
- If the patch is rejected by a member of the committee, or linux-kernel
members object to the patch by bringing up issues that the maintainer
and members did not realize, the patch will be dropped from the
queue.
- At the end of the review cycle, the acked patches will be added to
the latest -stable release, and a new -stable release will happen.
- Security patches will be accepted into the -stable tree directly from
the security kernel team, and not go through the normal review cycle.
Contact the kernel security team for more details on this procedure.
Review committe:
- This will be made up of a number of kernel developers who have
volunteered for this task, and a few that haven't.
2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
are from OSDL) and the development is as follow:
- As soon a new kernel is released a two weeks windows is open, during
this period of time maintainers can submit big diffs to Linus (usually
the patched sitted in -mm kernels for a few weeks). Preferred way to
submit big changes is using GIT ( more information about GIT at
http://git.or.cz/ and
http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
- After two weeks a -rc1 kernel is released and now is possible to
push only patches that do not include new functionalities)
- Aftwer two weeks -rc2 is released
- Process continues until the kernel is considered "ready", the
process should last three months ( 4 kernels per yeard should be
released).
I'm sure I'm missing lot of details so I would really appreciate if
you could support me in writing this article that hopefully will be
usefull outside my LUG as well :-)
Thanks in advance.
--
Paolo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Tony Luck
> 2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
> are from OSDL) and the development is as follow:
(which is used as a testing ground for anything non-trivial before it is
fed into Linus' tree).
> - As soon a new kernel is released a two weeks windows is open, during
> this period of time maintainers can submit big diffs to Linus (usually
> the patched sitted in -mm kernels for a few weeks). Preferred way to
> submit big changes is using GIT ( more information about GIT at
> http://git.or.cz/ and
> http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
> - After two weeks a -rc1 kernel is released and now is possible to
> push only patches that do not include new functionalities)
Initially Linus said that he would only accept e-mail patches after
rc1 ... but common sense prevailed and he later clarified to say that
git merges could still be used, but only to include bug fixes.
Also note that a whole new driver (or filesystem) might be accepted
after -rc1 because there is no risk of causing regressions with
such a change.
> - After two weeks -rc2 is released
There isn't really a hard schedule for when -rcN for N>1 are released.
> - Process continues until the kernel is considered "ready", the
> process should last three months ( 4 kernels per yeard should be
> released).
IIRC the goal was to make a release in around 8 weeks (which would
be closer to six per year). But you have the important part, which is
that Linus doesn't make the release until it is "ready". 2.6.13 was
released on August 28th, and 2.6.14 on October 27th ... so we
appear to have hit the goal this time through.
-Tony
Russell King
> > - Process continues until the kernel is considered "ready", the
> > process should last three months ( 4 kernels per yeard should be
> > released).
> IIRC the goal was to make a release in around 8 weeks (which would
> be closer to six per year). But you have the important part, which is
> that Linus doesn't make the release until it is "ready". 2.6.13 was
> released on August 28th, and 2.6.14 on October 27th ... so we
> appear to have hit the goal this time through.
However, three months is _far_ too long. Look at what is happening
post-2.6.14? There's loads of really huge changes going in which
were backed up over the last two months.
Continuing like this will just push the release of each kernel further
and further out as there's more stuff merged during the mayhem than
can possibly be properly reviewed and tested.
Shorter release cycles means that there's less mayhem, which in turn
means more time to review.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Linus Torvalds
On Sat, 29 Oct 2005, Russell King wrote:
> On Sat, Oct 29, 2005 at 11:57:30AM -0700, Tony Luck wrote:
> > IIRC the goal was to make a release in around 8 weeks (which would
> > be closer to six per year). But you have the important part, which is
> > that Linus doesn't make the release until it is "ready". 2.6.13 was
> > released on August 28th, and 2.6.14 on October 27th ... so we
> > appear to have hit the goal this time through.
>
> However, three months is _far_ too long. Look at what is happening
> post-2.6.14? There's loads of really huge changes going in which
> were backed up over the last two months.
Yes. Three months would be too much. I considered even 2.6.14 to be
delayed by at least one week. So 8 weeks is certainly better than it used
to be, but I think 6 weeks should be the goal-post.
But to hit that, we'd might need to shrink the "merge window" from two
weeks to just one, otherwise there's not enough time to calm down. With
most of the real development happening during the previous calm-down
stage, that might not be impossible: we'd only have one week for merges,
but that isn't the week when development actually happens, so who cares?
Everything said, I think 2.6.13->14 worked well enough, even if it's hard
to say how well a process works after one release. Considering that 2.6.13
had the painful PCI changes (you may not have noticed too much, since they
were x86 only) and there were some potentially painful SCSI changes in the
.14 early merges, so it's not like 13->14 was an "easy" release - so the
process certainly _seems_ to be workable.
So I'm planning on continuing with it unchanged for now. Two-week merge
window until -rc1, and then another -rc kernel roughly every week until
release. With the goal being 6 weeks, and 8 weeks being ok.
I don't think anybody has been really unhappy with this approach? Hmm?
Linus
Akula2
This is my first mail to the Kernel tree:-
> Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> to say how well a process works after one release. Considering that 2.6.13
> had the painful PCI changes (you may not have noticed too much, since they
> were x86 only) and there were some potentially painful SCSI changes in the
> .14 early merges, so it's not like 13->14 was an "easy" release - so the
> process certainly _seems_ to be workable.
Will you please throw more light on the *painful* PCI & SCSI changees?
> I don't think anybody has been really unhappy with this approach? Hmm?
honestly, I did download 14 yesterday and made the kernel ready to
compile. today I've seen 14git1 patch! Am trying to understand why is
it this kind of kernel development model...
thanks,
Akula
Andi Kleen
>
> But to hit that, we'd might need to shrink the "merge window" from two
> weeks to just one, otherwise there's not enough time to calm down. With
Please don't. Even the two weeks are too short IMHO, because it is
hard to digest so much code in such a short time and also it is not
always easy for maintainers to hit such short time windows for sending
patches.
> I don't think anybody has been really unhappy with this approach? Hmm?
The long freeze periods were nothing much happens are painful. It
would be better to have some more overlap of merging and stabilizing
(stable does that already kind of, but not enough)
-Andi
Russell King
> Linus Torvalds <torv...@osdl.org> writes:
> > I don't think anybody has been really unhappy with this approach? Hmm?
>
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)
Violently agree. I find the long freeze periods painful and very very
very boring, to the point of looking for other stuff to do (such as
cleaning up bits of the kernel and queuing mega-patches for the next
round of merging.)
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Greg KH
> Hi Linus,
>
> This is my first mail to the Kernel tree:-
>
> > Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> > to say how well a process works after one release. Considering that 2.6.13
> > had the painful PCI changes (you may not have noticed too much, since they
> > were x86 only) and there were some potentially painful SCSI changes in the
> > .14 early merges, so it's not like 13->14 was an "easy" release - so the
> > process certainly _seems_ to be workable.
>
> Will you please throw more light on the *painful* PCI & SCSI changees?
If it worked for you, it wasn't painful :)
But for PCI, we changed the way we did resource allocation for devices,
which caused a lot of odd problems on some machines.
thanks,
greg k-h
Jesper Juhl
> I would like to write a short article about the new development
> process discussed during last Linux Kernel Developers' Summit for my
> local LUG.
>
> Since I'm not able to find an accurate report of what has been
> discussed during that meeting I try to summariza what is my
> understanding of the current process:
>
> The are two kind of releases, 2.6.x kerneles and 2.6.x.y
>
apply patches for them. In that document I also give a description of
the various trees.
Perhaps that document will be useful to you. You can find it in a
recent kernel source tree as Documentation/applying-patches.txt or you
can read it online via lrx at this URL :
http://sosdg.org/~coywolf/lxr/source/Documentation/applying-patches.txt
Good luck on your article. Once you are done with it I'd love to read
it and I guess a lot of other people would too, so please supply an
URL to it at that time :-)
--
Jesper Juhl <jespe...@gmail.com>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Tony Luck
> But to hit that, we'd might need to shrink the "merge window" from two
> weeks to just one, otherwise there's not enough time to calm down.
The problem with just one week to merge would be if other stuff
happened that ate up the merge period (e.g. this coming week I'm
travelling and will spend all day Tuesday and Wednesday in meetings).
It would be easier to manage if I could be sure which week was going
to be the merge window ... but I know that it is hard to predict when
a release will happen. Overall I'd be much happier sticking with a
two week window.
-Tony
Andrew Morton
>
> On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > Linus Torvalds <torv...@osdl.org> writes:
> > > I don't think anybody has been really unhappy with this approach? Hmm?
> >
> > The long freeze periods were nothing much happens are painful. It
> > would be better to have some more overlap of merging and stabilizing
> > (stable does that already kind of, but not enough)
>
> Violently agree. I find the long freeze periods painful and very very
> very boring, to the point of looking for other stuff to do (such as
> cleaning up bits of the kernel and queuing mega-patches for the next
> round of merging.)
The freezes are for fixing bugs, especially recent regressions. There's no
shortage of them, you know.
I you can think of a better way to get kernel developers off their butts
and actually fixing bugs, I'm all ears.
Theodore Ts'o
> hard to digest so much code in such a short time and also it is not
> always easy for maintainers to hit such short time windows for sending
> patches.
>
> > I don't think anybody has been really unhappy with this approach? Hmm?
>
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)
I thought Andrew was accepting patches targeted at 2.6.n+1 into the
-mm tree during the freeze periods, yes? If so, why would it be a
case of "nothing much happens"? Nothing much might be happening in
Linus's git tree, but that doesn't that they can't be happening in
Andrew's -mm patchsets....
- Ted
Russell King
> Russell King <rmk+...@arm.linux.org.uk> wrote:
> >
> > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > Linus Torvalds <torv...@osdl.org> writes:
> > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > >
> > > The long freeze periods were nothing much happens are painful. It
> > > would be better to have some more overlap of merging and stabilizing
> > > (stable does that already kind of, but not enough)
> >
> > Violently agree. I find the long freeze periods painful and very very
> > very boring, to the point of looking for other stuff to do (such as
> > cleaning up bits of the kernel and queuing mega-patches for the next
> > round of merging.)
>
> The freezes are for fixing bugs, especially recent regressions.
Given my stated low activity during the -rc periods, well, you draw
your conclusion from that.
> There's no shortage of them, you know.
Please let me know when there's something in my area regresses.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Andrew Morton
>
> On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > Russell King <rmk+...@arm.linux.org.uk> wrote:
> > >
> > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > Linus Torvalds <torv...@osdl.org> writes:
> > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > >
> > > > The long freeze periods were nothing much happens are painful. It
> > > > would be better to have some more overlap of merging and stabilizing
> > > > (stable does that already kind of, but not enough)
> > >
> > > Violently agree. I find the long freeze periods painful and very very
> > > very boring, to the point of looking for other stuff to do (such as
> > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > round of merging.)
> >
> > The freezes are for fixing bugs, especially recent regressions.
>
> Given my stated low activity during the -rc periods, well, you draw
> your conclusion from that.
>
> > There's no shortage of them, you know.
>
> Please let me know when there's something in my area regresses.
>
a) you're sitting around feeling very very very bored while
b) the kernel is in long freeze due to the lack of kernel developer
attention to known bugs
The solution seems fairly obvious to me?
Russell King
> Russell King <rmk+...@arm.linux.org.uk> wrote:
> >
> > On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > > Russell King <rmk+...@arm.linux.org.uk> wrote:
> > > >
> > > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > > Linus Torvalds <torv...@osdl.org> writes:
> > > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > > >
> > > > > The long freeze periods were nothing much happens are painful. It
> > > > > would be better to have some more overlap of merging and stabilizing
> > > > > (stable does that already kind of, but not enough)
> > > >
> > > > Violently agree. I find the long freeze periods painful and very very
> > > > very boring, to the point of looking for other stuff to do (such as
> > > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > > round of merging.)
> > >
> > > The freezes are for fixing bugs, especially recent regressions.
> >
> > Given my stated low activity during the -rc periods, well, you draw
> > your conclusion from that.
> >
> > > There's no shortage of them, you know.
> >
> > Please let me know when there's something in my area regresses.
> >
>
> a) you're sitting around feeling very very very bored while
>
> b) the kernel is in long freeze due to the lack of kernel developer
> attention to known bugs
>
> The solution seems fairly obvious to me?
That's fine if you have the hardware to be able to debug these issues.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Andrew Morton
>
> That's fine if you have the hardware to be able to debug these issues.
Most driver bugs cannot be reproduced by the developer. Almost by
definition: if the patch had caused problems on the developer's machine, he
wouldn't have shipped it.
This is why we have this wonderful group of long-suffering people who
download and test development kernels for us, and who take the time to
report defects.
Yes, it's painful to get into a long-range debugging session, sending debug
patches, twelve-hour turnaround, etc. But what alternative have we?
Russell King
> Russell King <rmk+...@arm.linux.org.uk> wrote:
> >
> > That's fine if you have the hardware to be able to debug these issues.
>
> Most driver bugs cannot be reproduced by the developer. Almost by
> definition: if the patch had caused problems on the developer's machine, he
> wouldn't have shipped it.
>
> This is why we have this wonderful group of long-suffering people who
> download and test development kernels for us, and who take the time to
> report defects.
>
> Yes, it's painful to get into a long-range debugging session, sending debug
> patches, twelve-hour turnaround, etc. But what alternative have we?
However, it helps if you have a grasp of technologies like ACPI,
IO-APICs, APICs, PCI IRQ routing for x86 problems. Since I don't
work in the x86 field, I have _zero_ knowledge of such things
because they just don't apply when working on ARM.
That makes debugging x86 problems a nightmare. Remember I gave
up with trying to sort out peoples PCMCIA problems?
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Andi Kleen
> I thought Andrew was accepting patches targeted at 2.6.n+1 into the
> -mm tree during the freeze periods, yes? If so, why would it be a
> case of "nothing much happens"? Nothing much might be happening in
> Linus's git tree, but that doesn't that they can't be happening in
> Andrew's -mm patchsets....
The problem is that -mm* contains typically so many more or less
broken changes that any extensive work on there is futile
because you never know whose bugs you're debugging
(and if the patch that is broken will even make it anywhere)
In short mainline is frozen too long and -mm* is too unstable.
-Andi
Andi Kleen
> The freezes are for fixing bugs, especially recent regressions. There's no
> shortage of them, you know.
>
> I you can think of a better way to get kernel developers off their butts
> and actually fixing bugs, I'm all ears.
The problem is that you usually cannot do proper bug fixing because
the release might be just around the corner, so you typically
chose the ugly workaround or revert, or just reject changes for bugs that a
are too risky or the impact too low because there is not enough time to
properly test anymore.
It might work better if we were told when the releases would actually
happen and you don't need to fear that this not quite tested everywhere
bugfix you're about to submit might make it into the gold kernel, breaking
the world for some subset of users.
-Andi
Al Viro
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere)
>
> In short mainline is frozen too long and -mm* is too unstable.
Besides, -mm is changing so fscking fast that it doesn't build on a lot
of configs most of the time. And trying to keep track of it and at least
deal with build breakage at real time is, IME, hopeless.
Russell King
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
>
> > The freezes are for fixing bugs, especially recent regressions. There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
>
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
>
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Indeed - a prime example is the bootmem initialisation problem. Had
I known on 1st October that the final release wouldn't have been for
a long time, I'd have applied that patch and you wouldn't have had
that problem with ARM relying on the bootmem initialisation order
clashing with your ia64 (or x86_64) swiotlb fix.
But stupid rmk - again I hasten to add - was suckered into this "-rc
is frozen" idea again and decided it wasn't appropriate to submit it.
In hind-sight, there would have been plenty of time to sort out any
issues.
Hell, we held up 2.6.14 for swiotlb _anyway_.
Time to start writing those lines...
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core
Andrew Morton
>
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
>
> > The freezes are for fixing bugs, especially recent regressions. There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
>
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
There is absolutely nothing preventing people from working on bugs at any
time! It's not as if a bugfix can ever come too early.
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Nobody knows when a kernel will be released, because it's released
according to perceived bug status, not according to a preconceived
timeline.
I just don't buy what you're saying. Unless the kernel is at -rc3 or -rc4,
we *know* the release is weeks away - it's always been thus.
Andrew Morton
>
> > It might work better if we were told when the releases would actually
> > happen and you don't need to fear that this not quite tested everywhere
> > bugfix you're about to submit might make it into the gold kernel, breaking
> > the world for some subset of users.
>
> Indeed - a prime example is the bootmem initialisation problem.
No, I'd say that was an *exception*, not a "prime example".
I've filed away 322 unresolved bug reports here. The great majority are
busted drivers on random hardware dating back as far as 2.6.11. Many of
them are regressions.
There is nothing stopping anyone from working with the originators to get
these things fixed up at any time.
Why is it necessary for me to chase maintainers to get their bugs fixed?
Why are maintainers working on new features when they have unresolved bugs?
Why is it so often me who has to do the followup for un-responded-to bug
reports against subsystems which I don't know anything about?
(Those are rhetorical questions, btw).
Andi Kleen
> There is nothing stopping anyone from working with the originators to get
> these things fixed up at any time.
>
> Why is it necessary for me to chase maintainers to get their bugs fixed?
>
> Why are maintainers working on new features when they have unresolved bugs?
Because zero bugs is just unrealistic and they would never get anything done
if that was the requirement?
(especially considering that a lot of the bugs at least on x86-64 are
hardware/firmware bugs of some sort, so often it is not really a linux
bug but just a missing ha^w^wwork^w^w^w^wfix for something else)
I agree regressions are a problem and need to be addressed, but handling all
non regressions on a non trivial platforms is just impossible IMHO...
Perhaps it would be nice to have better bug classification: e.g.
regression/new hardware/reported by more than one person etc. I think
with some prioritization like that it would be much easier to keep the bugs
under control.
Sometimes bugs are less important than others.
e.g. on x86-64 the bootmem issue was obscure at best, affecting
only a very small part of the user base. Sure the few people hit by it
will be annoyed, but trying to make everyone happy is impossible so
one has to try to just make the majority of users happy. So it was imho
ok to just revert the patch to fix ARM again and not breaking IA64 (I cannot
assess how many users were on ARM/IA64 affected) for the next release and
try to sort it out later.
Regressions are important, everything else has to be prioritized based on the
impact on the user base (and this doesn't necessarily mean the most noisy
part of the userbase)
Perhaps some people could volunteer to set some flags in bugzilla for obvious
things, like regression or new hardware or missing basic information or for
really old kernel and no report for a new one and that could be used to
filter the queries better. Should be an relatively easy task.
-Andi
Paul Jackson
> > because you never know whose bugs you're debugging
> > (and if the patch that is broken will even make it anywhere)
> >
> > In short mainline is frozen too long and -mm* is too unstable.
>
> Besides, -mm is changing so fscking fast that it doesn't build on a lot
> of configs most of the time.
I think you are exagerating.
It builds on most configs most of the time in my experience. If I
haven't tried a crosstool rebuild of the several defconfig arch's in a
week, I might expect one of the less popular archs to drop out, usually
for something so easy even I can figure some sort of fix or workaround.
It's tough to get stable kernels if the only kernels people want to
test on are stable kernels. Linux is benefiting immensely from its
rapid evolution in various directions.
Granted - build and boot tested versions of *-mm, just a couple days
behind Andrew's patch set releases, would save people of dealing with
some of the simply stupid breakage.
Is there any corporate Linux supporter able to fund that? Clearing
off the simple breakage could help attract the more serious and
diverse testing of a wider group of users with various specialized
interests.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <p...@sgi.com> 1.925.600.0401
Al Viro
> I think you are exagerating.
>
> It builds on most configs most of the time in my experience. If I
> haven't tried a crosstool rebuild of the several defconfig arch's in a
> week, I might expect one of the less popular archs to drop out, usually
> for something so easy even I can figure some sort of fix or workaround.
Try allmodconfig for a change... I'm doing that for mainline on a regular
basis and even that turns into considerable amount of time. I have tried
that for -mm and had to give up.
Rob Landley
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere)
>
> In short mainline is frozen too long and -mm* is too unstable.
Are you implying that if mainline wasn't frozen so much, it would still be
more stable than -mm?
Rob
Rob Landley
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
>
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.
Hence the -mm tree, which takes stuff that may still need to be debugged.
Except that it has this nasty habit of taking stuff which still needs to be
debugged from people _other_than_you_, which screws you up. You seem to want
a tree where the only stuff likely to break is your stuff, which is another
popular option: maintaining your own developer tree. Getting people to _use_
such a tree takes a bit of work, but that's not news to anybody.
Think about what you're asking for here. Imagine that other people _also_ get
what you're asking for, at the same time. Is it still what you want?
Right now patches go from developer tree, to -mm tree, to -linus tree, with a
larger audience each time. The _reason_ linus's tree has a larger audience
is exactly _because_ the patches in it have had more testing so it's less
likely to break. And the releases have a way larger audience than Linus's
-rc releases, and the distro kernels have a larger audience than that...
> -Andi
Rob
Andrew Morton
>
> On Monday 31 October 2005 02:22, Andrew Morton wrote:
>
> > There is nothing stopping anyone from working with the originators to get
> > these things fixed up at any time.
> >
> > Why is it necessary for me to chase maintainers to get their bugs fixed?
> >
> > Why are maintainers working on new features when they have unresolved bugs?
>
> Because zero bugs is just unrealistic and they would never get anything done
> if that was the requirement?
>
> (especially considering that a lot of the bugs at least on x86-64 are
> hardware/firmware bugs of some sort, so often it is not really a linux
> bug but just a missing ha^w^wwork^w^w^w^wfix for something else)
>
> I agree regressions are a problem and need to be addressed, but handling all
> non regressions on a non trivial platforms is just impossible IMHO...
>
> Perhaps it would be nice to have better bug classification: e.g.
> regression/new hardware/reported by more than one person etc. I think
> with some prioritization like that it would be much easier to keep the bugs
> under control.
Well sure. But who does that? It should be the relevant maintainer, IMO.
If that's the way in which he chooses to work. Expecting some third party
to do this on a kenrel-wide basis won't fly.
> ...
> Sometimes bugs are less important than others.
I don't believe that what we're seeing is some prioritisation process. The
problem is that some bugs are *harder* than others, and people are just
ducking things which cannot be locally reproduced.
Andrew Morton
>
> You seem to want
> a tree where the only stuff likely to break is your stuff
That's what I was thinking ;)
The simple fact is that we have more developers doing more stuff faster
than they used to. All within a coupled system which has a lot of
interactions.
End result: yes, we do all need to spend more time looking at other
people's code and less time looking at our own. That's just life in a
large project.
I'm very careful to make sure that relevant developers are copied on
patches which go into -mm. In fact there's significantly better review
opportunity on patches which go developer->mm->Linus than there are on
patches which go developer->maintainer-git->Linus.
But the cc'ed people just _have_ to take time out to read the dang patch!
They almost always have multiple weeks in which to do this. But if they
just delete the thing while they work on their own stuff, well...
Paul Jackson
> Try allmodconfig for a change...
;). That's ok. What I do keeps me busy enough.
Still, it's too bad one of Linux's corporate friends can't find a way
to do this sort of thing, providing rapid turn around on build-blessed
*-mm versions.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <p...@sgi.com> 1.925.600.0401
Zwane Mwaikambo
> Perhaps some people could volunteer to set some flags in bugzilla for obvious
> things, like regression or new hardware or missing basic information or for
> really old kernel and no report for a new one and that could be used to
> filter the queries better. Should be an relatively easy task.
I don't think we need any special flags, we just need more people paying
attention to it. It doesn't take that much time to go through pending bugs
and trying to identify real ones, but the problem is that we need
knowledgeable people to do this.
Andrew Morton
>
> On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> > I think you are exagerating.
> >
> > It builds on most configs most of the time in my experience. If I
> > haven't tried a crosstool rebuild of the several defconfig arch's in a
> > week, I might expect one of the less popular archs to drop out, usually
> > for something so easy even I can figure some sort of fix or workaround.
>
> Try allmodconfig for a change... I'm doing that for mainline on a regular
> basis and even that turns into considerable amount of time. I have tried
> that for -mm and had to give up.
fud. Every -mm release is built with allmodconfig on x86 and on x86_64.
It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
arm and ia64. It'