current git kernel has strange problems during bisect
Christian Borntraeger
git bisect start
git bisect good a3a798c
git bisect bad v2.6.29-rc1
results in a repository without several files, e.g Makefile!
git describe also fails.
Any ideas how to fix this problem to continue with my bisect?
Christian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Johannes Schindelin
On Sun, 11 Jan 2009, Christian Borntraeger wrote:
> Am Sonntag 11 Januar 2009 schrieb Christian Borntraeger:
> > doing a
> > git bisect start
> > git bisect good a3a798c
> > git bisect bad v2.6.29-rc1
> >
> > results in a repository without several files, e.g Makefile!
> > git describe also fails.
>
> In fact, retesting with a clean repository shows, that there are only btrfs
> files - nothing else.
>
> Linus did you pull a broken btrfs repository?
I guess it is a subtree merge. So no, nothing went wrong
Use "git bisect skip" to skip over those.
Hth,
Dscho
Christian Borntraeger
Am Sonntag 11 Januar 2009 schrieben Sie:
> Hi,
>
> On Sun, 11 Jan 2009, Christian Borntraeger wrote:
>
> > Am Sonntag 11 Januar 2009 schrieb Christian Borntraeger:
> > > doing a
> > > git bisect start
> > > git bisect good a3a798c
> > > git bisect bad v2.6.29-rc1
> > >
> > > results in a repository without several files, e.g Makefile!
> > > git describe also fails.
> >
> > In fact, retesting with a clean repository shows, that there are only
btrfs
> > files - nothing else.
> >
> > Linus did you pull a broken btrfs repository?
>
> I guess it is a subtree merge. So no, nothing went wrong
>
> Use "git bisect skip" to skip over those.
I think we should really avoid merging subtrees to the linux kernel. It makes
bisecting a real PITA.
Furthermore, It is unlikely, but what if the problem is part of the 581
changesets from btrfs?
Christian
Boaz Harrosh
> Am Sonntag 11 Januar 2009 schrieben Sie:
>> Hi,
>>
>> On Sun, 11 Jan 2009, Christian Borntraeger wrote:
>>
>>> Am Sonntag 11 Januar 2009 schrieb Christian Borntraeger:
>>>> doing a
>>>> git bisect start
>>>> git bisect good a3a798c
>>>> git bisect bad v2.6.29-rc1
>>>>
>>>> results in a repository without several files, e.g Makefile!
>>>> git describe also fails.
>>> In fact, retesting with a clean repository shows, that there are only
> btrfs
>>> files - nothing else.
>>>
>>> Linus did you pull a broken btrfs repository?
>> I guess it is a subtree merge. So no, nothing went wrong
>>
>> Use "git bisect skip" to skip over those.
>
> I think we should really avoid merging subtrees to the linux kernel. It makes
> bisecting a real PITA.
Should is too soft, we cannot. What if it changes mainline files, they will
not have common ancestry. And also the sub-tree checkout un-checkout will take
ages. Chris must have merged with his subtree with a rebase not a merge. I suspect
Linus git tree will have to rebase. This is the first time I've seen such a merge.
for example see be0e5c097f it has no parent, and so on.
> Furthermore, It is unlikely, but what if the problem is part of the 581
> changesets from btrfs?
>
Exactly
> Christian
> --
Boaz
Linus Torvalds
On Sun, 11 Jan 2009, Christian Borntraeger wrote:
>
> I think we should really avoid merging subtrees to the linux kernel. It
> makes bisecting a real PITA. Furthermore, It is unlikely, but what if
> the problem is part of the 581 changesets from btrfs?
Umm, yes?
The thing is, btrfs was developed as an outside module. There are two
choices: import it with history, or import it without history. The history
is interesting, so importing _with_ it is a much nicer one. But that does
mean that btrfs introduces into the kernel tree the same behaviour we've
had in the git development tree for a long time - multiple root commits,
and "independent" branches that get merged.
It's actually very natural for git, and the btrfs tree actually was
re-done with "git filter-branch" to move all the history so that it is in
fs/btrfs, rather than moving around from the root like the _original_
development was done. So it's not technically a subtree merge, it's a
regular merge with just two different root commits - one for the original
base kernel development, one for the original btrfs kernel development.
For bisect, it's indeed somewhat annoying, and we could have perhaps done
some things a bit differently, but it's about the closest you can get to
"real history" without making the first btrfs merge-point a _total_
disaster.
For bisect purposes, if you know you're not chasing down a btrfs issue,
you can do
git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
where that commit 34353029 is the last one which has _just_ the btrfs
files. The next commit is when it does "Merge Btrfs into fs/btrfs", and
that one has the whole kernel tree again.
Linus
Sam Ravnborg
> For bisect, it's indeed somewhat annoying, and we could have perhaps done
> some things a bit differently, but it's about the closest you can get to
> "real history" without making the first btrfs merge-point a _total_
> disaster.
>
> For bisect purposes, if you know you're not chasing down a btrfs issue,
> you can do
>
> git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
>
> where that commit 34353029 is the last one which has _just_ the btrfs
> files. The next commit is when it does "Merge Btrfs into fs/btrfs", and
> that one has the whole kernel tree again.
The cost of moving this piece of history from one git tree to another
git tree is that we make it harder to debug the kernel for the advanced user
that knows how to do bisect.
It is not like this history would be lost - one just had to look
somewhere else to find it.
That may be a bad pain/benefit ratio - time will tell.
There should be a way to avoid such pain when bisecting without
having to mark a semi-random (for the average person) commit as good.
As in something that is present when the average bisect user pull the tree,
and not something the user has to do afterwards.
Sam
Alexey Zaytsev
>>
>> For bisect, it's indeed somewhat annoying, and we could have perhaps done
>> some things a bit differently, but it's about the closest you can get to
>> "real history" without making the first btrfs merge-point a _total_
>> disaster.
>>
>> For bisect purposes, if you know you're not chasing down a btrfs issue,
>> you can do
>>
>> git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
>>
>> where that commit 34353029 is the last one which has _just_ the btrfs
>> files. The next commit is when it does "Merge Btrfs into fs/btrfs", and
>> that one has the whole kernel tree again.
>
> The cost of moving this piece of history from one git tree to another
> git tree is that we make it harder to debug the kernel for the advanced user
> that knows how to do bisect.
And wasn't is trivial to avoid? Just exporting the commits as
patches and importing them into the kernel tree would preserve
the history, and not break bisection.
Linus Torvalds
On Sun, 11 Jan 2009, Sam Ravnborg wrote:
>
> The cost of moving this piece of history from one git tree to another
> git tree is that we make it harder to debug the kernel for the advanced user
> that knows how to do bisect.
>
> It is not like this history would be lost - one just had to look
> somewhere else to find it.
>
> That may be a bad pain/benefit ratio - time will tell.
Umm. No.
Time is exactly what makes it useful. It will make all the downsides
shrink, and the advantages stay.
> There should be a way to avoid such pain when bisecting without
> having to mark a semi-random (for the average person) commit as good.
Well, you don't actually have to mark that semi-random one as good either.
What you can do is to just mark anything that _only_ contains fs/btrfs as
good. IOW, you don't have to know the magic number - you just have to be
told that "oh, if you only have btrfs files, and you're not actively
bisecting a btrfs bug, just do 'git bisect good' and continue".
Yeah, you'll hit it a few times, but you don't even have to compile things
or boot anything, so it's not actually going to be all that much slower
than just knowing about the magic point either.
So now you can consider yourself told how to solve it. It wasn't that
hard. And the advantage is that we have real history.
Linus
Andi Kleen
>
> For bisect purposes, if you know you're not chasing down a btrfs issue,
> you can do
>
> git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
Could you perhaps add some standard tag for that commit? That
would make it easier than to always find the exact btrfs commit.
Just an idea.
-Andi
Johannes Schindelin
On Sun, 11 Jan 2009, Andi Kleen wrote:
> Linus Torvalds <torv...@linux-foundation.org> writes:
> >
> > For bisect purposes, if you know you're not chasing down a btrfs issue,
> > you can do
> >
> > git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
>
> Could you perhaps add some standard tag for that commit? That
> would make it easier than to always find the exact btrfs commit.
>
> Just an idea.
Well, AFAICT what Linus hinted at is that you do not need such a standard
tag. Indeed, you would only clutter the history with such tags, when it
usually is just a matter of saying "git bisect good" whenever you _know_
you are hitting known-good history.
Ciao,
Dscho
Christian Borntraeger
> Well, you don't actually have to mark that semi-random one as good either.
> What you can do is to just mark anything that _only_ contains fs/btrfs as
> good. IOW, you don't have to know the magic number - you just have to be
> told that "oh, if you only have btrfs files, and you're not actively
> bisecting a btrfs bug, just do 'git bisect good' and continue".
That should work.
<rant>
Still, I am a bit frustrated. During this weekend I reported 2 regressions
(wlan and ata) and I still try to find out why suspend/resume stopped
working. In the meantime I have identified 2 patches (one was already known,
I reported the 2nd to the usb maintainers) after 2.6.28 that caused suspend
to ram regressions. In rc1 S2R was broken again. So I tried bisecting the
third patch - which finally brought me to the btrfs bisect problem.
For me, this was the most annoying merge window ever.
In my opinion we should really avoid subtree merges in the future as a curtesy
to people who do the uncool work of testing, problem tracking and bisecting.
</rant>
Christian
Sam Ravnborg
>
>
> On Sun, 11 Jan 2009, Sam Ravnborg wrote:
> >
> > The cost of moving this piece of history from one git tree to another
> > git tree is that we make it harder to debug the kernel for the advanced user
> > that knows how to do bisect.
> >
> > It is not like this history would be lost - one just had to look
> > somewhere else to find it.
> >
> > That may be a bad pain/benefit ratio - time will tell.
>
> Umm. No.
>
> Time is exactly what makes it useful. It will make all the downsides
> shrink, and the advantages stay.
>
> > There should be a way to avoid such pain when bisecting without
> > having to mark a semi-random (for the average person) commit as good.
>
> Well, you don't actually have to mark that semi-random one as good either.
> What you can do is to just mark anything that _only_ contains fs/btrfs as
> good. IOW, you don't have to know the magic number - you just have to be
> told that "oh, if you only have btrfs files, and you're not actively
> bisecting a btrfs bug, just do 'git bisect good' and continue".
And we lost 24 hours due to timezone differences etc. and maybe
a few testers.
Thats my point.
There are other obvious ways to do this where we keep history in kernel
but do not impact bisect.
And we have one frustrated tester already - so this is not a made up example.
Sam
Alexey Zaytsev
<torv...@linux-foundation.org> wrote:
>
>
> On Sun, 11 Jan 2009, Sam Ravnborg wrote:
>>
>> The cost of moving this piece of history from one git tree to another
>> git tree is that we make it harder to debug the kernel for the advanced user
>> that knows how to do bisect.
>>
>> It is not like this history would be lost - one just had to look
>> somewhere else to find it.
>>
>> That may be a bad pain/benefit ratio - time will tell.
>
> Umm. No.
>
> Time is exactly what makes it useful. It will make all the downsides
> shrink, and the advantages stay.
>
>> There should be a way to avoid such pain when bisecting without
>> having to mark a semi-random (for the average person) commit as good.
>
> Well, you don't actually have to mark that semi-random one as good either.
> What you can do is to just mark anything that _only_ contains fs/btrfs as
> good. IOW, you don't have to know the magic number - you just have to be
> told that "oh, if you only have btrfs files, and you're not actively
> bisecting a btrfs bug, just do 'git bisect good' and continue".
>
> Yeah, you'll hit it a few times, but you don't even have to compile things
> or boot anything, so it's not actually going to be all that much slower
> than just knowing about the magic point either.
But would not such bug avoid being bisected if you blindly
mark btrfs commits as good?
v2.6.29 <-- bad
..
..
..
btrfs stuff <-- mark as good
..
the-real-bug
..
v2.6.28 <-- good
So you hit the btrfs commit, mark it as good, leaving the real bug below,
and the bisection continues, with both sides being actually bad.
Am I missing something?
Daniel Barkalow
> Am Sonntag 11 Januar 2009 schrieb Linus Torvalds:
> > Well, you don't actually have to mark that semi-random one as good either.
> > What you can do is to just mark anything that _only_ contains fs/btrfs as
> > good. IOW, you don't have to know the magic number - you just have to be
> > told that "oh, if you only have btrfs files, and you're not actively
> > bisecting a btrfs bug, just do 'git bisect good' and continue".
>
> That should work.
>
> <rant>
> Still, I am a bit frustrated. During this weekend I reported 2 regressions
> (wlan and ata) and I still try to find out why suspend/resume stopped
> working. In the meantime I have identified 2 patches (one was already known,
> I reported the 2nd to the usb maintainers) after 2.6.28 that caused suspend
> to ram regressions. In rc1 S2R was broken again. So I tried bisecting the
> third patch - which finally brought me to the btrfs bisect problem.
>
> For me, this was the most annoying merge window ever.
>
> In my opinion we should really avoid subtree merges in the future as a curtesy
> to people who do the uncool work of testing, problem tracking and bisecting.
> </rant>
I think hitting a version without the actual kernel source in it should
actually make bisecting easier, not harder; you can say without even
building the kernel that that version doesn't have the problem you're
trying to find, because it doesn't have anything in it.
The alternative to having that part of the tree empty would be to stick in
some kernel version there (probably 2.6.28), and then you'd build and test
2.6.28 again, completely wasting a bunch of time.
Probably the bisect documentation or messages need to make it clear what
you should do when you land on this sort of commit.
-Daniel
*This .sig left intentionally blank*
Sam Ravnborg
> btrfs stuff <-- mark as good
> the-real-bug
> ...
> v2.6.28 <-- good
>
> So you hit the btrfs commit, mark it as good, leaving the real bug below,
> and the bisection continues, with both sides being actually bad.
>
> Am I missing something?
Yep - you miss that people get confused when suddenly they have no kernel source.
Sam
Daniel Barkalow
> ...
> btrfs stuff <-- mark as good
> ...
> the-real-bug
> ...
> v2.6.28 <-- good
>
> So you hit the btrfs commit, mark it as good, leaving the real bug below,
> and the bisection continues, with both sides being actually bad.
>
> Am I missing something?
Yes, there are no kernel bugs below the btrfs stuff, because there's no
kernel at all below the btrfs stuff. The history is actually like:
A -- B -- C -- D -- G
/
F -- E
F and E are the btrfs stuff, while A-D and G are commit containing the
kernel source (D and G also containing btrfs). Marking E as good cuts off
F, but doesn't cut off anything at all on the top line. Of course, if
you're actually debugging a problem with btrfs that you somehow know to
have worked while btrfs was a separate module at so point, you would want
to get into this history (and would build it as a separate module in order
to do so).
-Daniel
*This .sig left intentionally blank*
Pierre Habouzit
> On Sun, Jan 11, 2009 at 22:42, Sam Ravnborg <s...@ravnborg.org> wrote:
> >>
> >> For bisect, it's indeed somewhat annoying, and we could have perhaps done
> >> some things a bit differently, but it's about the closest you can get to
> >> "real history" without making the first btrfs merge-point a _total_
> >> disaster.
> >>
> >> For bisect purposes, if you know you're not chasing down a btrfs issue,
> >> you can do
> >>
> >> git bisect good 34353029534a08e41cfb8be647d734b9ce9ebff8
> >>
> >> where that commit 34353029 is the last one which has _just_ the btrfs
> >> files. The next commit is when it does "Merge Btrfs into fs/btrfs", and
> >> that one has the whole kernel tree again.
> >
> > The cost of moving this piece of history from one git tree to another
> > git tree is that we make it harder to debug the kernel for the advanced user
> > that knows how to do bisect.
>
> And wasn't is trivial to avoid? Just exporting the commits as
> patches and importing them into the kernel tree would preserve
> the history, and not break bisection.
And would have brought a whole history of totally irrelevant stuff that
never exited for real, with probably a lot of non-compiling sub-steps
which would be even worse.
No, the two possible choices were to squash the whole stuff at once, or
do what has been done IMNSHO. People have to grok how to take shortcuts
with git-bisect. I know that git-bisect puts people on the brainless
course of actions where they git-bisect; configure; compile; boot; test;
mark as good/bad and retry. And that's what I sometimes don't like with
it. Because people trust git-bisect too much and forget how to think
right.
--
·O· Pierre Habouzit
··O madc...@debian.org
OOO http://www.madism.org
Christian Couder
Well "git bisect" can be more usefull if it can be fully automated (with git
bisect run) because then you can make the computer do all the boring work.
So I think putting people on the "brainless course of actions" can often be
a good thing.
Anyway it looks to me that this kind of problem could be avoided if one
could "replace" some commits only when bisecting. In this case what could
be done is that one could "replace" the commit where btrfs is merged with
one commit that cuts off the btrfs history. If the merge commit is only
replaced when bisecting, then you get the best of both worlds:
_ when you bisect, you don't see the btrfs history that breaks the kernel
build,
_ when you don't bisect, you see the full real history.
Of course if the bisection process finds out that the "replaced" commit is
the culprit, then you need to understand what this means...
Regards,
Christian.
Christian Couder
>
> Anyway it looks to me that this kind of problem could be avoided if one
> could "replace" some commits only when bisecting. In this case what could
> be done is that one could "replace" the commit where btrfs is merged with
> one commit that cuts off the btrfs history.
By the way, it possible right now to cut off the btrfs history in one's own
repository using a graft. One don't need to wait for me to finish the
replace stuff I am slowly working on. But on the other hand it will have
all the restrictions of the current graft mechanism.
Kyle Moffett
<bornt...@de.ibm.com> wrote:
> In my opinion we should really avoid subtree merges in the future as a curtesy
> to people who do the uncool work of testing, problem tracking and bisecting.
> </rant>
As an alternative, you can relatively easily rewrite the following
independent histories:
A -- B -- C
X -- Y -- Z
To look like this:
A -- B -- C -- X' -- Y' -- Z'
Where X' is (C + sub/dir/X), Y' is (C + sub/dir/Y), etc...
Assuming the following:
"master" branch points to commit C
"child" branch points to commit Z
"${KIDSTART}" is the SHA1 id of commit X
echo "${KIDSTART} $(git rev-parse --verify master)" >>.git/info/grafts
git filter-branch --index-filter 'git read-tree master && git
read-tree --prefix="sub/dir/" "${GIT_COMMIT}"' -- master..child
The one downside is then somebody actually has to *test* those commits
when doing a bisect, even though they did not materially change
anything. The upside is that there isn't any "what the hell just
happened?" when you *do* end up in the newly-created branch.
Cheers,
Kyle Moffett
Andreas Bombe
> On Sun, Jan 11, 2009 at 4:39 PM, Christian Borntraeger
> <bornt...@de.ibm.com> wrote:
> > In my opinion we should really avoid subtree merges in the future as a curtesy
> > to people who do the uncool work of testing, problem tracking and bisecting.
> > </rant>
>
> As an alternative, you can relatively easily rewrite the following
> independent histories:
>
> A -- B -- C
> X -- Y -- Z
>
> To look like this:
>
> A -- B -- C -- X' -- Y' -- Z'
>
> Where X' is (C + sub/dir/X), Y' is (C + sub/dir/Y), etc...
Given that the subtree may have been in development for a long time, it
is almost a certainty that the older commits may compile on A but not
on C. By basing it all on C you create a lot of uncompilable commits
which hurt bisection just as bad. At least with missing kernel sources
it is obvious that an attempt at compilation is futile and a waste of
time.
Kyle Moffett
> On Tue, Jan 13, 2009 at 03:26:09PM -0500, Kyle Moffett wrote:
>> On Sun, Jan 11, 2009 at 4:39 PM, Christian Borntraeger
>> <bornt...@de.ibm.com> wrote:
>> > In my opinion we should really avoid subtree merges in the future as a curtesy
>> > to people who do the uncool work of testing, problem tracking and bisecting.
>> > </rant>
>>
>> As an alternative, you can relatively easily rewrite the following
>> independent histories:
>>
>> A -- B -- C
>> X -- Y -- Z
>>
>> To look like this:
>>
>> A -- B -- C -- X' -- Y' -- Z'
>>
>> Where X' is (C + sub/dir/X), Y' is (C + sub/dir/Y), etc...
>
> Given that the subtree may have been in development for a long time, it
> is almost a certainty that the older commits may compile on A but not
> on C. By basing it all on C you create a lot of uncompilable commits
> which hurt bisection just as bad. At least with missing kernel sources
> it is obvious that an attempt at compilation is futile and a waste of
> time.
No, the older commits will compile just fine as they don't actually
reference the new code from any of the parent makefiles. It would
effectively be "dead code" until the "merge" in the commit *after* Z'
in which you add lines to "sub/Kconfig" and "sub/Kbuild" which
reference "sub/dir/*".
Cheers,
Kyle Moffett