Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

significant file rename patch landed incorrectly

27 views
Skip to first unread message

L. David Baron

unread,
Jun 5, 2010, 1:40:39 PM6/5/10
to dev-tree-...@lists.mozilla.org
Last night, Callek landed
http://hg.mozilla.org/mozilla-central/rev/d8dc49d5bd60 , which
contains a significant number of file renames (it renames most files
in netwerk/), without the correct rename metadata (instead as
adds/removes).

There are various possibilities for how we could try to fix this:
* land the patch correctly and then merge it with mozilla-central
(I have this done locally, but I don't plan to push it right
now). This fixes the output of hg log -f, but does not seem to
fix hg annotate. (Does hg annotate work with renames anyway?)

* land the patch correctly and replay all of current
mozilla-central on top of that, and then do a dummy merge (I
think this is possible but don't know the details) that
essentially marks current m-c as merged into that history taking
all the details from the new history

If we want to do something to fix this, I think it's better to do
something sooner rather than later since merges into another
repository (like tracemonkey) could complicate things.

hg is pretty bad at showing information about renames anyway, so I'm
not even sure how useful fixing it is...

Thoughts?

-David

--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/

Ehsan Akhgari

unread,
Jun 5, 2010, 4:43:23 PM6/5/10
to L. David Baron, dev-tree-...@lists.mozilla.org
On Sat, Jun 5, 2010 at 1:40 PM, L. David Baron <dba...@dbaron.org> wrote:

> * land the patch correctly and replay all of current
> mozilla-central on top of that, and then do a dummy merge (I
> think this is possible but don't know the details) that
> essentially marks current m-c as merged into that history taking
> all the details from the new history
>

I _think_ this should be possible. I'm going to try it locally to see what
happens, and I'll post when I have more information.


--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 5, 2010, 9:34:41 PM6/5/10
to L. David Baron, dev-tree-...@lists.mozilla.org
I ended up doing this, and pushed the results to mozilla-central.

Unfortunately, because hg annotate does not correctly follow up the
chain of revisions for a file, it means that this command will pretend
that our netwerk code became into being since revision d8dc49d5bd60
(see <http://hg.mozilla.org/mozilla-central/annotate/ac1ed3f6b2e7/netwerk/cache/nsCache.cpp>
for example.). On hgweb, revision logs are also broken, but you can
use hg log -f locally to get a correct revision log.

--
Ehsan
<http://ehsanakhgari.org/>

Justin Wood (Callek)

unread,
Jun 5, 2010, 10:30:09 PM6/5/10
to
On 6/5/2010 1:45 PM, L. David Baron wrote:

> On Saturday 2010-06-05 10:40 -0700, L. David Baron wrote:
>> * land the patch correctly and then merge it with mozilla-central
>> (I have this done locally, but I don't plan to push it right
>> now). This fixes the output of hg log -f, but does not seem to
>> fix hg annotate. (Does hg annotate work with renames anyway?)
>
> hg annotate does normally work with renames, so this fixes only half
> the problem. I'm inclined not to land it.

Just to be clear, "I'm very sorry."

Note to others, do not rebase with old HG versions.

(I'll soon be making a blog entry about my mistake).

--
~Justin Wood (Callek)

Boris Zbarsky

unread,
Jun 6, 2010, 10:34:24 AM6/6/10
to
On 6/5/10 9:34 PM, Ehsan Akhgari wrote:
> I ended up doing this, and pushed the results to mozilla-central.
>
> Unfortunately, because hg annotate does not correctly follow up the
> chain of revisions for a file, it means that this command will pretend
> that our netwerk code became into being since revision d8dc49d5bd60

So.. What can we do to get a working annotate here? Not having that is
really a _huge_ problem, especially since we never fixed our hgweb
tooling to handle annotation sanely across the cvs/hg divide. As things
stand, figuring out why a particular piece of necko code is there just
became a _lot_ more time-consuming, and this will interfere in a
significant way with future necko work.

Do push hooks get access to the list of revision ids being pushed? Can
we strip the buggy push (and of course everything that landed on top of
it) from m-c, reland the things that were supposed to be there, and add
a hook that blocks any push containing the buggy revision?

> (see<http://hg.mozilla.org/mozilla-central/annotate/ac1ed3f6b2e7/netwerk/cache/nsCache.cpp>
> for example.). On hgweb, revision logs are also broken, but you can
> use hg log -f locally to get a correct revision log.

Revision log is a heck of a lot less useful than annotate.

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 10:39:19 AM6/6/10
to
On 6/6/10 10:34 AM, Boris Zbarsky wrote:
> Do push hooks get access to the list of revision ids being pushed? Can
> we strip the buggy push (and of course everything that landed on top of
> it) from m-c, reland the things that were supposed to be there, and add
> a hook that blocks any push containing the buggy revision?

Alternately, would merging with the correct add/remove patch "the other
way" (with the first and second parent swapped) have helped with hg
annotate?

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 10:40:19 AM6/6/10
to
On 6/5/10 10:30 PM, Justin Wood (Callek) wrote:
> Note to others, do not rebase with old HG versions.

Is this limited to old hg versions? If so, what constitutes old?

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 10:42:38 AM6/6/10
to
On 6/6/10 8:02 AM, Mike Beltzner wrote:

> On 6/5/2010 10:30 PM, Justin Wood (Callek) wrote:
>> Note to others, do not rebase with old HG versions.
>
> Is there any way we can take this foot-gun away from people?

Can a push hook block a push containing a changeset that has the same
leafname being both added and removed?

-Boris

Benjamin Smedberg

unread,
Jun 6, 2010, 1:31:13 PM6/6/10
to Boris Zbarsky

I haven't read the patch, but it is very likely that this would have helped,
yes.

I am *very* concerned about how this affects the electrolysis merge.

--BDS

Ehsan Akhgari

unread,
Jun 6, 2010, 1:33:44 PM6/6/10
to Boris Zbarsky, Gavin Sharp, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 10:34 AM, Boris Zbarsky <bzba...@mit.edu> wrote:
> On 6/5/10 9:34 PM, Ehsan Akhgari wrote:
>>
>> I ended up doing this, and pushed the results to mozilla-central.
>>
>> Unfortunately, because hg annotate does not correctly follow up the
>> chain of revisions for a file, it means that this command will pretend
>> that our netwerk code became into being since revision d8dc49d5bd60
>
> So..  What can we do to get a working annotate here?  Not having that is
> really a _huge_ problem, especially since we never fixed our hgweb tooling
> to handle annotation sanely across the cvs/hg divide.  As things stand,
> figuring out why a particular piece of necko code is there just became a
> _lot_ more time-consuming, and this will interfere in a significant way with
> future necko work.

Yes, I understand that, and I tried my best to fix the issue, but my
other solution (which was putting the bad changeset on a dormant
branch and closing that branch) didn't work, because it's not possible
to put existing changesets in a new branch. That solution would fix
annotate, but it would cause the default branch to have two heads,
which was not acceptable with our push hooks in place.

> Do push hooks get access to the list of revision ids being pushed?  Can we
> strip the buggy push (and of course everything that landed on top of it)
> from m-c, reland the things that were supposed to be there, and add a hook
> that blocks any push containing the buggy revision?

This was also another suggestion of mine, but Gavin didn't like it.
The problem that this solution creates is that anyone that has pulled
a changeset in the range that we split on the server should also split
the correct range, and pull again. This includes Tinderbox machines,
other project branches, developers, etc.

However, I personally still think that this could be manageable, and I
think that the inconvenience that it would create would only be a
one-time thing, whereas the inconvenience of the current state of
things is an ongoing pain (until hg annotate is fixed, that is.)

But if we decide to do this, we need to do it as soon as possible,
before too many people pull.

And I don't think that we need any special push hooks on the server,
because you can't push a changeset to a repo which doesn't have the
changeset's parent(s), right?

>> (see<http://hg.mozilla.org/mozilla-central/annotate/ac1ed3f6b2e7/netwerk/cache/nsCache.cpp>
>> for example.).  On hgweb, revision logs are also broken, but you can
>> use hg log -f locally to get a correct revision log.
>
> Revision log is a heck of a lot less useful than annotate.

Yes. I was just pointing out the only remaining option.

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 6, 2010, 1:36:54 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org
The merge is done correctly, but hg annotate doesn't correctly
navigate the changeset hierarchy upwards when there are two paths.
Take this for example:

ehsanakhgari:~/moz/src [01:31:34]$ hg log -f netwerk/cache/nsCache.cpp
changeset: 43173:ac1ed3f6b2e7
tag: qparent
parent: 43172:09f273f2c759
parent: 43143:c1162c2cc26b
user: Ehsan Akhgari <eh...@mozilla.com>
date: Sat Jun 05 21:23:26 2010 -0400
summary: Merge resolving the bad rename changeset landed as part
of bug 542222

changeset: 43144:2d90590dabe6
parent: 43112:28086cf6ede8
user: Mitchell Field <mitchel...@live.com.au>
date: Sat Jun 05 21:18:12 2010 -0400
summary: Bug 542222 - Reduce recursion in netwerk makefiles.

changeset: 43113:d8dc49d5bd60
user: Mitchell Field <mitchel...@live.com.au>
date: Wed Jun 02 01:56:00 2010 -0400
summary: Bug 542222 - Reduce recursion in netwerk makefiles. r=biesi

changeset: 29050:96c833c60acb
user: Arpad Borsos <arpad....@googlemail.com>
date: Tue May 26 10:53:15 2009 +0200
summary: Bug 467948 - fix some deprecated conversion from string
constant warnings; r=dbaron

changeset: 1:9b2a99adc05e
user: h...@mozilla.com
date: Thu Mar 22 10:30:00 2007 -0700
summary: Free the (distributed) Lizard! Automatic merge from CVS:
Module mozilla: tag HG_REPO_INITIAL_IMPORT at 22 Mar 2007 10:30 PDT,

If you annotate this file, you'll see all lines coming from
d8dc49d5bd60. But if you navigate the left-hand side of merge
upwards, that revision is not in the path at all.

I think this is a bug in hg annotate, and when it gets fixed, we'll
get correct annotation, because the history of the files is correct in
itself.

--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
>

Ehsan Akhgari

unread,
Jun 6, 2010, 1:38:04 PM6/6/10
to Benjamin Smedberg, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 1:31 PM, Benjamin Smedberg <benj...@smedbergs.us> wrote:
> On 6/6/10 10:39 AM, Boris Zbarsky wrote:
>>
>> On 6/6/10 10:34 AM, Boris Zbarsky wrote:
>>>
>>> Do push hooks get access to the list of revision ids being pushed? Can
>>> we strip the buggy push (and of course everything that landed on top of
>>> it) from m-c, reland the things that were supposed to be there, and add
>>> a hook that blocks any push containing the buggy revision?
>>
>> Alternately, would merging with the correct add/remove patch "the other
>> way" (with the first and second parent swapped) have helped with hg
>> annotate?
>
> I haven't read the patch, but it is very likely that this would have helped,
> yes.

FWIW, I tried merges in both ways, and neither helped.

> I am *very* concerned about how this affects the electrolysis merge.

How come?

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 6, 2010, 2:12:15 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org

I think Callek said he was using 1.2.1...


--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 6, 2010, 2:16:44 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org

Should be very easy. Filed bug 570411 for that.

--
Ehsan
<http://ehsanakhgari.org/>

Boris Zbarsky

unread,
Jun 6, 2010, 2:25:29 PM6/6/10
to
On 6/6/10 1:33 PM, Ehsan Akhgari wrote:
> Yes, I understand that, and I tried my best to fix the issue

Right. I'm not attacking you here!

> This was also another suggestion of mine, but Gavin didn't like it.
> The problem that this solution creates is that anyone that has pulled
> a changeset in the range that we split on the server should also split
> the correct range, and pull again. This includes Tinderbox machines,
> other project branches, developers, etc.
>
> However, I personally still think that this could be manageable

Agreed.

> But if we decide to do this, we need to do it as soon as possible,
> before too many people pull.

Also agreed. Who can make that call?

> And I don't think that we need any special push hooks on the server,
> because you can't push a changeset to a repo which doesn't have the
> changeset's parent(s), right?

But the parent of the original botched commit would still be in, no? Or
would we strip that too?

-Boris

L. David Baron

unread,
Jun 6, 2010, 2:28:07 PM6/6/10
to Benjamin Smedberg, dev-tree-...@lists.mozilla.org
On Sunday 2010-06-06 13:31 -0400, Benjamin Smedberg wrote:
> I am *very* concerned about how this affects the electrolysis merge.

Does hg merging deal well with renames even if they're properly
recorded as renames? The last time I was affected by renames, I
think it didn't, but that was a while ago.

Mike Beltzner

unread,
Jun 6, 2010, 2:49:21 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org
We can all make that call, I think. I trust the people in this thread to know the best way to both fix the problems and communicate the required solution to people who may have pulled over the duration.

cheers,
mike

Benjamin Smedberg

unread,
Jun 6, 2010, 3:01:27 PM6/6/10
to
On 6/6/10 2:28 PM, L. David Baron wrote:
> On Sunday 2010-06-06 13:31 -0400, Benjamin Smedberg wrote:
>> I am *very* concerned about how this affects the electrolysis merge.
>
> Does hg merging deal well with renames even if they're properly
> recorded as renames? The last time I was affected by renames, I
> think it didn't, but that was a while ago.

Yes, merging works well with correctly-done renames.

--BDS

Benjamin Smedberg

unread,
Jun 6, 2010, 3:08:04 PM6/6/10
to Ehsan Akhgari, dev-tree-...@lists.mozilla.org
On 6/6/10 1:38 PM, Ehsan Akhgari wrote:

>> I am *very* concerned about how this affects the electrolysis merge.
>
> How come?

The electrolysis branch contains very significant changes to netwerk/, and
we're trying to converge to get this branch landed in mozilla-central this
week. If `hg merge` doesn't produce correct results here so that we end up
with .rej files, it will be a multi-day undertaking to fix it, or at least
that's what i think from prior experience.

--BDS

Benjamin Smedberg

unread,
Jun 6, 2010, 3:08:04 PM6/6/10
to Ehsan Akhgari, dev-tree-...@lists.mozilla.org
On 6/6/10 1:38 PM, Ehsan Akhgari wrote:

>> I am *very* concerned about how this affects the electrolysis merge.
>
> How come?

The electrolysis branch contains very significant changes to netwerk/, and

L. David Baron

unread,
Jun 6, 2010, 3:49:53 PM6/6/10
to Benjamin Smedberg, Ehsan Akhgari, dev-tree-...@lists.mozilla.org

I think that suggests we should close the tree, strip changesets out
of mozilla-central, and push the correct replay.

Axel Hecht

unread,
Jun 6, 2010, 3:55:23 PM6/6/10
to
I bet that annotate does the same thing as diff, it just takes the first
parent, probably by numeric id. Thus, the parent that's in the tree will
trump any succeeding landing and merge.

Not sure how annotate should resolve two parents, other than throwing dices?

Axel

Ehsan Akhgari

unread,
Jun 6, 2010, 4:23:54 PM6/6/10
to L. David Baron, Benjamin Smedberg, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 2:28 PM, L. David Baron <dba...@dbaron.org> wrote:
> On Sunday 2010-06-06 13:31 -0400, Benjamin Smedberg wrote:
>> I am *very* concerned about how this affects the electrolysis merge.
>
> Does hg merging deal well with renames even if they're properly
> recorded as renames?  The last time I was affected by renames, I
> think it didn't, but that was a while ago.

I'm not sure. But I did a null merge, which means that I kept "my"
changes and threw away "theirs", which should solve the problem. And
I think it does, judging by the fact that hg log -f knows how to
navigate the history across the renames now.

--
Ehsan
<http://ehsanakhgari.org/>

L. David Baron

unread,
Jun 6, 2010, 4:26:45 PM6/6/10
to Benjamin Smedberg, Ehsan Akhgari, dev-tree-...@lists.mozilla.org
On Sunday 2010-06-06 12:49 -0700, L. David Baron wrote:
> On Sunday 2010-06-06 15:08 -0400, Benjamin Smedberg wrote:
> I think that suggests we should close the tree, strip changesets out
> of mozilla-central, and push the correct replay.

Actually, never mind that. I forgot about the issues that would
cause with the pushlog.

I'd *guess* that electrolysis might come out OK if you first merge
with the branch of transplants that Ehsan pushed yesterday (i.e.,
the branch that has the correct rename but not the incorrect one),
and then merge with tip. But I'm not sure.

It might be worth keeping the tree closed until we are.

(It's also worth somebody doing experiments to see if a different
order of merge parents would make annotate work better. I fear
annotate might be based on the order of revisions in the repo rather
than the shape of the graph, though.)

Ehsan Akhgari

unread,
Jun 6, 2010, 4:36:47 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 2:25 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
> On 6/6/10 1:33 PM, Ehsan Akhgari wrote:
>>
>> Yes, I understand that, and I tried my best to fix the issue
>
> Right.  I'm not attacking you here!

Oh, I wasn't trying to imply that! I just wanted to say that I did
whatever I could think of, but others may have better suggestions.
:-)

>> This was also another suggestion of mine, but Gavin didn't like it.
>> The problem that this solution creates is that anyone that has pulled
>> a changeset in the range that we split on the server should also split
>> the correct range, and pull again.  This includes Tinderbox machines,
>> other project branches, developers, etc.
>>
>> However, I personally still think that this could be manageable
>
> Agreed.
>
>> But if we decide to do this, we need to do it as soon as possible,
>> before too many people pull.
>
> Also agreed.  Who can make that call?

Not sure. I don't think this is something that we have predicted as
part of our review/superreview requirements, so I wasn't sure who to
talk to yesterday. All we did yesterday was the result of the
consensus of people on irc at that time, which included me, dbaron,
jesse, and gavin (sorry if I'm forgetting someone.)

>> And I don't think that we need any special push hooks on the server,
>> because you can't push a changeset to a repo which doesn't have the
>> changeset's parent(s), right?
>
> But the parent of the original botched commit would still be in, no?  Or
> would we strip that too?

The parent of the original revision which caused the problem (i.e.,
28086cf6ede8) has two children, so I think we need to strip that too,
in order to be in a sane state afterwards. *All* changesets landed
after it (including d8dc49d5bd60, obviously) should be stripped. This
will mean that everyone should:

1. Check their local repository for revision 28086cf6ede8, using hg
log -r 28086cf6ede8).
2. If they don't have that changeset, they're not affected, they can
hg pull -u happily.
3. If they do have that changeset, they should preserve all their
local changes somehow (for example, by putting them in patch queues
and hg qpop -a), then run hg strip 28086cf6ede8, and then pull -u, and
then re-apply their local changes.
4. Tinderbox machines should be set to do a clobber build, which pulls
mozilla-central from scratch.
5. Nightly build machines are not affected (since they already do
clobber builds).
6. We should go through all the changesets which we're stripping, go
to their respective bugs, and put a comment in them stating the new
revision ID, in case someone needs to find that in future. I think
there are between 30-40 unique changesets in the range right now (not
counting the changesets that I replayed yesterday).
7. The entire thing should be documented somewhere other than this
thread, to give people a reference on what happened.
8. Project branches might possibly need to strip some changesets from
their repos as well, if they've pulled 28086cf6ede8 or any of its
descendants.

--
Ehsan
<http://ehsanakhgari.org/>

Boris Zbarsky

unread,
Jun 6, 2010, 4:37:19 PM6/6/10
to
On 6/6/10 4:26 PM, L. David Baron wrote:
>> I think that suggests we should close the tree, strip changesets out
>> of mozilla-central, and push the correct replay.
>
> Actually, never mind that. I forgot about the issues that would
> cause with the pushlog.

OK. Well, I just closed the tree for now until we sort this out....

It looks like at this point we'd need to strip out Ehsan's merge commit
and everything after it; there seem to be 8 changesets plus a
tracemonkey merge that just happened 20 minutes ago. Wish we'd closed
the tree when we discovered the problem... :(

So the open questions seem to be:

1) Can we sanely merge e10s in?
2) Can we recover sane annotate on the affected files, and how?

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 4:38:48 PM6/6/10
to
On 6/6/10 2:49 PM, Mike Beltzner wrote:
> We can all make that call, I think. I trust the people in this thread to know the best way to both fix the problems and communicate the required solution to people who may have pulled over the duration.

Well, I personally don't have the bits to perform the measures I
proposed (running hg strip on the mozilla-central repository, for
example, or adding push hooks). So at some point when we come to a
decision we'll have to talk to whoever does about doing that...

In any case, seems like the issue of what to do is still open; tree is
closed for now, pending that getting decided.

-Boris

Axel Hecht

unread,
Jun 6, 2010, 4:39:17 PM6/6/10
to
On 06.06.10 21:49, L. David Baron wrote:
> On Sunday 2010-06-06 15:08 -0400, Benjamin Smedberg wrote:
>> On 6/6/10 1:38 PM, Ehsan Akhgari wrote:
>>>> I am *very* concerned about how this affects the electrolysis merge.
>>>
>>> How come?
>>
>> The electrolysis branch contains very significant changes to
>> netwerk/, and we're trying to converge to get this branch landed in
>> mozilla-central this week. If `hg merge` doesn't produce correct
>> results here so that we end up with .rej files, it will be a
>> multi-day undertaking to fix it, or at least that's what i think
>> from prior experience.
>
> I think that suggests we should close the tree, strip changesets out
> of mozilla-central, and push the correct replay.
>
> -David
>

But that changeset is in clones all over the place that people have out
there. I doubt that we'll be successful in keeping that from getting
pushed back in eventually.

Axel

Ehsan Akhgari

unread,
Jun 6, 2010, 4:47:44 PM6/6/10
to dev-tree-...@lists.mozilla.org
Wait!

I may have a fix:

http://pastebin.mozilla.org/731830

Don't know why I couldn't get to work yesterday, but I'm working on
doing the same thing again now...

Ehsan Akhgari

unread,
Jun 6, 2010, 5:51:27 PM6/6/10
to dev-tree-...@lists.mozilla.org
For some reason, I can't get this to work on mozilla-central. No
matter which side I merge on, annotate always sees the bad changeset
as the first one.

--
Ehsan
<http://ehsanakhgari.org/>

Jesse Ruderman

unread,
Jun 6, 2010, 5:59:55 PM6/6/10
to
hg annotate isn't completely lost. you can still ask for an older
version to be annotated:

http://hg.mozilla.org/mozilla-central/annotate/5b3604a3cfbe/netwerk/cache/nsCache.cpp
vs
http://hg.mozilla.org/mozilla-central/annotate/28086cf6ede8/netwerk/cache/src/nsCache.cpp

Could we rely on this until "hg annotate" is fixed, or do we still
want to strip changesets from mozilla-central? (and now tracemonkey)

Zack Weinberg

unread,
Jun 6, 2010, 6:07:31 PM6/6/10
to Ehsan Akhgari, Boris Zbarsky, dev-tree-...@lists.mozilla.org
Ehsan Akhgari <ehsan....@gmail.com> wrote:

> Should be very easy. Filed bug 570411 for that.

I have a closely related but tangential question. I have been trying
to shepherd in some patches submitted by a community member. They
include a couple of file renames (and, unfortunately, simultaneous
modifications to the files) but the rename was not recorded properly so
if I import the patch with mq, I get a delete and an add. What is the
best way to fix this? (I don't know how the contributor generated the
diff.)

zw

Neil

unread,
Jun 6, 2010, 7:03:24 PM6/6/10
to
Zack Weinberg wrote:

>I have been trying to shepherd in some patches submitted by a community member. They include a couple of file renames (and, unfortunately, simultaneous modifications to the files) but the rename was not recorded properly so if I import the patch with mq, I get a delete and an add. What is the best way to fix this?
>

hg import --no-commit
hg rename --after
hg commit

I don't know what the mq equivalents are.

--
Warning: May contain traces of nuts.

Boris Zbarsky

unread,
Jun 6, 2010, 10:32:33 PM6/6/10
to
On 6/6/10 4:36 PM, Ehsan Akhgari wrote:
> The parent of the original revision which caused the problem (i.e.,
> 28086cf6ede8) has two children, so I think we need to strip that too,
> in order to be in a sane state afterwards.

Hmm... I would have thought strip could just remove a given node in the
DAG and all notes reachable from it without affecting anything else.
Maybe I don't understand the setup correctly, though....

> 1. Check their local repository for revision 28086cf6ede8, using hg
> log -r 28086cf6ede8).
> 2. If they don't have that changeset, they're not affected, they can
> hg pull -u happily.
> 3. If they do have that changeset, they should preserve all their
> local changes somehow (for example, by putting them in patch queues
> and hg qpop -a), then run hg strip 28086cf6ede8, and then pull -u, and
> then re-apply their local changes.
> 4. Tinderbox machines should be set to do a clobber build, which pulls
> mozilla-central from scratch.
> 5. Nightly build machines are not affected (since they already do
> clobber builds).
> 6. We should go through all the changesets which we're stripping, go
> to their respective bugs, and put a comment in them stating the new
> revision ID, in case someone needs to find that in future. I think
> there are between 30-40 unique changesets in the range right now (not
> counting the changesets that I replayed yesterday).
> 7. The entire thing should be documented somewhere other than this
> thread, to give people a reference on what happened.
> 8. Project branches might possibly need to strip some changesets from
> their repos as well, if they've pulled 28086cf6ede8 or any of its
> descendants.

Yes, exactly.

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 10:32:59 PM6/6/10
to
On 6/6/10 4:39 PM, Axel Hecht wrote:
> But that changeset is in clones all over the place that people have out
> there. I doubt that we'll be successful in keeping that from getting
> pushed back in eventually.

That's where a push hook checking for that changeset comes in, no?

-Boris

Boris Zbarsky

unread,
Jun 6, 2010, 10:35:26 PM6/6/10
to
On 6/6/10 5:59 PM, Jesse Ruderman wrote:
> hg annotate isn't completely lost. you can still ask for an older
> version to be annotated:
>
> http://hg.mozilla.org/mozilla-central/annotate/5b3604a3cfbe/netwerk/cache/nsCache.cpp
> vs
> http://hg.mozilla.org/mozilla-central/annotate/28086cf6ede8/netwerk/cache/src/nsCache.cpp

Yes, but what workflow will lead you to the correct version and location
to be annotated? The normal workflow is "mxr search files for
nsCache.cpp, then click annotate link". What would happen after that to
give you the second url above? And how long would it take to perform
those steps?

> Could we rely on this until "hg annotate" is fixed

Assuming there's something to be fixed about it, right?

> or do we still want to strip changesets from mozilla-central? (and now tracemonkey)

I doubt we care about stripping this out of tracemonkey. We do want to
strip it from m-c, I think, unless the resulting pushlog confusion is
bad enough to kill the idea.

-Boris

L. David Baron

unread,
Jun 6, 2010, 11:05:28 PM6/6/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Sunday 2010-06-06 22:32 -0400, Boris Zbarsky wrote:
> On 6/6/10 4:36 PM, Ehsan Akhgari wrote:
> >The parent of the original revision which caused the problem (i.e.,
> >28086cf6ede8) has two children, so I think we need to strip that too,
> >in order to be in a sane state afterwards.
>
> Hmm... I would have thought strip could just remove a given node in
> the DAG and all notes reachable from it without affecting anything
> else. Maybe I don't understand the setup correctly, though....

I don't think we would need to strip the parent. Stripping one of
its children would just change it back to having one child.

Ehsan Akhgari

unread,
Jun 7, 2010, 1:55:00 AM6/7/10
to L. David Baron, Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 11:05 PM, L. David Baron <dba...@dbaron.org> wrote:
> On Sunday 2010-06-06 22:32 -0400, Boris Zbarsky wrote:
>> On 6/6/10 4:36 PM, Ehsan Akhgari wrote:
>> >The parent of the original revision which caused the problem (i.e.,
>> >28086cf6ede8) has two children, so I think we need to strip that too,
>> >in order to be in a sane state afterwards.
>>
>> Hmm...  I would have thought strip could just remove a given node in
>> the DAG and all notes reachable from it without affecting anything
>> else. Maybe I don't understand the setup correctly, though....
>
> I don't think we would need to strip the parent.  Stripping one of
> its children would just change it back to having one child.

What worries me is that the two branches created by its children are
later merged with each other, so stripping only one child would
probably strip all its children _and_ the merge revision (as well as
all its children), so we would end up with the other branch until the
merge point. I guess we _could_ replay the rest of the changesets on
top of that new tip, but anyways, the bigger problem is determining
whether stripping this way is safe. Which changeset to strip is not
really the main issue here.

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 7, 2010, 2:00:53 AM6/7/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 10:35 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
> On 6/6/10 5:59 PM, Jesse Ruderman wrote:
>>
>> hg annotate isn't completely lost. you can still ask for an older
>> version to be annotated:
>>
>>
>> http://hg.mozilla.org/mozilla-central/annotate/5b3604a3cfbe/netwerk/cache/nsCache.cpp
>> vs
>>
>> http://hg.mozilla.org/mozilla-central/annotate/28086cf6ede8/netwerk/cache/src/nsCache.cpp
>
> Yes, but what workflow will lead you to the correct version and location to
> be annotated?  The normal workflow is "mxr search files for nsCache.cpp,
> then click annotate link".  What would happen after that to give you the
> second url above?  And how long would it take to perform those steps?

Here's what needs to be done right now.

1. Go to mxr, and click the annotate link.
2. In the annotate page, click on one of the lines representing the
faulty revision.
3. Click the changeset link at the top of the page.
4. In the changeset page, click the parent link to go to the parent revision.
5. In the parent revision, click browse and navigate back to the same file.

It's a major pain. Things will be much worse if part of the thing
that you're huntung down is landed after the faulty changeset. In
that case, you should look at two annotate results, and merge them in
your mind.

>> Could we rely on this until "hg annotate" is fixed
>
> Assuming there's something to be fixed about it, right?

I don't really know how annotate is supposed to work, but it seems to
me that it's broken right now. We have the correct history, and
mercurial knows how to get it (hg log -f) but apparently annotate
works in an entirely different way.

Could we ask one of the mercurial folks to look at the problem? Maybe
they can at least tell us that this is a problem which is going to be
fixed in version X, or it's something that they're not looking at
fixing any time soon?

>> or do we still want to strip changesets from mozilla-central? (and now
>> tracemonkey)
>
> I doubt we care about stripping this out of tracemonkey.  We do want to
> strip it from m-c, I think, unless the resulting pushlog confusion is bad
> enough to kill the idea.

If tracemonkey has the broken changeset (and any of its children),
those need to be stripped too. Otherwise, their next push would fail
(by the hook that we should put in place after stripping.)

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 7, 2010, 2:04:14 AM6/7/10
to Neil, dev-tree-...@lists.mozilla.org

I'm not sure if there are any. What I would do is to pop the mq
patch, hg import --no-commit and hg mv -A as you suggested, qrm the
old patch and qnew -f the new one.

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 7, 2010, 2:01:37 AM6/7/10
to L. David Baron, Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Sun, Jun 6, 2010 at 11:06 PM, L. David Baron <dba...@dbaron.org> wrote:

> On Sunday 2010-06-06 22:35 -0400, Boris Zbarsky wrote:
>> I doubt we care about stripping this out of tracemonkey.  We do want
>> to strip it from m-c, I think, unless the resulting pushlog
>> confusion is bad enough to kill the idea.
>
> The problem that I'm worried about isn't pushlog confusion.  It's
> that all the pushlog code will throw an exception rather than
> returning data, and we'll have no pushlog at all.

Why would that happen? Doesn't pushlog use the information that
mercurial stores about the changesets?

--
Ehsan
<http://ehsanakhgari.org/>

Axel Hecht

unread,
Jun 7, 2010, 4:32:03 AM6/7/10
to

No, it's using a separate database for the data about pushes.

Axel

Axel Hecht

unread,
Jun 7, 2010, 5:07:21 AM6/7/10
to

I looked at the source of annotate, and I think it's actually right.
What it shows is "the latest modification to this line", and the merge
is ignored. Which I think is the right thing.

There are bugs in the code, like
http://mercurial.selenic.com/bts/issue1839, but that doesn't affect us
in our case, as we're merged. That might be affected by
http://mercurial.selenic.com/bts/issue1327.

Anyway, I think stripping changesets is a bad idea, and if I look at the
revisions of nsCache.cpp, it's not a good balance to shake our ecosystem
that badly.

I for example do use copies of repos and repo data in a separate
database to work around some of our limitations here, and updating that
data for a removed changeset is gonna be tedious. No idea who else is
running stuff like that.

Also:

Newer versions of hg are much nicer in exposing multiple parents, I ran
an hg serve locally, it's not all that tricky. It'd be a nice RFE to
have a graph display for the file log on the web. That works locally,
but not on the web afaict.

Axel

Blair McBride

unread,
Jun 7, 2010, 8:17:01 AM6/7/10
to dev-tree-...@lists.mozilla.org
On 7/06/2010 11:49 p.m., Mike Shaver wrote:
>> I think Callek said he was using 1.2.1...
> Isn't that the version we're distributing in MozillaBuild? Oof.

Yep. See bug 557210 for updating it to something newer.

- Blair

Ted Mielczarek

unread,
Jun 7, 2010, 8:50:55 AM6/7/10
to Mike Shaver, dev-tree-...@lists.mozilla.org
On Mon, Jun 7, 2010 at 7:49 AM, Mike Shaver <mike....@gmail.com> wrote:

> On Sun, Jun 6, 2010 at 2:12 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>> I think Callek said he was using 1.2.1...
>
> Isn't that the version we're distributing in MozillaBuild?  Oof.

To be fair, "old" in this context means "15 months old".

http://mercurial.selenic.com/wiki/WhatsNew#Version_1.2.1_-_2009-03-20

I'll get a new MozillaBuild release out soon, there are a few other
things we need to update anyway.

-Ted

Boris Zbarsky

unread,
Jun 7, 2010, 9:25:47 AM6/7/10
to
On 6/7/10 5:07 AM, Axel Hecht wrote:
> Anyway, I think stripping changesets is a bad idea, and if I look at the
> revisions of nsCache.cpp, it's not a good balance to shake our ecosystem
> that badly.

It's not just nsCache.cpp; we've had single-file fails like this before
and let them slide (might have been a mistake; that precommit hook Ehsan
filed a bug on would have saved us here if we'd put it in place then).
But in this case, we're talking about breakage for every single file
under netwerk/ except the toplevel Makefile.in.

-Boris

Boris Zbarsky

unread,
Jun 7, 2010, 9:26:38 AM6/7/10
to
On 6/7/10 1:55 AM, Ehsan Akhgari wrote:
> so stripping only one child would
> probably strip all its children _and_ the merge revision (as well as
> all its children)

Yes, that's what it would have to do.

> I guess we _could_ replay the rest of the changesets on
> top of that new tip

Right.

-Boris

Boris Zbarsky

unread,
Jun 7, 2010, 9:28:12 AM6/7/10
to Robert Sayre
On 6/7/10 2:00 AM, Ehsan Akhgari wrote:
> 1. Go to mxr, and click the annotate link.
> 2. In the annotate page, click on one of the lines representing the
> faulty revision.
> 3. Click the changeset link at the top of the page.
> 4. In the changeset page, click the parent link to go to the parent revision.
> 5. In the parent revision, click browse and navigate back to the same file.

Note the same file, since it got added/removed. You have to navigate to
some other file at some other path, which you have to guess at.

> It's a major pain.

Amen.

> If tracemonkey has the broken changeset (and any of its children),
> those need to be stripped too. Otherwise, their next push would fail
> (by the hook that we should put in place after stripping.)

Tracemonkey doesn't do whole-repo pushes into m-c. They pull m-c
wholesale, but only push cherry-picked changesets. So they should, in
fact, be ok. But probably worth double-checking with sayrer.

-Boris

Neil

unread,
Jun 7, 2010, 9:56:29 AM6/7/10
to
L. David Baron wrote:

>On Sunday 2010-06-06 22:32 -0400, Boris Zbarsky wrote:
>
>
>>On 6/6/10 4:36 PM, Ehsan Akhgari wrote:
>>
>>
>>>The parent of the original revision which caused the problem (i.e., 28086cf6ede8) has two children, so I think we need to strip that too, in order to be in a sane state afterwards.
>>>
>>>
>>Hmm... I would have thought strip could just remove a given node in the DAG and all notes reachable from it without affecting anything else. Maybe I don't understand the setup correctly, though....
>>
>>
>I don't think we would need to strip the parent. Stripping one of its children would just change it back to having one child.
>
>

Assuming this is the approach taken, then am I correct in understanding
that people will fall in to one of the following categories:

1. Automatically destroys and reclones mozilla-central. Unaffected.
2. Not pulled any of the stripped changesets. Unaffected.
3. Pulled but not updated to any of the stripped changesets. Will
have local changesets which hg push will try (and fail) to push
which need to be stripped. (How?)
4. Updated to a stripped changeset, but uses mq or otherwise has no
local changes. Can update -C to the new head. (Or pull -rebase?)
5. Updated to a stripped changeset, and has local changes. Needs to
update -r 28086cf6ede8 then can update to the new head.

L. David Baron

unread,
Jun 7, 2010, 10:53:57 AM6/7/10
to Ehsan Akhgari, Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Monday 2010-06-07 02:01 -0400, Ehsan Akhgari wrote:

> On Sun, Jun 6, 2010 at 11:06 PM, L. David Baron <dba...@dbaron.org> wrote:
> > On Sunday 2010-06-06 22:35 -0400, Boris Zbarsky wrote:
> >> I doubt we care about stripping this out of tracemonkey.  We do want
> >> to strip it from m-c, I think, unless the resulting pushlog
> >> confusion is bad enough to kill the idea.
> >
> > The problem that I'm worried about isn't pushlog confusion.  It's
> > that all the pushlog code will throw an exception rather than
> > returning data, and we'll have no pushlog at all.
>
> Why would that happen? Doesn't pushlog use the information that
> mercurial stores about the changesets?

Pushlog has its own database for the push data; that's not stored in
the changesets. It then expects that the changesets that it has
data about are in the repository so that it can get the rest of the
information about them. I expect that if they're not in the
repository, it will get confused (but I'm not sure exactly how
badly). (I remember it being broken the last time we stripped a
changeset in mozilla-central. In that case, however, the problem
was that the changeset was somehow partially pushed, and we fixed
the problem by stripping the changeset and then re-pushing the exact
same changeset, so once the changeset was re-pushed it was in
mozilla-central again and pushlog was fine.)

Boris Zbarsky

unread,
Jun 7, 2010, 10:55:09 AM6/7/10
to
On 6/7/10 9:56 AM, Neil wrote:
> 3. Pulled but not updated to any of the stripped changesets. Will
> have local changesets which hg push will try (and fail) to push
> which need to be stripped. (How?)

hg strip rev

where "rev" is the broken changeset.

-Boris

Axel Hecht

unread,
Jun 7, 2010, 11:06:08 AM6/7/10
to

Detail that's gtk if we really really go down this route, hg strip is
part of mq. That should be part of the messaging.

Axel

Ehsan Akhgari

unread,
Jun 7, 2010, 11:17:21 AM6/7/10
to Mike Shaver, Boris Zbarsky, dev-tree-...@lists.mozilla.org
I'm not sure, but it's quite possible. We really need to update MozillaBuild much more frequently.

--
Ehsan Akhgari
eh...@mozilla.com

Sent from my Palm PreOn 7 Jun 2010 7:49 a.m., Mike Shaver &lt;mike....@gmail.com&gt; wrote:

On Sun, Jun 6, 2010 at 2:12 PM, Ehsan Akhgari &lt;ehsan....@gmail.com&gt; wrote:

&gt; On Sun, Jun 6, 2010 at 10:40 AM, Boris Zbarsky &lt;bzba...@mit.edu&gt; wrote:

&gt;&gt; On 6/5/10 10:30 PM, Justin Wood (Callek) wrote:

&gt;&gt;&gt;

&gt;&gt;&gt; Note to others, do not rebase with old HG versions.

&gt;&gt;

&gt;&gt; Is this limited to old hg versions? &nbsp;If so, what constitutes old?

&gt;

&gt; I think Callek said he was using 1.2.1...

Isn't that the version we're distributing in MozillaBuild? Oof.

Mike


Mike Beltzner

unread,
Jun 7, 2010, 11:26:22 AM6/7/10
to Ehsan Akhgari, Boris Zbarsky, dev-tree-...@lists.mozilla.org, Mike Shaver
Is there any update that should be given to our community at the Monday project meeting on this topic?

cheers,
mike

Ehsan Akhgari

unread,
Jun 7, 2010, 11:34:06 AM6/7/10
to Neil, dev-tree-...@lists.mozilla.org
On Mon, Jun 7, 2010 at 9:56 AM, Neil <ne...@parkwaycc.co.uk> wrote:
> L. David Baron wrote:
>
>> On Sunday 2010-06-06 22:32 -0400, Boris Zbarsky wrote:
>>
>>>
>>> On 6/6/10 4:36 PM, Ehsan Akhgari wrote:
>>>
>>>>
>>>> The parent of the original revision which caused the problem (i.e.,
>>>> 28086cf6ede8) has two children, so I think we need to strip that too, in
>>>> order to be in a sane state afterwards.
>>>>
>>>
>>> Hmm...  I would have thought strip could just remove a given node in the
>>> DAG and all notes reachable from it without affecting anything else. Maybe I
>>> don't understand the setup correctly, though....
>>>
>>
>> I don't think we would need to strip the parent.  Stripping one of its
>> children would just change it back to having one child.
>>
>
> Assuming this is the approach taken, then am I correct in understanding that
> people will fall in to one of the following categories:
>
>  1. Automatically destroys and reclones mozilla-central. Unaffected.

Right.

>  2. Not pulled any of the stripped changesets. Unaffected.

Right.

>  3. Pulled but not updated to any of the stripped changesets. Will
>     have local changesets which hg push will try (and fail) to push
>     which need to be stripped. (How?)

hg strip bad-rev-id

>  4. Updated to a stripped changeset, but uses mq or otherwise has no
>     local changes. Can update -C to the new head. (Or pull -rebase?)

hg update parent-of-bad-rev-id
hg strip bad-rev-id
hg pull -u

>  5. Updated to a stripped changeset, and has local changes. Needs to
>     update -r 28086cf6ede8 then can update to the new head.

In addition to somehow save their local changes and reapply them later, yes.

--
Ehsan
<http://ehsanakhgari.org/>

Ted Mielczarek

unread,
Jun 7, 2010, 11:38:24 AM6/7/10
to dev-tree-...@lists.mozilla.org
I'd just like to say personally that I think stripping changesets out
of m-c is going to create way more hassle than it's going to solve. It
breaks Mercurial's expectations as well as the expectations of other
parts of our tooling, and generally doesn't play well with distributed
VCS.

-Ted

Ehsan Akhgari

unread,
Jun 7, 2010, 11:38:08 AM6/7/10
to L. David Baron, Boris Zbarsky, dev-tree-...@lists.mozilla.org
On Mon, Jun 7, 2010 at 10:53 AM, L. David Baron <dba...@dbaron.org> wrote:
> On Monday 2010-06-07 02:01 -0400, Ehsan Akhgari wrote:
>> On Sun, Jun 6, 2010 at 11:06 PM, L. David Baron <dba...@dbaron.org> wrote:
>> > On Sunday 2010-06-06 22:35 -0400, Boris Zbarsky wrote:
>> >> I doubt we care about stripping this out of tracemonkey.  We do want
>> >> to strip it from m-c, I think, unless the resulting pushlog
>> >> confusion is bad enough to kill the idea.
>> >
>> > The problem that I'm worried about isn't pushlog confusion.  It's
>> > that all the pushlog code will throw an exception rather than
>> > returning data, and we'll have no pushlog at all.
>>
>> Why would that happen?  Doesn't pushlog use the information that
>> mercurial stores about the changesets?
>
> Pushlog has its own database for the push data; that's not stored in
> the changesets.  It then expects that the changesets that it has
> data about are in the repository so that it can get the rest of the
> information about them.  I expect that if they're not in the
> repository, it will get confused (but I'm not sure exactly how
> badly).  (I remember it being broken the last time we stripped a
> changeset in mozilla-central.  In that case, however, the problem
> was that the changeset was somehow partially pushed, and we fixed
> the problem by stripping the changeset and then re-pushing the exact
> same changeset, so once the changeset was re-pushed it was in
> mozilla-central again and pushlog was fine.)

It should be possible to generate a mapping of old to new changesets
and replace them inside the pushlog db, right?

The only tricky part is mimicking the exact same changeset graph in
the repository after stripping, so that we would have a 1:1 mapping
between old and new changeset IDs. Is it possible to "strip" the
changeset that are going to be stripped in mozilla-central and then
let the new db be built the normal way?

I expect that such modifications would break tools which use use the
pushlog db, such tinderboxpushlog. :(

--
Ehsan
<http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 7, 2010, 11:47:29 AM6/7/10
to Ted Mielczarek, dev-tree-...@lists.mozilla.org
One thing that I would like to point out is that there is still
another possibility: branching off the parent of the bad changeset,
applying the correct changeset, naming the existing branch and closing
it, and replaying (transplanting) all the existing changesets on top
of the bad one into the new branch (which would be the default for
mozilla-central.

I had done this experiment in a user repo on Saturday:

<http://hg.mozilla.org/users/eakhgari_mozilla.com/542222-fixed-repo>

The only downside with this approach is that the default branch would
end up having two heads (e.g.,
http://hg.mozilla.org/users/eakhgari_mozilla.com/542222-fixed-repo/rev/c590aa6c1153
and http://hg.mozilla.org/users/eakhgari_mozilla.com/542222-fixed-repo/rev/fdb1e4bc853d).
We have a hook in place which prevents people from pushing multiple
heads for the default branch, which is probably a good thing. I
_think_ that hook can be modified to make an exception this one time.

The upside of this approach is that we won't have to deal with
stripping anything from mozilla-central, and we'd get the correct
annotate output. The downside of this approach is that
mozilla-central would end up having two heads on the default branch
forever, and people who have updated their clones to any revision
below the bad changeset in the tree would have to hg update -C the
next time (because they will be updating across branches) which
requires them to save their local modification somewhere. And
Tinderboxen should probbaly be set to a clobber build anyway.

What do people think about this approach?

--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
>

Ehsan Akhgari

unread,
Jun 7, 2010, 11:49:24 AM6/7/10
to Mike Beltzner, Boris Zbarsky, dev-tree-...@lists.mozilla.org, Mike Shaver
On Mon, Jun 7, 2010 at 11:26 AM, Mike Beltzner <belt...@mozilla.com> wrote:
> Is there any update that should be given to our community at the Monday project meeting on this topic?

As far as updates go, for now we can only tell them that
mozilla-central is closed, and will remain closed until this issue is
resolved. But as far as I can tell, we don't yet have a solution
which satisfies everyone. But the meeting is in 2 hours, and a lot
can happen in that time!

--
Ehsan
<http://ehsanakhgari.org/>

Boris Zbarsky

unread,
Jun 7, 2010, 11:52:16 AM6/7/10
to

OK, then how do we get usable blame? We've already been drifting down
the path of blame being broken for a few years (e.g. we never hooked up
blame across the hg/cvs divide, which we'd been going to do); it's
getting to the point where looking up blame is hard enough, apparently,
that people commonly put up patches that change code in ways that are
clearly wrong given the blame on the code... and then reviewers end up
having to do the time-consuming blame-hunting to r- the patch.

-Boris

Jeff Muizelaar

unread,
Jun 7, 2010, 12:03:56 PM6/7/10
to Boris Zbarsky, dev-tree-...@lists.mozilla.org

On 6-Jun-10, at 10:34 AM, Boris Zbarsky wrote:

> On 6/5/10 9:34 PM, Ehsan Akhgari wrote:
>> I ended up doing this, and pushed the results to mozilla-central.
>>
>> Unfortunately, because hg annotate does not correctly follow up the
>> chain of revisions for a file, it means that this command will
>> pretend
>> that our netwerk code became into being since revision d8dc49d5bd60
>
> So.. What can we do to get a working annotate here? Not having
> that is really a _huge_ problem, especially since we never fixed our
> hgweb tooling to handle annotation sanely across the cvs/hg divide.
> As things stand, figuring out why a particular piece of necko code
> is there just became a _lot_ more time-consuming, and this will
> interfere in a significant way with future necko work.

For what it's worth, you could use the git mirror at git://
bluishcoder.co.nz/git/mozilla-central.git to work around this problem.
Git tracks renames by content instead of with metadata so it doesn't
have the same problem mercurial does here.

Plus you can graft on the the cvs history from this repository http://people.mozilla.org/~jmuizelaar/mozilla-cvs-history.git
and get annotations back to the beginning of the history. See http://pastebin.mozilla.org/732143

-Jeff


Neil

unread,
Jun 7, 2010, 12:27:08 PM6/7/10
to
Ehsan Akhgari wrote:

>The downside of this approach is that mozilla-central would end up having two heads on the default branch forever, and people who have updated their clones to any revision below the bad changeset in the tree would have to hg update -C the next time (because they will be updating across branches) which requires them to save their local modification somewhere.
>

Can we start by begging everyone to update -r 28086cf6ede8 right now, so
that when we do sort something out they won't have to do anything special?

Ehsan Akhgari

unread,
Jun 7, 2010, 12:50:51 PM6/7/10
to Neil, dev-tree-...@lists.mozilla.org

Sure. This is something that we can announce on the project meeting as well.

--
Ehsan
<http://ehsanakhgari.org/>

Benjamin Smedberg

unread,
Jun 7, 2010, 1:06:29 PM6/7/10
to

I don't think that we can go back in time, now that there has been a
tracemonkey merge. The bad revs have been pulled into TM and it would be
basically impossible to get them fully untangled now.

--BDS

Ehsan Akhgari

unread,
Jun 7, 2010, 1:07:33 PM6/7/10
to Neil, dev-tree-...@lists.mozilla.org
On Mon, Jun 7, 2010 at 12:50 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> On Mon, Jun 7, 2010 at 12:27 PM, Neil <ne...@parkwaycc.co.uk> wrote:
> Sure.  This is something that we can announce on the project meeting as well.

I added this to the project meeting notes:

<https://wiki.mozilla.org/WeeklyUpdates/2010-06-07#Gecko>

--
Ehsan
<http://ehsanakhgari.org/>

Smokey Ardisson

unread,
Jun 7, 2010, 2:11:42 PM6/7/10
to
On Jun 7, 11:52 am, Boris Zbarsky <bzbar...@mit.edu> wrote:

> OK, then how do we get usable blame?  We've already been drifting down
> the path of blame being broken for a few years (e.g. we never hooked up
> blame across the hg/cvs divide, which we'd been going to do); it's

As an aside, was there ever a bug on hooking up blame across the hg/
cvs divide (I've looked, but I can't find one)? This is easily one of
my biggest pain points with the Mozilla-Hg ecosystem, and I know I
don't use Gecko blame anywhere close to as often as Boris must.

Smokey

Benjamin Smedberg

unread,
Jun 7, 2010, 2:21:46 PM6/7/10
to
On 6/7/10 2:11 PM, Smokey Ardisson wrote:

> As an aside, was there ever a bug on hooking up blame across the hg/
> cvs divide (I've looked, but I can't find one)? This is easily one of
> my biggest pain points with the Mozilla-Hg ecosystem, and I know I
> don't use Gecko blame anywhere close to as often as Boris must.

Yes, there was a bug. We have the data necessary to do it, but hooking it up
with hgweb or normal hg annotate is hard, and so we haven't actually done
it. It's pretty trivial to do it with an extension.

--BDS


Axel Hecht

unread,
Jun 7, 2010, 3:08:15 PM6/7/10
to

What I think would really help is a rendition of

hg log -G -f somefile

Sadly, -G and -f (graph and follow) are not compat, at least in 1.5.4.

I'm not sure why, though.

Anyway, there could also be an external tool that does a more
computationally intensive blame computation than what's hg doing now,
detecting duplicate patches and backouts.

That seems to be helpful in more usecases (i.e., backouts), too, much
more than breaking into our revisions and creating hooks that keep
people from doing what hg tries to do by default.

Axel

Axel Hecht

unread,
Jun 7, 2010, 6:13:42 PM6/7/10
to
On 05.06.10 19:45, L. David Baron wrote:
> On Saturday 2010-06-05 10:40 -0700, L. David Baron wrote:
>> * land the patch correctly and then merge it with mozilla-central
>> (I have this done locally, but I don't plan to push it right
>> now). This fixes the output of hg log -f, but does not seem to
>> fix hg annotate. (Does hg annotate work with renames anyway?)
>
> hg annotate does normally work with renames, so this fixes only half
> the problem. I'm inclined not to land it.

I have been pondering about this for a bit more.

Here's my thinking as of now, subject to change on every other beer:

hg annotate as implemented right now gives you:

- When was this line of code changed last?

What folks are really interested here, and what the code doesn't deliver
in this scenario, is

- When was this code seen first?


I'm having a bit of an issue if "this code" is actually per-line, as a
single line might be doing something rather different with respect to
context, but I guess it'd be safe to assume that it's per line.


I'd be surprised if we couldn't hook up a view to hg via an extension
that would deliver that perspective. It'd be a good deal more expensive
than what hg is doing natively, as you don't have to just walk back the
graph until you got all lines covered, but you gotta always walk all
ancestors, and basically compare all lines to each other. Sounds like an
N^2*M problem, with N being the number of lines, M being the number of
revisions.

We got the pushlog hooks linked up, I guess we can get such a view
hooked up.

Axel

Ehsan Akhgari

unread,
Jun 7, 2010, 7:18:34 PM6/7/10
to Axel Hecht, dev-tree-...@lists.mozilla.org

Question is, when can we have such an extension? If it's in a couple
of weeks, then I'd vote for opening up mozilla-central as it is right
now. If it's some indefinite point in the future, then I'd vote to
resolve this problem in a way that existing hg tools can handle it the
way that we want.

Unless we have an owner working on this and it's very high priority on
their list, I doubt that the answer is anything but "some indefinite
point in the future".

--
Ehsan
<http://ehsanakhgari.org/>

Mike Shaver

unread,
Jun 7, 2010, 7:25:13 PM6/7/10
to Ehsan Akhgari, Axel Hecht, dev-tree-...@lists.mozilla.org
On Mon, Jun 7, 2010 at 7:18 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> Question is, when can we have such an extension?  If it's in a couple
> of weeks, then I'd vote for opening up mozilla-central as it is right
> now.  If it's some indefinite point in the future, then I'd vote to
> resolve this problem in a way that existing hg tools can handle it the
> way that we want.
>
> Unless we have an owner working on this and it's very high priority on
> their list, I doubt that the answer is anything but "some indefinite
> point in the future".

I should have an estimate for it tomorrow; I'm hoping that it can get
resolved in the next couple of weeks.

Mike

Axel Hecht

unread,
Jun 7, 2010, 7:46:00 PM6/7/10
to

Here's the code that does it command-line-style for me:

import sys
from mercurial.ui import ui as _ui
from mercurial.hg import repository
from mercurial.node import hex

file = sys.argv[1]
r = repository(_ui())

fctx = r.filectx(file,'default')
lines = fctx.data().splitlines()
_h = fctx.rev()
blame = map(lambda l: (_h, l), lines)
for ancestor in reversed(list(fctx.ancestors())):
alines = ancestor.data().splitlines()
_h = ancestor.rev()
for i, line in enumerate(lines):
if line is None:
continue
if line in alines:
blame[i] = (_h, line)
lines[i] = None

for rev, line in blame:
print rev, line

Axel Hecht

unread,
Jun 7, 2010, 9:24:36 PM6/7/10
to

PS: That code sucks, when it comes down to the "find that code" part,
should be written in terms of mercurial.bdiff.blocks.

Axel

Vladimir Vukicevic

unread,
Jun 7, 2010, 10:21:20 PM6/7/10
to
On Jun 6, 1:36 pm, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> 3. If they do have that changeset, they should preserve all their
> local changes somehow (for example, by putting them in patch queues
> and hg qpop -a), then run hg strip 28086cf6ede8, and then pull -u, and
> then re-apply their local changes.

Can we add a push hook to reject any pushes (with a message pointing
to a wiki page or something) with that changeset? Ideally we'd be
able to maintain a list of 'blacklisted' changsets so that we have a
way to deal with that in the future.

- Vlad

Ehsan Akhgari

unread,
Jun 7, 2010, 11:01:34 PM6/7/10
to Vladimir Vukicevic, dev-tree-...@lists.mozilla.org
Shaver said that he's probably going to have a plan on how to fix
Mercurial to be able to actually handle this case, and if that
happens, we won't need to make any changes to mozilla-central, and the
tree could just be reopened in its current status.

--
Ehsan
<http://ehsanakhgari.org/>

Boris Zbarsky

unread,
Jun 7, 2010, 11:38:56 PM6/7/10
to
On 6/7/10 6:13 PM, Axel Hecht wrote:
> hg annotate as implemented right now gives you:
>
> - When was this line of code changed last?
>
> What folks are really interested here, and what the code doesn't deliver
> in this scenario, is
>
> - When was this code seen first?

For some definition of "code" and "first". Typically what's needed in
your typical vcs archeology trip is figuring out why the code is the way
it is. This might mean "the last substantive change to this code"
(however substantive is defined), or "when the code was added". It
really depends on the situation.

> I'm having a bit of an issue if "this code" is actually per-line, as a
> single line might be doing something rather different with respect to
> context, but I guess it'd be safe to assume that it's per line.

So the question that you're proposing we answer is "what was the first
revision which had this line in the exact state it's in now"?

> I'd be surprised if we couldn't hook up a view to hg via an extension
> that would deliver that perspective. It'd be a good deal more expensive
> than what hg is doing natively, as you don't have to just walk back the
> graph until you got all lines covered, but you gotta always walk all
> ancestors, and basically compare all lines to each other. Sounds like an
> N^2*M problem, with N being the number of lines, M being the number of
> revisions.

That's really going to hurt on some files in our tree
(nsCSSFrameConstructor comes to mind: ~13000 lines, ~1000 revisions)....
And the point is that we don't need this information for the whole
file, typically; just for a small part of it. Does that help any?

-Boris

Axel Hecht

unread,
Jun 8, 2010, 5:19:35 AM6/8/10
to
On 08.06.10 05:38, Boris Zbarsky wrote:
> On 6/7/10 6:13 PM, Axel Hecht wrote:
>> hg annotate as implemented right now gives you:
>>
>> - When was this line of code changed last?
>>
>> What folks are really interested here, and what the code doesn't deliver
>> in this scenario, is
>>
>> - When was this code seen first?
>
> For some definition of "code" and "first". Typically what's needed in
> your typical vcs archeology trip is figuring out why the code is the way
> it is. This might mean "the last substantive change to this code"
> (however substantive is defined), or "when the code was added". It
> really depends on the situation.
>
>> I'm having a bit of an issue if "this code" is actually per-line, as a
>> single line might be doing something rather different with respect to
>> context, but I guess it'd be safe to assume that it's per line.
>
> So the question that you're proposing we answer is "what was the first
> revision which had this line in the exact state it's in now"?

Actually, fixing the code changed that. I'm now actually taking diffs
across all ancestors to the current code, so it's detecting sequences of
lines that showed up like this. That's the same code pattern that hg
annotate uses, just reversed the order, and thus I might walk across
more revisions, I guess.

>> I'd be surprised if we couldn't hook up a view to hg via an extension
>> that would deliver that perspective. It'd be a good deal more expensive
>> than what hg is doing natively, as you don't have to just walk back the
>> graph until you got all lines covered, but you gotta always walk all
>> ancestors, and basically compare all lines to each other. Sounds like an
>> N^2*M problem, with N being the number of lines, M being the number of
>> revisions.
>
> That's really going to hurt on some files in our tree
> (nsCSSFrameConstructor comes to mind: ~13000 lines, ~1000 revisions)....
> And the point is that we don't need this information for the whole file,
> typically; just for a small part of it. Does that help any?
>

It helped to just unsuck the code, luckily. I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=570676, which shows the
difference between what hg annotate does, and what my proposed code
does. It does show a slight slowdown for nsCSSFrameConstructor, but
moreso, it shows a different blame for it, I've put a review request on
that on you, if you don't mind. I guess it's someone with your
understanding of the history of that file than can tell whether that's a
bug or a feature.

Axel

David Bolter

unread,
Jun 8, 2010, 8:47:01 AM6/8/10
to
On Jun 7, 11:01 pm, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> Shaver said that he's probably going to have a plan on how to fix
> Mercurial to be able to actually handle this case, and if that
> happens, we won't need to make any changes to mozilla-central, and the
> tree could just be reopened in its current status.
>

Sounds good. Are we willing to open the tree now?

cheers,
David

>
>
>
>
>
>
> On Mon, Jun 7, 2010 at 10:21 PM, Vladimir Vukicevic <vladim...@gmail.com> wrote:
> > On Jun 6, 1:36 pm, Ehsan Akhgari <ehsan.akhg...@gmail.com> wrote:
> >> 3. If they do have that changeset, they should preserve all their
> >> local changes somehow (for example, by putting them in patch queues
> >> and hg qpop -a), then run hg strip 28086cf6ede8, and then pull -u, and
> >> then re-apply their local changes.
>
> > Can we add a push hook to reject any pushes (with a message pointing
> > to a wiki page or something) with that changeset?  Ideally we'd be
> > able to maintain a list of 'blacklisted' changsets so that we have a
> > way to deal with that in the future.
>
> >    - Vlad
> > _______________________________________________
> > dev-tree-management mailing list

> > dev-tree-managem...@lists.mozilla.org
> >https://lists.mozilla.org/listinfo/dev-tree-management

Mike Beltzner

unread,
Jun 8, 2010, 9:06:25 AM6/8/10
to David Bolter, dev-tree-...@lists.mozilla.org
On 2010-06-08, at 8:47 AM, David Bolter wrote:

> Sounds good. Are we willing to open the tree now?

The tree has now been closed for close to two days. I've seen a bunch of solutions go by, and now we're waiting for a Mercurial fix. I really do not pretend to know the best solution, but it feels like we can just block the bad changesets, put a note on hg that gets sent to committers / pullers, and get back to work?

Can we get a clear answer about who owns the plan, what the plan is, who's driving, or maybe just a bug number tracking the issue, as well as an indication of when the tree is likely to re-open?

cheers,
mike

Mike Shaver

unread,
Jun 8, 2010, 9:28:23 AM6/8/10
to Mike Beltzner, David Bolter, dev-tree-...@lists.mozilla.org
I should have a more detailed ETA on a mercurial fix today. With the proof
of concept from Axel as well, I think we can open the tree now, since "all"
that we're missing is annotate, and we have two different ways to solve that
in short order.

Mike

On Jun 8, 2010 9:06 AM, "Mike Beltzner" <belt...@mozilla.com> wrote:
> On 2010-06-08, at 8:47 AM, David Bolter wrote:
>

>> Sounds good. Are we willing to open the tree now?
>

> The tree has now been closed for close to two days. I've seen a bunch of
solutions go by, and now we're waiting for a Mercurial fix. I really do not
pretend to know the best solution, but it feels like we can just block the
bad changesets, put a note on hg that gets sent to committers / pullers, and
get back to work?
>
> Can we get a clear answer about who owns the plan, what the plan is, who's
driving, or maybe just a bug number tracking the issue, as well as an
indication of when the tree is likely to re-open?
>
> cheers,
> mike

David Bolter

unread,
Jun 8, 2010, 9:38:02 AM6/8/10
to
On Jun 8, 9:28 am, Mike Shaver <mike.sha...@gmail.com> wrote:
> I should have a more detailed ETA on a mercurial fix today. With the proof
> of concept from Axel as well, I think we can open the tree now, since "all"
> that we're missing is annotate, and we have two different ways to solve that
> in short order.

Tree is OPEN.

cheers,
David

>
> Mike


>
> On Jun 8, 2010 9:06 AM, "Mike Beltzner" <beltz...@mozilla.com> wrote:> On 2010-06-08, at 8:47 AM, David Bolter wrote:
>
> >> Sounds good. Are we willing to open the tree now?
>
> > The tree has now been closed for close to two days. I've seen a bunch of
>
> solutions go by, and now we're waiting for a Mercurial fix. I really do not
> pretend to know the best solution, but it feels like we can just block the
> bad changesets, put a note on hg that gets sent to committers / pullers, and
> get back to work?
>
> > Can we get a clear answer about who owns the plan, what the plan is, who's
>
> driving, or maybe just a bug number tracking the issue, as well as an
> indication of when the tree is likely to re-open?
>
>
>
>
>
>
>
>
>
> > cheers,
> > mike
> > _______________________________________________
> > dev-tree-management mailing list

> > dev-tree-managem...@lists.mozilla.org
> >https://lists.mozilla.org/listinfo/dev-tree-management

0 new messages