gclient runhooks updating projects from gyp files is absurdly slow?

[Shawn Singh]

Hi all,

Since yesterday, when I run "gclient sync"  or  "gclient runhooks", it hangs for an absurdly long time when "updating projects from gyp files".  I let it run last night and it seemed to finish in about 35 minutes.  It usually takes less than 20 seconds on my machines.  This seems to be happening on both my mac machine and linux machine.

Is there anything I should check/fix in my build setup?  It was working just yesterday, and I didn't see any discussions that hinted any confuguration change requirements. Did something get updated recently that is causing this?

Thanks

[Nico Weber]

You can patch in
http://codereview.chromium.org/10442058/diff/1/build/gyp_chromium and
then follow the steps in the CL description here
http://codereview.chromium.org/10442058/ to figure out where the
performance goes.

Nico

> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[Shawn Singh]

Thanks Nico - turns out not to be needed so far.  Looks like the script was hanging when it invokes "git svn info", there was a previous discussion on how to fix this, the subject title of that thread was "git cl upload hanging?"

I wonder if this occurred to me because I never migrated to the new git workflow.  I'm migrating now, and most likely the "git svn info" hanging will be fixed after that.

~Shawn

[Scott Graham]

It looks like there's a lot of runs of git svn info via third_party/webrtc/build/version.py. These look pretty expensive even if your git svn setup is working:

F:\src\cr\src>tim c:\src\depot_tools\python_bin\python.exe build\gyp_chromium
Enabled Psyco JIT.
Updating projects from gyp files...

real: 1m18.796s

F:\src\cr\src>vim third_party\webrtc\build\common.gypi
F:\src\cr\src>cd third_party\webrtc
F:\src\cr\src\third_party\webrtc>git diff build\common.gypi
diff --git a/build/common.gypi b/build/common.gypi
index 81c7560..1e7e50a 100644
--- a/build/common.gypi
+++ b/build/common.gypi
@@ -110,7 +110,7 @@
       '..','../..',
     ],
     'defines': [
-      'WEBRTC_SVNREVISION="<!(python <(webrtc_root)/build/version.py)"',
+      'WEBRTC_SVNREVISION="1.0"',
     ],
     'conditions': [
       ['build_with_chromium==1', {

F:\src\cr\src\third_party\webrtc>cd ..\..
F:\src\cr\src>tim c:\src\depot_tools\python_bin\python.exe build\gyp_chromium
Enabled Psyco JIT.
Updating projects from gyp files...

real: 0m54.163s


So, about 24s of calling git svn info.

Perhaps we could put that in gclient runhooks or something instead so that it's only run once instead of many times.

On Tue, Jul 24, 2012 at 10:09 AM, Shawn Singh <shawn...@chromium.org> wrote:

[Tony Chang]

Better yet, this code should be using src/build/util/lastchange.py or something like it to compute the version at compile time where it can be parallelized rather than at gyp time.

[Scott Graham]

I'd prefer not at compile time unless it can be made no-op friendly.
i.e. don't run stuff unconditionally every build.

Since the "input" is source control status, not a file, I think it
needs to happen at sync time?

But if there's some way around that I'm happy, however it works. I
logged a bug here:
http://code.google.com/p/webrtc/issues/detail?id=687

[Tony Chang]

Ah, you're right, lastchange.py runs during gclient runhooks, not compile time.  Sounds like it would need to happen at sync time.

[Andrew MacDonald]

Thanks for finding this. I disabled it for the time being (which we'll hopefully roll into Chromium shortly), and later move it to a gclient hook similar to lastchange.py.

We'll be more careful about this stuff in the future ;)

[Kyle Horimoto]

Is there a workaround/fix for this? I can't get gyp_chromium to finish running, so I haven't been able to compile Chromium yet this morning :(

[Shawn Singh]

The workaround suggested in the other discussion thread was:

  1. remove .git/svn

  2. remove the svn-remote stuff from .git/config, and reconfigure it according to the wiki.

copy-pasting what newt@ had written:

1. cd $chrome/src
   git config --remove-section svn-remote.svn
   rm -Rf .git/svn/
2. Follow wiki instructions to set up git svn again.

I personally can't say what the correct workaround would have been for my situation that started this thread; I just used it as an opportunity to do a fresh git clone and migrate to the new git workflow all at once, and things work fine for me now.

~Shawn

[Ben Hayden]

Is this line correct?

git config svn-remote.svn.fetch trunk/src:refs/remotes/origin/git-svn

Or should it be this?

git config svn-remote.svn.fetch trunk/src:refs/remotes/origin/trunk

I just setup a repo with /git-svn and it's taking forever, but I have an older repo that uses /trunk and it's fast.

I have no idea what the difference is, and it doesn't seem like something that I should need to know.

[Marc-Antoine Ruel]

trunk = old work flow, remote ends with chromium.git

git-svn = new work flow, remote ends with src.git

M-A

[Raymond Toy]

I've also just recently started having this problem.  I've followed the directions below and everything seems to be working, except for one thing.

The wiki instructions (for non-committers) say "git svn fetch".  That's taking forever.  After several hours, I'm at r19700 or so.  Is that expected?  I don't remember having to take this long when I initially did this.

Ray

[Scott Graham]

No, the fetch shouldn't take very long. I normally use the very effective "git rsleevi@ help-me" command on IRC when that happens.

If that command is unavailable, re-following the instructions on a fresh tree probably will work too.

[Raymond Toy]

On Wed, Jul 25, 2012 at 3:37 PM, Scott Graham <sco...@chromium.org> wrote:

No, the fetch shouldn't take very long. I normally use the very effective "git rsleevi@ help-me" command on IRC when that happens.

Heh, ok. 

If that command is unavailable, re-following the instructions on a fresh tree probably will work too.

I think I'll start over with a new tree, preserving my old setup, just in case I still need some ancient branch.

Thanks,

[Ryan Sleevi]

On Wed, Jul 25, 2012 at 3:27 PM, Raymond Toy <rt...@google.com> wrote:

> I've also just recently started having this problem. I've followed the
> directions below and everything seems to be working, except for one thing.
>
> The wiki instructions (for non-committers) say "git svn fetch". That's
> taking forever. After several hours, I'm at r19700 or so. Is that
> expected? I don't remember having to take this long when I initially did
> this.
>
> Ray

If git svn fetch is taking a while, that's usually a sign that git-svn
is not mapping the metadata from the git log (the git-svn-id) to your
local git-svn configuration.

Under the NewGit workflow, many of the git mirrors of the SVN
repository are pointed to the read/write SVN URLs. This means extra
work for those who are only wanting to do a read-only checkout. (Side
note: /many/, but not all - there are still a number of DEPS mirrors
that refer to the /public/ SVN, and thus extra work has to be
configured by comitters, but not by people doing read-only check-outs.
See the .pushurl comments under the UsingNewGit page)

Did you perhaps omit a step when setting up the UsingNewGit for
read-only? the rewriteUUID-and-friend commands are supposed to help
git-svn know that when it sees a git-svn-id for
svn://svn.chromium.org, it's the same as the https://src.chromium.org
version, and thus git-svn 'already has it', and should just no-op the
fetch. Depending on git log speed, a full fetch if you have a decent
git clone "should" just take a minute or two.

You "should" be able to remove the [svn] entries from .git/config, and
then fully rm -rf .git/.svn, and then follow those config steps again
from the UsingNewGit page. Saves having to re-clone.

[Raymond Toy]

On Wed, Jul 25, 2012 at 7:57 PM, Ryan Sleevi <rsl...@chromium.org> wrote:

Did you perhaps omit a step when setting up the UsingNewGit for
read-only? the rewriteUUID-and-friend commands are supposed to help
git-svn know that when it sees a git-svn-id for
svn://svn.chromium.org, it's the same as the https://src.chromium.org
version, and thus git-svn 'already has it', and should just no-op the
fetch. Depending on git log speed, a full fetch if you have a decent
git clone "should" just take a minute or two.

I followed all of the directions for read-only setup, including the rewriteUUID stuff.  When I run git svn fetch, I do see it printing out very quickly a bunch of (git?) hashes, but then it quickly reverts to pulling in every single revision.

In any case, since my box has plenty of disk space, I just created a new directory and followed the new git flow instructions from scratch.  Everything is working as expected and gyp_chromium is fast once again.

I may try again with my old tree to do an update.  I doubt I'm the only one who has run into this issue.

Thanks for the explanation of how things are supposed to work.

Ray

[Dan Beam]

On Thu, Jul 26, 2012 at 10:00 AM, Raymond Toy <rt...@google.com> wrote:

On Wed, Jul 25, 2012 at 7:57 PM, Ryan Sleevi <rsl...@chromium.org> wrote:

Did you perhaps omit a step when setting up the UsingNewGit for
read-only? the rewriteUUID-and-friend commands are supposed to help
git-svn know that when it sees a git-svn-id for
svn://svn.chromium.org, it's the same as the https://src.chromium.org
version, and thus git-svn 'already has it', and should just no-op the
fetch. Depending on git log speed, a full fetch if you have a decent
git clone "should" just take a minute or two.

I followed all of the directions for read-only setup, including the rewriteUUID stuff.  When I run git svn fetch, I do see it printing out very quickly a bunch of (git?) hashes, but then it quickly reverts to pulling in every single revision.

In any case, since my box has plenty of disk space, I just created a new directory and followed the new git flow instructions from scratch.  Everything is working as expected and gyp_chromium is fast once again.

I may try again with my old tree to do an update.  I doubt I'm the only one who has run into this issue.

Thanks for the explanation of how things are supposed to work.

tl;dr - the LKGR is a Subversion revision which means nothing to git [workflows].  If you want to sync to LKGR with your git checkout, the LKGR svn rev needs to be mapped to a git SHA1.  This means gclient sync needs to:

  1) git svn fetch # or

  2) git pull && git reset --hard `git log --grep '^git-svn-id:.*@<svn_rev> ' --pretty='format:%H'` # or something to this effect

The second way is hacky and fragile (and requires you to rebase than reset, as git fetch alone doesn't let you grep the git log as far as I know), so I chose the first.  If git svn fetch is used, there are 2 ways to speed this up:

  1) git svn fetch -r <svn_rev> # leaves huge holes in your git-svn history

  2) git fetch && git svn fetch # no holes, should be quick when configured correctly

We've also had an idea to sidestep the necessity to do this svn -> git mapping by making a remote lkgr branch on the Chromium infrastructure side, but isn't there yet (in Marc-Antoine's words):

  +maruel> dbeam: because ... nobody has done it yet?

Keeping a remote branch up to date would require modifying the LKGR bumping script to move the HEAD of the remote lkgr git branch (essentially only running git svn find-rev r<rev> once for all developers) and possibly adding a way to express a remote git branch in your .gclient solution (e.g. "git_branch": "lkgr"?).  This isn't necessary for Subversion as one can simply modify the URL as branches are just directories.  

Additionally, there is an open bug I haven't gotten to fixing yet that warns about or auto-fixes your git-svn setup to use the "snappy" way (http://code.google.com/p/chromium/issues/detail?id=109184).

I hope this provides more context,

Dan Beam

db...@chromium.org

P.S. I don't necessarily recommend this as a long term solution, but you can also simply remove your safesync_url if the latency is unacceptable (but obviously I'd try to optimize it first, up to you, should be minimal if configured correctly).

[Daniel Cheng]

Btw, instead of :

git log --grep '^git-svn-id:.*@<svn_rev>

Just say:

git svn find-rev r123456

Daniel

[Dan Beam]

On Thu, Jul 26, 2012 at 4:31 PM, Daniel Cheng <dch...@chromium.org> wrote:

Btw, instead of :

git log --grep '^git-svn-id:.*@<svn_rev>

Just say:

git svn find-rev r123456

Yes, but you need to call git svn fetch to do this.  git svn fetch && git svn find-rev r<rev> would have been more descriptive, sorry.

Dan Beam

db...@chromium.org

[Torne (Richard Coles)]

On 27 July 2012 00:21, Dan Beam <db...@chromium.org> wrote:

On Thu, Jul 26, 2012 at 10:00 AM, Raymond Toy <rt...@google.com> wrote:

On Wed, Jul 25, 2012 at 7:57 PM, Ryan Sleevi <rsl...@chromium.org> wrote:

Did you perhaps omit a step when setting up the UsingNewGit for
read-only? the rewriteUUID-and-friend commands are supposed to help
git-svn know that when it sees a git-svn-id for
svn://svn.chromium.org, it's the same as the https://src.chromium.org
version, and thus git-svn 'already has it', and should just no-op the
fetch. Depending on git log speed, a full fetch if you have a decent
git clone "should" just take a minute or two.

I followed all of the directions for read-only setup, including the rewriteUUID stuff.  When I run git svn fetch, I do see it printing out very quickly a bunch of (git?) hashes, but then it quickly reverts to pulling in every single revision.

In any case, since my box has plenty of disk space, I just created a new directory and followed the new git flow instructions from scratch.  Everything is working as expected and gyp_chromium is fast once again.

I may try again with my old tree to do an update.  I doubt I'm the only one who has run into this issue.

Thanks for the explanation of how things are supposed to work.

tl;dr - the LKGR is a Subversion revision which means nothing to git [workflows].  If you want to sync to LKGR with your git checkout, the LKGR svn rev needs to be mapped to a git SHA1.  This means gclient sync needs to:

  1) git svn fetch # or

  2) git pull && git reset --hard `git log --grep '^git-svn-id:.*@<svn_rev> ' --pretty='format:%H'` # or something to this effect

The second way is hacky and fragile (and requires you to rebase than reset, as git fetch alone doesn't let you grep the git log as far as I know), so I chose the first.  If git svn fetch is used, there are 2 ways to speed this up:

This doesn't solve the second way being hacky and fragile, but it does avoid the rebase then reset:

git remote update origin

git rebase `git log --grep '^git-svn-id:.*@<svn_rev> ' --pretty='format:%H' origin/git-svn`

i.e. update the remote, then just look in the log for the downloaded remote and use the SHA you find to rebase.

It will still pick the wrong thing out of the log sometimes, though, because of reverts/recommits/etc which may have included the previous commit's git-svn-id line in their own commit message.

--
Torne (Richard Coles)
to...@google.com