Tip of the day: repo sync -j

[Antoine Labour]

So, the git server has a limit on how many connections you can have open at the same time. It makes parallel repo sync fail if the -j parameter is too high. E.g.

$ repo sync -j 12

channel 10: open failed: administratively prohibited: open failed

channel 11: open failed: administratively prohibited: open failed

fatal: The remote end hung up unexpectedly

fatal: The remote end hung up unexpectedly

error: Cannot fetch chromeos-overlay

error: Cannot fetch login_manager

Fetching projects:  32% (9/28)  remote: Counting objects: 7, done.

remote: Compressing objects: 100% (4/4), done.

remote: Total 4 (delta 3), reused 0 (delta 0)

Unpacking objects: 100% (4/4), done.

From ssh://gitrw.chromium.org:9222/board-overlays

   55b7664..f7cc576  master     -> cros/master

Fetching projects:  35% (10/28)  

error: Exited sync due to fetch errors

The good thing is that repo can separate the "network" sync (the git fetch part, network bound) from the "local" sync (the git rebase part, compute & i/o bound).

So you can separate both phases and give a different -j argument, something like:

$ repo sync -n -j 4 && repo sync -l -j 12

Fetching projects:  96% (27/28)  remote: Counting objects: 9, done.

remote: Compressing objects: 100% (5/5), done.

remote: Total 5 (delta 3), reused 0 (delta 0)

Unpacking objects: 100% (5/5), done.

From ssh://gitrw.chromium.org:9222/chromiumos-overlay

   1ce5d7e..8deeb4c  master     -> cros/master

Fetching projects: 100% (28/28), done.  

Your sources have been sync'd successfully.

Antoine

[Taylor Hutt]

On Thu, Apr 7, 2011 at 1:44 PM, Antoine Labour <pi...@chromium.org> wrote:

<snip>

$ repo sync -n -j 4 && repo sync -l -j 12

Nice.  I measured the time to use just a 'repo sync' command with the little snippet above.  I executed each command twice and noted the numbers.

'repo sync'
  real 89.81
  user 18.09
  sys 5.67

  real 68.07
  user 15.36
  sys 5.07

repo sync -n -j 4 && repo sync -l -j 12

  real 22.58
  user 15.28
  sys 4.65

  real 24.63
  user 15.46
  sys 4.63

That's a nice speedup!  Thanks for the tip.

I'm assuming the behavior will be the same when there are conflicts?

thutt
--
Diagnosis: Dry, rough, tired hair caused by daily wear and tear.

[Anush Elangovan(அனுஷ்)]

On Thu, Apr 7, 2011 at 2:06 PM, Taylor Hutt <th...@google.com> wrote:

On Thu, Apr 7, 2011 at 1:44 PM, Antoine Labour <pi...@chromium.org> wrote:

<snip>

$ repo sync -n -j 4 && repo sync -l -j 12

Thanks for the tip. Also if you checkout the external manifest (repo init -u http://git.chromium.org/git/manifest) you checkout via http so you will be able to scale to much larger -j values (since we are load balanced on http). 

 

Nice.  I measured the time to use just a 'repo sync' command with the little snippet above.  I executed each command twice and noted the numbers.

'repo sync'
  real 89.81
  user 18.09
  sys 5.67

  real 68.07
  user 15.36
  sys 5.07

repo sync -n -j 4 && repo sync -l -j 12

  real 22.58
  user 15.28
  sys 4.65

  real 24.63
  user 15.46
  sys 4.63

That's a nice speedup!  Thanks for the tip.

I'm assuming the behavior will be the same when there are conflicts?

thutt
--
Diagnosis: Dry, rough, tired hair caused by daily wear and tear.

--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Vadim Bendebury]

On Thu, Apr 7, 2011 at 2:15 PM, Anush Elangovan(அனுஷ்)
<an...@chromium.org> wrote:
>
> On Thu, Apr 7, 2011 at 2:06 PM, Taylor Hutt <th...@google.com> wrote:
>>
>>
>> On Thu, Apr 7, 2011 at 1:44 PM, Antoine Labour <pi...@chromium.org> wrote:
>>
>> <snip>
>>>
>>> $ repo sync -n -j 4 && repo sync -l -j 12
>>
>
> Thanks for the tip. Also if you checkout the external manifest (repo init -u
> http://git.chromium.org/git/manifest) you checkout via http so you will be
> able to scale to much larger -j values (since we are load balanced on
> http).
>
>

is there a reason not to make it a default behavior?  Modulo
discovering the number of cores on the machines/http vs git fetch,
etc.?

cheers,
/v

--
Vadim Bendebury (mailto:vbe...@google.com)
Google ChromeOS - (650) 253-7486

[Chris Masone]

Because anyone who doesn't build internal bits (a lot of people, I suspect) is checking out over HTTP and this would slow them down.  I can do everything with -j12 or more, because I don't check out over ssh.