Question about "repo init" when using --reference option.

[SeongUk Baek]

Dear all.

I am testing about instruction execution time like "repo sync -n", "repo sync -l", "repo forall -c git checkout TAG" with & without using "—reference” options

It is observed that
"repo sync -n" and "repo sync -l" are faster when I use "--reference option"

But "repo forall -c git checkout TAG" is slow.

Does anyone know about this?

Thanks & Regards

[Magnus Bäck]

On Monday, May 06, 2013 at 00:59 EDT,

SeongUk Baek <baeks...@gmail.com> wrote:

> I am testing about instruction execution time like "repo sync -n",
> "repo sync -l", "repo forall -c git checkout TAG" with & without
> using "—reference” options
>
> It is observed that
> "repo sync -n" and "repo sync -l" are faster when I use "--reference
> option"

The purpose of --reference is to speed up the network part of the sync
operation, so 'repo sync -n' should indeed be faster with --reference.
'repo sync -l' should be unaffected.

> But "repo forall -c git checkout TAG" is slow.

That doesn't make sense since it's what 'repo sync -l' does (verify with
'repo --trace sync -l'). While the network half of the sync operation
may run threaded the local half IIRC won't.

Regardless of performance, it usually won't do what you want since
it doesn't take the manifest contents into account. I've seen numerous
howtos etc suggest the use of this 'forall' command but it's usually
misguided.

--
Magnus Bäck
ba...@google.com

[SeongUk Baek]

Thanks for reply.

2013년 5월 6일 월요일 오후 10시 42분 8초 UTC+9, Magnus Bäck 님의 말:

On Monday, May 06, 2013 at 00:59 EDT,
     SeongUk Baek <baeks...@gmail.com> wrote:

> I am testing about instruction execution time like "repo sync -n",
> "repo sync -l", "repo forall -c git checkout TAG" with & without
> using "—reference” options
>
> It is observed that
> "repo sync -n" and "repo sync -l" are faster when I use "--reference
> option"

The purpose of --reference is to speed up the network part of the sync
operation, so 'repo sync -n' should indeed be faster with --reference.
'repo sync -l' should be unaffected.

 

I see your point. 

> But "repo forall -c git checkout TAG" is slow.

That doesn't make sense since it's what 'repo sync -l' does (verify with
'repo --trace sync -l'). While the network half of the sync operation
may run threaded the local half IIRC won't.

Regardless of performance, it usually won't do what you want since
it doesn't take the manifest contents into account. I've seen numerous
howtos etc suggest the use of this 'forall' command but it's usually
misguided.

 

In case of using "repo forall -c"

1. When I want to reset Index area and Working directory area, using command "git reset --hard HEAD" or "git checkout -f"

If I want to apply all of repos, using "repo forall -c git reset --hard HEAD" or "repo forall -c git checkout -f"

2. When I want to make local branch <start-point>, using like "repo forall -c git branch <branchName> <TagName or BranchName>"

3. When I want to push local branch was created, using like "repo forall -c git push <remoteName> <branchName>"

In this case, using "repo forall" is correct? or any other way?

 

--
Magnus Bäck
ba...@google.com

[Magnus Bäck]

On Monday, May 06, 2013 at 22:08 EDT,

SeongUk Baek <baeks...@gmail.com> wrote:

> 2013년 5월 6일 월요일 오후 10시 42분 8초 UTC+9, Magnus Bäck 님의 말:

[...]

> > Regardless of performance, it usually won't do what you want since
> > it doesn't take the manifest contents into account. I've seen
> > numerous howtos etc suggest the use of this 'forall' command but
> > it's usually misguided.
>
> In case of using "repo forall -c"
>
> 1. When I want to reset Index area and Working directory area, using
> command "git reset --hard HEAD" or "git checkout -f"
> If I want to apply all of repos, using "repo forall -c git reset
> --hard HEAD" or "repo forall -c git checkout -f"
>
> 2. When I want to make local branch <start-point>, using like "repo
> forall -c git branch <branchName> <TagName or BranchName>"

Depending on what you want to do and what's in your manifest,
'repo start' might be a better idea. The command above assumes
that all gits in the manifest should be branched from the same
branch or tag. I know that at least in the past some tools-related
gits in the Android manifest have been on another branch than
the rest. If you're branching from a tagged release you can use
the tag instead and things will work fine, but that isn't necessarily
the case.

> 3. When I want to push local branch was created, using like "repo
> forall -c git push <remoteName> <branchName>"
>
> In this case, using "repo forall" is correct? or any other way?

I was specifically talking about using forall when you should really be
switching the manifest revision with 'repo init -b'. There are countless
other Git operations that you may want to do in all gits for which
there's no Repo counterpart.

--
Magnus Bäck
ba...@google.com

[SeongUk Baek]

I'll apply mirror configuration to all of developer's server like below steps.

1. Create bare repositories in storage.

2. Mount the storage in all developers server.

3. All developers are using the "--reference" option to the storage when they want to sync.

I really want to know that If I configured above steps, is there a problem that can be expected?

I really appreciate your reply.

2013년 5월 7일 화요일 오전 11시 39분 37초 UTC+9, Magnus Bäck 님의 말:

[Luke Diamand]

On 07/05/13 09:11, SeongUk Baek wrote:
> I'll apply mirror configuration to all of developer's server like below
> steps.
>
> 1. Create bare repositories in storage.
> 2. Mount the storage in all developers server.
> 3. All developers are using the "--reference" option to the storage when
> they want to sync.
>
> I really want to know that If I configured above steps, is there a
> problem that can be expected?

The thing I found was that even with this, the developer machines are
still making ssh:// or git:// RPCs to your repo server. This then
becomes the bottleneck. The git config "insteadof" helps with this.

>
> I really appreciate your reply.
>
> 2013년 5월 7일 화요일 오전 11시 39분 37초 UTC+9, Magnus Bäck 님의 말:
>
> On Monday, May 06, 2013 at 22:08 EDT,

> ba...@google.com <javascript:>
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Magnus Bäck]

On Tuesday, May 07, 2013 at 04:35 EDT,

Luke Diamand <lu...@diamand.org> wrote:

> On 07/05/13 09:11, SeongUk Baek wrote:
>
> > I'll apply mirror configuration to all of developer's server like
> > below steps.
> >
> > 1. Create bare repositories in storage.
> > 2. Mount the storage in all developers server.
> > 3. All developers are using the "--reference" option to the storage
> > when they want to sync.
> >
> > I really want to know that If I configured above steps, is there a
> > problem that can be expected?

AFAICT --reference only works with local directories.

$ git clone --reference /some/nfs/path/git https://github.com/git/git.git
Cloning into 'git'...
fatal: reference repository '/some/nfs/path/git' is not a local directory.

Anyway, having a central mirror that's NFS mounted defeats the purpose
of --reference, doesn't it? It will take some load off your Git/Gerrit
server (scaling up NFS servers might be easier), but the performance
will continue to be bound by the network capacity.

(For fun I tried --reference when cloning git.git, and the clone time
went down from 37 s to <1 s. That was with a brand-new clone, though.
Doing the same exercise for Android's platform/frameworks/base when
the git I used as reference was cloned on Jan 31 reduced clone time
from 2 min 39 s to 28 s.)

> The thing I found was that even with this, the developer machines
> are still making ssh:// or git:// RPCs to your repo server.

Yes, this is expected. The mirror is supposed to provide a solid base
for the clones, not an up to date replica.

> This then becomes the bottleneck. The git config "insteadof" helps
> with this.

How are you using insteadOf?

--
Magnus Bäck
ba...@google.com

[SeongUk Baek]

2013년 5월 7일 화요일 오후 10시 12분 14초 UTC+9, Magnus Bäck 님의 말:

AFAICT --reference only works with local directories.

Yes. I know it .

$ git clone --reference /some/nfs/path/git https://github.com/git/git.git
Cloning into 'git'...
fatal: reference repository '/some/nfs/path/git' is not a local directory.

Anyway, having a central mirror that's NFS mounted defeats the purpose
of --reference, doesn't it? It will take some load off your Git/Gerrit
server (scaling up NFS servers might be easier), but the performance
will continue to be bound by the network capacity.

Sure. It will be some load off Gerrit server and save disk capacity.

And network capacity also important.

But I worried that make some trouble because developers are using pack files shared.

For example, "repo forall" command is slower than without --reference option.

[Magnus Bäck]

On Tuesday, May 07, 2013 at 20:33 EDT,

SeongUk Baek <baeks...@gmail.com> wrote:

> 2013년 5월 7일 화요일 오후 10시 12분 14초 UTC+9, Magnus Bäck 님의 말:
>
> > AFAICT --reference only works with local directories.
>
> Yes. I know it .
>
> > $ git clone --reference /some/nfs/path/git https://github.com/git/git.git
> > Cloning into 'git'...
> > fatal: reference repository '/some/nfs/path/git' is not a local directory.
> >
> > Anyway, having a central mirror that's NFS mounted defeats the
> > purpose of --reference, doesn't it? It will take some load off your
> > Git/Gerrit server (scaling up NFS servers might be easier), but the
> > performance will continue to be bound by the network capacity.
>
> Sure. It will be some load off Gerrit server and save disk capacity.
> And network capacity also important.
>
> But I worried that make some trouble because developers are using pack
> files shared.
> For example, "repo forall" command is slower than without --reference
> option.

How are the files shared, if not via NFS?

[...]

--
Magnus Bäck
ba...@google.com

[SeongUk Baek]

2013년 5월 8일 수요일 오전 10시 18분 16초 UTC+9, Magnus Bäck 님의 말:

As I know if use "--reference" option, the developers use the pack files shared from central mirror repos.

Is not it?

After repo sync, there is no pack file in .git/objects/pack.

And only the files that refer to the central mirror repos in .git/objects/info/alternates.

[Magnus Bäck]

On Tuesday, May 07, 2013 at 21:37 EDT,

SeongUk Baek <baeks...@gmail.com> wrote:

> 2013년 5월 8일 수요일 오전 10시 18분 16초 UTC+9, Magnus Bäck 님의 말:
>

> > How are the files shared, if not via NFS?
>
> As I know if use "--reference" option, the developers use the pack
> files shared from central mirror repos.
> Is not it?
>
> After repo sync, there is no pack file in .git/objects/pack.
> And only the files that refer to the central mirror repos in
> .git/objects/info/alternates.

But how are you able to set that up, given that --reference doesn't
support NFS (or any other network file systems)?

--
Magnus Bäck
ba...@google.com

[SeongUk Baek]

Dear Magnus.

It was decided not to use the storage.

So, I will make bare repositories in every developer's server.

And they will use --reference to local directories.

2013년 5월 8일 수요일 오전 10시 46분 0초 UTC+9, Magnus Bäck 님의 말: