[LLVMdev] GIT mirroring

[Anton Korobeynikov]

Hello Everyone,

It seems given the decent amount of discussions it's time to make
small announcement.

So, official git mirrors are available for some subset of LLVM
projects. They were used by some LLVM developers for couple of months
already and seem to be stable enough.

Right now there is a mirroring for LLVM itself and clang. More can be
added per requests. The mirrors are made via running of git-svn on
llvm.org inside post-commit hook, so they should be in theory as fresh
as possible. Also, one can recreate git-svn metadata and enable such
nice features like "git svn dcommit". The mirrors track only trunk
mostly due to not so git'ish way of doing branches / tags in LLVM
repository.

The URLs for clone are:

http://llvm.org/git/llvm.git
http://llvm.org/git/clang.git

It's not yet decided whether we'll run e.g. gitweb for mirrors, most
probably - there won't be anything like this, viewvc should be enough
for almost all purposes.

PS: Patches to various sorts of documentation scattered here and there
are welcome :)

--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

[David A. Greene]

Anton Korobeynikov <an...@korobeynikov.info> writes:

> So, official git mirrors are available for some subset of LLVM
> projects. They were used by some LLVM developers for couple of months
> already and seem to be stable enough.

Woo hoo!

> as possible. Also, one can recreate git-svn metadata and enable such
> nice features like "git svn dcommit". The mirrors track only trunk

Yee haw! Is there some documentation on how to set this up for LLVM?

Thanks Anton and everyone who worked on this!

-Dave

[Jakob Stoklund Olesen]

On Feb 1, 2011, at 12:20 PM, Anton Korobeynikov wrote:

> Hello Everyone,
>
> It seems given the decent amount of discussions it's time to make
> small announcement.
>
> So, official git mirrors are available for some subset of LLVM
> projects. They were used by some LLVM developers for couple of months
> already and seem to be stable enough.

Thank you for setting this up, Anton!

This is how I use the Git mirror with git-svn:

For the initial clone and setup:

$ git clone http://llvm.org/git/llvm.git
$ cd llvm
$ git config --add remote.origin.fetch '+refs/remotes/git-svn:refs/remotes/git-svn'
$ git fetch
$ git svn init https://llvm.org/svn/llvm-project/llvm/trunk
$ git svn rebase -l

This will quickly build the git-svn metadata by using the magical remotes/git-svn branch fetched from the origin.

To update I run:

$ git fetch
$ git svn rebase -l

And to commit:

$ git svn dcommit
$ git fetch
$ git svn rebase -l

I have sometimes seen git-svn refusing to dcommit, claiming that I have uncommitted files in my tree. I think this happens when I forget to resynchronize the metadata after committing. Anyway, the solution is to wipe away all of .git/svn and rebuild it with "git svn rebase -l"

/jakob

[Mark Lacey]

On Tue, Feb 1, 2011 at 12:20 PM, Anton Korobeynikov <an...@korobeynikov.info> wrote:

So, official git mirrors are available for some subset of LLVM
projects.

Thanks so much for setting this up, Anton!

I notice that .gitignore in the root has tools/clang. This makes it necessary to use '--force' if using 'git submodule'. I do not know if there are any drawbacks to this - if someone knows, it would be interesting to know what those drawbacks are.

One alternative would be for individuals who clone into tools/clone without using submodules to instead add tools/clang explicitly to .git/info/exclude.

Thanks again,

Mark

[NAKAMURA Takumi]

To rebuild, it would be enough to remove
.git/svn/refs/remotes/git-svn/.rev_map.*

My usual way to resync;

$ git fetch llvm.org (is remote name)

$ git update-ref refs/remotes/git-svn llvm.org/master

$ git svn fetch
Partial-rebuilding
.git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8
...
Currently at 124651 = 071d3af0de273b1079d79f7f979264f28d567373
r124653 = cde5a400dbc9655eddf0f383585d3cf67c11c539
...
r124704 = e349d2c61256f11180b7112ab592b4ae700f3dcf
Done rebuilding
.git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8

...Takumi

[Peter Collingbourne]

On Wed, Feb 02, 2011 at 09:42:58AM +0900, NAKAMURA Takumi wrote:
> To rebuild, it would be enough to remove
> .git/svn/refs/remotes/git-svn/.rev_map.*
>
> My usual way to resync;
>
> $ git fetch llvm.org (is remote name)
>
> $ git update-ref refs/remotes/git-svn llvm.org/master
>
> $ git svn fetch
> Partial-rebuilding
> .git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8
> ...
> Currently at 124651 = 071d3af0de273b1079d79f7f979264f28d567373
> r124653 = cde5a400dbc9655eddf0f383585d3cf67c11c539
> ...
> r124704 = e349d2c61256f11180b7112ab592b4ae700f3dcf
> Done rebuilding
> .git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8

I have a small script (attached) in my $PATH so I can just do:

$ git svnmu llvm

--
Peter

[Bob Wilson]

I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.

I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

[Jakob Stoklund Olesen]

On Mar 21, 2011, at 1:47 PM, Bob Wilson wrote:

> I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.
>
> I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.

I now run 'git update-ref' every time I fetch from the mirror:

git fetch -p origin
git update-ref refs/remotes/git-svn origin/master
git svn rebase -l

Same for committing:

git svn dcommit
git fetch -p origin
git update-ref refs/remotes/git-svn origin/master
git svn rebase -l

For the initial clone and setup:

git clone http://llvm.org/git/llvm.git
cd llvm

git svn init https://llvm.org/svn/llvm-project/llvm/trunk

git update-ref refs/remotes/git-svn origin/master

[Anton Korobeynikov]

>> I've tried re-creating my git repos from scratch but that did not fix the problem.  Any ideas?
>
> The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.

I'm not a git-svn expert, but I suspect the real problem is that
git-svn automagically updates master from git-svn remote. Right now
we're exporting just master and thus stuff appears to form a cycle.
This is just a random thought though :)

--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

_______________________________________________

[Tobias Grosser]

On 03/21/2011 04:47 PM, Bob Wilson wrote:
> I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.
>
> I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

Yes, here the changes necessary:

>> $ git clone http://llvm.org/git/llvm.git
>> $ cd llvm
>> $ git config --add remote.origin.fetch '+refs/remotes/git-svn:refs/remotes/git-svn'
>> $ git fetch

Skip the last two lines line.

>> $ git svn init https://llvm.org/svn/llvm-project/llvm/trunk

Add here:
git config svn-remote.svn.fetch ':refs/remotes/origin/master'

>> $ git svn rebase -l

And here I personally just use a 'git svn fetch'

This is the complete sequence

$ git clone http://llvm.org/git/llvm.git
$ cd llvm

$ git svn init https://llvm.org/svn/llvm-project/llvm/trunk

$ git config svn-remote.svn.fetch ':refs/remotes/origin/master'
$ git svn fetch

LLVM trunk is now in origin/master and can be accessed e.g. by

$git log origin/master

It can be updated by using

$git remote update
$git fetch
$git pull

Cheers
Tobi

[Tobias Grosser]

On 03/21/2011 04:54 PM, Jakob Stoklund Olesen wrote:
>
> On Mar 21, 2011, at 1:47 PM, Bob Wilson wrote:
>
>> I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.
>>
>> I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?
>
> The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.
>
> I now run 'git update-ref' every time I fetch from the mirror:

You should be able to get rid of this by calling:

git config svn-remote.svn.fetch ':refs/remotes/origin/master'

Now the default svn branch points to refs/remotes/origin/master instead
of refs/remotes/git-svn and everything should work automatically.
Let me know if there are any problems with this approach.

Tobi

[Jakob Stoklund Olesen]

On Mar 21, 2011, at 2:29 PM, Tobias Grosser wrote:

> On 03/21/2011 04:54 PM, Jakob Stoklund Olesen wrote:
>>
>> On Mar 21, 2011, at 1:47 PM, Bob Wilson wrote:
>>
>>> I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.
>>>
>>> I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?
>>
>> The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.
>>
>> I now run 'git update-ref' every time I fetch from the mirror:
>
> You should be able to get rid of this by calling:
>
> git config svn-remote.svn.fetch ':refs/remotes/origin/master'

That would work too.

/jakob