[LLVMdev] Mirroring of LLVM repository
Anton Korobeynikov
Currently the load of llvm.org is too high. This influences the whole
project services like buildbots, bugzilla, etc. It was found that this
workload is possible caused by massive mirroring of LLVM SVN
repository into git/hg/whatever. Please don't do that :)
1. Usually one don't need full history, this makes the mirroring much
easier and faster (do you *really* need r5000? I guess - no)
2. There are some git mirrors already, which can be used for
'bootstrapping' git-svn - you can really save a lot of resources using
them. They might be not so up-to-date (often they are updated daily
during late night hours for US people), but as I already mentioned -
you will have bunch of information already imported.
These mirrors are:
Main LLVM repository (starting from r40000): git://repo.or.cz/llvm.git
(see http://repo.or.cz/w/llvm.git for more information)
llvm-gcc repository (starting from r40000):
git://repo.or.cz/llvm-gcc-4.2.git
(http://repo.or.cz/w/llvm-gcc-4.2.git for more information)
clang repository (starting from r40000): git://repo.or.cz/clang.git
(http://repo.or.cz/w/clang.git for more information)
As a bonus - you'll have 'normal' names / e-mails of committers from
that mirrors.
Please think twice if you will need to mirror the stuff directly from
the main LLVM SVN repository. Basically you'll preventing people from
getting real work done (since people can't update and takes forever).
Thank you for you patience and understanding!
--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University
_______________________________________________
LLVM Developers mailing list
LLV...@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Rafael Espindola
> 'bootstrapping' git-svn - you can really save a lot of resources using
> them. They might be not so up-to-date (often they are updated daily
> during late night hours for US people), but as I already mentioned -
> you will have bunch of information already imported.
How do I do this bootstrap?
I tried a "git clone" + "git svn init", but the next "git svn fetch" tries
to start from revision 1.
Cheers,
--
Rafael Avila de Espindola
Google | Gordon House | Barrow Street | Dublin 4 | Ireland
Registered in Dublin, Ireland | Registration Number: 368047
Andreas Bolka
> Currently the load of llvm.org is too high. This influences the whole
> project services like buildbots, bugzilla, etc. It was found that this
> workload is possible caused by massive mirroring of LLVM SVN
> repository into git/hg/whatever. Please don't do that :)
Let me add to what Anton said:
I can only speak for git-svn, but the main burden it imposes on LLVM's
SVN repository are attempts of mirroring the _complete_ history.
Unfortunatly, a naive `git svn clone` will do just that -- so _please_
avoid this at all cost.
If you want to use git-svn to work with LLVM's SVN repository, please
only clone the latest SVN revision using git-svn's `--revision <n>`
option (combined with a huge log window: `--log-window-size 999999`).
You may find the following script helpful:
#!/bin/bash
if [ $# -ne 2 ]; then echo "Usage: $0 <svn_url> <dir>" >&2; exit 64; fi
SVN_URL=$1
SVN_REV=`svn info $SVN_URL | awk '/Rev:/ {print $4}'`
git svn clone --log-window-size 999999 --revision $SVN_REV $SVN_URL $2
This makes the initial clone roughly as demanding as a `svn checkout`.
> [..] There are some git mirrors already [..]
Next to the mirrors mentioned (and kept up-to-date) by Anton, I've been
maintaining a mirror of LLVM's trunk (only llvm, no llvm-gcc or clang)
on Github for the last few months:
- http://github.com/earl/llvm-mirror/
This mirror is automatically updated once an hour and includes the
complete history of LLVM, back to the initial revision [1].
As Anton mentioned, you can use those public Git mirrors to bootstrap
your own git-svn repository. Here's how to do this using my Github
mirror:
git clone git://github.com/earl/llvm-mirror.git llvm
cd llvm
git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/*'
git fetch
git svn init https://llvm.org/svn/llvm-project/llvm/trunk
git svn rebase --local
Once that's done, you can work with git-svn just as you're used to.
Alternatively, you can keep up-to-date purely via the Git mirror by
using `git fetch; git svn rebase --local`. With that, the only time you
need to hit SVN is for committing. This is most effective with a recent
Git (1.6.1+) where you can freely intermix fetching via `git fetch` from
a Git mirror with `git svn fetch` directly from SVN (git-svn will
incrementally update it's revision map).
[1]
http://github.com/earl/llvm-mirror/tree/455694886d7a341a76164cc67d3fcc2741c73c52
--
Regards,
Andreas
Rafael Espindola
> cd llvm
> git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/*'
> git fetch
> git svn init https://llvm.org/svn/llvm-project/llvm/trunk
> git svn rebase --local
This one worked perfectly. Thanks!
I tried the same with the llvm-gcc-4.2 mirror, but "git svn rebase
--local" is running for
more than 1h with no output. Do I have to do something special for
git://repo.or.cz/llvm-gcc-4.2.git?
Thanks,
--
Rafael Avila de Espindola
Google | Gordon House | Barrow Street | Dublin 4 | Ireland
Registered in Dublin, Ireland | Registration Number: 368047
_______________________________________________
Anton Korobeynikov
> I tried the same with the llvm-gcc-4.2 mirror, but "git svn rebase
> --local" is running for
> more than 1h with no output. Do I have to do something special for
> git://repo.or.cz/llvm-gcc-4.2.git?
That's pretty strange. However this repository is large and contains
many files, maybe first run will be long due to need of regeneration
of metadata, etc.
--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University
Andreas Bolka
> > git clone git://github.com/earl/llvm-mirror.git llvm
> > cd llvm
> > git config --add remote.origin.fetch '+refs/remotes/*:refs/remotes/*'
> > git fetch
> > git svn init https://llvm.org/svn/llvm-project/llvm/trunk
> > git svn rebase --local
>
> This one worked perfectly. Thanks!
> Do I have to do something special for
> git://repo.or.cz/llvm-gcc-4.2.git?
Yes, my bad, the instructions were a bit too specific to my Github
mirror. The following scheme bootstraps git-svn and works with all
mirrors, including those on repo.or.cz:
mkdir llvm-gcc-4.2
cd llvm-gcc-4.2
git init
git pull git://repo.or.cz/llvm-gcc-4.2.git master:remotes/git-svn
git svn init https://llvm.org/svn/llvm-project/llvm-gcc-4.2/trunk
git svn rebase --local
--
Regards,
Andreas
Rafael Espindola
> many files, maybe first run will be long due to need of regeneration
> of metadata, etc.
One difference I noticed is that in the llvm repository (the one in
github) "git branch -r" lists
a git-svn branch that is missing in llvm-gcc-4.2.
I am not sure how git-svn works. Is that branch required?
Cheers,
--
Rafael Avila de Espindola
Google | Gordon House | Barrow Street | Dublin 4 | Ireland
Registered in Dublin, Ireland | Registration Number: 368047
Rafael Espindola
> mirror. The following scheme bootstraps git-svn and works with all
> mirrors, including those on repo.or.cz:
>
> mkdir llvm-gcc-4.2
> cd llvm-gcc-4.2
> git init
> git pull git://repo.or.cz/llvm-gcc-4.2.git master:remotes/git-svn
> git svn init https://llvm.org/svn/llvm-project/llvm-gcc-4.2/trunk
> git svn rebase --local
This works nicely, but I can only use "git svn rebase" for updating.
Is that I way I could use both "git pull" and "git svn rebase"? I
tried to adapt http://gcc.gnu.org/wiki/GitMirror, but I am lost. What
I tried was adding
[remote "origin"]
url = git://repo.or.cz/clang.git
fetch = master:remotes/git-svn
to the config file. It works until I do the first "git svn rebase".
After that I get
From git://repo.or.cz/clang
! [rejected] master -> git-svn (non fast forward)
> --
> Regards,
> Andreas
Cheers,
--
Rafael Ávila de Espíndola