Using repo for our own projects
1,349 views
Skip to first unread message
mha...@sandia.gov
Dec 16, 2015, 1:56:48 PM12/16/15
to Repo and Gerrit Discussion
We're looking at using repo for managing our own git repositories. I've seen this https://groups.google.com/forum/#!searchin/repo-discuss/How$20to$20use$20repo/repo-discuss/sUXdTJUcAeM/mvUXTun1BAAJ issue, but we're past that point.
We have a manifest repository set up and so far we are able to get things to work as we expect. However, we have a requirement to be able to work on machines which are not connected to the Internet, so we made a bare clone of https://gerrit.googlesource.com/git-repo and copied it to our server. We modified the REPO_URL variable in repo to reference that server instead. When we do a 'repo init', however, we get a number of '[new tag ...]' messages that we do not get when doing the same command on a connected machine which accesses the main git-repo repository.
When using the main git-repo repository we get this
I noticed that that there is an initial get of 'clone.bundle' that we don't get from our repository. Clearly I've missed something in our setup. Can someone tell me what?
Also, we haven't been able to find a lot of documentation on using repo in non-android projects. Is there a resource someone can point us to?
Thanks.
Mark E. Hamilton
We have a manifest repository set up and so far we are able to get things to work as we expect. However, we have a requirement to be able to work on machines which are not connected to the Internet, so we made a bare clone of https://gerrit.googlesource.com/git-repo and copied it to our server. We modified the REPO_URL variable in repo to reference that server instead. When we do a 'repo init', however, we get a number of '[new tag ...]' messages that we do not get when doing the same command on a connected machine which accesses the main git-repo repository.
% repo init -m base.xml -u our_server:/git/manifests
Get our_server:/git/git-repo
remote: Counting objects: 3444, done.
remote: Compressing objects: 100% (1574/1574), done.
remote: Total 3444 (delta 1814), reused 3444 (delta 1814)
Receiving objects: 100% (3444/3444), 3.15 MiB | 0 bytes/s, done.
Resolving deltas: 100% (1814/1814), done.
From sierra-git:/git/git-repo
* [new branch] maint -> origin/maint
* [new branch] master -> origin/master
* [new branch] stable -> origin/stable
* [new tag] v1.0 -> v1.0
* [new tag] v1.0.1 -> v1.0.1
* [new tag] v1.0.2 -> v1.0.2
...
Get our_server:/git/manifests
...When using the main git-repo repository we get this
% repo init -m base.xml -u our_server:/git/manifests
Get https://gerrit.googlesource.com/git-repo/clone.bundle
Get https://gerrit.googlesource.com/git-repo
Get our_server:/git/manifests
remote: Counting objects: 54, done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 54 (delta 25), reused 0 (delta 0)
From sierra-git:/git/manifests
* [new branch] master -> origin/master
Your identity is: Mark E. Hamilton <mhamilt@sandia.gov>
If you want to change this, please re-run 'repo init' with --config-name
repo has been initialized in /sierra/dev/mhamilt/sntools_votd_7
I noticed that that there is an initial get of 'clone.bundle' that we don't get from our repository. Clearly I've missed something in our setup. Can someone tell me what?
Also, we haven't been able to find a lot of documentation on using repo in non-android projects. Is there a resource someone can point us to?
Thanks.
Mark E. Hamilton
David Pursehouse
Dec 16, 2015, 7:35:30 PM12/16/15
to mha...@sandia.gov, Repo and Gerrit Discussion
The clone.bundle file is a git bundle (see documentation [1]). I don't know if there's documentation of how to set it up in combination with Gerrit, but I imagine it's just a case of running git bundle on the repository and making the resulting file available over http(s).
The tags/branches being not listed seems to be a difference in git's behaviour when cloning from a bundle file vs. from a remote repository. For example in the following steps:
$ git clone clone.bundle git-repo
the git-repo repository is initialised and cloned from the bundle file, the branches and tags are not listed to the console, but are present in the repo.
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Magnus Bäck
Dec 17, 2015, 1:45:41 AM12/17/15
to mha...@sandia.gov, Repo and Gerrit Discussion
On Wednesday, December 16, 2015 at 19:26 CET,
mha...@sandia.gov wrote:
[...]
> We have a manifest repository set up and so far we are able to get
> things to work as we expect. However, we have a requirement to be
> able to work on machines which are not connected to the Internet, so
> we made a bare clone of https://gerrit.googlesource.com/git-repo and
> copied it to our server. We modified the REPO_URL variable in repo to
> reference that server instead.
You could also use Git's insteadOf mechanism to substitute the normal
upstream URL with your local one, but in most cases I'd expect it to
be easier to provision a patched repo wrapper script than a couple of
.gitconfig lines to all users.
[...]
> Also, we haven't been able to find a lot of documentation on using
> repo in non-android projects. Is there a resource someone can point
> us to?
There's nothing Android-specific about Repo, but it was originally built
to cater to the open sourcing of Android which of course has influenced
its feature set. Therefore I don't think there's a need for documenting
the use of Repo for things non-Android, but I would agree that Repo is
generally underdocumented.
--
Magnus Bäck | Software Engineer, Firmware Platform Tools
magnu...@axis.com | Axis Communications
mha...@sandia.gov wrote:
[...]
> We have a manifest repository set up and so far we are able to get
> things to work as we expect. However, we have a requirement to be
> able to work on machines which are not connected to the Internet, so
> we made a bare clone of https://gerrit.googlesource.com/git-repo and
> copied it to our server. We modified the REPO_URL variable in repo to
> reference that server instead.
upstream URL with your local one, but in most cases I'd expect it to
be easier to provision a patched repo wrapper script than a couple of
.gitconfig lines to all users.
[...]
> Also, we haven't been able to find a lot of documentation on using
> repo in non-android projects. Is there a resource someone can point
> us to?
to cater to the open sourcing of Android which of course has influenced
its feature set. Therefore I don't think there's a need for documenting
the use of Repo for things non-Android, but I would agree that Repo is
generally underdocumented.
--
Magnus Bäck | Software Engineer, Firmware Platform Tools
magnu...@axis.com | Axis Communications
Yuxuan Wang
Dec 17, 2015, 12:48:45 PM12/17/15
to David Pursehouse, mha...@sandia.gov, Repo and Gerrit Discussion
Repo just always try to append "/clone.bundle" to the git url before
cloning (in this case it's
https://gerrit.googlesource.com/git-repo/clone.bundle), if that file
exists it will download that file and clone from the bundle first,
then do a fetch to the git url. That clone.bundle file will actually
redirect to Google's CDN server, so it will offload some bandwidth
away from Google's git server, and also that's resumable and probably
faster. We generate the clone.bundle file periodically through some
cron job.
Since your git url is already in your local network, the whole
clone.bundle thing might not be that necessary. Also it might be
non-trivial for you to generate the clone.bundle file and make the url
work. The end result will be the same.
>> Your identity is: Mark E. Hamilton <mha...@sandia.gov>
--
ʎɥsıɟ
cloning (in this case it's
https://gerrit.googlesource.com/git-repo/clone.bundle), if that file
exists it will download that file and clone from the bundle first,
then do a fetch to the git url. That clone.bundle file will actually
redirect to Google's CDN server, so it will offload some bandwidth
away from Google's git server, and also that's resumable and probably
faster. We generate the clone.bundle file periodically through some
cron job.
Since your git url is already in your local network, the whole
clone.bundle thing might not be that necessary. Also it might be
non-trivial for you to generate the clone.bundle file and make the url
work. The end result will be the same.
ʎɥsıɟ
Mark E. Hamilton
Dec 17, 2015, 5:38:36 PM12/17/15
to Repo and Gerrit Discussion
On 12/16/15 23:45, Magnus Bäck wrote:
> On Wednesday, December 16, 2015 at 19:26 CET,
> mha...@sandia.gov wrote:
>
> [...]
> On Wednesday, December 16, 2015 at 19:26 CET,
> mha...@sandia.gov wrote:
>
> [...]
>> copied it to our server. We modified the REPO_URL variable in repo to
>> reference that server instead.
>
> You could also use Git's insteadOf mechanism to substitute the normal
> upstream URL with your local one, but in most cases I'd expect it to
> be easier to provision a patched repo wrapper script than a couple of
> .gitconfig lines to all users.
Thanks. What I'm considering is checking an environment variable in
>> reference that server instead.
>
> You could also use Git's insteadOf mechanism to substitute the normal
> upstream URL with your local one, but in most cases I'd expect it to
> be easier to provision a patched repo wrapper script than a couple of
> .gitconfig lines to all users.
repo, and if it's set using its value instead of the default for
REPO_URL. I'm not sure of the process for submitting that sort of patch
back, however.
--
----------------
Mark E. Hamilton
Engineering Sciences Center
Senior Member of Technical Staff
Sandia National Laboratories
505-844-7666
Mark E. Hamilton
Dec 22, 2015, 2:05:09 PM12/22/15
to Yuxuan Wang, David Pursehouse, Repo and Gerrit Discussion
Thanks. I had a closer look at the repo script, and the issue we are
having seems to be that it assumes that the server is using http
protocol, which we are not. We use ssh, and the _DownloadBundle function
will simply return w/o downloading the bundle if the URL doesn't start
with http[s].
I considered creating out own bundle (and modifying repo to use scp for
our server,) but at you suggest is seems to be unnecessary work for us.
I could probably just delete the tags from the bare clone, if there's no
other way to reduce that output. All I really want to do is prevent our
users from complaining about all that unnecessary, confusing output.
On 12/17/15 10:47, Yuxuan Wang wrote:
> Repo just always try to append "/clone.bundle" to the git url before
> cloning (in this case it's
> https://gerrit.googlesource.com/git-repo/clone.bundle), if that file
> exists it will download that file and clone from the bundle first,
> then do a fetch to the git url. That clone.bundle file will actually
> redirect to Google's CDN server, so it will offload some bandwidth
> away from Google's git server, and also that's resumable and probably
> faster. We generate the clone.bundle file periodically through some
> cron job.
>
> Since your git url is already in your local network, the whole
> clone.bundle thing might not be that necessary. Also it might be
> non-trivial for you to generate the clone.bundle file and make the url
> work. The end result will be the same.
having seems to be that it assumes that the server is using http
protocol, which we are not. We use ssh, and the _DownloadBundle function
will simply return w/o downloading the bundle if the URL doesn't start
with http[s].
I considered creating out own bundle (and modifying repo to use scp for
our server,) but at you suggest is seems to be unnecessary work for us.
I could probably just delete the tags from the bare clone, if there's no
other way to reduce that output. All I really want to do is prevent our
users from complaining about all that unnecessary, confusing output.
On 12/17/15 10:47, Yuxuan Wang wrote:
> Repo just always try to append "/clone.bundle" to the git url before
> cloning (in this case it's
> https://gerrit.googlesource.com/git-repo/clone.bundle), if that file
> exists it will download that file and clone from the bundle first,
> then do a fetch to the git url. That clone.bundle file will actually
> redirect to Google's CDN server, so it will offload some bandwidth
> away from Google's git server, and also that's resumable and probably
> faster. We generate the clone.bundle file periodically through some
> cron job.
>
> Since your git url is already in your local network, the whole
> clone.bundle thing might not be that necessary. Also it might be
> non-trivial for you to generate the clone.bundle file and make the url
> work. The end result will be the same.
Mark E. Hamilton
Dec 22, 2015, 8:41:51 PM12/22/15
to David Pursehouse, Repo and Gerrit Discussion
Thanks. Interestingly, when I clone from our local clone I don't get the
tags messages:
% git clone server:/git/git-repo
Initialized empty Git repository in /home/mhamilt/git-repo/.git/
However, repos is not doing a clone. It's actually doing an
init/config/fetch combination, which does produce the tag messages:
% git init --quiet
% git config remote.origin.url server:/git/git-repo
% git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
% git fetch origin +refs/heads/*:refs/remotes/origin/*
refs/tags/*:refs/tags/*
Passing --no-tags to 'git fetch' doesn't stop this, and I'm not sure
this would be the right thing to do anyway. Seems to me that these tags
aren't really necessary in your .repo directory.
On 12/16/15 17:35, David Pursehouse wrote:
> The clone.bundle file is a git bundle (see documentation [1]). I don't
> know if there's documentation of how to set it up in combination with
> Gerrit, but I imagine it's just a case of running git bundle on the
> repository and making the resulting file available over http(s).
>
> The tags/branches being not listed seems to be a difference in git's
> behaviour when cloning from a bundle file vs. from a remote repository.
> For example in the following steps:
>
> $ wget https://gerrit.googlesource.com/git-repo/clone.bundle
> $ git clone clone.bundle git-repo
>
> the git-repo repository is initialised and cloned from the bundle file,
> the branches and tags are not listed to the console, but are present in
> the repo.
>
> [1] https://git-scm.com/docs/git-bundle
tags messages:
% git clone server:/git/git-repo
Initialized empty Git repository in /home/mhamilt/git-repo/.git/
remote: Counting objects: 3444, done.
remote: Compressing objects: 100% (1574/1574), done.
remote: Total 3444 (delta 1814), reused 3444 (delta 1814)
Receiving objects: 100% (3444/3444), 3.15 MiB, done.
remote: Compressing objects: 100% (1574/1574), done.
remote: Total 3444 (delta 1814), reused 3444 (delta 1814)
Resolving deltas: 100% (1814/1814), done.
%
However, repos is not doing a clone. It's actually doing an
init/config/fetch combination, which does produce the tag messages:
% git init --quiet
% git config remote.origin.url server:/git/git-repo
% git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
% git fetch origin +refs/heads/*:refs/remotes/origin/*
refs/tags/*:refs/tags/*
remote: Counting objects: 3444, done.
remote: Compressing objects: 100% (1574/1574), done.
remote: Total 3444 (delta 1814), reused 3444 (delta 1814)
Receiving objects: 100% (3444/3444), 3.15 MiB | 0 bytes/s, done.
Resolving deltas: 100% (1814/1814), done.
From server:/git/git-repo
remote: Compressing objects: 100% (1574/1574), done.
remote: Total 3444 (delta 1814), reused 3444 (delta 1814)
Receiving objects: 100% (3444/3444), 3.15 MiB | 0 bytes/s, done.
Resolving deltas: 100% (1814/1814), done.
* [new branch] maint -> origin/maint
* [new branch] master -> origin/master
* [new branch] stable -> origin/stable
...
* [new branch] master -> origin/master
* [new branch] stable -> origin/stable
Passing --no-tags to 'git fetch' doesn't stop this, and I'm not sure
this would be the right thing to do anyway. Seems to me that these tags
aren't really necessary in your .repo directory.
On 12/16/15 17:35, David Pursehouse wrote:
> The clone.bundle file is a git bundle (see documentation [1]). I don't
> know if there's documentation of how to set it up in combination with
> Gerrit, but I imagine it's just a case of running git bundle on the
> repository and making the resulting file available over http(s).
>
> The tags/branches being not listed seems to be a difference in git's
> behaviour when cloning from a bundle file vs. from a remote repository.
> For example in the following steps:
>
> $ wget https://gerrit.googlesource.com/git-repo/clone.bundle
> $ git clone clone.bundle git-repo
>
> the git-repo repository is initialised and cloned from the bundle file,
> the branches and tags are not listed to the console, but are present in
> the repo.
>
> [1] https://git-scm.com/docs/git-bundle
--
----------------
Mark E. Hamilton
----------------
Mark E. Hamilton
Reply all
Reply to author
Forward
0 new messages