failing to get partial-copy to work

75 views
Skip to first unread message

Tony Finch

unread,
Mar 8, 2016, 8:25:43 AM3/8/16
to gito...@googlegroups.com
I'm failing to get partial-copy working. I haven't used any VREFs so far
so this is new territory for me...

One of my users wants to make a repo public, but keep some branches
private. We can't restrict read permissions per branch with gitolite, so
we need a separate public copy of the repo containing only the public
branches. We should be able to do this automatically with partial-copy.

So I have tried to produce a small scale demo example based on the
documentation. I think this is all the relevant stuff from my
gitolite.conf:

@public = gitweb daemon
@ssh = fanf2

repo @all
option deny-rules = 1

repo git/pc-private
desc = testing partial-copy - private
RW+ = @ssh

repo git/pc-public
desc = testing partial-copy - public
- u/ = @public
R = @all
- VREF/partial-copy = @all
config gitolite.partialCopyOf = git/pc-private

I cloned the new empty pc-private repo, created a couple of commits on the
master branch, and a commit on the u/private branch, and pushed it. Then I
cloned pc-public, and it was populated from pc-private. However, pc-public
included all the branches from pc-private, including the u/private branch
which I thought would be excluded.

It seems that this setup is backwards from what we want. (Maybe that's
also where my error is?) We want to be able to work on pc-private as
before, and when we push to a public branch in pc-private, the changes are
automatically duplicated to pc-public. However in my test pc-public was
not automatically updated until I pulled from it.

Whereas the partial-copy documentation suggests that both repos can be
writable, what we want is simpler because we don't need pc-public to be
writable.

Also the config above has screwed up permissions so that pc-public isn't
readable by gitweb as it should be. This is because of the deny-rules
setting.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Viking, North Utsire: Southerly 4 or 5, increasing 6 to gale 8. Slight or
moderate, occasionally rough. Fair then occasional rain. Good, occasionally
poor.

milk

unread,
Mar 8, 2016, 9:00:45 AM3/8/16
to Tony Finch, gitolite
On Tue, Mar 8, 2016 at 5:25 AM, Tony Finch <d...@dotat.at> wrote:
I'm failing to get partial-copy working. I haven't used any VREFs so far
so this is new territory for me...

One of my users wants to make a repo public, but keep some branches
private. We can't restrict read permissions per branch with gitolite, so
we need a separate public copy of the repo containing only the public
branches. We should be able to do this automatically with partial-copy.

So I have tried to produce a small scale demo example based on the
documentation. I think this is all the relevant stuff from my
gitolite.conf:

There are two partial-copies - a VREF and a trigger. Since you don't need pc-public writable, you don't need the VREF because: https://github.com/sitaramc/gitolite/blob/master/src/VREF/partial-copy#L3.

You will also notice that the partial-copy trigger requires has arguments: repo and user: https://github.com/sitaramc/gitolite/blob/master/src/triggers/partial-copy#L17. If I'm reading this correctly, this means that the user that triggered the partial-copy also determines what refs are copied from the main repo. This means that only those that need to read the partial copy should be using the partial copy. All private users should _not_ use the partial copy at all.

 
It seems that this setup is backwards from what we want. (Maybe that's
also where my error is?) We want to be able to work on pc-private as
before, and when we push to a public branch in pc-private, the changes are
automatically duplicated to pc-public. However in my test pc-public was
not automatically updated until I pulled from it.

Recreate the repo but create a new user that does not belong to the @ssh group. 


Also the config above has screwed up permissions so that pc-public isn't
readable by gitweb as it should be. This is because of the deny-rules
setting.

You need to give explicit R access to gitweb and daemon. @all does not include them. See http://gitolite.com/gitolite/conf.html#example-1

-milki

Tony Finch

unread,
Mar 8, 2016, 9:16:53 AM3/8/16
to milk, gitolite
milk <nites...@gmail.com> wrote:
>
> There are two partial-copies - a VREF and a trigger. Since you don't need
> pc-public writable, you don't need the VREF because:
> https://github.com/sitaramc/gitolite/blob/master/src/VREF/partial-copy#L3.

Hmf. I Based on this I think the existing partial-copy stuff is backwards
from what I need.

The trigger isn't right because it updates the partial copy on demand,
when it is accessed through gitolite. But the partial copy will usually be
accessed through git-daemon and gitweb, which don't invoke gitolite's
triggers. And I want the partial copy to be updated as soon as the main
repo is changed, but the VREF is designed to push in the opposite
direction.

> > It seems that this setup is backwards from what we want. (Maybe that's
> > also where my error is?) We want to be able to work on pc-private as
> > before, and when we push to a public branch in pc-private, the changes are
> > automatically duplicated to pc-public. However in my test pc-public was
> > not automatically updated until I pulled from it.
>
> Recreate the repo but create a new user that does not belong to the @ssh
> group.

I'm sorry I don't understand where you think I should add this new user to
my configuration.

> You need to give explicit R access to gitweb and daemon. @all does not
> include them. See http://gitolite.com/gitolite/conf.html#example-1

@all includes gitweb and daemon in my setup.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Irish Sea: Southwest 5, veering northwest 6 to gale 8 later. Mainly slight,
becoming moderate or rough later, but occasionally very rough later in south.
Rain or showers. Good, occasionally poor.

Sitaram Chamarty

unread,
Mar 8, 2016, 10:55:58 AM3/8/16
to milk, Tony Finch, gitolite
On 08/03/16 19:30, milk wrote:

> You will also notice that the partial-copy trigger requires has arguments: repo and user: https://github.com/sitaramc/gitolite/blob/master/src/triggers/partial-copy#L17. If I'm reading this correctly, this means that the user that triggered the partial-copy also determines what refs are copied from the main repo. This means that only those that need to read the partial copy should be using the partial copy. All private users should _not_ use the partial copy at all.

Well... I've just fixed the docs (not pushed yet so don't go looking
right away) to fix this. The example should have been closer to this:

repo foo # private repo
# user u2 gets no access here
RW+ = u1

repo bar # public repo
# branch sb should be denied to all here
- sb = @all # <=== this was earlier just "u2"

# give anyone any access you like
RW+ = u1 u2

# standard stuff
- VREF/partial-copy = @all
config gitolite.partialCopyOf = foo

then it won't matter if private user do use the partial-copy

Sitaram Chamarty

unread,
Mar 8, 2016, 10:57:00 AM3/8/16
to Tony Finch, milk, gitolite
On 08/03/16 19:46, Tony Finch wrote:
> milk <nites...@gmail.com> wrote:

>> There are two partial-copies - a VREF and a trigger. Since you don't need
>> pc-public writable, you don't need the VREF because:
>> https://github.com/sitaramc/gitolite/blob/master/src/VREF/partial-copy#L3.
>
> Hmf. I Based on this I think the existing partial-copy stuff is backwards
> from what I need.
>
> The trigger isn't right because it updates the partial copy on demand,
> when it is accessed through gitolite. But the partial copy will usually be
> accessed through git-daemon and gitweb, which don't invoke gitolite's
> triggers. And I want the partial copy to be updated as soon as the main
> repo is changed, but the VREF is designed to push in the opposite
> direction.

yes indeed; this is not what you want, I think. The VREF is merely
superfluous but the trigger is not even invoked. I think what you may
need is a much simpler setup.

Forget all the partial copy stuff, and try this (smoke tested):

1. conf

repo foo
RW+ = u1
option hook.post-receive = bikini
config gitolite.pc = bar

repo bar
- u/ = @all
R = @all

2. using [1], add a repo-specific "post-receive" hook called "bikini".
(Yeah I know it's not Friday but really, what else can you call a
hook that keeps the important bits hidden ;-)

The code is

#!/bin/bash

public_user=gitweb # or daemon, or indeed anything

pc=`git config --file $GL_REPO_BASE/$GL_REPO.git/config --get gitolite.pc`;
[ -z "$pc" ] && exit 0

unset GIT_DIR
cd $GL_REPO_BASE/$pc.git

while read old new ref
do
gitolite access -q $pc $public_user R $ref || continue
git fetch -f $GL_REPO_BASE/$GL_REPO.git $ref:$ref
done

Now, note that this does not cover for two important situations which
*may* be important to you: what happens if a branch that was considered
"public ok" suddenly becomes private. That, and a similar question on
tags, are handled by the partial-copy trigger (mostly thanks to Jan
Weitzel), which you may want to include in this script.

regards
sitaram

[1]: http://gitolite.com/gitolite/cookbook.html#v3.6-variation-repo-specific-hooks

Tony Finch

unread,
Mar 8, 2016, 11:03:03 AM3/8/16
to Sitaram Chamarty, milk, gitolite
Sitaram Chamarty <sita...@gmail.com> wrote:
>
> Forget all the partial copy stuff, and try this (smoke tested):

Marvellous, thanks!

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Humber, Thames, Dover: Southwest 5, backing south 6 or 7, perhaps gale 8
later. Slight or moderate, occasionally rough later. Occasional rain. Good,
occasionally poor.

Tony Finch

unread,
Mar 9, 2016, 4:05:06 PM3/9/16
to Sitaram Chamarty, gitolite
OK, I have got this working OK, I think.

Documentation: https://git.csx.cam.ac.uk/admin.html#partial_copy
Code: https://git.csx.cam.ac.uk/x/ucs/git/gitcam.git/blob/HEAD:/etc/gitolite-local/hooks/local/post-receive-partial-copy

The main difference from your suggestion is that (instead of fetching) I
am pushing from the master (private) repo to the partial-copy (public)
repo, because that makes git's "remote:" messages more self-explanatory.
(There's an example in the docs.)

Also, I have my own multi-hook thing which predates the
repo-specific-hooks feature. Mine only handles post-receive hooks,
but it allows them to be used with wild repos, and it allows multiple
post-receive hooks on a repo (e.g. both multimail and partial-copy).

I've improved it a bit to make it easier to change the set of hooks.
Previously it required all hooks to be linked from all repos; now only
the multi-dispatch hook needs linking from every repo.

Code: https://git.csx.cam.ac.uk/x/ucs/git/gitcam.git/blob/HEAD:/etc/gitolite-local/hooks/common/post-receive

Thanks for your help!

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Portland: Northwest gale 8 to storm 10, veering north 5 or 6. Very rough or
high, becoming moderate or rough. Rain then showers. Good, occasionally poor.

Sitaram Chamarty

unread,
Mar 9, 2016, 6:53:21 PM3/9/16
to Tony Finch, gitolite
On 10/03/16 02:35, Tony Finch wrote:
> OK, I have got this working OK, I think.
>
> Documentation: https://git.csx.cam.ac.uk/admin.html#partial_copy
> Code: https://git.csx.cam.ac.uk/x/ucs/git/gitcam.git/blob/HEAD:/etc/gitolite-local/hooks/local/post-receive-partial-copy
>
> The main difference from your suggestion is that (instead of fetching) I
> am pushing from the master (private) repo to the partial-copy (public)
> repo, because that makes git's "remote:" messages more self-explanatory.
> (There's an example in the docs.)

Nice! If I want to add a link to gitolite docs pointing to this
alternate, which one should I use?

And what about pushes to the partial copy repo? Are they allowed?

Tony Finch

unread,
Mar 10, 2016, 5:50:54 AM3/10/16
to Sitaram Chamarty, gitolite
> Nice! If I want to add a link to gitolite docs pointing to this
> alternate, which one should I use?

I expect the link to the code will be most useful, but it isn't very
helpful without the docs :-)

> And what about pushes to the partial copy repo? Are they allowed?

Well, it's OK if you don't mind them bein overwritten :-)
I should probably make a note about that in the docs.

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Trafalgar: North 5 to 7 decreasing 4 at times. Moderate or rough. Mainly fair.
Good, occasionally moderate.

Jan Remmet

unread,
Mar 15, 2016, 6:18:02 AM3/15/16
to Tony Finch, Sitaram Chamarty, gitolite
Hi Tony,
I tried your hook and it works fine. I found an issue while pushing tags. For my
setup only tags pointing to readable branches should be pushed to the
partial-copy.

I skip now all tags and push the other refs with --follow-tags. lightweight
tags gets lost, buts thats ok for my this use case.

Jan


-----8<-----
From c47a55f32b4c31852dfa1e7689daaa48be2023ac Mon Sep 17 00:00:00 2001
From: Jan Remmet <j.re...@phytec.de>
Date: Tue, 15 Mar 2016 10:00:01 +0100
Subject: [PATCH] pr_partial-copy: skip pushing tags directly

Tags can point to read restricted branches. So don't push them directly.
This has the drawback that lightweight tags get lost.

Signed-off-by: Jan Remmet <j.re...@phytec.de>
---
local/hooks/repo-specific/pr_partial-copy | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/local/hooks/repo-specific/pr_partial-copy b/local/hooks/repo-specific/pr_partial-copy
index b6b0599..e0ad79d 100755
--- a/local/hooks/repo-specific/pr_partial-copy
+++ b/local/hooks/repo-specific/pr_partial-copy
@@ -38,8 +38,11 @@ die "unknown partial-copy target repo $target"

while read old new ref
do
+ case $ref in
+ (refs/tags/*) continue
+ esac
gitolite access -q $GL_REPO $reader R $ref &&
- git push -f $GL_REPO_BASE/$target.git $ref:$ref
+ git push -f --follow-tags $GL_REPO_BASE/$target.git $ref:$ref
done

exit 0
--
1.9.1

Tony Finch

unread,
Mar 15, 2016, 7:45:07 AM3/15/16
to Jan Remmet, Sitaram Chamarty, gitolite
Jan Remmet <J.Re...@phytec.de> wrote:
>
> I tried your hook and it works fine. I found an issue while pushing tags. For my
> setup only tags pointing to readable branches should be pushed to the
> partial-copy.
>
> I skip now all tags and push the other refs with --follow-tags. lightweight
> tags gets lost, buts thats ok for my this use case.

This is an excellent point, and I think I will adopt your patch in my
installation.

Thanks very much for the helpful feedback!

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Irish Sea: East or northeast 4 or 5, becoming variable 3 at times, increasing
6 or 7 later in north. Smooth or slight, occasionally moderate in south. Fair.
Good.

Jan Remmet

unread,
Mar 15, 2016, 7:51:08 AM3/15/16
to Tony Finch, Sitaram Chamarty, gitolite
On Tue, Mar 15, 2016 at 11:18:00AM +0100, Jan Remmet wrote:
> On Wed, Mar 09, 2016 at 09:05:04PM +0000, Tony Finch wrote:
> > Documentation: https://git.csx.cam.ac.uk/admin.html#partial_copy
> > Code: https://git.csx.cam.ac.uk/x/ucs/git/gitcam.git/blob/HEAD:/etc/gitolite-local/hooks/local/post-receive-partial-copy
> Hi Tony,
> I tried your hook and it works fine. I found an issue while pushing tags. For my
> setup only tags pointing to readable branches should be pushed to the
> partial-copy.
>
> I skip now all tags and push the other refs with --follow-tags. lightweight
> tags gets lost, buts thats ok for my this use case.
I missed the case of only pushing an annotated tag...

Jan

Tony Finch

unread,
Mar 15, 2016, 7:53:45 AM3/15/16
to Jan Remmet, Sitaram Chamarty, gitolite
Jan Remmet <J.Re...@phytec.de> wrote:

> I missed the case of only pushing an annotated tag...

Ah, yes, tricky.

I thought when writing the code that the refex permissions should be
specified to cover tags as well as branches. But I didn't test that so
maybe it is too simplistic (and awkward for people writing gitolite.conf
clauses).

Tony.
--
f.anthony.n.finch <d...@dotat.at> http://dotat.at/
Trafalgar: Westerly or southwesterly 4 or 5, increasing 6 at times, becoming
variable 3 or 4 later. Moderate or rough. Fair. Moderate or good.

Jan Remmet

unread,
Mar 18, 2016, 11:04:40 AM3/18/16
to gito...@googlegroups.com
valid tags are reachably from a readable branch.

Signed-off-by: Jan Remmet <j.re...@phytec.de>
---
local/hooks/repo-specific/pr_partial-copy | 12 ++++++++++++
1 file changed, 12 insertions(+)

diff --git a/local/hooks/repo-specific/pr_partial-copy b/local/hooks/repo-specific/pr_partial-copy
index b6b0599..af3b06e 100755
--- a/local/hooks/repo-specific/pr_partial-copy
+++ b/local/hooks/repo-specific/pr_partial-copy
@@ -15,6 +15,15 @@
# See also http://article.gmane.org/gmane.comp.version-control.gitolite/4243

die() { echo 1>&2 "$@"; exit 1; }
+valid_tag() {
+ sha=$(git rev-list -1 $ref)
+ for branch in $(git for-each-ref refs/heads '--format=%(refname)')
+ do
+ gitolite access -q $target $reader R $branch || continue
+ [ "$(git merge-base $sha $branch)" = "$sha" ] && return 0
+ done
+ return 1
+}

# because we are dealing with two repositories
unset GIT_DIR
@@ -38,6 +47,9 @@ die "unknown partial-copy target repo $target"

while read old new ref
do
+ case $ref in
+ refs/tags/*) valid_tag || continue
+ esac
gitolite access -q $GL_REPO $reader R $ref &&
git push -f $GL_REPO_BASE/$target.git $ref:$ref
done
--
1.9.1

Jan Remmet

unread,
Mar 18, 2016, 11:11:26 AM3/18/16
to Tony Finch, Sitaram Chamarty, gitolite
On Tue, Mar 15, 2016 at 11:53:42AM +0000, Tony Finch wrote:
> Jan Remmet <J.Re...@phytec.de> wrote:
>
> > I missed the case of only pushing an annotated tag...
>
> Ah, yes, tricky.
>
> I thought when writing the code that the refex permissions should be
> specified to cover tags as well as branches. But I didn't test that so
> maybe it is too simplistic (and awkward for people writing gitolite.conf
> clauses).
>
In our environment there is no rule about naming tags on different branches.
So an tag may be pushed and pointing to on "internal" work in progress branch.
I send a patch which adds a check if the tags are reachable from the readable
branches. But the cost is searching for each pushed tag.

Jan
Reply all
Reply to author
Forward
0 new messages