Cannot clone All-Projects over HTTP on 2.10.2

[Chris Mackey]

Hey guys, I'm running a test instance of Gerrit locally, but I'm having a problem with this version that I haven't had before. I do not seem to be able to clone All-Projects over HTTP. I can clone it over SSH, but I'm getting missing object errors during the checkout over HTTP. The gerrit.config settings:

[gerrit]
    basePath = git
    canonicalWebUrl = http://127.0.0.1:6003/
[database]
    type = h2
    database = db/ReviewDB
[index]
    type = LUCENE
[auth]
    type = DEVELOPMENT_BECOME_ANY_ACCOUNT
[sendemail]
    smtpServer = localhost
[container]
    user = chris
    javaHome = /usr/lib/jvm/java-7-oracle/jre
[sshd]
    listenAddress = *:6004
[httpd]
    listenUrl = http://*:6003/
[cache]
    directory = cache


So when I try to clone All-Projects, I get this:

git clone http://ad...@127.0.0.1:6003/All-Projects
Cloning into 'All-Projects'...
remote: Counting objects: 4, done
remote: Finding sources: 100% (4/4)
remote: Total 4 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
Checking connectivity... done.
error: Trying to write ref HEAD with nonexistent object 0000000000000000000000000000000000000000
fatal: Cannot update the ref 'HEAD'.
Unexpected end of command stream

I was pretty sure this was new behaviour, so I fired up an instance of Gerrit 2.9.4 with the same settings, but it worked fine:

git clone http://ad...@127.0.0.1:6001/All-Projects
Cloning into 'All-Projects'...
remote: Counting objects: 4, done
remote: Finding sources: 100% (4/4)
remote: Total 4 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
Checking connectivity... done.
Note: checking out '82aae1acc46c691fa0be700b8312b15ea6c18a7a'.

I suspect that I'm not being authenticated properly on clone over HTTP in either case however, because even when I use spurious usernames, it still works. Additionally, I can run ls-remote and fail to see refs/meta/config:

git ls-remote
From http://ad...@127.0.0.1:6001/All-Projects
82aae1acc46c691fa0be700b8312b15ea6c18a7a    HEAD


So this seems to be 2 different problems:

1. Git over HTTP reads are not being authenticated on either version
2. The behaviour of unauthenticated users has changed from 2.9.4 to 2.10.2. The newer release will prevent access where that did not happen before.

On 1, I'm suspecting some kind of configuration issue? I've tried setting various configuration parameters across various versions (somewhat haphazardly however, such that I cannot reliably list off the various combinations I've tried) but noticed no change in this behaviour. I went as far as stepping through the execution on the VM for both versions, but in either case, the user passed on to ProjectControl was always ANONYMOUS. Am I missing something terribly obvious here? It's behaving as if access controls are not being applied on the read level for HTTP, but at no point has any documentation I've seen indicated this is the case.

On 2, I compared the default access control rules shipped with 2.9.4 and 2.10.2 and could not see a difference between the two - but there clearly is a difference in behaviour. Again, I've looked over the release notes and couldn't find anything that immediately shouts out to me as being related.

I'd appreciate any insight into these two issues I'm having.

[Chris Mackey]

To keep this up to date, I've got a bit further with issue #1. It appears I'm falling back to the anonymous code path within Gerrit due to a failure of the client to send an Authorization header. This is verified below:

export GIT_CURL_VERBOSE=1
git clone http://127.0.0.1:6003/All-Projects
Cloning into 'All-Projects'...
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6003 (#0)
> GET /All-Projects/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: 127.0.0.1:6003
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Mon, 30 Mar 2015 17:06:28 GMT
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-advertisement
< Content-Length: 232
<
* Connection #0 to host 127.0.0.1 left intact
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6003 (#1)
> POST /All-Projects/git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: 127.0.0.1:6003
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 124

* upload completely sent off: 124 out of 124 bytes
< HTTP/1.1 200 OK
< Date: Mon, 30 Mar 2015 17:06:28 GMT
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked

<
remote: Counting objects: 4, done
remote: Finding sources: 100% (4/4)
remote: Total 4 (delta 0), reused 0 (delta 0)

* Connection #1 to host 127.0.0.1 left intact

Unpacking objects: 100% (4/4), done.
Checking connectivity... done.
error: Trying to write ref HEAD with nonexistent object 0000000000000000000000000000000000000000
fatal: Cannot update the ref 'HEAD'.
Unexpected end of command stream

[Jan Kundrát]

On Monday, 30 March 2015 19:09:03 CEST, Chris Mackey wrote:
> To keep this up to date, I've got a bit further with issue #1. It appears
> I'm falling back to the anonymous code path within Gerrit due to a failure
> of the client to send an Authorization header. This is verified below:

Are you sure that your Git configuration is set up to send authentication
even to this completely new server which listens on a different port? Check
your ~/.netrc .

With kind regards,
Jan

--
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

[Chris Mackey]

Hey Jan

I've tried both with and without an entry in ~/.netrc. The only difference that makes it the output of the Git client with GIT_CURL_VERBOSE=1, where it says when it cannot be found there. It appears to make no difference to the behaviour.

[Chris Mackey]

On a bit of a hunch, I've tried something a little unusual, with more unexpected results. It looks like when the Git client is sending its initial request, Gerrit is not returning with headers required for digest authentication to continue. So, remembering from the API documentation that REST calls that must be authenticated an "/a/" is required before the rest path, I tried the same thing with a new repository.

Firist I created two projects "a/WithAuthentication" and "WithoutAuthentication" as parent repos, and made a trivial change to the ACL rules in them just to ensure they did have commits in refs/meta/config.

Next, I try to clone the "WithoutAuthentication":

git clone http://ad...@127.0.0.1:6003/WithoutAuthentication
Cloning into 'WithoutAuthentication'...
remote: Counting objects: 6, done
remote: Finding sources: 100% (6/6)
remote: Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.

Checking connectivity... done.
error: Trying to write ref HEAD with nonexistent object 0000000000000000000000000000000000000000
fatal: Cannot update the ref 'HEAD'.
Unexpected end of command stream

Next, WithAuthentication:

git clone http://ad...@127.0.0.1:6003/a/WithAuthentication
Cloning into 'WithAuthentication'...
Password for 'http://ad...@127.0.0.1:6003':
remote: Counting objects: 6, done
remote: Finding sources: 100% (6/6)
remote: Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
Checking connectivity... done.
Note: checking out '0c7d7b5ee818aa5fc6588a5666777b690b20624e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

Of course, I can't have all my projects in a "/a/" subdirectory. I tried modifying the canonicalWebUrl to have the /a/ in it, in case that made any difference, but it did not.

At this point, I'm not really sure if this is something that I have done wrong in configuration or not. It seems very strange to me that you must manually create repositories in a "a/" subfolder to make use of http authentication - it's certainly not something that I'd picked up from the documentation.

[Jan Kundrát]

Hi Chris,
the default settings in a fresh installation allow anonymous read access to
all repos. A side effect of this is that Gerrit won't ask for password if
none is needed, no matter whether you add your username into the URL.

When I remove this rule which enables read-only access to everything by
default, Gerrit starts asking for credentials.

This gets interesting when I remove that default rule, but add another rule
which e.g. allows read-only access to, say, refs/tags/*. In that case,
Gerrit won't ask for my credentials when `git ls-remote`ing over HTTP.

Seems that you've found a bug. Confirmed on 2.10.2.

Cheers,

[Jan Kundrát]

So I digged into Git's source a bit. Git can configure curl to start the
HTTP auth even if not requested (that's the "int proactive_auth" argument
to http_init() in http.c. Turns out that it gets set only when doing
pushes, and all other places hardcode it to zero (false). So here's how I
understand the interaction of these things:

1) When doing read-only operations, git (the C-implementation which most
users have) sends credentials only when requested by the remote.
2) If a repo allows anonymous read access to at least some refs, Gerrit
doesn't really know whether it's supposed to request credentials via HTTP
error 401, so it simply assumes that it's a client's business to decide.
3) Gerrit wants to support both anonymous and authenticated access to a
given repo via a single URL endpoint.

It seems to me that these are in conflict *IF* you have a repo where some
refs are OK to be anonymously read, while others can only be accessed by
authenticated and authorized users. Do you really need to use that? Note
that it's OK to have repositories which are not readable by anonymous
users; what's broken is mixing anonymously-accessible and private refs in a
single repository.

There's also that mess where apparently "/a/" is used as a prefix for
authenticated access to the REST API, and that git-over-HTTP access to
projects is also allwoed at the root, nut just below /p/. This means that
there's a certain ambiguity which results in weird behavior. You've already
reported that stuff starts working "correctly" for you if a project is
named "a/something". A side effect of this is that users cannot access any
project whose name starts with "a/", at least when copy-pasting the URLs
from Gerrit's web.

Hope this helps,

[Chris Mackey]

Thanks Jan, that does make a lot more sense now!

What's curious about this is that if you have rule allowing an anonymous user access to a branch which doesn't even exist, this still manifests itself. All-Projects for example only has the refs/meta/config branch, but the anonymous rule that could be applied to branches that don't even exist still seems to prevent Gerrit for asking for credentials. It could be possible at this point to work out which ACL rules on a repo are actually not applicable by working out what actually exists on the repo, but I imagine this sort of thing would be quite expensive to do on repositories with a large number of refs.

It's also interesting that the behaviour of 2.9.4 is quite different! 2.9.4 also failed to authenticate users properly, but it served the All-Projects repository anyway - without prompting for credentials it would offer the contents of refs/meta/config regardless of the fact that that user didn't have access to them. The checked out repo could not see the refs/meta/config branch with ls-remote, but it would fetch HEAD regardless, which is effectively the same thing.

Do you happen to know if either the failure to authenticate or creating a repo under a "/a/" subdirectory are known bugs? I've been searching the issues list, but it's proven so far difficult to refine the search down to something manageable.