Gerrit 2.7: Can clone via ssh but not http

[Kate Ebneter]

Having solved my earlier problem, I now face a somewhat more intractable one.

Configuration:

Gerrit 2.7 in standalone daemon, installed according to the instructions. Behind an Apache HTTPD proxy server.
OS: Ubuntu oneiric 11.10 (running in AWS EC2)
Authentication: LDAP (working :-) )

gerrit.config (obfuscated, but nothing important changed):

[gerrit]
    basePath = /myco/source/git-repo
    canonicalWebUrl = http://source.b.myco.com
[database]
    type = mysql
    hostname = db.myco.com
    database = reviewdb2
    username = gerrit2
[core]
        packedGitOpenFiles = 4096
        packedGitLimit = 1g
        packedGitWindowSize = 16k
[receive]
        maxObjectSizeLimit = 25m
[auth]
        type = LDAP
        trustContainerAuth = true
[ldap]
        server = ldap://auth.int.myco.com
        referral = follow
        accountBase = ou=people,dc=myco,dc=com
        accountFullName = ${cn}
        accountEmailAddress = ${email}
        accountSshUserName = ${uid}
        groupBase = ou=groups,dc=myco,dc=com
        groupPattern = (cn=${groupname})
        groupMemberPattern = (uniqueMember=${dn})
[sendemail]
        enable = true
        from = Code Review ($user) <bu...@myco.com>
        smtpServer = localhost
        smtpServerPort = 25
        smtpEncryption = none
[container]
    user = gerrit2
    javaHome = /myco/build-tools/jdk1.6.0_30-x64/jre
[sshd]
    listenAddress = *:29418
        threads = 24
        batchThreads = 2
        # Maximum number of concurrent SSH sessions that a user account may have open. 0 -> no limit
        maxConnectionsPerUser = 0
[httpd]
    listenUrl = proxy-http://*:8080/
    requestLog = true
[cache]
    directory = cache
[cache "web_sessions"]
        maxAge = 30 days
[gitweb]
        url = https://source.myco.com/view/
[user]
        name = Gerrit Code Review
        email = bu...@myco.com

Apache proxy info (again obfuscated, but only where necessary):

<VirtualHost *:443>
  ServerName source.b.myco.com

  SSLEngine on
  SSLCertificateFile STAR_myco_com.crt
  SSLCertificateKeyFile STAR_myco_com.key
  SSLCertificateChainFile STAR_myco_com.ca-bundle

  ProxyRequests Off
  ProxyVia Off
  ProxyPreserveHost On

  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  RewriteEngine On

# Snipped out some gitweb and other stuff — we don't run gitweb out of gerrit but separately.

 # Gerrit itself
  # <project>.git -> /p/<project>.git (and extra stuff like /p/<project>.git/info/refs)
  AllowEncodedSlashes On
  #ProxyPassMatch /([[:alnum:]-]+)\.git(.*) http://source.b.int.myco.com:8080/r/$1.git$2
  ProxyPass / http://source.b.int.myco.com:8080/ nocanon
  ProxyPassReverse / http://source.b.int.myco.com:8080/
  #ProxyPass /r/ http://source.b.int.myco.com:/8080/r/ nocanon
</VirtualHost>

I should note that some of this has been copied from our original gerrit server (which is 2.4.x) and I've tried various options to get to this point; using the options with the /r/s in them (as in the installation documents) results in not being able to bring up the web page. :-|

What works:

I can log in to the UI at https://source.b.myco.com and do various gerrit-like things there.
I can clone using git clone ssh://source.b.myco.com/my-repo.git.

What doesn't work:

git clone https://source.b.myco.com/my-repo             # This is the URL suggested by the web page
git clone https://source.b.myco.com/my-repo.git

Response is:

$ git clone https://source.b.myco.com/my-repo
Cloning into 'my-repo'...
error: The requested URL returned error: 403 while accessing https://source.b.myco.com/my-repo/info/refs
fatal: HTTP request failed

It doesn't seem to matter if I use proxy-http:// or proxy-https:// in the httpd.listenUrl setting./httpd.

I know that others have encountered this before and there is an issue filed requesting better documentation of how to get this to work. Has anyone gotten this to work?

Many thanks,
Kate Ebneter, one very frustrated SCM engineer

[Matthias Sohn]

shouldn't the canonical URL be

s/canonicalWebUrl = http://source.b.myco.com/canonicalWebUrl = https://source.b.myco.com/

--

Matthias 

[Shawn Pearce]

On Thu, Sep 26, 2013 at 5:25 PM, Kate Ebneter <kebn...@gmail.com> wrote:
> Having solved my earlier problem, I now face a somewhat more intractable
> one.
>
> Configuration:
>
> Gerrit 2.7 in standalone daemon, installed according to the instructions.
> Behind an Apache HTTPD proxy server.
> OS: Ubuntu oneiric 11.10 (running in AWS EC2)
> Authentication: LDAP (working :-) )
>

...

> [auth]
> type = LDAP
> trustContainerAuth = true

With trustContainerAuth = true your Apache server needs to be
performing an authentication step. I see no authentication
configuration in your quoted Apache config:

This block should be doing some sort of authentication for Git
requests. Or turn off trustContainerAuth.

[Kate Ebneter]

On Thursday, September 26, 2013 11:59:09 PM UTC-7, Shawn Pearce wrote:

On Thu, Sep 26, 2013 at 5:25 PM, Kate Ebneter <kebn...@gmail.com> wrote:
> Having solved my earlier problem, I now face a somewhat more intractable
> one.
>
> Configuration:

... 

...
> [auth]
>         type = LDAP
>         trustContainerAuth = true

With trustContainerAuth = true your Apache server needs to be
performing an authentication step. I see no authentication
configuration in your quoted Apache config:

> Apache proxy info (again obfuscated, but only where necessary):

<snip>

 

This block should be doing some sort of authentication for Git
requests. Or turn off trustContainerAuth.

Ahhhh, thank you so much. The guy who set up the previous installation insisted that was needed — but that configuration was rather ... baroque, to put it mildly.

Removing trustContainerAuth did, indeed, fix it. Much obliged — and, by the way, very impressed with the latest release, it's a big improvement over 2.4 and it's come a long way since I first investigated gerrit way back when.

 

[Shawn Pearce]

On Fri, Sep 27, 2013 at 1:19 AM, Kate Ebneter <kebn...@gmail.com> wrote:
>
> On Thursday, September 26, 2013 11:59:09 PM UTC-7, Shawn Pearce wrote:
>>
>> On Thu, Sep 26, 2013 at 5:25 PM, Kate Ebneter <kebn...@gmail.com> wrote:
>> > Having solved my earlier problem, I now face a somewhat more intractable
>> > one.
>> >
>> > Configuration:
>> ...
>>
>> ...
>> > [auth]
>> > type = LDAP
>> > trustContainerAuth = true
>>
>> With trustContainerAuth = true your Apache server needs to be
>> performing an authentication step. I see no authentication
>> configuration in your quoted Apache config:
>>
>> > Apache proxy info (again obfuscated, but only where necessary):
>> <snip>
>
>
>>
>> This block should be doing some sort of authentication for Git
>> requests. Or turn off trustContainerAuth.
>
>
> Ahhhh, thank you so much. The guy who set up the previous installation
> insisted that was needed — but that configuration was rather ... baroque, to
> put it mildly.

He may have been insisting this is required to get behavior users expect.

With trustContainerAuth = true, git clients can use the user's LDAP
password, and Apache gets configured to check the LDAP password.

With trustContainerAuth = false/unset, git clients have to use the
Gerrit special HTTP password in the web UI. Which may be confusing to
some users, since it differs from the LDAP account.

> Removing trustContainerAuth did, indeed, fix it. Much obliged — and, by the
> way, very impressed with the latest release, it's a big improvement over 2.4
> and it's come a long way since I first investigated gerrit way back when.

Thanks. I think 2.8 is going to be even more interesting. :-)

[Kate Ebneter]

>> This block should be doing some sort of authentication for Git

>> requests. Or turn off trustContainerAuth.
>
>
> Ahhhh, thank you so much. The guy who set up the previous installation
> insisted that was needed — but that configuration was rather ... baroque, to
> put it mildly.

He may have been insisting this is required to get behavior users expect.

With trustContainerAuth = true, git clients can use the user's LDAP
password, and Apache gets configured to check the LDAP password.

With trustContainerAuth = false/unset, git clients have to use the
Gerrit special HTTP password in the web UI. Which may be confusing to
some users, since it differs from the LDAP account.

Ah, yes, that was, indeed, the case. And, alas, still is; I am told it is not acceptable to require separate passwords for HTTP and other access.

The current (gerrit 2.4) configuration runs gerrit inside Jetty, which handles the LDAP password checking. There is an Apache gateway that sits in front of the gerrit server and runs the reverse proxy (the whole thing is in an AWS Virtual Private Cloud; the gateways control access to the interior of the VPC). So I set up a clone of the existing configuration only with gerrit 2.7 instead. From the Apache side, the only differences are the inclusion of "AllowEncodedSlashes On" and the "nocanon" option on the ProxyPass directive. On the Jetty side, there are no differences except the user name, password, and database name. The only change from the recommended configuration for gerrit is that the WEB-INF/web.xml file includes the necessary configuration for LDAP and there are, of course, a few other configuration changes associated with that. This configuration worked perfectly with gerrit 2.4.

With gerrit 2.7, an attempt to connect to http://source.myco.com/ gets an infinite redirection loop. It looks like something is adding another '/' to the string, so that the browser address bar turns "http://source.myco.com/" into "http://source.myco.com//". Looking at various Apache logs suggests that this is jetty — jetty returns a "301":

source.myco.com:443 50.xxx.xxx.xxx - - [29/Sep/2013:20:33:05 -0700] "GET // HTTP/1.1" 301 277 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:23.0) Gecko/20100101 Firefox/23.0"

Has anyone seen anything similar? I promise when I get this working that I will document the configuration on my blog (buildengineer.org) and donate the documentation to the Gerrit project to help others in this situation!

Thanks,
Kate (who is not usually this helpless, really!)

[Shawn Pearce]

Stupid question, but gerrit.canonicalWebUrl is set to point to the
server? We have seen weird behavior when this was unset and LDAP is
used, but I thought we had fixed these issues with 2.7.

Is gerrit.canonicalWebUrl using https:// but the http.listenUrl isn't
proxy-https:// or Apache isn't telling Jetty its a secure connection?
Another place we do blind redirects is to try and upgrade from
plaintext HTTP to HTTPS.

[Kate Ebneter]

On Sunday, September 29, 2013 11:48:01 PM UTC-7, Shawn Pearce wrote:

On Sun, Sep 29, 2013 at 8:50 PM, Kate Ebneter <kebn...@gmail.com> wrote:

<snip>

gerrit.canonicalWebUrl is set to https://source.myco.com
 

Is gerrit.canonicalWebUrl using https:// but the http.listenUrl isn't
proxy-https:// or Apache isn't telling Jetty its a secure connection?
Another place we do blind redirects is to try and upgrade from
plaintext HTTP to HTTPS.

httpd.listenUrl is set to proxy-https://*:8080

This gave me a clue, however: I changed gerrit.canonicalWebUrl to http://source.myco.com (not https://) ... and it works! Oy, vey.

I freely admit that some of this is black magic to me — I'm a build engineer, not a network/web admin — but it clearly has something to do with the relationship between this:

<VirtualHost *:443>
  ServerName source.myco.com

  SSLEngine on
  SSLCertificateFile STAR_myco_com.crt
  SSLCertificateKeyFile STAR_myco_com.key
  SSLCertificateChainFile STAR_myco_com.ca-bundle

  ProxyRequests Off
  ProxyVia Off
  ProxyPreserveHost On

  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

...

  ## Gerrit itself

  AllowEncodedSlashes On
  ProxyPassMatch /([[:alnum:]-]+)\.git(.*) http://source.int.myco.com:8080/r/$1.git$2
  ProxyPass / http://source.int.myco.com:8080/ nocanon
  ProxyPassReverse / http://source.int.myco.com:8080/
</VirtualHost>

and the gerrit.config data. But it works, I'm happy, thank you thank you, and I *will* write this up on my blog!

Thanks,
Kate