HTTP pulls fail on 2.1.3

[Rob Heittman]

After moving from 2.1.2.4 to 2.1.3, projects that used to allow anonymous http fetch ... don't, anymore.  403 on old git (worked before upgrade):

76.161.124.98 - - [25/Jun/2010:19:46:41 +0000] "GET /p/partners/unccd-prais/info/refs HTTP/1.1" 403 1397 - "git/1.6.4.2"

Newer msysgit (worked before upgrade) shows this on client:

error: RPC failed; result=22, HTTP code = 503

this on server:

195.37.114.122 - - [25/Jun/2010:19:49:21 +0000] "GET /p/partners/unccd-prais/info/refs?service=git-upload-pack HTTP/1.1" 200 10593 - "git/1.7.0.2.msysgit.0.14.g956d7"

195.37.114.122 - - [25/Jun/2010:19:49:22 +0000] "POST /p/partners/unccd-prais/git-upload-pack HTTP/1.1" 200 - - "git/1.7.0.2.msysgit.0.14.g956d7"

The problem also appears with authenticated HTTP.  I'm seeing it in all my repositories.  Everything's working fine over SSH.

I'm a bit stumped.  Gerrit's installed straightforwardly using the internal Jetty; no complexities.  Project permissions allow +1 Read to Anonymous.  Anybody else seeing this with 2.1.3?

- R

[Shawn Pearce]

On Fri, Jun 25, 2010 at 12:59, Rob Heittman <rob.he...@solertium.com> wrote:
> After moving from 2.1.2.4 to 2.1.3, projects that used to allow anonymous
> http fetch ... don't, anymore. 403 on old git (worked before upgrade):
> 76.161.124.98 - - [25/Jun/2010:19:46:41 +0000] "GET
> /p/partners/unccd-prais/info/refs HTTP/1.1" 403 1397 - "git/1.6.4.2"

Oh, right. OK, I think I know what is going on.

If the project seems like it has branch level read permissions, it
denies older clients from fetching it over HTTP, because with the
older clients the server can't ensure the client is permitted to
access any particular resource. So we just blanket deny them.

Do you have any branch level READ (+1 or +2) permissions in the
project, or in any project it inherits from like -- All Projects --?

> Newer msysgit (worked before upgrade) shows this on client:
> error: RPC failed; result=22, HTTP code = 503
> this on server:
> 195.37.114.122 - - [25/Jun/2010:19:49:21 +0000] "GET
> /p/partners/unccd-prais/info/refs?service=git-upload-pack HTTP/1.1" 200
> 10593 - "git/1.7.0.2.msysgit.0.14.g956d7"
> 195.37.114.122 - - [25/Jun/2010:19:49:22 +0000] "POST
> /p/partners/unccd-prais/git-upload-pack HTTP/1.1" 200 - -
> "git/1.7.0.2.msysgit.0.14.g956d7"

I don't see how these correlate. The client says 503, but the server
log says 200. I don't think you are looking at the correct log
records here. Or, somebody isn't reporting accurately.

>  Anybody else seeing this with 2.1.3?

It does work on at least one 2.1.3 install, with a newer Git client.
http://egit.eclipse.org/r/ is a 2.1.3 server behind mod_proxy.

$ git clone http://egit.eclipse.org/r/p/jgit.git tmp_jgit_2
Cloning into tmp_jgit_2...
remote: Counting objects: 5347, done
remote: Compressing objects: 100% (5347/5347)
Receiving objects: 100% (5347/5347), 2.48 MiB | 875 KiB/s, done.
Resolving deltas: 100% (2586/2586), done.

[Shawn Pearce]

On Fri, Jun 25, 2010 at 13:49, Shawn Pearce <s...@google.com> wrote:
> On Fri, Jun 25, 2010 at 12:59, Rob Heittman <rob.he...@solertium.com> wrote:
>> After moving from 2.1.2.4 to 2.1.3, projects that used to allow anonymous
>> http fetch ... don't, anymore.  403 on old git (worked before upgrade):
>> 76.161.124.98 - - [25/Jun/2010:19:46:41 +0000] "GET
>> /p/partners/unccd-prais/info/refs HTTP/1.1" 403 1397 - "git/1.6.4.2"
>
> Oh, right.  OK, I think I know what is going on.
>
> If the project seems like it has branch level read permissions, it
> denies older clients from fetching it over HTTP, because with the
> older clients the server can't ensure the client is permitted to
> access any particular resource.  So we just blanket deny them.
>
> Do you have any branch level READ (+1 or +2) permissions in the
> project, or in any project it inherits from like -- All Projects --?

Ok, so I went back and looked at the code. We just disable support
for the older clients. It doesn't matter what the project access
controls say, its just disabled. According to git blame, that dates
back to the introduction of /p/ support, in Gerrit 2.1.2, and the
related code hasn't been touched since.

So, we've never supported the older clients. Its at least not a regression.

[Rob Heittman]

Thanks for the quick response, Shawn.  Short answer: OpenJDK was the culprit.

Longer answer:

The insane condition with the client reporting 503 and the server reporting 200 really happens with OpenJDK and 2.1.3, and right on up to git 1.7.1 on the client.

It didn't occur in previous Gerrit, since this particular instance had OpenJDK selected in its config file all along.  The instance was installed with Ubuntu Lucid, which doesn't have a Sun JDK by default, and although I added the Sun JDK after the fact, I never corrected the gerrit.config -- it kept on using OpenJDK.  Amazingly, this is the first problem I noticed, which is pretty good compared to my other experiences trying to use OpenJDK on-purpose.

If OpenJDK compatibility is a desirable thing, I'm happy to set this up reproducibly and do some further analysis on the crazy behavior.  But I'm not too fussed about it myself.

As you pointed out, 1.6 clients are still categorically blocked from http pulls with a 403, but I'm perfectly OK with that.

[Shawn Pearce]

On Sat, Jun 26, 2010 at 05:42, Rob Heittman <rob.he...@solertium.com> wrote:
> Thanks for the quick response, Shawn.  Short answer: OpenJDK was the
> culprit.

Ouch.

> Longer answer:
> The insane condition with the client reporting 503 and the server reporting
> 200 really happens with OpenJDK and 2.1.3, and right on up to git 1.7.1 on
> the client.

This shouldn't happen.

> If OpenJDK compatibility is a desirable thing, I'm happy to set this up
> reproducibly and do some further analysis on the crazy behavior.  But I'm
> not too fussed about it myself.

It is. We should be able to run on OpenJDK. But I suspect almost
everyone runs on the Sun JDK, and that's why we have never noticed
this. Its probably not a JGit or Gerrit Code Review issue, but a
problem with Jetty on OpenJDK, possibly caused by the way OpenJDK
implements NIO?

I don't have the time to do the legwork to figure out what is wrong
with OpenJDK here. But if someone else manages to understand the
problem and can point me in the right direction, I can try to get it
fixed.

[Rob Heittman]

I also suspect the Jetty update in 2.1.3 was the immediate trigger of the changed behavior.  What I'll do is clone my Gerrit instance and run a copy in OpenJDK, then do a little comparative wiresharking to see what's really going on, which might give me a hint.  I can revert Jetty pretty easily.  Lazyweb question: does Gerrit already have an easy way to dump request/response guts?  I'm looking for a way to spot any overt discrepancy between what's arriving in / sent from Gerrit vs. what's actually being transmitted on the wire.

[Shawn Pearce]

On Sat, Jun 26, 2010 at 20:50, Rob Heittman <rob.he...@solertium.com> wrote:
> I also suspect the Jetty update in 2.1.3 was the immediate trigger of the
> changed behavior.

Oh, that could be the cause, I forgot we had bumped up to a newer Jetty. :-)

> What I'll do is clone my Gerrit instance and run a copy
> in OpenJDK, then do a little comparative wiresharking to see what's really
> going on, which might give me a hint.  I can revert Jetty pretty easily.

Thanks. Let us know if you learn anything interesting.

>  Lazyweb question: does Gerrit already have an easy way to dump
> request/response guts?  I'm looking for a way to spot any overt discrepancy
> between what's arriving in / sent from Gerrit vs. what's actually being
> transmitted on the wire.

Unfortunately, no.

[kenwenzel]

We're also affected by this issue.
I filed a bug report on this: http://code.google.com/p/gerrit/issues/detail?id=614

Gerrit's error log contains the following entry:

[2010-06-29 09:10:38,209] ERROR org.eclipse.jetty.util.log : EXCEPTION
com.google.inject.ProvisionException: Guice provision errors:

1) Error in custom provider, com.google.inject.OutOfScopeException:
Cannot access scoped object. Either we are not currently inside an
HTTP Servlet request, or you may have forgotten to apply com.goog
le.inject.servlet.GuiceFilter as a servlet filter for this request.
while locating com.google.gerrit.server.CurrentUser

1 error
at com.google.inject.InjectorImpl$4.get(InjectorImpl.java:767)
at
com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.getExecutor(ProjectQoSFilter.java:
147)
at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.access
$100(ProjectQoSFilter.java:68)
at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter
$TaskThunk.onTimeout(ProjectQoSFilter.java:222)
at
org.eclipse.jetty.server.AsyncContinuation.expired(AsyncContinuation.java:
401)
at org.eclipse.jetty.server.AsyncContinuation
$AsyncEventState.run(AsyncContinuation.java:893)
at org.eclipse.jetty.util.thread.QueuedThreadPool
$2.run(QueuedThreadPool.java:437)
at java.lang.Thread.run(Thread.java:636)
Caused by: com.google.inject.OutOfScopeException: Cannot access scoped
object. Either we are not currently inside an HTTP Servlet request, or
you may have forgotten to apply com.google.inject.servlet.
GuiceFilter as a servlet filter for this request.
at
com.google.inject.servlet.GuiceFilter.getContext(GuiceFilter.java:132)
at
com.google.inject.servlet.GuiceFilter.getRequest(GuiceFilter.java:118)
at com.google.inject.servlet.ServletScopes
$1$1.get(ServletScopes.java:48)
at
com.google.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:
48)
at com.google.inject.InjectorImpl$4$1.call(InjectorImpl.java:
758)
at
com.google.inject.InjectorImpl.callInContext(InjectorImpl.java:804)
at com.google.inject.InjectorImpl$4.get(InjectorImpl.java:754)
... 7 more