HTTP 503 while fetch

[Mah]

Hello,

We are testing http fetching from gerrit by multiple number of hosts and we got problem that for 100 fetches in the same time part of them fails with this reason:

Fetching origin

error: RPC failed; result=22, HTTP code = 503

fatal: The remote end hung up unexpectedly

error: Could not fetch origin

First we did this with apache+jetty (proxy-http), next jetty, we changed config values for maxThreads, maxQueued and that didn't fix anything. 

Logs:

- error_log is clear

- httpd_log for gerrit shows 503 (when jetty)

- access_log for apache shows 503 (when proxy)

ssh -p 29418 {username}@{server-host} gerrit show-queue while fetching shows:

Task State Command
------------------------------------------------------------------------------
00d60327 git-upload-pack repo (-----)
801193ec git-upload-pack repo (-----)
e002cf9d git-upload-pack repo (-----)
c0078bad git-upload-pack repo (-----)
00fda3ac git-upload-pack repo (-----)
2000e796 git-upload-pack repo (-----)
40f39be5 git-upload-pack repo (-----)
a037f788 git-upload-pack repo (-----)
803c336e git-upload-pack repo (-----)
e03def6a git-upload-pack repo (-----)
204b07fd git-upload-pack repo (-----)
004843fe git-upload-pack repo (-----)
404e3b0f git-upload-pack repo (-----)
8023d354 git-upload-pack repo (-----)
4025db5c git-upload-pack repo (-----)
a0893768 git-upload-pack repo (-----)
808e736e git-upload-pack repo (-----)
e08f2f6b git-upload-pack repo (-----)
c0946b86 git-upload-pack repo (-----)
207d4734 git-upload-pack repo (-----)
007a833e git-upload-pack repo (-----)
60833f4f git-upload-pack repo (-----)
40807b42 git-upload-pack repo (-----)
3c2fa285 14:40:04.026 Log File Compressor
------------------------------------------------------------------------------
24 tasks
Task State Command
------------------------------------------------------------------------------
603b3fcc git-upload-pack repo (-----)
40387bcf git-upload-pack repo (-----)
802d1391 git-upload-pack repo (-----)
e01e4f5a git-upload-pack repo (-----)
c0230ba6 git-upload-pack repo (-----)
3c2fa285 14:40:04.026 Log File Compressor
------------------------------------------------------------------------------
6 tasks
Task State Command
------------------------------------------------------------------------------
3c2fa285 14:40:04.026 Log File Compressor
------------------------------------------------------------------------------
1 tasks

Any idea what we missed or how to fix that?

Client git: 1.7.4

Server git: 1.7.6

Gerrit: 2.2.1

java version "1.6.0"; IcedTea Runtime Environment (build 1.6.0-b08); OpenJDK 64-Bit Server VM (build 1.6.0-b08, mixed mode)

[Shawn Pearce]

On Fri, Jul 1, 2011 at 05:11, Mah <szymon...@gmail.com> wrote:
> We are testing http fetching from gerrit by multiple number of hosts and we
> got problem that for 100 fetches in the same time part of them fails with
> this reason:
> Fetching origin
> error: RPC failed; result=22, HTTP code = 503
> fatal: The remote end hung up unexpectedly
> error: Could not fetch origin

503 means the server is overloaded and does not want to respond to
this request right now.

> First we did this with apache+jetty (proxy-http), next jetty, we changed
> config values for maxThreads, maxQueued and that didn't fix anything.

I think you need several variables here.

httpd.maxThreads - number of Jetty threads, you don't need as many of
these as you think. Web UI and the *setup* for an HTTP Git fetch or
push uses a thread, but the actual fetch or push runs on a different
thread (see below).

httpd.maxQueued - number of requests Jetty has pending waiting for a
Jetty thread.

sshd.threads - number of Git fetch or push threads for the entire
server. These threads are shared between the SSHD and HTTPD. HTTP Git
fetch or push requests swap threads from a Jetty thread to an SSHD
thread to actually perform the work. If you have an insufficient
number of these threads, requests will queue up (as you see in
show-queue). After a timeout, Jetty will cancel the queue entry and
return a 503 to the client.

I would try increasing the sshd.threads parameter, instead of
httpd.maxThreads. Most web requests run in under 250 ms. The setup for
an HTTP Git fetch or push takes a lot less than that over HTTP, maybe
1 or 2 ms. So just httpd.maxThreads = 5 should be able to serve
somewhere around 20 requests/second. For Gerrit Code Review you need
to expend your thread resources on sshd.threads; having sshd.threads =
50-200 is not unheard of. :-)

One thing to keep in mind with sshd.threads is each thread needs a lot
more than just its thread stack (which is usually the limiting factor
of threads in a Java program). The thread stack is usually ~8 MB.
However the per-thread memory required to say complete a `git clone`
of the 2 million objects / 400 MB of data in the Linux kernel is well
over 120 MB. sshd.threads and the queue are server overload protection
mechanisms. You want sshd.threads to be somewhere in the range of the
number of requests your server can handle concurrently (given memory
and CPU constraints), while leaving a little bit of slop for slow
clients (as the thread has to spool data out at network speed to the
client, which is often bottlenecked by the client's network if they
are far away on a DSL modem). Allowing sshd.threads to be more than
you have actual capacity for (in terms of memory and CPU) means the
system will thrash and nobody will be able to complete a Git request.

You may want to look at tweaking httpd.maxWait. This sets how long an
HTTP request waits for an sshd.threads pool worker to become
available. A Git client will wait forever, but intermediate proxy
servers may also need to have their wait time increased if you
increase this value above 3-5 minutes. SSH clients don't notice this
queuing effect because they effectively have httpd.maxWait = 1 year.
:-)

[Mah]

Big thanks! :) Now it's working great :)