java.util.concurrent.TimeoutException when cloning over HTTPS

[Sebastiano Pilla]

I'm trying to clone a large repository (~ 4GB with several binary files) over HTTPS, Gitblit 1.6.0 running on JDK 1.8.0_05 behind a reverse HTTPS proxy (nginx). Each time the clone operation stops with an early EOF error:

[tomcat@server docs] $ GIT_SSL_NO_VERIFY=true git clone https://deployment@gitblit/r/docs.git .
Cloning into '.'...
Password for 'https://deployment@gitblit':
remote: Counting objects: 442483, done
remote: Finding sources: 100% (442483/442483)
remote: Getting sizes: 100% (179619/179619)
remote: Compressing objects:  99% (18093/18099)
fatal: The remote end hung up unexpectedly23.80 MiB | 4.53 MiB/s  
fatal: early EOF
fatal: index-pack failed

The clone operation downloads about 1GB and then stops. On the server I see a TimeoutException, several minutes earlier than the error on the client; the stack trace is:

2014-07-11 11:06:23 [WARN ] Failed login attempt for deployment, invalid credentials from 127.0.0.1
2014-07-11 11:06:28 [INFO ] ARF: docs.git/info/refs?service=git-upload-pack (100) authenticated
2014-07-11 11:06:44 [INFO ] ARF: docs.git/git-upload-pack (100) authenticated
2014-07-11 11:08:17 [WARN ] Internal error during upload-pack
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:193)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:129)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:361)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
        at org.eclipse.jgit.util.TemporaryBuffer.write(TemporaryBuffer.java:138)
        at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
        at org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:151)
        at org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:126)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:532)
        at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:356)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:190)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1512)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1489)
        at org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:164)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1477)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1465)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1034)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1415)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1269)
        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:717)
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:628)
        at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:187)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:237)
        at org.eclipse.jgit.http.server.UploadPackServlet$Factory.doFilter(UploadPackServlet.java:145)
        at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
        at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:151)
        at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
        at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:80)
        at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:235)
        at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:215)
        at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:101)
        at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:175)
        at com.gitblit.servlet.GitServlet.service(GitServlet.java:98)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1639)
        at com.gitblit.servlet.EnforceAuthenticationFilter.doFilter(EnforceAuthenticationFilter.java:87)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1639)
        at com.gitblit.servlet.AccessRestrictionFilter.doFilter(AccessRestrictionFilter.java:224)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1639)
        at com.gitblit.servlet.ProxyFilter$1.doFilter(ProxyFilter.java:74)
        at com.gitblit.servlet.ProxyFilter.doFilter(ProxyFilter.java:67)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:154)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more

Searching for this error I stumbled upon a Jetty bug reported against the 9.1.4.v20140401 version used in Gitblit 1.6.0: https://bugs.eclipse.org/bugs/show_bug.cgi?id=435322 . The problem seems limited to the HTTP(S) transport, since I've been able to clone this repository over SSH.

Any chance to see an updated Jetty version in the next Gitblit releases?

[James Moger]

Any chance to see an updated Jetty version in the next Gitblit releases?

The develop branch switched to Jetty 9.2.1 about a month ago.

-J

[Sebastiano Pilla]

On Friday, July 18, 2014 3:21:44 PM UTC+2, James Moger wrote:

Any chance to see an updated Jetty version in the next Gitblit releases?

The develop branch switched to Jetty 9.2.1 about a month ago.

FYI, I've just installed Gitblit 1.6.2, which contains Jetty 9.2.3, and the problem is still present. It looks like the bug in Jetty has not been fixed and is also present in the 9.3.0 versions.

I'll try deploying Gitblit as a WAR on Tomcat to see if the problem persists in this configuration as well or not.

[Sebastiano Pilla]

On Friday, July 18, 2014 3:21:44 PM UTC+2, James Moger wrote:

Any chance to see an updated Jetty version in the next Gitblit releases?

The develop branch switched to Jetty 9.2.1 about a month ago.

I've just installed Gitblit 1.6.2 and the problem persists. As suggested in the above mentioned Jetty bug, I've tried to put the latest jetty 9.2.4-201401103 in place of the 9.2.3 version that comes with Gitblit 1.6.2, but it failed with the same error.

I'll try running Gitblit WAR on Tomcat to see if the problem exists there too, it will take me some time because of file permission issues but I hope to have some results in the next days.

[Jan Gruncl]

Any news on this? This bug completely paralysed our GFX repository (cannot push latest commit, cannot clone), and we have to make it work as soon as possible :-(.

error: unpack failed: error java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

To http://X...@XX.XX.XX.XX:8080/r/XXX_XXX.git

 ! [remote rejected] master -> master (n/a (unpacker error))

error: failed to push some refs to 'http://X...@XX.XX.XX.XX:8080/r/XXX_XXX.git'

Any help would be very appreciated. We use Gitblit 1.6.2 running on Debian VPS.

[James Moger]

Hi Jan,

I haven't looked into it.  You could try building from master or develop and seeing if the dependency updates since 1.6.2 resolve this.

-J

--
You received this message because you are subscribed to the Google Groups "gitblit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gitblit+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastiano Pilla]

Coming back to this issue years later. I still have the problem, this time on 2 federated Gitblit instances where the 2nd one cannot pull large repositories because of this issue.

I believe that the problem isn't caused by a Jetty dependency, but rather by a hard-coded 30 sec timeout in GitBlitServer.java:

https://github.com/gitblit/gitblit/blob/4ece397d714946697bf911221b6168356a6c44c7/src/main/java/com/gitblit/GitBlitServer.java#L296

This 30 sec timeout is too short for a server to finish preparing all packs of a large repository. I hope to be able to send in a pull request in a few days to make this timeout configurable.

[Sebastiano Pilla]

Pull request created: https://github.com/gitblit/gitblit/pull/1243