gerrit too many open files - looks like a lfs issue

520 views
Skip to first unread message

ld...@audiokinetic.com

unread,
Nov 2, 2018, 9:45:37 AM11/2/18
to Repo and Gerrit Discussion
Hi,

We are 2.15.6 (upgraded last night after we got our first issue)
We have had this error twice since yesterday:

[2018-11-02 09:29:35,884] [HTTP-841] WARN  org.eclipse.jetty.io.ManagedSelector : Accept failed for channel java.nio.channels.SocketChannel[closed]
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.io.ManagedSelector.processAccept(ManagedSelector.java:393)
at org.eclipse.jetty.io.ManagedSelector.access$1000(ManagedSelector.java:56)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:295)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
[2018-11-02 09:29:35,889] [HTTP-939] WARN  org.eclipse.jetty.server.HttpChannel : //git.audiokinetic.inte/plugins/lfs/content/default/dfc04334bcade529dfc3d07a90369213792ce7ed3580dd19f042a798b5080ddc
java.nio.file.FileSystemException: /home/software/gerrit/data/lfs/df/c0/dfc04334bcade529dfc3d07a90369213792ce7ed3580dd19f042a798b5080ddc: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.eclipse.jgit.lfs.server.fs.FileLfsRepository.getReadChannel(FileLfsRepository.java:145)
at org.eclipse.jgit.lfs.server.fs.ObjectDownloadListener.<init>(ObjectDownloadListener.java:101)
at com.googlesource.gerrit.plugins.lfs.fs.LfsFsContentServlet.getObject(LfsFsContentServlet.java:141)
at com.googlesource.gerrit.plugins.lfs.fs.LfsFsContentServlet.doGet(LfsFsContentServlet.java:82)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:250)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:451)
at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:100)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:122)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:69)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:133)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:135)
at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:69)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
[2018-11-02 09:29:35,890] [HTTP-841] WARN  org.eclipse.jetty.io.ManagedSelector : Accept failed for channel null
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.io.ManagedSelector.processAccept(ManagedSelector.java:393)
at org.eclipse.jetty.io.ManagedSelector.access$1000(ManagedSelector.java:56)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:295)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)


Wasn't this fixed in 2.15.6?

David Ostrovsky

unread,
Nov 2, 2018, 9:59:36 AM11/2/18
to Repo and Gerrit Discussion

Am Freitag, 2. November 2018 14:45:37 UTC+1 schrieb ld...@audiokinetic.com:
Hi,

We are 2.15.6 (upgraded last night after we got our first issue)
We have had this error twice since yesterday:

[2018-11-02 09:29:35,884] [HTTP-841] WARN  org.eclipse.jetty.io.ManagedSelector : Accept failed for channel java.nio.channels.SocketChannel[closed]
java.io.IOException: Too many open files
[...]
Wasn't this fixed in 2.15.6?

Note, that the regression was in JGit's org.eclipse.jgit.lfs.server
library: [1], that is not part of Gerrit, but is shipped with lfs-plugin.
You need to install a new version of lfs-plugin: [2].


ld...@audiokinetic.com

unread,
Nov 2, 2018, 10:10:35 AM11/2/18
to Repo and Gerrit Discussion
Thanks, I'm trying to look at the release note and other information to see if I missed something that would have told me that the fix was in the plugin.
Or was it rightfully assumed that when someone update gerrit, they update the core-plugin also?

Luca Milanesio

unread,
Nov 2, 2018, 10:18:12 AM11/2/18
to ld...@audiokinetic.com, Luca Milanesio, Repo and Gerrit Discussion

On 2 Nov 2018, at 14:10, ld...@audiokinetic.com wrote:

Thanks, I'm trying to look at the release note and other information to see if I missed something that would have told me that the fix was in the plugin.
Or was it rightfully assumed that when someone update gerrit, they update the core-plugin also?

Git-LFS is not a core plugin.

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthias Sohn

unread,
Nov 2, 2018, 11:30:04 AM11/2/18
to Luca Milanesio, ld...@audiokinetic.com, repo-d...@googlegroups.com
On Fri, Nov 2, 2018 at 3:18 PM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 Nov 2018, at 14:10, ld...@audiokinetic.com wrote:

Thanks, I'm trying to look at the release note and other information to see if I missed something that would have told me that the fix was in the plugin.
Or was it rightfully assumed that when someone update gerrit, they update the core-plugin also?

Git-LFS is not a core plugin. 
Luca.

The 2.15.6 release notes [1] mention that this release fixes the file handle leak in JGit's lfs server
but do not mention that the lfs plugin also needs to be updated. See log of lfs plugin stable-2.15 branch [2]
containing the commit 5b8184dupdating to JGit 4.9.7.

Can some maintainer amend the 2.15.6 release notes and add a note that the lfs plugin needs to be updated ?

 
-Matthias

David Pursehouse

unread,
Nov 3, 2018, 12:35:00 AM11/3/18
to Matthias Sohn, Luca Milanesio, ld...@audiokinetic.com, repo-d...@googlegroups.com
On Sat, Nov 3, 2018 at 12:30 AM Matthias Sohn <matthi...@gmail.com> wrote:
On Fri, Nov 2, 2018 at 3:18 PM Luca Milanesio <luca.mi...@gmail.com> wrote:


On 2 Nov 2018, at 14:10, ld...@audiokinetic.com wrote:

Thanks, I'm trying to look at the release note and other information to see if I missed something that would have told me that the fix was in the plugin.
Or was it rightfully assumed that when someone update gerrit, they update the core-plugin also?

Git-LFS is not a core plugin. 
Luca.

The 2.15.6 release notes [1] mention that this release fixes the file handle leak in JGit's lfs server
but do not mention that the lfs plugin also needs to be updated. See log of lfs plugin stable-2.15 branch [2]
containing the commit 5b8184dupdating to JGit 4.9.7.

Can some maintainer amend the 2.15.6 release notes and add a note that the lfs plugin needs to be updated ?

Sorry, I intended to do this earlier but other work got in the way.  I've done it now:

Reply all
Reply to author
Forward
0 new messages