Puppet server throwing 'java.nio.channels.WritePendingException: null'

[Nikita Beleckij]

We have 700 servers and recently puppetserver stopped responce catalog to agents. The first few minutes it works fine, and then throws exceptions for all thread:

2017-12-16 21:42:19,963 WARN [qtp1380493193-555] [o.e.j.s.HttpChannel] /puppet/v3/catalog/db03st-ru.star-conflict.com?environment=production

java.nio.channels.WritePendingException: null

at org.eclipse.jetty.server.HttpConnection$SendCallback.reset(HttpConnection.java:624)

at org.eclipse.jetty.server.HttpConnection$SendCallback.access$100(HttpConnection.java:594)

at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:479)

at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768)

at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:135)

at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:166)

at org.eclipse.jetty.server.HttpWriter.close(HttpWriter.java:49)

at java.io.PrintWriter.close(PrintWriter.java:339)

at ring.util.servlet$set_body.invokeStatic(servlet.clj:89)

at ring.util.servlet$set_body.invoke(servlet.clj:84)

at ring.util.servlet$update_servlet_response.invokeStatic(servlet.clj:115)

at ring.util.servlet$update_servlet_response.invoke(servlet.clj:107)

at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__30506.invoke(jetty9_core.clj:426)

at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)

at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)

at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers$eval30081$normalize_uri_handler__30086$fn__30087$fn__30088.invoke(normalized_uri_helpers.clj:80)

at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:497)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

2017-12-17 00:10:17,030 WARN [qtp1943211232-575] [o.e.j.s.HttpChannel] /puppet/v3/catalog/pl-01.local.net?environment=production Agent Log: puppet-agent [19302]: Could not retrieve catalog from remote server: end of file reached It works again only after the restart.The puppetserver are Intel Xeon E312 machines, so they have 32 cores and 20GB of memory, and Intel SSDs. Server does not have high LA. How can I identify the source of the problem?