An I/O error has occurred while writing a response message entity to the container output stream

Paskal Karsenti

unread,

Oct 22, 2015, 3:48:14 AM10/22/15

to Presto

Hello ,

i got this error 3 days after i upgraded from 0.107 to 0.122 :

2015-10-22T07:17:17.812Z ERROR http-worker-77833 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.

org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException

at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)

at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)

at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)

at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)

at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)

at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)

at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)

at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:499)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.eclipse.jetty.io.EofException

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)

at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)

at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)

at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111)

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:640)

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:611)

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)

at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)

any idea ?

Thanks for your help

maxim.d...@gmail.com

unread,

Nov 5, 2015, 4:24:41 PM11/5/15

to Presto

We've got the same error - have you been able to get to the bottom of this?

smi...@twitter.com

unread,

Mar 10, 2016, 4:41:15 PM3/10/16

to Presto, maxim.d...@gmail.com

I am also getting this error after upgrading from 0.132 to 0.139. Were you guys able to resolve this?

Dain Sundstrom

unread,

Mar 10, 2016, 5:31:07 PM3/10/16

to presto...@googlegroups.com, maxim.d...@gmail.com

I have never seen that error. Do you have a stack trace?

-dain

> --
> You received this message because you are subscribed to the Google Groups "Presto" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

smi...@twitter.com

unread,

Mar 10, 2016, 5:41:12 PM3/10/16

to Presto, maxim.d...@gmail.com

Hi Dain,

It shows up as this error on cli and UI:

com.facebook.presto.operator.PageTransportErrorException: Expected response code to be 200, but was 500 Server Error: http://10.53.112.120:8080/v1/task/20160309_233955_00133_4nkfd.4.4/results/20160309_233955_00133_4nkfd.3.6/57
        at com.facebook.presto.operator.HttpPageBufferClient$PageResponseHandler.handle(HttpPageBufferClient.java:528)
        at com.facebook.presto.operator.HttpPageBufferClient$PageResponseHandler.handle(HttpPageBufferClient.java:501)
        at io.airlift.http.client.jetty.JettyHttpClient$JettyResponseFuture.processResponse(JettyHttpClient.java:787)
        at io.airlift.http.client.jetty.JettyHttpClient$JettyResponseFuture.completed(JettyHttpClient.java:764)
        at io.airlift.http.client.jetty.JettyHttpClient$BufferingResponseListener.onComplete(JettyHttpClient.java:1041)
        at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
        at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
        at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:454)
        at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:401)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:268)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1255)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:184)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:131)
        at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
        at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90)
        at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:113)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:745)

When I followed it to the erroneous host, got this errors:

2016-03-10T00:21:43.388Z        ERROR   async-http-response-296 org.glassfish.jersey.server.ServerRuntime$Responder     Error while closing the output
stream in order to commit response.
java.lang.NullPointerException
        at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:346)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:678)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:509)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:668)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:722)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:177)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:441)
        at io.airlift.http.server.TimingFilter$TimedServletOutputStream.write(TimingFilter.java:162)
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:298)
        at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276)
        at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:834)
        at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:691)


        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)


        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:828)


        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)


        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:858)
        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:820)
        at io.airlift.http.server.AsyncResponseHandler.lambda$toCompletionCallback$1(AsyncResponseHandler.java:101)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
        at io.airlift.concurrent.BoundedExecutor.executeOrMerge(BoundedExecutor.java:69)
        at io.airlift.concurrent.BoundedExecutor.access$000(BoundedExecutor.java:28)
        at io.airlift.concurrent.BoundedExecutor$1.run(BoundedExecutor.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

followed by a lot of these

2016-03-10T00:21:43.727Z        ERROR   http-worker-3871        org.glassfish.jersey.server.ServerRuntime$Responder     An I/O error has occurred while writing a response message entity to the container output stream.


org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)


        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:835)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
        at org.eclipse.jetty.server.Server.handle(Server.java:517)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


        at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jetty.io.EofException


        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:313)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:141)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:726)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:509)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:673)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:722)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:177)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:413)
        at io.airlift.http.server.TimingFilter$TimedServletOutputStream.write(TimingFilter.java:162)
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:298)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:1853)
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:443)
        at com.facebook.presto.block.BlockJsonSerde$Serializer.serialize(BlockJsonSerde.java:59)
        at com.facebook.presto.block.BlockJsonSerde$Serializer.serialize(BlockJsonSerde.java:41)
        at com.fasterxml.jackson.databind.SerializerProvider.defaultSerializeValue(SerializerProvider.java:896)
        at com.fasterxml.jackson.datatype.jdk8.OptionalSerializer.serialize(OptionalSerializer.java:92)
        at com.fasterxml.jackson.datatype.jdk8.OptionalSerializer.serialize(OptionalSerializer.java:19)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:639)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:152)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContentsUsing(IndexedListSerializer.java:124)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:69)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)
        at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:505)
        .....
        ... 46 more
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
        at sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:177)
        ... 133 more

Christopher Berner

unread,

Mar 10, 2016, 6:21:18 PM3/10/16

to Presto, maxim.d...@gmail.com

I believe that's the bug we fixed in 0.140 "Fix race which can cause queries to fail with a REMOTE_TASK_ERROR."

There was a bug in Jetty that was causing that NPE

Haozhun Jin

unread,

Mar 10, 2016, 6:28:05 PM3/10/16

to presto...@googlegroups.com, maxim.d...@gmail.com

The bug Chris mentioned is introduced in 0.133 (merged into trunk on 01/08/2016), and fixed in 0.140. The stack trace we see is exactly the same as the one smittal posted. I'm confident it's the same issue. The fix is here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=487714

Additionally, I'm sure this is unrelated to the original thread from last year. I checked our log when I was investigating this issue, and the stack trace smittal posted never appeared prior to our deployment of 0.133.

--

smi...@twitter.com

unread,

Mar 10, 2016, 6:53:25 PM3/10/16

to Presto, maxim.d...@gmail.com, hj...@fb.com

Thanks all. I will upgrade to 0.141. I have also never seen this error before or on 0.132.

...

smi...@twitter.com

unread,

Mar 22, 2016, 6:48:26 PM3/22/16

to Presto, maxim.d...@gmail.com, hj...@fb.com

Hi, resurrecting this thread because I saw the same error again but with a different stack trace as below:

2016-03-21T21:25:26.616Z        ERROR   async-http-response-0   org.glassfish.jersey.server.ServerRuntime$Responder     Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: AsyncContext completed
        at org.eclipse.jetty.server.AsyncContextState.state(AsyncContextState.java:54)
        at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
        at io.airlift.jaxrs.ServletAsyncContextDelegateProvider$ServletAsyncContextDelegate.complete(ServletAsyncContextDelegateProvider.java:60)
        at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:210)
        at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:413)

        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:691)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)

        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:828)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:858)
        at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:820)
        at io.airlift.http.server.AsyncResponseHandler.lambda$toCompletionCallback$1(AsyncResponseHandler.java:101)
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
        at io.airlift.concurrent.BoundedExecutor.executeOrMerge(BoundedExecutor.java:69)
        at io.airlift.concurrent.BoundedExecutor.access$000(BoundedExecutor.java:28)
        at io.airlift.concurrent.BoundedExecutor$1.run(BoundedExecutor.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

This resulted in the following error on coordinator:

PAGE_TRANSPORT_ERROR (65539)

com.facebook.presto.operator.PageTransportErrorException: Expected response code to be 200, but was 500 Server Error: http://xxx.xx.xxx.xxx:8080/v1/task/20160319_184355_00188_2ttt8.1.37/results/20160319_184355_00188_2ttt8.0.0/7009
	at com.facebook.presto.operator.HttpPageBufferClient$PageResponseHandler.handle(HttpPageBufferClient.java:528)
	at com.facebook.presto.operator.HttpPageBufferClient$PageResponseHandler.handle(HttpPageBufferClient.java:501)
	at io.airlift.http.client.jetty.JettyHttpClient$JettyResponseFuture.processResponse(JettyHttpClient.java:787)
	at io.airlift.http.client.jetty.JettyHttpClient$JettyResponseFuture.completed(JettyHttpClient.java:764)
	at io.airlift.http.client.jetty.JettyHttpClient$BufferingResponseListener.onComplete(JettyHttpClient.java:1041)
	at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
	at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
	at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:454)
	at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:401)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:268)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1324)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:184)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:131)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:114)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

...

Haozhun Jin

unread,

Mar 28, 2016, 9:01:05 PM3/28/16

to smi...@twitter.com, Presto, maxim.d...@gmail.com, nyigi...@netflix.com

I can confirm we are seeing this in our logs as well.

We see queries that fail with "Expected response code to be 200, but was 500 Server Error". And then a few minutes later (sometimes up to 2 hours), the "java.lang.IllegalStateException: AsyncContext completed" in our log.

Chris is looking into this right now. He has reached out to the Jetty people.

vignesh....@adroll.com

unread,

May 8, 2016, 10:34:53 PM5/8/16

to Presto

I'm also noticing this error on the worker nodes when I run queries.. I just upgraded to 0.146 and was running a few benchmark queries. Is anyone else who's > 0.141 facing this issue?

2016-05-09T02:18:36.309Z ERROR http-worker-265 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)

wang...@gmail.com

unread,

May 24, 2016, 5:00:16 AM5/24/16

to Presto, vignesh....@adroll.com

We face the same problem. We use teradata presto 0.141t

2016-05-24T16:26:39.510+0800 ERROR http-worker-248 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.

org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:835)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1685)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)

at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)

at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jetty.io.EofException

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)

at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:375)
at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:195)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:273)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
... 4 more

Caused by: java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175)
... 11 more

在 2016年5月9日星期一 UTC+8上午10:34:53，vignesh....@adroll.com写道：

Message has been deleted

john...@gmail.com

unread,

Jun 13, 2016, 4:55:43 AM6/13/16

to Presto

Same here, on 0.145:

2016-06-12T06:25:14.225-0500 ERROR http-worker-155926 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)

at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:396)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:314)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)

at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jetty.io.EofException

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:197)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:419)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:375)
at org.eclipse.jetty.io.SelectChannelEndPoint$3.run(SelectChannelEndPoint.java:107)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:195)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:273)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:171)

at org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.execute(ProduceExecuteConsume.java:51)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:146)
... 3 more

Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)

at sun.nio.ch.IOUtil.write(IOUtil.java:65)

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:175)
... 11 more

2016-06-12T06:25:14.230-0500 ERROR http-worker-155902 org.glassfish.jersey.server.ServerRuntime$Responder An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)