java.io.IOException: Broken pipe

[Paskal Karsenti]

Hello,

Presto version 0.73

we got a lot of exceptions like :

io.airlift.slice.RuntimeIOException: org.eclipse.jetty.io.EofException

        at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:144) ~[slice-0.6.jar:0.6]

.

Caused by: org.eclipse.jetty.io.EofException: null

        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:189) ~[jetty-io-9.1.4.v20140401.jar:9.1.4.v20140401]

        at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:400) ~[jetty-io-9.1.4.v20140401.jar:9.1.4.v20140401]

.

Caused by: java.io.IOException: Broken pipe

        at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_60]

        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_60]

2014-10-05T09:37:46.545+0000     WARN   http-worker-28981       org.eclipse.jetty.server.HttpChannel    Could not send response error 500: io.airlift.slice.RuntimeIOException: org.eclipse.jetty.io.EofException

2014-10-05T09:37:46.545+0000     WARN   http-worker-27807       org.eclipse.jetty.server.HttpChannel    Could not send response error 500: io.airlift.slice.RuntimeIOException: org.eclipse.jetty.io.EofException

2014-10-05T09:38:26.094+0000     WARN   http-client-shared-28885        org.eclipse.jetty.http.HttpParser       badMessage: java.lang.IllegalStateException: too much data after closed for HttpReceiverOverHTTP@7dfb5bf4 on HttpConnectionOverHTTP@15230ca(l:/192.168.250.4:55818 <-> r:/192.168.250.4:8090)

do you think it could be a network issue ? if no a lot of heavy requests can also produce that ?

Thanks for your help

[Dain Sundstrom]

This normally means that the server hung up on the client.  There are a lot of reasons this can happen.  You could run out of http worker threads.  You may have your task memory too high and the vm is struggling to find free memory (i.e, full GC pauses).  A node could have died.  

Some of these are caused by bugs we have fixed in the last few releases.  For example in 0.76, we made the http server fully async.  So I suggest you should upgrade to the latest release and try again.

-dain

[Paskal Karsenti]

understood

Thanks you so much 

--
You received this message because you are subscribed to the Google Groups "Presto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nezih Yigitbasi]

Hi,

With the latest master I am observing a similar issue (occasionally) on my test cluster (not overloaded at all) with a simple query of the form select * from T limit N. When I digged a little bit deeper it seems like one side of a channel is closed by a call that originates from Driver.destroyIfNecessary() (see stack trace below), and another thread after that tries to flush that channel’s buffers and fails with a broken pipe exception. Seems like a race condition, what do you think?

Here is the stacktrace of the close() call on a channel that can come before the flush() call on the same channel:

20141121_201222_00003_k2kj7.0.0-0-65@4462, prio=5, in group 'main', status: 'RUNNING'
      at org.eclipse.jetty.io.ChannelEndPoint.close(ChannelEndPoint.java:113)
      at org.eclipse.jetty.io.SelectChannelEndPoint.close(SelectChannelEndPoint.java:168)
      at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:133)
      at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:120)
      at org.eclipse.jetty.client.http.HttpChannelOverHTTP.exchangeTerminated(HttpChannelOverHTTP.java:100)
      at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:479)
      at org.eclipse.jetty.client.HttpReceiver.responseFailure(HttpReceiver.java:451)
      at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:516)
      at org.eclipse.jetty.client.http.HttpChannelOverHTTP.abort(HttpChannelOverHTTP.java:70)
      at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:200)
      at org.eclipse.jetty.client.HttpConversation.abort(HttpConversation.java:155)
      at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:674)
      at io.airlift.http.client.jetty.JettyHttpClient$JettyResponseFuture.cancel(JettyHttpClient.java:447)
      at com.facebook.presto.operator.HttpPageBufferClient.close(HttpPageBufferClient.java:212)
      at com.facebook.presto.operator.ExchangeClient.closeQuietly(ExchangeClient.java:403)
      at com.facebook.presto.operator.ExchangeClient.close(ExchangeClient.java:220)
      at com.facebook.presto.operator.ExchangeOperator.close(ExchangeOperator.java:191)
      at com.facebook.presto.operator.ExchangeOperator.finish(ExchangeOperator.java:147)
      at com.facebook.presto.operator.Driver.destroyIfNecessary(Driver.java:383)
      at com.facebook.presto.operator.Driver.access$400(Driver.java:53)
      at com.facebook.presto.operator.Driver$DriverLockResult.close(Driver.java:527)
      at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
      at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:557)
      at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:444)
      at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:578)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

And here is the flush() call that fails:

org.glassfish.jersey.server.internal.process.MappableException: io.airlift.slice.RuntimeIOException: org.eclipse.jetty.io.EofException
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91) ~[jersey-server-2.12.jar:na]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621) [jersey-server-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377) [jersey-server-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367) [jersey-server-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:828) [jersey-server-2.12.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) [jersey-common-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:858) [jersey-server-2.12.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:820) [jersey-server-2.12.jar:na]
    at io.airlift.http.server.AsyncResponseHandler$2.onSuccess(AsyncResponseHandler.java:107) [http-server-0.97.jar:0.97]
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319) [guava-18.0.jar:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_60]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_60]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_60]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_60]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: io.airlift.slice.RuntimeIOException: org.eclipse.jetty.io.EofException
    at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:144) ~[slice-0.7.jar:0.7]
    at com.facebook.presto.spi.block.VariableWidthBlockEncoding.writeBlock(VariableWidthBlockEncoding.java:66) ~[presto-spi-0.85-SNAPSHOT.jar:0.85-SNAPSHOT]
    at com.facebook.presto.block.PagesSerde$PagesWriter.append(PagesSerde.java:84) ~[presto-main-0.86-SNAPSHOT.jar:0.86-SNAPSHOT]
    at com.facebook.presto.block.PagesSerde.writePages(PagesSerde.java:53) ~[presto-main-0.86-SNAPSHOT.jar:0.86-SNAPSHOT]
    at com.facebook.presto.block.PagesSerde.writePages(PagesSerde.java:46) ~[presto-main-0.86-SNAPSHOT.jar:0.86-SNAPSHOT]
    at com.facebook.presto.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:88) ~[presto-main-0.86-SNAPSHOT.jar:0.86-SNAPSHOT]
    at com.facebook.presto.server.PagesResponseWriter.writeTo(PagesResponseWriter.java:39) ~[presto-main-0.86-SNAPSHOT.jar:0.86-SNAPSHOT]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[jersey-server-2.12.jar:na]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) ~[jersey-server-2.12.jar:na]
    ... 23 common frames omitted
Caused by: org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:337) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:128) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:645) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.util.IteratingCallback.processIterations(IteratingCallback.java:233) ~[jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:193) ~[jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:455) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:745) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:778) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:131) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:124) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:339) ~[jetty-server-9.2.2.v20140723.jar:9.2.2.v20140723]
    at io.airlift.http.server.TimingFilter$TimedServletOutputStream.write(TimingFilter.java:161) ~[http-server-0.97.jar:0.97]
    at org.eclipse.jetty.servlets.gzip.AbstractCompressedStream.write(AbstractCompressedStream.java:226) ~[jetty-servlets-9.2.2.v20140723.jar:9.2.2.v20140723]
    at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:298) ~[jersey-container-servlet-core-2.12.jar:na]
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[jersey-common-2.12.jar:na]
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[jersey-common-2.12.jar:na]
    at io.airlift.slice.CountingOutputStream.write(CountingOutputStream.java:56) ~[slice-0.7.jar:0.7]
    at java.io.DataOutputStream.write(DataOutputStream.java:107) ~[na:1.7.0_60]
    at io.airlift.slice.LittleEndianDataOutputStream.write(LittleEndianDataOutputStream.java:57) ~[slice-0.7.jar:0.7]
    at io.airlift.slice.Slice.getBytes(Slice.java:501) ~[slice-0.7.jar:0.7]
    at io.airlift.slice.OutputStreamSliceOutput.writeBytes(OutputStreamSliceOutput.java:141) ~[slice-0.7.jar:0.7]
    ... 35 common frames omitted
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[na:1.7.0_60]
    at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[na:1.7.0_60]
    at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[na:1.7.0_60]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524) ~[na:1.7.0_60]
    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:172) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

[Dain Sundstrom]

This is expected behavior.  The client code is doing a "force" disconnect of the http client, which causes the server to get an EOF.  The real problem is we are logging the exception on the server.  I posted a pull-request (https://github.com/facebook/presto/pull/2190) to ignore the EOF exception when in the PagesResponseWriter.

-dain

...