overlord errors in log, but tasks are fine.

68 views
Skip to first unread message

Vikram Ramakrishnan

unread,
Jul 19, 2014, 11:29:47 AM7/19/14
to druid-de...@googlegroups.com
Hi,

I see the following error coming up very frequently in the log..The tasks submitted to overlord are running fine without any issues though...
what is the reason behind these errors....

-V



[2014-07-18 23:59:20,331] DEBUG HttpConnection@6e038265{FILLING} onFillable HttpChannelState@3bbf33e8{s=IDLE i=true a=null} (org.eclipse.jetty.server.HttpConnection)

[2014-07-18 23:59:20,331] DEBUG  (org.eclipse.jetty.io.ChannelEndPoint)

java.io.IOException: Connection reset by peer

        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)

        at sun.nio.ch.IOUtil.read(IOUtil.java:197)

        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)

        at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:137)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:216)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)

        at java.lang.Thread.run(Thread.java:744)

[2014-07-18 23:59:20,332] DEBUG ishut SelectChannelEndPoint@314dd984{/10.65.76.25:56892<->8080,Open,in,OSHUT,-,-,300000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.ChannelEndPoint)

[2014-07-18 23:59:20,332] DEBUG onClose SelectChannelEndPoint@314dd984{/10.65.76.25:56892<->8080,CLOSED,ISHUT,OSHUT,-,-,300000,HttpConnection}{io=0,kio=0,kro=1} (org.eclipse.jetty.io.AbstractEndPoint)


Fangjin Yang

unread,
Jul 19, 2014, 12:23:36 PM7/19/14
to druid-de...@googlegroups.com
Hi Vikram, I

Fangjin Yang

unread,
Jul 19, 2014, 12:26:51 PM7/19/14
to druid-de...@googlegroups.com
Oops, pressed enter there too fast.

I am very surprised you would be getting this error often. Can you share some details about your jetty configuration? This error is from Jetty. When you say you get this error in the log, are you getting them in the log of the task, the broker, the overlord, or somewhere else?

-- FJ

Vikram Ramakrishnan

unread,
Jul 21, 2014, 7:03:12 AM7/21/14
to druid-de...@googlegroups.com
Hi FJ,

missed this over the weekend. The errors are in the overlord logs.. the task logs are fine.
I don't have any specific jetty configuration. Here's my overlord and middleManager runtime props.


-V

Fangjin Yang

unread,
Jul 21, 2014, 3:27:46 PM7/21/14
to druid-de...@googlegroups.com
Hi Vikram, not entirely sure what is causing this exception. It seems like some jetty connection is being severed. Do you guys have keep alives messages or anything similar? Also, this is not really impacting anything right?

Thanks,
FJ

Vikram Ramakrishnan

unread,
Jul 25, 2014, 8:12:30 AM7/25/14
to druid-de...@googlegroups.com
Hi FJ,

We have a haproxy vip setup on top of the indexers and haproxy does ECV check on these machines... would those be causing this?
is there a specific jetty configuration we should use for prod?

-V

Fangjin Yang

unread,
Jul 31, 2014, 2:28:59 PM7/31/14
to druid-de...@googlegroups.com
Hi Vikram, we don't really have a specific jetty configuration outside of setting the max number of http threads. I'm not entirely sure about haproxy and ECV...
Reply all
Reply to author
Forward
0 new messages