Query Exception : For MiddleManager : Druid 0.10

[Arpan Khagram]

Hi Team, I am started facing below exception in broker stating channel disconnected from middleManager. I have checked connectivity between broker and my middleManager and there is no issue with that. PFB the configuration of the middleManager (runtime.properties). I assume there is no need to set druid.server.http.numThreads if i have set  druid.indexer.fork.property.druid.server.http.numThreads. Kindly suggest what could be going wrong here ? Thanks in advance.

druid.host=10.52.208.85

druid.service=druid/middlemanager

druid.port=8091

# Number of tasks per middleManager

druid.worker.capacity=4

# Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx9g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid/task

druid.indexer.task.restoreTasksOnRestart=true

# HTTP server threads

#druid.server.http.numThreads=40

# Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numThreads=20

# Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.3.0"]

# Peon properties

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=1200000000

druid.indexer.fork.property.druid.processing.numThreads=3

druid.indexer.fork.property.druid.server.http.numThreads=50

Exception Stack Trace :

io.druid.java.util.common.RE: Failure getting results from[http://<milddleManager IP address>:8101/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Channel disconnecte

d]

        at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:489) ~[druid-server-0.10.0.jar:0.10.0]

        at io.druid.client.DirectDruidClient$JsonParserIterator.hasNext(DirectDruidClient.java:439) ~[druid-server-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:88) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:68) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:70) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:66) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:46) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:108) ~[druid-processing-0.10.0.jar:0.10.0]

        at io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80) ~[druid-common-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78) ~[druid-processing-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[java-util-0.10.0.jar:0.10.0]

        at io.druid.server.QueryResource.doPost(QueryResource.java:243) [druid-server-0.10.0.jar:0.10.0]

        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]

        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.j

ava:205) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]

        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]

        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [jetty-servlet-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.16.v20170120.jar:9.3.16.v20170120]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.16.v20170120.jar:9.3.16.v20170120]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

Caused by: java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException: Channel disconnected

        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

        at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:476) ~[druid-server-0.10.0.jar:0.10.0]

        ... 71 more

Caused by: org.jboss.netty.channel.ChannelException: Channel disconnected

        at com.metamx.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:317) ~[http-client-1.0.6.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.4.Final.jar:?]

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.4.Final.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_91]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_91]

        ... 1 more

2017-04-29T23:03:48,499 INFO [qtp190967225-124[groupBy_[DSLAM]_query49]] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2017-04-29T23:03:48.497Z","service":"druid/broker","host":"10.52.208.91:8082","version":"0.10.0","severity":"component-failure","description":"Exception handling request","data":{"class":"io.druid.server.QueryResource","exceptionType":"io.druid.java.util.common.RE","exceptionMessage":"Failure getting results from[http://10.52.208.85:8101/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Channel disconnected]","exceptionStackTrace":"io.druid.java.util.common.RE: Failure getting results from[http://10.52.208.85:8101/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Channel disconnected]\n\tat io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:489)\n\tat io.druid.client.DirectDruidClient$JsonParserIterator.hasNext(DirectDruidClient.java:439)\n\tat io.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:88)\n\tat io.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:68)\n\tat io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)\n\tat io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:70)\n\tat io.druid.java.util.common.guava.MergeSequence$2.accumulate(MergeSequence.java:66)\n\tat io.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:46)\n\tat io.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63)\n\tat io.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)\n\tat io.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:108)\n\tat io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:80)\n\tat io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)\n\tat io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87)\n\tat io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83)\n\tat io.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)\n\tat io.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82)\n\tat io.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)\n\tat io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:87)\n\tat io.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:83)\n\tat io.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78)\n\tat io.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:82)\n\tat io.druid.java.util.common.guava.Yielders.each(Yielders.java:32)\n\tat io.druid.server.QueryResource.doPost(QueryResource.java:243)\n\tat sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n\tat com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n\tat com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n\tat com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n\tat com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)\n\tat com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)\n\tat com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)\n\tat com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)\n\tat com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)\n\tat com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)\n\tat com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)\n\tat org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.util.concurrent.ExecutionException: org.jboss.netty.channel.ChannelException: Channel disconnected\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)\n\tat io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:476)\n\t... 71 more\nCaused by: org.jboss.netty.channel.ChannelException: Channel disconnected\n\tat com.metamx.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:317)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\t... 1 more\n","exception":"io.druid.java.util.common.RE: Failure getting results from[http://10.52.208.85:8101/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Channel disconnected]","query":"GroupByQuery{dataSource='DSLAM', querySegmentSpec=LegacySegmentSpec{intervals=[2017-04-28T20:00:00.000Z/2017-04-29T20:00:00.000Z]}, virtualColumns=[], limitSpec=NoopLimitSpec, dimFilter=((ELEMENT_NAME = NAAOTR && SUBELEMENT_NAME = /shelf=0/slot=8/port=2) || (ELEMENT_NAME = NAAOTR && SUBELEMENT_NAME = /shelf=0/slot=8/port=3) || (ELEMENT_NAME = NAAOTS && SUBELEMENT_NAME = /shelf=0/slot=0/port=19) || (ELEMENT_NAME = NAAOTS && SUBELEMENT_NAME = /shelf=0/slot=8/port=0) || (ELEMENT_NAME = NAAOTS && SUBELEMENT_NAME = /shelf=0/slot=8/port=1) || (ELEMENT_NAME = NAAOTS && SUBELEMENT_NAME = /shelf=0/slot=8/port=2) || (ELEMENT_NAME = NAAOTS && SUBELEMENT_NAME = /shelf=0/slot=8/port=3) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=1/port=2) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=1/port=4) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=2/port=21) || (ELEMENT_NAME = NAAOTV && SUBELEMENT_NAME = /shelf=0/slot=3/port=5) || (ELEMENT_NAME = NAAOTW && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOTW && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOTW && SUBELEMENT_NAME = /shelf=0/slot=2/port=19) || (ELEMENT_NAME = NAAOTW && SUBELEMENT_NAME = /shelf=0/slot=3/port=21) || (ELEMENT_NAME = NAAOTW && SUBELEMENT_NAME = /shelf=0/slot=3/port=23) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=0/port=12) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=0/port=5) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=1/port=10) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=1/port=19) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=1/port=5) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=1/port=7) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=8/port=0) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=8/port=1) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=8/port=2) || (ELEMENT_NAME = NAAOUA && SUBELEMENT_NAME = /shelf=0/slot=8/port=3) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=1/port=11) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=1/port=18) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=2/port=3) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=3/port=0) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=3/port=26) || (ELEMENT_NAME = NAAOUB && SUBELEMENT_NAME = /shelf=0/slot=3/port=3) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=1/port=2) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=1/port=23) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=1/port=4) || (ELEMENT_NAME = NAAOUE && SUBELEMENT_NAME = /shelf=0/slot=4/port=14) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=0/port=24) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=0/port=27) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=0/port=36) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=0/port=41) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=1/port=11) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=1/port=13) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=1/port=29) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=1/port=30) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=1/port=43) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=2/port=15) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=3/port=24) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=3/port=3) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=8/port=0) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=8/port=1) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=8/port=2) || (ELEMENT_NAME = NAAOVA && SUBELEMENT_NAME = /shelf=0/slot=8/port=3) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=0/port=11) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=0/port=23) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=0/port=9) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=1/port=2) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=8/port=0) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=8/port=1) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=8/port=2) || (ELEMENT_NAME = NAAOVF && SUBELEMENT_NAME = /shelf=0/slot=8/port=3) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=1/port=10) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=2/port=12) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=2/port=15) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=2/port=25) || (ELEMENT_NAME = NAAOVG && SUBELEMENT_NAME = /shelf=0/slot=2/port=27) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=1/port=16) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=1/port=23) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=2/port=0) || (ELEMENT_NAME = NAAOVN && SUBELEMENT_NAME = /shelf=0/slot=2/port=1) || (ELEMENT_NAME = NAAOVP && SUBELEMENT_NAME = /shelf=0/slot=0/port=0) || (ELEMENT_NAME = NAAOVP && SUBELEMENT_NAME = /shelf=0/slot=0/port=1) || (ELEMENT_NAME = NAAOVP && SUBELEMENT_NAME = /shelf=0/slot=1/port=0) || (ELEMENT_NAME = NAAOVP && SUBELEMENT_NAME = /shelf 

[Arpan Khagram]

Checked middleManager logs and did not notice any exception there.

[Arpan Khagram]

Did not face this issue again in recent times after i increased the number of http connections on Broker node (which is getting used to communicate with middlemanager and historical nodes)