Druid quickstart data ingestion failing

[Manish Trivedi]

Hi,

I am new to druid framework and playing around with Quickstart link here: http://druid.io/docs/0.9.0/tutorials/quickstart.html

Following the steps as per the quickstart seems OK until I start loading data using following step:

Load batch data

curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task

response from the curl:

{"task":"index_hadoop_wikiticker_2016-05-05T19:13:33.689Z"}

Upon starting this task, I see following error in my "overload" process log:

===========

2016-05-05T19:13:33,739 INFO [pool-12-thread-1] io.druid.indexing.overlord.RemoteTaskRunner - Task index_hadoop_wikiticker_2016-05-05T19:13:33.689Z switched from pending to running (on [0.1.190.41:8091])

2016-05-05T19:13:33,845 INFO [Curator-PathChildrenCache-0] io.druid.indexing.overlord.RemoteTaskRunner - Worker[0.1.190.41:8091] wrote RUNNING status for task: index_hadoop_wikiticker_2016-05-05T19:13:33.689Z

2016-05-05T19:13:58,503 INFO [qtp1311052922-50] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://0.1.190.41:8091

2016-05-05T19:13:58,534 WARN [qtp1311052922-50] org.jboss.netty.channel.SimpleChannelUpstreamHandler - EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught() for proper handling.

java.net.NoRouteToHostException: No route to host

at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_77]

at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_77]

at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_77]

at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_77]

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:54) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:97) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.Channels.connect(Channels.java:634) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:216) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [netty-3.10.4.Final.jar:?]

at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [netty-3.10.4.Final.jar:?]

at com.metamx.http.client.pool.ChannelResourceFactory.generate(ChannelResourceFactory.java:84) [http-client-1.0.4.jar:?]

at com.metamx.http.client.pool.ChannelResourceFactory.generate(ChannelResourceFactory.java:41) [http-client-1.0.4.jar:?]

at com.metamx.http.client.pool.ResourcePool$ImmediateCreationResourceHolder.<init>(ResourcePool.java:147) [http-client-1.0.4.jar:?]

at com.metamx.http.client.pool.ResourcePool$ImmediateCreationResourceHolder.<init>(ResourcePool.java:126) [http-client-1.0.4.jar:?]

at com.metamx.http.client.pool.ResourcePool$1.load(ResourcePool.java:52) [http-client-1.0.4.jar:?]

at com.metamx.http.client.pool.ResourcePool$1.load(ResourcePool.java:48) [http-client-1.0.4.jar:?]

at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache.get(LocalCache.java:3934) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) [guava-16.0.1.jar:?]

at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) [guava-16.0.1.jar:?]

at com.metamx.http.client.pool.ResourcePool.take(ResourcePool.java:71) [http-client-1.0.4.jar:?]

at com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:133) [http-client-1.0.4.jar:?]

at com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14) [http-client-1.0.4.jar:?]

at io.druid.indexing.overlord.RemoteTaskRunner$2.openStream(RemoteTaskRunner.java:475) [druid-indexing-service-0.9.0.jar:0.9.0]

at io.druid.indexing.overlord.http.OverlordResource.doGetLog(OverlordResource.java:455) [druid-indexing-service-0.9.0.jar:0.9.0]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) [guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) [guice-servlet-4.0-beta.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) [druid-server-0.9.0.jar:0.9.0]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

2016-05-05T19:13:58,543 INFO [qtp1311052922-50] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://0.1.190.41:8091

2016-05-05T19:13:58,544 WARN [qtp1311052922-50] org.jboss.netty.channel.SimpleChannelUpstreamHandler - EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught() for proper handling.

java.net.NoRouteToHostException: No route to host

===========

Please let me know as this error blocks me from moving further.

NOTE: I have not modified any of the jvm.config or properties file. They are all default values.

Druid version: druid-0.9.0, Zookeeper version: zookeeper-3.4.6

Any help will be really appreciated.

thanks,

Manish

[Fangjin Yang]

Hi Manish, are you by any chance behind any corporate firewalls?

[Manish Trivedi]

Thanks Fangjin for your reply. I am working on office mac and see that "Security & Privacy" option is disabled for me. 

It could be firewall issue but I am able to run other Java server processes on machines on various ports.

I will try to test again and update.

Thanks,

Manish

[Manish Trivedi]

Hi Fangjin,

I was finally able to resolve the issue. I have added following property in all the runtime.properties under conf-quickstart directory.

----------

druid.host=127.0.0.1

----------

├── _common

│   ├── common.runtime.properties

│   └── log4j2.xml

├── broker

│   ├── jvm.config

│   └── runtime.properties

├── coordinator

│   ├── jvm.config

│   └── runtime.properties

├── historical

│   ├── jvm.config

│   └── runtime.properties

├── middleManager

│   ├── jvm.config

│   └── runtime.properties

└── overlord

    ├── jvm.config

    └── runtime.properties

I am assuming that the issue was because of Netty creating dynamic IP when no configuration supplied.

Thanks,

Manish

On Thursday, May 5, 2016 at 12:29:47 PM UTC-7, Manish Trivedi wrote: