"Failure getting results" and OutOfMemoryError after loading data

Amy Troschinetz

unread,

Oct 6, 2014, 1:44:26 PM10/6/14

to druid-de...@googlegroups.com

I managed to load about a month of data into Druid, the dataset is only 64GB (compressed, but I'm only using a few of the columns for each event so the data we actually care about is much smaller).

Here's the description of the ingestion task:

{

"type": "index",

"dataSource": "simple_outclick",

"granularitySpec":

{

"type": "uniform",

"gran": "DAY",

"intervals": ["2014-09-01T21:57:47/2014-10-01T15:04:07"]

},

"indexGranularity": "second",

"aggregators": [

{"type": "count", "name": "count"}

],

"firehose":

{

"type": "local",

"baseDir": "/home/users/me/stuff",

"filter": ".csv",

"parser":

{

"timestampSpec":

{

"column": "clickDate",

"format": "yyyy-MM-dd HH:mm:ss"

},

"data":

{

"format": "tsv",

"columns": ["masterClickId", [...]],

"dimensions": ["masterClickId", "outclickTypeId", "siteId", "couponId", "calc_mediumCalc"]

}

I can provide the entire indexer task log if anyone thinks it will be enlightening, but it does contain the following:

2014-10-03 09:37:32,610 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

"id" : "index_simple_outclick_2014-10-01T17:46:21.967Z",

"status" : "SUCCESS",

"duration" : 143461732

}

When I do a timeBoundary query I get the following, which looks correct to me:

[

{

"result": {

"maxTime": "2014-10-01T15:05:06.000Z",

"minTime": "2014-09-01T21:50:02.000Z"

},

"timestamp": "2014-09-01T21:50:02.000Z"

}

]

However when I try to do a groupBy query I get this response from my broker node:

Failure getting results from[http://ip-of-one-of-myhistorical-nodes:8080/druid/v2/]

If I query ip-of-one-of-myhistorical-nodes or my 2nd historical node instead of my broker node, I get this response:

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory

I am currently running a cluster that looks like:

1x druid-overlord m1.xlarge

1x druid-coordinator m1.xlarge

1x druid-broker m3.2xlarge

2x druid-historical m3.2xlarge

1x druid-middleManager m3.2xlarge

1x zookeeper m1.small

1x druid-mysql m1.small

I'm not sure the best way to diagnose this error, I don't believe it should be possible that I'm running out of memory with such a manageable dataset. Each m3.2xlarge instance has 30GB of main memory. Is there some way to ask Druid how much memory it's currently using and for what, some kind of diagnostics panel or something? I keep trying to use the Coordinator Console but every control on those pages is unresponsive. The "View Information about the Cluster" page for example never actually returns any requested information.

PS: Here's the log from the broker node when it fails to service the request:

2014-10-06 17:12:23,411 INFO [qtp1197972673-54] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://10.7.150.217:8080

2014-10-06 17:12:23,413 INFO [qtp1197972673-54] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://10.71.179.186:8080

2014-10-06 17:12:24,398 WARN [qtp1197972673-54] io.druid.server.QueryResource - Exception occurred on request [GroupByQuery{limitSpec=NoopLimitSpec, dimFilter=null, granularity=DurationGranularity{length=86400000, origin=0}, dimensions=[], aggregatorSpecs=[LongSumAggregatorFactory{fieldName='count', name='totalClicks'}], postAggregatorSpecs=[], orderByLimitFn=identity}]

com.metamx.common.RE: Failure getting results from[http://10.71.179.186:8080/druid/v2/]

at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:294)

at io.druid.client.DirectDruidClient$JsonParserIterator.hasNext(DirectDruidClient.java:248)

at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:103)

at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:81)

at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67)

at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40)

at com.metamx.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:46)

at com.metamx.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:42)

at com.metamx.common.guava.YieldingAccumulators$1.accumulate(YieldingAccumulators.java:32)

at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:104)

at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:81)

at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67)

at com.metamx.common.guava.ConcatSequence.accumulate(ConcatSequence.java:40)

at com.metamx.common.guava.LazySequence.accumulate(LazySequence.java:37)

at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:87)

at io.druid.query.groupby.GroupByQueryQueryToolChest.makeIncrementalIndex(GroupByQueryQueryToolChest.java:148)

at io.druid.query.groupby.GroupByQueryQueryToolChest.mergeGroupByResults(GroupByQueryQueryToolChest.java:115)

at io.druid.query.groupby.GroupByQueryQueryToolChest.access$100(GroupByQueryQueryToolChest.java:56)

at io.druid.query.groupby.GroupByQueryQueryToolChest$2.run(GroupByQueryQueryToolChest.java:85)

at io.druid.query.FinalizeResultsQueryRunner.run(FinalizeResultsQueryRunner.java:96)

at io.druid.query.BaseQuery.run(BaseQuery.java:80)

at io.druid.query.BaseQuery.run(BaseQuery.java:75)

at io.druid.server.QueryResource.doPost(QueryResource.java:119)

at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$VoidOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:167)

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:298)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1622)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:462)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('j' (code 106)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

at [Source: com.metamx.http.client.io.AppendableByteArrayInputStream@714924b5; line: 1, column: 2]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1369)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:532)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:453)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2278)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:779)

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:665)

at io.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:286)

... 70 more

2014-10-06 17:12:24,400 ERROR [qtp1197972673-54] io.druid.server.QueryResource - Exception handling request: {class=io.druid.server.QueryResource, exceptionType=class com.metamx.common.RE, exceptionMessage=Failure getting results from[http://10.71.179.186:8080/druid/v2/], exception=com.metamx.common.RE: Failure getting results from[http://10.71.179.186:8080/druid/v2/], query=GroupByQuery{limitSpec=NoopLimitSpec, dimFilter=null, granularity=DurationGranularity{length=86400000, origin=0}, dimensions=[], aggregatorSpecs=[LongSumAggregatorFactory{fieldName='count', name='totalClicks'}], postAggregatorSpecs=[], orderByLimitFn=identity}, peer=209.163.206.98}