Broker node problem upgrading from 0.6.52 to 0.6.73

[Sunil Srivatsa]

Hi,

I tried switching my Druid version in my build script, but now the Broker node fails to start with this:

2014-03-22 00:19:25,423 ERROR [main] io.druid.cli.CliBroker - Error when starting up.  Failing.
com.google.inject.ProvisionException: Guice provision errors:

1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize or druid.computation.buffer.size: maxDirectMemory[4,187,619,328], memoryNeeded[25,769,803,776], druid.computation.buffer.size[1,073,741,824], numThreads[23]
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:85)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:85)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:76)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:74)
  while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 1 at io.druid.query.groupby.GroupByQueryQueryToolChest.<init>(GroupByQueryQueryToolChest.java:70)
  at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:71)
  while locating io.druid.query.groupby.GroupByQueryQueryToolChest
  while locating io.druid.query.QueryToolChest annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=10, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryToolChestBinder(DruidBinders.java:45)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryToolChest>
    for parameter 0 at io.druid.query.MapQueryToolChestWarehouse.<init>(MapQueryToolChestWarehouse.java:36)
  while locating io.druid.query.MapQueryToolChestWarehouse
  while locating io.druid.query.QueryToolChestWarehouse
    for parameter 0 at io.druid.client.CachingClusteredClient.<init>(CachingClusteredClient.java:92)
  at io.druid.cli.CliBroker$1.configure(CliBroker.java:79)
  while locating io.druid.client.CachingClusteredClient
    for parameter 1 at io.druid.server.ClientQuerySegmentWalker.<init>(ClientQuerySegmentWalker.java:53)
  while locating io.druid.server.ClientQuerySegmentWalker
  at io.druid.cli.CliBroker$1.configure(CliBroker.java:87)
  while locating io.druid.query.QuerySegmentWalker
    for parameter 2 at io.druid.server.QueryResource.<init>(QueryResource.java:76)
  while locating io.druid.server.QueryResource

I’m launching the broker with this script:

~/Development/druid master cat bin/runit/broker
#!/bin/bash

export PATH=/usr/java/jdk1.7.0_25/bin:$PATH

app_dir=/data/app/druid/current
log_dir=$app_dir/log

cd $(dirname $0)/../..

exec java -server \
  -Xmx4g \
  -Xms2g \
  -XX:NewSize=256m \
  -XX:MaxNewSize=1g \
  -XX:+UseConcMarkSweepGC \
  -Duser.timezone=UTC \
  -Dfile.encoding=UTF-8 \
  -Djava.io.tmpdir=/tmp/druid \
  -classpath lib/*:etc/broker io.druid.cli.Main server broker > $log_dir/broker.log 2>&1

With these properties:

~/Development/druid master cat etc/broker/runtime.properties
druid.host=aia24.sjc1b.square:29960
druid.port=29960
druid.service=druid/prod/broker

druid.zk.service.host=host1.square,host2.square,host3.square
druid.zk.paths.base=/druid/prod

druid.broker.http.readTimeout=PT5M

I can’t seem to find why it’s trying to use so much memory, and I didn’t see this problem in 0.6.52.  Does anyone have any idea why this is happening?  Thanks for your help!

Best,
Sunil

[Fangjin Yang]

Hi Sunil, can you set druid.processing.numThreads to something lower (e.g. 1, which was the default on the broker before). This change came out because druid.processing.numThreads defaulted to 1 before and the default is now numCores - 1.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAOKMzKDKZL-b9CMdp%3DS29x0j%3DGhiA9xX59%3D14TJNOdRMdyd64Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Fangjin Yang]

The broker doesn't actually use this config, but because of the way Druid validates processing configs, it's an unfortunate check and exception that gets thrown. The default value was changed because I noticed a general problem of people incorrectly configuring historical nodes and not really parallel processing segments at query time.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.