realtime node woes

[David Montgomery]

Hi,

I am giving up om kafka 8.  going back to 7.  Now..with the latest version of durid..I am now getting this error i n the realtime node.  How do I resolve? 


[{
  "schema" : { "dataSource":"datasource-pixel",
               "aggregators":[ {"type":"count", "name":"impressions"}],
               "indexGranularity":"minute",
                   "shardSpec" : { "type": "none" } },
  "config" : { "maxRowsInMemory" : 500000,
               "intermediatePersistPeriod" : "PT10m" },
  "firehose" : { "type" : "kafka-0.7.2",
                 "consumerProps" : { "zk.connect" : "1.dzk.do.development.singapore.testcom:2181",
                                     "zk.connectiontimeout.ms" : "15000",
                                     "zk.sessiontimeout.ms" : "15000",
                                     "zk.synctime.ms" : "5000",
                                     "groupid" : "topic-pixel",
                                     "autooffset.reset" : "largest",
                                     "autocommit.enable" : "false" },
                 "feed" : "topic-pixel",
                 "parser" : { "timestampSpec" : { "column" : "utcdt", "format" : "iso" },
                              "data" : { "format" : "json" },
                              "dimensionExclusions" : ["deleteme"] } },
  "plumber" : { "type" : "realtime",
                "windowPeriod" : "PT10m",
                "segmentGranularity":"hour",
                "basePersistDirectory" : "/tmp/realtime/basePersist" }

}
]





2014-02-28 12:01:38,857 INFO [main] kafka.consumer.ZookeeperConsumerConnector - topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 begin rebalancing consumer topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 try #2
2014-02-28 12:01:40,221 INFO [main] kafka.consumer.ZookeeperConsumerConnector - topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 exception during rebalance
java.lang.NumberFormatException: null
    at java.lang.Integer.parseInt(Integer.java:454)
    at java.lang.Integer.parseInt(Integer.java:527)
    at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:231)
    at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
    at kafka.utils.ZkUtils$$anonfun$getPartitionsForTopics$1$$anonfun$apply$2.apply(ZkUtils.scala:216)
    at kafka.utils.ZkUtils$$anonfun$getPartitionsForTopics$1$$anonfun$apply$2.apply(ZkUtils.scala:215)
    at scala.collection.Iterator$class.foreach(Iterator.scala:772)
    at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
    at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
    at kafka.utils.ZkUtils$$anonfun$getPartitionsForTopics$1.apply(ZkUtils.scala:215)
    at kafka.utils.ZkUtils$$anonfun$getPartitionsForTopics$1.apply(ZkUtils.scala:212)
    at scala.collection.Iterator$class.foreach(Iterator.scala:772)
    at scala.collection.mutable.HashMap$$anon$3.foreach(HashMap.scala:108)
    at kafka.utils.ZkUtils$.getPartitionsForTopics(ZkUtils.scala:212)
    at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:479)
    at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:449)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
    at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:444)
    at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:733)
    at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:207)
    at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:77)
    at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:91)
    at io.druid.firehose.kafka.KafkaSevenFirehoseFactory.connect(KafkaSevenFirehoseFactory.java:91)
    at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:90)
    at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:150)
    at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331)
    at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250)
    at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136)
    at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:72)
    at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40)
    at io.druid.cli.Main.main(Main.java:91)
2014-02-28 12:01:40,224 INFO [main] kafka.consumer.ZookeeperConsumerConnector - topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 end rebalancing consumer topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 try #2
2014-02-28 12:01:40,224 INFO [main] kafka.consumer.ZookeeperConsumerConnector - topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered
2014-02-28 12:01:40,225 INFO [main] kafka.consumer.Fetcher - Cleared all relevant queues for this fetcher
2014-02-28 12:01:40,225 INFO [main] kafka.consumer.ConsumerIterator - Clearing the current data chunk for this consumer iterator
2014-02-28 12:01:40,226 INFO [main] kafka.consumer.Fetcher - Cleared the data chunks in all the consumer message iterators
2014-02-28 12:01:40,226 INFO [main] kafka.consumer.ZookeeperConsumerConnector - topic-pixel_do-druidrealtime-sg-development-20140228111325-1393588884141-b7feaab3 Committing all offsets after clearing the fetcher queues

[David Montgomery]

Below is what I see in zookeeper.

print client.get_children('/consumers/druid/ids')

['druid_do-druidrealtime-sg-development-20140228111325-1393668893703-dd65fc42', 'druid_do-druidrealtime-sg-development-20140228111325-1393668930309-cd081643']

Using python I can read and write to kafka. 

Any ideas n how to resolve? 

[Fangjin Yang]

Hi David, I'm not entirely sure what is wrong, hopefully someone more knowledgable with Kafka than I can help, but you can also try the Kafka user-groups for help with Kafka setup:

https://kafka.apache.org/contact.html

The problem definitely seems to be a problem in the Kafka 7 setup. FWIW, Kafka 8 should work as far as I know. Other members of the community have it running, and hopefully someone else can assist here.

-- FJ

[Gian Merlino]

That exception looks like kafka 0.7 trying to read kafka 0.8 metadata. Kafka 0.7 and 0.8 zk metadata are not compatible and it will generally cause problems to use the same path with two different versions. You can deal with that by clearing out the path completely when switching versions (if you don't care about old kafka metadata) or by using a different base zk path for kafka 0.8 and kafka 0.7.

On Saturday, March 1, 2014 2:32:15 AM UTC-8, David Montgomery wrote:

[David Montgomery]

Hi,

I rebuilt realtime node, kafka, zookeeper.....that error is gone.now now this.  I have verified that kafka works via python producing and consuming usig brod kafka client.  I cant imagine there is anything wrong with kafka and zookeeper  I had druid working with druid 5.x.

6x is a different gain and finding it hard past the online docs.  There is clearly a configuration option that is missing that I guess everyone can figure out but me regarding kafka.

What is soooo different about setting up kafka that worked with 5x but not with 6x?

This how i install kafka
cd kafka-0.7.2-incubating-src
./sbt update
./sbt package
echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"' | tee -a /var/lib/kafka-0.7.2-incubating-src/bin/kafka-run-class.sh
echo 'export JMX_PORT=${JMX_PORT:-9999}' | tee -a /var/lib/kafka-0.7.2-incubating-src/bin/kafka-server-start.sh

this is how I start kafka
/var/lib/kafka-0.7.2-incubating-src/bin/kafka-server-start.sh /var/lib/kafka-0.7.2-incubating-src/config/server.properties

scala libs are here and I caped moved

/var/lib/kafka-0.7.2-incubating-src/core/lib_managed/scala_2.8.0/test/scalatest-1.2.jar
/var/lib/kafka-0.7.2-incubating-src/project/boot/scala-2.7.7/lib/scala-compiler.jar
/var/lib/kafka-0.7.2-incubating-src/project/boot/scala-2.7.7/lib/scala-library.jar
/var/lib/kafka-0.7.2-incubating-src/project/boot/scala-2.8.0/lib/scala-compiler.jar
/var/lib/kafka-0.7.2-incubating-src/project/boot/scala-2.8.0/lib/scala-library.jar

I get the below error......

2014-03-03 10:49:42,456 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=1.dzk.do.development.singapore.rtbhui.com:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@5af3a14
2014-03-03 10:49:42,568 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.announcement.Announcer.start()] on object[io.druid.curator.announcement.Announcer@398f573b].
2014-03-03 10:49:42,577 INFO [main-SendThread(128.199.238.103:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 128.199.238.103/128.199.238.103:2181. Will not attempt to authenticate using SASL (unknown error)
2014-03-03 10:49:42,578 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.client.ServerInventoryView.start() throws java.lang.Exception] on object[io.druid.client.SingleServerInventoryView@3e0ea213].
2014-03-03 10:49:42,610 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.segment.realtime.RealtimeManager.start() throws java.io.IOException] on object[io.druid.segment.realtime.RealtimeManager@60532a0a].
2014-03-03 10:49:42,614 INFO [main] io.druid.segment.realtime.RealtimeManager - Calling the FireDepartment and getting a Firehose.
2014-03-03 10:49:42,617 ERROR [main] io.druid.cli.CliBroker - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:331)
    at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:250)
    at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:136)
    at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:72)
    at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40)
    at io.druid.cli.Main.main(Main.java:91)

Caused by: java.lang.NoClassDefFoundError: scala/ScalaObject
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at io.druid.firehose.kafka.KafkaSevenFirehoseFactory.connect(KafkaSevenFirehoseFactory.java:89)

    at io.druid.segment.realtime.FireDepartment.connect(FireDepartment.java:90)
    at io.druid.segment.realtime.RealtimeManager$FireChief.init(RealtimeManager.java:150)
    at io.druid.segment.realtime.RealtimeManager.start(RealtimeManager.java:87)

    ... 10 more
Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 25 more
^Z

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b23f91fd-edfe-4c70-bf54-fcbb3cc7a79d%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Fangjin Yang]

Hi David, to test that your ZK can talk to Druid, even in a local setup, can you try following the steps in http://druid.io/docs/latest/Tutorial:-Loading-Your-Data-Part-1.html? That should outline what is needed to setup Kafka with Druid.

[David Montgomery]

I rand the below query from the realtime node.

/home/ubuntu/druid-services-0.6.52# curl -X 'POST' -H 'Content-Type:application/json' -d @examples/indexing/wikipedia_index_task.json 107.170.226.65:8082/druid/indexer/v1/task
{"task":"index_wikipedia_2014-03-04T04:29:12.092Z"}

In the logs on the overlord node i get the below..I am assume its this is the failure?  Then why?  Received FAILED status for task


druid.host=107.170.222.11
druid.port=8082
druid.service=overlord
druid.zk.service.host=1.dzk.do.development.singapore.test.com:2181
druid.db.connector.connectURI=jdbc:mysql://druid.mysql.do.development.singapore.rtestcom:3306/dry
druid.db.connector.user=test
druid.db.connector.password=test
druid.selectors.indexing.serviceName=overlord
druid.indexer.queue.startDelay=PT0M
druid.indexer.runner.javaOpts="-server -Xmx1g"
druid.indexer.runner.startPort=8082
druid.indexer.fork.property.druid.computation.buffer.size=268435456

2014-03-04 04:30:34,923 INFO [TaskQueue-Manager] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_wikipedia_2014-03-04T04:30:34.922Z]: LockTryAcquireAction{interval=2013-08-31T00:00:00.000Z/2013-09-01T00:00:00.000Z}
2014-03-04 04:30:34,923 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskLockbox - Created new TaskLockPosse: TaskLockPosse{taskLock=TaskLock{groupId=index_wikipedia_2014-03-04T04:30:34.922Z, dataSource=wikipedia, interval=2013-08-31T00:00:00.000Z/2013-09-01T00:00:00.000Z, version=2014-03-04T04:30:34.923Z}, taskIds=[]}
2014-03-04 04:30:34,923 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskLockbox - Added task[index_wikipedia_2014-03-04T04:30:34.922Z] to TaskLock[index_wikipedia_2014-03-04T04:30:34.922Z]
2014-03-04 04:30:34,924 INFO [TaskQueue-Manager] io.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_wikipedia_2014-03-04T04:30:34.922Z
2014-03-04 04:30:34,927 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Running command: java -cp overlord/:/home/ubuntu/druid-services-0.6.52/lib/server-metrics-0.0.5.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-io-2.0.1.jar:/home/ubuntu/druid-services-0.6.52/lib/hibernate-validator-5.0.1.Final.jar:/home/ubuntu/druid-services-0.6.52/lib/lz4-1.1.2.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-lang-2.6.jar:/home/ubuntu/druid-services-0.6.52/lib/asm-3.1.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-model-builder-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jdbi-2.32.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-jaxrs-base-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/aopalliance-1.0.jar:/home/ubuntu/druid-services-0.6.52/lib/jersey-servlet-1.17.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-datatype-guava-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/tesla-aether-0.0.5.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-spi-0.9.0.M2.jar:/home/ubuntu/druid-services-0.6.52/lib/curator-recipes-2.3.0.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-repository-metadata-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/sigar-1.6.5.132.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-common-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/java-util-0.25.2.jar:/home/ubuntu/druid-services-0.6.52/lib/extendedset-1.3.4.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-module-jaxb-annotations-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-core-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/javax.inject-1.jar:/home/ubuntu/druid-services-0.6.52/lib/classmate-0.8.0.jar:/home/ubuntu/druid-services-0.6.52/lib/curator-x-discovery-2.3.0.jar:/home/ubuntu/druid-services-0.6.52/lib/spymemcached-2.8.4.jar:/home/ubuntu/druid-services-0.6.52/lib/wagon-provider-api-2.4.jar:/home/ubuntu/druid-services-0.6.52/lib/curator-client-2.3.0.jar:/home/ubuntu/druid-services-0.6.52/lib/log4j-1.2.16.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-http-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-processing-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/irc-api-1.0-0011.jar:/home/ubuntu/druid-services-0.6.52/lib/httpclient-4.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-servlets-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-codec-1.7.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-cli-1.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jets3t-0.9.0.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-dataformat-smile-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/plexus-utils-3.0.15.jar:/home/ubuntu/druid-services-0.6.52/lib/guava-14.0.1.jar:/home/ubuntu/druid-services-0.6.52/lib/protobuf-java-2.4.0a.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-connector-file-0.9.0.M2.jar:/home/ubuntu/druid-services-0.6.52/lib/joda-time-2.1.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-connector-okhttp-0.0.9.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-indexing-service-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/maxminddb-0.2.0.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-server-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-api-0.9.0.M2.jar:/home/ubuntu/druid-services-0.6.52/lib/slf4j-api-1.6.4.jar:/home/ubuntu/druid-services-0.6.52/lib/compress-lzf-0.8.4.jar:/home/ubuntu/druid-services-0.6.52/lib/org.abego.treelayout.core-1.0.1.jar:/home/ubuntu/druid-services-0.6.52/lib/aws-java-sdk-1.6.0.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-security-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/antlr4-runtime-4.0.jar:/home/ubuntu/druid-services-0.6.52/lib/google-http-client-jackson2-1.15.0-rc.jar:/home/ubuntu/druid-services-0.6.52/lib/plexus-interpolation-1.19.jar:/home/ubuntu/druid-services-0.6.52/lib/http-client-0.8.5.jar:/home/ubuntu/druid-services-0.6.52/lib/guice-multibindings-4.0-beta.jar:/home/ubuntu/druid-services-0.6.52/lib/guice-4.0-beta.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-util-0.9.0.M2.jar:/home/ubuntu/druid-services-0.6.52/lib/javax.servlet-3.0.0.v201112011016.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-pool-1.6.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-indexing-hadoop-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-jaxrs-json-provider-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/validation-api-1.1.0.Final.jar:/home/ubuntu/druid-services-0.6.52/lib/jersey-server-1.17.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-continuation-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/okhttp-1.0.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jline-0.9.94.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-server-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-datatype-joda-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-api-0.1.7.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-aether-provider-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/httpcore-4.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-mapper-asl-1.9.11.jar:/home/ubuntu/druid-services-0.6.52/lib/curator-framework-2.3.0.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-settings-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/java-xmlbuilder-0.4.jar:/home/ubuntu/druid-services-0.6.52/lib/rhino-1.7R4.jar:/home/ubuntu/druid-services-0.6.52/lib/emitter-0.2.9.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-model-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/slf4j-log4j12-1.6.2.jar:/home/ubuntu/druid-services-0.6.52/lib/config-magic-0.9.jar:/home/ubuntu/druid-services-0.6.52/lib/maven-settings-builder-3.1.1.jar:/home/ubuntu/druid-services-0.6.52/lib/guice-servlet-4.0-beta.jar:/home/ubuntu/druid-services-0.6.52/lib/bytebuffer-collections-0.0.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jersey-guice-1.17.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-databind-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/airline-0.6.jar:/home/ubuntu/druid-services-0.6.52/lib/geoip2-0.4.0.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-util-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/zookeeper-3.4.5.jar:/home/ubuntu/druid-services-0.6.52/lib/icu4j-4.8.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-io-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-servlet-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/jboss-logging-3.1.1.GA.jar:/home/ubuntu/druid-services-0.6.52/lib/jsr305-2.0.1.jar:/home/ubuntu/druid-services-0.6.52/lib/jersey-core-1.17.1.jar:/home/ubuntu/druid-services-0.6.52/lib/aether-impl-0.9.0.M2.jar:/home/ubuntu/druid-services-0.6.52/lib/xpp3-1.1.4c.jar:/home/ubuntu/druid-services-0.6.52/lib/druid-services-0.6.52.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-dbcp-1.4.jar:/home/ubuntu/druid-services-0.6.52/lib/opencsv-2.3.jar:/home/ubuntu/druid-services-0.6.52/lib/jetty-client-8.1.11.v20130520.jar:/home/ubuntu/druid-services-0.6.52/lib/netty-3.2.4.Final.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-annotations-2.2.2.jar:/home/ubuntu/druid-services-0.6.52/lib/jackson-core-asl-1.9.11.jar:/home/ubuntu/druid-services-0.6.52/lib/google-http-client-1.15.0-rc.jar:/home/ubuntu/druid-services-0.6.52/lib/mysql-connector-java-5.1.18.jar:/home/ubuntu/druid-services-0.6.52/lib/commons-logging-1.1.1.jar:/home/ubuntu/druid-services-0.6.52/config/overlord -Ddruid.indexer.runner.startPort=8082 -Ddruid.db.connector.password=Rtbops808 -Dfile.encoding.pkg=sun.io -Djava.io.tmpdir=/tmp -Ddruid.db.connector.connectURI=jdbc:mysql://druid.mysql.do.development.singapore.rtbhui.com:3306/dry -Ddruid.port=8082 -Ddruid.db.connector.user=root -Ddruid.selectors.indexing.serviceName=overlord -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.indexer.queue.startDelay=PT0M -Ddruid.service=overlord -Ddruid.indexer.fork.property.druid.computation.buffer.size=268435456 -Ddruid.host=107.170.226.65 -Ddruid.indexer.runner.javaOpts="-server -Xmx1g" -Ddruid.zk.service.host=1.dzk.do.development.singapore.rtbhui.com:2181 -Ddruid.computation.buffer.size=268435456 -Ddruid.host=107.170.226.65:8082 -Ddruid.port=8082 io.druid.cli.Main internal peon /tmp/persistent/index_wikipedia_2014-03-04T04:30:34.922Z/722bf049-8b33-432d-b2ff-882f8e7f584f/task.json /tmp/persistent/index_wikipedia_2014-03-04T04:30:34.922Z/722bf049-8b33-432d-b2ff-882f8e7f584f/status.json
2014-03-04 04:30:34,936 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Logging task index_wikipedia_2014-03-04T04:30:34.922Z output to: /tmp/persistent/index_wikipedia_2014-03-04T04:30:34.922Z/722bf049-8b33-432d-b2ff-882f8e7f584f/log
2014-03-04 04:30:39,583 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[1] for task: index_wikipedia_2014-03-04T04:30:34.922Z
2014-03-04 04:30:39,585 INFO [pool-6-thread-1] io.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: log/index_wikipedia_2014-03-04T04:30:34.922Z.log
2014-03-04 04:30:39,585 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Removing temporary directory: /tmp/persistent/index_wikipedia_2014-03-04T04:30:34.922Z/722bf049-8b33-432d-b2ff-882f8e7f584f
2014-03-04 04:30:39,589 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_wikipedia_2014-03-04T04:30:34.922Z
2014-03-04 04:30:39,589 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_wikipedia_2014-03-04T04:30:34.922Z
2014-03-04 04:30:39,589 INFO [pool-6-thread-1] io.druid.indexing.overlord.HeapMemoryTaskStorage - Updating task index_wikipedia_2014-03-04T04:30:34.922Z to status: TaskStatus{id=index_wikipedia_2014-03-04T04:30:34.922Z, status=FAILED, duration=-1}
2014-03-04 04:30:39,589 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_wikipedia_2014-03-04T04:30:34.922Z] from TaskLock[index_wikipedia_2014-03-04T04:30:34.922Z]
2014-03-04 04:30:39,591 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_wikipedia_2014-03-04T04:30:34.922Z, dataSource=wikipedia, interval=2013-08-31T00:00:00.000Z/2013-09-01T00:00:00.000Z, version=2014-03-04T04:30:34.923Z}
2014-03-04 04:30:39,592 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Task done: IndexTask{id=index_wikipedia_2014-03-04T04:30:34.922Z, type=index, dataSource=wikipedia}
2014-03-04 04:30:39,592 INFO [pool-6-thread-1] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2014-03-04T04:30:39.592Z","service":"overlord","host":"107.170.226.65:8082","metric":"indexer/time/run/millis","value":-1,"user2":"wikipedia","user3":"FAILED","user4":"index"}]
2014-03-04 04:30:39,593 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Task FAILED: IndexTask{id=index_wikipedia_2014-03-04T04:30:34.922Z, type=index, dataSource=wikipedia} (-1 run duration)
2014-03-04 04:31:06,894 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).
2014-03-04 04:32:06,894 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).
2014-03-04 04:33:06,894 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).
2014-03-04 04:34:06,895 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b23fd55f-aea4-4c24-863a-e8f69ed52d2e%40googlegroups.com.

[David Montgomery]

Also,

In the in the console..I am seeing this......

http://107.170.1111.111:8082/druid/indexer/v1/task/index_wikipedia_2014-03-04T07%3A31%3A53.442Z/log

I am on a 1 gig ram box in digi ocean for the overload and still get the error. 

I changed to the below
druid.indexer.fork.property.druid.computation.buffer.size=536870912
druid.computation.buffer.size=536870912

I am not seenig the below errpr in the logs.  How to I enable so I can see tat level of detail? I had to go to the web console to see the error.

.google.inject.ProvisionException: Guice provision errors:

1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize or druid.computation.buffer.size: maxDirectMemory[249,364,480], memoryNeeded[536,870,912], druid.computation.buffer.size[268,435,456], numThreads[1]
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:76)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:76)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:76)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:71)
  while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:61)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:68)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=22, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:52)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:78)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:124)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:68)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:150)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.query.QuerySegmentWalker
    for parameter 2 at io.druid.server.QueryResource.<init>(QueryResource.java:75)
  while locating io.druid.server.QueryResource

[Fangjin Yang]

Hi David, the error is occurring because you don't have enough direct memory for your intermediate computation buffer size. To fix the problem, set your JVM max direct memory to 536,870,912.

BTW, if you are still seeing Kafka errors, I'd recommend checking out: http://druid.io/docs/0.6.52/Tutorial:-Loading-Your-Data-Part-2.html

I posted the wrong link before, but the first section of that tutorial talks about setting up a clean Kafka and getting a real-time node to work with it. It may be able to help you with our Kafka woes. Apologies for the difficulties you've been having so far. Druid is a young project so we definitely could use more documentation in places.