ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while run

862 views

Skip to first unread message

HoonDong Kim

unread,

Apr 22, 2015, 12:22:16 AM4/22/15

to druid...@googlegroups.com

Hi. I am testing the druid cluster for our company's realtime analytics platform.

I am testing at 5 Separate Node cluster. ( broker, historical , middlemanager , coordinator , overlord )

I am used zookeeper , mysql ,S3 storage and local file storage at this test.

In our production Hadoop eco system, we have hadoop and memcached layer. But, that system is production environment. So, At this testing, I don't use hadoop and memcached.

But, At this test, the batch data loading process alaways fail.

Is there any problem in my setting file????

I use this command.

./load-druid.sh druid05_domain:8080 /data01/druid/druid-0.7.0/examples/indexing/wikipedia_index_task.json

ps.

i don't know the baseKey's meaning . so, at below property, i use the s3 folder name.

druid.indexer.fork.property.druid.storage.baseKey = prod

---------_common --------

# Extensions

druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions","io.druid.extensions:druid-examples","io.druid.extensions:druid-histogram","io.druid.extensions:mysql-metadata-storage"]

# Zookeeper

druid.zk.service.host=vm-druid04

druid.zk.base=/druid/prod

druid.discovery.curator.path=/prod/discovery

# Request logging, monitoring, and metrics

druid.request.logging.type=emitter

druid.request.logging.feed=druid_requests

# Metadata Storage (mysql)

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI=jdbc\:mysql\://vm-druid05\:3306/druid

druid.metadata.storage.connector.user=druid

druid.metadata.storage.connector.password=***********

# Deep storage (local filesystem for examples - don't use this in production)

druid.storage.type=s3

druid.s3.accessKey=******

druid.s3.secretKey=**********

druid.storage.storage.storageDirectory=/mnt/resource/data02/druid/localStorage

# Query Cache (we use a simple 1000mb heap-based local cache on the broker)

druid.cache.type=local

druid.cache.sizeInBytes=1000000000

# Indexing service discovery

druid.selectors.indexing.serviceName=druid:prod:overlord

# Monitoring (disabled for examples)

# druid.monitoring.monitors=["com.metamx.metrics.SysMonitor","com.metamx.metrics.JvmMonitor"]

# Metrics logging (disabled for examples)

druid.emitter=noop

--------- broker ----------

druid.host=vm-broker01

druid.port=8080

druid.service=druid/prod/broker

# We enable using the local query cache here

druid.broker.cache.useCache=true

druid.broker.cache.populateCache=true

druid.broker.http.numConnections=20

druid.broker.http.readTimeout=PT5M

# Bump these up only for faster nested groupBy

druid.processing.buffer.sizeBytes=2147483647

druid.processing.numThreads=31

druid.server.http.numThreads=50

druid.announcer.type=batch

druid.curator.compress=true

---------middle manager --------

druid.host=vm-druid02.cloudapp.net:8080

druid.port=8080

druid.service=druid/prod/middlemanager

# Store task logs in deep storage

#druid.indexer.logs.type=s3

#druid.indexer.logs.s3Bucket=log-hoondongkim

#druid.indexer.logs.s3Prefix=prod/logs/v1

# Resources for peons

druid.indexer.runner.javaOpts=-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

druid.indexer.task.baseTaskDir=/mnt/resource/data02/persistent/task/

# Peon properties

druid.indexer.fork.property.druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.segmentCache.locations=[{"path": "/mnt/resource/data02/persistent/zk_druid", "maxSize": 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.storage.archiveBaseKey=prod

druid.indexer.fork.property.druid.storage.archiveBucket=archive-hoondongkim

druid.indexer.fork.property.druid.storage.baseKey=prod/v1

druid.indexer.fork.property.druid.storage.bucket=hoondongkim

druid.indexer.fork.property.druid.storage.type=s3

druid.worker.capacity=9

druid.worker.ip=vm-druid02

druid.worker.version=0

---------- overload ----------

druid.host=vm-druid05:8080

druid.port=8080

druid.service=druid/prod/overlord

# Run the overlord in local mode with a single peon to execute tasks

#druid.indexer.queue.startDelay=PT5M

#druid.indexer.runner.javaOpts="-server -Xmx16g"

#druid.indexer.fork.property.druid.processing.numThreads=1

#druid.indexer.fork.property.druid.computation.buffer.size=100000000

# Only required if you are autoscaling middle managers

druid.indexer.autoscale.doAutoscale=true

druid.indexer.autoscale.strategy=ec2

druid.indexer.autoscale.workerIdleTimeout=PT90m

druid.indexer.autoscale.terminatePeriod=PT5M

druid.indexer.autoscale.workerVersion=0

# Upload all task logs to deep storage

druid.indexer.logs.type=s3

druid.indexer.logs.s3Bucket=log-hoondongkim

druid.indexer.logs.s3Prefix=prod/logs/v1

# Run in remote mode

druid.indexer.runner.type=remote

druid.indexer.runner.minWorkerVersion=0

# Store all task state in the metadata storage

druid.indexer.storage.type=metadata

------------------------------ Error Message ---------------------------------

WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected.

2015-04-22T03:47:51,815 INFO [task-runner-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@8c7bf95]

2015-04-22T03:47:51,824 INFO [task-runner-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}]

2015-04-22T03:47:51,825 INFO [task-runner-0] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@5828da3d]

2015-04-22T03:47:51,848 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Estimated approximately [5.006113] rows of data.

2015-04-22T03:47:51,849 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Will require [1] shard(s).

2015-04-22T03:47:51,850 INFO [task-runner-0] io.druid.segment.realtime.firehose.LocalFirehoseFactory - Searching for all [wikipedia_data.json] in [/data01/druid/druid-0.7.0/examples/indexing]

2015-04-22T03:47:51,935 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[IndexTask{id=index_wikipedia_2015-04-22T03:47:42.851Z, type=index, dataSource=wikipedia}]

java.lang.ExceptionInInitializerError

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:323) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) [druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) [druid-indexing-service-0.7.0.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]

Caused by: java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1429674471861-0 to 1429674471861-9999)

at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]

at io.druid.segment.indexing.RealtimeTuningConfig.<clinit>(RealtimeTuningConfig.java:40) ~[druid-server-0.7.0.jar:0.7.0]

... 8 more

2015-04-22T03:47:51,947 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError

at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:162) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.cli.CliPeon.run(CliPeon.java:209) [druid-services-0.7.0.jar:0.7.0]

at io.druid.cli.Main.main(Main.java:88) [druid-services-0.7.0.jar:0.7.0]

Caused by: java.util.concurrent.ExecutionException: java.lang.ExceptionInInitializerError

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]

at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:159) ~[druid-indexing-service-0.7.0.jar:0.7.0]

... 2 more

Caused by: java.lang.ExceptionInInitializerError

at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:323) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:185) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:235) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:214) ~[druid-indexing-service-0.7.0.jar:0.7.0]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_75]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_75]

at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_75]