Historical nodes throwing an exception though are able to load segments

SAURABH VERMA

unread,

Feb 14, 2015, 7:05:29 AM2/14/15

to druid-de...@googlegroups.com

Hello,

The historical node is throwing an exception as below (Druid version 0.6.171):-

"Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[viddata_2015-02-14T03:00:00.000-07:00_2015-02-14T04:00:00.000-07:00_2015-02-14T03:36:31.779-07:00], segment=DataSegment{size=129167987, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[xCount], dimensions=[colo, pool, status, typename], version='2015-02-14T03:36:31.779-07:00', loadSpec={type=hdfs, path=hdfs://10.86.184.19:54310/druid/segments/viddata/20150214T030000.000-0700_20150214T040000.000-0700/2015-02-14T03_36_31.779-07_00/0/index.zip}, interval=2015-02-14T03:00:00.000-07:00/2015-02-14T04:00:00.000-07:00, dataSource='viddata', binaryVersion='9'}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[viddata_2015-02-14T03:00:00.000-07:00_2015-02-14T04:00:00.000-07:00_2015-02-14T03:36:31.779-07:00]

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:140)

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:165)

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:44)

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:127)

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516)

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:510)

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:507)

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:759)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:780)

Caused by: io.druid.segment.loading.SegmentLoadingException: /import/home/mon/druid/persistent/zk_druid/viddata/2015-02-14T03:00:00.000-07:00_2015-02-14T04:00:00.000-07:00/2015-02-14T03:36:31.779-07:00/0/index.drd (No such file or directory)

at io.druid.segment.loading.MMappedQueryableIndexFactory.factorize(MMappedQueryableIndexFactory.java:42)

at io.druid.segment.loading.OmniSegmentLoader.getSegment(OmniSegmentLoader.java:96)

at io.druid.server.coordination.ServerManager.loadSegment(ServerManager.java:145)

at io.druid.server.coordination.ZkCoordinator.loadSegment(ZkCoordinator.java:136)

... 20 more

Caused by: java.io.FileNotFoundException: /import/home/mon/druid/persistent/zk_druid/viddata/2015-02-14T03:00:00.000-07:00_2015-02-14T04:00:00.000-07:00/2015-02-14T03:36:31.779-07:00/0/index.drd (No such file or directory)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at io.druid.segment.SegmentUtils.getVersionFromDir(SegmentUtils.java:24)

at io.druid.segment.IndexIO.loadIndex(IndexIO.java:159)

at io.druid.segment.loading.MMappedQueryableIndexFactory.factorize(MMappedQueryableIndexFactory.java:39)

"

The historical runprops is :-

"druid.host=phxscal1100

druid.service=historical

druid.port=8081

druid.zk.service.host=phxscal1041:2181

druid.extensions.localRepository=/import/home/mon/druid171PullReps

druid.extensions.remoteRepositories=[]

druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage:0.6.171"]

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs://10.86.184.19:54310/druid/segments

druid.server.tier=_default_tier

druid.server.maxSize=10000000000

# Change these to make Druid faster

druid.processing.buffer.sizeBytes=1000000000

druid.processing.numThreads=4

druid.segmentCache.locations=[{"path": "/import/home/mon/druid/persistent/zk_druid", "maxSize"\: 40000000000}]

"

The middleManager runprops is :-

"druid.host=phxscal1111

druid.port=8080

druid.service=middleManager

druid.extensions.remoteRepositories=[]

#druid.extensions.localRepository=/import/home/mon/druidRepository

druid.extensions.localRepository=/import/home/mon/druid171PullReps

druid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight:0.6.171","io.druid.extensions:druid-hdfs-storage:0.6.171"]

druid.zk.service.host=phxscal1041:2181

druid.selectors.indexing.serviceName=overlord

# Dedicate more resources to peons

druid.indexer.runner.javaCommand=/home/mon/jdk1.6/bin/java

druid.indexer.runner.javaOpts=-server -Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:MaxDirectMemorySize=7g -Duser.timezone=MST -Dfile.encoding=UTF-8

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.4.0"]

druid.indexer.task.baseTaskDir=/home/mon/druid/persistent/task

druid.indexer.task.chathandler.type=announce

druid.indexer.runner.startPort=9080

#druid.indexer.fork.property.druid.storage.type=local

#druid.indexer.fork.property.druid.storage.storageDirectory=/import/home/mon/druid/localstorage

druid.indexer.fork.property.druid.storage.type=hdfs

druid.indexer.fork.property.druid.storage.storageDirectory=hdfs://10.86.184.19:54310/druid/segments

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

druid.indexer.fork.property.druid.processing.numThreads=6

druid.indexer.fork.property.druid.request.logging.type=file

druid.indexer.fork.property.druid.request.logging.dir=request_logs/

druid.indexer.fork.property.druid.segmentCache.locations=[{"path": "/import/home/mon/druid/persistent/zk_druid", "maxSize": 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.computation.buffer.size=268435456

druid.worker.capacity=4

#mysql

druid.db.connector.connectURI=jdbc:mysql://mymisc1.db.stratus.XYZ.com:3306/test

druid.db.connector.user=caljobs_app

druid.db.connector.password=caljobs_app

"

Please help resolve the issue

Thanks,

Saurabh

Gian Merlino

unread,

Feb 14, 2015, 10:39:50 AM2/14/15

to druid-de...@googlegroups.com

If you download and unpack the archive hdfs://10.86.184.19:54310/druid/segments/viddata/20150214T030000.000-0700_20150214T040000.000-0700/2015-02-14T03_36_31.779-07_00/0/index.zip, what files are in it and how big are they?

Is there anything in the /import/home/mon/druid/persistent/zk_druid/viddata/2015-02-14T03:00:00.000-07:00_2015-02-14T04:00:00.000-07:00/2015-02-14T03:36:31.779-07:00/ directory on your historical? Does that directory exist at all (even empty)?

SAURABH VERMA

unread,

Feb 14, 2015, 3:09:35 PM2/14/15

to druid-de...@googlegroups.com

Gian,

1. The index.zip has version.bin, meta.smoosh, 00000.smoosh

2. The path (/import/home/mon....) should be created by the middlemanager, (as of now that folder does not exist), i believe its a path that is accessible from middlemanager and historical both

Gian Merlino

unread,

Feb 14, 2015, 3:19:17 PM2/14/15

to druid-de...@googlegroups.com

The "druid.segmentCache.locations" should be a directory that is local to each historical node. It's used to locally cache segments and each historical node assumes it can add and remove segments from the cache whenever it wants. Sharing the mount between nodes might cause problems, so can you try changing it to a local path?

SAURABH VERMA

unread,

Feb 16, 2015, 10:37:26 AM2/16/15

to druid-de...@googlegroups.com

Gian, the error is still appearing in historical nodes console.

Right now I have changed segmentCacheLocation to a local path, below is the error log :-

"2015-02-16 08:31:34,317 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completely removing [viddata_2015-02-16T07:00:00.000-07:00_2015-02-16T08:00:00.000-07:00_2015-02-16T07:03:00.531-07:00] in [30,000] millis

2015-02-16 08:31:34,318 INFO [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: viddata_2015-02-16T07:00:00.000-07:00_2015-02-16T08:00:00.000-07:00_2015-02-16T07:03:00.531-07:00]

2015-02-16 08:31:34,318 ERROR [ZkCoordinator-0] io.druid.server.coordination.ZkCoordinator - Failed to load segment for dataSource: {class=io.druid.server.coordination.ZkCoordinator, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[viddata_2015-02-16T07:00:00.000-07:00_2015-02-16T08:00:00.000-07:00_2015-02-16T07:03:00.531-07:00], segment=DataSegment{size=510132161, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[xCount], dimensions=[colo, duration, machine, pool, status, type, typename], version='2015-02-16T07:03:00.531-07:00', loadSpec={type=hdfs, path=hdfs://10.86.184.19:54310/druid/segments/viddata/20150216T070000.000-0700_20150216T080000.000-0700/2015-02-16T07_03_00.531-07_00/0/index.zip}, interval=2015-02-16T07:00:00.000-07:00/2015-02-16T08:00:00.000-07:00, dataSource='viddata', binaryVersion='9'}}

io.druid.segment.loading.SegmentLoadingException: Exception loading segment[viddata_2015-02-16T07:00:00.000-07:00_2015-02-16T08:00:00.000-07:00_2015-02-16T07:03:00.531-07:00]