Exceptions during handoff

[SAURABH VERMA]

Hello,

 I am seeing the below exception peon console :-

 io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[VidData]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.io.IOException, exceptionMessage=No FileSystem for scheme: hdfs, interval=2015-02-13T11:00:00.000-07:00/2015-02-13T12:00:00.000-07:00}

java.io.IOException: No FileSystem for scheme: hdfs

        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)

        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)

        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)

        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)

        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

        at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:75)

        at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:362)

        at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:780)

Kindly guide how to resolve this issue,

Thanks,

Saurabh

[SAURABH VERMA]

Druid version : 0.6.171 

[SAURABH VERMA]

This is the runtime.properties for middlemanager:--

druid.host=phxscal1111

druid.port=8080

druid.service=middleManager

druid.extensions.remoteRepositories=[]

druid.extensions.localRepository=/import/home/mon/druidRepository

druid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight:0.6.171","io.druid.extensions:druid-hdfs-storage:0.6.171"]

druid.zk.service.host=phxscal1041:2181

druid.selectors.indexing.serviceName=overlord

# Dedicate more resources to peons

druid.indexer.runner.javaCommand=/home/mon/jdk1.6/bin/java

druid.indexer.runner.javaOpts=-server -Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:MaxDirectMemorySize=7g -Duser.timezone=MST -Dfile.encoding=UTF-8 -cp /import/home/mon/hadoop2.4.0

druid.indexer.task.baseTaskDir=/home/mon/druid/persistent/task

druid.indexer.task.chathandler.type=announce

druid.indexer.runner.startPort=9080

#druid.indexer.fork.property.druid.storage.type=local

#druid.indexer.fork.property.druid.storage.storageDirectory=/import/home/mon/druid/localstorage

druid.indexer.fork.property.druid.storage.type=hdfs

druid.indexer.fork.property.druid.storage.storageDirectory=hdfs://phxscal1077.phx.XYZ.com:54310/

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

druid.indexer.fork.property.druid.processing.numThreads=6

druid.indexer.fork.property.druid.request.logging.type=file

druid.indexer.fork.property.druid.request.logging.dir=request_logs/

druid.indexer.fork.property.druid.segmentCache.locations=[{"path": "/home/mon/druid/persistent/zk_druid", "maxSize": 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.computation.buffer.size=268435456

druid.worker.capacity=4

#mysql

druid.db.connector.connectURI=jdbc:mysql://mymisc1.db.stratus.XYZ.com:3306/test

druid.db.connector.user=jobs_app

druid.db.connector.password=jobs_app

[SAURABH VERMA]

Updated runprops for middlemanager :-

druid.host=phxscal1111

druid.port=8080

druid.service=middleManager

druid.extensions.remoteRepositories=[]

#druid.extensions.localRepository=/import/home/mon/druidRepository

druid.extensions.localRepository=/import/home/mon/druid171PullReps

druid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight:0.6.171","io.druid.extensions:druid-hdfs-storage:0.6.171"]

druid.zk.service.host=phxscal1041:2181

druid.selectors.indexing.serviceName=overlord

# Dedicate more resources to peons

druid.indexer.runner.javaCommand=/home/mon/jdk1.6/bin/java

druid.indexer.runner.javaOpts=-server -Xmx4g -Xms4g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:MaxDirectMemorySize=7g -Duser.timezone=MST -Dfile.encoding=UTF-8

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.4.0"]

druid.indexer.task.baseTaskDir=/home/mon/druid/persistent/task

druid.indexer.task.chathandler.type=announce

druid.indexer.runner.startPort=9080

#druid.indexer.fork.property.druid.storage.type=local

#druid.indexer.fork.property.druid.storage.storageDirectory=/import/home/mon/druid/localstorage

druid.indexer.fork.property.druid.storage.type=hdfs

druid.indexer.fork.property.druid.storage.storageDirectory=hdfs://phxscal1077.phx.xyz.com:54310/

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

druid.indexer.fork.property.druid.processing.numThreads=6

druid.indexer.fork.property.druid.request.logging.type=file

druid.indexer.fork.property.druid.request.logging.dir=request_logs/

druid.indexer.fork.property.druid.segmentCache.locations=[{"path": "/home/mon/druid/persistent/zk_druid", "maxSize": 0}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.indexer.fork.property.druid.computation.buffer.size=268435456

druid.worker.capacity=4

#mysql

druid.db.connector.connectURI=jdbc:mysql://mymisc1.db.stratus.xyz.com:3306/test

druid.db.connector.user=caljobs_app

druid.db.connector.password=caljobs_app

[SAURABH VERMA]

middlemanager command line :-

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/../config/middlemanager/mm_0

#For the kit

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/lib/*

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/config/middlemanager/mm_0:/import/home/appmon/hadoop2.4.0/

echo "Running command:"

(set -x; /home/appmon/jdk1.6/bin/java ${JAVA_ARGS} -classpath ${DRUID_CP}  io.druid.cli.Main server middleManager)

[SAURABH VERMA]

Thanks a lot FJ and Gian for solving the issue over IRC.

The resolution was adding the hadoop classpath jars and config xmls to  the classpath in the middlemanager.

so, after all changes my middlemanager command line looks like :-

JAVA_ARGS="-Xmx8g -Duser.timezone=MST -Dfile.encoding=UTF-8"

"DRUID_CP=${EXAMPLE_LOC}

#For a pull

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/../config/middlemanager/mm_0

#For the kit

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/lib/*

DRUID_CP=${DRUID_CP}:${SCRIPT_DIR}/config/middlemanager/mm_0:/import/home/appmon/hadoop2.4.0/jars/*:/import/home/appmon/hadoop2.4.0/conf"

pay attention to /import/home/appmon/hadoop2.4.0/jars/* and /import/home/appmon/hadoop2.4.0/conf

the first path is for all the *.jar files and the second one is for *.xml

One way to verify whether your jars are getting picked up by middlemgrs and peons is to see the command line logs, if you see hadoop jar files (for ex. hadoop-common-2.4.0.jar, hadoop-yarn-applications-unmanaged-am-launcher-2.4.0.jar, etc.) then that shows that your changes to the classpath are indeed picked properly.

Exact way I did :-

1. run 'hadoop classpath' from hadoop-x.y.z/bin folder

2. you will see a list of folders some ending with * and some ending without *

3. * ones are the folders from where you have to pick up the jars

4. without * ones are the folders from where *.xml and non-jar content will come

5. copied the jars and xmls in separate shared folders

6. put your jars shared path and xmls shared path in middlemanager classpath

7. verify command line console as mentioned above

If step 7 is verified fine, then you would not see the error saying "java.io.IOException: No FileSystem for scheme: hdfs" for sure :-)