Do I have to deploy a hadoop instance on the server on which I start the overlord node with hdfs

[Zuhai Jiang]

Do I have to deploy a hadoop instance on the server on which I start the overlord node with hdfs as deep storage?

I have started a druid cluster using a remote hadoop cluster's hdfs as deep storage. When I sent an index task of a batch file to the overlord node, it failed and gave the following exceptions:

2015-01-03 11:54:07,110 WARN [task-runner-0] io.druid.indexing.common.index.YeOldePlumberSchool - Failed to merge and upload
java.io.IOException: No FileSystem for scheme: hdfs
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
	at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:75)
	at io.druid.indexing.common.task.IndexTask$2.push(IndexTask.java:390)
	at io.druid.indexing.common.index.YeOldePlumberSchool$1.finishJob(YeOldePlumberSchool.java:179)
	at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:444)
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:198)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:218)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2015-01-03 11:54:07,114 INFO [task-runner-0] io.druid.indexing.common.index.YeOldePlumberSchool - Deleting Index File[/tmp/persistent/task/index_trafficbase_2015-01-03T11:53:36.676Z/work/trafficbase_2014-12-22T00:00:00.000Z_2014-12-23T00:00:00.000Z_2015-01-03T11:53:36.686Z_0/trafficbase_2014-12-22T00:00:00.000Z_2014-12-23T00:00:00.000Z_2015-01-03T11:53:36.686Z/spill0]
2015-01-03 11:54:07,115 INFO [task-runner-0] io.druid.indexing.common.task.IndexTask - Task[index_trafficbase_2015-01-03T11:53:36.676Z] interval[2014-12-22T00:00:00.000Z/2014-12-23T00:00:00.000Z] partition[0] took in 99,999 rows (99,999 processed, 0 unparseable, 0 thrown away) and output 99,405 rows
2015-01-03 11:54:07,117 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_trafficbase_2015-01-03T11:53:36.676Z, type=index, dataSource=trafficbase}]
java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at io.druid.indexing.common.index.YeOldePlumberSchool$1.finishJob(YeOldePlumberSchool.java:189)
	at io.druid.indexing.common.task.IndexTask.generateSegment(IndexTask.java:444)
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:198)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:218)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
	at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:75)
	at io.druid.indexing.common.task.IndexTask$2.push(IndexTask.java:390)
	at io.druid.indexing.common.index.YeOldePlumberSchool$1.finishJob(YeOldePlumberSchool.java:179)
	... 8 more

I searched for solution on the groups and noticed that when I started the node using hdfs, I should include hadoop conguration files in the classpath. The problem is that since I'm using a remote hadoop cluster, I don't have hadoop configuration files on the local servers, they are on the severs holding the hadoop cluster.So I wonder if I can use a remote hdfs as deep storage or I have to deploy a local hadoop instance and use this hdfs? 

[Gian Merlino]

You don't have to actually deploy hadoop on the druid machines, but you do need to include the hadoop configuration files on druid's classpath. You should be able to copy them over from your hadoop cluster.

[Zuhai Jiang]

I copied all the hadoop config files to the server on which I deployed overlord node and started the node with all the config files included in the classpath. However I still get the same exceptions in the task log after I sent the index task.

The remote hadoop's version is 2.4.1, is this version supported by druid? Or should I do something to adapt it?

在 2015年1月4日星期日UTC+8上午2时09分51秒，Gian Merlino写道：

[Zuhai Jiang]

The version of druid I deployed is 0.6.160, and here's my overlord node configuration:

-server

-Xmx256m

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

druid.host={host_ip}:8084

druid.port=8084

druid.service=overlord

druid.zk.service.host={zk_ip}

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs://{hdfs_ip}:{hdfs_port}/user/hadoop/druid/deep

druid.extensions.coordinates=["io.druid.extensions:druid-hdfs-storage:0.6.160"]

druid.db.connector.connectURI={mysql_connectURI}

druid.db.connector.user={mysql_username}

druid.db.connector.password={mysql_password}

druid.selectors.indexing.serviceName=overlord

druid.indexer.queue.startDelay=PT0M

# druid.indexer.runner.type=remote

# druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.javaOpts=-server -Xmx1g -Xms1g -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxDirectMemorySize=536870912

druid.indexer.runner.startPort=8088

druid.indexer.fork.property.druid.processing.numThreads=1

druid.indexer.fork.property.druid.computation.buffer.size=100000000

在 2015年1月4日星期日UTC+8上午2时09分51秒，Gian Merlino写道：

[Fangjin Yang]

2.4.1 is supported. Can you show us the command you use to start up the overlord?

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/a721350f-b8f7-487a-a7b1-27d4f899f7d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Zuhai Jiang]

the command line to start up the overlord:

java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:/home/betauser/hadoop/config:config/overlord io.druid.cli.Main server overlord

where /home/betauser/hadoop/config is the directory I place the hadoop config files

在 2015年1月4日星期日UTC+8上午10时47分20秒，Fangjin Yang写道：

[Fangjin Yang]

Can you also try adding:

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.4.1"] to the overlord runtime.properties so Druid will try to use that version by default. 

Also, did you include the hadoop 2.4 jar on the cp of the overlord?

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e172b0b1-d3ef-48a0-a8fe-9a03f7269f45%40googlegroups.com.

[Zuhai Jiang]

Thanks for advising and being patient to me. 

I just tried to add druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.4.1"] to the overlord runtime.properties and started up the overlord again. But I still find only hadoop 2.3.0 jars in my local maven repository and get the same error after I sent index task.

Do I need to copy hadoop 2.4.1 jars from hadoop cluster to druid machines and include them in the classpath?

在 2015年1月4日星期日UTC+8上午11时26分04秒，Fangjin Yang写道：

[Zuhai Jiang]

I copied the hadoop jars to the druid machines, included them in the classpaths and it finally found the hdfs.

Thanks for patiently answering my questions very much! 

Though I'm sure there are more problems waiting for me...

在 2015年1月4日星期日UTC+8上午11时58分24秒，Zuhai Jiang写道：

[Fangjin Yang]

Hi Zuhai, great to hear! They say the hardest challenge with big data technologies is to get different versions of Hadoop to work with them :)

Thanks for the update and please let us know if you have more problems.