Error when indexing using Hortonworks YARN cluster

[Richard Grossman]

Hi

When I try to launch an index task I get

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
	...
	at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:116) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
	at io.druid.indexer.JobHelper.runJobs(JobHelper.java:323) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
	at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
	at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_101]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_101]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
	at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]

The hadoop version is 2.7.1

I've tried all the combination regarding hadoop config file and I don't find a way to includes this in the classpath
I join the mapred-site.xml and core-site.xml

Could you help me to find out how to configure the index task

Thanks

[Gian Merlino]

Hey Richard,

Did you build Druid against 2.7.1 or did you adjust the hadoop-dependencies directory? If so you need to include hadoop-aws too; that was split out into its own package. It contains the NativeS3FileSystem.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/7b5920a0-e840-45b5-9df2-51b37398988e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Grossman]

Thanks it works !!

just to help other people that may want to run druid with hortonworks you need to add to your core-site.xml

<property>
      <name>mapreduce.job.classloader</name>
      <value>true</value>
    </property>

to prevent a package conflict