hdfs-storage - class not found: org.apache.hadoop.hdfs.DistributedFileSystem

[Roger Hoover]

Hi,

I'm trying to setup the hdfs-storage module on Druid 0.6.171.

During initialization of a real-time node, I see the hdfs jar being loaded as a dependency of the hdfs-storage extension.

2015-02-13 00:48:06 INFO  io.druid.initialization.Initialization:70 - Added URL[file:/foo/test/repo/org/apache/hadoop/hadoop-hdfs/2.3.0/hadoop-hdfs-2.3.0.jar]

But later it cannot load one of the classes inside.  Any ideas why?

I see the class in the jar:

jar tf /foo/test/repo/org/apache/hadoop/hadoop-hdfs/2.3.0/hadoop-hdfs-2.3.0.jar | grep DistributedFileSystem.class

org/apache/hadoop/hdfs/DistributedFileSystem.class

The exception is below:

2015-02-13 00:43:46 ERROR io.druid.segment.realtime.plumber.RealtimePlumber:98 - Failed to persist merged index[web]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.lang.RuntimeException, exceptionMessage=java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found, interval=2015-02-12T23:03:00.000Z/2015-02-12T23:04:00.000Z}

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1882)

at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2298)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:353)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

at io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:75)

at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:362)

at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:42)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1788)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1880)

... 15 more

I appreciate any suggestions.  Thanks!

Roger

[Gian Merlino]

Hi Roger,

I think you are probably running into this bug: https://github.com/druid-io/druid/issues/713. Can you try the workaround of including hadoop jars on the Druid historical node's classpath?

[Roger Hoover]

Gian,

Thank you for helping.  Adding hadoop jars to the classpath got it working.  I still don't understand why though.  The jars of interest seemed to be getting loaded but the classes don't resolve.

Anyway, thanks for the help.

Roger

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/613b3ddf-7d47-490d-be75-318fd4cd5d83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Olaf Krische]

hello,

can you try deleting the jar from the local maven repository? maybe you have a corrupt jar.

[Roger Hoover]

Olaf,

Thanks for the response.  Adding the hadoop jars to the classpath made it work.  Thanks,

Roger

On Sun, Feb 15, 2015 at 4:39 AM, Olaf Krische <okri...@gmail.com> wrote:

hello,

can you try deleting the jar from the local maven repository? maybe you have a corrupt jar.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/2e575d0e-e595-46b0-baf0-afd9f3d328a1%40googlegroups.com.