Unable to upgrade to 0.9 with HDFS sink

27 views
Skip to first unread message

sridh...@gmail.com

unread,
Oct 9, 2018, 3:13:17 PM10/9/18
to divolte-collector
Hi,

I just downloaded 0.9 and tried starting the server with same config that I used for 0.8 and encountering bellow error regarding HDFS sink.
Probably because of the update in hadoop dependencies. Can you please suggest how can I get this to work?

19:04:34.753 [main] DEBUG org.apache.hadoop.fs.FileSystem - looking for configuration option fs.hdfs.impl
19:04:34.753 [main] DEBUG org.apache.hadoop.fs.FileSystem - Looking in service filesystems for implementation class
19:04:34.755 [main] ERROR i.d.s.filesinks.hdfs.HdfsFileManager - Could not initialize HDFS filesystem or failed to check for existence of publish and / or working directories..
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at io.divolte.server.filesinks.hdfs.HdfsFileManager$HdfsFileManagerFactory.getFileSystemInstance(HdfsFileManager.java:219)
        at io.divolte.server.filesinks.hdfs.HdfsFileManager$HdfsFileManagerFactory.verifyFileSystemConfiguration(HdfsFileManager.java:154)
        at io.divolte.server.config.HdfsSinkConfiguration.lambda$getFactory$1(HdfsSinkConfiguration.java:59)
        at io.divolte.server.Server.lambda$new$7(Server.java:97)
        at com.google.common.collect.CollectCollectors.lambda$toImmutableMap$1(CollectCollectors.java:61)
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at io.divolte.server.Server.<init>(Server.java:94)
        at io.divolte.server.Server.<init>(Server.java:61)
        at io.divolte.server.Server.main(Server.java:225)
Exception in thread "main" java.io.UncheckedIOException: Could not initialize HDFS filesystem.
        at io.divolte.server.filesinks.hdfs.HdfsFileManager$HdfsFileManagerFactory.verifyFileSystemConfiguration(HdfsFileManager.java:182)
        at io.divolte.server.config.HdfsSinkConfiguration.lambda$getFactory$1(HdfsSinkConfiguration.java:59)
        at io.divolte.server.Server.lambda$new$7(Server.java:97)
        at com.google.common.collect.CollectCollectors.lambda$toImmutableMap$1(CollectCollectors.java:61)
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at io.divolte.server.Server.<init>(Server.java:94)
        at io.divolte.server.Server.<init>(Server.java:61)
        at io.divolte.server.Server.main(Server.java:225)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at io.divolte.server.filesinks.hdfs.HdfsFileManager$HdfsFileManagerFactory.getFileSystemInstance(HdfsFileManager.java:219)
        at io.divolte.server.filesinks.hdfs.HdfsFileManager$HdfsFileManagerFactory.verifyFileSystemConfiguration(HdfsFileManager.java:154)

Thanks,
Sridhar

andre...@godatadriven.com

unread,
Oct 10, 2018, 3:34:24 AM10/10/18
to divolte-collector
Hi Sridhar,

It's not obvious what might be going on here. The version of Hadoop that we build against changed between 0.8 and 0.9; previously we used Hadoop 2.9 and now we're on 3.1. It's possible this could be causing some issues. Can you tell us more about your environment?

Cheers,

 - Andrew

sridh...@gmail.com

unread,
Oct 10, 2018, 4:06:56 PM10/10/18
to divolte-collector
Hi Andrew,

Thanks for your reply.

Most likely hadoop version is causing the issue
We are running cloudera hadoop version - Hadoop 2.6.0-cdh5.14.4

A quick search on google pointed me to couple different possible fixes, but not sure if there is a config driven way to fix this in divolte without having to rebuild.

We did not have this problem with 0.8. Do you know how can we make this backward compatible?

Thanks again,

Sridhar

andre...@godatadriven.com

unread,
Nov 26, 2018, 2:26:01 PM11/26/18
to divolte-collector
Hi Sridhar,

I don't have an easy way to test this at the moment, sorry.

The simplest path would be to try changing the Hadoop version to something suitable in build.gradle and build a custom version.

Hope this helps,

 - Andrew
Reply all
Reply to author
Forward
0 new messages