java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found

[Syed Akram]

Hi,

I am using a custom connector to read orc file using hive orc reader, 

so when i try to read a file from hdfs

it throws below exception

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)

        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2415)

        at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:69)

        at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:61)

        at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:43)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)

        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)

        at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:53)

        at com.facebook.presto.example.ORCBatchReader.readBatch(ORCBatchReader.java:56)

        at com.facebook.presto.example.ExampleRecordSet.<init>(ExampleRecordSet.java:54)

        at com.facebook.presto.example.ExampleRecordSetProvider.getRecordSet(ExampleRecordSetProvider.java:54)

        at com.facebook.presto.split.RecordPageSourceProvider.createPageSource(RecordPageSourceProvider.java:41)

        at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:48)

        at com.facebook.presto.operator.TableScanOperator.createSourceIfNecessary(TableScanOperator.java:258)

        at com.facebook.presto.operator.TableScanOperator.isFinished(TableScanOperator.java:206)

        at com.facebook.presto.operator.Driver.processInternal(Driver.java:377)

        at com.facebook.presto.operator.Driver.processFor(Driver.java:303)

        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:587)

        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:505)

        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:640)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:744)

is this any jar version problem? or any other thing?

Thanks

Akram Syed

[Syed Akram]

Any one have any suggestions?
what may be the issue?

how to fix this.

[Dain Sundstrom]

Based on this part of the stack trace:

> at com.facebook.presto.example.ExampleRecordSetProvider.getRecordSet(ExampleRecordSetProvider.java:54)

It appears that you have modified the example http connector to support reading ORC files from HDFS. To get HDFS to work properly is a lot of work because it uses dynamic class loading to resolve things like the file system, and it has global statics. We have resolved all of these problem in the Hive connector, but it was a lot of work.

To fix class loader issue, you might be able to use the ClassLoaderSafe* wrappers in the SPI. See HiveConnectorFactory for an example on how to use them.

Additionally it looks like you are using an ORC reader under a RecordSet interface. This is much slower than using the PageSource interface, as the Presto ORC reader natively return Presto blocks (which become pages). See OrcPageSource for how to implement this.

-dain

> --
> You received this message because you are subscribed to the Google Groups "Presto" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Syed Akram]

Thanks Dain,

Could you please elaborate on the use of ClassLoaderSafe* wrappers.

Thanks

[Syed Akram]

If i'm not wrong, i have to use  ClassLoaderSafe* wrapper over here, i.e. in ExampleCOnnectorFactory.create() .

please suggest me, how to proceed, 

By the way i tried to use ClassLoaderSafe* as in HiveConnectorFactory, but no use,

public Connector create(final String connectorId, Map<String, String> requiredConfig)

    {

        requireNonNull(requiredConfig, "requiredConfig is null");

        requireNonNull(optionalConfig, "optionalConfig is null");

        try {

            // A plugin is not required to use Guice; it is just very convenient

            Bootstrap app = new Bootstrap(

                    new JsonModule(),

                    new ExampleModule(connectorId, typeManager));

        Injector injector = app

                    .strictConfig()

                    .doNotInitializeLogging()

                    .setRequiredConfigurationProperties(requiredConfig)

                    .setOptionalConfigurationProperties(optionalConfig)

                    .initialize();

       

            return injector.getInstance(ExampleConnector.class);

        }

Thanks

Akram Syed