Help with using delta and hadoop 2.9.1

[Vishal Venkatraman]

Hi,

I need some help with using delta with hadoop 2.9.1. 

I am having to write the dataframe on Azure storage container in a delta format. My on-prem cluster runs on hadoop version 2.9.1 

I am using the following dependencies but it keeps throwing the following error

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-azure</artifactId>
<version>2.9.1</version>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_2.11</artifactId>
<version>0.1.0</version>
</dependency>

Error : 

org.apache.hadoop.fs.UnsupportedFileSystemException: fs.AbstractFileSystem.wasbs.impl=null: No AbstractFileSystem configured for scheme: wasbs

Can someone please help me figure out the right delta core version to use for hadoop 2.9.1?

[Shixiong(Ryan) Zhu]

Looks you are missing a config: spark.delta.logStore.class=org.apache.spark.sql.delta.storage.AzureLogStore

You can take a look at https://docs.delta.io/latest/delta-storage.html#microsoft-azure-storage for more information.

Best Regards,

Ryan

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/7796bed0-7f74-46dc-806f-9ab68d8de45fn%40googlegroups.com.

[Vishal Venkatraman]

Ryan,

Thanks for responding. I am already using it. doing something like 

spark.conf.set("spark.delta.logStore.class","org.apache.spark.sql.delta.storage.AzureLogStore")

Still getting the error.Can you please help ?

[Shixiong(Ryan) Zhu]

"spark.delta.logStore.class" is not a SQL conf. It needs to be set before starting SparkContext/SparkSession. For example, you can set it in your spark-shell/spark-submit command using "--conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.AzureLogStore".

Best Regards,

Ryan

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/e6607f9d-93ec-43ad-9c91-f2bf90cb2c00n%40googlegroups.com.

[Vishal Venkatraman]

Ryan,

After i made this change and added it during spark session creation it works perfectly. 

I cant thank you enough for helping with this.

Vishal.