Connection to IBM COS

[Jose]

Hi, I am trying to read a parquet file as delta from the IBM COS, and I am having problems.

Code:

spark = pyspark.sql.SparkSession.builder.appName("MyApp").master('local[*]')\

    .config("spark.delta.logStore.class", "org.apache.spark.sql.delta.storage.COSLogStore")\

    .config("spark.jars.packages", "io.delta:delta-core_2.12:0.8.0") \

    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \

    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \

    .getOrCreate()

hconf = spark._jsc.hadoopConfiguration()

hconf.set('fs.stocator.scheme.list', 'cos')

hconf.set('fs.cos.impl', 'com.ibm.stocator.fs.ObjectStoreFileSystem')

hconf.set('fs.stocator.cos.impl', 'com.ibm.stocator.fs.cos.COSAPIClient')

hconf.set('fs.stocator.cos.scheme', 'cos')

hconf.set('fs.cos.atomic.write', 'true')

hconf.set("fs.cos.service.endpoint", COS_ENDPOINT)

hconf.set("fs.cos.service.iam.api.key",COS_API_KEY_ID)

from delta.tables import *

df_COS = spark.read \

.option("header", "true") \

.option("inferSchema", "true") \

.parquet("cos://bucket.service/")

df_COS.write.format("delta").save("cos://bucket.service/")

Error:

Py4JJavaError: An error occurred while calling o59.save. : java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: org.apache.spark.sql.delta.storage.COSLogStore

[Tathagata Das]

The LogStore for IBM COS is not yet released. But the good news is that it will be released within a week as part of the Delta 1.0 release.

Here is a sneak-peek of the documentation of the 1.0 release which explains how the configuration will work - https://docs.delta.io/1.0.0/delta-storage.html#ibm-cloud-object-storage

TD

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/49c60e62-b2e7-4d69-80b4-6dda9fdc5679n%40googlegroups.com.