Accessing delta tables in Thriftserver

[Ed P]

I was able to use Spark/scala to create parquet files and make available to a SQL tool via thriftserver but not so lucky with the delta table.

I am storing my data on HDFS, created the delta table and I can see it with my SQL tool but when I try to get a list of columns or do a "select" statement, I get errors. Similar to when I use Beeline.

To properly create a Delta table stored on HDFS and access via the thriftserver, what steps are required?

[Jacek Laskowski]

Hi,

Delta tables are regular Spark artifacts so they should be available in STS. You may however need to set up --packages and the two --conf's for STS to have all infra set up properly. How did you create delta tables? Is this a vanilla STS configuration?

Pozdrawiam,

Jacek Laskowski

----

https://about.me/JacekLaskowski

"The Internals Of" Online Books

Follow me on https://twitter.com/jaceklaskowski

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/bcc2399f-7c0f-490a-accc-9d0650e8f27en%40googlegroups.com.

[Ed P]

Hi Jacek,

Thank you for the quick response and info. I looked further into the issue after thinking about your comments and I was able to get it to work. 

I am submitting a Scala job and the config lines that made this work were;

.config("spark.sql.warehouse.directory", s"${mapConfig("warehouseLocation")}")
.enableHiveSupport()
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
.getOrCreate()

Appreciate the support, very excited to get this project moving forward.

Best,

Ed