Refresh Delta Lake metadata in Databricks

[Jose Madrid]

Hi,

I created a couple unmanaged delta lake tables in databricks, all was working fine, until our databricks resource was deleted and we had to recreate the databricks resource. My delta lake tables still exist in data lake, but I was wondering is there any way that I can refresh the metadata in the new databricks resource, so I don't have to specify the location when querying these tables. Having the metadata in databricks makes querying these tables easier.

Thanks in advance.

[Tathagata Das]

If all the files of the delta lake tables are intact in the file system, then you can always read the tables directly using the path. 

- SQL: select * from delta.`path` (path has to be inside backticks)

- DataFrame: spark.read.format("delta").load("path")

All the table's metadata (schema, properties, etc.) are present in the table's transaction log inside tableDir/_delta_log/ .

In addition, if you want to add that path as a table in the metastore, then you can use the standard CREATE TABLE ... LOCATION 'path' to create an external table on that path. With that all SQL commands using the table name should just work. 

TD

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/ca7242e2-60b8-454f-a532-acaf2f413675n%40googlegroups.com.

[Jose Madrid]

Thanks Tathagata.

That's exactly what I meant to explain, how to add the path in the metastore.

Do you know when I try to create the table in the metastore, I get the following error;

"Failure to initialize configuration"

I can query the delta table using the path.