Delta Tables crashes with error: org.apache.spark.sql.AnalysisException: `database_name`.`table_name` is not a Delta table.

[Allie Ubisse]

Hi Amazing Developers,

I came across the is not a Delta table error when I try to read my delta table in databricks.

I just wish to understand the problem with respect to delta'.

Error:

An error occurred while fetching the table:  table_name

com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: dbfs:/user/hive/warehouse/database_name .db/ table_name  doesn't exist;;

com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: `database_name`.`table_name` is not a Delta table.

scenario:

The tables received updates hourly, and specific pipelines take input as the table to perform some aggregation. The pipeline will run for 2 to 3 months without breaking then boom the above error. Once the Error hits, almost all our delta tables crash, even the ones that were not used for a long time.

This hits only our development Environment and Production still runs healthy.

Delta settings:

We are using the default settings in all the environments.

please let me know if more info is required.

Thanks for your help.

[Denny Lee]

This may be specific to the Databricks environment so you may want to ping Databricks support and/or ping the deltalake-databricks channel on the Delta Lake Users Slack (https://dbricks.co/delta-users-slack).

Saying this, out of curiosity by any chance are your development and production environments using different versions of the Databricks Runtime (DBR)?  More recent DBR versions default to Delta which may somehow be coming into play here.

Sent via Superhuman

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/91eaa7b7-6995-4288-8c62-ec6ed7e87af1n%40googlegroups.com.

[Allie Ubisse]

Hey Denny,

We are using the same DBR versions across all the environments. I will try to ping the  delta lake-databricks channel on the Delta Lake Users Slack.

To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.

[Aman Sehgal]

I faced the same issue. In the databricks notebook it all works fine, but reDash was showing this error.

@alliesilverubisse

You mentioned that "This hits only our development Environment and Production still runs healthy"

It looks like when you shut down your clusters for dev environment, it creates an unknown bug with hive meta store.

Not sure what's the reason.

I was able to fix the issue by:

1. Unmount and re-mount the bucket

2. Drop the table

2. Re-register the delta table

3. If you're using reDash, then restart your endpoint

Hope this helps