Upgrade
98 views
Skip to first unread message
Amy
Feb 19, 2021, 2:14:17 PM2/19/21
to Delta Lake Users and Developers
Hello, I am new to delta lake.
I created a delta table using Databricks with an older DBR runtime which has delta lake 0.5.0
Trying to upgrade to a newer DBR runtime with delta lake 0.7.0
Is there any specific process to upgrade delta lake ? I am currently getting an error trying to read an older delta version. Didn't have that problem before. Any insight is appreciated
-------------------------
df = spark.read.format("delta").load(dest_file_location)
AnalysisException: Multiple sources found for parquet (org.apache.spark.sql.execution.datasources.v2.parquet.ParquetDataSourceV2, org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat), please specify the fully qualified class name.;
Burak Yavuz
Feb 19, 2021, 3:00:52 PM2/19/21
to Amy, Delta Lake Users and Developers
Can you please check if you're accidently bundling Spark in your libraries that you upload to Databricks? You should be ignoring any Spark dependencies, when building fat jars.
--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/261de8fb-e3a8-486b-bb14-5ff74e05cf77n%40googlegroups.com.
Amy
Feb 19, 2021, 3:50:46 PM2/19/21
to Delta Lake Users and Developers
That works. Thanks! Also, do you recommend making a copy of the existing delta tables (DBR 6.4 delta 0.5.0) to a new folder location and then start writing with DBR 7.6 ( delta 0.7.0) ? or leaving things as is will work fine ? I am concern if there will be binary differences or compatibility issues. Never done thing before.
Burak Yavuz
Feb 19, 2021, 3:51:51 PM2/19/21
to Amy, Delta Lake Users and Developers
There should not be any compatibility issues, therefore you don't need to make any backups.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/b0fc6922-7ddd-461a-972c-bd81ae834e28n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages