Spark issues with Hive 4

federico jordán

unread,

Apr 29, 2025, 11:22:11 AMApr 29

to MR3

Greetings,
I have successfully migrated from Hive 3 on MR3 to Hive 4 on MR3. However, when trying to work with Spark 3.4.x or 3.5.x, we are encountering issues with the Hive Metastore, and from what I’ve seen, these Spark versions have compatibility problems with Hive 4.
We still have Hive 3 on MR3 in production, as we need our processes to continue running. Will the image mr3project/hive3:1.11 remain available? Is there any way to work with Spark on Hive 4?

Thank you

Sungwoo Park

unread,

Apr 29, 2025, 11:40:54 AMApr 29

to MR3

Hello --- first, thanks a lot for trying Hive 4 on MR3.

Metastore of Hive 4 is not compatible with Spark 3.x. That's why we had to run Spark 4.0.0-RC2 when benchmarking Hive 4 on MR3. (I confirm that Spark 4.0.0-RC2 has been updated to work with Metastore of Hive 4, although I don't know how thoroughly it has been tested by the Spark community.) So, I think you want to either upgrade Spark 3.x to Spark 4.0.0, or continue to use Metastore of Hive 3. (Spark 4.0.0 is not officially released yet, but you can download release candidates.)

mr3project/hive3:1.11 will remain available until we release Hive 3 on MR3 2.0 later.

Cheers,

--- Sungwoo

federico jordán

unread,

Apr 30, 2025, 11:01:27 AMApr 30

to MR3

Thank you for your help, I will try it.

El dia dimarts, 29 d’abril del 2025 a les 17:40:54 UTC+2, Sungwoo Park va escriure:

Sungwoo Park

unread,

Apr 30, 2025, 1:37:45 PMApr 30

to MR3

For the record, the incompatibility between Spark 3.x and Hive 4 stems from this commit in Hive:

HIVE-26537: Deprecate older APIs in the HMS thrift interface

HIVE-26537 is not included in Apache Hive 4.0.0, but it is included in Hive 4 on MR3. That's why Spark 3.x cannot use Metastore of Hive 4 on MR3.

Source: https://github.com/apache/iceberg-python/issues/1222

--- Sungwoo

federico jordán

unread,

May 6, 2025, 4:11:32 AMMay 6

to MR3

Hi,

I've noticed that the Hive3 branch is no longer being actively maintained. However, we still have Hive3MR3 running in our production environment.
Previously, we encountered an issue when restarting the mr3master pod where it would fail with the error message "License has expired", causing the mr3master to stop functioning completely. Could you confirm if we might experience the same problem with the latest mr3project/hive3:1.11 image?
Currently, we have critical production services that cannot be migrated to Hive4 at this time.

Best regards,

El dia dimecres, 30 d’abril del 2025 a les 19:37:45 UTC+2, Sungwoo Park va escriure:

Reply all

Reply to author

Forward