Hive 4.0.0 on MR3

[Sungwoo Park]

We are stabilizing Hive 4.0.0 on MR3. Here is a summary of preliminary experiments.

1. Loading 1TB TPC-DS ORC Hive tables (with 'transactional'=true), computing column statistics: success

2. Loading 1TB TPC-DS ORC Iceberg tables: success

Column statistics is automatically computed along with loading Iceberg tables.

3. Running 99 TPC-DS queries on cluster Blue

On Hive tables: total 1559 seconds, geo-mean 9.68 seconds

On Iceberg tables: total 1510 seconds, geo-mean 8.99 seconds

4. Running 99 TPC-DS queries on cluster Indigo

On Hive tables: total 2119 seconds, geo-mean 12.88 seconds

On Iceberg tables: total 1971 seconds, geo-mean 11.15 seconds

--- Sungwoo

[Sungwoo Park]

Here is an update.

1. All the integration tests included in Hive 4.0.0 now pass :-) Some work still remains, but we hope to release MR3 1.1 when Hive 4.0.1 is released. (Hive 4.0.1 fixes a few critical bugs in Hive 4.0.0.) While stabilizing Hive 4.0.0, we found a performance bug and submitted a fix (HIVE-28196).

2. If you would like to test or evaluate Hive 4.0.0 on MR3, let us know. From MR3 1.1, we are going to upload only a demo version of Hive-MR3.

3. After releasing MR3 1.1, we are not going to further maintain Hive 3.1.3 on MR3 and focus only on Hive 4 on MR3.

4. If you are using Apache Spark + Iceberg, try Hive + Iceberg (or Hive-MR3 + Iceberg).

--- Sungwoo

[Sungwoo Park]

We are quite close to releasing Hive 4 on MR3 1.11, along with the least release of Hive 3.1.3 on MR3. In addition to the many new features in Hive 4, LLAP IO cache is now working efficiently. For example, running the same query a few times usually gets the cache hit ratio close to 100% (if enough memory is allocated).

When MR3 1.11 is released, we will remove all previous releases and Docker images, so that we can focus on maintaining Hive 4 on MR3 and keeping the documentation (mr3docs) up-to-date with the latest release.

--- Sungwoo