hivemr3-metastore & hivemr3-hiveserver2 is crashlooping and mr3master pod not turning up

[Subhasmita Swain]

Hello,

We are attempting to set up a clustered hive metastore for usage with Trino, managed MySQL as a backend, and S3 persist. We have faithfully followed the documentation of MR3, but we still cannot pinpoint the precise cause of the pods' failure to activate. We've made an effort to debug, but we may be lacking some professional guidance.

For your reference, I've included the log files from both pods. Please let me know if there are any further requirements.

I wanted to highlight a few specific log lines. 

2023-04-04T14:17:28,384 INFO [main] conf.MetastoreConf: Unable to find config file: hivemetastore-site.xml 2023-04-04T14:17:28,384 INFO [main] conf.MetastoreConf: Unable to find config file: metastore-site.xml
 

2023-04-04T14:22:49,484 WARN [main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2023-04-04T14:22:49,492 INFO [main] metastore.ObjectStore: RawStore: org.apache.hadoop.hive.metastore.ObjectStore@5ef5c734, with PersistenceManager: null will be shutdown

Thank You

Regards, 

Subhasmita

[Sungwoo Park]

Welcome to MR3 Group!

1.

Metastore downloads mysql-connector-java-8.0.28.tar.gz from the following address (because the Docker image should not include a MySQL connector). Please check its compatibility with your MySQL database.

https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.28.tar.gz

2.

Your Metastore log reports that an incompatible database for Metastore already exists in MySQL:

Error: Table 'CTLGS' already exists (state=42S01,code=1050)
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

This is probably the reason for the following error message:

2023-04-05T07:47:34,336 ERROR [main] metastore.RetryingHMSHandler: MetaException(message:Version information not found in metastore.)

The database for Metastore is specified in HIVE_DATABASE_NAME in kubernetes/env.sh.

HIVE_DATABASE_NAME=hive3mr3

If a database already exists, try a different database name. If HIVE_DATABASE_NAME points to a database created by another version of Hive, you might also see the above message.

If you have already created a database for Metastore, do not append "--init-schema" option to containers.args[] in kubernetes/yaml/metastore.yaml.

Cheers,

Sungwoo

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/a4cbd402-838a-4c0a-aeff-19b4f304cb8an%40googlegroups.com.

[Sungwoo Park]

If you should use a different MySQL connector, please check out this page and the link to Troubleshooting thereof:

https://mr3docs.datamonad.com/docs/k8s/guide/run-metastore/#connecting-metastore-to-a-database-server

Sungwoo