Clickhouse upgrade facing multiple issues

[Rajesh Roy]

Hello,

We are using two shard and two replica. Clickhouse docker image is  yandex/clickhouse-server:19.16.14.65

But while upgrading the version  to yandex/clickhouse-server:21.9.2.17 we get errors :

• Code: 473, DB::Exception: WRITE locking attempt on "system.asynchronous_metric_log" has timed out! (120000ms) Possible deadlock avoided. (in query: ALTER TABLE system.asynchronous_metric_log MODIFY TTL event_date + INTERVAL 3 MONTH)

• Code: 517,  e.displayText() = DB::Exception: Metadata on replica is not up to date with common metadata in Zookeeper. Cannot alter

These are the errors while executing our SQL templates. It results clickhouse server failed.

Can anyone help me to find what is the reason of these errors and Is there any way to solve these errors ?

After ignoring (ignored-errors: [517, 473]) these error from templates, clickhouse server started, but it is producing continuous warning logs like:

 <Warning> db.table_name (ReplicatedMergeTreePartCheckThread): Checking part 20220401_324452_324452_0
 <Warning> db.table_name (ReplicatedMergeTreePartCheckThread): Checking if anyone has a part 20220401_324452_324452_0 or covering part.
 <Warning> db.table_name (ReplicatedMergeTreePartCheckThread): Found the missing part 20220401_324452_324452_0 at 20220401_324452_324452_0 on s1-r2

How can I resolve this huge never ending warning logs?