ZooKeeper Out of memory due to ClickHouse

[CBS Archi]

Hello Everyone

We are using Clickhouse version: 20.12.4 revision 54442

Number of replicas: 2

Number of shard on each replica: 2

We are getting a issue where zookeeper is going out of memory beyond 3 GiB.

The Number of children in one of replica : 2964027

It seems due to large number of children it is going OOM.

Looking help to prevent this issue.

Thanks

CBS

[CBS]

Hello Guys

Is there any clue if the below exception can cause a high number of zNode in zookeeper?

2022.02.23 12:04:30.000798 [ 132 ] {} <Error> DB_0.metrics_v2 (ReplicatedMergeTreeCleanupThread): void DB::ReplicatedMergeTreeCleanupThread::run(): Code: 999, e.displayText() = Coordination::Exception: Transaction failed (No node): Op #0, path: /clickhouse/tables/shard_0/metrics_v2/blocks/202202_7133862859622441240_16962971050174568009, Stack trace (when copying this message, always include the lines below): 

--
You received this message because you are subscribed to the Google Groups "ClickHouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/02e892cf-9c80-4949-a7de-9690c362739an%40googlegroups.com.

[CBS Archi]

Hello everyone,

Do anyone have a clue regarding these?

Thanks