Slow processing of the WAL

[Laurence Syree]

Hi All,

We updated our Zookeeper cluster earlier this week; we stopped Clickhouse for the duration of the upgrade; however, after starting everything back up, the WAL started to grow, along with the number of parts we are tracking.

We left the cluster for some time to recover from this, but it now appears to be processing the WAL slower than the incoming data flow. After stopping inbound data to the cluster today, we see an incredibly slow WAL recovery.

Our clickhouse servers have 24 cores and 128 GB of ram each. Resource utilisation on all six nodes is very low, but WAL processing is progressing painfully slowly.

We have looked through the optimisation options in the clickhouse documentation and haven't found anything that looks relevant to this issue.

Any advice on the topic would be greatly appreciated.

Thanks,

Laurence Syree

[Vladimir]

Hi Laurence,

Could you please provide any extra info, did you update only ZK or ClickHouse server as well, and which versions of ZK and ClickHouse do you use? Also, are there any Warnings or Errors in logs? The values from `system.settings` and `system.merge_tree_settings` may also be helpful. Also you may check if anything was changed in `system.metric_log` or `asynchronous_metric_log` after upgrade.