Fetch part data consistency during rebalance

[Dongxiao Yang]

Hi I'm trying to figure out a property solution for huge size ClickHouse cluster rebalance. Our design is to have some scripts which will fetch/attach/detach parts from old shard to new one .  My question is , during ClickHouse fetch part process between old shard  to new one , is there any lock for that part which will prevent it be involved any merge or mutation process , so part on new node will have safe data consistency with old one ? 

Thanks. 

[Dima Novik]

Hi!

> is there any lock for that part which will prevent it be involved any merge or mutation process , so part on new node will have safe data consistency with old one ? 

There is no lock on the part level. You can stop all merges for the specific table using SYSTEM STOP MERGES query (don't forget to start them again after copy operation is done). But this approach won't work well if you ingest data into the table during copying, because you'll end up with TOO MANY PARTS.

You may also take a look at the clickhouse-copier.

[Dongxiao Yang]

So it's better to stop ingest before SYSTEM STOP MERGES , this is same requirement for clickhouse-copier if I understand right . 

I'm thing if there is a way which will not stop ingest during add/remove new shard and data rebalance.

Thanks !