Error codes 252 and 242

[Oscar Campos]

Hi everyone,

We are testing ClickHouse to determine if it is a good fit for our use case and if it could replace our current statistics system, we installed a cluster with 4 nodes with the following configuration:

• 1 Layer
  
• 2 Shards
  
• 2 Replicas per Shard
  

So we have four nodes in total and a node that makes all the ingestion of data inserting CSV format files concurrently (split the files in 16 chunks and insert them in each working node in round robin fashion) . It is working fine but some times we get the pair of error codes 252 and 242 in one of the shards.

I guess that is probably some network issues between the two nodes that makes the replication process to lag but I don't really know as I could not find any related information over the internet regarding this problem. The errors are:

Error code 252 DB::Exception: Too much parts. Merges are processing significantly slower than inserts..  (this one happens in layer1-shard2-replica1)

Error code 242 DB::Exception: Table is in readonly mode. (this one happens in layer1-shard2-replica2)

The Shard 1 never show the problem.

Any ideas?

Thanks.

[Oscar Campos]

Looks like the problem is related with Zookeeper, we have a zookeeper instance running in each node and ClickHouse connects to Zookeeper using localhost, we should probably use a proxy so if the local node dies for some reason we still can connect to a healthy node,said that, there is no errors in the Zookeeper logs.

[man...@gmail.com]

You should specify all ZooKeeper nodes in configuration file on each server.
No proxy required: if you specify all nodes, server will select a healthy node automatically.

It is recommended to have separate ZK cluster with three nodes.

Table is in readonly mode.

It means, that table has temporarily lost a session with ZK. It will try to restore a session.
You could find related messages in error log. You will also find messages related to libzookeeper in /var/log/clickhouse-server/stderr.

[Oscar Campos]

Thanks