Extend single node to a cluster

[Siddharth Jain]

Hello,

we are new to clickhouse. we setup a single node on a powerful VM with 8 vCPUs and 32 GB RAM but the performance was not as good as BigQuery on a 250 GB dataset. we would like to extend the single node and form a cluster to see if it gives improvement in performance. How can this be done? we don't want to create a new cluster then load data into it. we want to re-use the data that has been loaded onto the single node - so we are looking to extend the current setup. Loading the data is very expensive for a 200 GB dataset with billions of rows in it. Thanks.

[kranthi kumar]

is it extending the cluster by adding a Replica or adding a new shard? 

Adding server as replica is very simple, whereas Shard need to be re-balance the data manually. 

also without increasingthe server CPU and Memory, with same sizes of node as Replica or shard will not be good idea. 

the minimum requirements from vendor is 64 GB RAM and 16 CPU core. 

On Sat, Oct 15, 2022 at 10:51 PM Siddharth Jain <sidd...@gmail.com> wrote:

Hello,

we are new to clickhouse. we setup a single node on a powerful VM with 8 vCPUs and 32 GB RAM but the performance was not as good as BigQuery on a 250 GB dataset. we would like to extend the single node and form a cluster to see if it gives improvement in performance. How can this be done? we don't want to create a new cluster then load data into it. we want to re-use the data that has been loaded onto the single node - so we are looking to extend the current setup. Loading the data is very expensive for a 200 GB dataset with billions of rows in it. Thanks.

--
You received this message because you are subscribed to the Google Groups "ClickHouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/CAPqV3pR4B%3DLro5sWMNavPatd2Ga3KWV0xF5PA7O8McLdU0Sz9A%40mail.gmail.com.

--

------------------------

Thanks,

Kranthi Kumar.M

[Vanya Blinkov]

Hi Siddharth,

This node spec is barely on par with a modern laptop, thus by no means powerful. Mainstream baremetal servers go all the way to 256 vCPU and multiple terabytes of RAM nowadays.

You can use 

https://benchmark.clickhouse.com/hardware/ to get a ballpark of how ClickHouse will perform on various hardware.

For sharding see documentation on the "Distributed" table engine, for replication (high availability) you'll need a variation of "ReplicatedMergeTree" table engine. However, I'd recommend increasing the node size before introducing sharing.

--

Ivan Blinkov

To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/CAPtHTSGdd-1xb-W9ELkfDhBS-4YnmPMP-T%2B1gusb2gj1g95hpw%40mail.gmail.com.

--

--

Ivan Blinkov