Maximum number of shards
25 views
Skip to first unread message
Maxim Fridental
Apr 11, 2022, 5:17:58 AM4/11/22
to ClickHouse
Hi there,
what is the practical maximum number of shards for a distributed table? Each shard has only one replica and the distributed table is only used for reading, never for writing.
Altinity blog believes we can have dozens of shards. But can we go as high as 1000 shards? 100000 shards?
Best,
Maxim
Alexey Milovidov
Apr 14, 2022, 5:55:35 PM4/14/22
to Maxim Fridental, ClickHouse
600 shards configuration is proven to work.
There is evidence that 1000 shards work.
It starts to slow down short queries significantly. On 600 shards
config, the queries always take no less than 100ms.
If you use only one replica, you will also have to turn on the
"skip_unavailable_shards" setting.
10 000 shards may work but overhead for a single query will be too high.
> --
> You received this message because you are subscribed to the Google Groups "ClickHouse" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/35b902cb-132d-4c54-b4e5-0ffec60e6f20n%40googlegroups.com.
There is evidence that 1000 shards work.
It starts to slow down short queries significantly. On 600 shards
config, the queries always take no less than 100ms.
If you use only one replica, you will also have to turn on the
"skip_unavailable_shards" setting.
10 000 shards may work but overhead for a single query will be too high.
> You received this message because you are subscribed to the Google Groups "ClickHouse" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/35b902cb-132d-4c54-b4e5-0ffec60e6f20n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages