Scalability in a cluster ?

95 views
Skip to first unread message

Dorian Hoxha

unread,
Jun 12, 2016, 11:17:34 AM6/12/16
to CockroachDB user group
So::

1. max number of nodes in a cluster (ex: rethinkdb supports up to ideally 20)
1.1. Does the scalability increase by 1 with each node (assuming even usage on all ranges) or something <1 like in postgresql-xl ?
2. max storage per node (some kind of limit on the lower storage layer, ex like pg has)
3. max rows/kv per node/range/table
4. max number of columns in a table (ex: we store column-id as 1-byte so 256 max or something similar)
5. max number of tables (ex: more than x tables it's an antipattern since each table has overhead in ram)
6. max number of ranges (each range has overhead, does each node know the position of each range, or only range.ranges?)

These I think should also be in the faq/docs ?

jesse

unread,
Jun 13, 2016, 5:53:53 PM6/13/16
to CockroachDB user group

Hi Dorian,


In general, CockroachDB is designed to support thousands of nodes, and there are no hard limits in terms of max nodes, max storage per node, max columns in a table, etc. There may be limits in practice, though. For example, all table metadata is currently held in memory, so you don’t want to have too many tables. As we get closer to production readiness, we'll be doing larger scale tests that will let us discover the practical limits of the system, and we'll be sure to document those. 


Best,
Jesse

Peter Mattis

unread,
Jun 14, 2016, 9:56:36 AM6/14/16
to Dorian Hoxha, CockroachDB user group
Hi Dorian,

On Sun, Jun 12, 2016 at 11:17 AM, Dorian Hoxha <dorian...@gmail.com> wrote:
So::

1. max number of nodes in a cluster (ex: rethinkdb supports up to ideally 20)

CockroachDB is designed to scale to hundreds and even thousands of nodes. In practice, 10s of nodes is more realistic at this time. We're working hard at making larger clusters work smoothly.
 
1.1. Does the scalability increase by 1 with each node (assuming even usage on all ranges) or something <1 like in postgresql-xl ?

The scalability depends on your workload, but the short answer is that you're going to see <1 performance increase for each additional node if you have any sort of realistic schema with multiple tables and indexes. I don't have a more concrete answer for you at this time, but agree it would be good to provide guidance here.
 
2. max storage per node (some kind of limit on the lower storage layer, ex like pg has)

Postgres has a limit on the disk storage it will use? There is no hardcoded limit in CockroachDB.
 
3. max rows/kv per node/range/table

There is no limit on the number of rows/kvs per node or table. Ranges hold up to 64MB of row/kv data, but will split automatically. The 64MB size is configurable, but you shouldn't set it larger right now as larger ranges have some problems which we're working to resolve (i.e. certain operations use memory proportional to the range size). This also implies that row/kv data should be significantly smaller than a range. 
 
4. max number of columns in a table (ex: we store column-id as 1-byte so 256 max or something similar)

There is no hardcoded limit. Column IDs are 32-bit variable length encoded integers. 
 
5. max number of tables (ex: more than x tables it's an antipattern since each table has overhead in ram)

There is no hardcoded limit, though each table will consume a small amount of memory (a few KB) on every node.
 
6. max number of ranges (each range has overhead, does each node know the position of each range, or only range.ranges?)

There is no hardcoded limit. The range indexing information is cached on demand. The design here is similar to BigTable/HBase and is intended to scale to very large numbers of ranges. 
 
These I think should also be in the faq/docs ?

 Agreed that the above should be in faq/docs. 

--
You received this message because you are subscribed to the Google Groups "CockroachDB user group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroachdb-us...@googlegroups.com.
To post to this group, send email to cockroac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cockroachdb-users/182bc3b3-a4b0-4af6-9329-2483067e8b71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Spencer Kimball

unread,
Jun 14, 2016, 10:09:04 AM6/14/16
to Peter Mattis, Dorian Hoxha, CockroachDB user group, Jesse Seldess
+jesse for adding these answers to the FAQ.


For more options, visit https://groups.google.com/d/optout.



--
Spencer Kimball | Co-founder & CEO
Cockroach Labs

Jesse Seldess

unread,
Jun 14, 2016, 10:26:08 AM6/14/16
to Spencer Kimball, Peter Mattis, Dorian Hoxha, CockroachDB user group
Thanks, Spencer. Created https://github.com/cockroachdb/docs/issues/380. I'll work these into the FAQ soon.

Dorian Hoxha

unread,
Jun 14, 2016, 1:05:09 PM6/14/16
to CockroachDB user group
@ Peter, pg limits:
Limit

Value
Maximum Database SizeUnlimited
Maximum Table Size32 TB
Maximum Row Size1.6 TB
Maximum Field Size1 GB
Maximum Rows per TableUnlimited
Maximum Columns per Table250 - 1600 depending on column types
Maximum Indexes per TableUnlimited

Peter Mattis

unread,
Jun 14, 2016, 3:38:10 PM6/14/16
to Dorian Hoxha, CockroachDB user group
Thanks, Dorian.

I wonder where the Postgres table size limitation comes from given that tables can contain an unlimited number of rows.

--
You received this message because you are subscribed to the Google Groups "CockroachDB user group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cockroachdb-us...@googlegroups.com.
To post to this group, send email to cockroac...@googlegroups.com.

Dorian Hoxha

unread,
Jun 15, 2016, 4:29:03 AM6/15/16
to Peter Mattis, CockroachDB user group
Pg rows are stored in 8kb(default) pages and there can be up to 32bit signed integer pages. Pages can be up to 64kb in size

Peter Mattis

unread,
Jun 15, 2016, 8:21:21 AM6/15/16
to Dorian Hoxha, CockroachDB user group
Got it. Thanks for the details.
Reply all
Reply to author
Forward
0 new messages