Are there any limit on number of columns in ClickHouse

2,751 views
Skip to first unread message

Edward

unread,
Jun 28, 2016, 7:19:26 AM6/28/16
to ClickHouse
For instance, will ClickHouse be able to support 10 000 columns in a single table? What about 100 000? Are there any limits?

Edward

unread,
Jun 29, 2016, 12:28:40 PM6/29/16
to ClickHouse
I'm storing time-series data in matrix format, so max number of columns is very important for me.
Any ideas about max number of columns?

man...@gmail.com

unread,
Jun 29, 2016, 4:03:44 PM6/29/16
to ClickHouse
Number of columns is not limited explicitly.
1000 columns will work Ok.
More than 1000 columns is not tested in production.

I assume that 10 000 will work poor.
It because in MergeTree table, there are two files for each column in each data part. Data part is created for each INSERT, and data parts are merged in background.
It should work normally if you will INSERT data in large enough batches.
(Maybe, you will need to increase max_memory_usage)

I suggest you don't try 100 000 columns... Maybe it will work for table of type Log and StripeLog. Most likely, there will be not enough memory to insert data in large batches, and if you insert in small batches, performance will be worse. Consider batches of 8192 rows. Then, if each value is UInt32, it is 3.2 GB. Seems possible, though.

Edward

unread,
Jun 30, 2016, 5:52:28 AM6/30/16
to ClickHouse
Thanks a lot!
Reply all
Reply to author
Forward
0 new messages