ScyllaDB takes 10x more disk space compared to Cassandra?
1,240 views
Skip to first unread message
Cho Ten Van
<chotenvan@gmail.com>Aug 29, 2018, 8:48:49 AM8/29/18
to ScyllaDB users
Hello!
I'm in the process of picking cassandra-like database to migrate 100mln records from Postgres.
So I'm trying to compare Cassandra, ScyllaDB and Rocksandra on similar hardware (HDD disks, for now - I know, Scylla prefers SSD, but I don't have them yet, and I need to know how much space I need).
I've noticed that Scylla takes 10x times more disk space than Cassandra/Rocksandra, so I quickly run out of disk space.
I wonder what is the reason of this?
The data is evenly distributed across partitions and record size is small (see cfhistograms below).
It seems that for every partition (user_id - see table structure below) Scylla reserves like 1Mb of disk space, and it doesn't matter how many records are stored in this partition. While Cassandra stores only what is need.
Probably, this is expected behaviour, but I couldn't find much information about it.
Is there any setting that configures this reserved disk space?
I expect that I will have roughly numRecords = numPartitions*10, and each partition to hold normally 1-300 records for 95% of data, and 300-10000 records for 5% of data.
And there will be like like 5-10 similar denormalized tables, so disk space will be multiplied even more, so...
I would be grateful for any advice!
Thanks in advance!
Cho
==============================
My table structure is the following:
CREATE TABLE element_by_user (
user_id bigint,
element_id text,
...
+ 3 text fields (not more than 100 characters)
+ 10 timestamp fields (sparsely filled, mostly nulls)
+ 1 map field (text,timestamp - not filled yet, all nulls)
+ 1 map field (text, UDT - not filled yet, all nulls. UDT has 10 fields of primitive type, if it matters)
....
PRIMARY KEY ((user_id), element_id)
);
=========== SCYLLA (30% of expected data, experiment was not finished because I run out of disk space)
nodetool cfhistograms usercontent element_by_user
usercontent/element_by_user histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 0.00 15.00 304.00 2759 42
75% 0.00 21.00 440.50 6866 103
95% 0.00 38.00 482.00 61214 770
98% 0.00 43.72 482.00 126934 1916
99% 0.00 49.72 482.00 219342 3311
Min 0.00 5.00 215.00 1110 18
Max 0.00 157256.00 482.00 7007506 88148
nodetool cfstats usercontent.element_by_user
Keyspace: usercontent
Read Count: 23
Read Latency: 3.0434782608695655E-4 ms.
Write Count: 24437989
Write Latency: 2.7676868174382107E-5 ms.
Pending Flushes: 0
Table: element_by_user
SSTable count: 16
SSTables in each level: [16/4]
Space used (live): 31881364688
Space used (total): 31881364688
Space used by snapshots (total): 0
Off heap memory used (total): 571206144
SSTable Compression Ratio: 0.0
Number of keys (estimate): 2262801
Memtable cell count: 81026
Memtable data size: 400627206
Memtable off heap memory used: 549715968
Memtable switch count: 149
Local read count: 23
Local read latency: 0.319 ms
Local write count: 24437989
Local write latency: 0.028 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 1125024
Bloom filter off heap memory used: 1310768
Index summary off heap memory used: 20179408
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 1110
Compacted partition maximum bytes: 7007506
Compacted partition mean bytes: 15313
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0
=========== ROCKSANDRA (100% of data)
nodetool cfhistograms usercontent element_by_user
usercontent/element_by_user histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 0.00 0.00 0.00 215 10
75% 0.00 0.00 0.00 924 50
95% 0.00 0.00 0.00 11864 770
98% 0.00 0.00 0.00 29521 1916
99% 0.00 0.00 0.00 61214 3311
Min 0.00 0.00 0.00 73 2
Max 0.00 0.00 0.00 4055269 219342
nodetool cfstats usercontent.element_by_user
Keyspace : usercontent
Read Count: 3331
Read Latency: 0.39014380066046234 ms
Write Count: 67814678
Write Latency: 0.04589383727516925 ms
Pending Flushes: 0
Table: element_by_user
SSTable count: 1
Space used (live): 4793646958
Space used (total): 4793646958
Space used by snapshots (total): 0
Off heap memory used (total): 5179264
SSTable Compression Ratio: 0.5818218966302031
Number of partitions (estimate): 2976396
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 1546
Local read count: 3331
Local read latency: NaN ms
Local write count: 67814678
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 3720512
Bloom filter off heap memory used: 3720504
Index summary off heap memory used: 465080
Compression metadata off heap memory used: 993680
Compacted partition minimum bytes: 73
Compacted partition maximum bytes: 4055269
Compacted partition mean bytes: 3001
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Glauber Costa
<glauber@scylladb.com>Aug 29, 2018, 9:15:10 AM8/29/18
to ScyllaDB users
Unfortunately this is normal, and is a result of the file format Scylla uses: Cassandra 2.1's.
Cassandra 3.0 uses a new file format - which we will have in a couple of months (almost code ready).
There are two options to reduce the disk usage. One of them is to wait until we adopt the Cassandra 3.0 format.
The other is: one of the many inefficiencies of the Cassandra 2.1 format is that it repeats the name of your regular columns many times
throughout the file: they are used as keys for the values.
So if your column is called "my_column_with_a_big_name", you will see your disk usage going up a lot.
But if you call it "m", you don't see it.
An old trick in the book from the Cassandra 2.x days was to do exactly that, and we routinely come across schemas like that in the field
from people using older cassandra versions.
I'm the first to admit this is pretty ugly, but it's a small price to pay to get the other benefits of Scylla for the time being.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/b3197d5f-669a-4d95-91f9-aa48c2fcb881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nadav Har'El
<nyh@scylladb.com>Aug 29, 2018, 9:37:32 AM8/29/18
to scylladb-users@googlegroups.com
On Wed, Aug 29, 2018 at 3:48 PM, Cho Ten Van <chot...@gmail.com> wrote:
Hello!I'm in the process of picking cassandra-like database to migrate 100mln records from Postgres.So I'm trying to compare Cassandra, ScyllaDB and Rocksandra on similar hardware (HDD disks, for now - I know, Scylla prefers SSD, but I don't have them yet, and I need to know how much space I need).I've noticed that Scylla takes 10x times more disk space than Cassandra/Rocksandra, so I quickly run out of disk space.I wonder what is the reason of this?
As Glauber explained, one of the biggest wastes of Scylla's current sstable format (which is identical to Cassandra 2.x's sstable format) is that the names of columns, as well as clustering keys, are repeated over and over in the disk format. You can reduce some space usage by using shorter field names. Another likely curprit is the length of your clustering key values (the values of element_id) - if these are long, they are repeated for each and every column in that row...
All of this will be improved when we release Scylla with sstable 3 format support (the code is already mostly finished), but before that, turning on *compression* may provide you with significant space. Compression will get rid of at least some of the duplication in the disk format. See https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsConfigCompress.html (for example) on how to turn compression on in Cassandra or Scylla (in Scylla and Cassandra 2.x it's significantly more important than in Cassandra 3.x).
The data is evenly distributed across partitions and record size is small (see cfhistograms below).It seems that for every partition (user_id - see table structure below) Scylla reserves like 1Mb of disk space, and it doesn't matter how many records are stored in this partition.
This should not be the case. How did you get this impression?
Small partitions only take the amount of space they need - they are not rounded to 1MB or anything of that sort.
While Cassandra stores only what is need.Probably, this is expected behaviour, but I couldn't find much information about it.Is there any setting that configures this reserved disk space?I expect that I will have roughly numRecords = numPartitions*10, and each partition to hold normally 1-300 records for 95% of data, and 300-10000 records for 5% of data.And there will be like like 5-10 similar denormalized tables, so disk space will be multiplied even more, so...I would be grateful for any advice!Thanks in advance!Cho==============================My table structure is the following:CREATE TABLE element_by_user (user_id bigint,element_id text,...+ 3 text fields (not more than 100 characters)+ 10 timestamp fields (sparsely filled, mostly nulls)
Note that unset values should not take space on disk.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Cho Ten Van
<chotenvan@gmail.com>Aug 29, 2018, 9:38:29 AM8/29/18
to ScyllaDB users
Thanks a lot for quick reply!
I guess we will wait for new version :)
We are still in RnD stage, so maybe by the time we are closer to production new version will be ready, looking forward for it!
So far will test everything else with Cassandra/Rocksandra.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Avi Kivity
<avi@scylladb.com>Aug 29, 2018, 9:44:18 AM8/29/18
to scylladb-users@googlegroups.com, Glauber Costa
10X is not normal or expected if you have 3 text fields that have data, unless:
- the field names are very long, or
- the amount of data in the field is very small, or
- element_id contents is very large
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAD-J%3Dzb3HkfRpM5FnWoUXU4Yj5KZEAgaTCg0A6%2B35-nYoht%2BwQ%40mail.gmail.com.
Cho Ten Van
<chotenvan@gmail.com>Aug 29, 2018, 9:45:04 AM8/29/18
to ScyllaDB users
Nadav, thanks for further clarification!
As for my wild guess about 1Mb per partition - I admit this was just some explanation I tried to come up with while looking at disk space growing during data migration:) Didn't expect key/column name lengths could affect that.
Thanks again, guys
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
Nadav Har'El
<nyh@scylladb.com>Aug 29, 2018, 9:48:36 AM8/29/18
to scylladb-users@googlegroups.com, Glauber Costa
On Wed, Aug 29, 2018 at 4:44 PM, Avi Kivity <a...@scylladb.com> wrote:
10X is not normal or expected if you have 3 text fields that have data, unless:
- the field names are very long, or
- the amount of data in the field is very small, or
- element_id contents is very large
I think the most likely explanation (but I'm still guessing :-)) is large element_id (clustering key) values, in addition to large number of fields in each clustering row - the clustering key is repeated for each of those fields :-(
If that is the case, compression should help significantly.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/b3197d5f-669a-4d95-91f9-aa48c2fcb881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAD-J%3Dzb3HkfRpM5FnWoUXU4Yj5KZEAgaTCg0A6%2B35-nYoht%2BwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/ef201c14-941b-15d8-a9f0-e0c4dce95af7%40scylladb.com.
Cho Ten Van
<chotenvan@gmail.com>Aug 29, 2018, 9:55:50 AM8/29/18
to ScyllaDB users
Avi, thanks for reply!
element_id,license_id,transaction_id is in fact UUIDs (like 123e4567-e89b-12d3-a456-426655440000), I just haven't updated the schema yet to use UUIDs.
element_type,consumption_mode is enums with values like 5-10 symbols.
I don't know, maybe UDTs add length too (I thought they are internally mapped to "wide row" columns like "license_map|123e4567-e89b-12d3-a456-426655440000|activate_ts"), but they are null, so there is no such keys anyway.
CREATE TYPE element_nested_license (license_id text, transaction_id text, consumption_mode text, quality text, subscription_id text, create_ts timestamp, activate_ts timestamp, expire_ts timestamp, last_start_playback_ts timestamp, created_for_free boolean, window_extended boolean, free_preview_period bigint, last_drm_error text, play_position bigint);
CREATE TABLE element_by_user (user_id bigint, element_id text, element_type text, parent_element_id text, parent_element_type text, purchase_ts timestamp, rate_ts timestamp, watch_ts timestamp, bookmark_ts timestamp, pin_ts timestamp, remove_ts timestamp, last_start_ts timestamp, consume_ts timestamp, free_preview_ts timestamp, reserve_ts timestamp, watch_count int, rating float, expire_ts timestamp, play_position bigint, license_map map<text, FROZEN<element_nested_license>>, bookmarked_in_subscription map<text, timestamp>, PRIMARY KEY ((user_id), element_id));
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/b3197d5f-669a-4d95-91f9-aa48c2fcb881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Cho Ten Van
<chotenvan@gmail.com>Sep 3, 2018, 9:13:50 AM9/3/18
to ScyllaDB users
Hi again!
Just wanted to let you know - I've changed element_id type to UUID (instead of text which stored uuids as strings) and space used is about 1.5x less now.
The rest probably is indeed long field names.
Reply all
Reply to author
Forward
0 new messages