How load sstables fastly

[Raphaël MAZELIER]

Hi scylladb communauty,

At work we are in the process to test/benchmark scylla as a drop in replacement for cassandra.

Compatibility test are ok ; and now we want to stress test with real data.

On cassandra we are using sstableloader to daily ingest 300G of new sstable generated offline from spark and stored on s3. Generally it take around 2 hours to load into our cluster. 

Trying to load with the same exact script works but it's painfully slow.

Good thing is that our generated sstable seems ok. 
Bad thing is that time is absolutely not acceptable for us. At this rate it would more than one day.

Reading the output of sstable loader (and debugging some fw issue) I suspect this tool is absolutely not acting as the cassandra counterpart.

It seems to use the cqlsh port ?! 

and the output confirm this :

===== Using optimized driver!!! =====
WARN  19:01:23,795 dclocal_read_repair_chance and read_repair_chance table options have been deprecated and will be removed in version 4.0
Adding sstable /sstdir/1/01c39218-98b1-43fa-9560-98044af34cc7/ks_user_counters/user_counters_batch/me-2-big
 25% done.  1807425 statements sent (in  1378914 batches,        0 failed).
 1807427 statements generated.
 2758969 cql rows processed in  1378914 partitions.
       0 cql rows and        0 partitions deleted.
       0 local and        0 remote counter shards where skipped.

real    2m46.709s
user    3m14.151s
sys     0m54.377s

for example this one take 3min + for a one 120meg sstable. this is super bad.

Actually doing more test it's even worst. Sometime the sstable just ignore sstable and load nothing. Also notice the 25% done.. 

Is there another option to quickly load data into scylla? 
I looked at nodetool refresh -las but again it's not documented and does not seems to work...

any help appreciated.

best

[Avi Kivity]

sstableloader indeed uses the CQL port rather than joining the cluster.

It should not be particularly slow (it's not too fast either). Perhaps --use-unset will help, don't know why it's not the default. This helps if you have many columns, of which only a different subset are used in different rows.

The trick to getting good performance is to have enough concurrency. Don't load one sstable, load many at the same time. IIRC (not sure) sstableloader will do this if you give it a directory to process, see also --threads-count. You should also use multiple clients.

load-and-stream will be faster. It is documented [1], and it works, we use it extensively. Again the trick is to load multiple sstables per node, and to load on all nodes in parallel.

[1] https://opensource.docs.scylladb.com/stable/operating-scylla/nodetool-commands/refresh.html

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/f5d1b49f-258b-4f02-af35-c2a34252f64cn%40googlegroups.com.

[Yaniv Kaul]

sstableloader I believe is also destined to be retired, as it doesn't support some features (UUID based and such).

Y.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/f119cdd4f5cfe6c902ad635fe087d33234ebe642.camel%40scylladb.com.

[Raphaël MAZELIER]

Hey thanks for the answer;

Indeed C* sstableloader stream sstable in // which make it fast.

> load-and-stream will be faster. It is documented [1], and it works, we use it extensively.

I should be stupid but I just don't understand where are supposed to be placed sstable.
The doc stated that this function can stream to another cluster ; so I guess I can put my files somewhere on another box and then do a nodetool ; but it's just doing nothing ?!

If I had to put the sstable on one of my cluster node it will made the whole thing harder. Any hints?

Best,

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/xn-ZOfaSy0I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/f119cdd4f5cfe6c902ad635fe087d33234ebe642.camel%40scylladb.com.

[Avi Kivity]

On Mon, 2024-03-25 at 12:43 +0100, Raphaël MAZELIER wrote:

Hey thanks for the answer;

Indeed C* sstableloader stream sstable in // which make it fast.

> load-and-stream will be faster. It is documented [1], and it works, we use it extensively.

I should be stupid but I just don't understand where are supposed to be placed sstable.

The /upload subdirectory of the table's directory.

The doc stated that this function can stream to another cluster ; so I guess I can put my files somewhere on another box and then do a nodetool ; but it's just doing nothing ?!

If I had to put the sstable on one of my cluster node it will made the whole thing harder. Any hints?

Maybe it's harder, but that's what we have.

[Raphaël MAZELIER]

> Maybe it's harder, but that's what we have.

Indeed it's working. But it's super non convenient for multiple reason.
First of all we need to be intra cluster to use it. Second the directory name is not fixed since it depends of the name of the sstable dir with an uuid. Third we can just hope than nodetool finish without error (like repair)/ Progress can only be found in server logs. I understand that scylla mimics c* behaviour but maybe there places for user friendly improvements?

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/xn-ZOfaSy0I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/0a1b5481b3f21fd897096a4fac92beeaa0a63329.camel%40scylladb.com.

[Avi Kivity]

On Sun, 2024-03-31 at 21:06 +0200, Raphaël MAZELIER wrote:

> Maybe it's harder, but that's what we have.

Indeed it's working. But it's super non convenient for multiple reason.
First of all we need to be intra cluster to use it. Second the directory name is not fixed since it depends of the name of the sstable dir with an uuid. Third we can just hope than nodetool finish without error (like repair)/ Progress can only be found in server logs. I understand that scylla mimics c* behaviour but maybe there places for user friendly improvements?

You can use sstableloader if load-and-stream is inconvenient for you.

[Raphaël MAZELIER]

> You can use sstableloader if load-and-stream is inconvenient for you.

Things is that scylla sstableloader is super slow (contrary to the C* one, I suspect the implementation highly differ). 
Currently I search to fast load 2 billions rows from sstable to scylla. I have sstables from other data pipeline jobs (currently we don't want to rewrite them but let see).

You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/xn-ZOfaSy0I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/d5cea64891db4f8a95c2b4d1e4e6bac1baa3d323.camel%40scylladb.com.

[Avi Kivity]

On Sun, 2024-03-31 at 22:55 +0200, Raphaël MAZELIER wrote:

> You can use sstableloader if load-and-stream is inconvenient for you.

Things is that scylla sstableloader is super slow (contrary to the C* one, I suspect the implementation highly differ). 
Currently I search to fast load 2 billions rows from sstable to scylla. I have sstables from other data pipeline jobs (currently we don't want to rewrite them but let see).

It won't be super slow if you use it with sufficient concurrency.

[Raphaël MAZELIER]

Quick follow up on this thread.

Concerning scylla ssttableloader it's still super slow despite putting enough concurrency. This is strange cause I test with a small insertion dev (on golang with 1000x goroutine concurrency) and I achieve correct result at inserting (100k insert/s).

And also testing sstableloader with dummy mode, still super slow. So I'm really wondering why this tool is so slow (ok sstable format is hmm special )

So as mentioned in another thread/topic/discussion I start creating a golang "sstable loader" for fun. I made considerable progress at the point I can decode a simple sstable (meaning a table with "simple" schema). 

I would have some questions about the format tough but in a separate thread.

Best,

[Tzach Livyatan]

On Fri, Apr 12, 2024 at 7:27 PM Raphaël MAZELIER <rap...@adikteev.com> wrote:

Quick follow up on this thread.

Concerning scylla ssttableloader it's still super slow despite putting enough concurrency. This is strange cause I test with a small insertion dev (on golang with 1000x goroutine concurrency) and I achieve correct result at inserting (100k insert/s).

And also testing sstableloader with dummy mode, still super slow. So I'm really wondering why this tool is so slow (ok sstable format is hmm special )

I wonder where the bottleneck is.

Is it on the parsing, network bandwidth, or ScyllaDB?

If it's the first, more concurrency might help.

If it's the last, we need to understand why, and updating the loader might not help.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/eb6feedf-26d4-4df2-95e1-5a8c218bd58dn%40googlegroups.com.