Problems with Scylla Migrator for DynamoDB

[johan...@gmail.com]

Hi. I'm trying to copy over a DynamoDB table to a test Scylla node. It looks like the workers are getting killed? See attached logs. This is the command I tried:

/path/to/spark/bin/spark-submit --class com.scylladb.migrator.Validator   --master spark://ip-xxx-xx-xx-xx.us-east-2.compute.internal:7077    --conf spark.scylla.config=./config.yaml  ./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar

(Also tried com.scylladb.migrator.Migrator)

My config.yaml is patterned after the example at https://www.scylladb.com/2020/09/02/one-step-streaming-migration-from-dynamodb-into-scylla/ 

Can someone tell me what I'm doing incorrectly?  Migrator is on a T3.Large instance (8G mem). Thanks!

Johann

[johan...@gmail.com]

Anyone? I know people are busy with the summit but I hope someone can respond to this. Thanks!

[Ľuboš Koščo]

Hello Johann,

definitely use com.scylladb.migrator.Migrator , Validator is for afterwards to compare the rows from source to target and report any diffs (so to validate the migration, if it was fully OK).

So please try with Migrator class

your workers need to have access to both source and target from your config.yaml

Streams also need to be enabled in DynamoDB as per the blog first.

Let us know how that goes.

xing the fingers

L

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/c6a17bcc-686d-463a-a820-75b00574acf9n%40googlegroups.com.