re-join node with existing data in filesystem?

[Martin Brzenska]

Hi, 

I had a kind of a disaster these days. Several Nodes has hardware issues and I removed those nodes from the cluster (using `nodetool removenode`). 

Several nodes failed almost the same time - more then RF.

Now, it looks like the data in the filesystem can be rescued - so my question is:

Can I re-join the Nodes with the old data in the datadir and the data will be available again? If so, how should i proceed in such a case?

Thanks in advance

Martin

[Asias He]

Hello Martin,

Once a node is removed from the cluster, you can not rejoining the old nodes. If you have the data recovered, you can copy the sstables into the upload directory and run nodetool refresh to load that data back to the cluster.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/36a5593e-13b1-4969-95cf-789c398a7e09n%40googlegroups.com.

--

Asias

[Martin Brzenska]

Hi Asias,

that sounds great!

If i understand correctly, i would go each node at a time and for each node do:

- Add the new and empty node to the cluster

- Wait for it to be UN

- Copy the recovered files from the old FS under (old)/var/lib/scylla/data/$KEYSPACE/$TABLENAME/ to (new)/var/lib/scylla/data/$KEYSPACE/$TABLENAME/upload/ 

[Asias He]

On Mon, Oct 18, 2021 at 4:55 PM 'Martin Brzenska' via ScyllaDB users <scyllad...@googlegroups.com> wrote:

Hi Asias,

that sounds great!

If i understand correctly, i would go each node at a time and for each node do:

- Add the new and empty node to the cluster

- Wait for it to be UN

- Copy the recovered files from the old FS under (old)/var/lib/scylla/data/$KEYSPACE/$TABLENAME/ to (new)/var/lib/scylla/data/$KEYSPACE/$TABLENAME/upload/ 

I assume you still have some nodes from the existing cluster. For example, you have n1,n2,n3,n4,n5. Disks on n4 and n5 are broken. You ran nodetool removenode to remove n4 and n5. Files from n4 and n5 are recovered. To load the recovered data, you can

Put files from n4 and n5 to upload directory of n1, n2, n3.

Run nodetool refresh on n1, n2 n3.

That's it.

Then you can bootstrap new nodes if needed.

With scylla, data will always be streamed to the new node, so there will be no "empty" node.

 

On Monday, October 18, 2021 at 10:26:26 AM UTC+2 as...@scylladb.com wrote:

Hello Martin,

Once a node is removed from the cluster, you can not rejoining the old nodes. If you have the data recovered, you can copy the sstables into the upload directory and run nodetool refresh to load that data back to the cluster.

On Mon, Oct 18, 2021 at 4:20 PM 'Martin Brzenska' via ScyllaDB users <scyllad...@googlegroups.com> wrote:

Hi, 

I had a kind of a disaster these days. Several Nodes has hardware issues and I removed those nodes from the cluster (using `nodetool removenode`). 

Several nodes failed almost the same time - more then RF.

Now, it looks like the data in the filesystem can be rescued - so my question is:

Can I re-join the Nodes with the old data in the datadir and the data will be available again? If so, how should i proceed in such a case?

Thanks in advance

Martin

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/36a5593e-13b1-4969-95cf-789c398a7e09n%40googlegroups.com.

--

Asias

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/ddf54841-9ce3-4cd6-9a23-e433b8158028n%40googlegroups.com.

--

Asias

[Martin Brzenska]

Out of curiosity: Has sstableloader a disadvantage to the nodetool refresh approach? 

[Benny Halevy]

On Fri, 2021-10-29 at 13:14 -0700, 'Martin Brzenska' via ScyllaDB users wrote:

Out of curiosity: Has sstableloader a disadvantage to the nodetool refresh approach? 

sstableloader is typically slower since it's using CQL to load the sstable constents

vs. nodetool refresh that loads the sstable contents directly on each node using

with no need to parse and generate queries as well as no need for messaging

over the network.