nodetool repair takes too much time

[Jakub Liska]

Hey,

I have a 3 x i3.xlarge cluster with RF=3, 4 tables, each with a few hundreds of millions of small rows and properly sized partittions.

"nodetool repair" is running for almost 4 hours already, in past 3 hours the cluster is utilized only on 25%.

Is repair time expected to be that long? 

Is it better to do this on an idling cluster or can I run it weekly on a busy cluster as I cannot really have maintenance windows that wide.

[Jakub Liska]

I used ONE write consistency and I can tell that there is at least 0.001% of rows not present on all replicas at the same time.

Which is fine but I'd have to do repair at least weekly, but it would not be acceptable if it took 5-6 hours.

[Jakub Liska]

I think that there is some kind of dead lock because it is running for almost 6 hours and I'm getting a feeling that it will never end ...

The cluster is idle though, no requests are made...

[Dor Laor]

Jakub, repair is designed to run outside of the maintenance window.

We recently slew it down to make sure it doesn't add latency to the foreground

operations. We also added a command to estimate the remaining work. 

Asias, can you please give Jakub a hand here?

We have an active work around repair and we intend to make it faster and more efficient.

On Mon, Jun 12, 2017 at 6:43 AM, Jakub Liska <liska...@gmail.com> wrote:

I think that there is some kind of dead lock because it is running for almost 6 hours and I'm getting a feeling that it will never end ...

The cluster is idle though, no requests are made...

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/7c9b9f8c-e63e-42bc-a530-f6dce2e407eb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jakub Liska]

Hi,

it took almost 8 hours in the end and it needed ~ 30% of network and disk IO and CPU on average.

However it's a good news that it can serve R/W requests when it's is being repaired. 

Please let me know what is the command to estimate remaining work.

Thank you, Jakub

[Asias He]

On Jun 13, 2017 00:22, "Jakub Liska" <liska...@gmail.com> wrote:

Hi,

it took almost 8 hours in the end and it needed ~ 30% of network and disk IO and CPU on average.

However it's a good news that it can serve R/W requests when it's is being repaired. 

Repair is designed to work with cql requests in  flights and not affect them much thanks to the workload conditioning. So you can run repair outside the maintenance window. 

Please let me know what is the command to estimate remaining work.

We recently have a patch in this area to support report the repair progress in the log. Which scylla version are you using?

Thank you, Jakub

[Jakub Liska]

I'm on 1.7 rc3... On the AMI for i3 instances.

[Jakub Liska]

Btw this nodetool repair only http://i.imgur.com/6CYqRZy.png

Yesterday, one of my nodes became unreachable due to heavy querying (no indexing) so I had to restart it and after it started up, nodetool status showed it was missing like 15% of load than it had before that crash.

So I had to run repair again on that node.

[Jakub Liska]

Btw when I have a spare cluster time for a longer maintenance window, is it safe to run repair on all nodes simultaneously ? 

[Dor Laor]

On Tue, Jun 13, 2017 at 1:31 PM, Jakub Liska <liska...@gmail.com> wrote:

Btw when I have a spare cluster time for a longer maintenance window, is it safe to run repair on all nodes simultaneously ? 

Yes, if you're not serving requests and have idle time by all mean run repair in parallel.

Note that there are flags which makes it more efficient so you wouldn't repair replica data which

was already repaired by the other replicas

 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/eb078385-e925-4503-8975-679a0adb9765%40googlegroups.com.

[Jakub Liska]

Hey Dor,

just to be sure, on a 3 node cluster, is this :

nodetool repair -seq -hosts ccda8c3f-ec55-4020-9591-945fad0c100c,e1465f3a-9d8b-48ef-9b16-4c6ff5557c50,bfd7ba99-6e20-454c-8263-d0c7ef349b2c myKeyspace

the same thing as 

nodetool repair -full myKeyspace

Both these commands should repair the whole cluster sequentially, right?

If I do this : 

nodetool repair myKeyspace

Only the node from which I issue this command gets fixed, right?