Redis Cluster Timeout

[sirbadger]

With the release of version 3 we're trying to move an application from using twemproxy to redis cluster.  

Our application currently purges our redis instances overnight and loads data for the following day, during the data load random nodes are being ejected from the cluster and then after a short time are appearing back as slaves (we have 6 masters and 6 slaves).  The process we are runnings is:

1) flushdb on each master instance

2) load data

3) run "return #redis.call('keys', 'moo.*')"

Could the time taken to run flushdb or the lua script be causing the cluster to eject nodes?

FYI, we are using StackExchange.Redis as the client

[Salvatore Sanfilippo]

Both issues are possible, if you want to do very slow operations which
should not be considered as downtime by Redis Cluster, the simplest
thing to do is to use a bigger node-timeout.
You may even do hack the node timeout with CONFIG SET just before
those slow operations, setting it to a few minutes, and after the
cleanup is finished re-set it to a value which makes the cluster not
so slow to react on real failures.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--
Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Marc Gravell]

As an aside: don't use KEYS if possible. SCAN has full support in SE.Redis; and also note that your Lua at the moment has no specific target - KEYS and SCAN only work inside a single server. For that reason, on SE.Redis this is on the IServer interface (via GetServer()), not the IDatabase interface.

You can probably use a fairly large page size to SCAN (to reduce the trips) without causing too much harm.

--

[Marc Gravell]

For full examples (and explanation) of using SCAN in SE.Redis: see https://github.com/StackExchange/StackExchange.Redis/blob/master/Docs/KeysScan.md

Note that even though the method is called `.Keys(...)`, it decides internally whether to use `KEYS` or `SCAN` based on what it knows about the server. If `SCAN` is available, the iterator that it returns automatically pages through the data as you iterate (it does also support explicit resume from a known cursor position etc, but that is a bit more advanced)

[sirbadger]

Thanks for the reply.  I assumed that would be the problem.  We only run the slow operations overnight when our apps are offline so I think we'll try the CONFIG SET approach.

[sirbadger]

Thanks for the tip, will take a look at SCAN.  I should have been more clear about the lua step, we run it against each target and total them up.