--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
- Josiah
Why not a config file option? Because configuration files are the
kinds of things that you set and never look at again, then 6 months
down the line someone is digging through it and asking "wtf did we do
that for?" I could get behind "SLAVEOF host port ONCE" if it was only
available via a remote command, and not with a configuration option.
Regards,
- Josiah
I might suggest the ability to configure a global lock file that is
shared by all Redis instances on a single physical server. The lock
could be used to block and/or delay [BG]SAVEs and/or SLAVEOF commands.
This would allow a set of instances to insure only a single instance
is attempting to save and/or sync at any given moment, reducing
contention for these high-resource commands.
Ideally, you could configure different lock files for BG[SAVE] and
SLAVEOF commands, although they might point to the same file.
The lock file could be a simple PID file. If the file existing (and
the process exists), the system is locked. If no file exists, a
process can grab the lock by simply writing out the file. Race
conditions can be avoided with the proper flags to open(2).
In the case of SAVE, I would have the command immediately return an
error if the lock cannot be acquired. For SLAVEOF, the command would
simply go idle until the lock can be acquired. BGSAVE might go idle
or might return... I'm not sure which makes more sense.
In the case of BGSAVE and SLAVEOF, the fork() would not be allowed until
the instance owns the lock. The lock would then be released as soon as
the child process exits (or, in the case of a SLAVEOF, when the
initial bulk transfer is complete). You might also be able to
configure a time-out, so that SLAVEOF returns an error after 300
seconds or something. Any time a command is outstanding, the system
would check for the lock ever 250ms or some other configurable value.
If we really want to get fancy, we could allow a set of lock files,
say .../redis-lock-[1-4].pid, to allow up to four operations at one
time. This might be useful for very larger servers with, for
example, a dozen instances. The lock files could still be used to
limit "overhead" resource usage, but would allow more than one
high-usage operation at a time.
Thoughts?
-j
--
Jay A. Kreibich < J A Y @ K R E I B I.C H >
"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson
--
--
Maybe she awoke to see the roommate's boyfriend swinging from the
chandelier wearing a boar's head.
Something which you, I, and everyone else would call "Tuesday", of course.
On Wed, Feb 22, 2012 at 10:11 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> The problem is that occasionally the WAN link between the data centers is
> interrupted and the slaves in data center B decide they need to re-sync with
> their masters. Unfortunately, *all* the instances on each slave try to do
> this AT THE SAME TIME and that causes too much stress on the masters.
> Effectively, the masters are DoSd by the slaves all re-syncing at the same
> time.
I understand the problem this is causing your system but I believe the
solution you are presenting is targeting the symptom and not the root
cause of the problem.
There is one event here that triggers a chain of two problems/symptoms:
* Event: connectivity loss between master and slave;
* Problem 1: slave needs full re-sync with master;
* Problem 2: N slaves doing this at the same time will cause a DoS on masters.
The solutions presented on this thread try to tackle Problem 2, how to
prevent the DoS of the master, and although it is a valid problem and
should be solved (I'm particularly fond of SLAVEOF host port ONCE
myself), it doesn't fix the initial problem: the need for a full
re-sync.
I would propose that, for each slave, a rotating AOF file should be
kept, based on time or size, with older files being removed when
slaves ACK back synchronization points reached.
For example, when a slave connects, it tells you what was the last
sync point it saw, and the master only has to send the AOF's since
that sync point. Every time the master rotates a slave AOF, it sends
the new name to the SLAVE. Every time a slave ACKs a specific AOF sync
point, all AOFs up-to that one can be removed (or archived, if your
business rules require that).
I'm sure that this simplistic approach has holes in it, I didn't
thought it out thoroughly yet, but my initial point still stands: you
are fixing a symptom, not the cause. It might be enough, and thats
fine, just pointing it out though :).
Best regards,
--
Pedro Melo
@pedromelo
http://www.simplicidade.org/
http://about.me/melo
xmpp:me...@simplicidade.org
mailto:me...@simplicidade.org
I agree with your analysis. Jeremy's proposal, while could be actually
useful to mitigate the problem, does not fix the root cause.
I also agree about incremental resync as a solution to many of this issues.
However I think the implementation of incremental resync should use
the implementation proposed here:
https://github.com/antirez/redis/issues/189
In short it uses the trick of still accumulating the output buffer of
the slaves for some time (or for some space) while the slave is not
connected. Moreover there is a sliding window so that we don't discard
the buffer sent to the slaves but take it for some time, since a slave
may want to resync from an offset that is already flushed on the
socket.
But back to the root cause for a moment, the problem is: "currently
Redis does not handle well the case when multiple slaves want sync at
once". I trust you about that, but I would understand why this
happens.
I mean, even without partial resync, Redis should handle that better.
Full resync should just be slower, but not a DoS.
Redis is already optimized to do a single BSAVE on reconnection of
multiple slaves, so what is actually DosSing it? Maybe the multiple
bulk transfers generate too much I/O and we should trottle this stuff?
Please if you have some information on this matter and how I can
reproduce it I would love to insert this fix into 2.6 if possible.
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
On Thu, Feb 23, 2012 at 12:28 PM, Salvatore Sanfilippo
<ant...@gmail.com> wrote:
> However I think the implementation of incremental resync should use
> the implementation proposed here:
>
> https://github.com/antirez/redis/issues/189
I knew I'd read something about partial resync, but forgot to search
the issues :)
I like it, although it would only survive small downtimes (which will
probably cover most of the situations, so no worries there), and I
really like the use of bytes written/read as a sync marker. Simple and
effective.
> In short it uses the trick of still accumulating the output buffer of
> the slaves for some time (or for some space) while the slave is not
> connected. Moreover there is a sliding window so that we don't discard
> the buffer sent to the slaves but take it for some time, since a slave
> may want to resync from an offset that is already flushed on the
> socket.
I don't know if you can use the same buffer for all clients, unless
you send the current byte count after the full dump.
> I mean, even without partial resync, Redis should handle that better.
> Full resync should just be slower, but not a DoS.
> Redis is already optimized to do a single BSAVE on reconnection of
> multiple slaves, so what is actually DosSing it? Maybe the multiple
> bulk transfers generate too much I/O and we should trottle this stuff?
>
> Please if you have some information on this matter and how I can
> reproduce it I would love to insert this fix into 2.6 if possible.
I assume this last two paragraphs are for Jeremy, since he is the one
with the problem.
Bye,
Hello Pedro,
I agree with your analysis. Jeremy's proposal, while could be actually
useful to mitigate the problem, does not fix the root cause.
I also agree about incremental resync as a solution to many of this issues.
However I think the implementation of incremental resync should use
the implementation proposed here:
https://github.com/antirez/redis/issues/189
In short it uses the trick of still accumulating the output buffer of
the slaves for some time (or for some space) while the slave is not
connected. Moreover there is a sliding window so that we don't discard
the buffer sent to the slaves but take it for some time, since a slave
may want to resync from an offset that is already flushed on the
socket.
But back to the root cause for a moment, the problem is: "currently
Redis does not handle well the case when multiple slaves want sync at
once". I trust you about that, but I would understand why this
happens.
I mean, even without partial resync, Redis should handle that better.
Full resync should just be slower, but not a DoS.
Redis is already optimized to do a single BSAVE on reconnection of
multiple slaves, so what is actually DosSing it? Maybe the multiple
bulk transfers generate too much I/O and we should trottle this stuff?
Please if you have some information on this matter and how I can
reproduce it I would love to insert this fix into 2.6 if possible.