question about presharding

[Jak Sprats]

First off nice blog, we have been discussing a poor-mans-cluster for a
while and it is nice to have an official recommendation. Also
presharding at app-level is the most scalable solution possible (most
of the time), it takes the burden off of the redis, puts it in the app
(i.e. very scalable). Its cool what you can do w/ lightweight
software, treat it like data almost (move it here and there simply),
very forward thinking, and not possible w/o lightweight code/
architecture. When I read the blog I thought of redis instances as a
swarm of bees, when they got too dense, make a new hive, move some
over, but in end effect they still stay bees, no bloating :)

One thing struck me while reading the blog and I am not sure here, but
I think there may be a consistency race condition in the migration
flow.

Here is an excerpt: (the HOWTO)
1.) Start the spare Redis instances in the new virtual machine.
2.) Set this instances are replicas for the old instances you want
to move.
3.) When the initial synchronization is done and all the slaves are
working, change the configuration of your clients to use the new
instances.
4.) Elect the new instances as masters with SLAVEOF NO ONE.
5.) Finally shut down all the old instances.
6.) Upgrade your shell scripts configs with the new IP/PORT info for
every instance.

In step3, in the middle of it, at the point where clients are cut over
from old to new instance, there will be some updates (e.g. RPOP pops
elementX) that have just happened at the old instance that have not
yet been replicated to the new instance. If before that update gets
replicated a read is done on the new master on the modified data (e.g.
RPOP pops elementX - should pop element [X - 1]), there will be some
consistency issues.

This is a known issue w/ async replication lag and non-idempotent
operations (e.g. Mysql w/ bin log, slave promotion).

Maybe I misunderstood the HOWTO, maybe I am just wrong, but my
developer warning light went off, thought I would post what is was
telling me

- Jak

[Tim Lossen]

jak,

your "developer warning light" is spot on. we tried the zero-downtime
approach when we switched datacenters -- admittedly from the states
to europe, so the replication lag was particularly bad in that case.
anyway, we still do not fully understand what went wrong, but we have
broken / inconsistent data left over from this exercise.

in hindsight, we now wish we had scheduled a half-hour of planned
downtime instead, which would have saved us tons of trouble.

tim

> --
> You received this message because you are subscribed to the Google
> Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en
> .
>

--
http://tim.lossen.de

[Jeremy Zawodny]

Speaking of Salvatore's presharding post, I got inspired to write up our sharding implementation and config a bit:

Redis Sharding at Craigslist

http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/

Jeremy

[Jak Sprats]

Hi Jeremy,

i read your blog, you have a knack for presenting information in a
straightforward manner ... anyways, it got me thinking on a somewhat
related tangent.

I am using your architecture, as this idea works only in the
distributed case (also I am writing this up quick, hopefully it
illustrates the point)
A.) you have 20 masters taking up 120GB of RAM for storage.
B.) similarly you have 20 slaves that also need 120GB of RAM.

Here is what I was thinking, and it is a bit hacky, but it does save
RAM, so maybe it is justified
1.) do a one time RDB dump on all 20 masters
2.) scp ALL 20 rdbs to replay_server
3.) change masters to run using AOF and set "appendfsync no" -
hopefully fast enough
4.) On each node: at hourly instances, mv the appendonly file to
hourX.aof
5.) the master will now be writing to a new aof file and hourX.aof
will be a delta
6.) scp (e.g. from box3) hourX.aof to the replay_server (file
box3_hourX.aof)
7.) start up a redis instance w/ rdb3 and box3_hourX.aof
8.) once the instance has started, do a SAVE (will save to rdb3,
box3_hourX.aof can be deleted)

Steps 7-8 can be done sequentially for each master (on the
replay_server). So the replay_server will be constantly replaying rdbs
and aofs from specific masters.

This gives you disk persistence w/o using 120GB RAM (replay_server
will use 6GB RAM [max-mem across masters]).

I could be off on the specifics (e.g. what happens if you "mv" the
appendonly file, simultaneous load of rdb & aof :), but the general
concept seems sound.

make sense?

- Jak

On Feb 26, 3:59 pm, Jeremy Zawodny <Jer...@Zawodny.com> wrote:
> Speaking of Salvatore's presharding post, I got inspired to write up our
> sharding implementation and config a bit:
>

> Redis Sharding at Craigslisthttp://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/

[Jeremy Zawodny]

Jak,

Before we deployed Redis for production use, I thought about a couple ideas that are very similar to what you describe.  Each one of them should have worked but would have required a development time, testing, and writing of extra tools for our sysadmins.

Ultimately, I went for the approach that was operationally simple and predictable.  But if I was really trying to economize on RAM, I would have implemented a sort of relay server with masters doing AOF.

I also considered a consolidated slave server which ran a redis-server slave instance for each production one, but used VM instead of keeping everything in RAM.  But I wasn't to thrilled with what I saw in the VM usage.  I suspect that diskstore may be a reasonable alternative here when it's stable.

Tangentially related, I'm planning to test diskstore on boxes with fusion-io SSD cards at some point.  I could be a great way to build *massive* redis boxes that far exceed RAM but don't sacrifice as much performance as you otherwise might.

Jeremy

[Jak Sprats]

Hi Jeremy,

devtime and operations complexity are not cheap at all.

sometimes i drop ideas to see if anyone has tried these things or
developed a framework for it, seems like a brilliant use of a spare
box, but again it only has benefits in the distributed case, so its
application is limited (could work good in the cloud too, spin up a
replication server once a day, do persistent backups, save money, not
paying for slaves ... for very relaxed persistence apps :).

On the Fusion-IO cards, that will be VERY interesting. The FusionIO
cards use a Btree (on-card) to buffer writes (all SSDs have a RAM
buffer for writes), what would be truly kick-ass is to be able to plug
diskstore into that btree directly, it would be a native software/
firmware solution on the best SSDs to date. If you are buying from
FusionIO ask them about this btree and if its possible to access it
via software, I have always wanted them to open their firmware Btree,
they have such a brilliant product, I want to develop directly on
it :)

- Jak

On Feb 26, 7:29 pm, Jeremy Zawodny <Jer...@Zawodny.com> wrote:
> Jak,
>
> Before we deployed Redis for production use, I thought about a couple ideas
> that are very similar to what you describe.  Each one of them should have
> worked but would have required a development time, testing, and writing of
> extra tools for our sysadmins.
>
> Ultimately, I went for the approach that was operationally simple and
> predictable.  But if I was really trying to economize on RAM, I would have
> implemented a sort of relay server with masters doing AOF.
>
> I also considered a consolidated slave server which ran a redis-server slave
> instance for each production one, but used VM instead of keeping everything
> in RAM.  But I wasn't to thrilled with what I saw in the VM usage.  I
> suspect that diskstore may be a reasonable alternative here when it's
> stable.
>
> Tangentially related, I'm planning to test diskstore on boxes with fusion-io
> SSD cards at some point.  I could be a great way to build *massive* redis
> boxes that far exceed RAM but don't sacrifice as much performance as you
> otherwise might.
>
> Jeremy
>

[Tim Lossen]

whoa, please stop -- you make me drool already ;)

On 2011-02-27, at 04:56 , Jak Sprats wrote:

> On the Fusion-IO cards, that will be VERY interesting. The FusionIO
> cards use a Btree (on-card) to buffer writes (all SSDs have a RAM
> buffer for writes), what would be truly kick-ass is to be able to plug
> diskstore into that btree directly, it would be a native software/
> firmware solution on the best SSDs to date.

> On Feb 26, 7:29 pm, Jeremy Zawodny <Jer...@Zawodny.com> wrote:

>> Tangentially related, I'm planning to test diskstore on boxes with fusion-io
>> SSD cards at some point. I could be a great way to build *massive* redis
>> boxes that far exceed RAM but don't sacrifice as much performance as you
>> otherwise might.

--
http://tim.lossen.de

[Salvatore Sanfilippo]

On Fri, Feb 25, 2011 at 10:08 PM, Jak Sprats <jaks...@gmail.com> wrote:
> In step3, in the middle of it, at the point where clients are cut over
> from old to new instance, there will be some updates (e.g. RPOP pops
> elementX) that have just happened at the old instance that have not
> yet been replicated to the new instance. If before that update gets
> replicated a read is done on the new master on the modified data (e.g.
> RPOP pops elementX - should pop element [X - 1]), there will be some
> consistency issues.

Ok let's make your scenario more concrete:

master1 list => a b c
slave1 list => a b c

client 1: RPOP gainst master1, get element 'c'
master1 list => a b
slave1 list => a b c (operation yet not replicated)

client 2, with updated config: RPOP against slave1 list, get element 'c'
slave1 list => a b
previos RPOP gets replicated, slave 1 list => a

'c' was popped two times, while 'b' is lost.

So you are right, this can happen. The window of exposure is tiny if
the two Redis instances are "near" (low latency), but as there is more
latency the risk is increased (replication over internet for
instance).

However how to avoid this problem? The trivial fix is to stop the
application for a time that is at least equivalent to the replicaiton
latency, so one should stop the application, change configuration,
restart the app again.
The replication latency is usually sub millisecond, so well, just a
stop/start is enough.
If it's over internet, it's better to wait a few seconds before
restarting the app again.

An interesting alternative that can fix this problems is however to
support a command that is able to block all the new connections. so
for instance I can do:

1) Create a slave, and wait for the synchronization.
2) Issue the command BLOCKNC (block new clients) against the slave.
The replication will work as usually, but new connections will be just
accepted without being processed (we just unregister the listening
socket read handler).
3) Change the configuration of the clients to use the slave.
4) wait two seconds. The final replication data will go from master to
slave. Everything is now in sync.
5) Issue UNBLOCKNC against the slave.

Sounds good? Better ideas?

Thanks for your contribution Jak, that was very useful.

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Didier Spezia]

Hi,

one problem is some large applications are very slow to start and
stop. Here, we routinely have applications whose stop+start time is
about 10 min per node (and some applications run on 200 nodes). So
you can imagine a complete outage of this application is expensive ...

Rather than just blocking the incoming connections on the slave,
I would much prefer a flexible quiesce command to alter the behavior
of Redis (on the master, and possibly on the slave as well).
Something like:

QUIESCE [START level mode|STOP]
with:
level - 0 means incoming connections
1 means write/update operations
2 means all operations (even reads) except admin ones
mode - 0 means quiesced operations are blocked
1 means quiesced operations return an error without blocking

Beyond the migration issue, I think it could be useful in various
situations, because it gives more options to operation people.
For instance, there is already a feature request to have a read-only
mode. It could be covered by such a command as well.

Regards,
Didier.


On Feb 27, 1:57 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:

[Jak Sprats]

Hi Salvatore,

The alternative is a good one:

1) Create a slave, and wait for the synchronization.
2) Issue the command BLOCKNC (block new clients) against the
slave.
The replication will work as usually, but new connections will be
just
accepted without being processed (we just unregister the listening
socket read handler).
3) Change the configuration of the clients to use the slave.
4) wait two seconds. The final replication data will go from
master to
slave. Everything is now in sync.
5) Issue UNBLOCKNC against the slave.

Steps 3 & 4 can be automated w/ the following change
3) issue FLUSH command against the master (block new clients then
flush replications streams then shutdown)
4) Change the configuration of the clients to use the slave.

This would effectively be a distributed mutex:
(lock, do-something, unlock)
(BLOCKNC, FLUSH, UNBLOCKNC)
which is the best way to guarantee behavior.

This should open up the possibilities for a poor-mans-cluster.

The only down side to this is that mutexes lock, but replication
flushing is fast, and if app-servers are smart enough to handle
switching from master to new-master gracefully, it should result in a
few millisecond hiccup.

- Jak

On Feb 27, 4:57 am, Salvatore Sanfilippo <anti...@gmail.com> wrote:

[Jak Sprats]

Hi All,

something that dawned on me revisiting the whole BLOCKNC UNBLOCKNC
cutover for the poor-mans cluster.

If this cutover were in place, it would also be an effective means of
fighting any instance that (for whatever reason) suffers from high
memory fragmentation OVER TIME.

If one were to cut a redis instance over to a new instance (running on
the same machine), the new instance would have all data very recently
malloc'ed and the probability for memory fragmentation should be
lower.

This would also push clients to follow the "fail often" strategy that
lots of cluster people advocate. "Fail often" is a great way to test
replication/failover during non-crisis times.

This single-box-cutover would require 2X RAM, and is kind of like a
windows-reboot solution, but it should have some positive effect on
memory fragmentation, which is a very difficult problem to tackle.

- Jak

p.s. on the topic of memory fragmentation, there are facebook
presentations about jemalloc, claiming it stands up when malloc and
tcmalloc fall down (due to memory fragmentation) ... has anyone
compiled/run/tested w/ jemalloc?

On Feb 27, 2:59 pm, Jak Sprats <jakspr...@gmail.com> wrote:
> Hi Salvatore,
>
> The alternative is a good one:
>     1) Create a slave, and wait for the synchronization.
>     2) Issue the command BLOCKNC (block new clients) against the
> slave.

>     Thereplicationwill work as usually, but new connections will be

> just
> accepted without being processed (we just unregister the listening
> socket read handler).
>     3) Change the configuration of the clients to use the slave.

>     4) wait two seconds. The finalreplicationdata will go from

> > latency the risk is increased (replicationover internet for

> > instance).
>
> > However how to avoid this problem? The trivial fix is to stop the
> > application for a time that is at least equivalent to the replicaiton
> > latency, so one should stop the application, change configuration,
> > restart the app again.

> > Thereplicationlatency is usually sub millisecond, so well, just a

> > stop/start is enough.
> > If it's over internet, it's better to wait a few seconds before
> > restarting the app again.
>
> > An interesting alternative that can fix this problems is however to
> > support a command that is able to block all the new connections. so
> > for instance I can do:
>
> > 1) Create a slave, and wait for the synchronization.
> > 2) Issue the command BLOCKNC (block new clients) against the slave.

> > Thereplicationwill work as usually, but new connections will be just

> > accepted without being processed (we just unregister the listening
> > socket read handler).
> > 3) Change the configuration of the clients to use the slave.

> > 4) wait two seconds. The finalreplicationdata will go from master to