Re: Thoughts on Large Redis Instances

[Jeremy Zawodny]

The approach we take (at craigslist) on our 72GB boxes is to run multiple smaller redis procs.  That makes the persistence issues easier to deal with and also allows us to take advantage of multiple cores.  You do lose the ability to use some commands since you don't necessarily know if the data you want to query (say in an intersection) will be on the same nodes.  But overall it has worked quite well for us.

http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/

http://news.ycombinator.com/item?id=2266632

Jeremy

On Tue, Jan 22, 2013 at 8:43 AM, Stephen O'Donnell <stephen....@gmail.com> wrote:

Guys,

I have been doing a bit of a evaluation of Redis for my own interests and to see if it would be good for a few applications where I work. Right now we use Oracle for just about everything.

If I was to use Redis, I would be looking at a fairly large dataset. Would it be generally advised to run Redis as many small shards, or one big (say 64GB+) instance?

What problems have people run into with large Redis instances?

One thing that worries me about a busy large instance is persistence. I have read about the AOF and RDB files. If I had a 64GB data set and a lot of changes, I don't think RDB files are going to give me the data persistence I would require. The AOF is going to get big fast, which will affect startup times, and I get the feeling that frequent rebuilds on a database of that size are going to be slow.

The other thing that worries me about large instances is creating slaves. Pushing 64GB of data to a slave is going to take a long time, and then if the slave goes down it will take a long time to recover too.

I also noted at comment in an earlier thread titled 'Durability' that stated:

> If you decide to go the Redis route, please only use it
> for data that you can lose (or recover from a primary data source).

Do I take it from this that it is seem as best practice to use Redis as a semi-permeanant data store, but ensure the data is persisted elsewhere incase it needs to be recovered in the future?

Thanks,

Stephen.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/1GSNWriPxjAJ.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Josiah Carlson]

On Tue, Jan 22, 2013 at 8:43 AM, Stephen O'Donnell
<stephen....@gmail.com> wrote:
> Guys,
>
> I have been doing a bit of a evaluation of Redis for my own interests and to
> see if it would be good for a few applications where I work. Right now we
> use Oracle for just about everything.
>
> If I was to use Redis, I would be looking at a fairly large dataset. Would
> it be generally advised to run Redis as many small shards, or one big (say
> 64GB+) instance?

Redis really only uses 1-2 cores (the 2nd core during snapshot and/or
aof rewriting), so if you've got a really big machine, you're going to
be under-utilizing your processors. You can run multiple instances on
a single machine, or multiple instances on multiple machines. Also,
because of NUMA, running a single Redis instance on huge memory won't
be as fast as running smaller instances assigned to certain cores, or
running Redis on multiple machines.

> What problems have people run into with large Redis instances?

Most people run into problems where the snapshot and/or AOF gets to be
very large, creating a new snapshot or rewriting an old AOF takes a
long time, and loading during restart takes a long time. Multiple
smaller instances mitigate this generally through rotating
snapshots/rewrites (or having multiple machines, thus multiple disks).
As a point of data, when I was running a ~50 gig Redis instance in
EC2, snapshotting took roughly 20 minutes with BGSAVE (fork + write),
but only 3 minutes with SAVE (blocking all other operations). To deal
with this, we paused the world at 3AM every night for snapshot +
backup. Also, starting that 50 gig instance from scratch took roughly
20 minutes.

> One thing that worries me about a busy large instance is persistence. I have
> read about the AOF and RDB files. If I had a 64GB data set and a lot of
> changes, I don't think RDB files are going to give me the data persistence I
> would require. The AOF is going to get big fast, which will affect startup
> times, and I get the feeling that frequent rebuilds on a database of that
> size are going to be slow.

Yes and no. Depending on your data, you can expect on-disk RDB files
to be roughly 10-20% of your in-memory usage, and freshly rewritten
AOFs to be roughly the same size as a fresh RDB. If you've got enough
disk IO, you should be able to write the RDB and/or AOF reasonably
quickly. But you will still run into issues like I already mentioned.

> The other thing that worries me about large instances is creating slaves.
> Pushing 64GB of data to a slave is going to take a long time, and then if
> the slave goes down it will take a long time to recover too.

This is really going to be the killer for you. Even at a 1:10 ratio of
on-disk vs. in-memory storage, slaving will still be moving 6
gigabytes. At gigabit ethernet speeds, that's 48 seconds to transfer
the data, but still probably 10-20 minutes just to load the data from
the snapshot. In that time, your master will probably be collecting
millions of write commands to send to the slave, and may disconnect
the slave or crash after running out of memory, neither of which you
want.

> I also noted at comment in an earlier thread titled 'Durability' that
> stated:
>
>> If you decide to go the Redis route, please only use it
>> for data that you can lose (or recover from a primary data source).
>
> Do I take it from this that it is seem as best practice to use Redis as a
> semi-permeanant data store, but ensure the data is persisted elsewhere
> incase it needs to be recovered in the future?

It depends on your needs. The aforementioned 50 gig Redis instance I
mentioned earlier was a Twitter Analytics database. Everything was
stored there (including some task queues for subsequent fetches). We
did the daily snapshot and backup because we could recover 24 hours of
data in 8 hours by re-fetching from Twitter. And in the worst-case, if
we were to lose the entire database, we would lose some historical
data, but could regenerate "current" data within 7 days running at
full tilt.

I said what I said because financial data (of the kind that Square
has) is important. More important than ego. More important than "look
at this awesome stuff that people are using Redis for." If your data
is important enough not to lose, you can mitigate risks with sharding,
slaving, AOF, and backed up snapshots. But risk mitigation only goes
so far when you're talking about financial data. And the last thing
that I (and I hope anyone else here) wants is for someone making a
mistake, losing data stored in Redis, and blaming Redis for not
holding their hand enough to prevent that loss.

Redis is this crazy blend between a cache, a database, remote data
structure API, and an API for data modeling with your application. It
does things that no other database does. But it also does things that
others don't do. Understanding the power, and limitations, aren't
necessarily straightforward. Which is why I advise being conservative.

Regards,
- Josiah

[CharSyam]

As already jeremy and Josiah said, there are some points you should consider.

Generally,

1. multiple instances are fast than one large instance.(redis just uses one core except, bgsave)

2. large memory takes large time to save rdb

3. you need more efforts to run multiple servers comparing to one large instance.

4. and if you use multiple redis instance on muliple ec2 server

 like jeremy, cross master/slave node to another server. It can give more robust

 against failures.

 

2013/1/22 Josiah Carlson <josiah....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

[Stephen O'Donnell]

Most people run into problems where the snapshot and/or AOF gets to be
very large, creating a new snapshot or rewriting an old AOF takes a
long time, and loading during restart takes a long time. Multiple
smaller instances mitigate this generally through rotating
snapshots/rewrites (or having multiple machines, thus multiple disks).
As a point of data, when I was running a ~50 gig Redis instance in
EC2, snapshotting took roughly 20 minutes with BGSAVE (fork + write),
but only 3 minutes with SAVE (blocking all other operations). To deal
with this, we paused the world at 3AM every night for snapshot +
backup. Also, starting that 50 gig instance from scratch took roughly
20 minutes.

This pretty much lines up with some testing I was doing. On a dedicated machine I have access to, (24GB 8 core Xenon 3.57Ghz) it took about 1 minute to load a 5.5GB instance from disk, so I would be looking at 10+ minutes to startup a 50GB one. Its highly likely the disk on this machine is faster than EC2, but I am only guessing.
 

> The other thing that worries me about large instances is creating slaves.
> Pushing 64GB of data to a slave is going to take a long time, and then if
> the slave goes down it will take a long time to recover too.

This is really going to be the killer for you. Even at a 1:10 ratio of
on-disk vs. in-memory storage, slaving will still be moving 6
gigabytes. At gigabit ethernet speeds, that's 48 seconds to transfer
the data, but still probably 10-20 minutes just to load the data from
the snapshot. In that time, your master will probably be collecting
millions of write commands to send to the slave, and may disconnect
the slave or crash after running out of memory, neither of which you
want.

I found that out during my testing. I guess you need to size the buffer (and leave enough spare RAM) for you incoming data per second * the number of seconds to bring up the slave, which could be quite a lot of data.
 

> I also noted at comment in an earlier thread titled 'Durability' that
> stated:
>
>> If you decide to go the Redis route, please only use it
>> for data that you can lose (or recover from a primary data source).
>
> Do I take it from this that it is seem as best practice to use Redis as a
> semi-permeanant data store, but ensure the data is persisted elsewhere
> incase it needs to be recovered in the future?

It depends on your needs. The aforementioned 50 gig Redis instance I
mentioned earlier was a Twitter Analytics database. Everything was
stored there (including some task queues for subsequent fetches). We
did the daily snapshot and backup because we could recover 24 hours of
data in 8 hours by re-fetching from Twitter. And in the worst-case, if
we were to lose the entire database, we would lose some historical
data, but could regenerate "current" data within 7 days running at
full tilt.

I said what I said because financial data (of the kind that Square
has) is important. More important than ego. More important than "look
at this awesome stuff that people are using Redis for." If your data
is important enough not to lose, you can mitigate risks with sharding,
slaving, AOF, and backed up snapshots. But risk mitigation only goes
so far when you're talking about financial data. And the last thing
that I (and I hope anyone else here) wants is for someone making a
mistake, losing data stored in Redis, and blaming Redis for not
holding their hand enough to prevent that loss.

Some of the data in the company I work for certainly falls into must-never-lose it category. My thoughts on using Redis are for an area that is important, but the data could all be rebuilt from the source in the event of a disaster. That would still cause unacceptable downtime, but I have seen some pretty good down time caused by Oracle failures / miss-configurations over the years too.
 

Redis is this crazy blend between a cache, a database, remote data
structure API, and an API for data modeling with your application. It
does things that no other database does. But it also does things that
others don't do. Understanding the power, and limitations, aren't
necessarily straightforward. Which is why I advise being conservative.

Thanks for the info. This post / thread was very informative, and kind of backed up many of my own thoughts from reading around on Redis. Sounds like many smaller instances with application sharding is the way to go, at least until Redis cluster comes out.
 

Regards,
 - Josiah

[Stephen O'Donnell]

On Tuesday, January 22, 2013 6:09:18 PM UTC, Jeremy Zawodny wrote:

The approach we take (at craigslist) on our 72GB boxes is to run multiple smaller redis procs.  That makes the persistence issues easier to deal with and also allows us to take advantage of multiple cores.  You do lose the ability to use some commands since you don't necessarily know if the data you want to query (say in an intersection) will be on the same nodes.  But overall it has worked quite well for us.

http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/

http://news.ycombinator.com/item?id=2266632

Jeremy

 Thanks for sharing this. Your blog post was very interesting.

[Yiftach Shoolman]

My 2 cents:

• Disk I/O bottlenecks is mostly seen during AOF rewrite and BGSave. We tested AOF w/o rewrite (every 1sec) on AWS in so many scenarios and didn't find much differences versus pure in-memory (more details here) 
• The good thing about AOF rewrite is that you can control when to trigger it and where to trigger it. I.e. trigger AOF rewrite on your slave and not too often can solve a lot of your performance issues
• +1 to all the other tips that were given here. Sharding/Clustering is the right approach to go with large Redis datasets 

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/l0GEwfAVqRQJ.

To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Yiftach Shoolman
+972-54-7634621