Re: Production plan

[Yiftach Shoolman]

Hi Raz,

My 2 cents:

1.  Practically what you want to do is sharding, which is a valid approach. If you haven't, please read this nice blog from Jeremy on Redis Sharding at Craigslist

2. Dividing your dataset to 2 main datastores is a good idea. For your large structures instances, make sure your app knows how to deal with union/intersect commands across Redis instances

3. I guess you are aware that due to COW and AOF rewrites, you will never be able to fully utilize your 68.4GB memory of the EC2 m2.4xlarge instances

Yiftach 

On Thu, Jul 19, 2012 at 10:41 AM, raz lachyani <r...@vtaggo.com> wrote:

Hi Guy,

I decided to make redis my main data store and your opinion is very important to me.

I build this decision on the fact that soon the Redis sentinel will be ready.

I divided my redis into two instances types : one for small structures and the other for large ones (like sorted set of all the users in my system).

I want to start with 10 small structures instances and 4 large structures instances (that is the number that will fit my needs) + 14 slaves (one for each master) on one machine. When my users number will grow I will move each instance to its own machine. So my potential storage (without re sharding) is 14x68.4G on AWS EC2.

In addition I will send all the data to MySQL in async manner for convenient offline queries and use mongoDB to log events.

What do you think ? Am I in the right direction ? Is there anything im missing here ? am I and YouPorn are the only ones that use Redis as their main Data Store ?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/mBc1ctarO28J.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
Yiftach Shoolman
+972-54-7634621

[Josiah Carlson]

As you are going down this path, understand the one truth: machines
crash and disks disappear (especially in EC2 with ephemeral OR EBS
volumes). Have a backup strategy, and remember that it is possible
that you could lose all of the changed data since your last backup. If
your system cannot deal with this kind of data loss, you shouldn't be
using Redis. Redis cluster can somewhat mitigate this (with multiple
slaves per master), but there isn't a lot that can be done if/when an
entire availability zone disappears due to some catastrophic event
(which occurs roughly once every 6-9 months, according to my memory).

In the past, I have used Redis as the only storage for a large
collection of data. Daily at 3AM local time, the system would pause,
dump to disk, backup to S3, then continue. We did this because we
could recover from 24 hours of lost data in about 4-6 hours. We could
also seed the system from the database, and build up 95% of our data
in 3-5 days (the 5% lost would have been nice to have, but wasn't
crucial).


Also, MongoDB is not magic. Claims to the contrary are demonstrably
false. You can tune Postgres or MySQL to perform as well as (if not
better than) MongoDB with the same data reliability. For logging, I'd
actually just recommend syslog-ng + backing up your logs to S3. If you
ever need to do queries against your logs, you can either perform a
mapreduce over them, or just download and process them offline.
Syslog-ng will be able to handle millions of logs per second without
breaking a sweat, where Mongo will be having serious issues with just
a few hundred (unless you configure Mongo in such a way that you don't
care about your data, and your Mongo loggers will be backed up waiting
for Mongo to catch up). Even worse, unless you have indexes (which
slows writes even more), your queries will grind your log insertion to
a halt on Mongo. If you really need multi-machine logging and
arbitrary querying; look at Riak. It's better than Mongo at just about
everything.

Regards,
- Josiah