real life usage discussion, say disqus was to use redis, how would they use it?

[S Ahmed]

I want to learn or ask you guys how a real world usage of redis might be, so I thought i'd pick a company that most people know of.

disqus http://www.disqus.com

So they provide blog commenting, so let's take this page as an example:

http://techcrunch.com/2010/12/17/spin-this-yahoo/

So the data model is probably like:

Table Websites:

websiteID

domain

Table Thread:

websiteID

threadID

url

Table Posts:

threadID

postID

So it figures out the website by the domain, and it looks up the request url to know the thread (i.e. the thread will contain all comments for the blog post).

That was just background, not relavent too much :)

Now, the comments can be ordered by:

rating (integer)

newest (datetime)

oldest (datetime)

Now this is a read heavy, so there must be 100 reads before a write let's say.

1.  Would you store JUST the id's in redis, or the entire comment or ALL the comments?

     Or maybe store:  id, rating, timestamp

2.  How would you be able to sort by rating, newest or oldest?

3. In light of the above questions, when a new comment is posted, how would you go about re-generating the cache and/or expiring other caches?

[David Cramer]

This is how we use Redis:

http://disqus.com/analytics/

You probably wouldn't use Redis from something like Disqus (its not a
one-size fits all after all).

On Dec 18, 9:32 am, S Ahmed <sahmed1...@gmail.com> wrote:
> I want to learn or ask you guys how a real world usage of redis might be, so
> I thought i'd pick a company that most people know of.
>

> disqushttp://www.disqus.com

[dz]

Just a note that though we're not using Redis as our primary comments
datastore, we are using it for many things :)

-- dz from disqus

On Dec 18, 9:32 am, S Ahmed <sahmed1...@gmail.com> wrote:

> I want to learn or ask you guys how a real world usage of redis might be, so
> I thought i'd pick a company that most people know of.
>

> disqushttp://www.disqus.com

[S Ahmed]

From what I understood, you guys use memcache to store the comment tree etc,, couldn't redis be used for the exact same usage scenerio?

Any other comments?   redis is not ideal for the scenerio I outlined?  maybe someone can tell me why so I can understand? :)

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[David Cramer]

We use memcache to cache the results of the comment tree. If you were going to use Redis you'd just be duplicating what you'd do in Postgres without the relational support.

So let me give a brief rundown on our history with the comment tree.

- Nested sets/intervals were too write heavy, but they would work great in Redis or Postgres.

- Pulling in the entire tree (which is one method we employ currently, as does Reddit and others) would work great in Redis and Postgres.

- The most efficient way we've found to pull in the tree (in a limited scope, ordered) is by using arrays in Postgres to store sortable column values. This *could* be achieved in Redis by flattening the array to a floating point number so its sortable, but is difficult to achieve. This is the reason we're using the RDBMS for storing the tree.

All in all, if we couldn't find a way to store the comments in a highly sortable fashion in Redis, we couldn't find any reason to store the data in Redis.

We however are using Redis entirely for our analytics datastore. This process begins with buffering data from a various data sources (such as a queue and Hadoop) into a Python collector. The collector then partitions the data based on our forum cluster, and keys it based on a time interval. For something like this, other than some odd tuning issues that we've seen (and the fact that we have to partition manually), Redis has worked very well.

-- 
David Cramer

[Salvatore Sanfilippo]

On Wed, Dec 22, 2010 at 5:42 PM, S Ahmed <sahme...@gmail.com> wrote:

> Any other comments?   redis is not ideal for the scenerio I outlined?  maybe
> someone can tell me why so I can understand? :)

For sure Redis *can* be used to implement something like Disqus, but
can does not mean must have.
Depends on many parameters. Is a comment system DB-bound? Or given
that you can cache the whole stuff until there is no a new comment
added this is a non issue so it's better to use another on disk store
that is perfect for many reads and seldon writes?

What is the amount of comments?

And so forth. But for sure you can use Redis efficiently for such a task.
I could do this: store all the comments for a given post as a single
hash if the total messages are <= a given threshold (the majority).
This is very space efficient in Redis.

Switch to a different graph-alike representation just for uncommonly
long comments.

Five minutes idea. People that design this systems for work know better :)

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Salvatore Sanfilippo]

On Wed, Dec 22, 2010 at 11:03 PM, Salvatore Sanfilippo
<ant...@gmail.com> wrote:

> Switch to a different graph-alike representation just for uncommonly
> long comments.

I mean uncommonly long threads, not comments. Sorry for the mistake.

Cheers,
Salvatore

[Salvatore Sanfilippo]

Thanks David,

do you mind if I add Disqus as "testimonial" in the new Redis site ;)
Linking the Disqus logo to the analytics page in this page:
http://redis.io/topics/whos-using-redis

Cheers,
Salvatore

--

[David Cramer]

Sure that sounds great :)

-- 
David Cramer

[Salvatore Sanfilippo]

On Wed, Dec 22, 2010 at 11:08 PM, David Cramer <dcr...@gmail.com> wrote:
> Sure that sounds great :)

Cool thank you :)

Salvatore

[David Zhou]

To elaborate on Analytics, we're making heavy use of set unions and zset unions.  Nearly any metric we measure can be expressed as a bunch of time-series based set or zset unions.

-- dz@disqus

[S Ahmed]

Hold your horses folks!

I by no means meant to actually store all the data into redis, I was just referring to certain things like the comment tree (just the id's and nesting information) or something simliar to that.

David mentioned:

>>>We use memcache to cache the results of the comment tree.

What is the difference between storing this in memcache over redis?  In this specific usage you are currently using memcache for, could redis just as easily be used or there is a reason not too?

[David Cramer]

The difference is only the difference in the software. Why use Redis if Memcache solves your problems? Memcache is faster in these general operations, has been around longer, and the libraries have been very tested.

(Not saying Redis isn't great, it's just new)

-- 
David Cramer

[Salvatore Sanfilippo]

On Thu, Dec 23, 2010 at 1:11 AM, David Cramer <dcr...@gmail.com> wrote:
> The difference is only the difference in the software. Why use Redis if
> Memcache solves your problems? Memcache is faster in these general
> operations, has been around longer, and the libraries have been very tested.

I completely agree for being conservative and don't do switches if
it's not worth it. Also memcached does very well what it advertises
doing.
But I think there is a reason for a big percentage of memcached users
to consider switching to Redis, that is, you no longer cache and
invalidate, but cache and *modify* your old value as the DB side
changes, without the need of recreating it from scratch. So for
caching HTML fragments this is not an issue, but for caching
everything more complex, like objects (use Hashes), timelines (use
Lists), and so forth, Redis can completely shift the paradigm, and
remove a lot of load from the DB. Persistence is another interesting
feature for many users using Redis just like a cache. On the other
side memcached benefits from being more specialized, for instance it
uses less data for caching very small strings, from what I saw in some
test I did time ago.

Cheers,

[Demis Bellot]

Well put Salvatore.

If all you will ever need is a dumb fragment caching and you have memcached there is no need to switch (Although we originally switched from memcached because of the db partitioning and intelligent caching made possible with SortedSets).

If however you want your data persisted in intelligent comp-sci like structures or plan to utilize the messaging features from a number of different client bindings - Redis is the way to go!

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
- Demis

http://twitter.com/demisbellot

http://www.servicestack.net/mythz_blog