Youporn.com is now a 100% Redis Site

[Eric Pickup]

I just want to let the devs know that Youporn.com relaunched two weeks
ago with Redis as our primary database. With @100 million page views
per day, our cluster of Redis slaves are handling over 300k queries
per second.

After the switchover we had to add some additional Redis nodes but not
because Redis was overworked but because the network cards couldn't
keep up with Redis.

Everyone involved has been more than impressed with Redis'
performance.

All reads come from Redis; we are maintaining MySQL just to allow us
to build new sorted sets as our requirement change.

I'm going to be doing the keynote a major Canadian tech conference
(confoo.ca) about the youporn.com rewrite and one of the major topics
will be Redis and how much it helped us.

Do you guys have a logo that would look good on a white background? I
want to make you guys stand out.

Thanks for the great software!

~Eric

[Dvir Volk]

that's very cool to hear!

you can find the logo here:

https://github.com/antirez/redis-io/raw/master/public/images/redis-300dpi.png

~Eric

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Dvir Volk

System Architect, The Everything Project (formerly DoAT)

http://everything.me

[cyx]

Hi Eric,

That's great news! I'd love to learn about some of the details if
you're willing
to share:

1. Are you using AOF or RDB (or both)?
2. What redis version are you using?
3. Did you have to change any OS specific configuration (like the
maximum number of open files?)

Thanks in advance!

cyx

[Salvatore Sanfilippo]

On Thu, Feb 16, 2012 at 9:51 AM, Dvir Volk <dv...@doat.com> wrote:
> that's very cool to hear!
> you can find the logo here:
> https://github.com/antirez/redis-io/raw/master/public/images/redis-300dpi.png

Thanks Dvir, I just added an SVG logo as well:
http://redis.io/images/redis-logo.svg

Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Salvatore Sanfilippo]

Hello Eric, these are serious numbers indeed! Very cool.

I would love to see the slides after your talk, don't want you to
reveal too much things before the event ;) But I'm super curious about
what kind of operatons you use!

Thanks for sharing, if you happen to have issues with Redis we are here to help.

[Salvatore Sanfilippo]

Eric maybe I found a bug with the new implementation: apparently in
the category page the first page is always empty, with just the four
videos people are watching right now, but without the big listing.
From page 2 everything starts to look ok again.
Maybe related to the parameters passed to ZRANGE?

Cheers,
Salvatore

On Thu, Feb 16, 2012 at 2:57 AM, Eric Pickup <eric....@manwin.com> wrote:

[Gerhard Lazu]

This is very impressive indeed! Is it just replication to multiple
slaves and then reads spread across those or is it something more
involved. Would love to find out more about your setup.

[Tim Lossen]

cool! -- although i currently get this:

"Error 310 (net::ERR_TOO_MANY_REDIRECTS): There were too many redirects."

surely not related to redis ;)

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>

--
http://tim.lossen.de

[Haoran Chen]

Amazing! Can't wait for your slide.
How many nodes are you using? As I know, weibo.com use 200+ nodes to
cache.

[Eric Pickup]

Thanks for the pointer. Someone introduced a logic bug that didn't show up right away. It was creating an invalid offset for the ZRANGE as you suspected.

Fixed.

Eric Pickup
YouPorn Lead Developer
Manwin Canada
Office: +1.514.359.3555
Fax: +1.514.359.3556
Email: eric....@manwin.com

Cheers,
Salvatore

This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courrier électronique est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courrier électronique par erreur, veuillez m'en aviser immédiatement, par retour de courrier électronique ou par un autre moyen.

[Salvatore Sanfilippo]

On Thu, Feb 16, 2012 at 4:03 PM, Eric Pickup <Eric....@manwin.com> wrote:

> Thanks for the pointer.  Someone introduced a logic bug that didn't show up right away.  It was creating an invalid offset for the ZRANGE as you suspected.

Yes! I helped fixing youporn.com :)

Salvatore

[Dvir Volk]

redis is the pornstar of databases :)

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

[Arek Bochinski]

This thread is a mighty good read ;)

[Andrea Campi]

On Feb 16, 2012, at 10:06 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

> On Thu, Feb 16, 2012 at 4:03 PM, Eric Pickup <Eric....@manwin.com> wrote:
>
>> Thanks for the pointer. Someone introduced a logic bug that didn't show up right away. It was creating an invalid offset for the ZRANGE as you suspected.
>
> Yes! I helped fixing youporn.com :)

You should have that printed on a T-Shirt :D

>
> Salvatore
>
> --
> Salvatore 'antirez' Sanfilippo
> open source developer - VMware
>
> http://invece.org
> "We are what we repeatedly do. Excellence, therefore, is not an act,
> but a habit." -- Aristotele
>

[Timothy Broder]

Awesome Eric, congrats on the rewrite!

[Eric Pickup]

Thanks for the logos. I'll be posting the slides online after the talk and will be working on some articles.

The talk won't cover much of anything in depth. It is only an hour long and will serve as a high-level overview of all techs used (strengths, weaknesses, etc). I'll be doing future talks and articles about the specific techs used.

Most of our data itself is found in hashes with ordered sets used to know what data to show. A typical lookup would be an zInterStore on:

videos:filters:released
Videos:filters:orientation:straight
Videos:filters:categories:{category_id}
Videos:ordering:rating

We would then do a zRange to get the pages we want and get the list of video_ids back. We then start a pipeline and get all the videos from hashes.

We do use some key/value lookups and some lists but the majority of our operations are using the above pattern.

Eric Pickup
YouPorn Lead Developer
Manwin Canada
Office: +1.514.359.3555
Fax: +1.514.359.3556
Email: eric....@manwin.com

-----Original Message-----
From: redi...@googlegroups.com [mailto:redi...@googlegroups.com] On Behalf Of Salvatore Sanfilippo
Sent: February-16-12 4:01 AM
To: redi...@googlegroups.com
Subject: Re: Youporn.com is now a 100% Redis Site

This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. Ce courrier électronique est confidentiel et protégé. L'expéditeur ne renonce pas aux droits et obligations qui s'y rapportent. Toute diffusion, utilisation ou copie de ce message ou des renseignements qu'il contient par une personne autre que le (les) destinataire(s) désigné(s) est interdite. Si vous recevez ce courrier électronique par erreur, veuillez m'en aviser immédiatement, par retour de courrier électronique ou par un autre moyen.

[hr]

On Feb 16, 11:06 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:

> On Thu, Feb 16, 2012 at 4:03 PM, Eric Pickup <Eric.Pic...@manwin.com> wrote:
> > Thanks for the pointer.  Someone introduced a logic bug that didn't show up right away.  It was creating an invalid offset for the ZRANGE as you suspected.
>
> Yes! I helped fixing youporn.com :)

But I can't see youporn on the "who's using redis" page!!!

[MC Andre]

I vectorized the Redis logo so you can use any size you want.

http://www.yellosoft.us/logos

[Sebastian Stadil]

Congrats on the success!

I was calculating the number of queries you make per pageview, and get
the following:

Queries per sec * seconds in day / pageviews in day = 300k * (3600 *
24) / 100M = 260.

This seems to be on the high side, so I'm guessing the 300k queries/
sec is peak?

[emiretsk]

Hi Eric,

I would imagine that your sets are pretty long. How long does the
intersect take?

Regards,
Eugene

On Feb 16, 1:04 pm, Eric Pickup <Eric.Pic...@manwin.com> wrote:
> Thanks for the logos.  I'll be posting the slides online after the talk and will be working on some articles.
>
> The talk won't cover much of anything in depth.  It is only an hour long and will serve as a high-level overview of all techs used (strengths, weaknesses, etc).  I'll be doing future talks and articles about the specific techs used.
>
> Most of our data itself is found in hashes with ordered sets used to know what data to show.   A typical lookup would be an zInterStore on:
>
> videos:filters:released
> Videos:filters:orientation:straight
> Videos:filters:categories:{category_id}
> Videos:ordering:rating
>
> We would then do a zRange to get the pages we want and get the list of video_ids back.  We then start a pipeline and get all the videos from hashes.
>
> We do use some key/value lookups and some lists but the majority of our operations are using the above pattern.
>

> Eric PickupYouPornLead Developer

> Manwin Canada
> Office: +1.514.359.3555
> Fax: +1.514.359.3556

> Email: eric.pic...@manwin.com

>
>
>
>
>
>
>
> -----Original Message-----
> From: redi...@googlegroups.com [mailto:redi...@googlegroups.com] On Behalf Of Salvatore Sanfilippo
> Sent: February-16-12 4:01 AM
> To: redi...@googlegroups.com
> Subject: Re:Youporn.com is now a 100% Redis Site
>
> Hello Eric, these are serious numbers indeed! Very cool.
>
> I would love to see the slides after your talk, don't want you to reveal too much things before the event ;) But I'm super curious about what kind of operatons you use!
>
> Thanks for sharing, if you happen to have issues with Redis we are here to help.
>

> On Thu, Feb 16, 2012 at 2:57 AM, Eric Pickup <eric.pic...@manwin.com> wrote:
> > I just want to let the devs know thatYouporn.com relaunched two weeks

> > ago with Redis as our primary database.  With @100 million page views
> > per day, our cluster of Redis slaves are handling over 300k queries
> > per second.
>
> > After the switchover we had to add some additional Redis nodes but not
> > because Redis was overworked but because the network cards couldn't
> > keep up with Redis.
>
> > Everyone involved has been more than impressed with Redis'
> > performance.
>
> > All reads come from Redis;  we are maintaining MySQL just to allow us
> > to build new sorted sets as our requirement change.
>
> > I'm going to be doing the keynote a major Canadian tech conference

> > (confoo.ca) about theyouporn.com rewrite and one of the major topics

> > will be Redis and how much it helped us.
>
> > Do you guys have a logo that would look good on a white background?  I
> > want to make you guys stand out.
>
> > Thanks for the great software!
>
> > ~Eric
>
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

>
> --
> Salvatore 'antirez' Sanfilippo
> open source developer - VMware
>
> http://invece.org
> "We are what we repeatedly do. Excellence, therefore, is not an act, but a habit." -- Aristotele
>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

[Eric Pickup]

To be honest our we did our benchmarks so long ago that I no longer find the data. But even under quite heavy load, Redis is extremely responsive even with large sorted sets - we usually intersecting multiple sets of 100K or more.

My first talk will be an overview of how to put it all together and Redis will only be part of it. I plan future talks specifically about Redis for which I'll build some more detailed data.

~Eric

________________________________________
From: redi...@googlegroups.com [redi...@googlegroups.com] on behalf of emiretsk [eugene....@gmail.com]
Sent: Friday, February 17, 2012 1:44 PM
To: Redis DB

Hi Eric,

Regards,
Eugene

For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Eric Pickup]

Hi Sebastien,
That is peak load but Redis handled it for a sustained period of time (hours) with no problems.

~Eric



________________________________________
From: redi...@googlegroups.com [redi...@googlegroups.com] on behalf of Sebastian Stadil [seba...@scalr.com]
Sent: Friday, February 17, 2012 12:55 PM

To: Redis DB
Subject: Re: Youporn.com is now a 100% Redis Site

Congrats on the success!

I was calculating the number of queries you make per pageview, and get
the following:

Queries per sec * seconds in day / pageviews in day = 300k * (3600 *
24) / 100M = 260.

This seems to be on the high side, so I'm guessing the 300k queries/
sec is peak?

[catwell]

Fail or fake? http://pastebin.com/ieC6eTB7

[Stefan Scholl]

German news site Golem.de:
http://www.golem.de/news/nosql-youporn-wechselt-von-mysql-zu-redis-1202-89929.html

NoSQL: Youporn wechselt von MySQL zu Redis

[Bobby Toure]

Hi Eric, Hi all

It would be nice to no know on which architecture you did the bench
(OS, hardware spécification <cpu memory....), network interface,Bus
communication beetween the node , network interface, number of node )

Kinds Regards
Bobby

[Cris]

Hi Eric,
Unfortunatelly i cannot attend to your presentation from Montreal with subject 'Building a Website to Scale to 100 Million Page Views Per Day and Beyond' and this looks for me very interesting.

Do you have a powerpoint of this presentation on web so i can read it?

If not can you send me on my email?

thanks,
Cris

[Bouncner]

Hi guys,

I just wrote an email to the confoo guys asking when we can expect the slides of the presenters.

They gave me that link: https://joind.in/talk/view/6123

Enjoy! ;)