Twitter scaling problems and Ruby on Rails
33 views
Skip to first unread message
RogerV
May 26, 2008, 4:40:43 PM5/26/08
to The Java Posse
In Java Posse episode #188, the Posse basically retreated from an
earlier report regarding Twitter service outages and scaling problems
being due to Ruby on Rails.
In the end this was chalked up to the Internet rumour mill. Evidently
the Twitter PR machine is trying to circulate the notion that Ruby on
Rails doesn't really play a substantial role in their data centre
architecture and all this was overblown and insubstantial.
Well, I suspect this is just the typical corporate attempt to manage
bad news and keep investors calm about technical competence matters in
respect to the Twitter technical management. (Talk about a social
network that has slim opportunity to monetize its user base -
regardless of the site's popularity. No doubt Twitter has a very
delicate dance with investors.)
Now consider recent Internet headlines in juxtaposition to this
interview from about a year ago (where I've excerpted one of the
relevant sections in regard to today's events):
5 Question Interview with Twitter Developer Alex Payne
http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/
[Question] How has Ruby on Rails been holding up to the increased
load?
[Alex Payne] By various metrics Twitter is the biggest Rails site on
the net right now. Running on Rails has forced us to deal with scaling
issues - issues that any growing site eventually contends with - far
sooner than I think we would on another framework.
The common wisdom in the Rails community at this time is that scaling
Rails is a matter of cost: just throw more CPUs at it. The problem is
that more instances of Rails (running as part of a Mongrel cluster, in
our case) means more requests to your database. At this point in time
there’s no facility in Rails to talk to more than one database at a
time. The solutions to this are caching the hell out of everything and
setting up multiple read-only slave databases, neither of which are
quick fixes to implement. So it’s not just cost, it’s time, and time
is that much more precious when people can['t] reach your site.
None of these scaling approaches are as fun and easy as developing for
Rails. All the convenience methods and syntactical sugar that makes
Rails such a pleasure for coders ends up being absolutely punishing,
performance-wise. Once you hit a certain threshold of traffic, either
you need to strip out all the costly neat stuff that Rails does for
you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of
your application out of Rails, or both.
It’s also worth mentioning that there shouldn’t be doubt in anybody’s
mind at this point that Ruby itself is slow. It’s great that people
are hard at work on faster implementations of the language, but right
now, it’s tough. If you’re looking to deploy a big web application and
you’re language-agnostic, realize that the same operation in Ruby will
take less time in Python. All of us working on Twitter are big Ruby
fans, but I think it’s worth being frank that this isn’t one of those
relativistic language issues. Ruby is slow.
earlier report regarding Twitter service outages and scaling problems
being due to Ruby on Rails.
In the end this was chalked up to the Internet rumour mill. Evidently
the Twitter PR machine is trying to circulate the notion that Ruby on
Rails doesn't really play a substantial role in their data centre
architecture and all this was overblown and insubstantial.
Well, I suspect this is just the typical corporate attempt to manage
bad news and keep investors calm about technical competence matters in
respect to the Twitter technical management. (Talk about a social
network that has slim opportunity to monetize its user base -
regardless of the site's popularity. No doubt Twitter has a very
delicate dance with investors.)
Now consider recent Internet headlines in juxtaposition to this
interview from about a year ago (where I've excerpted one of the
relevant sections in regard to today's events):
5 Question Interview with Twitter Developer Alex Payne
http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/
[Question] How has Ruby on Rails been holding up to the increased
load?
[Alex Payne] By various metrics Twitter is the biggest Rails site on
the net right now. Running on Rails has forced us to deal with scaling
issues - issues that any growing site eventually contends with - far
sooner than I think we would on another framework.
The common wisdom in the Rails community at this time is that scaling
Rails is a matter of cost: just throw more CPUs at it. The problem is
that more instances of Rails (running as part of a Mongrel cluster, in
our case) means more requests to your database. At this point in time
there’s no facility in Rails to talk to more than one database at a
time. The solutions to this are caching the hell out of everything and
setting up multiple read-only slave databases, neither of which are
quick fixes to implement. So it’s not just cost, it’s time, and time
is that much more precious when people can['t] reach your site.
None of these scaling approaches are as fun and easy as developing for
Rails. All the convenience methods and syntactical sugar that makes
Rails such a pleasure for coders ends up being absolutely punishing,
performance-wise. Once you hit a certain threshold of traffic, either
you need to strip out all the costly neat stuff that Rails does for
you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of
your application out of Rails, or both.
It’s also worth mentioning that there shouldn’t be doubt in anybody’s
mind at this point that Ruby itself is slow. It’s great that people
are hard at work on faster implementations of the language, but right
now, it’s tough. If you’re looking to deploy a big web application and
you’re language-agnostic, realize that the same operation in Ruby will
take less time in Python. All of us working on Twitter are big Ruby
fans, but I think it’s worth being frank that this isn’t one of those
relativistic language issues. Ruby is slow.
sherod
May 27, 2008, 2:52:56 AM5/27/08
to The Java Posse
I've been trying to use twitter to follow the Mars Phoenix landers
progress (Yes, its twittering its mission....)
http://twitter.com/MarsPhoenix
We can land a probe on a distant planet, but Twitter can't display a
second page of posts because its 'feeling stressed'.
Maybe you do have to be a rocket scientist to build a social
networking site :)
On May 27, 6:40 am, RogerV <rog...@qwest.net> wrote:
> In Java Posse episode #188, the Posse basically retreated from an
> earlier report regarding Twitter service outages and scaling problems
> being due to Ruby on Rails.
>
> In the end this was chalked up to the Internet rumour mill. Evidently
> the Twitter PR machine is trying to circulate the notion that Ruby on
> Rails doesn't really play a substantial role in their data centre
> architecture and all this was overblown and insubstantial.
>
> Well, I suspect this is just the typical corporate attempt to manage
> bad news and keep investors calm about technical competence matters in
> respect to the Twitter technical management. (Talk about a social
> network that has slim opportunity to monetize its user base -
> regardless of the site's popularity. No doubt Twitter has a very
> delicate dance with investors.)
>
> Now consider recent Internet headlines in juxtaposition to this
> interview from about a year ago (where I've excerpted one of the
> relevant sections in regard to today's events):
>
> 5 Question Interview with Twitter Developer Alex Paynehttp://www.radicalbehavior.com/5-question-interview-with-twitter-deve...
progress (Yes, its twittering its mission....)
http://twitter.com/MarsPhoenix
We can land a probe on a distant planet, but Twitter can't display a
second page of posts because its 'feeling stressed'.
Maybe you do have to be a rocket scientist to build a social
networking site :)
On May 27, 6:40 am, RogerV <rog...@qwest.net> wrote:
> In Java Posse episode #188, the Posse basically retreated from an
> earlier report regarding Twitter service outages and scaling problems
> being due to Ruby on Rails.
>
> In the end this was chalked up to the Internet rumour mill. Evidently
> the Twitter PR machine is trying to circulate the notion that Ruby on
> Rails doesn't really play a substantial role in their data centre
> architecture and all this was overblown and insubstantial.
>
> Well, I suspect this is just the typical corporate attempt to manage
> bad news and keep investors calm about technical competence matters in
> respect to the Twitter technical management. (Talk about a social
> network that has slim opportunity to monetize its user base -
> regardless of the site's popularity. No doubt Twitter has a very
> delicate dance with investors.)
>
> Now consider recent Internet headlines in juxtaposition to this
> interview from about a year ago (where I've excerpted one of the
> relevant sections in regard to today's events):
>
Frederic Simon
May 27, 2008, 3:43:24 AM5/27/08
to java...@googlegroups.com
Scaling, and scaling at today's internet level is VERY hard. I use to tune Java systems in 1999 for 50,000 requests per seconds it was unique and at the limit of what Java AS can do (even today). It's amazing to see, that most Web Sites, did not need to deal with ultra scalability until recently.
We are avoiding the thought of great fast success: The bubble got busted.
But startups today are reaching the popularity level that needs ultra scalability VERY fast. For me Twitter problem is a proof, of the Web success we were selling to investors in the 90s!
And by the way, standard JEE architecture cannot and will not solve twitter problems. So, the Java community should not jump on the issue too fast without looking at ultra scallable Java Web Site (which are???)...
"Why most large-scale Web sites are not written in Java": http://tinyurl.com/ysj656
Nati Shalom about twitter: http://tinyurl.com/6zxp9e
I'm not selling GigaSpaces, but Nati knows what ultra scallabilty means. Also Terracota, Amazon EC2 and Google App Engine, are lowering the cost of making a very sucessful site, a success :-)
By the way, I heard someone rewrotte twitter on GAE! Did you heard about that?--
http://freddy33.blogspot.com/
http://www.jfrog.org/
We are avoiding the thought of great fast success: The bubble got busted.
But startups today are reaching the popularity level that needs ultra scalability VERY fast. For me Twitter problem is a proof, of the Web success we were selling to investors in the 90s!
And by the way, standard JEE architecture cannot and will not solve twitter problems. So, the Java community should not jump on the issue too fast without looking at ultra scallable Java Web Site (which are???)...
"Why most large-scale Web sites are not written in Java": http://tinyurl.com/ysj656
Nati Shalom about twitter: http://tinyurl.com/6zxp9eI'm not selling GigaSpaces, but Nati knows what ultra scallabilty means. Also Terracota, Amazon EC2 and Google App Engine, are lowering the cost of making a very sucessful site, a success :-)
By the way, I heard someone rewrotte twitter on GAE! Did you heard about that?
http://freddy33.blogspot.com/
http://www.jfrog.org/
sherod
May 27, 2008, 6:32:23 AM5/27/08
to The Java Posse
One more thing - maybe it's more the traditional case of just not
having enough money or management....
http://www.techcrunch.com/2008/04/24/lee-mighdoll-out-at-twitter-business-plan-still-mia/
Those two things tend to hit you harder than the technical issues :)
having enough money or management....
http://www.techcrunch.com/2008/04/24/lee-mighdoll-out-at-twitter-business-plan-still-mia/
Those two things tend to hit you harder than the technical issues :)
Frederic Simon
May 27, 2008, 7:00:18 AM5/27/08
to java...@googlegroups.com
Well, for me it's more let's save by building a site without including experienced people.
We are currently refactoring a very successful site where all the developers are directly out of University. It's an extremely talented team, but the lack of experience made them do ALL the mistakes that lead to low performance and scallability.
So, now they need to pay for a VERY big hardware stack, and heavy consulting fee (I'm not going to complain :) to fix it. Frankly, if one experienced guy was there from the beginning it will have save a lot.
But expensive fee at the beginning it's tough!
By the way, this is how I use to sell WebLogic Server to startup (I'm kind of ashame of it now): Just buy this expensive stack, and without trained developers you will have a highly scallable software!
Sorry for everything...
We are currently refactoring a very successful site where all the developers are directly out of University. It's an extremely talented team, but the lack of experience made them do ALL the mistakes that lead to low performance and scallability.
So, now they need to pay for a VERY big hardware stack, and heavy consulting fee (I'm not going to complain :) to fix it. Frankly, if one experienced guy was there from the beginning it will have save a lot.
But expensive fee at the beginning it's tough!
By the way, this is how I use to sell WebLogic Server to startup (I'm kind of ashame of it now): Just buy this expensive stack, and without trained developers you will have a highly scallable software!
Sorry for everything...
Dick Wall
May 27, 2008, 11:37:06 AM5/27/08
to The Java Posse
Hi Folks,
While I agree that RoR probably can't scale to anything like what
Twitter can do, I also know that in this particular case Ruby and
Rails are not the problem. Unfortunately I just can't reveal how I
know that, but it is information we did not have during JavaOne :-).
Suffice to say that RoR is a tiny part of Twitter now, and in fact
small enough that it looks a bit disingenuous to hold up Twitter as an
example of a large RoR app...
Cheers
Dick
On May 26, 1:40 pm, RogerV <rog...@qwest.net> wrote:
> In Java Posse episode #188, the Posse basically retreated from an
> earlier report regarding Twitter service outages and scaling problems
> being due to Ruby on Rails.
>
> In the end this was chalked up to the Internet rumour mill. Evidently
> the Twitter PR machine is trying to circulate the notion that Ruby on
> Rails doesn't really play a substantial role in their data centre
> architecture and all this was overblown and insubstantial.
>
> Well, I suspect this is just the typical corporate attempt to manage
> bad news and keep investors calm about technical competence matters in
> respect to the Twitter technical management. (Talk about a social
> network that has slim opportunity to monetize its user base -
> regardless of the site's popularity. No doubt Twitter has a very
> delicate dance with investors.)
>
> Now consider recent Internet headlines in juxtaposition to this
> interview from about a year ago (where I've excerpted one of the
> relevant sections in regard to today's events):
>
> 5 Question Interview with Twitter Developer Alex Paynehttp://www.radicalbehavior.com/5-question-interview-with-twitter-deve...
While I agree that RoR probably can't scale to anything like what
Twitter can do, I also know that in this particular case Ruby and
Rails are not the problem. Unfortunately I just can't reveal how I
know that, but it is information we did not have during JavaOne :-).
Suffice to say that RoR is a tiny part of Twitter now, and in fact
small enough that it looks a bit disingenuous to hold up Twitter as an
example of a large RoR app...
Cheers
Dick
On May 26, 1:40 pm, RogerV <rog...@qwest.net> wrote:
> In Java Posse episode #188, the Posse basically retreated from an
> earlier report regarding Twitter service outages and scaling problems
> being due to Ruby on Rails.
>
> In the end this was chalked up to the Internet rumour mill. Evidently
> the Twitter PR machine is trying to circulate the notion that Ruby on
> Rails doesn't really play a substantial role in their data centre
> architecture and all this was overblown and insubstantial.
>
> Well, I suspect this is just the typical corporate attempt to manage
> bad news and keep investors calm about technical competence matters in
> respect to the Twitter technical management. (Talk about a social
> network that has slim opportunity to monetize its user base -
> regardless of the site's popularity. No doubt Twitter has a very
> delicate dance with investors.)
>
> Now consider recent Internet headlines in juxtaposition to this
> interview from about a year ago (where I've excerpted one of the
> relevant sections in regard to today's events):
>
Frederic Simon
May 27, 2008, 11:51:30 AM5/27/08
to java...@googlegroups.com
And did you see the Twitter clone on GAE?
Jess Holle
May 27, 2008, 1:14:27 PM5/27/08
to java...@googlegroups.com
There have been articles elsewhere (cnet as I recall) on the notion
that Twitter's entire architecture is ill-suited for its requirements
and is being reworked.
I think their issues are a lot bigger than any use of RoR they might have made.
--
Jess Holle
I think their issues are a lot bigger than any use of RoR they might have made.
--
Jess Holle
phil.s...@gmail.com
May 27, 2008, 7:08:57 PM5/27/08
to The Java Posse
On May 27, 9:37 am, Dick Wall <dickw...@gmail.com> wrote:
> Hi Folks,
>
> While I agree that RoR probably can't scale to anything like what
> Twitter can do
Ruby is quite slow.... but the RoR architecture is stateless so it
> Hi Folks,
>
> While I agree that RoR probably can't scale to anything like what
> Twitter can do
scales horizontally quite well. You just need to add many more
machines that you would for a Java equivalent. see
http://en.wikipedia.org/wiki/Shared_nothing_architecture
phil.s...@gmail.com
May 27, 2008, 7:13:01 PM5/27/08
to The Java Posse
>The common wisdom in the Rails community at this time is that scaling
>Rails is a matter of cost: just throw more CPUs at it. The problem is
>that more instances of Rails (running as part of a Mongrel cluster, in
>our case) means more requests to your database. At this point in time
>there’s no facility in Rails to talk to more than one database at a
>time. The solutions to this are caching the hell out of everything and
>setting up multiple read-only slave databases, neither of which are
>quick fixes to implement.
Just a comment on this:
>Rails is a matter of cost: just throw more CPUs at it. The problem is
>that more instances of Rails (running as part of a Mongrel cluster, in
>our case) means more requests to your database. At this point in time
>there’s no facility in Rails to talk to more than one database at a
>time. The solutions to this are caching the hell out of everything and
>setting up multiple read-only slave databases, neither of which are
>quick fixes to implement.
http://www.google.com/search?q=multiple+databases+ruby+on+rails&sourceid=navclient-ff&ie=UTF-8&rlz=1B3GGGL_enUS268US268
RogerV
May 27, 2008, 9:32:29 PM5/27/08
to The Java Posse
On May 27, 8:37 am, Dick Wall <dickw...@gmail.com> wrote:
> Hi Folks,
>
> While I agree that RoR probably can't scale to anything like what
> Twitter can do, I also know that in this particular case Ruby and
> Rails are not the problem. Unfortunately I just can't reveal how I
> know that, but it is information we did not have during JavaOne :-).
> Suffice to say that RoR is a tiny part of Twitter now, and in fact
> small enough that it looks a bit disingenuous to hold up Twitter as an
> example of a large RoR app...
>
> Cheers
>
> Dick
Not sure that does anything to bring RoR out of the line of fire,
> Hi Folks,
>
> While I agree that RoR probably can't scale to anything like what
> Twitter can do, I also know that in this particular case Ruby and
> Rails are not the problem. Unfortunately I just can't reveal how I
> know that, but it is information we did not have during JavaOne :-).
> Suffice to say that RoR is a tiny part of Twitter now, and in fact
> small enough that it looks a bit disingenuous to hold up Twitter as an
> example of a large RoR app...
>
> Cheers
>
> Dick
though.
As Twitter developer, Alex Payne, said in that interview, "Twitter is
the biggest Rails site on
the net right now".
And then Payne goes on to elaborate at some length the woes they
the net right now".
encountered in having developed heavily on RoR.
So now we hear RoR plays an insignificant role in Twitter...All this
says is that a year ago they discovered they had a serious problem
with RoR and decided to then begin moving away from it for doing the
serious work of their site.
Based on Payne's interview, and Dick's current inside information, we
can therefore say that Twitter was _formerly_ one of the biggest RoR
sites on the net.
Hmm, that doesn't exactly come off as a glowing endorsement for RoR
either.
Somehow or another it looks like the matters of a year ago - per RoR
problems - and the current outage woes, have become conflated.
RogerV
May 27, 2008, 9:34:19 PM5/27/08
to The Java Posse
On May 27, 4:13 pm, "phil.swen...@gmail.com" <phil.swen...@gmail.com>
wrote:
The interview dated back to March 2007.
The postings for the technique for connecting to multiple databases
start showing up April/May 2007. RoR was originally architected around
single database connections.
So at the time of the interview, the problem likely existed as
described.
phil.s...@gmail.com
May 28, 2008, 12:55:04 AM5/28/08
to The Java Posse
solutions were available. It just has never been supported out of the
box...
Michael
May 28, 2008, 5:03:45 PM5/28/08
to The Java Posse
The problem that Twitter encountered last year was data federation,
i.e. multiple databases associated to the same model class. I think
this is a problem for just about every ORM out there, including
popular Java ones like Hibernate (Shards tries to address this, but
not that well.)
I talked to Alex and Blaine last year about some of their issues. In
particular they mentioned a "inbox" style solution as suggested here:
http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi.html .
They could then split DBs across users. Each user's inbox could have a
full copy of each tweet they would see. Obviously it would be more
efficient to just put an ID, and put the updates in their own database
as well. That DB would need to be split as well eventually, and doing
it temporarily (each database contains all updates for the past N days/
seconds) seemed to be the way to go there. Some of the Rails plugins
seemed to be able to handle the user split, but the update split.
On May 27, 9:55 pm, "phil.swen...@gmail.com" <phil.swen...@gmail.com>
wrote:
i.e. multiple databases associated to the same model class. I think
this is a problem for just about every ORM out there, including
popular Java ones like Hibernate (Shards tries to address this, but
not that well.)
I talked to Alex and Blaine last year about some of their issues. In
particular they mentioned a "inbox" style solution as suggested here:
http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi.html .
They could then split DBs across users. Each user's inbox could have a
full copy of each tweet they would see. Obviously it would be more
efficient to just put an ID, and put the updates in their own database
as well. That DB would need to be split as well eventually, and doing
it temporarily (each database contains all updates for the past N days/
seconds) seemed to be the way to go there. Some of the Rails plugins
seemed to be able to handle the user split, but the update split.
On May 27, 9:55 pm, "phil.swen...@gmail.com" <phil.swen...@gmail.com>
wrote:
RogerV
May 29, 2008, 8:24:08 PM5/29/08
to The Java Posse
None-the-less, it remains most interesting that Twitter decided to
deemphasize use of RoR since a year ago.
deemphasize use of RoR since a year ago.
Tom Eastmond
May 30, 2008, 1:51:05 PM5/30/08
to java...@googlegroups.com
Not to drag this on forever but I thought their response blog post from yesterday was interesting and remarkably open:
http://dev.twitter.com/2008/05/youve-got-qs-weve-got-as.html
Twitter is great - I hope they get things figured out - ruby/rails or not.
Tom
http://dev.twitter.com/2008/05/youve-got-qs-weve-got-as.html
Twitter is great - I hope they get things figured out - ruby/rails or not.
Tom
Reply all
Reply to author
Forward
0 new messages