Site Maintenance, Instability, and API Limits
Alex Payne
8:30PM PST. We're adding additional read-slave databases and tuning
the configuration on our existing databases.
You may have noticed some downtime and intermittent availability over
the last 24 hours. This was the result of some additional caching
measures that we put in place which caused an unexpected memory leak,
triggering cascading failures across our cluster. We've since taken
steps to correct this issue. Performance both internal and external
to the site has been stable since noon PST, and we're watching it
closely.
Earlier this week we had a different incident in which some users
appeared to be signed in as other users. This occurred while were
testing a more recent version of Ruby and an alternative application
server on one machine in our cluster. Due to our load balancing
configuration, the issue quickly spread. We disabled the site and
resolved the issue within fifteen minutes. It should not have
effected API clients.
Both of these changes were the result of our testing potential
performance improvements in preparation for South by Southwest (SXSW),
a large annual multi-disciplinary conference held every March in
Austin, Texas. This time last year, our growth started to take off
around SXSW, and we want to be as reliable as possible during the
event this year. We will be taking more cautious steps to improve our
performance and reliability over the next week.
As part of those steps, I intend to decrease the number of allowed
authenticated API requests per hour from 70 to 50 from Thursday, March
6th through Wednesday, March 12th. While we are taking steps to
greatly increase our capacity (and have been doing so continuously,
particular since our move to our current host), the API is our
foremost source of traffic, and as such is the first place we look
when trying to create some breathing room for our cluster. I
appreciate your understanding, and I hope that 20 fewer requests per
hour don't impact your applications too drastically for the duration
of the modified rate limit.
We also intend to put some extra abuse-prevention measures in place
before the event. We've seen a general increase in abusive traffic
over the last several months, and we simply can't afford it during a
heavy-traffic event. If you've been scraping Twitter or consuming
public API feeds unfairly, be prepared for an unpleasant surprise.
Thanks, as always, for your patience and understanding. If you have
any questions, concerns, or suggestions, please do let me know.
--
Alex Payne
http://twitter.com/al3x
Gene Smith
> public API feeds unfairly, be prepared for an unpleasant surprise.
Can you clarify this... what constitutes unfair or abusive scraping?
Some guidelines would definitely help--I think most of us who use
scraping, public RSS feeds, etc., want to do it in a respectful way
that doesn't affect other Twitter users.
Gene
Marco Kaiser
you say, most of your traffic comes from API usage. You also tell us, some of us (API users) might experience "an unpleasent surprise" soon, because they used the API "unfairly".
Raising a general suspicion like this without being specific, do you think it is a good way to treat those people who are responsible for most of your traffic? I guess the vast majority of us does not use the API in an unfair way. If we do, tell us specific what we do wrong, from your point of view. But please, DON'T change something on your side that might break clients WITHOUT telling us in detail before.
Just my 2c...
Aaron B. Hockley
abusers, but you're also going to piss off your most active "real"
users, as well.
-Aaron
Ed Finkler
>
> When you lower the API rate limit, sure you'll probably catch some
> abusers, but you're also going to piss off your most active "real"
> users, as well.
Good thing you let them know!
Ed Finkler
> Raising a general suspicion like this without being specific, do you think
> it is a good way to treat those people who are responsible for most of your
> traffic?
How do you know this is true?
> I guess the vast majority of us does not use the API in an unfair
> way.
I suspect this is the case. I think we can give alex the benefit of
the doubt, no? Asking for more details is fine, but lets not flare up
just yet.
-Ed
Marco Kaiser
How do you know this is true?
On Sat, Mar 1, 2008 at 10:22 AM, Marco Kaiser <kaiser...@gmail.com> wrote:
> Raising a general suspicion like this without being specific, do you think
> it is a good way to treat those people who are responsible for most of your
> traffic?
Alex told us, in his original posting in this thread.
I suspect this is the case. I think we can give alex the benefit of
> I guess the vast majority of us does not use the API in an unfair
> way.
the doubt, no? Asking for more details is fine, but lets not flare up
just yet.
I simply don't understand why he wants to "surprise" those who play unfair, and maybe affect others with what he does in reaction, too. I'd like to see a little more transparency about this, that's all. Sure he can have doubts, but then he should tell what is going wrong, and what he wants to change.
Don't want to start a flame here. It was my 2c, and I just questioned their (Twitter's) communication to the developer's community. Nothing personal.
Andrew Badera
Personally, I don't care if Alex "surprises" people or not. The -20 API limit is all the info I need; why do you, or does anyone, need more? If you're abusively scraping, you deserve to have the rug pulled out from under you. And if you can't tell whether or not you're being abusive, assume you're a culprit, and modify your app behavior appropriately.
--Andy Badera
http://andrew.badera.us/
and...@badera.us
(518) 641-1280
Google me: http://www.google.com/search?q=andrew+badera
deali...@gmail.com
> appeared to be signed in as other users.
As for the SXSW-specific API throttling, that seems unfair to
applications that have carefully parsed out their calls to the
different parts of their application. If you are going to push back
some of the scaling responsibility to your 3rd party developer
community, at least just make it permanent; and "scraping Twitter or
target of this? Nonspecific threats are usually unhelpful and often
scary for the innocent (as testified by the response you have already
received).
I know you guys are Good People and frustrated with abuse, but Twitter
is frustrating enough as a user. No need to condescend to the
community that is taking Twitter from awesome to killer app.
Marco Kaiser
Personally, I care, not because I have fear to be discovered as an abusive user (I'm not, for sure), but because I fear their actions could also affect those clients that try to be fair. Having a bit more knowledge what they plan to do simply would be nice.
Alex Payne
I know what policies we're putting in place, I'll let the list know.
The only reason I didn't is because we're still working that out.
--
Alex Payne
http://twitter.com/al3x
Cameron Kaiser
> > abusers, but you're also going to piss off your most active "real"
> > users, as well.
>
> Good thing you let them know!
And if you have some sort of support channel or Twitter feed, you can give
people who don't read this list advance warning. In fact, that's what I'll
be doing periodically for TTYtter users. It's good support policy for your
client base.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Even cabbage more sense than you! -- Shampoo, "Ranma 1/2" ------------------
Ed Finkler
> Alex told us, in his original posting in this thread.
Ah, I can see that -- API users in general. Gotcha.
Marco Kaiser
Alex Payne
of the concerns raised in this thread. Thanks for your feedback, all.
--
Alex Payne
http://twitter.com/al3x