DDoS Status Update

45 views
Skip to first unread message

Ryan Sarver

unread,
Aug 7, 2009, 2:05:32 PM8/7/09
to twitter-deve...@googlegroups.com
I wanted to send everyone an update to let you know what has been happening, the known issues, some suggestions on how to resolve them and some idea of how to move forward.

Whats been happening
As you know all too well Twitter, among other services, has been getting hit pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the attack come in a number of waves and from a number of different vectors increasing in intensity along the way. We were able to stabilize our own service for a bit, hence Biz's post saying all was well, but that didn't mean the attacks had ceased. In fact, at around 3am PST today the attacks intensified to almost 10x of what it was yesterday. In order for us to defend from the attack we have had to put a number of services in place and we know that some of you have gotten caught in the crossfire. Please know we are as frustrated as you are and wish there was more we could have communicated along the way.

Known Issues
 - HTTP 300 response codes - One of the measures in thwarting the onslaught requires that all traffic respect HTTP 30x response codes. This will help us identify the good traffic from the bad.
 - General throttling - Try to throttle your services back as much as possible for you to continue operating. We are working on our end to better understand the logic used in throttling traffic on the edge of the network and will communicate what we can, but the best idea is to just throttle back as much as you can in the mean time.
 - Streaming API - as part of the edge throttling we know requests to the Streaming API with lists of keywords or uses are getting dropped because the request is too large. We are working to get this filter removed and will update the list when we know more.
- Unexpected HTTP response codes - we know people are seeing a lot of other weirdness and we aren't exactly sure what to attribute the various issues to, but know that you aren't alone.

As the attacks change our tactics for defense will likely need to change as well, so stay active on the list and let us know what problems you are seeing and we will do our best to help guide you along.

Moving forward
We will try to communicate as much as we can so you guys are up to speed as things change and progress. I personally apologize for not communicating more in the mean time but there hasn't been much guidance we have been able to give other than hold tight with us. We fully appreciate all the long hours you are putting in to keep your apps running and supporting your users and know we are frustrated with you. Continue to watch this list, status.twitter.com and @twitterapi for updates

Thanks for your patience, Ryan

PM, Platform Team
@rsarver


Rich

unread,
Aug 7, 2009, 2:28:29 PM8/7/09
to Twitter Development Talk
Thanks for the update, however PLEASE get oAuth back up and running
ASAP please!

On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
>
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
>
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
>
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
>
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
>
> Thanks for your patience, Ryan
>
> PM, Platform Team
> @rsarver <http://twitter.com/rsarver>

Goblin

unread,
Aug 7, 2009, 2:32:36 PM8/7/09
to Twitter Development Talk
OAuth is working fine for my site. To be honest, for something that
does nothing but interact with Twitter I haven't seen much of a drop
in activity.

Joe Bowman

unread,
Aug 7, 2009, 2:34:36 PM8/7/09
to Twitter Development Talk
Applications in cloud hosting environments may be unable to throttle
anything, due to the fact that if it's IP based checking, the cloud
IPs are stlll going to be sending a lot of requests. ie: Appengine
applications.

Rich

unread,
Aug 7, 2009, 2:38:57 PM8/7/09
to Twitter Development Talk
oAuth worked for me on testing this morning, but trying to
authenticate three seperate accounts, right now... all of them timeout
on clicking the 'Allow' button

Greg Avola

unread,
Aug 7, 2009, 2:49:42 PM8/7/09
to Twitter Development Talk
This is happening all my applications.

Clicking Allow - just causes the App to timeout.

This reminds of the OAuth outage we had last time - which begs the
question, is OAuth ready for production applications?

Vincent Nguyen

unread,
Aug 7, 2009, 2:51:53 PM8/7/09
to twitter-deve...@googlegroups.com
Yes! Me too!
I think we must stop out service temporarily while waitng twitter team solve it!
Be patient for all of us!

2009/8/7 Greg Avola <gregor...@gmail.com>

Rich

unread,
Aug 7, 2009, 2:50:38 PM8/7/09
to Twitter Development Talk
Except if you want from [source] on your posts for 'newer' apps you
can only use oAuth!

Mario Menti

unread,
Aug 7, 2009, 2:53:48 PM8/7/09
to twitter-deve...@googlegroups.com
Thanks for the update Ryan.

One thing I don't quite understand is why it's not an option to allow whitelisted applications to post. I will try and throttle our (twitterfeed.com) service back, but with nearly half a million of active feeds in the system, I can't quite see how this will help, as even a fraction of requests will be way over any non-whitelist limits you have in place.

Mario.

narendra

unread,
Aug 7, 2009, 2:59:19 PM8/7/09
to Twitter Development Talk
Is there an insight into the hanging (posts, favorites) that is
happening on the twitter.com website?

Joe Bowman

unread,
Aug 7, 2009, 3:00:15 PM8/7/09
to Twitter Development Talk
All my oauth requests are failing with an invalid token exception, and
the response to the request for the token appears to be null. This is
using the twitter python client and from appengine. I don't even get
to the point of redirecting users to the login page.

Jesse Stay

unread,
Aug 7, 2009, 3:02:57 PM8/7/09
to twitter-deve...@googlegroups.com
Thanks for the communication - this is good.  Just curious - with entire businesses being put out of place, and rumors that the Russian Gov't may be behind such attacks, is Twitter communicating with Homeland Security about this?  To me this seems like a matter of national security even more than it is a Twitter issue.  The US economy is being attacked because of this.

Not to sound too radical - I'm just genuinely curious when the Government is going to get involved.  (and thank you for doing what you can - I'm sure I speak for all when I say we feel your pain)

Jesse

Rich

unread,
Aug 7, 2009, 3:05:54 PM8/7/09
to Twitter Development Talk
I agree with this, although it's not just the US economy... hurts many
other countries too... well businesses within those countries anyway!

On Aug 7, 8:02 pm, Jesse Stay <jesses...@gmail.com> wrote:
> Thanks for the communication - this is good.  Just curious - with entire
> businesses being put out of place, and rumors that the Russian Gov't may be
> behind such attacks, is Twitter communicating with Homeland Security about
> this?  To me this seems like a matter of national security even more than it
> is a Twitter issue.  The US economy is being attacked because of this.
> Not to sound too radical - I'm just genuinely curious when the Government is
> going to get involved.  (and thank you for doing what you can - I'm sure I
> speak for all when I say we feel your pain)
>
> Jesse
>
>
>
> On Fri, Aug 7, 2009 at 2:05 PM, Ryan Sarver <rsar...@twitter.com> wrote:
> > I wanted to send everyone an update to let you know what has been
> > happening, the known issues, some suggestions on how to resolve them and
> > some idea of how to move forward.
>
> > *Whats been happening*
> > As you know all too well Twitter, among other services, has been getting
> > hit pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw
> > the attack come in a number of waves and from a number of different vectors
> > increasing in intensity along the way. We were able to stabilize our own
> > service for a bit, hence Biz's post saying all was well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > today the attacks intensified to almost 10x of what it was yesterday. In
> > order for us to defend from the attack we have had to put a number of
> > services in place and we know that some of you have gotten caught in the
> > crossfire. Please know we are as frustrated as you are and wish there was
> > more we could have communicated along the way.
>
> > *Known Issues*
> > * - HTTP 300 response codes* - One of the measures in thwarting the
> > onslaught requires that all traffic respect HTTP 30x response codes. This
> > will help us identify the good traffic from the bad.
> > * - General throttling* - Try to throttle your services back as much as
> > possible for you to continue operating. We are working on our end to better
> > understand the logic used in throttling traffic on the edge of the network
> > and will communicate what we can, but the best idea is to just throttle back
> > as much as you can in the mean time.
> > * - Streaming API* - as part of the edge throttling we know requests to
> > the Streaming API with lists of keywords or uses are getting dropped because
> > the request is too large. We are working to get this filter removed and will
> > update the list when we know more.
> > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > other weirdness and we aren't exactly sure what to attribute the various
> > issues to, but know that you aren't alone.
>
> > As the attacks change our tactics for defense will likely need to change as
> > well, so stay active on the list and let us know what problems you are
> > seeing and we will do our best to help guide you along.
>
> > *Moving forward *
> > We will try to communicate as much as we can so you guys are up to speed as
> > things change and progress. I personally apologize for not communicating
> > more in the mean time but there hasn't been much guidance we have been able
> > to give other than hold tight with us. We fully appreciate all the long
> > hours you are putting in to keep your apps running and supporting your users
> > and know we are frustrated with you. Continue to watch this list,
> > status.twitter.com and @twitterapi for updates
>
> > Thanks for your patience, Ryan
>
> > PM, Platform Team
> > @rsarver <http://twitter.com/rsarver>

Marco Kaiser

unread,
Aug 7, 2009, 3:09:55 PM8/7/09
to twitter-deve...@googlegroups.com
I'm sure they would let you know first... 

Get real. 

Sent from my iPhone

Chris

unread,
Aug 7, 2009, 3:10:56 PM8/7/09
to Twitter Development Talk
Thank you for updating us!

I have still a problem with getting search results via curl like
described here: http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search
This was working pretty good before the DDoS attack, but now I don't
get any results just http_code of 302.

An example url, I'm trying to get is:
http://search.twitter.com/search.atom?q=twitter&lang=de&rpp=15&since_id=3170060360&show_user=true

Can you give me a hint what I'm doing wrong or is this part of the API
still not working correct?


Thanks and best regards,
Chris

Genevate

unread,
Aug 7, 2009, 3:12:44 PM8/7/09
to Twitter Development Talk
Ryan,

First, thanks for finally posting such a message. It has been pretty
frustrating when there is no communication for you guys. Especially
when we developers rely on your service and you also rely on us
promoting your service. It makes us third party developers look stupid
when Biz/Twitter tells half truths to the public says that everything
is running great when in reality it is not. I understand you want to
look like things are fine so people don't loose confidence in your
service but it doesn't say much to us developers and give us much
confidence about your transparency and what the actual truth you tell
us is. If anyone should be updated it should be your developer
community first then the users, you owe us that much as we have built
your service up ever user we send your way. How about you make a more
public announcement to all your users to let them know that your
issues are affecting all the third party vendors and the issues is not
fixed.

Now that I have been able to finally get a message through to someone
at Twitter, how about getting oAuth working so that our users can at
least login consistently to our applications. Right now oAuth login
works some times and not others. You make us move over to something
that obviously is not production ready.

Also why are us white listed apps having troubles and getting a rate
limit of 150 like average users. I understand you want us to throttle
it back but, can't you give us more than that? Basically your idea of
throttling is cutting our usual allowed consumption down to less than
1%!? Serious, why not just shut it off entirely.

Chris-


On Aug 7, 2:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
>
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
>
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
>
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
>
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
>
> Thanks for your patience, Ryan
>
> PM, Platform Team
> @rsarver <http://twitter.com/rsarver>

Tiago Pinto

unread,
Aug 7, 2009, 2:56:33 PM8/7/09
to Twitter Development Talk
Hello Ryan,

Thanks for that update.

currently I can ping twitter.com but I can't access http on it

tpinto@vm:~/app$ ping twitter.com -c4
PING twitter.com (168.143.162.116) 56(84) bytes of data.
64 bytes from 168.143.162.116: icmp_seq=1 ttl=241 time=212 ms
64 bytes from 168.143.162.116: icmp_seq=2 ttl=241 time=243 ms
64 bytes from 168.143.162.116: icmp_seq=3 ttl=241 time=216 ms
64 bytes from 168.143.162.116: icmp_seq=4 ttl=241 time=214 ms
--- twitter.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3017ms
rtt min/avg/max/mdev = 212.389/221.455/243.174/12.611 ms

tpinto@vm:~/app$ curl http://twitter.com
curl: (7) couldn't connect to host

This app/ip is whitelisted and we're using mbleigh's twitter-auth.

Any thoughts on this one? Is it a problem on our end (i.e. our
network, dns cache etc.)?

Thanks,
Tiago

On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
>
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
>
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
>
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
>
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
>
> Thanks for your patience, Ryan
>
> PM, Platform Team
> @rsarver <http://twitter.com/rsarver>

Seth

unread,
Aug 7, 2009, 2:58:16 PM8/7/09
to Twitter Development Talk
DMs seem to be down as well. Haven't been able to get any to go out.
Tweets seem to be fine though.

briantroy

unread,
Aug 7, 2009, 3:00:17 PM8/7/09
to Twitter Development Talk
I have a php/memcache based Twitter Throttle if anyone needs a
reference implementation. Just drop me an email at brian dot roy at
cosinity dot com

Tadeu Andrade

unread,
Aug 7, 2009, 3:04:11 PM8/7/09
to Twitter Development Talk
Same with me. OAuth doesn't work at all. Even the login page is showed
up =\

Nick Arnett

unread,
Aug 7, 2009, 3:25:32 PM8/7/09
to twitter-deve...@googlegroups.com
On Fri, Aug 7, 2009 at 12:02 PM, Jesse Stay <jess...@gmail.com> wrote:
Thanks for the communication - this is good.  Just curious - with entire businesses being put out of place, and rumors that the Russian Gov't may be behind such attacks, is Twitter communicating with Homeland Security about this?  To me this seems like a matter of national security even more than it is a Twitter issue.  The US economy is being attacked because of this.

Not to sound too radical - I'm just genuinely curious when the Government is going to get involved.  (and thank you for doing what you can - I'm sure I speak for all when I say we feel your pain)

They probably already are and almost certainly will not say so, at least for a while.

Meanwhile, here's a page to keep an eye on in that regard: http://www.us-cert.gov/current/

Nick

Justin Hart

unread,
Aug 7, 2009, 3:27:08 PM8/7/09
to Twitter Development Talk
Comments inline.

On Aug 7, 12:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.

Does this affect POST as well as GET? The issue here is the way
clients handle 30x after POST. Most clients (now by convention) do
not respect the RFC (http://www.w3.org/Protocols/rfc2616/rfc2616-
sec10.html#sec10.3) and will send a GET after POST always. Some
clients will respect the method, but not re-post any data. We need to
be sure we are all expecting the right things.

How does this affect OAuth signed requests? OAuth requires knowing
the HTTP Method as well as the full URI and parameters to generate
signatures a priori. Most (all?) clients and libraries do not know
how to handle changes in these during a redirect.

> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.

Is there anything else we can do on the protocol level? Keepalives on/
off? Specific headers?

Thanks for all your hard work. Having spent yesterday battling a rash
of spambots is nothing compared to what you guys are doing.

--Justin

Chad Etzel

unread,
Aug 7, 2009, 3:27:03 PM8/7/09
to twitter-deve...@googlegroups.com
As stated in Ryan's email, you should respect 302 responses.

In curl this can be accomplished with the --location flag. See the man
page for more details.

-Chad

Beier

unread,
Aug 7, 2009, 3:38:16 PM8/7/09
to Twitter Development Talk
We have multiple servers running and they are getting different
response codes. some servers getting 302 for GETs and 408 for POSTs,
other servers getting 503s...

We can modify the code to respect 302s, but what about 503s?

On Aug 7, 11:05 am, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
>
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
>
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
>
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
>
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
>
> Thanks for your patience, Ryan
>
> PM, Platform Team
> @rsarver <http://twitter.com/rsarver>

Chad Etzel

unread,
Aug 7, 2009, 3:44:53 PM8/7/09
to twitter-deve...@googlegroups.com
Inline about 302's. I'm trying to get info on your other questions:

On Fri, Aug 7, 2009 at 3:27 PM, Justin Hart<onyx...@gmail.com> wrote:
> Does this affect POST as well as GET?  The issue here is the way
> clients handle 30x after POST.  Most clients (now by convention) do
> not respect the RFC (http://www.w3.org/Protocols/rfc2616/rfc2616-
> sec10.html#sec10.3) and will send a GET after POST always.  Some
> clients will respect the method, but not re-post any data.  We need to
> be sure we are all expecting the right things.

You are correct that lots of clients now redirect with a GET instead
of a POST, even curl does this, which is a bummer.

The best thing to do in that case is to catch the response code
(without following the redirect automatically) and manually re-attempt
the POST with the new location.

We know it's a pain, but that's one way around the POST->GET problem.

-Chad

Simon Tiplady

unread,
Aug 7, 2009, 3:22:38 PM8/7/09
to Twitter Development Talk
Chris,

A 302 header means you need to request the location that twitter has
sent back to you with that header.
It is part of their attempts at spotting the real requests from the
fake ones.
How you handle it all depends on what language you are programming in!

Simon

On Aug 7, 8:10 pm, Chris <kiraili...@gmail.com> wrote:
> Thank you for updating us!
>
> I have still a problem with getting search results via curl like
> described here:http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search
> This was working pretty good before the DDoS attack, but now I don't
> get any results just http_code of 302.
>
> An example url, I'm trying to get is:http://search.twitter.com/search.atom?q=twitter〈=de&rpp=15&since_...

Genevate

unread,
Aug 7, 2009, 3:25:53 PM8/7/09
to Twitter Development Talk
Obviously the issue is far larger than a normal DDoS attack.

Think about it. Why would they stop white listed Apps and rate limit
these as well as take down oAuth.
There is something else going on and my guess is that besides the
DDoS, it has something to do with spam and or a third party app which
has either been compromised or blatantly cause major issues.

I just wish that Twitter would be honest.

On Aug 7, 2:53 pm, Mario Menti <mme...@gmail.com> wrote:

Chris

unread,
Aug 7, 2009, 3:34:48 PM8/7/09
to Twitter Development Talk
Thanks a lot!
Everything is now working well again for me (I'm just a small guy
compared to your big application :) )!

Chris

JDG

unread,
Aug 7, 2009, 4:01:37 PM8/7/09
to twitter-deve...@googlegroups.com
THEY may not be stopping whitelisted IPs -- it could be coming from upstream.
--
Internets. Serious business.

hansamann

unread,
Aug 7, 2009, 4:35:25 PM8/7/09
to Twitter Development Talk
I saw some examples for those redirects and they seem to send even an
invalid Location header:

Location: /?somekey

It's illegal for the Location header to contain a relative URL:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.30

This causes APIs like twitter4j on Google App Engine to fail. Making
this a absolute URL would work... or would that somehow break the
security measuers taken?

Maybe the Twitter dev team can consider to make the Location URLs
absolute, a lot of us would then at least be able to do some stuff
with Twitter while these attacks take place.

Cheers
Sven

On Aug 7, 12:44 pm, Chad Etzel <c...@twitter.com> wrote:
> Inline about 302's. I'm trying to get info on your other questions:
>

Beier

unread,
Aug 7, 2009, 4:38:24 PM8/7/09
to Twitter Development Talk
After following 30x redirects, our servers are doing a little better,
at least we are getting results once in a few times. But we are still
getting lots of 503s for Search and 400s for REST. All of our servers
are supposed to be whitelisted, what could we do here?

On Aug 7, 12:44 pm, Chad Etzel <c...@twitter.com> wrote:
> Inline about 302's. I'm trying to get info on your other questions:
>

goodtest

unread,
Aug 7, 2009, 4:42:30 PM8/7/09
to Twitter Development Talk
I am getting 404 just for search api, it was working just fine all
along. I get 404 on production server but works fine on dev and qa
boxes - not sure why. My assumption is our production server was
active during DDos attack and has been blacklisted. Am I right? how
can I whitelist/fix it?

chavan

unread,
Aug 7, 2009, 4:56:28 PM8/7/09
to Twitter Development Talk
I couldn't even authenticate the twitter account from my server. but i
could do it in my localhost. May i know the reason why? does this
anything related to Ongoing denial-of-service attack

can't authenticate with the Oauth

Dewald Pretorius

unread,
Aug 7, 2009, 5:09:23 PM8/7/09
to Twitter Development Talk
Chad,

I need more info on the 30x responses, please.

Are these responses given only occasionally, or are they given
consistently and predictably?

Is it only on GET or only on POST, or both?

I've throttled back my API calls, and now when I run tests with both
GET and POST, I get 200 OK regardless of whether I have set the cURL
location flag or not.

Dewald

On Aug 7, 4:44 pm, Chad Etzel <c...@twitter.com> wrote:
> Inline about 302's. I'm trying to get info on your other questions:
>

Joe Bowman

unread,
Aug 7, 2009, 7:59:42 PM8/7/09
to Twitter Development Talk
Wow, just as I sit down to determine if there's any issue with my
oauth client not following redirects, or anything else within my
code... it all just started working again. That's after being down for
oauth and timelines since the DDoS began, and having the search API
stop working sometime last night. Trends pretty much always worked
though. Hope it stays

Kyle Mulka

unread,
Aug 7, 2009, 5:24:37 PM8/7/09
to Twitter Development Talk
OAuth appears to be completely broken, along with my app which uses
it. After clicking allow, Twitter takes forever to respond, and then
when it does it just comes back with a completely blank response.

--
Kyle Mulka
http://twilk.com

xzela

unread,
Aug 7, 2009, 8:35:25 PM8/7/09
to Twitter Development Talk
have you tried removing the OAuth code and replacing it with basic
authentication? If it works, then it could be a simple 'hack' to keep
your product working for the time being.

On Aug 7, 2:24 pm, Kyle Mulka <repalvigla...@yahoo.com> wrote:
> OAuth appears to be completely broken, along with my app which uses
> it. After clicking allow, Twitter takes forever to respond, and then
> when it does it just comes back with a completely blank response.
>
> --
> Kyle Mulkahttp://twilk.com

Vincent Nguyen

unread,
Aug 7, 2009, 9:49:35 PM8/7/09
to twitter-deve...@googlegroups.com
Stop  asking Twitter Team everybody!
Everyone has the same issue and Twitter is working hard to solve it!
Please be patiente!

2009/8/8 xzela <zelafe...@gmail.com>

Anupam

unread,
Aug 8, 2009, 1:29:50 AM8/8/09
to Twitter Development Talk
Thanks for the info Ryan,

we hope it would be over soon..

Anupam

unread,
Aug 8, 2009, 3:08:01 AM8/8/09
to Twitter Development Talk
Hey good news, my apps seems to perform ok again..

thanks for the superb job twitter :)

cheers,

Chris Babcock

unread,
Aug 8, 2009, 4:19:08 AM8/8/09
to twitter-deve...@googlegroups.com
On Fri, 7 Aug 2009 11:05:32 -0700
Ryan Sarver <rsa...@twitter.com> wrote:

> I wanted to send everyone an update to let you know what has been
> happening, the known issues, some suggestions on how to resolve them
> and some idea of how to move forward.

This was really appreciated. When the dust clears, maybe one more
suggestion? An API to check on the network status? I think
infrastucture attacks aren't going away anytime soon. We've got a
diversity of applications here, some of which can chew up the bandwidth
pretty well and some of those just don't make sense to run if other
users can't get on-line. Instead of answering, "Is it OK to restart my
cron jobs?" the cron jobs could shut themselves down for increments of
so many hours.

Chris Babcock

PS - Of course it could be misused, but I think the benefit is net.

signature.asc

Naveen Ayyagari

unread,
Aug 8, 2009, 5:54:09 AM8/8/09
to twitter-deve...@googlegroups.com
Chris ,
We implemented something like this "network status" using the
rate_limit_status call (for the IP), while some of the numbers are
sometimes wonky with this api right now we poll this every 5 minutes
and set a flag to enable or disable all twitter requests from the
server depending on what is returned. This has the effect of backing
off twitter when we get limited, as well as automatically bring
twitter service up when the blackout period subsides.. Even with the
wonky numbers, we blindly accept what the call returns and disable
when api credits say they are 0 and if the call times out you know for
sure its time to back off. ( I know these criteria are overly
aggressive, but we figure its better to aggressively back off in the
hopes it helps things recover quicker)

When the server is in blackout, simply report that twitter is still
having ongoing issues to the user with a note and a link to the
twitter status page, keeps them informed as to what is going on
without flooding your inbox or you having to respond to all the users
emails, If the message in the UI saves you from having to read through
and answer 10 emails, it is worth the work to implement. Its all about
keeping the user informed, just like we like to be informed by twitter
about what the status is.

Another tip is to make sure you set a resonable timeout for all your
twitter requests or people get stuck waiting for your page to load and
think its broken, better to timeout and again report that twitter is
still recovering for DDoS attack and direct them to the twitter status
page..

While this may seem like unnecessary work if twitter is going to get
back to normal, it makes your app more robust to issues in the future,
keeps your users better informed as to what is happening (without
contacting you), and also helps reduce number of requests to twitter
that they have to ignore while they are recovering

On that note, it seems our servers are disabling significantly less
frequently over the past few hours. Heres hoping it lasts.. Not sure
if it was the back off or if Twitter has just managed to cope better
with the traffic issues.. Doesn't really matter to me which..

Oh and thanks for the update Chad, it really is nice just to hear
something, even if its not great news.

dwight wallace

unread,
Aug 8, 2009, 3:33:49 PM8/8/09
to Twitter Development Talk
Great job :) Hopefully you can crate a security environment to
preclude future attacks.

On Aug 7, 11:05 am, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
>
Reply all
Reply to author
Forward
0 new messages