I wanted to send everyone an update to let you know what has been happening, the known issues, some suggestions on how to resolve them and some idea of how to move forward.
*Whats been happening* As you know all too well Twitter, among other services, has been getting hit pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the attack come in a number of waves and from a number of different vectors increasing in intensity along the way. We were able to stabilize our own service for a bit, hence Biz's post saying all was well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>, but that didn't mean the attacks had ceased. In fact, at around 3am PST today the attacks intensified to almost 10x of what it was yesterday. In order for us to defend from the attack we have had to put a number of services in place and we know that some of you have gotten caught in the crossfire. Please know we are as frustrated as you are and wish there was more we could have communicated along the way.
*Known Issues* * - HTTP 300 response codes* - One of the measures in thwarting the onslaught requires that all traffic respect HTTP 30x response codes. This will help us identify the good traffic from the bad. * - General throttling* - Try to throttle your services back as much as possible for you to continue operating. We are working on our end to better understand the logic used in throttling traffic on the edge of the network and will communicate what we can, but the best idea is to just throttle back as much as you can in the mean time. * - Streaming API* - as part of the edge throttling we know requests to the Streaming API with lists of keywords or uses are getting dropped because the request is too large. We are working to get this filter removed and will update the list when we know more. - *Unexpected HTTP response codes* - we know people are seeing a lot of other weirdness and we aren't exactly sure what to attribute the various issues to, but know that you aren't alone.
As the attacks change our tactics for defense will likely need to change as well, so stay active on the list and let us know what problems you are seeing and we will do our best to help guide you along.
*Moving forward * We will try to communicate as much as we can so you guys are up to speed as things change and progress. I personally apologize for not communicating more in the mean time but there hasn't been much guidance we have been able to give other than hold tight with us. We fully appreciate all the long hours you are putting in to keep your apps running and supporting your users and know we are frustrated with you. Continue to watch this list, status.twitter.com and @twitterapi for updates
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
> Thanks for the update, however PLEASE get oAuth back up and running
> ASAP please!
> On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > I wanted to send everyone an update to let you know what has been happening,
> > the known issues, some suggestions on how to resolve them and some idea of
> > how to move forward.
> > *Whats been happening*
> > As you know all too well Twitter, among other services, has been getting hit
> > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > attack come in a number of waves and from a number of different vectors
> > increasing in intensity along the way. We were able to stabilize our own
> > service for a bit, hence Biz's post saying all was
> > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > today the attacks intensified to almost 10x of what it was yesterday. In
> > order for us to defend from the attack we have had to put a number of
> > services in place and we know that some of you have gotten caught in the
> > crossfire. Please know we are as frustrated as you are and wish there was
> > more we could have communicated along the way.
> > *Known Issues*
> > * - HTTP 300 response codes* - One of the measures in thwarting the
> > onslaught requires that all traffic respect HTTP 30x response codes. This
> > will help us identify the good traffic from the bad.
> > * - General throttling* - Try to throttle your services back as much as
> > possible for you to continue operating. We are working on our end to better
> > understand the logic used in throttling traffic on the edge of the network
> > and will communicate what we can, but the best idea is to just throttle back
> > as much as you can in the mean time.
> > * - Streaming API* - as part of the edge throttling we know requests to the
> > Streaming API with lists of keywords or uses are getting dropped because the
> > request is too large. We are working to get this filter removed and will
> > update the list when we know more.
> > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > other weirdness and we aren't exactly sure what to attribute the various
> > issues to, but know that you aren't alone.
> > As the attacks change our tactics for defense will likely need to change as
> > well, so stay active on the list and let us know what problems you are
> > seeing and we will do our best to help guide you along.
> > *Moving forward *
> > We will try to communicate as much as we can so you guys are up to speed as
> > things change and progress. I personally apologize for not communicating
> > more in the mean time but there hasn't been much guidance we have been able
> > to give other than hold tight with us. We fully appreciate all the long
> > hours you are putting in to keep your apps running and supporting your users
> > and know we are frustrated with you. Continue to watch this list,
> > status.twitter.com and @twitterapi for updates
Applications in cloud hosting environments may be unable to throttle
anything, due to the fact that if it's IP based checking, the cloud
IPs are stlll going to be sending a lot of requests. ie: Appengine
applications.
On Aug 7, 2:28 pm, Rich <rhyl...@gmail.com> wrote:
> Thanks for the update, however PLEASE get oAuth back up and running
> ASAP please!
> On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > I wanted to send everyone an update to let you know what has been happening,
> > the known issues, some suggestions on how to resolve them and some idea of
> > how to move forward.
> > *Whats been happening*
> > As you know all too well Twitter, among other services, has been getting hit
> > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > attack come in a number of waves and from a number of different vectors
> > increasing in intensity along the way. We were able to stabilize our own
> > service for a bit, hence Biz's post saying all was
> > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > today the attacks intensified to almost 10x of what it was yesterday. In
> > order for us to defend from the attack we have had to put a number of
> > services in place and we know that some of you have gotten caught in the
> > crossfire. Please know we are as frustrated as you are and wish there was
> > more we could have communicated along the way.
> > *Known Issues*
> > * - HTTP 300 response codes* - One of the measures in thwarting the
> > onslaught requires that all traffic respect HTTP 30x response codes. This
> > will help us identify the good traffic from the bad.
> > * - General throttling* - Try to throttle your services back as much as
> > possible for you to continue operating. We are working on our end to better
> > understand the logic used in throttling traffic on the edge of the network
> > and will communicate what we can, but the best idea is to just throttle back
> > as much as you can in the mean time.
> > * - Streaming API* - as part of the edge throttling we know requests to the
> > Streaming API with lists of keywords or uses are getting dropped because the
> > request is too large. We are working to get this filter removed and will
> > update the list when we know more.
> > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > other weirdness and we aren't exactly sure what to attribute the various
> > issues to, but know that you aren't alone.
> > As the attacks change our tactics for defense will likely need to change as
> > well, so stay active on the list and let us know what problems you are
> > seeing and we will do our best to help guide you along.
> > *Moving forward *
> > We will try to communicate as much as we can so you guys are up to speed as
> > things change and progress. I personally apologize for not communicating
> > more in the mean time but there hasn't been much guidance we have been able
> > to give other than hold tight with us. We fully appreciate all the long
> > hours you are putting in to keep your apps running and supporting your users
> > and know we are frustrated with you. Continue to watch this list,
> > status.twitter.com and @twitterapi for updates
oAuth worked for me on testing this morning, but trying to
authenticate three seperate accounts, right now... all of them timeout
on clicking the 'Allow' button
On Aug 7, 7:32 pm, Goblin <stu...@abovetheinternet.org> wrote:
> OAuth is working fine for my site. To be honest, for something that
> does nothing but interact with Twitter I haven't seen much of a drop
> in activity.
> On Aug 7, 7:28 pm, Rich <rhyl...@gmail.com> wrote:
> > Thanks for the update, however PLEASE get oAuth back up and running
> > ASAP please!
> > On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > > I wanted to send everyone an update to let you know what has been happening,
> > > the known issues, some suggestions on how to resolve them and some idea of
> > > how to move forward.
> > > *Whats been happening*
> > > As you know all too well Twitter, among other services, has been getting hit
> > > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > > attack come in a number of waves and from a number of different vectors
> > > increasing in intensity along the way. We were able to stabilize our own
> > > service for a bit, hence Biz's post saying all was
> > > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > > today the attacks intensified to almost 10x of what it was yesterday. In
> > > order for us to defend from the attack we have had to put a number of
> > > services in place and we know that some of you have gotten caught in the
> > > crossfire. Please know we are as frustrated as you are and wish there was
> > > more we could have communicated along the way.
> > > *Known Issues*
> > > * - HTTP 300 response codes* - One of the measures in thwarting the
> > > onslaught requires that all traffic respect HTTP 30x response codes. This
> > > will help us identify the good traffic from the bad.
> > > * - General throttling* - Try to throttle your services back as much as
> > > possible for you to continue operating. We are working on our end to better
> > > understand the logic used in throttling traffic on the edge of the network
> > > and will communicate what we can, but the best idea is to just throttle back
> > > as much as you can in the mean time.
> > > * - Streaming API* - as part of the edge throttling we know requests to the
> > > Streaming API with lists of keywords or uses are getting dropped because the
> > > request is too large. We are working to get this filter removed and will
> > > update the list when we know more.
> > > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > > other weirdness and we aren't exactly sure what to attribute the various
> > > issues to, but know that you aren't alone.
> > > As the attacks change our tactics for defense will likely need to change as
> > > well, so stay active on the list and let us know what problems you are
> > > seeing and we will do our best to help guide you along.
> > > *Moving forward *
> > > We will try to communicate as much as we can so you guys are up to speed as
> > > things change and progress. I personally apologize for not communicating
> > > more in the mean time but there hasn't been much guidance we have been able
> > > to give other than hold tight with us. We fully appreciate all the long
> > > hours you are putting in to keep your apps running and supporting your users
> > > and know we are frustrated with you. Continue to watch this list,
> > > status.twitter.com and @twitterapi for updates
> oAuth worked for me on testing this morning, but trying to
> authenticate three seperate accounts, right now... all of them timeout
> on clicking the 'Allow' button
> On Aug 7, 7:32 pm, Goblin <stu...@abovetheinternet.org> wrote:
> > OAuth is working fine for my site. To be honest, for something that
> > does nothing but interact with Twitter I haven't seen much of a drop
> > in activity.
> > On Aug 7, 7:28 pm, Rich <rhyl...@gmail.com> wrote:
> > > Thanks for the update, however PLEASE get oAuth back up and running
> > > ASAP please!
> > > On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > > > I wanted to send everyone an update to let you know what has been happening,
> > > > the known issues, some suggestions on how to resolve them and some idea of
> > > > how to move forward.
> > > > *Whats been happening*
> > > > As you know all too well Twitter, among other services, has been getting hit
> > > > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > > > attack come in a number of waves and from a number of different vectors
> > > > increasing in intensity along the way. We were able to stabilize our own
> > > > service for a bit, hence Biz's post saying all was
> > > > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > > > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > > > today the attacks intensified to almost 10x of what it was yesterday. In
> > > > order for us to defend from the attack we have had to put a number of
> > > > services in place and we know that some of you have gotten caught in the
> > > > crossfire. Please know we are as frustrated as you are and wish there was
> > > > more we could have communicated along the way.
> > > > *Known Issues*
> > > > * - HTTP 300 response codes* - One of the measures in thwarting the
> > > > onslaught requires that all traffic respect HTTP 30x response codes. This
> > > > will help us identify the good traffic from the bad.
> > > > * - General throttling* - Try to throttle your services back as much as
> > > > possible for you to continue operating. We are working on our end to better
> > > > understand the logic used in throttling traffic on the edge of the network
> > > > and will communicate what we can, but the best idea is to just throttle back
> > > > as much as you can in the mean time.
> > > > * - Streaming API* - as part of the edge throttling we know requests to the
> > > > Streaming API with lists of keywords or uses are getting dropped because the
> > > > request is too large. We are working to get this filter removed and will
> > > > update the list when we know more.
> > > > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > > > other weirdness and we aren't exactly sure what to attribute the various
> > > > issues to, but know that you aren't alone.
> > > > As the attacks change our tactics for defense will likely need to change as
> > > > well, so stay active on the list and let us know what problems you are
> > > > seeing and we will do our best to help guide you along.
> > > > *Moving forward *
> > > > We will try to communicate as much as we can so you guys are up to speed as
> > > > things change and progress. I personally apologize for not communicating
> > > > more in the mean time but there hasn't been much guidance we have been able
> > > > to give other than hold tight with us. We fully appreciate all the long
> > > > hours you are putting in to keep your apps running and supporting your users
> > > > and know we are frustrated with you. Continue to watch this list,
> > > > status.twitter.com and @twitterapi for updates
> Clicking Allow - just causes the App to timeout.
> This reminds of the OAuth outage we had last time - which begs the > question, is OAuth ready for production applications?
> On Aug 7, 2:38 pm, Rich <rhyl...@gmail.com> wrote: > > oAuth worked for me on testing this morning, but trying to > > authenticate three seperate accounts, right now... all of them timeout > > on clicking the 'Allow' button
> > On Aug 7, 7:32 pm, Goblin <stu...@abovetheinternet.org> wrote:
> > > OAuth is working fine for my site. To be honest, for something that > > > does nothing but interact with Twitter I haven't seen much of a drop > > > in activity.
> > > On Aug 7, 7:28 pm, Rich <rhyl...@gmail.com> wrote:
> > > > Thanks for the update, however PLEASE get oAuth back up and running > > > > ASAP please!
> > > > On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > > > > I wanted to send everyone an update to let you know what has been > happening, > > > > > the known issues, some suggestions on how to resolve them and some > idea of > > > > > how to move forward.
> > > > > *Whats been happening* > > > > > As you know all too well Twitter, among other services, has been > getting hit > > > > > pretty hard with a DDoS attack over the past 24+ hours. Yesterday > we saw the > > > > > attack come in a number of waves and from a number of different > vectors > > > > > increasing in intensity along the way. We were able to stabilize > our own > > > > > service for a bit, hence Biz's post saying all was > > > > > well< > http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>, > > > > > but that didn't mean the attacks had ceased. In fact, at around 3am > PST > > > > > today the attacks intensified to almost 10x of what it was > yesterday. In > > > > > order for us to defend from the attack we have had to put a number > of > > > > > services in place and we know that some of you have gotten caught > in the > > > > > crossfire. Please know we are as frustrated as you are and wish > there was > > > > > more we could have communicated along the way.
> > > > > *Known Issues* > > > > > * - HTTP 300 response codes* - One of the measures in thwarting the > > > > > onslaught requires that all traffic respect HTTP 30x response > codes. This > > > > > will help us identify the good traffic from the bad. > > > > > * - General throttling* - Try to throttle your services back as > much as > > > > > possible for you to continue operating. We are working on our end > to better > > > > > understand the logic used in throttling traffic on the edge of the > network > > > > > and will communicate what we can, but the best idea is to just > throttle back > > > > > as much as you can in the mean time. > > > > > * - Streaming API* - as part of the edge throttling we know > requests to the > > > > > Streaming API with lists of keywords or uses are getting dropped > because the > > > > > request is too large. We are working to get this filter removed and > will > > > > > update the list when we know more. > > > > > - *Unexpected HTTP response codes* - we know people are seeing a > lot of > > > > > other weirdness and we aren't exactly sure what to attribute the > various > > > > > issues to, but know that you aren't alone.
> > > > > As the attacks change our tactics for defense will likely need to > change as > > > > > well, so stay active on the list and let us know what problems you > are > > > > > seeing and we will do our best to help guide you along.
> > > > > *Moving forward * > > > > > We will try to communicate as much as we can so you guys are up to > speed as > > > > > things change and progress. I personally apologize for not > communicating > > > > > more in the mean time but there hasn't been much guidance we have > been able > > > > > to give other than hold tight with us. We fully appreciate all the > long > > > > > hours you are putting in to keep your apps running and supporting > your users > > > > > and know we are frustrated with you. Continue to watch this list, > > > > > status.twitter.com and @twitterapi for updates
> Clicking Allow - just causes the App to timeout.
> This reminds of the OAuth outage we had last time - which begs the
> question, is OAuth ready for production applications?
> On Aug 7, 2:38 pm, Rich <rhyl...@gmail.com> wrote:
> > oAuth worked for me on testing this morning, but trying to
> > authenticate three seperate accounts, right now... all of them timeout
> > on clicking the 'Allow' button
> > On Aug 7, 7:32 pm, Goblin <stu...@abovetheinternet.org> wrote:
> > > OAuth is working fine for my site. To be honest, for something that
> > > does nothing but interact with Twitter I haven't seen much of a drop
> > > in activity.
> > > On Aug 7, 7:28 pm, Rich <rhyl...@gmail.com> wrote:
> > > > Thanks for the update, however PLEASE get oAuth back up and running
> > > > ASAP please!
> > > > On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > > > > I wanted to send everyone an update to let you know what has been happening,
> > > > > the known issues, some suggestions on how to resolve them and some idea of
> > > > > how to move forward.
> > > > > *Whats been happening*
> > > > > As you know all too well Twitter, among other services, has been getting hit
> > > > > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > > > > attack come in a number of waves and from a number of different vectors
> > > > > increasing in intensity along the way. We were able to stabilize our own
> > > > > service for a bit, hence Biz's post saying all was
> > > > > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > > > > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > > > > today the attacks intensified to almost 10x of what it was yesterday. In
> > > > > order for us to defend from the attack we have had to put a number of
> > > > > services in place and we know that some of you have gotten caught in the
> > > > > crossfire. Please know we are as frustrated as you are and wish there was
> > > > > more we could have communicated along the way.
> > > > > *Known Issues*
> > > > > * - HTTP 300 response codes* - One of the measures in thwarting the
> > > > > onslaught requires that all traffic respect HTTP 30x response codes. This
> > > > > will help us identify the good traffic from the bad.
> > > > > * - General throttling* - Try to throttle your services back as much as
> > > > > possible for you to continue operating. We are working on our end to better
> > > > > understand the logic used in throttling traffic on the edge of the network
> > > > > and will communicate what we can, but the best idea is to just throttle back
> > > > > as much as you can in the mean time.
> > > > > * - Streaming API* - as part of the edge throttling we know requests to the
> > > > > Streaming API with lists of keywords or uses are getting dropped because the
> > > > > request is too large. We are working to get this filter removed and will
> > > > > update the list when we know more.
> > > > > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > > > > other weirdness and we aren't exactly sure what to attribute the various
> > > > > issues to, but know that you aren't alone.
> > > > > As the attacks change our tactics for defense will likely need to change as
> > > > > well, so stay active on the list and let us know what problems you are
> > > > > seeing and we will do our best to help guide you along.
> > > > > *Moving forward *
> > > > > We will try to communicate as much as we can so you guys are up to speed as
> > > > > things change and progress. I personally apologize for not communicating
> > > > > more in the mean time but there hasn't been much guidance we have been able
> > > > > to give other than hold tight with us. We fully appreciate all the long
> > > > > hours you are putting in to keep your apps running and supporting your users
> > > > > and know we are frustrated with you. Continue to watch this list,
> > > > > status.twitter.com and @twitterapi for updates
Thanks for the update Ryan. One thing I don't quite understand is why it's not an option to allow whitelisted applications to post. I will try and throttle our ( twitterfeed.com) service back, but with nearly half a million of active feeds in the system, I can't quite see how this will help, as even a fraction of requests will be way over any non-whitelist limits you have in place.
All my oauth requests are failing with an invalid token exception, and
the response to the request for the token appears to be null. This is
using the twitter python client and from appengine. I don't even get
to the point of redirecting users to the login page.
On Aug 7, 2:53 pm, Mario Menti <mme...@gmail.com> wrote:
> Thanks for the update Ryan.
> One thing I don't quite understand is why it's not an option to allow
> whitelisted applications to post. I will try and throttle our (
> twitterfeed.com) service back, but with nearly half a million of active
> feeds in the system, I can't quite see how this will help, as even a
> fraction of requests will be way over any non-whitelist limits you have in
> place.
Thanks for the communication - this is good. Just curious - with entire businesses being put out of place, and rumors that the Russian Gov't may be behind such attacks, is Twitter communicating with Homeland Security about this? To me this seems like a matter of national security even more than it is a Twitter issue. The US economy is being attacked because of this. Not to sound too radical - I'm just genuinely curious when the Government is going to get involved. (and thank you for doing what you can - I'm sure I speak for all when I say we feel your pain)
On Fri, Aug 7, 2009 at 2:05 PM, Ryan Sarver <rsar...@twitter.com> wrote: > I wanted to send everyone an update to let you know what has been > happening, the known issues, some suggestions on how to resolve them and > some idea of how to move forward.
> *Whats been happening* > As you know all too well Twitter, among other services, has been getting > hit pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw > the attack come in a number of waves and from a number of different vectors > increasing in intensity along the way. We were able to stabilize our own > service for a bit, hence Biz's post saying all was well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>, > but that didn't mean the attacks had ceased. In fact, at around 3am PST > today the attacks intensified to almost 10x of what it was yesterday. In > order for us to defend from the attack we have had to put a number of > services in place and we know that some of you have gotten caught in the > crossfire. Please know we are as frustrated as you are and wish there was > more we could have communicated along the way.
> *Known Issues* > * - HTTP 300 response codes* - One of the measures in thwarting the > onslaught requires that all traffic respect HTTP 30x response codes. This > will help us identify the good traffic from the bad. > * - General throttling* - Try to throttle your services back as much as > possible for you to continue operating. We are working on our end to better > understand the logic used in throttling traffic on the edge of the network > and will communicate what we can, but the best idea is to just throttle back > as much as you can in the mean time. > * - Streaming API* - as part of the edge throttling we know requests to > the Streaming API with lists of keywords or uses are getting dropped because > the request is too large. We are working to get this filter removed and will > update the list when we know more. > - *Unexpected HTTP response codes* - we know people are seeing a lot of > other weirdness and we aren't exactly sure what to attribute the various > issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to change as > well, so stay active on the list and let us know what problems you are > seeing and we will do our best to help guide you along.
> *Moving forward * > We will try to communicate as much as we can so you guys are up to speed as > things change and progress. I personally apologize for not communicating > more in the mean time but there hasn't been much guidance we have been able > to give other than hold tight with us. We fully appreciate all the long > hours you are putting in to keep your apps running and supporting your users > and know we are frustrated with you. Continue to watch this list, > status.twitter.com and @twitterapi for updates
> Thanks for the communication - this is good. Just curious - with entire
> businesses being put out of place, and rumors that the Russian Gov't may be
> behind such attacks, is Twitter communicating with Homeland Security about
> this? To me this seems like a matter of national security even more than it
> is a Twitter issue. The US economy is being attacked because of this.
> Not to sound too radical - I'm just genuinely curious when the Government is
> going to get involved. (and thank you for doing what you can - I'm sure I
> speak for all when I say we feel your pain)
> Jesse
> On Fri, Aug 7, 2009 at 2:05 PM, Ryan Sarver <rsar...@twitter.com> wrote:
> > I wanted to send everyone an update to let you know what has been
> > happening, the known issues, some suggestions on how to resolve them and
> > some idea of how to move forward.
> > *Whats been happening*
> > As you know all too well Twitter, among other services, has been getting
> > hit pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw
> > the attack come in a number of waves and from a number of different vectors
> > increasing in intensity along the way. We were able to stabilize our own
> > service for a bit, hence Biz's post saying all was well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > today the attacks intensified to almost 10x of what it was yesterday. In
> > order for us to defend from the attack we have had to put a number of
> > services in place and we know that some of you have gotten caught in the
> > crossfire. Please know we are as frustrated as you are and wish there was
> > more we could have communicated along the way.
> > *Known Issues*
> > * - HTTP 300 response codes* - One of the measures in thwarting the
> > onslaught requires that all traffic respect HTTP 30x response codes. This
> > will help us identify the good traffic from the bad.
> > * - General throttling* - Try to throttle your services back as much as
> > possible for you to continue operating. We are working on our end to better
> > understand the logic used in throttling traffic on the edge of the network
> > and will communicate what we can, but the best idea is to just throttle back
> > as much as you can in the mean time.
> > * - Streaming API* - as part of the edge throttling we know requests to
> > the Streaming API with lists of keywords or uses are getting dropped because
> > the request is too large. We are working to get this filter removed and will
> > update the list when we know more.
> > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > other weirdness and we aren't exactly sure what to attribute the various
> > issues to, but know that you aren't alone.
> > As the attacks change our tactics for defense will likely need to change as
> > well, so stay active on the list and let us know what problems you are
> > seeing and we will do our best to help guide you along.
> > *Moving forward *
> > We will try to communicate as much as we can so you guys are up to speed as
> > things change and progress. I personally apologize for not communicating
> > more in the mean time but there hasn't been much guidance we have been able
> > to give other than hold tight with us. We fully appreciate all the long
> > hours you are putting in to keep your apps running and supporting your users
> > and know we are frustrated with you. Continue to watch this list,
> > status.twitter.com and @twitterapi for updates
> Thanks for the communication - this is good. Just curious - with > entire businesses being put out of place, and rumors that the > Russian Gov't may be behind such attacks, is Twitter communicating > with Homeland Security about this? To me this seems like a matter > of national security even more than it is a Twitter issue. The US > economy is being attacked because of this.
> Not to sound too radical - I'm just genuinely curious when the > Government is going to get involved. (and thank you for doing what > you can - I'm sure I speak for all when I say we feel your pain)
> Jesse
> On Fri, Aug 7, 2009 at 2:05 PM, Ryan Sarver <rsar...@twitter.com> > wrote: > I wanted to send everyone an update to let you know what has been > happening, the known issues, some suggestions on how to resolve them > and some idea of how to move forward.
> Whats been happening > As you know all too well Twitter, among other services, has been > getting hit pretty hard with a DDoS attack over the past 24+ hours. > Yesterday we saw the attack come in a number of waves and from a > number of different vectors increasing in intensity along the way. > We were able to stabilize our own service for a bit, hence Biz's > post saying all was well, but that didn't mean the attacks had > ceased. In fact, at around 3am PST today the attacks intensified to > almost 10x of what it was yesterday. In order for us to defend from > the attack we have had to put a number of services in place and we > know that some of you have gotten caught in the crossfire. Please > know we are as frustrated as you are and wish there was more we > could have communicated along the way.
> Known Issues > - HTTP 300 response codes - One of the measures in thwarting the > onslaught requires that all traffic respect HTTP 30x response codes. > This will help us identify the good traffic from the bad. > - General throttling - Try to throttle your services back as much > as possible for you to continue operating. We are working on our end > to better understand the logic used in throttling traffic on the > edge of the network and will communicate what we can, but the best > idea is to just throttle back as much as you can in the mean time. > - Streaming API - as part of the edge throttling we know requests > to the Streaming API with lists of keywords or uses are getting > dropped because the request is too large. We are working to get this > filter removed and will update the list when we know more. > - Unexpected HTTP response codes - we know people are seeing a lot > of other weirdness and we aren't exactly sure what to attribute the > various issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to > change as well, so stay active on the list and let us know what > problems you are seeing and we will do our best to help guide you > along.
> Moving forward > We will try to communicate as much as we can so you guys are up to > speed as things change and progress. I personally apologize for not > communicating more in the mean time but there hasn't been much > guidance we have been able to give other than hold tight with us. We > fully appreciate all the long hours you are putting in to keep your > apps running and supporting your users and know we are frustrated > with you. Continue to watch this list, status.twitter.com and > @twitterapi for updates
I have still a problem with getting search results via curl like
described here: http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search This was working pretty good before the DDoS attack, but now I don't
get any results just http_code of 302.
First, thanks for finally posting such a message. It has been pretty
frustrating when there is no communication for you guys. Especially
when we developers rely on your service and you also rely on us
promoting your service. It makes us third party developers look stupid
when Biz/Twitter tells half truths to the public says that everything
is running great when in reality it is not. I understand you want to
look like things are fine so people don't loose confidence in your
service but it doesn't say much to us developers and give us much
confidence about your transparency and what the actual truth you tell
us is. If anyone should be updated it should be your developer
community first then the users, you owe us that much as we have built
your service up ever user we send your way. How about you make a more
public announcement to all your users to let them know that your
issues are affecting all the third party vendors and the issues is not
fixed.
Now that I have been able to finally get a message through to someone
at Twitter, how about getting oAuth working so that our users can at
least login consistently to our applications. Right now oAuth login
works some times and not others. You make us move over to something
that obviously is not production ready.
Also why are us white listed apps having troubles and getting a rate
limit of 150 like average users. I understand you want us to throttle
it back but, can't you give us more than that? Basically your idea of
throttling is cutting our usual allowed consumption down to less than
1%!? Serious, why not just shut it off entirely.
Chris-
On Aug 7, 2:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
> Thanks for the update Ryan.
> One thing I don't quite understand is why it's not an option to allow
> whitelisted applications to post. I will try and throttle our (
> twitterfeed.com) service back, but with nearly half a million of active
> feeds in the system, I can't quite see how this will help, as even a
> fraction of requests will be way over any non-whitelist limits you have in
> place.
> Clicking Allow - just causes the App to timeout.
> This reminds of the OAuth outage we had last time - which begs the
> question, is OAuth ready for production applications?
> On Aug 7, 2:38 pm, Rich <rhyl...@gmail.com> wrote:
> > oAuth worked for me on testing this morning, but trying to
> > authenticate three seperate accounts, right now... all of them timeout
> > on clicking the 'Allow' button
> > On Aug 7, 7:32 pm, Goblin <stu...@abovetheinternet.org> wrote:
> > > OAuth is working fine for my site. To be honest, for something that
> > > does nothing but interact with Twitter I haven't seen much of a drop
> > > in activity.
> > > On Aug 7, 7:28 pm, Rich <rhyl...@gmail.com> wrote:
> > > > Thanks for the update, however PLEASE get oAuth back up and running
> > > > ASAP please!
> > > > On Aug 7, 7:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> > > > > I wanted to send everyone an update to let you know what has been happening,
> > > > > the known issues, some suggestions on how to resolve them and some idea of
> > > > > how to move forward.
> > > > > *Whats been happening*
> > > > > As you know all too well Twitter, among other services, has been getting hit
> > > > > pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> > > > > attack come in a number of waves and from a number of different vectors
> > > > > increasing in intensity along the way. We were able to stabilize our own
> > > > > service for a bit, hence Biz's post saying all was
> > > > > well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> > > > > but that didn't mean the attacks had ceased. In fact, at around 3am PST
> > > > > today the attacks intensified to almost 10x of what it was yesterday. In
> > > > > order for us to defend from the attack we have had to put a number of
> > > > > services in place and we know that some of you have gotten caught in the
> > > > > crossfire. Please know we are as frustrated as you are and wish there was
> > > > > more we could have communicated along the way.
> > > > > *Known Issues*
> > > > > * - HTTP 300 response codes* - One of the measures in thwarting the
> > > > > onslaught requires that all traffic respect HTTP 30x response codes. This
> > > > > will help us identify the good traffic from the bad.
> > > > > * - General throttling* - Try to throttle your services back as much as
> > > > > possible for you to continue operating. We are working on our end to better
> > > > > understand the logic used in throttling traffic on the edge of the network
> > > > > and will communicate what we can, but the best idea is to just throttle back
> > > > > as much as you can in the mean time.
> > > > > * - Streaming API* - as part of the edge throttling we know requests to the
> > > > > Streaming API with lists of keywords or uses are getting dropped because the
> > > > > request is too large. We are working to get this filter removed and will
> > > > > update the list when we know more.
> > > > > - *Unexpected HTTP response codes* - we know people are seeing a lot of
> > > > > other weirdness and we aren't exactly sure what to attribute the various
> > > > > issues to, but know that you aren't alone.
> > > > > As the attacks change our tactics for defense will likely need to change as
> > > > > well, so stay active on the list and let us know what problems you are
> > > > > seeing and we will do our best to help guide you along.
> > > > > *Moving forward *
> > > > > We will try to communicate as much as we can so you guys are up to speed as
> > > > > things change and progress. I personally apologize for not communicating
> > > > > more in the mean time but there hasn't been much guidance we have been able
> > > > > to give other than hold tight with us. We fully appreciate all the long
> > > > > hours you are putting in to keep your apps running and supporting your users
> > > > > and know we are frustrated with you. Continue to watch this list,
> > > > > status.twitter.com and @twitterapi for updates
> All my oauth requests are failing with an invalid token exception, and
> the response to the request for the token appears to be null. This is
> using the twitter python client and from appengine. I don't even get
> to the point of redirecting users to the login page.
> On Aug 7, 2:53 pm, Mario Menti <mme...@gmail.com> wrote:
> > Thanks for the update Ryan.
> > One thing I don't quite understand is why it's not an option to allow
> > whitelisted applications to post. I will try and throttle our (
> > twitterfeed.com) service back, but with nearly half a million of active
> > feeds in the system, I can't quite see how this will help, as even a
> > fraction of requests will be way over any non-whitelist limits you have in
> > place.
On Fri, Aug 7, 2009 at 12:02 PM, Jesse Stay <jesses...@gmail.com> wrote: > Thanks for the communication - this is good. Just curious - with entire > businesses being put out of place, and rumors that the Russian Gov't may be > behind such attacks, is Twitter communicating with Homeland Security about > this? To me this seems like a matter of national security even more than it > is a Twitter issue. The US economy is being attacked because of this. > Not to sound too radical - I'm just genuinely curious when the Government > is going to get involved. (and thank you for doing what you can - I'm sure > I speak for all when I say we feel your pain)
They probably already are and almost certainly will not say so, at least for a while.
On Aug 7, 12:05 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
Does this affect POST as well as GET? The issue here is the way
clients handle 30x after POST. Most clients (now by convention) do
not respect the RFC (http://www.w3.org/Protocols/rfc2616/rfc2616- sec10.html#sec10.3) and will send a GET after POST always. Some
clients will respect the method, but not re-post any data. We need to
be sure we are all expecting the right things.
How does this affect OAuth signed requests? OAuth requires knowing
the HTTP Method as well as the full URI and parameters to generate
signatures a priori. Most (all?) clients and libraries do not know
how to handle changes in these during a redirect.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
Is there anything else we can do on the protocol level? Keepalives on/
off? Specific headers?
Thanks for all your hard work. Having spent yesterday battling a rash
of spambots is nothing compared to what you guys are doing.
On Fri, Aug 7, 2009 at 3:10 PM, Chris<kiraili...@gmail.com> wrote:
> Thank you for updating us!
> I have still a problem with getting search results via curl like > described here: http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search > This was working pretty good before the DDoS attack, but now I don't > get any results just http_code of 302.
We have multiple servers running and they are getting different
response codes. some servers getting 302 for GETs and 408 for POSTs,
other servers getting 503s...
We can modify the code to respect 302s, but what about 503s?
On Aug 7, 11:05 am, Ryan Sarver <rsar...@twitter.com> wrote:
> I wanted to send everyone an update to let you know what has been happening,
> the known issues, some suggestions on how to resolve them and some idea of
> how to move forward.
> *Whats been happening*
> As you know all too well Twitter, among other services, has been getting hit
> pretty hard with a DDoS attack over the past 24+ hours. Yesterday we saw the
> attack come in a number of waves and from a number of different vectors
> increasing in intensity along the way. We were able to stabilize our own
> service for a bit, hence Biz's post saying all was
> well<http://blog.twitter.com/2009/08/update-on-todays-dos-attacks.html>,
> but that didn't mean the attacks had ceased. In fact, at around 3am PST
> today the attacks intensified to almost 10x of what it was yesterday. In
> order for us to defend from the attack we have had to put a number of
> services in place and we know that some of you have gotten caught in the
> crossfire. Please know we are as frustrated as you are and wish there was
> more we could have communicated along the way.
> *Known Issues*
> * - HTTP 300 response codes* - One of the measures in thwarting the
> onslaught requires that all traffic respect HTTP 30x response codes. This
> will help us identify the good traffic from the bad.
> * - General throttling* - Try to throttle your services back as much as
> possible for you to continue operating. We are working on our end to better
> understand the logic used in throttling traffic on the edge of the network
> and will communicate what we can, but the best idea is to just throttle back
> as much as you can in the mean time.
> * - Streaming API* - as part of the edge throttling we know requests to the
> Streaming API with lists of keywords or uses are getting dropped because the
> request is too large. We are working to get this filter removed and will
> update the list when we know more.
> - *Unexpected HTTP response codes* - we know people are seeing a lot of
> other weirdness and we aren't exactly sure what to attribute the various
> issues to, but know that you aren't alone.
> As the attacks change our tactics for defense will likely need to change as
> well, so stay active on the list and let us know what problems you are
> seeing and we will do our best to help guide you along.
> *Moving forward *
> We will try to communicate as much as we can so you guys are up to speed as
> things change and progress. I personally apologize for not communicating
> more in the mean time but there hasn't been much guidance we have been able
> to give other than hold tight with us. We fully appreciate all the long
> hours you are putting in to keep your apps running and supporting your users
> and know we are frustrated with you. Continue to watch this list,
> status.twitter.com and @twitterapi for updates
Inline about 302's. I'm trying to get info on your other questions:
On Fri, Aug 7, 2009 at 3:27 PM, Justin Hart<onyxra...@gmail.com> wrote: > Does this affect POST as well as GET? The issue here is the way > clients handle 30x after POST. Most clients (now by convention) do > not respect the RFC (http://www.w3.org/Protocols/rfc2616/rfc2616- > sec10.html#sec10.3) and will send a GET after POST always. Some > clients will respect the method, but not re-post any data. We need to > be sure we are all expecting the right things.
You are correct that lots of clients now redirect with a GET instead of a POST, even curl does this, which is a bummer.
The best thing to do in that case is to catch the response code (without following the redirect automatically) and manually re-attempt the POST with the new location.
We know it's a pain, but that's one way around the POST->GET problem.
|
