Timeouts and API Errors, Tuesday August 11th

6 views
Skip to first unread message

Alex Payne

unread,
Aug 11, 2009, 3:23:52 PM8/11/09
to Twitter Development Talk, twitter-ap...@googlegroups.com
We're currently experiencing another wave of Distributed Denial of Service (DDoS) attacks against our system. Expect periodic slowness and errors until the attack passes or is countered by our operations team and hosting provider. Updates will be provided as we get them.

Thanks for your patience.

--
Alex Payne - Platform Lead, Twitter, Inc.
http://twitter.com/al3x

Dewald Pretorius

unread,
Aug 11, 2009, 3:28:40 PM8/11/09
to Twitter Development Talk
I have a spare bazooka in my basement. Let me know. I can FedEx it to
you.

Dewald

Alex Payne

unread,
Aug 11, 2009, 4:11:51 PM8/11/09
to Twitter Development Talk, twitter-ap...@googlegroups.com
Our operations staff has informed me that the attack ceased several minutes ago. Site performance should be returning to normal.

Tim Haines

unread,
Aug 11, 2009, 4:19:57 PM8/11/09
to twitter-deve...@googlegroups.com
I'm so happy gmail has a star feature, that deserved one.

Tim.

Andrew Badera

unread,
Aug 11, 2009, 4:37:47 PM8/11/09
to twitter-deve...@googlegroups.com
On Tue, Aug 11, 2009 at 4:11 PM, Alex Payne<al...@twitter.com> wrote:
> Our operations staff has informed me that the attack ceased several minutes
> ago. Site performance should be returning to normal.
>
> On Tue, Aug 11, 2009 at 12:23, Alex Payne <al...@twitter.com> wrote:
>>
>> We're currently experiencing another wave of Distributed Denial of Service
>> (DDoS) attacks against our system. Expect periodic slowness and errors until
>> the attack passes or is countered by our operations team and hosting
>> provider. Updates will be provided as we get them.
>> Thanks for your patience.
>>
>> --
>> Alex Payne - Platform Lead, Twitter, Inc.
>> http://twitter.com/al3x

Has it resumed? Still getting lots of intermittency here.

∞ Andy Badera
∞ This email is: [ ] bloggable [x] ask first [ ] private
∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)

Dewald Pretorius

unread,
Aug 11, 2009, 4:54:18 PM8/11/09
to Twitter Development Talk
My guess is it's still ongoing. I'm seeing far more rejections per
second, and the number of backed-off retries have also increased.

Dewald

On Aug 11, 5:37 pm, Andrew Badera <and...@badera.us> wrote:
> On Tue, Aug 11, 2009 at 4:11 PM, Alex Payne<a...@twitter.com> wrote:
> > Our operations staff has informed me that the attack ceased several minutes
> > ago. Site performance should be returning to normal.
>

Alex Payne

unread,
Aug 11, 2009, 4:57:13 PM8/11/09
to twitter-deve...@googlegroups.com
Yes, I've just been informed that the attack has resumed, and that our service provider is putting network hardware in place to counter the attack. We're trying to work with them to ensure minimal impact to the API, but in the near term there may be issues with OAuth and the Streaming API.

This is a bit of a juggling act, as we're trying to coordinate our team, the operations team, our service provider's staff, and specialists that they've brought in for this issue. Please bear with us.

Sean Callahan

unread,
Aug 11, 2009, 4:58:50 PM8/11/09
to Twitter Development Talk
Alex,

Did not see this post and posted a new message. Still receiving lots
of errors and no one can login on our site, tweetphoto.com, right now
along with a handful of others (that I've tried myself). Just wanted
to give you a heads up. Thanks!

Sean

On Aug 11, 1:11 pm, Alex Payne <a...@twitter.com> wrote:
> Our operations staff has informed me that the attack ceased several minutes
> ago. Site performance should be returning to normal.
>

Alex Payne

unread,
Aug 11, 2009, 5:00:32 PM8/11/09
to twitter-deve...@googlegroups.com
We're aware of these issues; sorry.

Our ops team tells me that the countermeasures that are being put in place should not cause the 302 redirect behavior that impacted OAuth and other services late last week. If you're seeing that behavior, please post here and we'll coordinate with them to eliminate it.

Ryan Sarver

unread,
Aug 11, 2009, 5:02:17 PM8/11/09
to twitter-deve...@googlegroups.com
Also, please be sure to provide packet dumps and full headers where
possible so we can more easily determine the source of the issue.

Thanks, Ryan

Andrew Badera

unread,
Aug 11, 2009, 5:01:59 PM8/11/09
to twitter-deve...@googlegroups.com
On Tue, Aug 11, 2009 at 4:57 PM, Alex Payne<al...@twitter.com> wrote:
> Yes, I've just been informed that the attack has resumed, and that our
> service provider is putting network hardware in place to counter the attack.
> We're trying to work with them to ensure minimal impact to the API, but in
> the near term there may be issues with OAuth and the Streaming API.
> This is a bit of a juggling act, as we're trying to coordinate our team, the
> operations team, our service provider's staff, and specialists that they've
> brought in for this issue. Please bear with us.
>


Thanks for the update Alex. No worries here, you all seem to be
keeping us much more comfortably updated than most of Friday.

--ab

Andrew Badera

unread,
Aug 11, 2009, 5:03:11 PM8/11/09
to twitter-deve...@googlegroups.com


*Thursday, not Friday

Alex Payne

unread,
Aug 11, 2009, 5:22:40 PM8/11/09
to twitter-deve...@googlegroups.com
Just found out that our hosting provider put some hardware in place that may cause disruptions. Our operations team just spoke them, and they should be taking it down in 15 - 30 minutes.

Dewald Pretorius

unread,
Aug 11, 2009, 6:05:47 PM8/11/09
to Twitter Development Talk
For me the API behavior is now back again to this morning, with around
2 to 5 rejections per second, and most being successful on the first
or second backed-off retry.

Dewald

On Aug 11, 6:22 pm, Alex Payne <a...@twitter.com> wrote:
> Just found out that our hosting provider put some hardware in place that may
> cause disruptions. Our operations team just spoke them, and they should be
> taking it down in 15 - 30 minutes.
>
>
>
> On Tue, Aug 11, 2009 at 14:03, Andrew Badera <and...@badera.us> wrote:
>
> > On Tue, Aug 11, 2009 at 5:01 PM, Andrew Badera<and...@badera.us> wrote:
Reply all
Reply to author
Forward
0 new messages