Twitter Update, 8/9 10am PST

Skip to first unread message

Ryan Sarver

Aug 9, 2009, 1:34:14 PM8/9/09
I wanted to send an update to everyone who is monitoring this thread and keep you abreast of where we stand.

First of all, the attack is still on going. We continue to work with our service provider and the other companies who are being attacked to resolve the issue as best we can. But it is continuing and as Chad stated, we are thus going to have continued and varied issues with access to the Twitter APIs.

Second, we have been monitoring every email and Chad and John have done a great job responding to the threads where we have new information to add or where we can clarify. As Chad stated in his last email there are known issues that we are continuing to work on. We are trying to provide updates when we have new information, but sadly there isn't a lot new to report. Please know that we can't respond to each and every thread as our teams time, in your best interest, should be put towards getting you all back online as soon as possible. Please help each other answer questions if someone already knows the answer. I will continue to give ongoing updates every 5-6 hours throughout the day even if nothing has changed so that you know we are still focused on it.

To be clear we have a large group of people working on this throughout the weekend and our developer ecosystem is incredibly important to us. Please believe us that this hurts us as much as it hurts you and we want to get you back online as soon as possible. We are endlessly appreciative for the hard work that each of you put into your applications and supporting Twitter users and we want those services to be online and back to normal.

With that being said, this attack is continuing, almost 3 days later, and we need to continue to do all that we can to defend it. Have faith that we have the best people on it both at Twitter and at our partner's companies.

With all of the emails flying around, its hard for us to track exactly what issues are still remaining so I wanted to try to capture them here and have you confirm or update me if new issues have cropped up or if others are passed. So please let me know which of the following issues you are still having or if there are any I missed:

1. OAuth rarely works - I tried a number of your apps and it seems to work 1 out of 6-7 times. As a note, it worked better with Safari, but not every time.
2. 302 redirect
3. General request timeouts
4. HTML in responses
5. Unexpected rate limiting / blacklisting

So again, thank you for your patience and know that we are hard at work to get you guys back up and running.

Best, Ryan

Dewald Pretorius

Aug 9, 2009, 1:42:41 PM8/9/09
to Twitter Development Talk
On Aug 9, 2:34 pm, Ryan Sarver <> wrote:
>I will continue to give ongoing updates
> every 5-6 hours throughout the day even if nothing has changed so that you
> know we are still focused on it.

Now THAT'S what we're talking about!

Thank you Ryan. It may not seem important to busy Twitter folks to
report status even when there is nothing new to report, but with an
outage of this duration it is absolutely essential, because it keeps
everyone's temperatures down.


Adam Cloud

Aug 9, 2009, 1:52:06 PM8/9/09

Neil Ellis

Aug 9, 2009, 1:52:04 PM8/9/09
It is true, perception at this time is very important - even more

Annoyed people can make hasty choices, write unfavorable articles etc.
So as crappy as it is to keep feeding us all updates, it's worth it a
100 fold in the long run especially since Twitter's business is based
effectively on kudos (since it is not technically unique to my


Duane Roelands

Aug 9, 2009, 1:53:33 PM8/9/09
to Twitter Development Talk
When calling via OAuth
(no id passed, looking for information on the authenticated user), I
am seeing this response:
<?xml version="1.0" encoding="UTF-8"?> <statuses type="array"> </

Not sure if that's in the list of things being monitored, but there
you go.

Naveen Ayyagari

Aug 9, 2009, 2:13:05 PM8/9/09

> 1. OAuth rarely works - I tried a number of your apps and it seems
> to work 1 out of 6-7 times. As a note, it worked better with Safari,
> but not every time.
-Not applicable
> 2. 302 redirect
- not sure anymore since our code has been updated to follow them
> 3. General request timeouts
- still seeing it but, not sure if it when we get temp blacklisted.
> 4. HTML in responses
Haven't seen it today actually, but was fairly frequent last night.

> 5. Unexpected rate limiting / blacklisting
- less frequent, but still happening.

Paul Kinlan

Aug 9, 2009, 2:18:23 PM8/9/09
Not to mention that still appears to be completly blocked from the app engine.


2009/8/9 Naveen Ayyagari <>

Jonathan Joyce (Storm ID)

Aug 9, 2009, 2:48:24 PM8/9/09
to Twitter Development Talk
Hi Ryan,

Some details from my perspective...

1. OAuth rarely works - I tried a number of your apps and it seems
to work 1
out of 6-7 times. As a note, it worked better with Safari, but not

Sporadic all day today. At its worst I'd agree that about 1 in 7
succeeds, but it often performs better than this too, whilst still
failing occassionally.
Using Safari does seem to improve things.

2. 302 redirect


3. General request timeouts

Yes, we are seeing response times fluctuating and sometimes timing
out. Fluctuations are from 0.3 secs to 6.5 secs, average around 1
Worst periods over the last 2 days for us were:
Sat 7am to 6pm (GMT), Sunday 3pm to 5pm (GMT)

4. HTML in responses


5. Unexpected rate limiting / blacklisting

Yes, rate limited from 20,000 -> 150, 3 times in last 24hours. We are
using several servers to spread load, and have found it safer to leave
some servers out in the wings so that when limiting occurs to any
active IP the reserves can be added back in to take the strain whilst
we wait for the restricted IPs to recover, which they generally do in
approximately 20 minutes.



Aug 9, 2009, 3:10:43 PM8/9/09
to Twitter Development Talk
Appreciate your updates Ryan, here are the issue we are still
experiencing at HootSuite

1. 302 redirect. We still have to follow redirects in order to
retrieve proper contents, but not a big deal
2. Rate Limiting. All of our server IPs (supposed to be whitelisted)
are bound with 150 limit
3. General request timeouts, but only occasionally
4. HTML in reponse, occasionally.


Aug 9, 2009, 1:42:27 PM8/9/09
to Twitter Development Talk
Hi Ryan,

Many thanks for the update.

On twitscoop, we have issues with the search api, which apparently
returns erroneous or ill-formed timestamps.
eg. you'll see the latest
tweet will be dated "14465 days ago" :))



Aug 9, 2009, 1:50:26 PM8/9/09
to Twitter Development Talk

I'm actually clean - no problems for the last 2 days. I do, however
have my app just following all re-directs. If it would help I can pull
the follow on 302 and let you know what happens.

Brian Roy

Mr Blog

Aug 9, 2009, 1:52:21 PM8/9/09
to Twitter Development Talk is not working for me from either API or directly
in the browser. Is it working for others? If I am blacklisted, how
does one go about getting it fixed?

Clay Loveless

Aug 9, 2009, 1:52:44 PM8/9/09
to Twitter Development Talk

Many thanks for the continued updates. You guys are handling this far
better than most would.

Regarding OAuth: I'm also having difficulty updating OAuth app
properties on the website -- the update form continues to return a
"400 Bad Request" when it's submitted.

Obviously not a huge priority issue, since OAuth behavior itself is
not working well, but just wanted to get it on your list.

Thanks again, and good luck to the team.



Aug 9, 2009, 2:09:32 PM8/9/09
to Twitter Development Talk
We are getting timeouts via appengine for search api calls for All our data aggregation activities are ending up with
almost no data for the past 3 days.

Vivek Puri


Aug 9, 2009, 2:23:10 PM8/9/09
to Twitter Development Talk
The 302 issue has not been made very clear:
1. does the redirect need to be re-signed?
2. is this only the token calls, or all calls to the API methods?

Also, I don't know if it's related or not, but paging on the blocks/
blocking method does not work. i.e. all page parameters return a full


Aug 9, 2009, 3:03:15 PM8/9/09
to Twitter Development Talk
1. I'm finding that OAuth breaks for periods, then works for periods
(not 1/6 requests, more like works 6 times, then breaks 6 times). When
it doesn't work in Safari, if I dump my cookies it usually
works the next try.

Benjamin Rubinstein

Aug 9, 2009, 3:09:22 PM8/9/09
to Twitter Development Talk
> 5. Unexpected rate limiting / blacklisting

Mostly I'm noticing rate limiting.

- Ben


Aug 9, 2009, 7:02:51 PM8/9/09
to Twitter Development Talk
> 5. Unexpected rate limiting / blacklisting
Most annoyed issue for me. Rate is 150/hour for my whitelist ip now.


Aug 10, 2009, 5:19:53 AM8/10/09
to Twitter Development Talk
I am using the search API,
-- getting 409 conflict, once in a while.
-- getting "no json could be decoded" sometimes, I guess thats coz of
the HTML in response error.

Reply all
Reply to author
0 new messages