Twitter Update, 8/9 noon PST

10 views
Skip to first unread message

Ryan Sarver

unread,
Aug 9, 2009, 3:13:41 PM8/9/09
to twitter-deve...@googlegroups.com
*Finally* have what we hope is good news for everyone. As of about 10 minutes ago we have been able to restore critical parts of API operation that should have great affect on your apps. As such, most of your apps should begin to function normally again. I have tested a few OAuth apps and they seem to be working as expected.

Please test your apps from their standard configs to see what results you get and let us know. I am primarily interested in unexpected throttling and issues with OAuth.

I look forward to hearing the results and thanks again for your assistance.

Best, Ryan

Bill Kocik

unread,
Aug 9, 2009, 3:19:05 PM8/9/09
to Twitter Development Talk


On Aug 9, 3:13 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Please test your apps from their standard configs to see what results you
> get and let us know. I am primarily interested in unexpected throttling and
> issues with OAuth.

OAuth appears to be working for my app. Thanks!

Dossy Shiobara

unread,
Aug 9, 2009, 3:31:15 PM8/9/09
to twitter-deve...@googlegroups.com
Thank you for the good news, Ryan. It looks like Twitter Karma is
starting to work again, too.


--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

Paul Kinlan

unread,
Aug 9, 2009, 3:32:16 PM8/9/09
to twitter-deve...@googlegroups.com
OAuth, Search and the friendship methods are working for me...

Paul

2009/8/9 Bill Kocik <bko...@gmail.com>

Dewald Pretorius

unread,
Aug 9, 2009, 3:35:02 PM8/9/09
to Twitter Development Talk
Ryan,

Are there any new requirements we have to comply with for calls to the
Search API?

I presume we have to handle 302s there as well? Anything else?

Dewald

Naveen Ayyagari

unread,
Aug 9, 2009, 3:49:09 PM8/9/09
to twitter-deve...@googlegroups.com
Most calls seem to be working much better for us.

in response to the rate_limit_status call, I get HTML back
occasionally.. looks like the fail whale page and 502 twitter over
capacity..

Got this one about 10 min ago.

08-09-09 19:40:15rate_limit_status response(502): <!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd
">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8" />
<meta http-equiv="Content-Language" content="en-us" />
<title>Twitter / Over capacity</title>
<link rel="icon" href="http://static.twitter.com/favicon.ico"
type="image/ico" />
<style type="text/css">
body{background:#9AE4E8 url(http://static.twitter.com/images/
bg.gif) no-repeat fixed left top;color:#333;font:0.75em
Helvetica,Arial,sans-serif;text-align:center}
#container { width: 755px; margin: 0 auto; padding: 0px 0; text-
align: left; position: relative; }
#content { width: 100%; margin-top: 0px; float: left; padding-
bottom: 15px; background: transparent url(http://static.twitter.com/images/arr2.gif
) no-repeat scroll 25px 0px;}
.subpage #content .wrapper { background-color: #fff; padding:
20px 10px 15px 10px; -moz-border-radius: 5px;-webkit-border-radius:
5px;}
.subpage #content h1, .subpage #content h2, .subpage #content
h3, .subpage #content h4, .subpage #content h5 { margin: 3px 0pt 4px; }
.subpage #content p { line-height: 1.2; margin: 5px 0; }
.subpage #content ul { padding-left: 30px; }
.subpage #content ol, #side ol { padding-left: 30px; }
a{text-decoration:none;color: #0084b4;}
#content div.desc { margin: 11px 0px 10px 0px; }
a img{border:0;}
ul{list-style:square;padding-left:20px;}

#navigation { position: absolute; top: 12px; right: 0; padding:
0px 5px 0px 5px; line-height: 0.5em; text-align: center; }
#navigation ul {
list-style:none
margin: 0; padding: 0px;
width: auto; height: 100%;
}
#navigation li { display:inline; padding: 0 0 0 5px; }
#navigation li:before { content: ' '; padding-right: 0; }
#navigation li.first:before { content: ''; padding-right: 0; }
#navigation, #footer { background-color: #fff; -moz-border-
radius: 5px;-webkit-border-radius: 5px;}

#footer { clear: left; width: 555px; text-align: center; padding:
1px 0; line-height: 1; }
#footer li { display: inline; padding: 0 0 0 5px; }
#footer li.first:before { content: ''; padding-right: 0; }

</style>
</head>
<body>

<div id="container" class="subpage">
<div id="navigation">
<ul>
<li class="first"><a href="http://twitter.com">Home</a></li>
<li><a href="http://twitter.com/public_timeline">Public
Timeline</a></li>
<li><a href="http://help.twitter.com">Help</a></li>
</ul>
</div>
<h1 id="header"><a href="http://twitter.com"><img src="http://static.twitter.com/images/twitter_logo_s.gif
" /></a></h1>
<div id="content">
<div class="desc"></div>
<div class="wrapper">
<span style="font-size:1.8em; font-weight:bold">Twitter is
over capacity.</span><br />
<div style="font-size:1.2em;margin-top:
2px;color:#b6b6a3">Too many tweets! Please wait a moment and try
again.</div><br />

<p style="margin-bottom:10px;text-align:center"><img src="http://static.twitter.com/images/whale.png
" /></p>
</div>
</div>

<div id="footer" style="width:100%">
<ul>
<li class="first">&copy; 2009 Twitter</li>
<li><a href="/help/aboutus">About Us</a></li>
<li><a href="/help/contact">Contact</a></li>
<li><a href="http://blog.twitter.com/">Blog</a></li>
<li><a href="http://status.twitter.com/">Status</a></li>
<li><a href="/help/api">API</a></li>
<li><a href="http://help.twitter.com/">Help</a></li>
<li><a href="/help/jobs">Jobs</a></li>
<li><a href="/help/tos">TOS</a></li>
<li><a href="/help/privacy">Privacy</a></li>
</ul>
</div>

</div>

<!-- BEGIN google analytics -->
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl
." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-
analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
var pageTracker = _gat._getTracker("UA-30775-6");
pageTracker._setDomainName("twitter.com");
pageTracker._trackPageview('503 Error');
</script>
<!-- END google analytics -->

</body>
</html>

chinaski007

unread,
Aug 9, 2009, 3:49:44 PM8/9/09
to Twitter Development Talk

I am still getting these bizarre "OK: 200" errors with returned
content like...

"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>

Dewald Pretorius

unread,
Aug 9, 2009, 4:00:03 PM8/9/09
to Twitter Development Talk
I am slowly firing up my cron jobs, and so far so good.

Dewald

Dewald Pretorius

unread,
Aug 9, 2009, 4:11:49 PM8/9/09
to Twitter Development Talk
Holding steady after 2,000 GETs in rapid succession, and heaven knows
how many POSTs. So, it looks like we are back in business.

Dewald

Mario Menti

unread,
Aug 9, 2009, 4:33:13 PM8/9/09
to twitter-deve...@googlegroups.com
Seeing similar improvements here (twitterfeed) - feeds with error statuses have decreased from 120,000 a few hours ago to just over 10,000 now, so am assuming it's safe to revert to our default settings and stop the throttling we put into place a couple of days ago.

Mario.

Rich

unread,
Aug 9, 2009, 4:39:41 PM8/9/09
to Twitter Development Talk
Thanks Ryan, just tested my oAuth system and it work perfectly.

Thanks for all your work guys & gals!

Arik Fraimovich

unread,
Aug 9, 2009, 4:51:45 PM8/9/09
to Twitter Development Talk
On Aug 9, 10:13 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> *Finally* have what we hope is good news for everyone.

Everything seems to behave much better now.

Thank you and the rest of the team !

--
Arik Fraimovich

Vincent Nguyen

unread,
Aug 9, 2009, 5:02:19 PM8/9/09
to twitter-deve...@googlegroups.com
My App works fine now!
Good news for everybody!
Finally, we can run our app again!


2009/8/9 Arik Fraimovich <ari...@gmail.com>:

Jesse Stay

unread,
Aug 9, 2009, 7:26:12 PM8/9/09
to twitter-deve...@googlegroups.com
Are there any new limits with verify_credentials() now?  I'm showing it only works half the time, even under the 15 requests per hour limit.  Anyone else seeing this?

Jesse

Sean Cashin

unread,
Aug 9, 2009, 6:14:24 PM8/9/09
to Twitter Development Talk
No longer being rate_limited at 150. Back to us being whitelisted.
Thanks for the fix.

Kyle Mulka

unread,
Aug 9, 2009, 7:55:53 PM8/9/09
to Twitter Development Talk
http://twilk.com works consistently now with OAuth and everything.
Whatever you guys did seems to have fixed the problems. Is the DOS
attack still as strong as it was before?

--
Kyle Mulka

Vignesh

unread,
Aug 9, 2009, 10:42:44 PM8/9/09
to Twitter Development Talk
My app is also working fine ... just beginning to fire up the crons...

On Aug 9, 4:55 pm, Kyle Mulka <repalvigla...@yahoo.com> wrote:
> http://twilk.comworks consistently now with OAuth and everything.

AccountingSoftwareGuy

unread,
Aug 9, 2009, 11:08:29 PM8/9/09
to Twitter Development Talk
Really weird...using the exact same .NET code and oAuth. From our
website if we try to post a tweet directly from our webform we get a
401 Unauthorized error. But if a tweet gets sent out via a windows
service we have that works behind the scenes the tweets are going out
just fine.

Any ideas??

Adam Loving

unread,
Aug 9, 2009, 10:54:36 PM8/9/09
to Twitter Development Talk
I just tested OAuth from Twibes.com, it worked twice in a row. Thanks
guys for whatever you're doing.

On Aug 9, 7:42 pm, Vignesh <vignesh.isqu...@gmail.com> wrote:
> My app is also working fine ... just beginning to fire up the crons...
>
> On Aug 9, 4:55 pm, Kyle Mulka <repalvigla...@yahoo.com> wrote:
>
> >http://twilk.comworksconsistently now with OAuth and everything.

Patrick

unread,
Aug 10, 2009, 12:11:53 AM8/10/09
to Twitter Development Talk
I am still having problems logging in using Basic Authentication.

Because I don't use OAuth I cannot give you feedback on that. Sorry.

kozen

Vignesh

unread,
Aug 10, 2009, 1:54:08 AM8/10/09
to Twitter Development Talk
25% of my requests are still getting timed out..is there any rate
limit in place?

Jesse Stay

unread,
Aug 10, 2009, 4:47:43 AM8/10/09
to twitter-deve...@googlegroups.com
I just started getting timeouts again. (the verify_credentials issue I mentioned before never got fixed either)

Jesse

Jesse Stay

unread,
Aug 10, 2009, 4:49:28 AM8/10/09
to twitter-deve...@googlegroups.com
Sorry (it's early and I'm tired), not timeouts - it's only allowing 150 requests per hour again.

Jesse

Dewald Pretorius

unread,
Aug 10, 2009, 9:00:37 AM8/10/09
to Twitter Development Talk
I am seeing tons of requests that return 200 OK plus the expected JSON
data, but the headers have no X-RateLimit entries.

Dewald

Dewald Pretorius

unread,
Aug 10, 2009, 10:22:11 AM8/10/09
to Twitter Development Talk
Just to let you guys know that 502 Bad Gateway responses are coming
thick and fast this morning (Monday).

Dewald

JV

unread,
Aug 10, 2009, 9:14:44 AM8/10/09
to Twitter Development Talk
I am still getting

"HTTP Error 409: Conflict"

from the search API, once every 15mins on an average.

JV

timothy willan

unread,
Aug 10, 2009, 10:59:49 AM8/10/09
to twitter-deve...@googlegroups.com
Hello Ryan I'm timothy not surwe why I'm receiving all twitter Development emails, let me know Thanks


On Sun, Aug 9, 2009 at 3:13 PM, Ryan Sarver <rsa...@twitter.com> wrote:
*Finally* have what we hope is good news for everyone. As of about 10 minutes ago we have been able to restore critical parts of API operation that should have great affect on your apps. As such, most of your apps should begin to function normally again. I have tested a few OAuth apps and they seem to be working as expected.

Please test your apps from their standard configs to see what results you get and let us know. I am primarily interested in unexpected throttling and issues with OAuth.

I look forward to hearing the results and thanks again for your assistance.

Best, Ryan



--
"The solution to building a strong country is to make its citizens strong...To make U.S. citizens strong you need to make them their own boss... in control of their own destiny... happy about life and the direction that life is taking them... that is what will make the U.S. citizens strong again. THAT is what built this country into the FREE nation that it is today{."Winning means being unafraid to lose."timoth...@gmail.com
Timothy Willan
2214 Ardenwood dr.
Spring Hill Fl. 346109
352-585-1264  
   

Marco Kaiser

unread,
Aug 10, 2009, 11:02:21 AM8/10/09
to twitter-deve...@googlegroups.com
most likely because you subscribed to the group

you can go to http://groups.google.com/group/twitter-development-talk/ to manage your subscription status

Marco

2009/8/10 timothy willan <timoth...@gmail.com>

JDG

unread,
Aug 10, 2009, 11:03:08 AM8/10/09
to twitter-deve...@googlegroups.com
you're obviously subscribed to the group. either you subscribed or someone did for you. log in to groups.google.com and unsubscribe.
--
Internets. Serious business.

Duane Roelands

unread,
Aug 10, 2009, 11:15:42 AM8/10/09
to Twitter Development Talk
My users are seeing these as well.

Ryan Sarver

unread,
Aug 10, 2009, 11:19:11 AM8/10/09
to twitter-deve...@googlegroups.com
Can you all provide full packet dumps of the issues you are seeing? That will help us debug the type of request you are making and what the full response looks like.

Thanks, Ryan

Dewald Pretorius

unread,
Aug 10, 2009, 11:35:23 AM8/10/09
to Twitter Development Talk
Ryan,

The 502s don't really bother me. It just slows me down a bit. My
Twitter lib automatically retries a request that 502'd.

The 200 OK responses that come back with the X-RateLimit headers
missing happen on all kinds of requests, both GETs and POSTs (more on
GETs as far as I can tell), and as far as I can see there is no
pattern. One request would return headers, and the very next request
to the same API method would not return the headers. Are these perhaps
requests have have been 302'd? I can't really check because cURL is
following the redirects on GETs.

Dewald

On Aug 10, 12:19 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Can you all provide full packet dumps of the issues you are seeing? That
> will help us debug the type of request you are making and what the full
> response looks like.
> Thanks, Ryan
>

Ryan Sarver

unread,
Aug 10, 2009, 2:40:25 PM8/10/09
to twitter-deve...@googlegroups.com
Thanks Dewald. Generally speaking if you see a 502/503 you should do a geometric back off instead of just immediately re-requesting.

Is anyone else seeing the X-RateLimit header not showing up some times?

Thanks, Ryan

chinaski007

unread,
Aug 10, 2009, 8:19:23 PM8/10/09
to Twitter Development Talk

Yes, I am also having x-ratelimit not show up after I have received
"OK: 200" errors.

I believe this is due to timeout issues.

Specifically, I often find that I am returned partial JSON (lack of
closing syntax) without the ratelimit header, e.g.:

'_rc' => 200,
'_headers' => bless( {
'connection' => 'close',
'cache-control' => 'max-age=60, must-revalidate, max-age=300',
'status' => '200 OK',
'date' => 'Mon, 10 Aug 2009 23:47:29 GMT',
'vary' => 'Accept-Encoding',
'client-peer' => '168.143.162.123:80',
'age' => '0',
'client-date' => 'Mon, 10 Aug 2009 23:47:21 GMT',
'x-died' => 'read failed: Connection reset by peer at /usr/local/lib/
perl5/site_perl/5.8.8/LWP/Protocol/http.pm line 382.',
'content-type' => 'application/json; charset=utf-8',
'x-cache-svr' => 'c091.twitter.com',
'server' => 'hi',
'x-timeline-cache-hit' => 'Miss',
'x-served-by' => 'c091.twitter.com',
'client-response-num' => 1,
'content-length' => '43452',
'client-aborted' => 'die',
'via' => '1.1 varnish',
'x-cache' => 'MISS',
'x-served-from' => 'b032',
'expires' => 'Mon, 10 Aug 2009 23:52:28 GMT'
}, 'HTTP::Headers' ),
'_msg' => 'OK',

Dewald Pretorius

unread,
Aug 11, 2009, 9:26:30 AM8/11/09
to Twitter Development Talk
Ryan,

I am now doing geometric backoffs on 502s and connection refuses.

I did not do exact math, but I seem to be getting between 2 and 5
connection refuses and 502s (combined) per second, and 95%+ of those
requests get a 200 OK at the first retry.

I think it's really good performance given the strain that your system
is under.

Dewald

Damon P. Cortesi

unread,
Aug 11, 2009, 3:17:29 PM8/11/09
to Twitter Development Talk
I appear to still be getting request timeouts on one of my servers.

Ryan Sarver

unread,
Aug 11, 2009, 4:08:41 PM8/11/09
to twitter-deve...@googlegroups.com
Thanks for the update Dewald. Keep us posted if things change.

Best, Ryan

Reply all
Reply to author
Forward
0 new messages