Platform Status Update, Monday 4:30pm PST

1 view
Skip to first unread message

Ryan Sarver

unread,
Aug 17, 2009, 7:41:50 PM8/17/09
to Twitter Development Talk, twitter-ap...@googlegroups.com
I wanted to email the list and update everyone on the current status
of the platform. Over the past ten days we have been dealing with a
lot of stress on our network and that has caused a number of our
partners to be knocked offline for extended periods of time. This is
obviously not something we want to happen and the platform and ops
teams have been working hard throughout that time to address the needs
of our ecosystem while protecting the system as a whole. We have made
some great progress today in tuning the system to a point that should
allow our partners to operate as they were before the recent issue
began.

However, the system is still under stress so we will need to keep an
eye on the platform and make sure everyone is operational. If you were
experiencing issues over the weekend and into today, please recheck
the issues to see if they have been resolved. If you are still
experiencing problems, please provide the requested details to help us
debug your specific problem. If you don't provide at least some of the
additional information we will not be able to help you - so please do
some leg work and help us help you:

1. The IP of the machine making requests to the Twitter API. If you're
behind NAT, please be sure to send us your *external* IP.
2. The IP address of the machine you're contacting in the Twitter
cluster. You can find this on UNIX machines via the "host" or
"nslookup" commands, and on Windows machines via the "nslookup"
command.
3. The Twitter API URL (method) you're requesting and any other
details about the request (GET vs. POST, parameters, headers, etc.).
4. Your host operating system, browser (including version), relevant
cookies, and any other pertinent information about your environment.
5. What kind of network connection you have and from which provider,
and what kind of network connectivity devices you're using.

We will continue to monitor the situation and the mailing list for any
new issues. We appreciate your patience and your support in helping us
get through the situation. Thank you to all the developers who
provided us with information to help us tune the system and get
everyone back online.

Thanks again, Ryan
@rsarver

Chad Etzel

unread,
Aug 17, 2009, 8:01:31 PM8/17/09
to twitter-deve...@googlegroups.com
Hi all,

I'd just like to add that if you can also provide HTTP
request/response headers and/or packet captures/dumps of actual API
requests, that would be really really helpful in troubleshooting
specific situations. Any patterns we can see emerging from this data
will help us debug even faster.

Thanks,
-Chad

Chad Etzel

unread,
Aug 17, 2009, 8:21:56 PM8/17/09
to twitter-deve...@googlegroups.com
To help find your external IP, I've setup
http://jazzychad.net/ip.php
(basically a noiseless whatismyip.com)

Or if you want to get only your external IP from the command line, try:
curl http://jazzychad.com/ipecho.php

-Chad

djc8080

unread,
Aug 17, 2009, 9:14:18 PM8/17/09
to Twitter Development Talk
Our IP: 69.107.71.66 Your IP: 168.143.162.68 Via AT&T Dynamic DSL

We host Twitter accounts for enterprise clients - each via basic
authentication 24x7 via FF3.5. No memory leak observed on FF3.5 -
despite the problems. We're staring at dozens of identical Vista
machines - some work happily, others fail.

After the Sat reset and Mon corrections, here's what we're seeing:

Reliable performance: (e.g. 99%+ correct)
- search
- statuses/update - POST
- statuses/user_timeline
- account/rate_limit_status
- ..ids
- friendships/create - POST

Unreliable with frequent fails:
- statuses/followers (e.g. 10% correct. Every few hours, one request
responds correctly ;-)
- statuses/friends (e.g. 80% correct)

On problem accounts, cookies have been deleted without helping.

On error, returns XML rate limit error. JSON calls get no-server msg.
Other api calls/results & rate_limit_status calls contradict the
returned error messages.

Hope that helps.

djc8080

unread,
Aug 18, 2009, 2:38:53 PM8/18/09
to Twitter Development Talk
Here's the typical errors. Call 1 says 132 hits remaining; call 2
returns over the limit. Once an hour, call 2 slips through and returns
the right result. Otherwise, on some accounts - the rate_limit bug
blocks access. On other accounts, there is no problem at all.
Sometimes, the problem resolves on one account; and moves to another.
In my humble opinion, the bugs trace to out-of-control, unpredictable
rate-limiting statements.

Also, note reset time in seconds makes no sense.

1. http://twitter.com/account/rate_limit_status.xml

Results:
<hash>
<remaining-hits type="integer">132</remaining-hits>
<hourly-limit type="integer">150</hourly-limit>
<reset-time type="datetime">2009-08-18T17:32:14+00:00</reset-time>
<reset-time-in-seconds type="integer">1250616734</reset-time-in-
seconds>
</hash>


2. http://twitter.com/statuses/followers/scotmckay.xml
Results:
<hash>
<request>/statuses/followers/scotmckay.xml</request>
-
<error>
Rate limit exceeded. Clients may not make more than 150 requests per
hour.
</error>
</hash>
Reply all
Reply to author
Forward
0 new messages