non json response

ben

unread,

Aug 26, 2009, 10:00:45 AM8/26/09

to Twitter Development Talk

Occassionally i get back a 200 status html response from the json
search api which look like this, most times the same search works
fine, it just happens occassionally:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
TR/1999/REC-html401-19991224/strict.dtd">

<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY></BODY>
</HTML>

Does anyone recognise what this kind of response means? Is it normal,
or just beta-ish quirks?

Ryan Sarver

unread,

Aug 26, 2009, 12:27:54 PM8/26/09

to twitter-deve...@googlegroups.com

Ben,

It's a known issue and we are trying to hunt it down. Can you please
provide us with your source IP and an approximate time of when you saw
it?

Thanks, Ryan

Ben Eliott

unread,

Aug 27, 2009, 5:45:11 AM8/27/09

to twitter-deve...@googlegroups.com

H Ryan,
Thanks for coming back on this. Here's a bit of logging which has the
url query and the precise time of the returned search, the timezone is
Europe/London. The ip is 67.23.28.168. I might be able to get more for
other times, let me know if you need 'em. This app is in development
so it may yet be some bug with me.
Ben

DEBUG - urlopen success on http://search.twitter.com/search.json?q=this&rpp=100&geocode=51.51076%2C-0.13126%2C23mi&since_id=3561793228
, code 200
2009-08-26 19:55:01,783 - twittersearch.models - DEBUG - search answer:
2009-08-26 19:55:01,783 - twittersearch.models - CRITICAL - Search did
not reutrn a json object! code = 200 answer = <!DOCTYPE html PUBLIC

"-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd

">^M

2009-08-26 20:10:02,548 - twittersearch.models - DEBUG - urlopen
success on http://search.twitter.com/search.json?q=this&rpp=100&geocode=51.51076%2C-0.13126%2C23mi&since_id=3561793228
, code 200
2009-08-26 20:10:02,548 - twittersearch.models - DEBUG - search
answer: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">^M

2009-08-26 20:25:02,439 - twittersearch.models - DEBUG - urlopen
success on http://search.twitter.com/search.json?q=this&rpp=100&geocode=51.51076%2C-0.13126%2C23mi&since_id=3561793228
, code 200
2009-08-26 20:25:02,439 - twittersearch.models - DEBUG - search
answer: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">^M
2009-08-26 20:25:02,439 - twittersearch.models - CRITICAL - Search did
not reutrn a json object! code = 200 answer = <!DOCTYPE html PUBLIC

"-//W3C//DTD HTML 4.01//EN" "http://www.w3.or

2009-08-26 20:40:07,334 - twittersearch.models - DEBUG - urlopen
success on http://search.twitter.com/search.json?q=this&rpp=100&geocode=51.51076%2C-0.13126%2C23mi&since_id=3561793228
, code 200
2009-08-26 20:40:07,334 - twittersearch.models - DEBUG - search
answer: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">^M
2009-08-26 20:40:07,334 - twittersearch.models - CRITICAL - Search did
not reutrn a json object! code = 200 answer = <!DOCTYPE html PUBLIC

"-//W3C//DTD HTML 4.01//EN" "http://www.w3.or

The response is always:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">

<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY></BODY>
</HTML>

Anirudh

unread,

Aug 28, 2009, 4:58:58 AM8/28/09

to Twitter Development Talk

Hi,

Just wanted to add that this problem is not restricted to json, but
exists with the atom api too. I'll try to find some more info. Will
post it here if I get it. It has happened previously too. We had to
write a function to prevent to check for html and if yes prevent it
from being parsed by PHP's simplexml parser because it threw up a lot
of errors.

Best,
Anirudh S

Anirudh

unread,

Aug 28, 2009, 5:12:07 AM8/28/09

to Twitter Development Talk

Hi,

Just got hold of a log entry. We had this issue on Friday, August 28,
2009 - 11:02am Indian Standard Time (+0530 UTC). Our site is
brandadda.com .

Hope this is useful.

Best,
Anirudh S

On Aug 26, 9:27 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Ben,
>
> It's a known issue and we are trying to hunt it down. Can you please
> provide us with your source IP and an approximate time of when you saw
> it?
>
> Thanks, Ryan
>

Steven Wilkin

unread,

Aug 30, 2009, 7:20:02 AM8/30/09

to Twitter Development Talk

I'm consistently getting the same response when accessing
http://search.twitter.com/trends.json from 209.40.204.183

Steve

On Aug 26, 5:27 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Ben,
>
> It's a known issue and we are trying to hunt it down. Can you please
> provide us with your source IP and an approximate time of when you saw
> it?
>

Rudifa

unread,

Sep 6, 2009, 3:35:20 PM9/6/09

to Twitter Development Talk

I have seen this same http page with empty body

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
TR/1999/REC-html401-19991224/strict.dtd">

<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY></BODY>
</HTML>

a number of times in the last few days (but intermittently - a good
response may come after several attempts),
in response to http://twitter.com/users/show/rudifa.json

The most recent one was on UTC time
2009-09-06 18:55:38.262
My IP is 84.227.186.88 as reported by http://www.whatismyip.com/

Could someone at twitter.com please tell us what does this mean? Server
(s) overloaded?

On Aug 30, 1:20 pm, Steven Wilkin <iamthebisc...@gmail.com> wrote:
> I'm consistently getting the same response when accessinghttp://search.twitter.com/trends.jsonfrom 209.40.204.183

Rich

unread,

Sep 7, 2009, 2:21:34 AM9/7/09

to Twitter Development Talk

I think this is the same issue we've been having for a while at
http://groups.google.com/group/twitter-development-talk/browse_thread/thread/e88f0f3182b0a0e7/e5f6b2a5678598f1

I never used to get it on search APIs but now I get it on both REST
and search APIs

On Sep 6, 8:35 pm, Rudifa <rudi.far...@gmail.com> wrote:
> I have seen this same http page with empty body
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
> TR/1999/REC-html401-19991224/strict.dtd">
> 
> <HTML>
> <HEAD>
> <META HTTP-EQUIV="Refresh" CONTENT="0.1">
> <META HTTP-EQUIV="Pragma" CONTENT="no-cache">
> <META HTTP-EQUIV="Expires" CONTENT="-1">
> <TITLE></TITLE>
> </HEAD>
> <BODY></BODY>
> </HTML>
> a number of times in the last few days (but intermittently - a good
> response may come after several attempts),

> in response tohttp://twitter.com/users/show/rudifa.json

>
> The most recent one was on UTC time
> 2009-09-06 18:55:38.262

> My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/

Ben Eliott

unread,

Sep 7, 2009, 6:25:18 AM9/7/09

to twitter-deve...@googlegroups.com

IP: 67.23.28.168, time is Europe/London

2009-09-07 11:19:48,014 - twittersearch.models - CRITICAL - Search did
not reutrn a json object! code = 200 answer = <!DOCTYPE html PUBLIC

"-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">

<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY></BODY>
</HTML>

Starting to wonder whether this is connected to auth/user agent.

Rich

unread,

Sep 7, 2009, 8:04:15 AM9/7/09

to Twitter Development Talk

Either way an XML or JSON feed should NEVER return HTML!

> > in response tohttp://twitter.com/users/show/rudifa.json

>
> > The most recent one was on UTC time
> > 2009-09-06 18:55:38.262

> > My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/

archF6

unread,

Sep 6, 2009, 11:10:18 PM9/6/09

to Twitter Development Talk

I am able to consistently reproduce this error -- I get this response
almost without fail after periods of inactivity greater than 2 hours.
I am requesting XML via PHP, it's a GET request. The requests are
coming from 96.30.16.192. Let me know if I can provide any additional
info that might help, or if you have any suggestions about how best to
handle this response.

On Sep 6, 3:35 pm, Rudifa <rudi.far...@gmail.com> wrote:
> I have seen this same http page with empty body
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
> TR/1999/REC-html401-19991224/strict.dtd">
> 
> <HTML>
> <HEAD>
> <META HTTP-EQUIV="Refresh" CONTENT="0.1">
> <META HTTP-EQUIV="Pragma" CONTENT="no-cache">
> <META HTTP-EQUIV="Expires" CONTENT="-1">
> <TITLE></TITLE>
> </HEAD>
> <BODY></BODY>
> </HTML>
> a number of times in the last few days (but intermittently - a good
> response may come after several attempts),

> in response tohttp://twitter.com/users/show/rudifa.json

>
> The most recent one was on UTC time
> 2009-09-06 18:55:38.262

> My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/

archF6

unread,

Sep 6, 2009, 11:14:23 PM9/6/09

to Twitter Development Talk

I am able to consistently reproduce this error. I am making GET
requests via PHP from IP: 96.30.16.192. I receive the error without
fail after periods of inactivity lasting 2 hours or more. The header
response code is 200. Please let me know if I can provide any
additional info that might help you diagnose the problem, or if you
have suggestions about how best to handle.

Thanks.

On Sep 6, 3:35 pm, Rudifa <rudi.far...@gmail.com> wrote:

> I have seen this same http page with empty body
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
> TR/1999/REC-html401-19991224/strict.dtd">
> 
> <HTML>
> <HEAD>
> <META HTTP-EQUIV="Refresh" CONTENT="0.1">
> <META HTTP-EQUIV="Pragma" CONTENT="no-cache">
> <META HTTP-EQUIV="Expires" CONTENT="-1">
> <TITLE></TITLE>
> </HEAD>
> <BODY></BODY>
> </HTML>
> a number of times in the last few days (but intermittently - a good
> response may come after several attempts),

> in response tohttp://twitter.com/users/show/rudifa.json

>
> The most recent one was on UTC time
> 2009-09-06 18:55:38.262

> My IP is 84.227.186.88 as reported byhttp://www.whatismyip.com/

Monica Keller

unread,

Sep 10, 2009, 5:09:38 PM9/10/09

to Twitter Development Talk

We see this error 75% of the time. Have you guys made an progress on
resolving the issue ?

> > > > > Occassionally i get back a 200 statushtmlresponse from the json

> > > > > search api which look like this, most times the same search works
> > > > > fine, it just happens occassionally:
>
> > > > > <!DOCTYPEhtmlPUBLIC "-//W3C//DTDHTML4.01//EN" "http://www.w3.org/
> > > > > TR/1999/REC-html401-19991224/strict.dtd">
> > > > > 
> > > > > <HTML>
> > > > > <HEAD>
> > > > > <META HTTP-EQUIV="Refresh" CONTENT="0.1">
> > > > > <META HTTP-EQUIV="Pragma" CONTENT="no-cache">
> > > > > <META HTTP-EQUIV="Expires" CONTENT="-1">
> > > > > <TITLE></TITLE>
> > > > > </HEAD>
> > > > > <BODY></BODY>
> > > > > </HTML>
>
> > > > > Does anyone recognise what this kind of response means? Is it normal,

> > > > > or just beta-ish quirks?- Hide quoted text -
>
> - Show quoted text -

John Kalucki

unread,

Sep 10, 2009, 5:12:21 PM9/10/09

to Twitter Development Talk

People are working on this as a high-priority issue. I'd imagine that
the API team will have an update soon.

Matthew Ranney

unread,

Sep 11, 2009, 1:05:30 AM9/11/09

to twitter-deve...@googlegroups.com

I'm seeing tons of these as well.

However, I've found that if you follow the suggestion of the META tag to simply "refresh" in 0.1 seconds if you get this bogus response, you can hide most of this from users, especially if they are on a fast network.

Raffi Krikorian

unread,

Sep 12, 2009, 12:57:12 PM9/12/09

to twitter-deve...@googlegroups.com

Hi all.

This is an extremely high priority problem for us, and for me personally, to fix.

If you're having this problem, please free to reach out to me, but please try to include:

* the IP address your request is coming from

* the request you're making

* the response you got back

* the time that the request/response was made/received

(or, a tcpdump or curl output when doing -vvv would be sufficient also)

Rudifa

unread,

Sep 13, 2009, 6:01:37 AM9/13/09

to Twitter Development Talk

I just had one non-json response, in the middle of about 10 requests
made with curl -vvv (other responses were correct)

Below are 3 requests and the non-json response bracketted by 2 good
responses which contain the response time and other logging data.

HTH
Rudi

rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvv
http://twitter.com/users/show/rudifa.json
* About to connect() to twitter.com port 80 (#0)
* Trying 168.143.161.20... connected
* Connected to twitter.com (168.143.161.20) port 80 (#0)
> GET /users/show/rudifa.json HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: twitter.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 13 Sep 2009 09:45:23 GMT
< Server: hi
< X-RateLimit-Limit: 150
< X-Transaction: 1252835123-2408-31139
< Status: 200 OK
< ETag: "df090f6c8147e20ba7fe81315a66b9af"
< Last-Modified: Sun, 13 Sep 2009 09:45:23 GMT
< X-RateLimit-Remaining: 124
< Content-Type: application/json; charset=utf-8
< Pragma: no-cache
< Content-Length: 1176
< Cache-Control: no-cache, no-store, must-revalidate, pre-check=0,
post-check=0
< Expires: Tue, 31 Mar 1981 05:00:00 GMT
< X-Revision: a62881015b2c2fb6f795bf931bd56bd494f37254
< X-RateLimit-Reset: 1252836853
< Set-Cookie: lang=en; path=/
< Set-Cookie:
_twitter_sess=BAh7CDoRdHJhbnNfcHJvbXB0MDoHaWQiJWU5OGQyZmU3NWVkY2RhZjhkYTk5%250ANTBlNTA4OTk0MzhhIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFz
%250AaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--66931156c75554797fc576876bdec52dc705736e;
domain=.twitter.com; path=/
< Vary: Accept-Encoding
< Connection: close
<
* Closing connection #0
{"profile_sidebar_border_color":"BDDCAD","description":"Wrote firmware
for world-class osciloscopes for many years. Now learning iPhone
programming tricks. Loves
skiing.","url":null,"screen_name":"rudifa","status":
{"in_reply_to_status_id":null,"favorited":false,"in_reply_to_user_id":null,"source":"<a
href=\"http://apiwiki.twitter.com/\" rel=\"nofollow\">API</
a>","created_at":"Thu Sep 10 16:49:49 +0000
2009","in_reply_to_screen_name":null,"id":
3890997267,"truncated":false,"text":"De retour de la T\u00eate de
Parmelan"},"following":null,"verified":false,"profile_text_color":"333333","followers_count":
9,"profile_background_image_url":"http://a1.twimg.com/
profile_background_images/17762518/
DSC01211-63-2.jpeg","created_at":"Thu Apr 30 22:42:35 +0000
2009","notifications":null,"friends_count":
29,"profile_link_color":"0084B4","profile_background_tile":false,"favourites_count":
0,"profile_background_color":"9AE4E8","protected":false,"time_zone":"Bern","location":"Geneva","name":"Rudi
Farkas","profile_sidebar_fill_color":"DDFFCC","id":
36797542,"statuses_count":52,"utc_offset":
3600,"profile_image_url":"http://a1.twimg.com/profile_images/311858510/
me-bsp-6a_normal.png"}

rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvv
http://twitter.com/users/show/rudifa.json
* About to connect() to twitter.com port 80 (#0)
* Trying 168.143.161.20... connected
* Connected to twitter.com (168.143.161.20) port 80 (#0)
> GET /users/show/rudifa.json HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: twitter.com
> Accept: */*
>
< HTTP/1.0 200 OK
< Connection: Close
< Pragma: no-cache
< cache-control: no-cache
< Refresh: 0.1
< Content-Type: text/html; charset=iso-8859-1

<
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
TR/1999/REC-html401-19991224/strict.dtd">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"

"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY></BODY>
</HTML>

* Closing connection #0

rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvv
http://twitter.com/users/show/rudifa.json
* About to connect() to twitter.com port 80 (#0)
* Trying 168.143.161.20... connected
* Connected to twitter.com (168.143.161.20) port 80 (#0)
> GET /users/show/rudifa.json HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: twitter.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 13 Sep 2009 09:45:35 GMT
< Server: hi
< X-RateLimit-Limit: 150
< X-Transaction: 1252835135-57410-29384
< Status: 200 OK
< ETag: "df090f6c8147e20ba7fe81315a66b9af"
< Last-Modified: Sun, 13 Sep 2009 09:45:35 GMT
< X-RateLimit-Remaining: 123
< Content-Type: application/json; charset=utf-8
< Pragma: no-cache
< Content-Length: 1176
< Cache-Control: no-cache, no-store, must-revalidate, pre-check=0,
post-check=0
< Expires: Tue, 31 Mar 1981 05:00:00 GMT
< X-Revision: a62881015b2c2fb6f795bf931bd56bd494f37254
< X-RateLimit-Reset: 1252836853
< Set-Cookie: lang=en; path=/
< Set-Cookie:
_twitter_sess=BAh7CDoRdHJhbnNfcHJvbXB0MDoHaWQiJWY0NDM0ODQ3YWI2NDk5OTg4OTY3%250AYWQ1ZjU5Njg3OWIwIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFz
%250AaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--
d454f0c630e2e82ace19b7056947734184aea7ed; domain=.twitter.com; path=/
< Vary: Accept-Encoding
< Connection: close
<
* Closing connection #0
{"profile_sidebar_border_color":"BDDCAD","description":"Wrote firmware
for world-class osciloscopes for many years. Now learning iPhone
programming tricks. Loves
skiing.","url":null,"screen_name":"rudifa","status":
{"in_reply_to_status_id":null,"favorited":false,"in_reply_to_user_id":null,"source":"<a
href=\"http://apiwiki.twitter.com/\" rel=\"nofollow\">API</
a>","created_at":"Thu Sep 10 16:49:49 +0000
2009","in_reply_to_screen_name":null,"id":
3890997267,"truncated":false,"text":"De retour de la T\u00eate de
Parmelan"},"following":null,"verified":false,"profile_text_color":"333333","followers_count":
9,"profile_background_image_url":"http://a1.twimg.com/
profile_background_images/17762518/
DSC01211-63-2.jpeg","created_at":"Thu Apr 30 22:42:35 +0000
2009","notifications":null,"friends_count":
29,"profile_link_color":"0084B4","profile_background_tile":false,"favourites_count":
0,"profile_background_color":"9AE4E8","protected":false,"time_zone":"Bern","location":"Geneva","name":"Rudi
Farkas","profile_sidebar_fill_color":"DDFFCC","id":
36797542,"statuses_count":52,"utc_offset":
3600,"profile_image_url":"http://a1.twimg.com/profile_images/311858510/
me-bsp-6a_normal.png"}

On Sep 12, 6:57 pm, Raffi Krikorian <ra...@twitter.com> wrote:
> Hi all.
>
> This is an extremely high priority problem for us, and for me
> personally, to fix.
>
> If you're having this problem, please free to reach out to me, but
> please try to include:
>
> * the IP address your request is coming from
> * the request you're making
> * the response you got back
> * the time that the request/response was made/received
>
> (or, a tcpdump or curl output when doing -vvv would be sufficient also)
>
> On Sep 10, 2009, at 10:05 PM, Matthew Ranney <m...@ranney.com> wrote:
>
> > I'm seeing tons of these as well.
>
> > However, I've found that if you follow the suggestion of the META
> > tag to simply "refresh" in 0.1 seconds if you get this bogus
> > response, you can hide most of this from users, especially if they
> > are on a fast network.
>

> > On Thu, Sep 10, 2009 at 2:09 PM, Monica Keller <monica.kel...@gmail.com

Naveen A

unread,

Sep 16, 2009, 1:23:52 PM9/16/09

to Twitter Development Talk

Is there a specific way we can construct our request to mitigate the
non-json response? I have used a few different twitter clients on the
same mobile device and some of them do not seem to be plagued with the
bad data like we are? Does including something in the header help get
us through whatever filter is returning the bad data?

Maybe the Twitter cookies that are returned back on each request?
Currently, we don't pass them back on subsequent requests because they
shouldn't be necessary, but if it will make some of the bad JSON
responses go away, I'll spend the time to implement it..

These bad json responses have been a problem for over a month now and
while I realize it is a difficult problem to track down, the fact
remains that the API is not functioning correctly.

A response from the twitter team would be greatly appreciated.

On Sep 13, 6:01 am, Rudifa <rudi.far...@gmail.com> wrote:
> I just had one non-json response, in the middle of about 10 requests
> made with curl -vvv (other responses were correct)
>
> Below are 3 requests and the non-json response bracketted by 2 good
> responses which contain the response time and other logging data.
>
> HTH
> Rudi
>

> rudolf-farkass-macbook-pro:TwitterWeb rudifarkas$ curl -vvvhttp://twitter.com/users/show/rudifa.json

> ...
>
> read more »

Raffi Krikorian

unread,

Sep 16, 2009, 1:50:11 PM9/16/09

to twitter-deve...@googlegroups.com

hi naveen.

we are most certainly working on it.

the best way to mitigate the error case is to actually do what the
response tells you to do -- in all cases that i've seen the "http 200
error", there has been a refresh header ("Refresh: 0.1"). simply obey
the header, make a subsequent request in 0.1 seconds, and more likely
than not, you will receive your request. in general, "be strict in
what you send out, and be lenient in what you accept" is a pretty good
philosophy to follow. an additional strategy to try: if you receive
an error, then try again with some form of exponential back-off.

if you do see the error, please either send me, personally, a tcpdump
of the issue, or attach it to the issue on http://code.google.com/p/twitter-api/issues/detail?id=1024
.

thanks!

--
Raffi Krikorian
Twitter Platform Team
ra...@twitter.com | @raffi

John Kalucki

unread,

Sep 16, 2009, 1:56:10 PM9/16/09

to Twitter Development Talk

I've been following the internal dialog on tracking this issue down.
Given what we know, I don't think there's anything that you can change
to the request parameters to reduce the chances of this happening.
From a given client, the chances of this happening to a request are
pretty close to random. Different clients, however, seem from the
outside to operate differently, as they tend to routed through
different infrastructure. There also may be differences in the quality
of the code, especially around how errors are handled.

If you want higher reliability, I'd suggest wrapping nearly all
network API calls in a retry loop. If you get any sort of error: tcp,
http, parser, etc. retry with linear backoff.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.

> ...
>
> read more »

Jim Renkel

unread,

Sep 16, 2009, 2:36:22 PM9/16/09

to twitter-deve...@googlegroups.com

I agree with John that to achieve higher user visible reliability, API
requests should be wrapped in a retry loop.

However, PLEASE, PLEASE, PLEASE, do NOT use linear backoff, i.e.,
subsequent retries are delayed by an amount of time chosen uniformly at
random up to the same maximum amount for each retry. This will lead to
disasters for all API users as failed API requests, when retried, will
tend to bunch up in ever increasing bunches, leading to a higher, not
lower, failure rate.

You should instead use exponential backoff, where the maximum amount of
delay time increases for each retry. Queuing theory and experience
indicate that the optimum factor used to increase the maximum delay for
each retry should be 2.0.

The earliest implementations of both Ethernet and TCP, and I'm sure
other protocols, tried linear backoff and experienced this problem in
spades. When the backoff was changed to exponential, the problems
miraculously went away.

Jim Renkel

_twitter_sess=BAh7CDoRdHJhbnNfcHJvbXB0MDoHaWQiJWU5OGQyZmU3NWVkY2RhZjhkYT
k5% 250ANTBlNTA4OTk0MzhhIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFz
> >
%250AaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--66931156c75554797fc576876bdec52dc

alfrhnsby

unread,

Sep 17, 2009, 12:31:16 PM9/17/09

to Twitter Development Talk

Yes ...I agree for your opinion

> -John Kaluckihttp://twitter.com/jkalucki

> ...
>
> read more »- Hide quoted text -

Reply all

Reply to author

Forward