How do I handle 302 redirects with curl?

15,286 views
Skip to first unread message

TjL

unread,
Aug 8, 2009, 1:31:31 AM8/8/09
to twitter-deve...@googlegroups.com
All of my scripts check for "Status 200" before proceeding.

Now we are (sometimes) getting a 302, but when I try

curl --netrc -s -D - http://twitter.com/account/rate_limit_status.xml

Gave me a 302 with a Location of:

http://twitter.com/account/rate_limit_status.xml?c73f7db0

but when I tried

curl --netrc -s -D - 'http://twitter.com/account/rate_limit_status.xml?c73f7db0'

it seemed to want to redirect me to

http://twitter.com/account/rate_limit_status.xml

If "accepting 30x" is a requirement now, I'd like some advice on how to do so.

TjL

Chad Etzel

unread,
Aug 8, 2009, 2:37:10 AM8/8/09
to twitter-deve...@googlegroups.com
You may have to follow redirects more than once *wink wink nudge nudge*

with curl you can add --location flag. There's a good bit of info in
the man page as well.

If using curl with PHP, you can set:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);

HTH,
-Chad

Rich

unread,
Aug 8, 2009, 3:14:38 AM8/8/09
to Twitter Development Talk
Excellent our client now supports the 302's :)

On Aug 8, 7:37 am, Chad Etzel <c...@twitter.com> wrote:
> You may have to follow redirects more than once *wink wink nudge nudge*
>
> with curl you can add --location flag. There's a good bit of info in
> the man page as well.
>
> If using curl with PHP, you can set:
> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
>
> HTH,
> -Chad
>
>
>
> On Sat, Aug 8, 2009 at 1:31 AM, TjL<luo...@gmail.com> wrote:
>
> > All of my scripts check for "Status 200" before proceeding.
>
> > Now we are (sometimes) getting a 302, but when I try
>
> > curl --netrc -s -D -http://twitter.com/account/rate_limit_status.xml

timwhitlock

unread,
Aug 8, 2009, 6:31:08 AM8/8/09
to Twitter Development Talk
I've seem the 302 Location headers having invalid URLs... i.e. two "?"
symbols.
The original query string and then an additional "?" for the token at
the end.
Following this redirect blindly has resulted in a Forbidden response.

Also it is unclear whether the redirect location needs to be re-
signed? (I am not doing so, but may explain the 403?)

Chad Etzel

unread,
Aug 8, 2009, 11:20:32 AM8/8/09
to twitter-deve...@googlegroups.com
Hmm,

Unfortunately this 302 business will completely goof OAuth calls.

If you are able to programmatically see that you are getting these
redirects, try calling the account/rate_limit_status call [1] (it
could be any call, but this one is "free" and is a GET). You should
still get a 302 (I'm pretty sure). Then if you jump through the
redirect hoops with this call, it should clear you from more 302's for
a while.

I'm out today, but if someone could try this and report back if it
works that would be helpful.

[1] http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0rate_limit_status

Thanks,
-Chad

Naveen Ayyagari

unread,
Aug 8, 2009, 1:02:55 PM8/8/09
to twitter-deve...@googlegroups.com
Sometimes the rate_limit_status call is not returning a 302 to
redirect, or the rate_limit_status xml, but HTML with a meta refresh
in it (which curl doesnt understand to follow redirect/retry).

Its not huge problem for us, but it can affect some throttling code
people may or may not be implementing. (when you get this response, a
subsequent retry request usually succeeds 95% of the time)


Here is an example response I am talking about:

curl http://twitter.com/account/rate_limit_status.xml -L -v
* About to connect() to twitter.com port 80 (#0)
* Trying 168.143.162.68... connected
* Connected to twitter.com (168.143.162.68) port 80 (#0)
> GET /account/rate_limit_status.xml HTTP/1.1
> User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2
OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8
> Host: twitter.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: Close
< Pragma: no-cache
< cache-control: no-cache
< Refresh: 0.1
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>

Chad Etzel

unread,
Aug 8, 2009, 6:25:51 PM8/8/09
to twitter-deve...@googlegroups.com
Hmm, it shouldn't be spitting back HTML. How often are you seeing this?
-Chad

CaMason

unread,
Aug 8, 2009, 9:01:19 PM8/8/09
to Twitter Development Talk
To confirm, I am also seeing this behaviour. Some output I've received
on numerous occasions this evening:


-bash-3.2# curl --interface eth0 http://twitter.com/account/rate_limit_status.xml
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
TR/1999/REC-html401-19991224/strict.dtd">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>

-Craig

On Aug 8, 11:25 pm, Chad Etzel <c...@twitter.com> wrote:
> Hmm, it shouldn't be spitting back HTML. How often are you seeing this?
> -Chad
>
>
>
> On Sat, Aug 8, 2009 at 1:02 PM, Naveen Ayyagari<knig...@gmail.com> wrote:
>
> > Sometimes the rate_limit_status call is not returning a 302 to
> > redirect, or the rate_limit_status xml, but HTML with a meta refresh
> > in it (which curl doesnt understand to follow redirect/retry).
>
> > Its not huge problem for us, but it can affect some throttling code
> > people may or may not be implementing. (when you get this response, a
> > subsequent retry request usually succeeds 95% of the time)
>
> > Here is an example response I am talking about:
>
> > curlhttp://twitter.com/account/rate_limit_status.xml-L -v
> >> [1]http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0ra...

Chris Babcock

unread,
Aug 9, 2009, 2:26:45 AM8/9/09
to twitter-deve...@googlegroups.com

> Hmm, it shouldn't be spitting back HTML. How often are you seeing
> this?

I've seen it a couple times tonight and I'm not even doing any real
testing, just trying a few variations manually to see if I might fine
tune some of the calls in the library that I'm writing.

Chris Babcock

Naveen Ayyagari

unread,
Aug 9, 2009, 1:16:53 PM8/9/09
to twitter-deve...@googlegroups.com
I see this behavior 1/4 times I call rate_limit_status and I call
rate_limit_status every 5 minutes..

Rich

unread,
Aug 9, 2009, 2:02:16 PM8/9/09
to Twitter Development Talk
The one client we are having the most issues with is simple a
timeline / DM checker for iPhone push notifications... because these
are ALL get request I'ved made sure that they all have the follow
location enabled now.

Just oAuth and random blocking/rate limiting that I hope they work on
next :)

On Aug 9, 6:16 pm, Naveen Ayyagari <knig...@gmail.com> wrote:
> I see this behavior 1/4 times I call rate_limit_status and I call  
> rate_limit_status every 5 minutes..
> On Aug 8, 2009, at 9:01 PM, CaMason wrote:
>
>
>
>
>
> > To confirm, I am also seeing this behaviour. Some output I've received
> > on numerous occasions this evening:
>
> > -bash-3.2# curl --interface eth0http://twitter.com/account/rate_limit_status.xml

TjL

unread,
Aug 9, 2009, 3:42:29 PM8/9/09
to twitter-deve...@googlegroups.com
On Sat, Aug 8, 2009 at 2:37 AM, Chad Etzel<ch...@twitter.com> wrote:
> You may have to follow redirects more than once *wink wink nudge nudge*
>
> with curl you can add --location flag. There's a good bit of info in
> the man page as well.

So instead of doing

I should be doing

curl --location --referer ";auto" --netrc -s -D -
http://twitter.com/account/rate_limit_status.xml

(where "http://twitter.com/account/rate_limit_status.xml" is just one example)

TjL

ps - I'm not doing this through PHP, it's all on the commandline

Reply all
Reply to author
Forward
0 new messages