How do I handle 302 redirects with curl?

Showing 1-12 of 12 messages
How do I handle 302 redirects with curl? TjL 8/7/09 10:31 PM
All of my scripts check for "Status 200" before proceeding.

Now we are (sometimes) getting a 302, but when I try

curl --netrc -s -D - http://twitter.com/account/rate_limit_status.xml

Gave me a 302 with a Location of:

http://twitter.com/account/rate_limit_status.xml?c73f7db0

but when I tried

curl --netrc -s -D - 'http://twitter.com/account/rate_limit_status.xml?c73f7db0'

it seemed to want to redirect me to

http://twitter.com/account/rate_limit_status.xml

If "accepting 30x" is a requirement now, I'd like some advice on how to do so.

TjL

Re: [twitter-dev] How do I handle 302 redirects with curl? Chad Etzel 8/7/09 11:37 PM
You may have to follow redirects more than once *wink wink nudge nudge*

with curl you can add --location flag. There's a good bit of info in
the man page as well.

If using curl with PHP, you can set:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);

HTH,
-Chad

Re: How do I handle 302 redirects with curl? Rich 8/8/09 12:14 AM
Excellent our client now supports the 302's :)

On Aug 8, 7:37 am, Chad Etzel <c...@twitter.com> wrote:
> You may have to follow redirects more than once *wink wink nudge nudge*
>
> with curl you can add --location flag. There's a good bit of info in
> the man page as well.
>
> If using curl with PHP, you can set:
> curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
>
> HTH,
> -Chad
>
>
>
> On Sat, Aug 8, 2009 at 1:31 AM, TjL<luo...@gmail.com> wrote:
>
> > All of my scripts check for "Status 200" before proceeding.
>
> > Now we are (sometimes) getting a 302, but when I try
>
> > curl --netrc -s -D -http://twitter.com/account/rate_limit_status.xml
Re: How do I handle 302 redirects with curl? timwhitlock 8/8/09 3:31 AM
I've seem the 302 Location headers having invalid URLs... i.e. two "?"
symbols.
The original query string and then an additional "?" for the token at
the end.
Following this redirect blindly has resulted in a Forbidden response.

Also it is unclear whether the redirect location needs to be re-
signed? (I am not doing so, but may explain the 403?)
Re: [twitter-dev] Re: How do I handle 302 redirects with curl? Chad Etzel 8/8/09 8:20 AM
Hmm,

Unfortunately this 302 business will completely goof OAuth calls.

If you are able to programmatically see that you are getting these
redirects, try calling the account/rate_limit_status call [1] (it
could be any call, but this one is "free" and is a GET). You should
still get a 302 (I'm pretty sure). Then if you jump through the
redirect hoops with this call, it should clear you from more 302's for
a while.

I'm out today, but if someone could try this and report back if it
works that would be helpful.

[1] http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0rate_limit_status

Thanks,
-Chad

Re: [twitter-dev] Re: How do I handle 302 redirects with curl? Naveen 8/8/09 10:02 AM
Sometimes the rate_limit_status call is not returning a 302 to  
redirect, or the rate_limit_status xml, but HTML with a meta refresh  
in it (which curl doesnt understand to follow redirect/retry).

Its not huge problem for us, but it can affect some throttling code  
people may or may not be implementing. (when you get this response, a  
subsequent retry request usually succeeds 95% of the time)


Here is an example response I am talking about:

curl http://twitter.com/account/rate_limit_status.xml -L -v
* About to connect() to twitter.com port 80 (#0)
*   Trying 168.143.162.68... connected
* Connected to twitter.com (168.143.162.68) port 80 (#0)
 > GET /account/rate_limit_status.xml HTTP/1.1
 > User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2  
OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8
 > Host: twitter.com
 > Accept: */*
 >
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Connection: Close
< Pragma: no-cache
< cache-control: no-cache
< Refresh: 0.1
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd
">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>

Re: [twitter-dev] Re: How do I handle 302 redirects with curl? Chad Etzel 8/8/09 3:25 PM
Hmm, it shouldn't be spitting back HTML. How often are you seeing this?
-Chad
Re: How do I handle 302 redirects with curl? Craig Mason 8/8/09 6:01 PM
To confirm, I am also seeing this behaviour. Some output I've received
on numerous occasions this evening:


-bash-3.2# curl --interface eth0 http://twitter.com/account/rate_limit_status.xml
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/
TR/1999/REC-html401-19991224/strict.dtd
">
<!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"> -->
<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0.1">
<META HTTP-EQUIV="Pragma" CONTENT="no-cache">
<META HTTP-EQUIV="Expires" CONTENT="-1">
<TITLE></TITLE>
</HEAD>
<BODY><P></BODY>
</HTML>

-Craig

On Aug 8, 11:25 pm, Chad Etzel <c...@twitter.com> wrote:
> Hmm, it shouldn't be spitting back HTML. How often are you seeing this?
> -Chad
>
>
>
> On Sat, Aug 8, 2009 at 1:02 PM, Naveen Ayyagari<knig...@gmail.com> wrote:
>
> > Sometimes the rate_limit_status call is not returning a 302 to
> > redirect, or the rate_limit_status xml, but HTML with a meta refresh
> > in it (which curl doesnt understand to follow redirect/retry).
>
> > Its not huge problem for us, but it can affect some throttling code
> > people may or may not be implementing. (when you get this response, a
> > subsequent retry request usually succeeds 95% of the time)
>
> > Here is an example response I am talking about:
>
> > curlhttp://twitter.com/account/rate_limit_status.xml-L -v
> >> [1]http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-account%C2%A0ra...
>
> >> Thanks,
> >> -Chad
>
> >> On Sat, Aug 8, 2009 at 6:31 AM,
Re: [twitter-dev] Re: How do I handle 302 redirects with curl? Chris Babcock 8/8/09 11:26 PM

> Hmm, it shouldn't be spitting back HTML. How often are you seeing
> this?

I've seen it a couple times tonight and I'm not even doing any real
testing, just trying a few variations manually to see if I might fine
tune some of the calls in the library that I'm writing.

Chris Babcock

Re: [twitter-dev] Re: How do I handle 302 redirects with curl? Naveen 8/9/09 10:16 AM
I see this behavior 1/4 times I call rate_limit_status and I call  
rate_limit_status every 5 minutes..
Re: How do I handle 302 redirects with curl? Rich 8/9/09 11:02 AM
The one client we are having the most issues with is simple a
timeline / DM checker for iPhone push notifications...  because these
are ALL get request I'ved made sure that they all have the follow
location enabled now.

Just oAuth and random blocking/rate limiting that I hope they work on
next :)

On Aug 9, 6:16 pm, Naveen Ayyagari <knig...@gmail.com> wrote:
> I see this behavior 1/4 times I call rate_limit_status and I call  
> rate_limit_status every 5 minutes..
> On Aug 8, 2009, at 9:01 PM, CaMason wrote:
>
>
>
>
>
> > To confirm, I am also seeing this behaviour. Some output I've received
> > on numerous occasions this evening:
>
> > -bash-3.2# curl --interface eth0http://twitter.com/account/rate_limit_status.xml
Re: [twitter-dev] Re: How do I handle 302 redirects with curl? TjL 8/9/09 12:42 PM
On Sat, Aug 8, 2009 at 2:37 AM, Chad Etzel<ch...@twitter.com> wrote:
> You may have to follow redirects more than once *wink wink nudge nudge*
>
> with curl you can add --location flag. There's a good bit of info in
> the man page as well.

So instead of doing

curl --netrc -s -D - http://twitter.com/account/rate_limit_status.xml

I should be doing

curl --location --referer ";auto" --netrc -s -D -
http://twitter.com/account/rate_limit_status.xml

(where "http://twitter.com/account/rate_limit_status.xml" is just one example)

TjL

ps - I'm not doing this through PHP, it's all on the commandline