/statuses/user_timeline.format missing in action?

[angusmci]

A part of our app that made use of the '/statuses/
user_timeline.format' endpoint has broken, and tests show that we're
now getting a 404 result from that (using 'curl', or the Perl
Net::Twitter module).

Has the API changed recently?

Angus

[Alex Payne]

Nope, hasn't changed. Is it possible that the user you're requesting
was deleted or marked as a spammer?

--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x

[angusmci]

The problem was reported by one of our users, but when I tested our
software with my own Twitter account - which is alive and in good
standing - I got the same 404 message.

If I use 'curl', i.e.

curl -u user:pass http://twitter.com/statuses/user_timeline.json

It works fine. However, if I use the Perl Net::Twitter equivalent,
i.e.

use Net::Twitter;
my $twitter = Net::Twitter->new(username => "user", password =>
"pass");
my $timeline = $twitter->user_timeline({ count => 1 });
print $twitter->http_code, " ", $twitter->http_message();

I get the 404 Not Found error.

Both my simple Net::Twitter test case and our actual application have
worked well in the past, and our Net::Twitter module is up-to-date.

Is it possible that Twitter's behavior depends in some way on other
headers sent by the client?

Thanks for your help,

Angus

[Alex Payne]

Certain headers like If-Modified-Since will effect the responses we
return. Can you please provide full header output?

[angusmci]

I don't have a way currently to monitor what's actually going out over
the wire, but I've scattered Data::Dumper calls through the
Net::Twitter code to inspect the data structures used by the libwww
user agent. According to that, Net::Twitter is requesting:

http://twitter.com/statuses/user_timeline.json?count=1&

and it's sending the headers:

'user-agent' => 'Net::Twitter/1.17 (PERL)',
'x-twitter-client-version:' => '1.17',
'x-twitter-client:' => 'Perl Net::Twitter',
'x-twitter-client-url:' => 'http://x4.net/twitter/meta.xml'

The authentication information that it should be sending corresponds
to a valid Twitter account. It's supposedly using
'basic_authentication' against 'twitter.com:80', with realm set to
'Twitter API'.

The response headers sent back by Twitter are:

'connection' => 'close',
'client-response-num' => 1,
'set-cookie' => '_twitter_sess=...; domain=.twitter.com; path=/',
'cache-control' => 'no-cache, no-store, must-revalidate, pre-check=0,
post-check=0',
'last-modified' => 'Mon, 20 Oct 2008 18:17:52 GMT',
'status' => '404 Not Found',
'date' => 'Mon, 20 Oct 2008 18:17:52 GMT',
'client-peer' => '128.121.146.100:80',
'content-length' => '73',
'client-date' => 'Mon, 20 Oct 2008 18:17:49 GMT',
'pragma' => 'no-cache',
'content-type' => 'application/json; charset=utf-8',
'server' => 'hi',
'expires' => 'Tue, 31 Mar 1981 05:00:00 GMT'

(I've elided the Twitter session cookie just in case it contains
anything sensitive).

Some more data points:

- the same request submitted to the corresponding API endpoint on
identi.ca works correctly.
- if I add an id parameter to my call to Net::Twitter's
user_timeline() method, Twitter returns a valid result.
- if I substitute 'friends' for 'user_timeline', I get the same
behavior: 404 with no 'id' parameter, 200 if I supply an 'id'
parameter.
- if I substitute 'friends_timeline' for 'user_timeline', it works
without problems: I get a 200 response, and appropriate JSON data
- the code implementing 'friends_timeline' and 'user_timeline' in
Net::Twitter seems to be effectively identical.
- if I send friends_timeline the wrong password, it replies '401 Not
authorized'.
- if I send user_timeline the wrong password, it replies '404 Not
Found'.
- testing from a different host/network produces the same results
- requesting 'http://twitter.com/statuses/user_timeline.json?count=1&'
using curl and the proper authentication credentials works
- this worked successfully at least as recently as September 22nd

The fact that sending an 'id' parameter (which asks Twitter to return
information for the user identified by 'id', and which may not depend
on authentication) made me think that it was an authentication issue
and that Net::Twitter was somehow fumbling authentication (although
since Net::Twitter is using Perl LWP, I don't know how that could
happen). However, 'friends_timeline' works as expected.

Thanks again for your help,

Angus

On Oct 20, 1:51 pm, "Alex Payne" <a...@twitter.com> wrote:
> Certain headers like If-Modified-Since will effect the responses we
> return.  Can you please provide full header output?
>
>
>
> On Sat, Oct 18, 2008 at 11:44 AM, angusmci <an...@pobox.com> wrote:
>
> > The problem was reported by one of our users, but when I tested our
> > software with my own Twitter account - which is alive and in good
> > standing - I got the same 404 message.
>
> > If I use 'curl', i.e.
>

> >    curl -u user:passhttp://twitter.com/statuses/user_timeline.json

[Alex Payne]

Are you quite sure that you're making the request authenticated? It
will return a 404 if it can't authenticate you, because that URL
doesn't specify a user ID to retrieve a timeline for and thus assumes
that you want the timeline for the requesting user.

[angusmci]

I'm pretty sure that the request is being made authenticated. Reasons
for thinking so include:

1. I'm passing the correct authentication credentials to the
Net::Twitter object.
2. Dumping the data structure corresponding to the Perl libwww user
agent shows an 'authentication' structure containing those
credentials; this should mean that those are the credentials that
libwww will send to Twitter.
3. While the 'user_timeline' method returns 404 whatever I send it, if
I switch to using the 'friends_timeline' method, I will get either a
'200' or a '401' response, depending on whether I pass in a correct
username/password combination or not.

The 'following_timeline' method (which wraps a call to '/statuses/
friends_timeline' in the Twitter API) and the 'user_timeline' method
(which wraps a call to '/statuses/user_timeline') are line-by-line
identical except for the URL, so I'd expect them to behave the same
way in the case of an authentication failure.

Are there any headers or arguments that are mandatory for
'user_timeline' that are not required for other, similar API
functions?

Thanks, Angus

[Alex Payne]

> Are there any headers or arguments that are mandatory for
> 'user_timeline' that are not required for other, similar API
> functions?

No, none. If you want the user timeline for the authenticating user,
you need to provide authentication. If you want the user timeline for
another user, you must specify it as part of the URL. That's all
there is to it.

[angusmci]

Thanks for your answer.

I decided to bypass the Net::Twitter library altogether, and just talk
to Twitter directly using LWP (Perl's web programming library).

I sent two requests to Twitter, one for "http://twitter.com/statuses/
user_timeline.json", and one for "http://twitter.com/statuses/
friends_timeline.json". In each case, I sent Twitter valid credentials
(i.e. the username and password that I use to log in to Twitter). I
set the LWP user agent up in exactly the same way each case.

The request for 'user_timeline' returned 404 Not Found; the request
for 'friends_timeline' returned a list of updates from people I'm
following.

If the authentication credentials I'm sending weren't valid, then the
request for 'friends_timeline' should return a 401 error. The fact
that it didn't suggests that the test script is authenticating it
successfully at least in some cases.

I'm sorry to keep bothering you with this, but it really looks as if
some parts - but only some parts - of the Twitter API have a
mysterious prejudice against Perl ('curl' works fine). I'll try with
Ruby and Python test scripts and see if that makes any difference.

Angus

[Matt Sanford]

Hi Angus,

Back in the day I was known to write some Perl so I took a crack at
user_timeline. The following works fine for me:

use LWP::UserAgent;
use HTTP::Request;

my $ua = LWP::UserAgent->new;
my $req=HTTP::Request->new(GET => "http://twitter.com/statuses/
user_timeline.json");
$req->authorization_basic('username here', 'password here');
print $ua->request($req)->as_string

Or, the exact command I used:

perl -MLWP::UserAgent -e '$ua = LWP::UserAgent->new;
$req=HTTP::Request->new(GET => "http://twitter.com/statuses/
user_timeline.json"); $req->authorization_basic($ARGV[0],$ARGV[1]);
print $ua->request($req)->as_string' USERNAME PASSWORD

Where USERNAME and PASSWORD are your user information. The LWP
library is finicky about authentication.

— Matt

[Cameron Kaiser]

> I'm sorry to keep bothering you with this, but it really looks as if
> some parts - but only some parts - of the Twitter API have a
> mysterious prejudice against Perl ('curl' works fine).

TTYtter hasn't ever had a problem, though admittedly it uses curl, not LWP.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Don't Be Evil. -- Paul Buchheit --------------------------------------------

[angusmci]

Thanks Matt

I think when you said "The LWP library is finicky about
authentication", you nailed it. There's more than one way to tell LWP
how to authenticate, and calling 'basic_authorization' on the request
object seems to make Twitter happy, whereas calling 'credentials' on
the user agent object apparently doesn't. Net::Twitter uses
'credentials' rather than 'basic_authorization'. I'm not sure I want
to do the necessary LWP spelunking to figure out what the difference
is between the two approaches as far as the consumer is concerned.

Curiously, the 'credentials' technique appears to have worked in the
past, still works for 'friends_timeline' on Twitter, and works for
both 'friends_timeline' and 'user_timeline' on the Laconica
implementation of the Twitter API. Go figure.

One thing I notice: if you send the wrong credentials to
'user_timeline', Twitter gives back a 404 Not Found code (rather than
401 Not Authorized). If you send the wrong credentials to
'friends_timeline', Twitter gives back a 401 Not Authorized. This
apparent inconsistency may be a deliberate choice on your part,
though.

I'm going to switch our application code over to using raw LWP rather
than Net::Twitter for the small number of calls we need, and that will
probably take care of it.

Thanks to you and Alex for your help,

Angus

[Chris Thompson]

I am the developer of Net::Twitter.

Or, at least, I was before I handed it off because I grew tired of trying to keep up with the foibles of the API. But, since the new guy hasn't released anything, my name is still on the most recent version. So I get emails from people, and questions on irc.perl.org about this.

The problem in this case lies squarely on Twitter's side.

Alex says:

>Are you quite sure that you're making the request authenticated?  It
>will return a 404 if it can't authenticate you, because that URL
>doesn't specify a user ID to retrieve a timeline for and thus assumes
>that you want the timeline for the requesting user.

This is not how HTTP Auth works.

The correct handshake for a URL that needs Auth is:

1) I request, with no WWW-Authenticate: header
2) Server responds with a 401: Unauthorized and a WWW-Authenticate header containing the realm
3) I re-request with the WWW-Authenticate header containing user/pass
4) Server decides that auth header is good, responds with a 200, or decides it's bad and goes back to #2

Net::Twitter uses perl's libwww (LWP) which, in turn, implements the HTTP protocol to spec. It doesn't send the WWW-Authenticate header until it sees a 401. This is a specific part of HTTP as defined in RFC2617.

If you think about it in terms of a browser like firefox, the browser CAN'T send an auth header until it is told it needs one, and it puts up an auth popup with the Realm listed that it got from the 401.

LWP is doing the right thing, Twitter simply isn't asking for the auth.

If you use curl or wget from the command line to hit the user_timeline url, it works. The reason for this is, you specify user and pass on the command line and both curl or wget just jam the WWW-Authenticate header in there whether it ever gets asked for it or not, violating RFC.

Same with Matt Sanford's perl using authorization_basic. This is not part of LWP::UserAgent, but part of HTTP::Headers and what it does is force the WWW-Authorize header into the request, always-on, just like curl and wget, and yet again violating the RFC.

LWP is only "being finicky" if by finicky you mean "Implementing RFC2617 as written".

I hate to be a pest on this, but the credentials code in Net::Twitter hasn't changed at all since Net::Twitter 1.0.0 way back in March of 2007. You guys are doing the right thing everywhere except user_timeline. If you had it throw the 401 first, you'd get the auth. 404's just flat wrong here.

--
------------------------
Chris Thompson

[FrankieShakes]

I'm experiencing a similar issue with my Dashboard Twitter client
using XMLHttpRequest.

I'm using JSON as my format and receive a "Not Found" response when
requesting the authenticating user's timeline. I pass the user
credentials as part of the URL. ie:
http://username:pass...@twitter.com/statuses/user_timeline.json?count=1

Any ideas why this would be happening? I see I'm not the only one.


Thanks,
Frank

On Oct 27, 10:10 pm, "Chris Thompson" <chris.thomp...@gmail.com>
wrote:

[Alex Payne]

You entirely right Chris. The onus is on us. I'll get this fixed up
tomorrow. Sorry to anyone who lost time on this bug!

--

[Alex Payne]

I'm tracking this issue here, for the record:
http://code.google.com/p/twitter-api/issues/detail?id=135

[Chris Thompson]

Alex,

Thanks. Of course, you've got me listed as Chris Anderson, but hey, I'm not in it for the fame :)

On Tue, Oct 28, 2008 at 2:25 PM, Alex Payne <al...@twitter.com> wrote:

I'm tracking this issue here, for the record:
http://code.google.com/p/twitter-api/issues/detail?id=135

--
------------------------
Chris Thompson

[Alex Payne]

Great, now I have to open an issue for that issue.

--

[Jesse Stay]

Chris, one more thing I noticed is that at times Net::Twitter seems to be returning 200 OK in the status_line returned for the object (set by $req->status_line when I set it in sub follow()), when the as_string Twitter sends back has a Status of 403 forbidden.  (in this case it's a suspended account)  Is this a Net::Twitter issue or is it something similar to what happened here, where Twitter's returning the data wrong, or am I just doing it wrong?  For the following authenticated URL:

http://twitter.com/favorites/create/MartinAllsop.xml

as_string() returns this:

HTTP/1.1 403 Forbidden

Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0

Connection: close

Date: Wed, 29 Oct 2008 00:59:26 GMT

Pragma: no-cache

Server: hi

Content-Length: 174

Content-Type: application/json; charset=utf-8

Expires: Tue, 31 Mar 1981 05:00:00 GMT

Last-Modified: Wed, 29 Oct 2008 00:59:26 GMT

Client-Date: Wed, 29 Oct 2008 00:59:23 GMT

Client-Peer: 128.121.146.100:80

Client-Response-Num: 1

Set-Cookie: _twitter_sess=BAh7BzoHaWQiJWY2M2VjOTE5Njg1Mzg1Y2FiOTJhZTJjNTAxNjA5NTVlIgpm%250AbGFzaElDOidBY3Rpb25Db250cm9sbGVyOjpGbGFzaDo6Rmxhc2hIYXNoewAG%250AOgpAdXNlZHsA--4b387873eb95ac39c23aa0ff692e360eb197079b; domain=.twitter.com; path=/

Status: 403 Forbidden

{"request":"\/friendships\/create\/MartinAllsop.json","error":"Could not follow user: This account is currently suspended and is being investigated due to strange activity."}