Using curl with Twitter

4 views
Skip to first unread message

TjL

unread,
Mar 9, 2009, 3:14:01 PM3/9/09
to twitter-deve...@googlegroups.com
I'm not sure how many other people know this, so my apologies if this
is "Duh" material, but I've never worked much with curl before (always
been a lynx man :-)

If you use 'curl --netrc' curl will check ~/.netrc for your Twitter
login information.

Just enter your Twitter login information like this:

machine twitter.com
login twitreport
password SuPerSEkret

Obviously change 'twitreport' to your Twittername and 'SuPerSEkret' to
your password. It can all be on one line if you prefer.

The nice thing is that this keeps your password from appearing in 'ps'
and you don't have to type it in each time, just add --netrc and it
will happen automatically.

TjL

Doug Williams

unread,
Mar 9, 2009, 3:25:07 PM3/9/09
to twitter-deve...@googlegroups.com
Great tip! This should become a great thread with cURL tips.

Below shows how to send POST data (note the -d flag):

curl -u username:password -d status="Updating with cURL"
http://twitter.com/statuses/update.xml

You can also send GET requests:

curl -u username:password http://twitter.com/statuses/friends_timeline.xml

What else does the community have to offer when it comes to our
favorite API unit test framework :)


Doug Williams
Twitter API Developer Support
http://twitter.com/dougw

Chad Etzel

unread,
Mar 9, 2009, 3:28:42 PM3/9/09
to twitter-deve...@googlegroups.com
> What else does the community have to offer when it comes to our
> favorite API unit test framework :)

I was just going to comment on this.... what happens when OAuth hits?
I fear that this very simple diagnostic mechanism will become
obsolete. Now asking someone to run a quick curl test becomes (I'm
assuming) more complicated.

-Chad

On Mon, Mar 9, 2009 at 3:25 PM, Doug Williams <do...@twitter.com> wrote:
>
> Great tip! This should become a great thread with cURL tips.
>
> Below shows how to send POST data (note the -d flag):
>
> curl -u username:password -d status="Updating with cURL"
> http://twitter.com/statuses/update.xml
>
> You can also send GET requests:
>
> curl -u username:password http://twitter.com/statuses/friends_timeline.xml
>
>
>

Alex Payne

unread,
Mar 9, 2009, 6:05:11 PM3/9/09
to twitter-deve...@googlegroups.com
We're discussing a local proxy that could be used for testing. It's
definitely a known problem.

--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x

TjL

unread,
Mar 10, 2009, 10:37:28 AM3/10/09
to twitter-deve...@googlegroups.com
On Mon, Mar 9, 2009 at 6:05 PM, Alex Payne <al...@twitter.com> wrote:
> We're discussing a local proxy that could be used for testing. It's
> definitely a known problem.

Um, am I reading this correctly?

is 'curl --netrc' not going to work anymore once OAuth is implemented?

TjL

Cameron Kaiser

unread,
Mar 10, 2009, 10:40:12 AM3/10/09
to twitter-deve...@googlegroups.com
> > We're discussing a local proxy that could be used for testing. It's
> > definitely a known problem.
>
> Um, am I reading this correctly?
> is 'curl --netrc' not going to work anymore once OAuth is implemented?

More accurately, it's not going to work when Basic Auth is discontinued.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- MOVIE IDEA: From Russia with E-mail Signature ------------------------------

Doug Williams

unread,
Mar 10, 2009, 10:45:30 AM3/10/09
to twitter-deve...@googlegroups.com
As Alex stated above, we know cURL usage will break if and when basic
authentication support is wholly discontinued. It's something we're
equally concerned about and something we would like to avoid. Stay
tuned.

Doug Williams
Twitter API Support
http://twitter.com/dougw

Abraham Williams

unread,
Mar 10, 2009, 10:54:19 AM3/10/09
to twitter-deve...@googlegroups.com
I believe it has been suggested on the list before Twitter implement a dev sandbox with "fake" data that continues to support BasicAuth similar to how BrightKite(?) does. Using Curl to get started is another reason to implement this.

Abraham
--
Abraham Williams | http://the.hackerconundrum.com
Web608 | Community Evangelist | http://web608.org
This email is: [ ] blogable [x] ask first [ ] private.

TjL

unread,
Mar 10, 2009, 10:55:07 AM3/10/09
to twitter-deve...@googlegroups.com
On Tue, Mar 10, 2009 at 10:45 AM, Doug Williams <do...@twitter.com> wrote:
> As Alex stated above, we know cURL usage will break if and when basic
> authentication support is wholly discontinued. It's something we're
> equally concerned about and something we would like to avoid. Stay
> tuned.

OK, I guess my next question is this:

Why turn off basic auth once OAuth is enabled?

Why not just leave them both?

(Not trying to be flip, I don't understand what's wrong with having
both "doors".)

TjL

Matt Sanford

unread,
Mar 10, 2009, 11:03:41 AM3/10/09
to twitter-deve...@googlegroups.com
The problem isn't having two doors; it's having one door with a security lock and alarm, and one held shut with some string :). In all seriousness, we would like to keep some way for people to easily test and we're trying to come up with something. We're also trying to figure out some sort of shared credentials for things like clients TwitPic (suggestions welcome off list at a...@twitter.com). We're hoping to deprecate BasicAuth in the future, but that may simply mean restricting its access to known clients/IPs. Here's the thinking:

1. As long as we have world-accessible BasicAuth it will be easier  to use than OAuth. I think we can agree that's a fact.
2. if it's easier, developers will prefer it.
3. if developers use it users will never be broken of the habit of giving out their passwords.
4. when passwords are misused, compromising user privacy, users will blame every app and twitter. This is a Bad Thing™

    I don't think any of this is news to anyone, but it's useful to keep in mind when suggesting we keep BasicAuth. 

Thanks;
  — Matt Sanford

TjL

unread,
Mar 10, 2009, 11:17:40 AM3/10/09
to twitter-deve...@googlegroups.com
Ok, well, I'll just try to stay current with future developments.

The only password that I ever need with BasicAuth and my programs is my own.

My programs need to be able to run unattended on the commandline
(triggered my procmail, for those who know what that is).

If there isn't a way to do that, I don't have a plan B.

I realize you're already aware of these potential problems, I'm just
processing it now.

TjL

Cameron Kaiser

unread,
Mar 10, 2009, 11:33:13 AM3/10/09
to twitter-deve...@googlegroups.com
> Ok, well, I'll just try to stay current with future developments.
>
> The only password that I ever need with BasicAuth and my programs is my own.
>
> My programs need to be able to run unattended on the commandline
> (triggered my procmail, for those who know what that is).
>
> If there isn't a way to do that, I don't have a plan B.

This is the same issue I'm working with for TTYtter, and I think I have a
solution which I plan to implement when OAuth goes open. I'll post more about
the workflow (it's open source) when I get a working beta operating.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- What the large print giveth, the small print taketh away. ------------------

Chad Etzel

unread,
Mar 10, 2009, 11:47:57 AM3/10/09
to twitter-deve...@googlegroups.com
On Tue, Mar 10, 2009 at 11:33 AM, Cameron Kaiser <spe...@floodgap.com> wrote:
>
>
> This is the same issue I'm working with for TTYtter, and I think I have a
> solution which I plan to implement when OAuth goes open. I'll post more about
> the workflow (it's open source) when I get a working beta operating.
>
> --

+1 on being interested in your workflow solution as I have a lot of
commandline stuff going on, too.
-Chad

Brian Gilham

unread,
Mar 10, 2009, 11:49:00 AM3/10/09
to twitter-deve...@googlegroups.com
Ditto.
-----Original Message-----
From: Chad Etzel <jazz...@gmail.com>

Date: Tue, 10 Mar 2009 11:47:57
To: <twitter-deve...@googlegroups.com>
Subject: [twitter-dev] Re: Using curl with Twitter

Cameron Kaiser

unread,
Mar 10, 2009, 12:08:19 PM3/10/09
to twitter-deve...@googlegroups.com
> > > This is the same issue I'm working with for TTYtter, and I think I have a
> > > solution which I plan to implement when OAuth goes open. I'll post more
> > > about the workflow (it's open source) when I get a working beta
> > > operating.
> >
> > +1 on being interested in your workflow solution as I have a lot of
> > commandline stuff going on, too.
>
> Ditto.

Well, since people are asking, the workflow doesn't significantly differ
from other OAuth applications and depends on the fact that access tokens
don't expire. When people start TTYtter up for the first time without an
access token (or TTYtter tries the access token and it fails), it asks for
the usual request token, prints the access URL with the request token it
wants the user to authorize, and waits for the user to authorize. Twitter,
presumably, will say, "ok, tell your program to continue." Back on TTYtter's
side, the user hits ENTER, and TTYtter exchanges its request token for an
access token *and caches it* once it has verified it can successfully hit
the user timeline for data. So far, this is not significantly different than
any other OAuth app.

The OAuth request token is then saved to disk and now TTYtter can run
unattended -- all that needs to happen is the user tells TTYtter which
credentials file to use, and TTYtter does not require further intervention
(I use TTYtter to automatically fetch replies from a number of my accounts
and mail them to me, so I wanted to make sure this would still work).

The real trick was what to do with keys in an open source app, but fortunately
the key issue is already solved because Twitter is presumably not relying on
oauth_consumer_key to unambiguously or securely identify consumer clients,
and the key needed for HMAC-SHA1 signatures can simply be randomly generated
at the point of asking for a request token. I hashed this out with Blaine
Cook in E-mail, so I am fairly confident this should work, and I appreciate
his help.

Note to Twitter: it would be nice if the OAuth authorization page were
viewable in something like Lynx, even if it were only trivially so. :) Good
for screen readers too.

When I have working code running, I'll announce. I plan to start working on
it right away when the beta opens.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- Don't Be Evil. -- Paul Buchheit --------------------------------------------

Cameron Kaiser

unread,
Mar 10, 2009, 12:09:29 PM3/10/09
to twitter-deve...@googlegroups.com
> The OAuth request token is then saved to disk and now TTYtter can run

s/request/access/

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- "I'd love to go out with you, but I'll be at the opening of my garage door."

Chad Etzel

unread,
Mar 10, 2009, 12:19:57 PM3/10/09
to twitter-deve...@googlegroups.com
> The real trick was what to do with keys in an open source app, but fortunately
> the key issue is already solved because Twitter is presumably not relying on
> oauth_consumer_key to unambiguously or securely identify consumer clients,

But then how could Twitter revoke access for a rogue app? I thought
the point of the app keys *is* to identify the app itself.

Also for Open Source apps, their keys are going to have to be locally
accessible, so now anyone could see the secret key.

With web apps (and to some extent client apps that can "obfuscate"
their keys), the keys are "safely" used behind a proxy to twitter
(i.e. my webapp server creates the actual network request using its
app keys) and the users never have access to them. With
commandline/open source/shell script/etc sorts of access, the app keys
will necessarily have to exist locally unless every request is sent
through some grand "Proxy Server In The Cloud(tm)"... ?

Just thinking out loud here,
-Chad

Cameron Kaiser

unread,
Mar 10, 2009, 12:25:13 PM3/10/09
to twitter-deve...@googlegroups.com
> > The real trick was what to do with keys in an open source app, but
> > fortunately the key issue is already solved because Twitter is presumably
> > not relying on oauth_consumer_key to unambiguously or securely identify
> > consumer clients,
>
> But then how could Twitter revoke access for a rogue app? I thought
> the point of the app keys *is* to identify the app itself.
> Also for Open Source apps, their keys are going to have to be locally
> accessible, so now anyone could see the secret key.

I thought so as well, but from section 4 of the OAuth Core 1.0 spec:

"Service Providers SHOULD NOT rely on the Consumer Secret as a method to
verify the Consumer identity, unless the Consumer Secret is known to be
inaccessible to anyone other than the Consumer and the Service Provider.
The Consumer Secret MAY be an empty string (for example when no Consumer
verification is needed, or when verification is achieved through other means
such as RSA)."

So the fact that situations exist where consumer keys cannot be considered
secure is considered in the OAuth spec. This was something Blaine set me
straight on.

My guess is Twitter will deal with rogues on a token-by-token basis rather
than a consumer key basis. As they (and I, and others) have been saying all
along, OAuth isn't insurance against bad apps, it's insurance against bad apps
having your password. Authorizing a bad app still allows it to do bad things,
but at least now you can revoke only that app's access token and not affect
other apps.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- BOND THEME NOW PLAYING: "The Living Daylights" -----------------------------

Adam

unread,
Mar 27, 2009, 7:28:00 PM3/27/09
to Twitter Development Talk, Cameron Kaiser
Does anyone have working code on how an unattended request from an app
would work?

-Adam

On Mar 10, 11:25 am, Cameron Kaiser <spe...@floodgap.com> wrote:
> > > The real trick was what to do with keys in an open source app, but
> > > fortunately the key issue is already solved because Twitter is presumably
> > > not relying on oauth_consumer_key to unambiguously or securely identify
> > > consumer clients,
>
> > But then how could Twitter revoke access for a rogue app? I thought
> > the point of the app keys *is* to identify the app itself.
> > Also for Open Source apps, their keys are going to have to be locally
> > accessible, so now anyone could see the secret key.
>
> I thought so as well, but from section 4 of theOAuthCore 1.0 spec:
>
> "Service Providers SHOULD NOT rely on the Consumer Secret as a method to
> verify the Consumer identity, unless the Consumer Secret is known to be
> inaccessible to anyone other than the Consumer and the Service Provider.
> The Consumer Secret MAY be an empty string (for example when no Consumer
> verification is needed, or when verification is achieved through other means
> such as RSA)."
>
> So the fact that situations exist where consumer keys cannot be considered
> secure is considered in theOAuthspec. This was something Blaine set me
> straight on.
>
> My guess is Twitter will deal with rogues on a token-by-token basis rather
> than a consumer key basis. As they (and I, and others) have been saying all
> along,OAuthisn't insurance against bad apps, it's insurance against bad apps
> having your password. Authorizing a bad app still allows it to do bad things,
> but at least now you can revoke only that app's access token and not affect
> other apps.
>
> --
> ------------------------------------ personal:http://www.cameronkaiser.com/--
>   Cameron Kaiser * Floodgap Systems *www.floodgap.com* cka...@floodgap.com
Reply all
Reply to author
Forward
0 new messages