Twitter streaming API using oauth with tracks that have spaces?

771 views
Skip to first unread message

dan

unread,
Nov 9, 2010, 1:11:00 PM11/9/10
to Twitter Development Talk
I've been having trouble connecting to the streaming API using oauth
if my tracks have spaces. I get 401s (unauthorized). In all cases, the
same code works if the tracks don't have spaces.

In Java: tried twitter4j (http://twitter4j.org/jira/browse/TFJ-420)
and tweetstream4j (http://stackoverflow.com/questions/4129622/
connecting-to-twitter-streaming-api-with-tracks-with-spaces-using-
apache-httpclie)

In Python: tried tweepy (https://github.com/joshthecoder/tweepy/
issues#issue/64)

The Twitter example using curl (http://dev.twitter.com/pages/
streaming_api_methods#track) works with tracks that have spaces, but
it's basic auth.

I am wondering if some oauth encoding versus POST param encoding is
not working out.

Can someone point me to a code example in Java or Python that is known
to work connecting to the Twitter streaming API using oauth that has
spaces in its tracks?

Thanks in advance.

Ciaran

unread,
Nov 10, 2010, 4:38:08 AM11/10/10
to twitter-deve...@googlegroups.com
Try ui-encoding them first, my understanding of the Twitter OAuth
signature validation is that it is non-standard (although there
appears to be debate about this) I suspect if you encode them first
before signing the url it will start to work
-cj.

> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group: http://groups.google.com/group/twitter-development-talk
>

Taylor Singletary

unread,
Nov 10, 2010, 9:56:54 AM11/10/10
to twitter-deve...@googlegroups.com
Think of it this way.. a valid POST body already must contain application/x-www-form-urlencoded encoded values for the body to be valid. Normalizing spaces to %20, and avoiding "+" is also a best practice. OAuth kicks in after you've already constructed a valid POST body.

Here's an example of tracking a term with a space character in it: "twitter api"

== POST Body
track=twitter%20api

== signature_base_string 

POST&http%3A%2F%2Fstream.twitter.com%2F1%2Fstatuses%2Ffilter.json&oauth_consumer_key%3Dri8JxYK2ddwSV5xIUfNNvQ%26oauth_nonce%3DQKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1289400791%26oauth_token%3D819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw%26oauth_version%3D1.0%26track%3Dtwitter%2520api

== Authorization Header
Authorization: OAuth oauth_nonce="QKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1289400791", oauth_consumer_key="ri8JxYK2ddwSV5xIUfNNvQ", oauth_token="819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw", oauth_signature="jaEvelmcrQOkHdWADBvwZsQeGiQ%3D", oauth_version="1.0"

Taylor

dan

unread,
Nov 12, 2010, 6:29:51 PM11/12/10
to Twitter Development Talk
Taylor,

Thanks for your response. The crux might be

> Normalizing spaces to %20, and avoiding "+" is also a best practice.

tweetstream4j uses Apache's HttpClient 4.0 (see
http://hc.apache.org/httpcomponents-client-ga/index.html). I believe
if a request has application/x-www-form-urlencoded params, HttpClient
URL-encodes (i.e. space => +), and one can't percent-encode beforehand
because the param will be double-encoded, e.g. foo%20bar => foo
%2520bar instead of "foo bar". One also can't avoid the HttpClient URL
encoding by using a different type of param, because then the params
are not labeled application/x-www-form-urlencoded.

I wouldn't be surprised if twitter4j and tweepy were in a similar sort
of bind, though I have not verified.

Can you elaborate on why avoiding + is so important? I would hate to
have to patch Apache's HttpClient.

Also, do you know of any Java or Python library that gets this right?


On Nov 10, 8:56 am, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:
> Think of it this way.. a valid POST body already must contain
> application/x-www-form-urlencoded encoded values for the body to be valid.
> Normalizing spaces to %20, and avoiding "+" is also a best practice. OAuth
> kicks in after you've already constructed a valid POST body.
>
> Here's an example of tracking a term with a space character in it: "twitter
> api"
>
> == POST Body
> track=twitter%20api
>
> == signature_base_string
> POST&http%3A%2F%2Fstream.twitter.com
> %2F1%2Fstatuses%2Ffilter.json&oauth_consumer_key%3Dri8JxYK2ddwSV5xIUfNNvQ%26oauth_nonce%3DQKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1289400791%26oauth_token%3D819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw%26oauth_version%3D1.0%26track%3Dtwitter%2520api
>
> == Authorization Header
> Authorization: OAuth
> oauth_nonce="QKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM",
> oauth_signature_method="HMAC-SHA1", oauth_timestamp="1289400791",
> oauth_consumer_key="ri8JxYK2ddwSV5xIUfNNvQ",
> oauth_token="819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw",
> oauth_signature="jaEvelmcrQOkHdWADBvwZsQeGiQ%3D", oauth_version="1.0"
>
> Taylor
>
> On Wed, Nov 10, 2010 at 1:38 AM, Ciaran <ciar...@gmail.com> wrote:
> > Try ui-encoding them first, my understanding of the Twitter OAuth
> > signature validation is that it is non-standard (although there
> > appears to be debate about this) I suspect if you encode them first
> > before signing the url it will start to work
> > -cj.
>

dan

unread,
Nov 12, 2010, 6:31:21 PM11/12/10
to Twitter Development Talk
By ui-encoding you mean percent-encoding?

On Nov 10, 3:38 am, Ciaran <ciar...@gmail.com> wrote:
> Try ui-encoding them first, my understanding of the Twitter OAuth
> signature validation is that it is non-standard (although there
> appears to be debate about this) I suspect if you encode them first
> before signing the url it will start to work
> -cj.
>

Taylor Singletary

unread,
Nov 12, 2010, 7:01:25 PM11/12/10
to twitter-deve...@googlegroups.com
The differences between "+" are "%20" are subtle. 

When you are using + instead, it should still work as long as + is represented in your signature base string as: %2B

Are you able to see what + yields in your signature base string?

A little background on the difference between + and %20 in URLs...

"+" only means space in a URL when in the context of a field name or field value -- otherwise, within a URL it's literally a plus.. for this reason, it's a best practice to always be clear with intent. Use + when you mean plus in a URL component that is not a key/value pair, use %2B when you literally mean "+" in a string, and always use %20 when you mean a "space" within a string.

Ideally, a web server will consider %20 and + within the context of a URL value the same -- but this gets complicated by OAuth when calculating signature verification -- you don't want to be "fuzzy" on what you consider is valid or not.

Taylor

Ciaran

unread,
Nov 13, 2010, 9:17:23 AM11/13/10
to twitter-deve...@googlegroups.com
On Fri, Nov 12, 2010 at 11:31 PM, dan <dfra...@gmail.com> wrote:
> By ui-encoding you mean percent-encoding?
I did mean uri-encoding ;) iphone's suck :)
-cj.

deepa

unread,
Aug 10, 2011, 6:31:07 AM8/10/11
to twitter-deve...@googlegroups.com
can anyone post the sample using streaimg API..
im not gettng how to start with this.......


modsaid

unread,
Aug 11, 2011, 12:49:23 AM8/11/11
to Twitter Development Talk
I'm an early beginning myself.

you can try following the stream of a hashtag at
http://stream.twitter.com/1/statuses/filter.json?track=%23DamnItsTrue
for #DamnItsTrue

It require basic authentication with a valid twitter account
Reply all
Reply to author
Forward
0 new messages