Twitter streaming API using oauth with tracks that have spaces?

[dan]

I've been having trouble connecting to the streaming API using oauth
if my tracks have spaces. I get 401s (unauthorized). In all cases, the
same code works if the tracks don't have spaces.

In Java: tried twitter4j (http://twitter4j.org/jira/browse/TFJ-420)
and tweetstream4j (http://stackoverflow.com/questions/4129622/
connecting-to-twitter-streaming-api-with-tracks-with-spaces-using-
apache-httpclie)

In Python: tried tweepy (https://github.com/joshthecoder/tweepy/
issues#issue/64)

The Twitter example using curl (http://dev.twitter.com/pages/
streaming_api_methods#track) works with tracks that have spaces, but
it's basic auth.

I am wondering if some oauth encoding versus POST param encoding is
not working out.

Can someone point me to a code example in Java or Python that is known
to work connecting to the Twitter streaming API using oauth that has
spaces in its tracks?

Thanks in advance.

[Ciaran]

Try ui-encoding them first, my understanding of the Twitter OAuth
signature validation is that it is non-standard (although there
appears to be debate about this) I suspect if you encode them first
before signing the url it will start to work
-cj.

> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group: http://groups.google.com/group/twitter-development-talk
>

[Taylor Singletary]

Think of it this way.. a valid POST body already must contain application/x-www-form-urlencoded encoded values for the body to be valid. Normalizing spaces to %20, and avoiding "+" is also a best practice. OAuth kicks in after you've already constructed a valid POST body.

Here's an example of tracking a term with a space character in it: "twitter api"

== POST Body

track=twitter%20api

== signature_base_string 


POST&http%3A%2F%2Fstream.twitter.com%2F1%2Fstatuses%2Ffilter.json&oauth_consumer_key%3Dri8JxYK2ddwSV5xIUfNNvQ%26oauth_nonce%3DQKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1289400791%26oauth_token%3D819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw%26oauth_version%3D1.0%26track%3Dtwitter%2520api

== Authorization Header

Authorization: OAuth oauth_nonce="QKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1289400791", oauth_consumer_key="ri8JxYK2ddwSV5xIUfNNvQ", oauth_token="819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw", oauth_signature="jaEvelmcrQOkHdWADBvwZsQeGiQ%3D", oauth_version="1.0"

Taylor

[dan]

Taylor,

Thanks for your response. The crux might be

> Normalizing spaces to %20, and avoiding "+" is also a best practice.

tweetstream4j uses Apache's HttpClient 4.0 (see
http://hc.apache.org/httpcomponents-client-ga/index.html). I believe
if a request has application/x-www-form-urlencoded params, HttpClient
URL-encodes (i.e. space => +), and one can't percent-encode beforehand
because the param will be double-encoded, e.g. foo%20bar => foo
%2520bar instead of "foo bar". One also can't avoid the HttpClient URL
encoding by using a different type of param, because then the params
are not labeled application/x-www-form-urlencoded.

I wouldn't be surprised if twitter4j and tweepy were in a similar sort
of bind, though I have not verified.

Can you elaborate on why avoiding + is so important? I would hate to
have to patch Apache's HttpClient.

Also, do you know of any Java or Python library that gets this right?


On Nov 10, 8:56 am, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> Think of it this way.. a valid POST body already must contain
> application/x-www-form-urlencoded encoded values for the body to be valid.
> Normalizing spaces to %20, and avoiding "+" is also a best practice. OAuth
> kicks in after you've already constructed a valid POST body.
>
> Here's an example of tracking a term with a space character in it: "twitter
> api"
>
> == POST Body
> track=twitter%20api
>
> == signature_base_string
> POST&http%3A%2F%2Fstream.twitter.com
> %2F1%2Fstatuses%2Ffilter.json&oauth_consumer_key%3Dri8JxYK2ddwSV5xIUfNNvQ%26oauth_nonce%3DQKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1289400791%26oauth_token%3D819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw%26oauth_version%3D1.0%26track%3Dtwitter%2520api
>
> == Authorization Header
> Authorization: OAuth
> oauth_nonce="QKWqIP8eEedgOPk5ujopscNxqeoafDNC0r6TZyLFM",
> oauth_signature_method="HMAC-SHA1", oauth_timestamp="1289400791",
> oauth_consumer_key="ri8JxYK2ddwSV5xIUfNNvQ",
> oauth_token="819797-torCkTs0XK7H2Y2i1ee5iofqkMC4p7aayeEXRTmlw",
> oauth_signature="jaEvelmcrQOkHdWADBvwZsQeGiQ%3D", oauth_version="1.0"
>
> Taylor
>

> On Wed, Nov 10, 2010 at 1:38 AM, Ciaran <ciar...@gmail.com> wrote:
> > Try ui-encoding them first, my understanding of the Twitter OAuth
> > signature validation is that it is non-standard (although there
> > appears to be debate about this) I suspect if you encode them first
> > before signing the url it will start to work
> > -cj.
>

[dan]

By ui-encoding you mean percent-encoding?

On Nov 10, 3:38 am, Ciaran <ciar...@gmail.com> wrote:
> Try ui-encoding them first, my understanding of the Twitter OAuth
> signature validation is that it is non-standard (although there
> appears to be debate about this) I suspect if you encode them first
> before signing the url it will start to work
> -cj.
>

[Taylor Singletary]

The differences between "+" are "%20" are subtle. 

When you are using + instead, it should still work as long as + is represented in your signature base string as: %2B

Are you able to see what + yields in your signature base string?

A little background on the difference between + and %20 in URLs...

"+" only means space in a URL when in the context of a field name or field value -- otherwise, within a URL it's literally a plus.. for this reason, it's a best practice to always be clear with intent. Use + when you mean plus in a URL component that is not a key/value pair, use %2B when you literally mean "+" in a string, and always use %20 when you mean a "space" within a string.

Ideally, a web server will consider %20 and + within the context of a URL value the same -- but this gets complicated by OAuth when calculating signature verification -- you don't want to be "fuzzy" on what you consider is valid or not.

Taylor

[Ciaran]

On Fri, Nov 12, 2010 at 11:31 PM, dan <dfra...@gmail.com> wrote:
> By ui-encoding you mean percent-encoding?

I did mean uri-encoding ;) iphone's suck :)
-cj.

[deepa]

can anyone post the sample using streaimg API..
im not gettng how to start with this.......

[modsaid]

I'm an early beginning myself.

you can try following the stream of a hashtag at
http://stream.twitter.com/1/statuses/filter.json?track=%23DamnItsTrue
for #DamnItsTrue

It require basic authentication with a valid twitter account