Twitter search API: "Unauthorized" through HTTParty but successful via Curl

[Pascal Lindelauf]

Working on our Rails application, I'm trying to retrieve tweets using the search API v1.1 through HTTParty. I've carefully gone through all steps and unit tested our code with the Twitter examples to setup the right OAuth authorization, so I'm very confident all code is working properly. However, no matter what I try, I keep getting a `"{"errors":[{"message":"Could not authenticate you","code":32}]}""` response.

Using the OAuth Tool, I generated the request that I need. I pasted the nonce and timestamp into my code, to replicate the exact same request in our application. This is the resulting log:

    opening connection to api.twitter.com...
    opened
    <- "GET /1.1/search/tweets.json?q=from:plindelauf&include_entities=true HTTP/1.1\r\nAuthorization: OAuth oauth_consumer_key=\"DOP3re9nlTPeyemlmcqg\", oauth_nonce=\"05d29110d282b24e516dae1ae2b54e1a\", oauth_signature=\"k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1370691632\", oauth_token=\"288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f\", oauth_version=\"1.0\"\r\nConnection: close\r\nHost: api.twitter.com\r\n\r\n"
    -> "HTTP/1.1 401 Unauthorized\r\n"
    -> "content-length: 63\r\n"
    -> "content-type: application/json; charset=utf-8\r\n"
    -> "date: Sat, 08 Jun 2013 11:41:41 UTC\r\n"
    -> "server: tfe\r\n"
    -> "set-cookie: guest_id=v1%3A137069170192515887; Domain=.twitter.com; Path=/; Expires=Mon, 08-Jun-2015 11:41:41 UTC\r\n"
    -> "strict-transport-security: max-age=631138519\r\n"
    -> "Connection: close\r\n"
    -> "\r\n"
    reading 63 bytes...
    -> ""
    -> "{\"errors\":[{\"message\":\"Could not authenticate you\",\"code\":32}]}"
    read 63 bytes
    Conn close

Note that even though the above dump does not show it, I have verified that the request is sent via https.

Then executing the CURL command from the OAuth Tool works just fine:

    curl --get 'https://api.twitter.com/1.1/search/tweets.json' --data 'include_entities=true&q=from%3Aplindelauf' --header 'Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"' --verbose
    * About to connect() to api.twitter.com port 443 (#0)
    *   Trying 199.16.156.8...
    * connected
    * Connected to api.twitter.com (199.16.156.8) port 443 (#0)
    * SSLv3, TLS handshake, Client hello (1):
    * SSLv3, TLS handshake, Server hello (2):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Server finished (14):
    * SSLv3, TLS handshake, Client key exchange (16):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSL connection using RC4-SHA
    * Server certificate:
    *      subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
    *      start date: 2013-04-08 00:00:00 GMT
    *      expire date: 2013-12-31 23:59:59 GMT
    *      subjectAltName: api.twitter.com matched
    *      issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
    *      SSL certificate verify ok.
    > GET /1.1/search/tweets.json?include_entities=true&q=from%3Aplindelauf HTTP/1.1
    > User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
    > Host: api.twitter.com
    > Accept: */*
    > Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"
    >
    < HTTP/1.1 200 OK
    < cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
    < content-length: 14040
    < content-type: application/json;charset=utf-8
    < date: Sat, 08 Jun 2013 11:42:20 GMT
    < expires: Tue, 31 Mar 1981 05:00:00 GMT
    < last-modified: Sat, 08 Jun 2013 11:42:20 GMT
    < pragma: no-cache
    < server: tfe
    < set-cookie: lang=en
    < set-cookie: guest_id=v1%3A137069174010710877; Domain=.twitter.com; Path=/; Expires=Mon, 08-Jun-2015 11:42:20 UTC
    < status: 200 OK
    < strict-transport-security: max-age=631138519
    < x-access-level: read
    < x-frame-options: SAMEORIGIN
    < x-rate-limit-limit: 180
    < x-rate-limit-remaining: 179
    < x-rate-limit-reset: 1370692640
    < x-transaction: 4b4022555f454e2b
    < x-xss-protection: 1; mode=block
    ...[the whole response body with all found tweets]...

I've also tried to see if it makes a difference when I send the query parameters separately, but that does not seem to be the case, since this curl command works just fine as well:

    curl --get 'https://api.twitter.com/1.1/search/tweets.json?include_entities=true&q=from%3Aplindelauf' --header 'Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"' --verbose

This has been driving me crazy for many days now! Can someone please help?!

[Pascal Lindelauf]

I found the problem! I did not percent encode the query parameters in my HTTP request, like I did when creating the signature. Note that I was sending the query string q=from:plindelauf&include_entities=true. This needed to explicitly be include_entities=true&q=from%3Aplindelauf. Darn!

Op zondag 9 juni 2013 23:29:28 UTC+2 schreef Pascal Lindelauf het volgende: