Important Notice on Incorrect API Endpoints for Search, REST, & OAuth (+ some general tips!)

239 views
Skip to first unread message

Taylor Singletary

unread,
May 27, 2010, 6:12:19 PM5/27/10
to twitter-development-talk
Hi Developers,

A few quick points before I go into more detail:

  * For the Search API, you should only be using http://search.twitter.com to execute search requests. Not http://api.twitter.com/1/search or any other variation.

  * Next week, we plan to remove the erroneous, unsupported endpoint at http://api.twitter.com/1/search

  * All REST requests to the API should use the fully qualified hostname and API version in URLs: http://api.twitter.com/1/* -- no other version is valid at this time. 

  * All OAuth negotiation steps should be over SSL and also at http://api.twitter.com -- but without a version.

  * Don't execute the same search query more often than every 20s and always use since_id on subsequent requests

  * Consider the streaming API if you're relying on search heavily to power your application

The Long-winded Approach

The only endpoint you should be using for search operations in the Twitter API today is http://search.twitter.com -- it doesn't require user authentication or OAuth -- simply identify yourself with a user-agent that is unique to your application.


Though we should have rejected traffic to that end point long ago to avoid confusion, it was never intended as a valid resource for search queries.

Next week, we'll be properly closing off this end point to avoid further confusion. If you have code today that uses the http://api.twitter.com or http:/twitter.com domains to execute search requests, be sure and update your code for the proper end point.

You can find the Search API documentation at http://bit.ly/twitter-search-api

Many users of the Search API are better served by using the Streaming API. If you use the search API to track the tweets of specific users, hashtags, or simple keyword queries, it is highly recommended that you use the Streaming API instead.

You shouldn't issue the same request to the search API more frequently than once every 20 seconds -- if you issue the same query more frequently than that, you're in danger of getting blacklisted. In addition, if you find yourself repeating the same query frequently, be sure and make use of the since_id parameter on subsequent requests -- without it, you put undue stress on the search infrastructure and will also be in danger of blacklisting.

While we're on the topic of using the proper endpoints, a general reminder about endpoints with the Twitter API:

All REST resource requests, with the exception of Search, should be pointed at http://api.twitter.com/1/* -- always use the api subdomain and specify the version number ("1"). No other version number will be accepted for the API at this time and your requests will fail if you provide a different string or integer.


Let us know if you have any concerns about the removal of the unofficial/unsupported search end point. We don't want to break people, but we also don't want you using unofficial API calls with substandard and unpredictable responses.

Thanks!

Taylor Singletary
Developer Advocate, Twitter
http://twitter.com/episod

Mageuzi

unread,
May 27, 2010, 9:55:22 PM5/27/10
to Twitter Development Talk
Are you all aware of this bug?
http://code.google.com/p/twitter-api/issues/detail?id=1154

We can't reliably use since_id for searches until this is fixed.


On May 27, 6:12 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:
> Hi Developers,
>
> A few quick points before I go into more detail:
>
>   * For the Search API, you should *only* be usinghttp://search.twitter.comtoexecute search requests.
> *Not*http://api.twitter.com/1/searchor any other variation.
>
>   * *Next week*, we plan to remove the erroneous, unsupported endpoint athttp://api.twitter.com/1/search
>
>   * All REST requests to the API should use the fully qualified hostname and
> API version in URLs:http://api.twitter.com/1/*-- no other version is valid
> at this time.
>
>   * All OAuth negotiation steps should be over SSL and also athttp://api.twitter.com-- but without a version.
>
>   * Don't execute the same search query more often than every 20s and always
> use since_id on subsequent requests
>
>   * Consider the streaming API if you're relying on search heavily to power
> your application
>
> *The Long-winded Approach*
> *
> *
> The only endpoint you should be using for search operations in the Twitter
> API today ishttp://search.twitter.com-- it doesn't require user
> authentication or OAuth -- simply identify yourself with a user-agent that
> is unique to your application.
>
> For those usinghttp://twitter.com/search,http://api.twitter.com/search, orhttp://api.twitter.com/1/search-- you've been doing it wrong :)
>
> Though we should have rejected traffic to that end point long ago to avoid
> confusion, it was never intended as a valid resource for search queries.
>
> Next week, we'll be properly closing off this end point to avoid further
> confusion. If you have code today that uses thehttp://api.twitter.comor
> http:/twitter.com domains to execute search requests, be sure and update
> your code for the proper end point.
>
> You can find the Search API documentation athttp://bit.ly/twitter-search-api
>
> Many users of the Search API are better served by using the Streaming API.
> If you use the search API to track the tweets of specific users, hashtags,
> or simple keyword queries, it is highly recommended that you use the
> Streaming API instead.
>
> You shouldn't issue the same request to the search API more frequently than
> once every 20 seconds -- if you issue the same query more frequently than
> that, you're in danger of getting blacklisted. In addition, if you find
> yourself repeating the same query frequently, be sure and make use of the
> since_id parameter on subsequent requests -- without it, you put undue
> stress on the search infrastructure and will also be in danger of
> blacklisting.
>
> While we're on the topic of using the proper endpoints, a general reminder
> about endpoints with the Twitter API:
>
> All REST resource requests, with the exception of Search, should be pointed
> athttp://api.twitter.com/1/*-- always use the api subdomain and specify
> the version number ("1"). No other version number will be accepted for the
> API at this time and your requests will fail if you provide a different
> string or integer.
>
> All OAuth negotiation steps should be over SSL athttps://api.twitter.com/oauth/*("https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/authorize", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authenticate")

janole

unread,
May 28, 2010, 1:41:29 AM5/28/10
to Twitter Development Talk
Hi Taylor,

I've filed a bug report about using since_id with search 6 months ago
after my client users complained about empty / no search results:

http://bit.ly/since_id

Is this bug fixed already? If not, how can we use since_id with the
Search API so that it is giving us some results and not just an empty
set?

Also, I'm using a proxy for my users from China and have therefore
used the user auth'ed version of the Search api (http://
api.twitter.com/1/search...)

If I'm switching to search.twitter.com, will I run into API limits for
my proxy then?

Thanks for any help
Ole @ mobileways.de / #Gravity Twitter Client for S60 Symbian

--
http://twitter.com/janole

On 28 Mai, 00:12, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:
> Hi Developers,
>
> A few quick points before I go into more detail:
>
>   * For the Search API, you should *only* be usinghttp://search.twitter.comtoexecute search requests.
> *Not*http://api.twitter.com/1/searchor any other variation.
>
>   * *Next week*, we plan to remove the erroneous, unsupported endpoint athttp://api.twitter.com/1/search
>
>   * All REST requests to the API should use the fully qualified hostname and
> API version in URLs:http://api.twitter.com/1/*-- no other version is valid
> at this time.
>
>   * All OAuth negotiation steps should be over SSL and also athttp://api.twitter.com-- but without a version.
>
>   * Don't execute the same search query more often than every 20s and always
> use since_id on subsequent requests
>
>   * Consider the streaming API if you're relying on search heavily to power
> your application
>
> *The Long-winded Approach*
> *
> *
> The only endpoint you should be using for search operations in the Twitter
> API today ishttp://search.twitter.com-- it doesn't require user
> authentication or OAuth -- simply identify yourself with a user-agent that
> is unique to your application.
>
> For those usinghttp://twitter.com/search,http://api.twitter.com/search, orhttp://api.twitter.com/1/search-- you've been doing it wrong :)
>
> Though we should have rejected traffic to that end point long ago to avoid
> confusion, it was never intended as a valid resource for search queries.
>
> Next week, we'll be properly closing off this end point to avoid further
> confusion. If you have code today that uses thehttp://api.twitter.comor
> http:/twitter.com domains to execute search requests, be sure and update
> your code for the proper end point.
>
> You can find the Search API documentation athttp://bit.ly/twitter-search-api
>
> Many users of the Search API are better served by using the Streaming API.
> If you use the search API to track the tweets of specific users, hashtags,
> or simple keyword queries, it is highly recommended that you use the
> Streaming API instead.
>
> You shouldn't issue the same request to the search API more frequently than
> once every 20 seconds -- if you issue the same query more frequently than
> that, you're in danger of getting blacklisted. In addition, if you find
> yourself repeating the same query frequently, be sure and make use of the
> since_id parameter on subsequent requests -- without it, you put undue
> stress on the search infrastructure and will also be in danger of
> blacklisting.
>
> While we're on the topic of using the proper endpoints, a general reminder
> about endpoints with the Twitter API:
>
> All REST resource requests, with the exception of Search, should be pointed
> athttp://api.twitter.com/1/*-- always use the api subdomain and specify
> the version number ("1"). No other version number will be accepted for the
> API at this time and your requests will fail if you provide a different
> string or integer.
>
> All OAuth negotiation steps should be over SSL athttps://api.twitter.com/oauth/*("https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/authorize", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authenticate")

Adam Green

unread,
May 28, 2010, 5:36:13 PM5/28/10
to twitter-deve...@googlegroups.com
Taylor:

Can we get clarification on the since_id issue? You have warned us
that not using since_id will get us blacklisted, and at the same time
since_id appears to still be broken according to others on this list.
Please advise.

What I do is request 100 responses per page, and then manually check
each tweet against the most recent tweet_id I've received for this
query. I stop asking for more pages when I reach a tweet that I
already have. Is this acceptable, or is since_id working now?

schammy

unread,
May 27, 2010, 8:24:10 PM5/27/10
to Twitter Development Talk
Twitter, your "since_id" feature has been broken since October 2009,
and it is STILL broken. And yet you warn us that not using it will
result in blacklisting? Your search API is unreliable when since_id is
used. Someone at Twitter mistakenly closed the bug in December but oh
yes, it still exists, it still gets plenty of comments. Fix that
before requiring it to be used. Unacceptable.

http://code.google.com/p/twitter-api/issues/detail?id=1154


On May 27, 3:12 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:
> Hi Developers,
>
> A few quick points before I go into more detail:
>
>   * For the Search API, you should *only* be usinghttp://search.twitter.comtoexecute search requests.
> *Not*http://api.twitter.com/1/searchor any other variation.
>
>   * *Next week*, we plan to remove the erroneous, unsupported endpoint athttp://api.twitter.com/1/search
>
>   * All REST requests to the API should use the fully qualified hostname and
> API version in URLs:http://api.twitter.com/1/*-- no other version is valid
> at this time.
>
>   * All OAuth negotiation steps should be over SSL and also athttp://api.twitter.com-- but without a version.
>
>   * Don't execute the same search query more often than every 20s and always
> use since_id on subsequent requests
>
>   * Consider the streaming API if you're relying on search heavily to power
> your application
>
> *The Long-winded Approach*
> *
> *
> The only endpoint you should be using for search operations in the Twitter
> API today ishttp://search.twitter.com-- it doesn't require user
> authentication or OAuth -- simply identify yourself with a user-agent that
> is unique to your application.
>
> For those usinghttp://twitter.com/search,http://api.twitter.com/search, orhttp://api.twitter.com/1/search-- you've been doing it wrong :)
>
> Though we should have rejected traffic to that end point long ago to avoid
> confusion, it was never intended as a valid resource for search queries.
>
> Next week, we'll be properly closing off this end point to avoid further
> confusion. If you have code today that uses thehttp://api.twitter.comor
> http:/twitter.com domains to execute search requests, be sure and update
> your code for the proper end point.
>
> You can find the Search API documentation athttp://bit.ly/twitter-search-api
>
> Many users of the Search API are better served by using the Streaming API.
> If you use the search API to track the tweets of specific users, hashtags,
> or simple keyword queries, it is highly recommended that you use the
> Streaming API instead.
>
> You shouldn't issue the same request to the search API more frequently than
> once every 20 seconds -- if you issue the same query more frequently than
> that, you're in danger of getting blacklisted. In addition, if you find
> yourself repeating the same query frequently, be sure and make use of the
> since_id parameter on subsequent requests -- without it, you put undue
> stress on the search infrastructure and will also be in danger of
> blacklisting.
>
> While we're on the topic of using the proper endpoints, a general reminder
> about endpoints with the Twitter API:
>
> All REST resource requests, with the exception of Search, should be pointed
> athttp://api.twitter.com/1/*-- always use the api subdomain and specify
> the version number ("1"). No other version number will be accepted for the
> API at this time and your requests will fail if you provide a different
> string or integer.
>
> All OAuth negotiation steps should be over SSL athttps://api.twitter.com/oauth/*("https://api.twitter.com/oauth/request_token", "https://api.twitter.com/oauth/authorize", "https://api.twitter.com/oauth/access_token", "https://api.twitter.com/oauth/authenticate")

themattharris

unread,
May 28, 2010, 7:33:42 PM5/28/10
to Twitter Development Talk
I just wanted to make everyone aware that this issue is open and being
tracked [1]. Any progress or developments will be posted on that
thread.

If you are experiencing a problem with since_id I encourage you to
read my comment [2].

Thank you,
Matt

1. http://code.google.com/p/twitter-api/issues/detail?id=1154
2. http://code.google.com/p/twitter-api/issues/detail?id=1154#c19

On May 27, 5:24 pm, schammy <scha...@gmail.com> wrote:
> Twitter, your "since_id" feature has been broken since October 2009,
> and it is STILL broken. And yet you warn us that not using it will
> result in blacklisting? Your search API is unreliable when since_id is
> used. Someone at Twitter mistakenly closed the bug in December but oh
> yes, it still exists, it still gets plenty of comments. Fix that
> before requiring it to be used. Unacceptable.
>
> http://code.google.com/p/twitter-api/issues/detail?id=1154
>
> On May 27, 3:12 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> wrote:
>
>
>
> > Hi Developers,
>
> > A few quick points before I go into more detail:
>
> >   * For the Search API, you should *only* be usinghttp://search.twitter.comtoexecutesearch requests.
> > *Not*http://api.twitter.com/1/searchorany other variation.
>
> >   * *Next week*, we plan to remove the erroneous, unsupported endpoint athttp://api.twitter.com/1/search
>
> >   * All REST requests to the API should use the fully qualified hostname and
> > API version in URLs:http://api.twitter.com/1/*--no other version is valid
> > at this time.
>
> >   * All OAuth negotiation steps should be over SSL and also athttp://api.twitter.com--but without a version.
>
> >   * Don't execute the same search query more often than every 20s and always
> > use since_id on subsequent requests
>
> >   * Consider the streaming API if you're relying on search heavily to power
> > your application
>
> > *The Long-winded Approach*
> > *
> > *
> > The only endpoint you should be using for search operations in the Twitter
> > API today ishttp://search.twitter.com--it doesn't require user
> > authentication or OAuth -- simply identify yourself with a user-agent that
> > is unique to your application.
>
> > For those usinghttp://twitter.com/search,http://api.twitter.com/search, orhttp://api.twitter.com/1/search--you've been doing it wrong :)
>
> > Though we should have rejected traffic to that end point long ago to avoid
> > confusion, it was never intended as a valid resource for search queries.
>
> > Next week, we'll be properly closing off this end point to avoid further
> > confusion. If you have code today that uses thehttp://api.twitter.comor
> > http:/twitter.com domains to execute search requests, be sure and update
> > your code for the proper end point.
>
> > You can find the Search API documentation athttp://bit.ly/twitter-search-api
>
> > Many users of the Search API are better served by using the Streaming API.
> > If you use the search API to track the tweets of specific users, hashtags,
> > or simple keyword queries, it is highly recommended that you use the
> > Streaming API instead.
>
> > You shouldn't issue the same request to the search API more frequently than
> > once every 20 seconds -- if you issue the same query more frequently than
> > that, you're in danger of getting blacklisted. In addition, if you find
> > yourself repeating the same query frequently, be sure and make use of the
> > since_id parameter on subsequent requests -- without it, you put undue
> > stress on the search infrastructure and will also be in danger of
> > blacklisting.
>
> > While we're on the topic of using the proper endpoints, a general reminder
> > about endpoints with the Twitter API:
>
> > All REST resource requests, with the exception of Search, should be pointed
> > athttp://api.twitter.com/1/*--always use the api subdomain and specify

Guillermo

unread,
Jun 4, 2010, 4:31:57 PM6/4/10
to Twitter Development Talk
I'm also having issues on not receiving new tweets when I use the
since_id value. Pretty much the same problem described in the above
comments.
Reply all
Reply to author
Forward
0 new messages