Search API Rate limiting - App Engine (again)
169 views
Skip to first unread message
Paul Kinlan
Sep 30, 2009, 6:45:27 PM9/30/09
to twitter-deve...@googlegroups.com
Hi Guys,
I have an app on the App engine using the search API and it is getting heavily rate limited again this past couple of days.
I know that we are on a shared set of IP addresses and someone else could be hammering the system, but it seems to run for weeks without seeing the rate limit being hit and then all of a sudden only about 60% of the searches I perform will be rate limited. This seems to occur every two months or so.
Has something changed recently?
Paul
steel
Oct 5, 2009, 2:59:23 AM10/5/09
to Twitter Development Talk
Hi. I have this problem too.
My application does two request per hour and it get "rate limit".
What is wrong? I think it is twitter's problems....
My application does two request per hour and it get "rate limit".
What is wrong? I think it is twitter's problems....
elkelk
Oct 5, 2009, 1:55:19 PM10/5/09
to Twitter Development Talk
Hi all,
I am having the same issue. I have tried setting a custom user-agent,
but this doesn't seem to affect the fact that twitter is limiting
based on I.P. address. I'm only making about 5 searches an hour and
80% of them are failing on app engine due to a 503 rate limit.
Twitter needs to determine a better way to let cloud clients access
their search API. It seems like they have really started blocking
search requests in the last week or so.
If anyone has any idea about how to better identify my app engine app
please let let me know.
I am having the same issue. I have tried setting a custom user-agent,
but this doesn't seem to affect the fact that twitter is limiting
based on I.P. address. I'm only making about 5 searches an hour and
80% of them are failing on app engine due to a 503 rate limit.
Twitter needs to determine a better way to let cloud clients access
their search API. It seems like they have really started blocking
search requests in the last week or so.
If anyone has any idea about how to better identify my app engine app
please let let me know.
elkelk
Oct 5, 2009, 12:58:09 PM10/5/09
to Twitter Development Talk
I'm noticing this problem as well. I'm making only a couple requests
per hour. I have tried setting the user-agent and the HTTP_REFERER
headers to a custom name, but Twitter doesn't seem to care.
per hour. I have tried setting the user-agent and the HTTP_REFERER
headers to a custom name, but Twitter doesn't seem to care.
Paul Kinlan
Oct 5, 2009, 4:53:43 PM10/5/09
to twitter-deve...@googlegroups.com
I am pretty sure there are custom headers on the App Engine that indicate the application that is sending the request.
2009/10/5 elkelk <danielsh...@gmail.com>
Martin Omander
Oct 6, 2009, 12:45:02 PM10/6/09
to Twitter Development Talk
Same here; my app runs on Google App Engine and 40% of the requests to
the Twitter Search API get the 503 error message indicating rate
limiting.
Is there anything we as app authors can do on our side to alleviate
the problem?
/Martin
On Oct 5, 1:53 pm, Paul Kinlan <paul.kin...@gmail.com> wrote:
> I am pretty sure there are custom headers on the App Engine that indicate
> the application that is sending the request.
>
Chad Etzel
Oct 6, 2009, 12:50:06 PM10/6/09
to twitter-deve...@googlegroups.com
Hi All,
GAE sites are problematic for the Twitter/Search API because the IPs
making outgoing requests are fluid and cannot as such be easily
allowed for access. Also, since most IPs are shared, other
applications on the same IPs making requests mean that fewer requests
per app get through.
One work around would be to spin up a server in EC2 or Rackspace Cloud
or something and use it as a proxy for your requests. That way you
have a dedicated IP that will have its full share of resources talking
with the Twitter servers.
HTH,
-Chad
Paul Kinlan
Oct 6, 2009, 2:13:24 PM10/6/09
to twitter-deve...@googlegroups.com
Hi Chad,
I am sorry but that doesn't even help in the slightest.
You are essentially saying that we shouldn't develop on the App
Engine, since would now have to also buy a proxy. Which is completely
unfeasible and defeats the purpose of why people are using the app
engine.
I understand that this might also be an App Engine issue - for
instance they could have reduced the number of IP addresses they pool
from to make external requests.
This is a very noticeable change in rate limiting in the last few
weeks. For instance I could run roughly 2 searches a second, then all
of a sudden I would be lucky to run 2 every 15 seconds. User-Agent
strings were supposed to allievate this issue. There are more than
enough pieces of meta data on an App Engine request that Identify the
exact application that is making the requests - I guess it is too much
effort to take these into account.
I am in the fortunate position that allowed me to set up a nginx proxy
quickly, but I suspect a lot of other people couldn't do that.
I hope something can be sorted for the large number of GAE based
Twitter apps.
Paul Kinlan
Aid
Oct 7, 2009, 1:23:43 PM10/7/09
to Twitter Development Talk
I am also facing this issue. I'm only making a couple of requests
from GAE (about 3-4) and none of them are getting through, I keep
getting the following using Twitter4J....
Twitter Exception while retrieving status
twitter4j.TwitterException: 400:The request was invalid. An
accompanying error message will explain why. This is the status code
will be returned during rate limiting.
<?xml version="1.0" encoding="UTF-8"?>
<hash>
<request>/statuses/show/200000000.xml</request>
<error>Rate limit exceeded. Clients may not make more than 150
requests per hour.</error>
</hash>
On Oct 6, 7:13 pm, Paul Kinlan <paul.kin...@gmail.com> wrote:
> Hi Chad,
>
> I am sorry but that doesn't even help in the slightest.
>
> You are essentially saying that we shouldn't develop on the App
> Engine, since would now have to also buy a proxy. Which is completely
> unfeasible and defeats the purpose of why people are using the app
> engine.
>
> I understand that this might also be an App Engine issue - for
> instance they could have reduced the number of IP addresses they pool
> from to make external requests.
>
> This is a very noticeable change inratelimiting in the last few
> >>>>> My application does two request per hour and it get "ratelimit".
> >>>>>> searches
> >>>>>> I perform will beratelimited. This seems to occur every two
from GAE (about 3-4) and none of them are getting through, I keep
getting the following using Twitter4J....
Twitter Exception while retrieving status
twitter4j.TwitterException: 400:The request was invalid. An
accompanying error message will explain why. This is the status code
will be returned during rate limiting.
<?xml version="1.0" encoding="UTF-8"?>
<hash>
<request>/statuses/show/200000000.xml</request>
<error>Rate limit exceeded. Clients may not make more than 150
requests per hour.</error>
</hash>
On Oct 6, 7:13 pm, Paul Kinlan <paul.kin...@gmail.com> wrote:
> Hi Chad,
>
> I am sorry but that doesn't even help in the slightest.
>
> You are essentially saying that we shouldn't develop on the App
> Engine, since would now have to also buy a proxy. Which is completely
> unfeasible and defeats the purpose of why people are using the app
> engine.
>
> I understand that this might also be an App Engine issue - for
> instance they could have reduced the number of IP addresses they pool
> from to make external requests.
>
> weeks. For instance I could run roughly 2 searches a second, then all
> of a sudden I would be lucky to run 2 every 15 seconds. User-Agent
> strings were supposed to allievate this issue. There are more than
> enough pieces of meta data on an App Engine request that Identify the
> exact application that is making the requests - I guess it is too much
> effort to take these into account.
>
> I am in the fortunate position that allowed me to set up a nginx proxy
> quickly, but I suspect a lot of other people couldn't do that.
>
> I hope something can be sorted for the large number of GAE based
> Twitter apps.
>
> Paul Kinlan
>
> of a sudden I would be lucky to run 2 every 15 seconds. User-Agent
> strings were supposed to allievate this issue. There are more than
> enough pieces of meta data on an App Engine request that Identify the
> exact application that is making the requests - I guess it is too much
> effort to take these into account.
>
> I am in the fortunate position that allowed me to set up a nginx proxy
> quickly, but I suspect a lot of other people couldn't do that.
>
> I hope something can be sorted for the large number of GAE based
> Twitter apps.
>
> Paul Kinlan
>
> >>>>> What is wrong? I think it is twitter's problems....
>
> >>>>> On 1 окт, 01:45, Paul Kinlan <paul.kin...@gmail.com> wrote:
>
> >>>>>> Hi Guys,
> >>>>>> I have an app on the App engine using the search API and it is
> >>>>>> getting
> >>>>>> heavilyratelimited again this past couple of days.
>
> >>>>> On 1 окт, 01:45, Paul Kinlan <paul.kin...@gmail.com> wrote:
>
> >>>>>> Hi Guys,
> >>>>>> I have an app on the App engine using the search API and it is
> >>>>>> getting
>
> >>>>>> I know that we are on a shared set of IP addresses and someone
> >>>>>> else
> >>>> could be
> >>>>>> hammering the system, but it seems to run for weeks without
> >>>>>> seeing the
> >>>>rate
> >>>>>>limitbeing hit and then all of a sudden only about 60% of the
> >>>>>> I know that we are on a shared set of IP addresses and someone
> >>>>>> else
> >>>> could be
> >>>>>> hammering the system, but it seems to run for weeks without
> >>>>>> seeing the
> >>>>rate
> >>>>>> searches
> >>>>>> I perform will beratelimited. This seems to occur every two
Josh Roesslein
Oct 7, 2009, 1:34:12 PM10/7/09
to twitter-deve...@googlegroups.com
Twitter should really in this case either white list all GAE IPs (I'm
sure an email to Google could get all IPs they use) or allow charging
API requests to an authenticated account rather than by IP (much like
the REST API does). This way each GAE application would just set up a
twitter account and each gets its own 150 request per hour.
sure an email to Google could get all IPs they use) or allow charging
API requests to an authenticated account rather than by IP (much like
the REST API does). This way each GAE application would just set up a
twitter account and each gets its own 150 request per hour.
Josh
steel
Oct 8, 2009, 4:03:53 AM10/8/09
to Twitter Development Talk
I have solved a problem like that:
While I receive an error 503 - my application continue knocking to
twitter with query.
Everything works ;)
While I receive an error 503 - my application continue knocking to
twitter with query.
Everything works ;)
Akshar
Oct 8, 2009, 5:09:45 PM10/8/09
to Twitter Development Talk
Any other solutions available for app engine folks stuck out here?
Please help!
I'm noticing this exact problem as well. I'm making only a few
requests per hour. I have tried setting the user-agent but it did not
help.
Akshar
Please help!
I'm noticing this exact problem as well. I'm making only a few
requests per hour. I have tried setting the user-agent but it did not
help.
Akshar
Akshar
Oct 8, 2009, 7:26:38 PM10/8/09
to Twitter Development Talk
http://apiwiki.twitter.com/Rate-limiting states that "for cloud
platforms like Google App Engine, applications without a static IP
addresses cannot receive Search whitelisting."
Does that mean there is no way to avoid getting HTTP 503 response
codes to search requests from app engine?
On Oct 8, 2:09 pm, Akshar <akshar.d...@gmail.com> wrote:
> Any other solutions available for app engine folks stuck out here?
> Please help!
>
> I'm noticing this exact problem as well. I'm making only a few
> requests per hour. I have tried setting the user-agent but it did not
> help.
>
> Akshar
>
> On Oct 6, 9:50 am, Chad Etzel <c...@twitter.com> wrote:
>
>
>
> > Hi All,
>
> > GAE sites are problematic for the Twitter/SearchAPIbecause the IPs
> > >> >searchrequests in the last week or so.
platforms like Google App Engine, applications without a static IP
addresses cannot receive Search whitelisting."
Does that mean there is no way to avoid getting HTTP 503 response
codes to search requests from app engine?
On Oct 8, 2:09 pm, Akshar <akshar.d...@gmail.com> wrote:
> Any other solutions available for app engine folks stuck out here?
> Please help!
>
> I'm noticing this exact problem as well. I'm making only a few
> requests per hour. I have tried setting the user-agent but it did not
> help.
>
> Akshar
>
> On Oct 6, 9:50 am, Chad Etzel <c...@twitter.com> wrote:
>
>
>
> > Hi All,
>
> > making outgoing requests are fluid and cannot as such be easily
> > allowed for access. Also, since most IPs are shared, other
> > applications on the same IPs making requests mean that fewer requests
> > per app get through.
>
> > One work around would be to spin up a server in EC2 or Rackspace Cloud
> > or something and use it as a proxy for your requests. That way you
> > have a dedicated IP that will have its full share of resources talking
> > with the Twitter servers.
>
> > HTH,
> > -Chad
>
> > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander <moman...@google.com> wrote:
>
> > > Same here; my app runs on Google App Engine and 40% of the requests to
> > > the TwitterSearchAPIget the 503 error message indicating rate
> > allowed for access. Also, since most IPs are shared, other
> > applications on the same IPs making requests mean that fewer requests
> > per app get through.
>
> > One work around would be to spin up a server in EC2 or Rackspace Cloud
> > or something and use it as a proxy for your requests. That way you
> > have a dedicated IP that will have its full share of resources talking
> > with the Twitter servers.
>
> > HTH,
> > -Chad
>
> > On Tue, Oct 6, 2009 at 12:45 PM, Martin Omander <moman...@google.com> wrote:
>
> > > Same here; my app runs on Google App Engine and 40% of the requests to
> > > limiting.
>
> > > Is there anything we as app authors can do on our side to alleviate
> > > the problem?
>
> > > /Martin
>
> > > On Oct 5, 1:53 pm, Paul Kinlan <paul.kin...@gmail.com> wrote:
> > >> I am pretty sure there are custom headers on the App Engine that indicate
> > >> the application that is sending the request.
>
> > >> 2009/10/5 elkelk <danielshaneup...@gmail.com>
>
> > >> > Hi all,
>
> > >> > I am having the same issue. I have tried setting a custom user-agent,
> > >> > but this doesn't seem to affect the fact that twitter is limiting
> > >> > based on I.P. address. I'm only making about 5 searches an hour and
> > >> > 80% of them are failing on app engine due to a 503 rate limit.
> > >> > Twitter needs to determine a better way to let cloud clients access
> > >> > theirsearchAPI. It seems like they have really started blocking
>
> > > Is there anything we as app authors can do on our side to alleviate
> > > the problem?
>
> > > /Martin
>
> > > On Oct 5, 1:53 pm, Paul Kinlan <paul.kin...@gmail.com> wrote:
> > >> I am pretty sure there are custom headers on the App Engine that indicate
> > >> the application that is sending the request.
>
> > >> 2009/10/5 elkelk <danielshaneup...@gmail.com>
>
> > >> > Hi all,
>
> > >> > I am having the same issue. I have tried setting a custom user-agent,
> > >> > but this doesn't seem to affect the fact that twitter is limiting
> > >> > based on I.P. address. I'm only making about 5 searches an hour and
> > >> > 80% of them are failing on app engine due to a 503 rate limit.
> > >> > Twitter needs to determine a better way to let cloud clients access
> > >> >searchrequests in the last week or so.
>
> > >> > If anyone has any idea about how to better identify my app engine app
> > >> > please let let me know.
>
> > >> > On Oct 5, 2:59 am, steel <steel...@gmail.com> wrote:
> > >> > > Hi. I have this problem too.
> > >> > > My application does two request per hour and it get "rate limit".
> > >> > > What is wrong? I think it is twitter's problems....
>
> > >> > > On 1 окт, 01:45, Paul Kinlan <paul.kin...@gmail.com> wrote:
>
> > >> > > > Hi Guys,
> > >> > > > I have an app on the App engine using thesearchAPIand it is getting
> > >> > If anyone has any idea about how to better identify my app engine app
> > >> > please let let me know.
>
> > >> > On Oct 5, 2:59 am, steel <steel...@gmail.com> wrote:
> > >> > > Hi. I have this problem too.
> > >> > > My application does two request per hour and it get "rate limit".
> > >> > > What is wrong? I think it is twitter's problems....
>
> > >> > > On 1 окт, 01:45, Paul Kinlan <paul.kin...@gmail.com> wrote:
>
> > >> > > > Hi Guys,
Abraham Williams
Oct 8, 2009, 9:07:54 PM10/8/09
to twitter-deve...@googlegroups.com
Pretty much. You have limited options:
--
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
http://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite
This email is: [ ] blogable [x] ask first [ ] private.
Sent from Madison, WI, United States
1) Run your Search API requests through a proxy where you will have exclusive access to the IP.
2) Wait for V2 of the Twitter API where the REST and Search APIs get combined so you can have authenticated search queries.
3) Hope Twitter slaps some duct tape on the issue and rolls out a whitelisting method for the Search API that includes passkeys in your user agent or some such thing.
4) Develop on non cloud base infrastructure.
5) Something else.
Abraham
2009/10/8 Akshar <aksha...@gmail.com>
--
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
http://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite
This email is: [ ] blogable [x] ask first [ ] private.
Sent from Madison, WI, United States
Akshar
Oct 9, 2009, 7:11:32 PM10/9/09
to Twitter Development Talk
Thanks Abraham.
Any pointers on how to setup a proxy on amazon ec2 for GAE?
On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote:
> Pretty much. You have limited options:
> 1) Run your Search API requests through a proxy where you will have
> exclusive access to the IP.
> 2) Wait for V2 of the Twitter API where the REST and Search APIs get
> combined so you can have authenticated search queries.
> 3) Hope Twitter slaps some duct tape on the issue and rolls out a
> whitelisting method for the Search API that includes passkeys in your user
> agent or some such thing.
> 4) Develop on non cloud base infrastructure.
> 5) Something else.
>
> Abraham
>
> 2009/10/8 Akshar <akshar.d...@gmail.com>
>
>
>
>
>
> >http://apiwiki.twitter.com/Rate-limitingstates that "for cloud
> Hacker |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/10/03/win-google-wave-invite
Any pointers on how to setup a proxy on amazon ec2 for GAE?
On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote:
> Pretty much. You have limited options:
> 1) Run your Search API requests through a proxy where you will have
> exclusive access to the IP.
> 2) Wait for V2 of the Twitter API where the REST and Search APIs get
> combined so you can have authenticated search queries.
> 3) Hope Twitter slaps some duct tape on the issue and rolls out a
> whitelisting method for the Search API that includes passkeys in your user
> agent or some such thing.
> 4) Develop on non cloud base infrastructure.
> 5) Something else.
>
> Abraham
>
>
>
>
>
>
> >http://apiwiki.twitter.com/Rate-limitingstates that "for cloud
elkelk
Oct 11, 2009, 12:05:38 PM10/11/09
to Twitter Development Talk
I would recommend just using a physical server and uploading a simple
php proxy script. If you have existing webspace, it will save you the
trouble of setting up an complete ec2 build just to run a proxy
script.
On Oct 9, 7:11 pm, Akshar <akshar.d...@gmail.com> wrote:
> Thanks Abraham.
>
> Any pointers on how to setup a proxy on amazon ec2 for GAE?
>
> On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote:
>
> > Pretty much. You have limited options:
> > 1) Run your Search API requests through a proxy where you will have
> > exclusive access to the IP.
> > 2) Wait for V2 of the Twitter API where the REST and Search APIs get
> > combined so you can have authenticated search queries.
> > 3) Hope Twitter slaps some duct tape on the issue and rolls out a
> > whitelisting method for the Search API that includes passkeys in your user
> > agent or some such thing.
> > 4) Develop on non cloud base infrastructure.
> > 5) Something else.
>
> > Abraham
>
> > 2009/10/8 Akshar <akshar.d...@gmail.com>
>
> > >http://apiwiki.twitter.com/Rate-limitingstatesthat "for cloud
> > Hacker |http://abrah.am|http://twitter.com/abrahamhttp://web608.org/geeks/abraham/blogs/2009/...
php proxy script. If you have existing webspace, it will save you the
trouble of setting up an complete ec2 build just to run a proxy
script.
On Oct 9, 7:11 pm, Akshar <akshar.d...@gmail.com> wrote:
> Thanks Abraham.
>
> Any pointers on how to setup a proxy on amazon ec2 for GAE?
>
> On Oct 8, 6:07 pm, Abraham Williams <4bra...@gmail.com> wrote:
>
> > Pretty much. You have limited options:
> > 1) Run your Search API requests through a proxy where you will have
> > exclusive access to the IP.
> > 2) Wait for V2 of the Twitter API where the REST and Search APIs get
> > combined so you can have authenticated search queries.
> > 3) Hope Twitter slaps some duct tape on the issue and rolls out a
> > whitelisting method for the Search API that includes passkeys in your user
> > agent or some such thing.
> > 4) Develop on non cloud base infrastructure.
> > 5) Something else.
>
> > Abraham
>
> > 2009/10/8 Akshar <akshar.d...@gmail.com>
>
vivekpuri
Oct 18, 2009, 8:09:11 AM10/18/09
to Twitter Development Talk
Will someone from Twitter please respond if there is an ETA to resolve
this issue. Work arounds can never be really as effective as the real
deal.
this issue. Work arounds can never be really as effective as the real
deal.
Chad Etzel
Oct 18, 2009, 3:12:17 PM10/18/09
to twitter-deve...@googlegroups.com
Sorry, I thought it was clear from the previous email. There is no ETA
because it's not going to be resolved. GAE does not use an IP
infrastructure that is amicable to our rate-limiting logic, so if you
want to integrate IP rate-limited calls into your web-based
applications, you will need to either use the workaround stated
earlier or use a hosting service that will let you use a static IP.
-Chad
vivekpuri
Oct 18, 2009, 9:39:56 PM10/18/09
to Twitter Development Talk
Chad,
Sorry for not being clear. I was thinking about Abraham William's
suggestion above where Twitter Search API works with authenticated
sessions+rate limiting, instead of IP based rate filtering. Just so
you know, AppEngine has 30 second timeout on request to all AppEngine
urls, and 10 second timeout on each individual HTTP request made
within an AppEngine request. In case you are making multiple HTTP
requests to Twitter within each individual AppEngine request, all the
communication microseconds, from AppEngine to Proxy and Proxy to
Twitter and then Twitter to Proxy and Proxy to AppEngine, quickly
addup leading to timeouts. Personally i have tried quite a few
scenarios to catch all the data i can, but from my experience, i can
catch only 30%(sometimes better, sometimes almost nothing) of what i
want, and rest just ends up with 503 and eventually since_id/max_id
getting too old to get response from the Twitter Search API. So, right
now Twitter is putting it's resources to offer a very robust Search
API, but we as developers cannot use it effectively just 'cause of the
way the hits are counted. Not to mention we are also investing funds
to keep our apps running. Hope you understand our position.
Thanks
Sorry for not being clear. I was thinking about Abraham William's
suggestion above where Twitter Search API works with authenticated
sessions+rate limiting, instead of IP based rate filtering. Just so
you know, AppEngine has 30 second timeout on request to all AppEngine
urls, and 10 second timeout on each individual HTTP request made
within an AppEngine request. In case you are making multiple HTTP
requests to Twitter within each individual AppEngine request, all the
communication microseconds, from AppEngine to Proxy and Proxy to
Twitter and then Twitter to Proxy and Proxy to AppEngine, quickly
addup leading to timeouts. Personally i have tried quite a few
scenarios to catch all the data i can, but from my experience, i can
catch only 30%(sometimes better, sometimes almost nothing) of what i
want, and rest just ends up with 503 and eventually since_id/max_id
getting too old to get response from the Twitter Search API. So, right
now Twitter is putting it's resources to offer a very robust Search
API, but we as developers cannot use it effectively just 'cause of the
way the hits are counted. Not to mention we are also investing funds
to keep our apps running. Hope you understand our position.
Thanks
Reply all
Reply to author
Forward
0 new messages