app engine, shared IP and twitter api

[enes akar]

Hi;

I have just deployed an application to app engine which use twitter search api.

But there is a problem. Twitter blocks some of  my requests saying "You have been rate limited. Enhance your calm."

Of course I have asked about this to twitter men, waiting for their reply.

But I want to ask you, whether following scenerio is possible:

May app engine give the same IP to different applications?

If so another application which we share the same IP, may be spamming twitter api; and because of this spammer application I am blocked too.

Is this possible?

Have you seen similar problem, and is there a solution?

Note: It is not possible to exceed the rate limits of twitter, because there is no traffic in my site. 

Thanks in advance.

--
Enes Akar
http://www.linkedin.com/pub/enes-akar/7/835/3aa

[Nick Johnson (Google)]

Hi,

App Engine uses a shared pool of IPs for outgoing urlfetch requests. Unfortunately, as you observe, some services such as Twitter enforce per-ip ratelimiting.

In the case of Twitter, most of their APIs that support anonymous access also support authenticated access. You can submit authenticated requests instead, which are limited by your account, rather than by your IP.

-Nick Johnson

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[enes akar]

Thanks Nick I will try to find authenticated version of search api.

By the way I am really thankful about the immediate responses of this group.

Enes Akar
http://www.linkedin.com/pub/enes-akar/7/835/3aa

[Ryan]

Make sure you set your User Agent string to something unique as well.
You'll still get Rate Limited, but it should be a slightly higher
limit.

To answer your question, yes, you are being rate limited because of
other App Engine Twitter search API users. I wouldn't suggest using
App Engine and Twitter Search for a production project because of
this. Twitter does not have authenticated search API, and only
whitelists search users by IP address, and explicitly says in their
docs that they can't/won't whitelist App Engine apps:

http://apiwiki.twitter.com/Rate-limiting

"The Search API is only able to whitelist IP addresses, not user
accounts. This works in most situations but for cloud platforms like
Google App Engine, applications without a static IP addresses cannot
receive Search whitelisting."

Ryan

On Feb 9, 8:05 am, enes akar <enesa...@gmail.com> wrote:
> Thanks Nick I will try to find authenticated version of search api.
>
> By the way I am really thankful about the immediate responses of this group.
>
> On Tue, Feb 9, 2010 at 4:50 PM, Nick Johnson (Google) <
>
>
>
>
>
> nick.john...@google.com> wrote:
> > Hi,
>
> > App Engine uses a shared pool of IPs for outgoing urlfetch requests.
> > Unfortunately, as you observe, some services such as Twitter enforce per-ip
> > ratelimiting.
>
> > In the case of Twitter, most of their APIs that support anonymous access
> > also support authenticated access. You can submit authenticated requests
> > instead, which are limited by your account, rather than by your IP.
>
> > -Nick Johnson
>

> > On Tue, Feb 9, 2010 at 2:33 PM, enes akar <enesa...@gmail.com> wrote:
>
> >> Hi;
>
> >> I have just deployed an application to app engine which use twitter search
> >> api.
>
> >> But there is a problem. Twitter blocks some of  my requests saying "You
> >> have been rate limited. Enhance your calm."
>
> >> Of course I have asked about this to twitter men, waiting for their reply.
>
> >> But I want to ask you, whether following scenerio is possible:
> >> May app engine give the same IP to different applications?
> >> If so another application which we share the same IP, may be spamming
> >> twitter api; and because of this spammer application I am blocked too.
>
> >> Is this possible?
> >> Have you seen similar problem, and is there a solution?
>
> >> Note: It is not possible to exceed the rate limits of twitter, because
> >> there is no traffic in my site.
>
> >> Thanks in advance.
>
> >> --
> >> Enes Akar
> >>http://www.linkedin.com/pub/enes-akar/7/835/3aa
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-a...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > Nick Johnson, Developer Programs Engineer, App Engine
> > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> > 368047
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

[enes akar]

Thanks Ryan, yes I have an unique User Agent. 

Also people from twitter answered my question. As I guessed, all app engine applications have the same IP, we are blocked easily. 

They are offering to migrate my app to another platform. I wanted them to propose app engine's users a special solution. It is very depressing after all app engine specific work.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.