Not able to connect to twitter API from Google Appengine

[nischalshetty]

My app http://www.justunfollow.com is just not able to connect to
twitter from Google Appengine. It's most probably an app engine issue
(none of the app engine apps seem to be able to connect to twitter),
but nevertheless I'm writing here to see if it so happened that
twitter has blocked access to app engine.

The error says : "Could not fecth URL" for http calls made to twitter
from google Appengine

-Nischal

[Livid]

I'm getting the same error for my community (with a built-in Twitter
OAuth client) running on GAE: http://v2ex.appspot.com

Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/webapp/__init__.py", line 511, in __call__
handler.get(*groups)
File "/base/data/home/apps/v2ex/1.343564127440067233/t.py", line
157, in get
statuses = twitter.GetHomeTimeline(count = 100)
File "/base/data/home/apps/v2ex/1.343564127440067233/twitter/
twitter.py", line 1451, in GetHomeTimeline
json = self._FetchUrl(url, parameters=parameters)
File "/base/data/home/apps/v2ex/1.343564127440067233/twitter/
oauthtwitter.py", line 101, in _FetchUrl
url_data = opener.open(url).read()
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 381, in open
response = self._open(req, data)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 399, in _open
'_open', req)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 360, in _call_chain
result = func(*args)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 1115, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 1080, in do_open
r = h.getresponse()
File "/base/python_runtime/python_dist/lib/python2.5/httplib.py",
line 197, in getresponse
self._allow_truncated, self._follow_redirects)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/urlfetch.py", line 241, in fetch
return rpc.get_result()
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/apiproxy_stub_map.py", line 501, in get_result
return self.__get_result_hook(self)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/urlfetch.py", line 325, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 2

It was fine several days ago.

On Jul 23, 2:15 pm, nischalshetty <nischalshett...@gmail.com> wrote:
> My apphttp://www.justunfollow.comis just not able to connect to

[Taylor Singletary]

Hi folks on Google App Engine experiencing difficulties,

We're looking into it!

Taylor

[nischalshetty]

@Taylor

Ah! You're my hero! I've been frantically trying to get in touch with
anyone and everyone over twitter and google app engine. The App engine
folks are yet to read the long thread I've started in their forum.

I hope if the issue is on your end you find a fix soon. It's been well
over 15-20 hours since my app has been unusable, it hurts!

-Nischal

On Jul 23, 7:06 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> >> My apphttp://www.justunfollow.comisjust not able to connect to

[Taylor Singletary]

Hey all,

We're still looking into this. To help us eliminate some possibile
issues, can someone who's working behind the Google App Engine IP
addresses attempt to connect to both
http://api.twitter.com/oauth/request_token and
https://api.twitter.com/oauth/request_token and let us know if you're
seeing a difference between the two? (I'm trying to rule out that the
SSL wildcard certificate is to blame or not).

Thanks,
Taylor

[nischalshetty]

@Taylor

The problem is even with the simple search request. So basically its
for all API calls to twitter.

-Nischal

On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> Hey all,
>
> We're still looking into this. To help us eliminate some possibile
> issues, can someone who's working behind the Google App Engine IP

> addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/oauth/request_tokenand let us know if you're

> >> >> My apphttp://www.justunfollow.comisjustnot able to connect to

[nischalshetty]

@Taylor

Just checked. I can make calls to google.com through the appengine. So
I guess its only twitter calls that are failing which means the issue
is on twitter's side.

I guess the appengine IPs are being blacklisted? or blocked by
twitter?

-Nischal

On Jul 23, 9:32 pm, nischalshetty <nischalshett...@gmail.com> wrote:
> @Taylor
>
> The problem is even with the simple search request. So basically its
> for all API calls to twitter.
>
> -Nischal
>
> On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> wrote:
>
>
>
> > Hey all,
>
> > We're still looking into this. To help us eliminate some possibile
> > issues, can someone who's working behind the Google App Engine IP

> > addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/...let us know if you're

> > >> >> My apphttp://www.justunfollow.comisjustnotable to connect to

[Greg Jones]

Hi Taylor,

It doesn't connect to either http or https. Happy to help testing
anything else...app's not live yet, but was a bit of a scare this
morning!

cheers,

Greg

On Jul 23, 5:32 pm, nischalshetty <nischalshett...@gmail.com> wrote:
> @Taylor
>
> The problem is even with the simple search request. So basically its
> for all API calls to twitter.
>
> -Nischal
>
> On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> wrote:
>
>
>
> > Hey all,
>
> > We're still looking into this. To help us eliminate some possibile
> > issues, can someone who's working behind the Google App Engine IP

> > addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/...let us know if you're

> > >> >> My apphttp://www.justunfollow.comisjustnotable to connect to

[nischalshetty]

Oh my GOD! I can see it working! Yippe....

Thank you so much. A post or update on what caused the issue would be
welcome!

-Nischal

On Jul 23, 9:51 pm, Greg Jones <psycle....@gmail.com> wrote:
> Hi Taylor,
>
> It doesn't connect to either http or https. Happy to help testing
> anything else...app's not live yet, but was a bit of a scare this
> morning!
>
> cheers,
>
> Greg
>
> On Jul 23, 5:32 pm, nischalshetty <nischalshett...@gmail.com> wrote:
>
>
>
> > @Taylor
>
> > The problem is even with the simple search request. So basically its
> > for all API calls to twitter.
>
> > -Nischal
>
> > On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> > wrote:
>
> > > Hey all,
>
> > > We're still looking into this. To help us eliminate some possibile
> > > issues, can someone who's working behind the Google App Engine IP

> > > addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/...us know if you're

> > > >> >> My apphttp://www.justunfollow.comisjustnotableto connect to

[nischalshetty]

Alrite, I can see intermittent errors. So all's not well yet...

-Nischal

On Jul 23, 11:35 pm, nischalshetty <nischalshett...@gmail.com> wrote:
> Oh my GOD! I can see it working! Yippe....
>
> Thank you so much. A post or update on what caused the issue would be
> welcome!
>
> -Nischal
>
> On Jul 23, 9:51 pm, Greg Jones <psycle....@gmail.com> wrote:
>
>
>
> > Hi Taylor,
>
> > It doesn't connect to either http or https. Happy to help testing
> > anything else...app's not live yet, but was a bit of a scare this
> > morning!
>
> > cheers,
>
> > Greg
>
> > On Jul 23, 5:32 pm, nischalshetty <nischalshett...@gmail.com> wrote:
>
> > > @Taylor
>
> > > The problem is even with the simple search request. So basically its
> > > for all API calls to twitter.
>
> > > -Nischal
>
> > > On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> > > wrote:
>
> > > > Hey all,
>
> > > > We're still looking into this. To help us eliminate some possibile
> > > > issues, can someone who's working behind the Google App Engine IP

> > > > addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/...know if you're

> > > > >> >> My apphttp://www.justunfollow.comisjustnotabletoconnect to

[John Adams]

Please post or forward your app's IP range so we can investigate. Thanks.

-j

[nischalshetty]

@John

It's hosted on the Google Appengine. I guess you guys are already on
it to fix the issue.

-Nischal

On Jul 23, 11:55 pm, John Adams <j...@twitter.com> wrote:
> Please post or forward your app's IP range so we can investigate. Thanks.
> -j
>
> On Fri, Jul 23, 2010 at 11:50 AM, nischalshetty

> <nischalshett...@gmail.com>wrote:

[Taylor Singletary]

Hi Everyone,

Here are the details on the issues with Google App Engine.

Twitter blocked a portion of the GAE network because an unknown user
set up a large proxy farm, forwarding large amounts of traffic to
twitter.com. This was probably an attempt to avoid our rate limits,
which is against the Twitter terms of service, among other privacy and
security issues.

We recognize that those in shared hosting environments like Google App
Engine are often held hostage by the actions of their peers and will
continue to investigate ways that we can deal with issues like this
without necessarily cutting off all traffic from a shared hosting
services, but those operating under such circumstances should be aware
that this kind of blacklisting will occur from time to time.

If you continue to experience issues with your Google App Engine
application, please reply to this thread with a link to your
application, and, if possible, the IP address from which your remote
requests are originating.

Thanks,
Taylor

[planetjones]

Still not working for me. That's been all day. Mine is GAE too.

[planetjones]

I'm still getting this and have been all day. Please can someone
advise what's happening and why Twitter seemingly doesn't want to talk
to GAE?

On Jul 23, 5:51 pm, Greg Jones <psycle....@gmail.com> wrote:
> Hi Taylor,
>
> It doesn't connect to either http or https. Happy to help testing
> anything else...app's not live yet, but was a bit of a scare this
> morning!
>
> cheers,
>
> Greg
>
> On Jul 23, 5:32 pm, nischalshetty <nischalshett...@gmail.com> wrote:
>
>
>
> > @Taylor
>
> > The problem is even with the simple search request. So basically its
> > for all API calls to twitter.
>
> > -Nischal
>
> > On Jul 23, 8:56 pm, Taylor Singletary <taylorsinglet...@twitter.com>
> > wrote:
>
> > > Hey all,
>
> > > We're still looking into this. To help us eliminate some possibile
> > > issues, can someone who's working behind the Google App Engine IP

> > > addresses attempt to connect to bothhttp://api.twitter.com/oauth/request_tokenandhttps://api.twitter.com/...us know if you're

> > > >> >> My apphttp://www.justunfollow.comisjustnotableto connect to

[nischalshetty]

I'm still facing issues. Please help me - http://www.justunfollow.com

I have no idea about the IP addresses though.

-Nischal

On Jul 24, 12:26 am, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> Hi Everyone,
>
> Here are the details on the issues with Google App Engine.
>
> Twitter blocked a portion of the GAE network because an unknown user
> set up a large proxy farm, forwarding large amounts of traffic to
> twitter.com. This was probably an attempt to avoid our rate limits,
> which is against the Twitter terms of service, among other privacy and
> security issues.
>
> We recognize that those in shared hosting environments like Google App
> Engine are often held hostage by the actions of their peers and will
> continue to investigate ways that we can deal with issues like this
> without necessarily cutting off all traffic from a shared hosting
> services, but those operating under such circumstances should be aware
> that this kind of blacklisting will occur from time to time.
>
> If you continue to experience issues with your Google App Engine
> application, please reply to this thread with a link to your
> application, and, if possible, the IP address from which your remote
> requests are originating.
>
> Thanks,
> Taylor
>

[Livid]

Hi Taylor,

I think it's almost impossible to determine the originated IP, because
every time Google uses a different server to request, and in theory
only the requested ends (Twitter) can see the originated IPs.

My app is http://v2ex.appspot.com/ , now is still experiencing the
problem from time to time.

I guess the problematic requests are made via basic auth, or you can
easily deactivate its consumer key, right?

Now I'm long for August 16.

Thanks,

@livid

On Jul 24, 3:26 am, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> Hi Everyone,
>
> Here are the details on the issues with Google App Engine.
>
> Twitter blocked a portion of the GAE network because an unknown user
> set up a large proxy farm, forwarding large amounts of traffic to
> twitter.com. This was probably an attempt to avoid our rate limits,
> which is against the Twitter terms of service, among other privacy and
> security issues.
>
> We recognize that those in shared hosting environments like Google App
> Engine are often held hostage by the actions of their peers and will
> continue to investigate ways that we can deal with issues like this
> without necessarily cutting off all traffic from a shared hosting
> services, but those operating under such circumstances should be aware
> that this kind of blacklisting will occur from time to time.
>
> If you continue to experience issues with your Google App Engine
> application, please reply to this thread with a link to your
> application, and, if possible, the IP address from which your remote
> requests are originating.
>
> Thanks,
> Taylor
>
> On Fri, Jul 23, 2010 at 12:08 PM, nischalshetty
>

[Marco Gomes]

I am with same problema, DownloadError: ApplicationError: 2

On both my GAE apps:
http://apoiomaisfeliz.appspot.com/
http://apoio.minhamarina.org.br/

Can Twitter API block the proxy farm without stopping our permitted
apps?

On Jul 23, 4:26 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

[John Adams]

About thirty minutes ago we lifted all of the blocks on Google App engine IPs; You should no longer have issues connecting from GAE to us.

-j

[nischalshetty]

@John

Thank you very much for lifting the ban. It's working now.

Our apps depend completely on Twitter. I have been working and putting
in a lot of time and effort (and money on app engine for the
resources) to manage my app http://justunfollow.com

My app gets a good number of users and I have been thinking of making
some money off it. But, if something like this happens, all my efforts
would take a beating!

Can't you guys be able to determine if the app making the request is
legit or not. I mean, we all use our consumer keys, at least the apps
using their consumer key must be allowed to make requests. IP should
come into play only when the consumer key is missing.

Can this be done?

-Nischal

[Aurélio Carlos]

My application www.ctwittlike.net (http://ctwittlike.appspot.com)
continues without access to Twitter's API.

Thank you.

Aurélio Carlos
lelinh...@gmail.com


On Jul 23, 4:26 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

> Hi Everyone,
>
> Here are the details on the issues with Google App Engine.
>
> Twitter blocked a portion of the GAE network because an unknown user
> set up a large proxy farm, forwarding large amounts of traffic to
> twitter.com. This was probably an attempt to avoid our rate limits,
> which is against the Twitter terms of service, among other privacy and
> security issues.
>
> We recognize that those in shared hosting environments like Google App
> Engine are often held hostage by the actions of their peers and will
> continue to investigate ways that we can deal with issues like this
> without necessarily cutting off all traffic from a shared hosting
> services, but those operating under such circumstances should be aware
> that this kind of blacklisting will occur from time to time.
>
> If you continue to experience issues with your Google App Engine
> application, please reply to this thread with a link to your
> application, and, if possible, the IP address from which your remote
> requests are originating.
>
> Thanks,
> Taylor
>

[Marco Gomes]

My apps are working again. Everything is fine.

Great work guys! Thank you :)


Marco Gomes
from Brazil
founder
http://boo-box.com