def __call__(self, environ, start_response):
logging.info('__call__ sees UA: "%s"', environ['HTTP_USER_AGENT'])
if environ['HTTP_USER_AGENT'].startswith('curl'):
2. Deploy that
3. Check your logs. If you are being hit with lots of inbound curl calls, you should get some clues right away. But you can also try hitting it with curl yourself.
On Aug 2, 2012, at 8:39 PM, Kate <mss.ka...@gmail.com> wrote:
> On Thursday, August 2, 2012 6:55:37 PM UTC-4, Kyle Finley wrote:
>> How do I restart the instance?
> at appengine.google.com in the instance section you should see a list of instance. They each have a "Shutdown" button
>> Yes it is in the root directory. I am stumped! I didn't think it was working as I tried testing for different browsers and it didn't catch them.
> And if you have included the file in your project, it should at the very least stop me from using curl to access your site.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/1jEROOXORQMJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
def check_for_curl(self): if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): return self.error(401)
class MainHandler(webapp.RequestHandler): def get(self): check_for_curl(self) # handle request
The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
> def check_for_curl(self):
> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
> return self.error(401)
> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
> - Kyle
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> First, she's getting HEAD not GET requests, so you need to use different handler.
> Also, you aren't returning, so if you were in a GET request, it would proceed to handle the request regardless.
> Something more like this (untested):
> class MainHandler(webapp.RequestHandler):
> def head(self):
> self.error(401)
> def get(self):
> if (self.request.headers['User-Agent'].startswith('curl'))
> self.error(401)
> return
> # rest of the get handler
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Hi Joshua,
>> Thank you, that's a good thought.
>> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
>> def check_for_curl(self):
>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>> return self.error(401)
>> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
>> - Kyle
>> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Friday, August 03, 2012 8:02 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests
Yes, thank you. Do you have any thoughts on how to return error code 429?
On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:
There are couple problems with your snippet.
First, she's getting HEAD not GET requests, so you need to use different
handler.
Also, you aren't returning, so if you were in a GET request, it would
proceed to handle the request regardless.
Something more like this (untested):
class MainHandler(webapp.RequestHandler):
def head(self):
self.error(401)
def get(self):
if (self.request.headers['User-Agent'].startswith('curl'))
self.error(401)
return
# rest of the get handler
On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
Hi Joshua,
Thank you, that's a good thought.
Kate sent me some files offline, and I believe we've figured out the
problem. For the middleware to work you must be using WSGI not CGI. Someone
please correct me if I'm wrong, but I believe she would have to upgrade here
App to python27 to use it. The alternative is to do the check in the webapp
request handler:
def check_for_curl(self):
if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
return self.error(401)
class MainHandler(webapp.RequestHandler):
def get(self):
check_for_curl(self)
# handle request
The problem is that webapp doesn't recognize error code 429 so we have to
use something else. Unless there's a simple way to make it write 429?
- Kyle
-- You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
-- You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
-- You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
I wish it was that easy. Webapp2 uses dictionary to return the status code / message and 429 didn't make the list.
<pre>Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp pEngine-default.bundle/Contents/Resources/google_appengine/google/appengine /ext/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/finley/dev/scotch/operation_curl_block/main.py", line 7, in get
return self.error(429)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp pEngine-default.bundle/Contents/Resources/google_appengine/google/appengine /ext/webapp/_webapp25.py", line 435, in error
self.response.set_status(code)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp pEngine-default.bundle/Contents/Resources/google_appengine/google/appengine /ext/webapp/_webapp25.py", line 279, in set_status
message = Response.http_status_message(code)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp pEngine-default.bundle/Contents/Resources/google_appengine/google/appengine /ext/webapp/_webapp25.py", line 341, in http_status_message
raise Error('Invalid HTTP status code: %d' % code)
Error: Invalid HTTP status code: 429
Ah, I hadn't checked. I usually return a permission denied Error, or a Busy
Error, 503 I think (sorry not at my desk)
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Kyle Finley
Sent: Friday, August 03, 2012 8:43 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How can I block curl requests
I think you change 401 in this code to 429
I wish it was that easy. Webapp2 uses dictionary to return the status code /
message and 429 didn't make the list.
<pre>Traceback (most recent call last):
File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp p
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/ e
xt/webapp/_webapp25.py", line 701, in __call__
handler.get(*groups)
File "/Users/finley/dev/scotch/operation_curl_block/main.py",
line 7, in get
return self.error(429)
File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp p
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/ e
xt/webapp/_webapp25.py", line 435, in error
self.response.set_status(code)
File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp p
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/ e
xt/webapp/_webapp25.py", line 279, in set_status
message = Response.http_status_message(code)
File
"/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAp p
Engine-default.bundle/Contents/Resources/google_appengine/google/appengine/ e
xt/webapp/_webapp25.py", line 341, in http_status_message
raise Error('Invalid HTTP status code: %d' % code)
Error: Invalid HTTP status code: 429
-- You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
> Yes, thank you. Do you have any thoughts on how to return error code 429?
> On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:
>> There are couple problems with your snippet.
>> First, she's getting HEAD not GET requests, so you need to use different handler.
>> Also, you aren't returning, so if you were in a GET request, it would proceed to handle the request regardless.
>> Something more like this (untested):
>> class MainHandler(webapp.RequestHandler):
>> def head(self):
>> self.error(401)
>> def get(self):
>> if (self.request.headers['User-Agent'].startswith('curl'))
>> self.error(401)
>> return
>> # rest of the get handler
>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>>> Hi Joshua,
>>> Thank you, that's a good thought.
>>> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
>>> def check_for_curl(self):
>>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>>> return self.error(401)
>>> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
>>> - Kyle
>>> -- >>> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>>> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> I would have thought self.error(429). That doesn't work? Is there a doc that says what codes are are allowed to return?
> On Aug 3, 2012, at 11:02 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Yes, thank you. Do you have any thoughts on how to return error code 429?
>> On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:
>>> There are couple problems with your snippet.
>>> First, she's getting HEAD not GET requests, so you need to use different handler.
>>> Also, you aren't returning, so if you were in a GET request, it would proceed to handle the request regardless.
>>> Something more like this (untested):
>>> class MainHandler(webapp.RequestHandler):
>>> def head(self):
>>> self.error(401)
>>> def get(self):
>>> if (self.request.headers['User-Agent'].startswith('curl'))
>>> self.error(401)
>>> return
>>> # rest of the get handler
>>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>>>> Hi Joshua,
>>>> Thank you, that's a good thought.
>>>> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
>>>> def check_for_curl(self):
>>>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>>>> return self.error(401)
>>>> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
>>>> - Kyle
>>>> -- >>>> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>>>> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>>>> To post to this group, send email to google-appengine@googlegroups.com.
>>>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>>>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> On Aug 3, 2012, at 10:45 AM, Joshua Smith wrote:
> I would have thought self.error(429). That doesn't work? Is there a doc > that says what codes are are allowed to return?
> On Aug 3, 2012, at 11:02 AM, Kyle Finley <kylefin...@gmail.com> wrote:
> Yes, thank you. Do you have any thoughts on how to return error code 429?
> On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:
> There are couple problems with your snippet.
> First, she's getting HEAD not GET requests, so you need to use different > handler.
> Also, you aren't returning, so if you were in a GET request, it would > proceed to handle the request regardless.
> Something more like this (untested):
> class MainHandler(webapp.RequestHandler): > def head(self): > self.error(401)
> def get(self): > if (self.request.headers['User-Agent'].startswith('curl')) > self.error(401) > return > # rest of the get handler
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
> Hi Joshua,
> Thank you, that's a good thought.
> Kate sent me some files offline, and I believe we've figured out the > problem. For the middleware to work you must be using WSGI not CGI. Someone > please correct me if I'm wrong, but I believe she would have to upgrade > here App to python27 to use it. The alternative is to do the check in the > webapp request handler:
> def check_for_curl(self): > if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): > return self.error(401)
> The problem is that webapp doesn't recognize error code 429 so we have to > use something else. Unless there's a simple way to make it write 429?
> - Kyle
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
>> On Aug 3, 2012, at 10:45 AM, Joshua Smith wrote:
>> I would have thought self.error(429). That doesn't work? Is there a doc >> that says what codes are are allowed to return?
>> On Aug 3, 2012, at 11:02 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Yes, thank you. Do you have any thoughts on how to return error code 429?
>> On Aug 3, 2012, at 9:51 AM, Joshua Smith wrote:
>> There are couple problems with your snippet.
>> First, she's getting HEAD not GET requests, so you need to use different >> handler.
>> Also, you aren't returning, so if you were in a GET request, it would >> proceed to handle the request regardless.
>> Something more like this (untested):
>> class MainHandler(webapp.RequestHandler): >> def head(self): >> self.error(401)
>> def get(self): >> if (self.request.headers['User-Agent'].startswith('curl')) >> self.error(401) >> return >> # rest of the get handler
>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Hi Joshua,
>> Thank you, that's a good thought.
>> Kate sent me some files offline, and I believe we've figured out the >> problem. For the middleware to work you must be using WSGI not CGI. Someone >> please correct me if I'm wrong, but I believe she would have to upgrade >> here App to python27 to use it. The alternative is to do the check in the >> webapp request handler:
>> def check_for_curl(self): >> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): >> return self.error(401)
>> The problem is that webapp doesn't recognize error code 429 so we have to >> use something else. Unless there's a simple way to make it write 429?
>> - Kyle
>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
> There are couple problems with your snippet.
> First, she's getting HEAD not GET requests, so you need to use different > handler.
> Also, you aren't returning, so if you were in a GET request, it would > proceed to handle the request regardless.
> Something more like this (untested):
> class MainHandler(webapp.RequestHandler): > def head(self): > self.error(401)
> def get(self): > if (self.request.headers['User-Agent'].startswith('curl')) > self.error(401) > return > # rest of the get handler
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
> Hi Joshua,
> Thank you, that's a good thought.
> Kate sent me some files offline, and I believe we've figured out the > problem. For the middleware to work you must be using WSGI not CGI. Someone > please correct me if I'm wrong, but I believe she would have to upgrade > here App to python27 to use it. The alternative is to do the check in the > webapp request handler:
> def check_for_curl(self): > if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): > return self.error(401)
> The problem is that webapp doesn't recognize error code 429 so we have to > use something else. Unless there's a simple way to make it write 429?
> - Kyle
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
> Just read this. Thanks.
> Looks like it works now.
> Thanks to all!!!!!
> Kate
> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
>> There are couple problems with your snippet.
>> First, she's getting HEAD not GET requests, so you need to use different >> handler.
>> Also, you aren't returning, so if you were in a GET request, it would >> proceed to handle the request regardless.
>> Something more like this (untested):
>> class MainHandler(webapp.RequestHandler): >> def head(self): >> self.error(401)
>> def get(self): >> if (self.request.headers['User-Agent'].startswith('curl')) >> self.error(401) >> return >> # rest of the get handler
>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Hi Joshua,
>> Thank you, that's a good thought.
>> Kate sent me some files offline, and I believe we've figured out the >> problem. For the middleware to work you must be using WSGI not CGI. Someone >> please correct me if I'm wrong, but I believe she would have to upgrade >> here App to python27 to use it. The alternative is to do the check in the >> webapp request handler:
>> def check_for_curl(self): >> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): >> return self.error(401)
>> The problem is that webapp doesn't recognize error code 429 so we have to >> use something else. Unless there's a simple way to make it write 429?
>> - Kyle
>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
Hopefully they will stop once they realizes they're being blocked - if it's a legitimate service. If they are not, and they are reading this thread, they will probably just change their user agent, though. The only way to stop them from reaching your app entirely would be to use a service like CloudFlare. I have no personal experience with CloudFlare, however, so I can not state definitively that it will solve your problem. And adding an additional layer can result in it's own issues, as demonstrated by the August 1st CloudFlare block.
> But I still don't like them hitting my site every 500 ms! e.g.
> 2012-08-06 13:56:02.725 / 401 32ms 0kb curl/7.18.2 (i386-redhat-linux-gnu) libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
> On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
> Just read this. Thanks.
> Looks like it works now.
> Thanks to all!!!!!
> Kate
> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
> There are couple problems with your snippet.
> First, she's getting HEAD not GET requests, so you need to use different handler.
> Also, you aren't returning, so if you were in a GET request, it would proceed to handle the request regardless.
> Something more like this (untested):
> class MainHandler(webapp.RequestHandler):
> def head(self):
> self.error(401)
> def get(self):
> if (self.request.headers['User-Agent'].startswith('curl'))
> self.error(401)
> return
> # rest of the get handler
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Hi Joshua,
>> Thank you, that's a good thought.
>> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
>> def check_for_curl(self):
>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>> return self.error(401)
>> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
>> - Kyle
>> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ClwBVQQTxesJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
for example. Curl does follow redirects by default, but it doesn't have to, so this may or may not have an effect.
If they are following redirects, you could set up a backend with a handler that just sleeps for 30 seconds. Then redirect them all to that one backend, where they can all sit in line waiting.
On Aug 6, 2012, at 2:58 PM, Kate <mss.ka...@gmail.com> wrote:
> But I still don't like them hitting my site every 500 ms! e.g.
> 2012-08-06 13:56:02.725 / 401 32ms 0kb curl/7.18.2 (i386-redhat-linux-gnu) libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18
> On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
> Just read this. Thanks.
> Looks like it works now.
> Thanks to all!!!!!
> Kate
> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
> There are couple problems with your snippet.
> First, she's getting HEAD not GET requests, so you need to use different handler.
> Also, you aren't returning, so if you were in a GET request, it would proceed to handle the request regardless.
> Something more like this (untested):
> class MainHandler(webapp.RequestHandler):
> def head(self):
> self.error(401)
> def get(self):
> if (self.request.headers['User-Agent'].startswith('curl'))
> self.error(401)
> return
> # rest of the get handler
> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>> Hi Joshua,
>> Thank you, that's a good thought.
>> Kate sent me some files offline, and I believe we've figured out the problem. For the middleware to work you must be using WSGI not CGI. Someone please correct me if I'm wrong, but I believe she would have to upgrade here App to python27 to use it. The alternative is to do the check in the webapp request handler:
>> def check_for_curl(self):
>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'):
>> return self.error(401)
>> The problem is that webapp doesn't recognize error code 429 so we have to use something else. Unless there's a simple way to make it write 429?
>> - Kyle
>> -- >> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
>> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ClwBVQQTxesJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
> for example. Curl does follow redirects by default, but it doesn't have > to, so this may or may not have an effect.
> If they are following redirects, you could set up a backend with a handler > that just sleeps for 30 seconds. Then redirect them all to that one > backend, where they can all sit in line waiting.
> On Aug 6, 2012, at 2:58 PM, Kate <mss.ka...@gmail.com> wrote:
> But I still don't like them hitting my site every 500 ms! e.g.
> On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
>> Just read this. Thanks.
>> Looks like it works now.
>> Thanks to all!!!!!
>> Kate
>> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
>>> There are couple problems with your snippet.
>>> First, she's getting HEAD not GET requests, so you need to use different >>> handler.
>>> Also, you aren't returning, so if you were in a GET request, it would >>> proceed to handle the request regardless.
>>> Something more like this (untested):
>>> class MainHandler(webapp.RequestHandler): >>> def head(self): >>> self.error(401)
>>> def get(self): >>> if (self.request.headers['User-Agent'].startswith('curl')) >>> self.error(401) >>> return >>> # rest of the get handler
>>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>>> Hi Joshua,
>>> Thank you, that's a good thought.
>>> Kate sent me some files offline, and I believe we've figured out the >>> problem. For the middleware to work you must be using WSGI not CGI. Someone >>> please correct me if I'm wrong, but I believe she would have to upgrade >>> here App to python27 to use it. The alternative is to do the check in the >>> webapp request handler:
>>> def check_for_curl(self): >>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): >>> return self.error(401)
>>> The problem is that webapp doesn't recognize error code 429 so we have >>> to use something else. Unless there's a simple way to make it write 429?
>>> - Kyle
>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/ClwBVQQTxesJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
> for example. Curl does follow redirects by default, but it doesn't have > to, so this may or may not have an effect.
> If they are following redirects, you could set up a backend with a handler > that just sleeps for 30 seconds. Then redirect them all to that one > backend, where they can all sit in line waiting.
> On Aug 6, 2012, at 2:58 PM, Kate <mss.ka...@gmail.com> wrote:
> But I still don't like them hitting my site every 500 ms! e.g.
> On Monday, August 6, 2012 2:50:55 PM UTC-4, Kate wrote:
>> Just read this. Thanks.
>> Looks like it works now.
>> Thanks to all!!!!!
>> Kate
>> On Friday, August 3, 2012 10:51:23 AM UTC-4, Joshua Smith wrote:
>>> There are couple problems with your snippet.
>>> First, she's getting HEAD not GET requests, so you need to use different >>> handler.
>>> Also, you aren't returning, so if you were in a GET request, it would >>> proceed to handle the request regardless.
>>> Something more like this (untested):
>>> class MainHandler(webapp.RequestHandler): >>> def head(self): >>> self.error(401)
>>> def get(self): >>> if (self.request.headers['User-Agent'].startswith('curl')) >>> self.error(401) >>> return >>> # rest of the get handler
>>> On Aug 3, 2012, at 10:34 AM, Kyle Finley <kylefin...@gmail.com> wrote:
>>> Hi Joshua,
>>> Thank you, that's a good thought.
>>> Kate sent me some files offline, and I believe we've figured out the >>> problem. For the middleware to work you must be using WSGI not CGI. Someone >>> please correct me if I'm wrong, but I believe she would have to upgrade >>> here App to python27 to use it. The alternative is to do the check in the >>> webapp request handler:
>>> def check_for_curl(self): >>> if self.request.environ['HTTP_USER_AGENT'].startswith('curl'): >>> return self.error(401)
>>> The problem is that webapp doesn't recognize error code 429 so we have >>> to use something else. Unless there's a simple way to make it write 429?
>>> - Kyle
>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/TQuZYYR0wrAJ. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/ClwBVQQTxesJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en.
It isn't stopping them. I am just not getting errors. What is troubling is that these curl requests are counting as hits and there are so many of tens of thousands of them it is hard for me to analyze site traffic as genuine requests are buried in the curl stats.
> It isn't stopping them. I am just not getting errors. What is troubling is that these curl requests are counting as hits and there are so many of tens of thousands of them it is hard for me to analyze site traffic as genuine requests are buried in the curl stats.
> -- > You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/p4fPoQaAACIJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
My site is now down as I'm over quota. I can't tun billing on as it is too expensive to pay for these dos attacks.
Thanks everyone for being helpful but I think I'm beaten on this. It seems a pity that a non profit site could be brought down by this but that's the case. Google doesn't seem to care as there has been no response on my production issue. I suppose it isn't in their interest as I either pay for the attacks or lose my site. V discouraging.
All attempts at blocking the attacks has only increased their volume.
On Wednesday, August 8, 2012 1:24:36 PM UTC-4, Kyle Finley wrote:
> In the admin logs - under options - you can filter by regular expression. > Does that help?
> On Aug 8, 2012, at 12:11 PM, Kate wrote:
> > It isn't stopping them. I am just not getting errors. What is troubling > is that these curl requests are counting as hits and there are so many of > tens of thousands of them it is hard for me to analyze site traffic as > genuine requests are buried in the curl stats.
> > -- > > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/p4fPoQaAACIJ. > > To post to this group, send email to google-a...@googlegroups.com<javascript:>.
On Thursday, August 9, 2012 6:17:32 AM UTC+3, Kate wrote:
> My site is now down as I'm over quota. I can't tun billing on as it is > too expensive to pay for these dos attacks.
> Thanks everyone for being helpful but I think I'm beaten on this. It seems > a pity that a non profit site could be brought down by this but that's the > case. > Google doesn't seem to care as there has been no response on my production > issue. I suppose it isn't in their interest as I either pay for the attacks > or lose my site. V discouraging.
> All attempts at blocking the attacks has only increased their volume.
> Kate
> On Wednesday, August 8, 2012 1:24:36 PM UTC-4, Kyle Finley wrote:
>> In the admin logs - under options - you can filter by regular expression. >> Does that help?
>> On Aug 8, 2012, at 12:11 PM, Kate wrote:
>> > It isn't stopping them. I am just not getting errors. What is troubling >> is that these curl requests are counting as hits and there are so many of >> tens of thousands of them it is hard for me to analyze site traffic as >> genuine requests are buried in the curl stats.
>> > -- >> > You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> > To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/p4fPoQaAACIJ. >> > To post to this group, send email to google-a...@googlegroups.com. >> > To unsubscribe from this group, send email to >> google-appengi...@googlegroups.com. >> > For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en.
But the ip address varies. My code forces 302 response. Before I added the code they were throwing errors head method not found. But even though I am doing the 303 I am still getting front end time exceeded and these requests are taking up about 95% of my quota. So to keep the site alive I would have to pay for them, I have lost most of my European and Australian visitors because the site is down every night during those places daylight hours. Obviously I can't continue like this and so will have to move to a provider capable of blocking these requests,
But the ip address varies. My code forces 302 response. Before I added the code they were throwing errors head method not found. But even though I am doing the 303 I am still getting front end time exceeded and these requests are taking up about 95% of my quota. So to keep the site alive I would have to pay for them, I have lost most of my European and Australian visitors because the site is down every night during those places daylight hours. Obviously I can't continue like this and so will have to move to a provider capable of blocking these requests,
