Why I am I getting these HEAD / HTTP/1.1 errors?

[Kate]

I can't work out why I am getting these errors. In 13 hours I have  26,205 of them. I have no forms in my code. 

I do have  urlfetch

If as I have been told the error means that the head method is noot implemented, how do I set up my app to support the Head methiod? I cannot find anywhere I am using it,

Here is an example of the error message.

2012-08-01 14:25:52.668 / 405 51ms 0kb curl/7.18.2 (i386-redhat-linux-gnu) libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18

193.136.19.13 - - [01/Aug/2012:12:25:52 -0700] "HEAD / HTTP/1.1" 405 124 - "curl/7.18.2 (i386-redhat-linux-gnu) libcurl/7.18.2 NSS/3.12.2.0 zlib/1.2.3 libidn/0.6.14 libssh2/0.18" "aussieclouds.appspot.com" ms=51 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.000045 instance=00c61b117c01c1a2dec223c75dcfeaab60fbd7

Kate

[Joshua Smith]

Looks to me like somebody is doing head requests to your app using curl. curl is a command line program that downloads web pages, similar to wget. This particular user is coming from linux, possibly from here: Portugal, Braga, Universidade do Minho (although reverse IP search results aren't always accurate).

If I were you, I'd just ignore the errors. You aren't in business to make some college kid's half-baked web scraper work correctly.

However, if you do want to handle the HEAD requests, you just add a method analogous to the post or get handlers you have implemented already in your main.py.

-Joshua

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TG2a3CId3bwJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kate]

Thanks for that, but these requests are coming from many different ip addresses and I can see no pattern to them.

I am just wondering why there are so many and if it is affecting my quota.

Also I do not have a main.py. I have an index.py. I'm wondering now how it works at all!

[Jay]

HEAD requests like this often come from social sites or search engines to test a url. This may or may not be the case for you, but that is my experience.

Depending on the framework you are using , you should be able to handle HEAD requests if you want to.

Handling the request, even if it errors does consume some resources, so there is likely some small impact to quota. If you don't want the systems hitting your site, probably the best option is robots.txt

[Kate]

The problem is I am getting tens of thousands of these requests per day and they ARE impacting my quota and they come from different IPs tho most seem to be in Europe. They are all curl requests so they are not genuine.

I'll have to move to a new ISP who can help me in blocking them as I can't afford this. My "real" requests are about .01% of these curl requests!