Using IGNORABLE_404_URLS

[shacker]

Our Django sites get literally hundreds of bogus 404 requests per day. For example:

Referrer: http://domain.edu/
Requested URL: /signup/
User agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)
IP address: 168.9.86.2

The "referrer" line is a lie because nowhere on our site do we point to "/signup" . I've given up trying to figure out how these things are generated or how to block them. But I would like to limit the number of daily emails to just the actual/legit 404s. So I started using IGNORABLE_404_URLS, per:

https://docs.djangoproject.com/en/dev/howto/error-reporting/#errors

IGNORABLE_404_URLS = (
   re.compile(r'\.(php|cgi)$'),
    re.compile(r'^/forums'),
    re.compile(r'^/signup'),
    re.compile(r'/src/'),    
    re.compile(r'/pdf/'),        
)

Unfortunately this seems to have no effect. Shouldn't the regex pattern there catch the bogus request domain.edu/signup ? Or is this not working because the way the requests are being submitted somehow bypasses Django's ability to catch it as an error? I'm just not clear what's going on here.

Thanks.

[Kurtis Mullins]

Hey, I don't think I'll be able to help you much with figuring this out in Django-land. I do have one suggestion though. You could manually block these using your front-end server (Nginx, Apache, etc...) so that way it doesn't even reach Django. Not only would this be a hypothetically easy fix, you'd save yourself some server load as well.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/7xxDzuRZue4J.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Sam Walters]

I second this. Not something you would implement in django. Would
handle that with the web server front end. Eg: i used to get requests
to phpmyadmin login url requests (i assume thats because that software
is a security issue). So i got nginx to return a minimal 404 message
instead of the full 404 page.

[shacker]

Thanks guys. Yes, after thinking about this more, there are really two things going on here:

1) Figuring out why entries in Django's IGNORABLE_404_URLS seem to be completely ignored.

2) Blocking bad behavior.

Since most of these requests are bogus, #2 is really what I want. Set up a bunch of 403 mod_rewrite rules in Apache and have dramatically cut down on the number of bad reports. 

I'm still curious about #1, but this is a big improvement. Thanks.