DoS susceptability via email logging

[Sam Berry]

Hello there,

I am currently running a number of small sites using logging via email
to notify me of 404s and server errors. The volume of email produced
is perfectly manageable due to the low level of traffic.

It does occur to me that it would be very easy to disrupt the site by
simply sending repeated requests which trigger a 404 Not Found. I'm
sure 10 404s a second would completely shutdown the process to normal
requests due to the time taken to send the log emails. I'm not using a
message queueing system.

I would like to keep the option of receiving logging emails. Is there
anything I can do to throttle or limit the email sending without
writing my own logging handler?

Any thoughts would be much appreciated.

Many thanks,
Sam Berry

[Nikolas Stevenson-Molnar]

I imagine you can use something like Celery ( has been recommended on
this list before: http://celeryproject.org/ ) to queue email tasks and
handle them in a single background process.

_Nik

[creecode]

Hello Sam,

You might want to do something like detect the situation and then at the firewall for your router or server, cut off the offending addresses.  No specific details to offer.  The advantage to getting a rule onto your firewall would be reduced load on the software side of request processing.

Another approach would be to use server monitoring software such as Nagios to regularlly check that the server/websites are up and running and give you a heads up when there is a problem.  The advantage here is that you can then find out what the problem is and take care of it manually.  If there is some kind of pattern to the problem then you might be able to automate a solution.  Going the monitoring route generally requires one or more other computers to be effective.

Toodle-looooooooooo............
creecode

[Russell Keith-Magee]

On Thu, Dec 8, 2011 at 8:42 PM, Sam Berry <samk...@googlemail.com> wrote:
> Hello there,
>
> I am currently running a number of small sites using logging via email
> to notify me of 404s and server errors. The volume of email produced
> is perfectly manageable due to the low level of traffic.

First off, I'd point out that the 404 emails aren't sent every time
your server generates a 404 -- they're only generated as a result of a
404 *from an internal link on your site*. If John Q Hacker hits
http://yoursite.com/this_page_doesnt_exist, you won't get a 404 email.

If you're getting *any* 404 emails, it means there are internal links
on your site that don't resolve. If this is something that is
happening systematically, at a rate that has given you cause for
concern about the potential for DOS, I'd suggest you have a bigger
problem with your site.

> It does occur to me that it would be very easy to disrupt the site by
> simply sending repeated requests which trigger a 404 Not Found. I'm
> sure 10 404s a second would completely shutdown the process to normal
> requests due to the time taken to send the log emails. I'm not using a
> message queueing system.
>
> I would like to keep the option of receiving logging emails. Is there
> anything I can do to throttle or limit the email sending without
> writing my own logging handler?

Interestingly, 404 emails aren't handled by a logging handler (unlike
500 messages) -- they're handled as a direct mail to site managers.
This is something that is probably worth a ticket in itself; using the
logging framework for 404 mails would allow for much more flexibility.

So - what you want here isn't a custom logging handler, but a custom
*mail* handler -- django-mailer [1] is a good candidate here. This
makes "sending" an email a very short lived operation from the
perspective of your web server -- all the web request does is queue
the email for sending. The actual sending of mail is handled out of
the request-response cycle. This removes the possibility of "DOS
attack by 404" because the expensive operation is taken out of the
request-response cycle.

Of course, you could still end up being flooded with email -- so you
either need to (a) monitor the size of your mail queue to make sure it
isn't getting flooded, or (b) put a processor in place to
merge/throttle the contents of the mail queue. This could also be
handled with a custom mail handler; however, I can't point you at a
ready-to-use candidate for this.

Another option is to use a mechanism other than the 404 emails to log
and handle errors. Django-sentry [2] and Arecibo [3] are two
candidates here, both of which are easy to install, and provide much
richer analytics than a full mailbox :-)

[1] https://github.com/jtauber/django-mailer/
[2] https://github.com/dcramer/django-sentry/
[3] http://areciboapp.com/

Yours,
Russ Magee %-)

[Donald Stufft]

On Thursday, December 8, 2011 at 7:33 PM, Russell Keith-Magee wrote:

On Thu, Dec 8, 2011 at 8:42 PM, Sam Berry <samk...@googlemail.com> wrote:

Hello there,

I am currently running a number of small sites using logging via email

to notify me of 404s and server errors. The volume of email produced

is perfectly manageable due to the low level of traffic.

First off, I'd point out that the 404 emails aren't sent every time

your server generates a 404 -- they're only generated as a result of a

404 *from an internal link on your site*. If John Q Hacker hits

http://yoursite.com/this_page_doesnt_exist, you won't get a 404 email.

This isn't exactly accurate. It sends them for any 404 response where the HTTP_REFERER

header is not in the ignorable urls setting, is not empty, and does not contain a ?

This is also trivial to get around and if someone is attempting to trigger a boatload of

404 emails this check is only going to prevent someone who knows nothing about Django (in

which case how do they know about 404 email sending?). 

If you're getting *any* 404 emails, it means there are internal links

on your site that don't resolve. If this is something that is

happening systematically, at a rate that has given you cause for

concern about the potential for DOS, I'd suggest you have a bigger

problem with your site.

It does occur to me that it would be very easy to disrupt the site by

simply sending repeated requests which trigger a 404 Not Found. I'm

sure 10 404s a second would completely shutdown the process to normal

requests due to the time taken to send the log emails. I'm not using a

message queueing system.

I would like to keep the option of receiving logging emails. Is there

anything I can do to throttle or limit the email sending without

writing my own logging handler?

Interestingly, 404 emails aren't handled by a logging handler (unlike

500 messages) -- they're handled as a direct mail to site managers.

This is something that is probably worth a ticket in itself; using the

logging framework for 404 mails would allow for much more flexibility.

Agreed, 404 should be configurable via Logging. 

So - what you want here isn't a custom logging handler, but a custom

*mail* handler -- django-mailer [1] is a good candidate here. This

makes "sending" an email a very short lived operation from the

perspective of your web server -- all the web request does is queue

the email for sending. The actual sending of mail is handled out of

the request-response cycle. This removes the possibility of "DOS

attack by 404" because the expensive operation is taken out of the

request-response cycle.

This is True.  You could also move email sending to a celery task

or something similar.

Of course, you could still end up being flooded with email -- so you

either need to (a) monitor the size of your mail queue to make sure it

isn't getting flooded, or (b) put a processor in place to

merge/throttle the contents of the mail queue. This could also be

handled with a custom mail handler; however, I can't point you at a

ready-to-use candidate for this.

Another option is to use a mechanism other than the 404 emails to log

and handle errors. Django-sentry [2] and Arecibo [3] are two

candidates here, both of which are easy to install, and provide much

richer analytics than a full mailbox :-)

This is, in my opinion, the best option if you wish to track 404's. 

[1] https://github.com/jtauber/django-mailer/

[2] https://github.com/dcramer/django-sentry/

[3] http://areciboapp.com/

Yours,

Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.