DOS attack via log bloat

[Chris McCann]

SD Ruby,

This morning I checked on a Rails app I've had in production for many years and found that the app was dumping a full stacktrace to the screen instead of either rendering the expected page or showing the "Oops.  Something went wrong" error page.

I immediately tried to do a cap web:deploy:disable to at least put up the maintenance page but that failed, too.  It was slightly unnerving since nothing like this had happened to me before. 

Looking into it I saw a MySQL error with an error code of 28 in the message.  A quick Google of that showed that's what MySQL does when it's out of disk space.

A quick "df -h" and sure enough, the 6 GB disk showed 0% available.  WTF, I wondered.

I headed over to the Rails log directory to see what the production log showed.  Well, the first thing it showed was the production.log file was almost 1 GB.  That's not normal, as logrotate is set to rotate the log every week and I've barely ever seen one even 1/10 that size.

I gzipped the log and pulled it down locally for a little spelunking.  Before long I found a suspicious IP address hitting the Site#index page over, and over, and over -- 2,131,853 times, to be exact.  That caused the production log to blow way past the available disk space on the slice.  Truncating the log immediately restored the apps functionality.

As a quick reaction measure I added the offending IP to be dropped via iptables.  The customer support techs at Rimuhosting.com (who are, frankly, the most responsive I've ever worked with) were very helpful, and they even let me know I was eligible for a free upgrade of over 1/2 GB of RAM and 6 GB of disk space, which they promptly added.

I modified my logrotate conf file for the app to rotate every 100MB instead of weekly.  If you're rotating on time like I was you're setting yourself up for what I'm calling a "log-bloat denial of service" attack.

A slightly scary but instructive lesson, which leads me to my questions:

- has anyone dealt with a similar style attack in a Rails app?  

- are there some security best practices I'm not implementing?

- has anyone used fail2ban or similar software with a Rails app to automatically block nefarious traffic?  

- what are your favorite monitoring solutions to get early warnings of disk space, RAM, or other impending problems?

Thanks,

Chris McCann

[Matt Aimonetti]

Hey Chris,

Thanks for sharing, interesting issue I've also seen with people simply forgetting to set logrotate.

I wouldn't call that an attack, the IP could be a proxy, a scrapper, a response time monitor or indeed someone doing a very slow, single IP based DDOS.

I think that really the issue isn't with the traffic you were getting but with your log management. 

You can use logrotate with min/maxsize and timeperiod to avoid these kind of issues.

You might also want to not use the default very verbose Rails output and instead use a more compact, more easily parsable log format. 

Another option is to use something like logentries.com and not keep any logs locally (or logrotate more aggressively).

Let me know if have any questions and best of luck.

- Matt

--
--
SD Ruby mailing list
sdr...@googlegroups.com
http://groups.google.com/group/sdruby
---
You received this message because you are subscribed to the Google Groups "SD Ruby" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdruby+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris McCann]

Matt,

Yeah, I hesitated to call it an "attack" per se but the sheer volume of requests for the apps default page, several per second, is clearly not anything resembling normal, and the end result was my server going down.  Sure, it could be something misconfigured on the sending end but very abnormal nonetheless.

I did modify the logrotate configuration to use file size.  Thanks for the suggestion on tweaking the verbosity and format of the Rails log output -- I'll definitely do that.  

I am looking into using logentries.com, too.  It looks like a nice service.

Cheers,

Chris

You received this message because you are subscribed to a topic in the Google Groups "SD Ruby" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sdruby/P7-OnkqYffA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sdruby+un...@googlegroups.com.

[John Lynch]

Chris, check out fail2ban for something simple , or modsecurity for a (much) more complicated but thorough solution.  Also, papertrailapp.com is a very nice logging-as-a-service.

- John

[Bill Vieux]

Chris,

Thanks for sharing. Always interesting to see the troubleshooting steps and resolution for production issues. 

I've used logentries and papertrail (via Heroku addons) and they have both been great. Papertrail has some nice recommendations regarding Rails log verbosity here: http://help.papertrailapp.com/kb/configuration/controlling-verbosity