Low hitting site experiencing denial of service attack - site opffline!

[Kate]

My site has been on GAE for several years but am now experiencing denial of service attacks. I don't want to pay for them so my site is now down. I put in a production issue last week and have tried blocking the user agent (curl). But I can't block by IP as all IPs are different.

Apart from moving from GAE what can  I do.

Thanks in advance,

Kate

[Kristopher Giesing]

Did your attacker change lines of attack after put in the curl blocking?  Or did the blocking itself not solve the problem?

Here's a thought: use node.js to set up a simple filter (node.js because it's high throughput for very lightweight operations).  For valid traffic, proxy to your real GAE site, otherwise proxy to a honeypot that will black-hole the attacker's requests.  I say proxy rather than redirect so that your GAE site can whitelist the IP of your filter - I think HTTP referrer can be spoofed, which would defeat the purpose.

Writing node.js proxies is pretty straightforward (there's code for it on the net) so the real trick is then making sure you can distinguish attacks from normal traffic - hence my original question.

This is all off the top of my head, there may be holes in this plan I hadn't thought of.  But it seems like it could work.

- Kris

[Kristopher Giesing]

PS.  I think it would also be possible to hide the identity of the GAE app the filter talks to.  That way you can move your GAE app to a different app ID, so that the attacker can't hit it directly without going through your filter.  DNS would also point to your filter.

[Kristopher Giesing]

PPS. The point of doing this outside GAE is so that you can run the filter on a fixed cost VM solution like Linode.  Then the only question becomes whether your attacker can overwhelm your node.js filter... hopefully, it won't reach that point.

- Kris

[Kate]

I don't know how to write the node.js. I am surprised that google cannot help me here. I do appreciate your help but I just can't do this without spending hours of study. To answer your other question, the blocking attempt didn't solve the problem. The curl requests come about 3 times per second. I have been working on trying to stop them for over a week now. I suppose I can just change my domain name or move to another hosting service but it seems a pity.

I have had this site since 1996 with never a problem.

[Kristopher Giesing]

There's a short example of a node.js proxy server here:

http://www.catonmat.net/http-proxy-in-nodejs/

It might not be as hard as you imagine to get this working.  I'd hate to see the attackers win :(

- Kris

[Ernesto Oltra]

Have you tried CloudFlare? It's basically the same, a proxy to your app, and you don't have to make/admin it. It has a reputation system, and some others pretty good settings to deal with DOS attacks.

BTW, I'm not affiliated at all with the company, I'm only other web dev.

[Kate]

I made a cloudflare account but it can't resolve my domain name, www.australiansabroad.com as I don't have a dns entry at network solutions.com where I register my sites. I have a special entry that google resolves. And if I put in my appspot site name cloudflare says it cannot accept that.

[Kate]

I will have a go and then I will have to give up on google apps. I am very surprised they allow dos attacks. No web provider has allowed this to happen before.

[Barry Hunter]

On Thu, Aug 9, 2012 at 12:48 PM, Kate <mss....@gmail.com> wrote:

I will have a go and then I will have to give up on google apps. I am very surprised they allow dos attacks. No web provider has allowed this to happen before.

You probably have just been lucky. ddos attacks are actully pretty rare, so its never been an issue before. Very  few providers are proactive about blocking. Many will just throw you out (sadly). 

The design of AppEngine, does make it hard to block the requests before hitting instances. On other providers maybe could of used something like rewrite rules to block them before they hit dynamic instances (like php). But also the different way of paying (many providers only charge for bandwidth and/or requests) means the instance time adds up, which woudlnt be a problem elsewhere.  

Something not sure if been mentioned before is Google do have a means to contact them to be reimbused for changes assocated with a DOS

https://developers.google.com/appengine/kb/billing#dos

Have you tried that? 

One final thing, it might not actully be a deliberate dos. It might just be a misbehavioed crawler. Its getting stuck in a loop, on your error messages - it doesnt expect a HEAD to fail, so its just retrying. Hence the suggestion to actully handle head requests in the first place. (although at the time didnt appreciate how many of these requests you getting) Turning them into 200 OK, might cause them to go away satisfied. 

Couple it with Cache-Control headers, to allow the Google edge cache to cache the requests, means the edgecache might be able to forfil the requests without touching dynamic instances. 

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/0RfQ_4DZg3oJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kate]

I tried sending back a 302 when I get a head request and that didn't help. All the ips asre different and not all come from planet lab. But I will contact planet lab. You would think they would stop however anyway as my site is down about 1/8th of the day now.

I will try the suggestions to date and if that doesn't work will seek out a new provider or just rename the site and get a new ip. I don't think it is planet lab looping as why would they keep changing the ip address?

I googled the problem and most people suggest getting the provider to help but google isn't interested as they have nothing to gain.

I can ask new providers their policy before inning up.

I did find another person on GAE with an identical problem but apparently he gave up.

Today I am already in to 20% of my quota and will soon lose. My only advertiser. I have already lost one, and my Adsense revenue is right down because of all the downtime.

I also use other google services not related to this site being attacked so if no resolution I will vote with my feet.

[Kate]

[Jeff Schnitzer]

Someone else asked this, but I'll ask it again - have you tried using
CloudFlare? Seems like this is what they are supposed to be for. I'd
like to know if it works.

Google can't magically make your DDOS go away. I don't know of any
hosting providers that can. You need to use some sort of filtering
service like CF to handle it - there are several, but I gather most
are fairly expensive. At least CF is free.

Sorry, but this is just the reality of the internet. Moving off of
GAE might produce a somewhat cheaper solution, but it won't solve your
DDOS problem if your attacker is paying any attention.

Jeff

On Wed, Aug 8, 2012 at 10:06 PM, Kate <mss....@gmail.com> wrote:
> I don't know how to write the node.js. I am surprised that google cannot help me here. I do appreciate your help but I just can't do this without spending hours of study. To answer your other question, the blocking attempt didn't solve the problem. The curl requests come about 3 times per second. I have been working on trying to stop them for over a week now. I suppose I can just change my domain name or move to another hosting service but it seems a pity.
>
> I have had this site since 1996 with never a problem.
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/RLTiLlM5cTsJ.

[Kate]

I made a cloudflare account but it can't resolve my domain name, www.australiansabroad.com as I don't have a dns entry at network solutions.com where I register my sites. I have a special entry that google resolves. And if I put in my appspot site name cloudflare says it cannot accept that.

[Kate]

My site is in Python.

I am afraid the attackers have won.

It seems nothing can be done and we on GAE have to put up with it. Or pay. I will increase my quota while I move my site and cop it sweet as they say in Australia. Thanks for everyone who tried to help.


Kate