Denial of Service Attack on a GAE Application

[Tony Smith]

Hi,

How would one protect themselves against a denial of service attack?
It is very, very easy to study how an application works, write some
requests in a script and then run the scripts from a few locations in
a never-ending loop. In no time, the GAE app will go over quota and
start serving 403 error pages.


Is there anything we can do to protect against this?

Thanks,
Tony

[Rick Thomas]

I noticed this code

http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py

[uprise78]

Rich, correct me if I'm wrong but that code you sent will add a new
read and a new write to every page hit and on top of that if the
person is banned they will still be able to reach the webpage and
cause it to perform this same read/write. I think DDOS attach
protection needs to be done on a much lower level than that.

[scottxu]

Also a big concern to me, especially when triggering httpmr-type
requests.

DDOS protection better be provided at low level, or infrastrature
level. If not
provided by the platform, then application developers have to know
much
details of AppEngine to do it.

Scott

> >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py- Hide quoted text -
>
> - Show quoted text -

[Tony Smith]

Yes, it needs to be at a very low level.

In my infrastructure, when the system is detecting a DOS attack it
creates a firewall rule on the offending IP address. This rule will
expire in a few minutes. If the DOS continues after the few minutes
then An hour long restriction is imposed. If after an hour the DOS is
still active then a permanent restriction is created and an admin is
notified. The admin then researched the attack and if it proves to be
of malicious intent (not some misconfiguration on the software) then
we take it up with the ISP.

Does Google have (currently or in plan) something similar that will
help address the DOS issues? You go through all the trouble to launch
an application and then they take it down through DOS... this would
really look bad for a start-up website not to mention Google. Well,
actually Google's GAE wouldn't be that affected because the users will
assume that the application went over it's quota. The normal users
would not know of the DOS attack.

Thanks,
Tony

[scottxu]

I don't know the status of these issues. Timeout and DDOS are two
concerns for me.

To prevent timeout, applications have to check timer very quickly
and break the request processing aggressively, which looks like
a DDOS :-). Even worse, it's difficult to guarrantee the loop between
timer-checking is less than timeout. Failing this, the request will
not be processed unless you modify the application.

Hope I am wrong, or hope to see solutions for these issues will
come out soon.

Scott

[Tony Smith]

Timeout is not an issue if the requests are minimal. And you can
control what the user can request from your application.

The MAJOR concern is DOS.
I have created a script that would request the main page of my app
(which now it is a static page that says: home).
I basically simulated 100 users requesting the static home page over
and over again at the same time. In less than 3 minutes my application
was down (over quota). I didn't even go through the trouble to have
several hosts hit the servers. Everything was done from my laptop...
It is TOO easy to take down a GAE application.

I understand that the AppEngine is offered as a free, "as-is" service,
but being at the mercy of any kid with a laptop and a DSL line is not
really good for anybody.


Thanks,
Tony

> > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide quoted text -

[scottxu]

Don't know why people pay less attention to timeout. To keep requests
minimal, then much application logic has to be removed from AppEngine.
Httpmr actually increases client-side logic, or you can say "move"
logic
complication from server to client.

AppEngine and httpmr are good at data-collection. However guess Google
wants them to do more than data-collection. Applications based on
large-scale
data potentially could need complicated logic. More features, more
logic.

For DOS, if people have deep knowledge of AppEngine, they may find
some
solutions. However, such solutions could be complicated and cause
timeout.
So, that's why I am concerned by timeout.

Scott

> > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hidequoted text -
>
> > > - Show quoted text -- Hide quoted text -

[Tony Smith]

It's not about paying less attention. About the timeout you can do
something (split the requests etc.).

About the Denial of Service attack you can't really do anything. If
somebody wants you out of the picture all they have to do is simulate
100 users and in 10 minutes you're out.

Are there any plans to block certain IPs from accessing an
application?

Thanks,
Tony

> > > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide...text -

[max7]

I think Google has anti dos techniques already.
ddos is usually done from botnets. Botnets might be used to make fake
clicks on paid ads.
Such kind of fraud is top problem for google.
Most likely google has some secret list with botted PCs around the
world.

Any fraud activity like dos or fake clicks go to antifraud team.
Most likely google will monitor GAE to detect botted PCs.

5.000.000 hit limit is not so critical. It is important to have
unbeaten hosting once it would be paid service.
Some companies pay huge money to akamai and other company for ddos
friendly hosting.
Right now small companies could be kicked out of business by big
companies by ddosing them.

Google is able to detect most botnets in the world and ban these ips.
I hope google will offer first ddos freindly hosting for the regular
price.

That may make botnets less profitable at the end and stop people to
create them.
Unfortunately there is concern that some governments supports botnets :
(

Max

[scottxu]

I don't know about DOS protection. Both timeout and DOS are well known
issues. Which is more urgent may depend on the comparison with
services from other companies.

The only unique issue of DOS on AppEngine is quotas. However, quotas
will be soft limits you can cross, and they are individual
configuration numbers which may be easily adjusted.

Timeout is hard limit which means terminated. Also timeout may relate
to whole AppEngine infrastructure and performance, and can not be
adjusted easily.

Guess Google already has some efforts going on. Just don't know their
strategy and roadmap.
So don't know how to prepare my plans on AppEngine accordingly.

Scott

> > > > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide...-

[Sharp-Developer.Net]

Should not it be raised as a an issue so we could get a feedback from
Google?
--
Alex

[bowman...@gmail.com]

Do you know which quota you hit? I'm wondering if it was the request
quota.

> > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hidequoted text -

[Tony Smith]

It was the high CPU request.
This request used a high amount of CPU, and was roughly 1.1 times over
the average request CPU limit. High CPU requests have a small quota,
and if you exceed this quota, your app will be temporarily disabled.

AGAIN, this is NOT a python page. It is a static webpage accessed by
hundreds of simulated users at the same time. It takes a few minutes
to bring a GAE application down.

Thanks,
Tony


On Sep 15, 2:37 pm, "bowman.jos...@gmail.com"

> > > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide...text -

[iceanfire]

That's a good point. I have pages that don't go over the high-cpu
quota during normal traffic. But when I test it out under a larger
than normal load, I suddenly get high-cpu errors. Like you said, this
problem can be used by rivals to effectively shut you down for close
to 0 cost to them.

> > > > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide...-

[bowman...@gmail.com]

Interesting. Looks like the first problem that should be figured out
is why a static page can break the CPU quota before request quota. Can
you submit an issue on this?

> > > > > > >http://code.google.com/p/pyib/source/browse/trunk/usercontrol.py-Hide...-

[Tony Smith]

I now have a working on a script that will bring down any GAE
application. You fill in the url and some post/get params if you have
them then the script will simulate thousands of users with thousand of
requests.

Right now it takes less than 3 minutes to bring down a GAE app. It's
disappointing that I can't do anything about it... I tried something
with memcache. I save the user's IP in memcache and then block his
requests when it gets too obvious that it's a DOS attack. It's
somewhat better but it still goes over the high CPU quota so I am
stuck again. It's disappointing.

It's disappointing because I know that once you become a little
popular somebody will start flooding your application and the
embarrassment of being down and vulnerable will most likely kill your
adventure.

Tony

[Sharp-Developer.Net]

Tony,

Have your reported this as an issue?

I think a lot of people would star it if someone create such and post
a link here. I definetly would.
--
Alex

[Tony Smith]

Hi,

I created an issue for this request. Please star it if you feel it's
important to you.

http://code.google.com/p/googleappengine/issues/detail?id=718


Thanks,
Tony

On Sep 18, 3:59 pm, "Sharp-Developer.Net"

[Sharp-Developer.Net]

Starred - I think it's gonna be even more impotant when we get paid
service.
--
Alex
http://sharp-developer.net/

[Marzia Niccolai]

Hi,

We've identified an issue that can cause an application to hit one of our short-term quotas after a very sudden spike in traffic, which would prevent it from serving for a short time.  We're currently working on a fix to address this issue and expect to have it out shortly.

On the broader issue of denial-of-service attacks, these are an unfortunate reality in the web world.  While we don't currently offer applications any specific protections against attacks of this nature, this is something we're interested in looking into for the future.  In the near-term, when we begin allowing developers to purchase computing resources beyond our free limits, we will provide a mechanism for reimbursement in the event of a DOS attack.

-Marzia

[Thomas Johansson]

Marzia -

That is great news.

Will this also help with genuine traffic spikes? A simple application
I use to test, which does a single datastore fetch for 5 items that
are a few bytes each, and stores it in memcache for 10 seconds, can go
over quota by a simple ab -c 30 -n 10000. I've tried with a very
gradual ramp up, being very careful not to trigger high cpu spawn
warnings, but regardless, after a while, it will just start spewing
them all over and the app dies. This is with a constant load after
build up, not a spike.

I'm hoping you can confirm that said fix will also solve that issue?

- Thomas

On Sep 24, 6:42 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> We've identified an issue that can cause an application to hit one of our
> short-term quotas after a very sudden spike in traffic, which would prevent
> it from serving for a short time.  We're currently working on a fix to
> address this issue and expect to have it out shortly.
>
> On the broader issue of denial-of-service attacks, these are an unfortunate
> reality in the web world.  While we don't currently offer applications any
> specific protections against attacks of this nature, this is something we're
> interested in looking into for the future.  In the near-term, when we begin
> allowing developers to purchase computing resources beyond our free limits,
> we will provide a mechanism for reimbursement in the event of a DOS attack.
>
> -Marzia
>
> On Mon, Sep 22, 2008 at 3:41 PM, Sharp-Developer.Net <
>

[Tony Smith]

There was a similar issue logged. The one that I created was marked as
duplicate.

Please *STAR* the new issue if you think it's important for you to
have firewall control in your application. Firewall control is the
first step towards resolving the denial of service attack problem.


http://code.google.com/p/googleappengine/issues/detail?id=644


Many thanks to all for your support and thanks Google for looking into
this.

Tony

On Sep 24, 12:42 pm, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>
> We've identified an issue that can cause an application to hit one of our
> short-term quotas after a very sudden spike in traffic, which would prevent
> it from serving for a short time.  We're currently working on a fix to
> address this issue and expect to have it out shortly.
>
> On the broader issue of denial-of-service attacks, these are an unfortunate
> reality in the web world.  While we don't currently offer applications any
> specific protections against attacks of this nature, this is something we're
> interested in looking into for the future.  In the near-term, when we begin
> allowing developers to purchase computing resources beyond our free limits,
> we will provide a mechanism for reimbursement in the event of a DOS attack.
>
> -Marzia
>
> On Mon, Sep 22, 2008 at 3:41 PM, Sharp-Developer.Net <
>