insane number of warmup requests exhaust entire daily quota

[Saurabh Gupta]

Hello Google,

My app was hit with 4.43k warmup requests in a 6 hr period that totally exhausted my daily budget: there was no other significant activity going on at that time.

Can you please help explain what was going on and how I can prevent this in the future?

Thanks,

Saurabh

[Francois Masurel]

Hi Saurabh,

There have been a few posts recently about the problem :

https://groups.google.com/d/topic/google-appengine/Z6EFDyv3MRk/discussion

https://groups.google.com/d/topic/google-appengine/3M164n46MkM/discussion

https://groups.google.com/d/topic/google-appengine/cJcOrOE4JDc/discussion

You might find some useful information in the above threads.

François

PS:

And, as Takashi from Google suggested, you can star the following issues :

https://code.google.com/p/googleappengine/issues/detail?id=7865

https://code.google.com/p/googleappengine/issues/detail?id=5858

[Saurabh Gupta]

Hi Francois,

Thanks for your email; what I'm talking about goes way beyond scheduler issues and routing of requests to dynamic instead of resident instances: I'm talking about 50-60 instances getting started up in response to a barrage of /_ah/warmup requests that drain even a highly funded app completely of quota.

Is this a DOS attack?

Here are some images to convey the situation:

[Jeff Schnitzer]

Look at your logs. Are your instances are dying on startup? Perhaps
your warmup handler is exceeding the 60s deadline, or blowing the heap
limit? That could cause lots of cycling, especially if you have lots
of resident instances.

Do you have any instances successfully serving user requests? Did
they go through /_ah/warmup or were they started by user facing
requests?

Just for kicks, check that the /_ah/warmup requests are actually being
called from Google and not by some third party. I doubt this is an
issue but it's worth making sure.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Cv-LioR0VCAJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Saurabh Gupta]

Hey guys

My startup takes ~3 seconds

this in the web.xml change that fixed it:

  <security-constraint>

    <web-resource-collection>

      <web-resource-name>Google Warmup</web-resource-name>

      <url-pattern>/_ah/warmup</url-pattern>

    </web-resource-collection>

    <auth-constraint>

      <role-name>admin</role-name>

    </auth-constraint>

  </security-constraint>

If the warmup endpoint is not protected by the admin privilege then folks can start hitting it.

Google's documentation doesn't say that this has to be done (implying that access to /_ah/warmup is implicitly protected)

Thanks,

Saurabh