Google Groups

Google Bot Is Still the enemy...

Brandon Wirtz Jan 9, 2012 1:24 AM
Posted in group: Google App Engine

Those Are F4 Instances.  The app averages 2.1 instance hours / hour at normal levels.  We usually see Google bot spike us to 6 for 15 minutes. I believe this was 35 F4’s  (those of you who run free apps, this is 10 days of free quota in 15 minutes) (and currently we are getting hit like this every 30 hours)


F4’s served that pages 2.5x faster on average (mainly because with more memory we get a lot more cache hits)  Faster pages meant faster bot crawl. At peak we were serving just shy of 100 Google Bot requests per second.


We really appreciate that the Crawl team helps us prove that we could serve 8.6M users a day with our product, (and against unique page requests no less)  but it would be nice if they could do it once a month rather than once a day.