Gzip on App Engine? Apparently automatic, but webmaster tools complains

[Danny Tuppeny]

Hi all,

According to the docs, it seems that gzip is handled automatically on
App Engine. However, in the Google Webmaster Tools "Site Performance"
section, it says to me:

Enable gzip compression
Compressing the following resources with gzip could reduce their
transfer size by 73.3 KB:
http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js (36.6
KB)
http://www.google-analytics.com/ga.js (13.7 KB)
http://blog.dantup.com/js/main.js (725 bytes)
http://blog.dantup.com/js/code/prettify.js (8.61 KB)
http://blog.dantup.com/css/prettify.css (161 bytes)
http://blog.dantup.com/css/main.css (2.02 KB)

The bottom four of these are served from App Engine (I can't speak for
the Google stuff at the top). Any ideas why it wouldn't be served
zipped if App Engine supports it, and presumably so does this
Webmaster Tool?

[Wesley Chun (Google)]

yes, App Engine should be compressing, as outlined here:
http://code.google.com/appengine/kb/general.html#compression

most browsers are recognized and compressed data returned... just
confirm your response headers. below are one example of request and
response headers (with FF3 as the client):

REQUEST

User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:
1.9.0.8) Gecko/2009032608 Firefox/3.0.8 GTB6
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate

RESPONSE

Content-Type text/plain
Cache-Control public, max-age=0
Content-Encoding gzip
Date Tue, 19 Jan 2010 23:37:23 GMT
Server Google Frontend
Content-Length 1571

what do your headers show?

thanks,
-- wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"Core Python Programming", Prentice Hall, (c)2007,2001
"Python Fundamentals", Prentice Hall, (c)2009
http://corepython.com

wesley.j.chun :: wesc...@google.com
developer relations :: google app engine

[Danny Tuppeny]

Hi Wesley,

The problem is shown in Google Webmaster Tools, not my browser. In
Webmaster Tools (labs -> site performance) it shows ways that you can
speed up your site. This includes telling you if GZip isn't enabled. I
can't check what headers GoogleBot/Webmaster Tools (?) are sending,
but I imagine if they're checking for GZip, they're sending headers to
say they support it (though exactly what, I can't say).

If it's just a case of App Engine not understanding that GoogleBot
supports GZip properly, then it's not a big issue (since it'll still
be zipped for visitors), but it would be nice to confirm that. Is it
possible you can contact the person/people that worked on the Google
Webmaster Tools Labs / page speed tool to verify? I don't think our
App Engine logs have enough info to diagnose the problem (short of
adding code that checks for GoogleBot and logs the headers, but I'm
sure you can find that out faster working for Google ;-))

Danny

On Jan 20, 3:21 am, "Wesley Chun (Google)" <wesc+...@google.com>
wrote:

> wesley.j.chun :: wesc+...@google.com

[Barry Hunter]

2010/1/21 Danny Tuppeny <da...@tuppeny.com>:

> Hi Wesley,
>
> The problem is shown in Google Webmaster Tools, not my browser. In
> Webmaster Tools (labs -> site performance) it shows ways that you can
> speed up your site. This includes telling you if GZip isn't enabled. I
> can't check what headers GoogleBot/Webmaster Tools (?) are sending,

Use 'View as Googlebot' ? (its also in labs)

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
>
>

[Danny Tuppeny]

2010/1/21 Barry Hunter <barryb...@googlemail.com>

Use 'View as Googlebot' ? (its also in labs)

Good point :) Though it only shows returned headers, not sent.

Returned this:

HTTP/1.1 200 OK
Date: Thu, 21 Jan 2010 17:20:47 GMT
Expires: Tue, 26 Jan 2010 17:20:47 GMT
Cache-Control: public, max-age=432000
Content-Type: application/x-javascript
Server: Google Frontend
X-XSS-Protection: 0
Transfer-Encoding: chunked

And in the GAE logs, it mentions gzip:

66.249.71.74 - - [21/Jan/2010:09:20:47 -0800] "GET /js/main.js HTTP/1.1" 200 0 - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html),gzip(gfe),gzip(gfe)"

Maybe a bug in Webmaster Tools?

[N. Rosencrantz]

Ask engineering here (or me) and our response is that nothing is
automatic in a double sense. The only automatic is the empty set.
Remember if you make it they can break it.

[Wesley C (Google)]

hmmm, it's possible, but i would try the GWT forums to see what they
say. i think App Engine is doing the right thing. :-)

-wesley

--

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"Core Python Programming", Prentice Hall, (c)2007,2001
"Python Fundamentals", Prentice Hall, (c)2009
http://corepython.com

wesley.j.chun :: wesc...@google.com

[Wesley Chun (Google)]

> Maybe a bug in Webmaster Tools?

this is likely as someone else reported the same issue with the
webmaster tools. anyway, i just wanted to let you know that i closed
the issue and provided some links that may be helpful:

http://code.google.com/p/googleappengine/issues/detail?id=2657#c11

--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"Core Python Programming", Prentice Hall, (c)2007,2001
"Python Fundamentals", Prentice Hall, (c)2009
   http://corepython.com

wesley.j.chun :: wesc+...@google.com

[Danny Tuppeny]

Hi Wesley,

I had a reply from the Webmaster Tools forum saying that there is an issue with many Google services not sending Googlebot zipped content. I suspect, as you say, that the headers it sends aren't recognised by App Engine (which I suspect are the same rules used by Analytics and other services showing the same problem). It sounds like a bug (though which side should fix it, I don't know), though it's only with that reporting, and it seems gzip is working for normal users :-)

Danny

2010/1/22 Wesley Chun (Google) <wesc...@google.com>