I've noticed for when Twitterbot crawls my app on GAE, the response does not appear to be gzipped (as seen by the response bytes size in GAE logs). I've tested this with other apps deployed on the *.
appspot.com, for example
https://ga-dev-tools.appspot.com/.
To illustrate, I'm using a test user agent "Twitterbot/9.0", although the actual Twitter user agent is "Twitterbot/1.0".
# Test case 1: With a generic Mozilla useragent Mozilla/9.0 + gzip headers, response returned is gzipped
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Encoding: gzip
Vary: Accept-Encoding
Date: Mon, 29 Jun 2015 10:11:35 GMT
Server: Google Frontend
Alternate-Protocol: 443:quic,p=1
Transfer-Encoding: chunked
# Test case 2: With a Twitterbot useragent Twitterbot/9.0 + gzip headers, response returned is not gzipped
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Date: Mon, 29 Jun 2015 10:12:06 GMT
Server: Google Frontend
Content-Length: 7956
Alternate-Protocol: 443:quic,p=1
# Test case 3: With a Twitterbot useragent Twitterbot/9.0 + no other headers, response returned is not gzipped
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Date: Mon, 29 Jun 2015 10:13:17 GMT
Server: Google Frontend
Content-Length: 7956
Alternate-Protocol: 443:quic,p=1
You will noticed that GAE is returning identical responses for test #2 (Twitterbot) and #3 (uncompressed request). This is unexpected and rather puzzling. Any idea why?