Google Frontend inconsistent caching

[Matt Mohebbi]

I have a nodejs application I deploy to Appengine Flexible Environment via a docker custom runtime. When I deploy to one project (iodine-test), I get the great caching features of the Google Frontend:

air:/tmp/plum/deploy-20160727-124350% curl -w "@/Users/matt/bin/curl-time-format.txt" -o /dev/null -s "http://iodine-test.appspot.com/drug/zoloft" -v

*   Trying 216.58.194.209...

* Connected to iodine-test.appspot.com (216.58.194.209) port 80 (#0)

> GET /drug/zoloft HTTP/1.1

> Host: iodine-test.appspot.com

> User-Agent: curl/7.43.0

> Accept: */*

> 

< HTTP/1.1 200 OK

< Content-Type: text/html; charset=utf-8

< ETag: W/"30a48-XHteNhi+a5i5072zCcVDyg"

< Vary: Accept-Encoding

< Via: irbk129:14580

< X-Cloud-Trace-Context: 02917deaade7c823eed970361cd13ded;o=1

< Date: Mon, 01 Aug 2016 15:15:49 GMT

< Server: Google Frontend

< Content-Length: 199240

< Cache-Control: public, max-age=604800

< Age: 7

< 

{ [6754 bytes data]

* Connection #0 to host iodine-test.appspot.com left intact

    time_namelookup:  0.004

       time_connect:  0.011

    time_appconnect:  0.000

   time_pretransfer:  0.011

      time_redirect:  0.000

 time_starttransfer:  0.023

                    ----------

         time_total:  0.048

However, when I deploy this same application to a different project (iodine-web-prod), caching doesn't occur on *any* of the requests, regardless of how often I request the same resource. 

air:/tmp/plum/deploy-20160727-124350% curl -w "@/Users/matt/bin/curl-time-format.txt" -o /dev/null -s "http://iodine-web-prod.appspot.com/drug/zoloft" -v

*   Trying 216.58.195.241...

* Connected to iodine-web-prod.appspot.com (216.58.195.241) port 80 (#0)

> GET /drug/zoloft HTTP/1.1

> Host: iodine-web-prod.appspot.com

> User-Agent: curl/7.43.0

> Accept: */*

> 

< HTTP/1.1 200 OK

< Content-Type: text/html; charset=utf-8

< Cache-Control: public, max-age=604800

< ETag: W/"30a48-XHteNhi+a5i5072zCcVDyg"

< Vary: Accept-Encoding

< Via: iwer4:25789

< X-Cloud-Trace-Context: fd82e108c58b92e3114028fdfcd22105;o=1

< Date: Mon, 01 Aug 2016 15:16:46 GMT

< Server: Google Frontend

< Content-Length: 199240

< 

{ [15256 bytes data]

* Connection #0 to host iodine-web-prod.appspot.com left intact

    time_namelookup:  0.005

       time_connect:  0.010

    time_appconnect:  0.000

   time_pretransfer:  0.010

      time_redirect:  0.000

 time_starttransfer:  1.503

                    ----------

         time_total:  1.536

I've combed through the bits and pieces of unofficial documentation:

https://code.google.com/p/googleappengine/issues/detail?id=2258

https://web.archive.org/web/20110826081218/http://www.kyle-jensen.com/proxy-caching-on-google-appengine

https://groups.google.com/forum/#!topic/google-appengine/6xAV2Q5x8AU/discussion

And also reviewed the documentation for Google Cloud CDN assuming that the policies outlined there are the same as those applied to Appengine. 

https://cloud.google.com/cdn/docs/caching

However, nothing explains why I would be getting this inconsistent behavior between projects. Any ideas?

[Nick (Cloud Platform Support)]

Hey Matt,

This is very interesting, and although I'd almost say it qualifies for a post to the Public Issue Tracker, we can also discuss it here, since it's somewhat broad, and it would be inconvenient to migrate all this info.

Are you making your requests from behind any kind of proxy? This is something which comes to mind. 

I'll see if we can see anything from our end in the meantime.

Cheers!

Nick
Cloud Platform Community Support

[Christian F. Howes]

While we have not tried this with flexible runtimes, on standard GAE we are using cache-control, and pragma headers.  we do not use the vary header.  while not documented the GAE support folks confirmed that google's caching proxy will not cache responses with certain values for the vary header (like custom headers and language headers), so you might try removing that from both systems.

also last i checked the cache was not guaranteed, and it's possible if your second app has low traffic that it's deemed not worth caching?  this area is a black box to me.

cfh

[Matt Mohebbi]

Hi Nick, 

Happy to move this over to the Public Issue tracker if you like. Should I start a new bug or post on the existing one regarding lack of documentation?

The proxy point is an interesting one. We were thinking the same thing (both my home and work use Webpass for internet) but when I try this through the cloud shell, I get the same result.

It is also worth noting that as a test, we built a stack using Google Compute Engine and the HTTP load balancer with the Google Cloud CDN enabled. There we see the same results as the iodine-test appengine instance: consistent caching behavior of the Google Frontend.

This is a really strange bug. Would love to get to the bottom of it as we'd prefer to use Appengine. Let me know if you dig up anything on your end!

Thanks,

Matt

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/s2BQjce16lU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/20ea9416-eb68-46c0-8fb9-7ebb44eabf82%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Matt Mohebbi]

Hi Christian, 

Thanks for the response. Yes, we have read about the Google Frontend being picky about Vary params (I imagine they don't want people blowing through their cache by putting dumb things in there). As a result, we are only sending vary for content type. 

And what's odd is that the same code running in two different appengine projects give different behavior (and this has been consistent now for two weeks!). As I mentioned before, we also pushed the same code to GCE + Google Cloud CDN and it works just fine there too. 

My understanding is that the cache is not guaranteed but what's odd is that both projects are for testing are are hardly used (despite the name, iodine-web-prod is not in use until we can figure out this bug!). 

-matt

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/s2BQjce16lU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2e3caab9-d78f-477a-aec2-c35f2da17fbe%40googlegroups.com.

[Simon Green]

Maybe related but I just noticed that my dashboard reports no cached content being served for the last day or so and the Age header is missing from the responses. Even static content defined in app.yaml isn't being cached (it seems).

This is happening on the same project with content served from both standard and flex services.

[Nick (Cloud Platform Support)]

Hey Matt,

This appears worthy of its own thread. It seems very strange that identical apps on different projects should have different caching behaviour.

Feel free to send the thread link to me at pay...@google.com when you've created it with label "Restrict-View-EditIssue". I'll be happy to help!

As for any other users, you can star the thread for more documentation to receive updates on the progress on that front.

Cheers,

Nick
Cloud Platform Community Support

On Thursday, August 4, 2016 at 3:50:27 PM UTC-4, Matt Mohebbi wrote:

Hi Nick, 

Happy to move this over to the Public Issue tracker if you like. Should I start a new bug or post on the existing one regarding lack of documentation?

The proxy point is an interesting one. We were thinking the same thing (both my home and work use Webpass for internet) but when I try this through the cloud shell, I get the same result.

It is also worth noting that as a test, we built a stack using Google Compute Engine and the HTTP load balancer with the Google Cloud CDN enabled. There we see the same results as the iodine-test appengine instance: consistent caching behavior of the Google Frontend.

This is a really strange bug. Would love to get to the bottom of it as we'd prefer to use Appengine. Let me know if you dig up anything on your end!

Thanks,

Matt

[Nick (Cloud Platform Support)]

Hey Matt,

Is this issue still occurring? Let me know if I can assist in any way!

Regards, 

Nick
Cloud Platform Community Support

On Thursday, August 4, 2016 at 3:56:04 PM UTC-4, Matt Mohebbi wrote:

Hi Christian, 

Thanks for the response. Yes, we have read about the Google Frontend being picky about Vary params (I imagine they don't want people blowing through their cache by putting dumb things in there). As a result, we are only sending vary for content type. 

And what's odd is that the same code running in two different appengine projects give different behavior (and this has been consistent now for two weeks!). As I mentioned before, we also pushed the same code to GCE + Google Cloud CDN and it works just fine there too. 

My understanding is that the cache is not guaranteed but what's odd is that both projects are for testing are are hardly used (despite the name, iodine-web-prod is not in use until we can figure out this bug!). 

-matt

[Matt Mohebbi]

Apologies for the delay. We had a hard deadline so we ended up just going with Google Compute Engine and Google Cloud CDN. We ended up taking down these trees to reduce server costs but for the record, the issue remains unresolved and I saw some other folks talking about it in the public slack group. 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/38777119-4f20-4963-934d-f7b0aaac3602%40googlegroups.com.

[Nick (Cloud Platform Support)]

Hey Matt,

Thanks for updating this thread. We'll continue to look into this issue as much as we can.

In future, if anybody else experiencing this issue comes across this thread, please be sure to file a Public Issue Tracker report so we can get as much information as possible.

Thanks again Matt for notifying us of this behaviour, and feel free to post to the Compute Engine group forum for any advice. We also have a Compute Engine issue tracker, and of course there's Serverfault for all other questions which aren't issue reports or high level general discussion.

Regards,

Nick
Cloud Platform Community Support

On Wednesday, August 17, 2016 at 3:31:13 PM UTC-4, Matt Mohebbi wrote:

Apologies for the delay. We had a hard deadline so we ended up just going with Google Compute Engine and Google Cloud CDN. We ended up taking down these trees to reduce server costs but for the record, the issue remains unresolved and I saw some other folks talking about it in the public slack group. 

[Jeff Deskins]

I am seeing a similar issue between HTTP and HTTPS on a custom domain.  The HTTP version of the page gets cached by the Google Frontend.

curl -sD - -o /dev/null http://football.tvstats.com/games/mobile/wbtv

HTTP/1.1 200 OK

Content-Type: text/html; charset=utf-8

Last-Modified: Fri, 19 Aug 2016 02:43:02 GMT

X-Cloud-Trace-Context: 661b5efe97828853eed520725bfa1ab2;o=5

Date: Fri, 19 Aug 2016 02:43:05 GMT

Server: Google Frontend

Content-Length: 56413

Cache-Control: public, max-age=120

Age: 7

However, when accessing the HTTPS version of the same page, it does not cache - even after several attempts.

curl -sD - -o /dev/null https://football.tvstats.com/games/mobile/wbtv

HTTP/1.1 200 OK

Content-Type: text/html; charset=utf-8

Last-Modified: Fri, 19 Aug 2016 02:42:37 GMT

Cache-Control: public, max-age=120

X-Cloud-Trace-Context: d082ac955b5cacc96e8e0fd83907b292;o=5

Date: Fri, 19 Aug 2016 02:44:32 GMT

Server: Google Frontend

Content-Length: 56413

Do you know if an issue was created?  I can add this info if needed.

Jeff

On Wednesday, August 17, 2016 at 6:26:22 PM UTC-5, Nick (Cloud Platform Support) wrote:

Hey Matt,

Thanks for updating this thread. We'll continue to look into this issue as much as we can.

In future, if anybody else experiencing this issue comes across this thread, please be sure to file a Public Issue Tracker report so we can get as much information as possible.

Thanks again Matt for notifying us of this behaviour, and feel free to post to the Compute Engine group forum for any advice. We also have a Compute Engine issue tracker, and of course there's Serverfault for all other questions which aren't issue reports or high level general discussion.

Regards,

Nick
Cloud Platform Community Support

On Wednesday, August 17, 2016 at 3:31:13 PM UTC-4, Matt Mohebbi wrote:

Apologies for the delay. We had a hard deadline so we ended up just going with Google Compute Engine and Google Cloud CDN. We ended up taking down these trees to reduce server costs but for the record, the issue remains unresolved and I saw some other folks talking about it in the public slack group. 

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2e3caab9-d78f-477a-aec2-c35f2da17fbe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/s2BQjce16lU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Simon Green]

Hi Jeff

I see the same thing. Edge caching doesn't seem to operate when using a custom domain AND HTTPS. Using either the appspot domain or HTTP only makes it work so I'm sure the server is sending the correct headers (I think I've tried every combination). It seems the same on Flex or classic.

I was planning on raising a separate issue for it but haven't got round to documenting all the examples I've tried yet.

- Simon