Downstream caching

[Carlos Mendes]

Hi,

I'm using downstream caching with Nginx proxy_cache.

Is there a way to configure the max-age for the HTML pages?

Currently is being set to:

"max-age=3589, public"

Thanks in advance

[Otto van der Schaaf]

The cache control that the html pages are served with probably look the same as without ngx_pagespeed ?

If so, your application controls those directives.

Otto

[Centmin Mod George]

doesn't downstream official docs for proxy_cache override that so html plaintext gets no cache headers

  # Define a mapping used to mark HTML as uncacheable.

  map $upstream_http_content_type $new_cache_control_header_val {

    default $upstream_http_cache_control;

    "~*text/html" "no-cache, max-age=0";

  }

is there a way to proxy cache/cache just the .html files with downstream caching ?

[Otto van der Schaaf]

On Monday, February 22, 2016 at 2:02:52 PM UTC+1, Centmin Mod George wrote:

doesn't downstream official docs for proxy_cache override that so html plaintext gets no cache headers

  # Define a mapping used to mark HTML as uncacheable.

  map $upstream_http_content_type $new_cache_control_header_val {

    default $upstream_http_cache_control;

    "~*text/html" "no-cache, max-age=0";

  }

I was assuming that Carlos was referring to the Cache-Control: emitted by the pagespeed-enabled origin which is to be received by the caching proxy running in front. The caching proxy should indeed forward "no-cache, max-age=0" according to the docs.

 

is there a way to proxy cache/cache just the .html files with downstream caching ?

I think you can achieve that by setting $bypass_cache when the response content-type does not look like html. 

Why do you want that if I may ask?

Otto

 

[Centmin Mod George]

sorry need to rephrase that question is there a way for downstream caching to include .html files in cached response served to visitors rather than exclude with no-cache, max-age=0

[Otto van der Schaaf]

I think that when you would run PageSpeed in OptimizeForBandwidth mode it would be ok to pass on the original cache control headers

for the html. 

However, things get a lot trickier when html filters are enabled that write out .pagespeed. urls and even more so when there are differences in the optimized html

that vary per user-agent. There are also filters that sometimes need to write out beaconing javascript in the html, and there's also the fact that you may

sometimes want to change configuration which may have consequences too as with caching there would be browsers having a cached version of html written

by a configuration that no longer matches the one on the pagespeed-enabled origin. Lastly, you won't know which responses are fully optimized at the caching proxy

you control and serving partially optimized responses seems like a bad idea to me.

So in short, my 2p is you would have to look in-detail at what each enabled filter does when considering serving html as cacheable, 

and that you would have to be very careful when attempting to serve optimized html as cacheable to the outside world.

Otto

[Joshua Marantz]

Have you looked at this?

https://developers.google.com/speed/pagespeed/module/downstream-caching

Generally we deal with partial optimizations by sending purge requests to the downstream cache. 

UA-specific optimizations can be dealt with by fragmenting the downstream cache by a UA-class that's computed (say) in the varnish configuration file.  However, we've changed our recommendation to simply disable UA-specific optimizations when you want to use a downstream cache for HTML.  Since we can now transcode to webp in-place with a Vary:Accept response, it's perhaps less important to make the HTML UA-specific.

-Josh

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Centmin Mod George]

thanks Otto yeah i can understand how tricky it is 

but leaves the dilemma of how you scale for html files or php if you can't put a caching layer in front of it as the downstream cache as outlined at https://developers.google.com/speed/pagespeed/module/downstream-caching which i have implemented doesn't help for scaling for .html or php files and particularly php where the brunt of the work and usage load can come from

[Centmin Mod George]

Thanks Josh yes https://developers.google.com/speed/pagespeed/module/downstream-caching is what I have implemented but also did per user agent implementation from ngx_pagespeed example template too

works well for static files but .html and .php is left without a caching layer i.e. nginx fastcgi_cache of php files or for my wordpress sites using redis caching at nginx level - I have to disable ngx_pagespeed as the no-cache header won't cache the html or php.

[Otto van der Schaaf]

On Thu, Feb 25, 2016 at 2:32 AM, Centmin Mod George <ng...@centminmod.com> wrote:

I have to disable ngx_pagespeed as the no-cache header won't cache the html or php.

I'm confused: the goal with downstream caching is that the proxy in front of the webserver receives cachable html. This allows it to store these responses in cache.

When a browser requests a page from the proxy, the proxy will pull the response from its cache (assuming it has a valid cached copy).

But either way, just before the proxy puts any html response on the wire, it will modify cachability for the response so browsers (and shared caches) will not cache it.

From what you are saying it sounds like the proxy is not caching html at all? Could you post or mail the proxy configuration you ended up with?

 

On fastcgi_cache and redis caching: can you clarify how you observe ngx_pagespeed to render those caching layers ineffective?

Otto

 

[Centmin Mod George]

Doing siege, blitz.io and loader.io type load tests on the .html pages the server load is much higher even with proxy_cache based downstream caching. With regular non-ngx_pagespeed proxy_cache for .html pages, the cpu load is negligible when .html page is cached. The test was done against the downstream proxy_cached site at http://centminmod.com

proxy_cache status for html page does show a Hit (X-Pcache: Hit) but cpu load is still way higher under stress than what i normally observe for non-ngx_pagespeed proxy_cache html pages

1. HTTP/1.1 200 OK Date: Fri, 26 Feb 2016 05:06:01 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block X-Content-Type-Options: nosniff Link: <http://centminmod.com/>; rel="canonical" Content-Encoding: gzip Server: nginx centminmod X-Powered-By: centminmod Cache-Control: no-cache, max-age=0 PS-CapabilityList: ll,ii,dj,jw,ws: X-Pcache: HIT

my config is a multitude of include files make it easier to manage instead everything in one file.

[Otto van der Schaaf]

Re: Date header changing on proxy cache hits

I don't think that is related to ngx_pagespeed or downstream caching.

The nginx docs state that nginx does not pass the Date: header from the origin by default.

After reading the RFC's and https://www.mnot.net/blog/2011/07/11/what_proxies_must_do, I think that is questionable behavior,

but perhaps there is a good reason why nginx defaults to this, I don't know.

You can override that behavior using proxy_pass_header if you want.

Re: X-Pcache: Hit load testing vs CPU-load

The response headers you posted specify "Transfer-Encoding: chunked".

I think that is indicative of the cache hits not being delivered via the optimized path in nginx. 

I would have expected a Content-Length:xxx header.

Is there a filter postprocessing the response bodies of cached proxy responses? That may explain

the higher cpu-usage during stress tests that you observed.

To figure that out, having a look at error.log with the log level set to debug may be clarifying.

Otto

On Fri, Feb 26, 2016 at 6:24 AM, Centmin Mod George <ng...@centminmod.com> wrote:

if you do a single repeated curl header checks against http://centminmod.com/ you will notice proxy_cache status says hit but the date value changes each curl request. Due to cache control = no-cache from the documented downstream cache setup for proxy_cache ?

--

[Jeff Kaufman]

Here's one thing to try:

* Fetch a .html page on your site with your downstream caching setup enabled.
* Fetch the same .html page again.
* Look in your access log for the origin server (running pagespeed):
do you see one request or two?

With downstream caching, what you should see is:

* Two requests to the proxy cache server at times A and B.
* One request to the origin server at time A.

If you see something else that's somewhere to go from.

[Centmin Mod George]

thanks Otto and Jeff, updating to 1.10.33.6 beta and doing some more tests to see