Cache-Control header - conflicting information from mod_pagespeed docs

[Thom Dunaway]

Hi,

I have noticed that my Pagespeed Admin>Console has a high percentage of "Resources not rewritten because of restrictive Cache-Control headers". The docs for this issue is the following:

• Resources not rewritten because of restrictive Cache-Control headers: Resources could not be rewritten because they had restrictive Cache-Control headers explicitly set (for example: Cache-Control: private, Cache-Control: max-age=0 or Cache-Control: no-transform). Remedy: Reconfigure your server to serve these resources with public caching headers (or no Cache-Control headers at all). For example, Header set Cache-Control "max-age=600" in Apache config.

 

I spent most of yesterday trying to figure out why my headers were being sent as "no-cache". After checking Apache, PHP and Codeigniter, I figured it was mod_pagespeed that was overwriting my "public " headers. This page says.

Page: https://developers.google.com/speed/pagespeed/module/install

By default, mod_pagespeed serves all HTML with Cache-Control: no-cache, max-age=0 because the transformations made to the page may not be cacheable for extended periods of time.

I can tell mod_pagespeed to leave the HTML caching headers, but this is not recommended. ModPagespeedModifyCachingHeaders off

Note: We do not suggest you turn this option off. It breaks mod_pagespeed's caching assumptions and can lead to unoptimized HTML being served from a proxy caches set up in front of the server. If you do turn it off, we suggest that you do not set long caching headers to HTML or users may receive stale or unoptimized content.

So this leaves me confused. Why is this an issue in the Console with a recommendation to set the header to something besides "no-cache" BUT the module itself overrides the header and changes it to "no-cache" and recommends not the change it.

Can someone please explain this to me and what is the actual solution?

Thom

[Joshua Marantz]

The confusion here is the different ways mod_pagespeed treats HTML and resources (CSS, JS, Images).

For HTML, independent of the incoming cache-control headers, MPS by default rewrites the cache-control headers to be more restrictive.  This is because (a) MPS changes (depending on options) the HTML to be user-agent specific and (b) MPS doesn't usually finish optimizing a page until a few seconds after it is visited, so we need to have downstream caches avoid capturing the not-yet-optimized output for very long.  See https://developers.google.com/speed/pagespeed/module/downstream-caching for more details.

For resources, mod_pagespeed optimizes them in the background so that the cost of (e.g.) image rewrites is amortized over all the uses of that image over time.  Moreover, the image rewrites typically happen in the background so that they don't delay the first HTML request, punishing the first user.  So if we can't re-use that optimization because the headers say we can't publicly cache an image or keep it for more than (say) 1 minute, then we don't bother.

For HTML, however, mod_pagespeed doesn't cache any rewrites, but does them on the fly for every request (unless you set up downstream caching).  So MPS doesn't care what the HTML cache headers are coming into the module.

Does that make sense?

-Josh

[Thom Dunaway]

Thanks for the reply.

So, you are saying that only HTML pages are having their cache control headers rewritten? So my other resources are keeping the expires that I set in Apache? I have done some tests on CSS and some images and find this to be true. BUT, why do my "Resources not rewritten because of restrictive Cache-Control headers" and "cache misses" in the console climb to 100%. I have tons to resources besides my HTML page.

I also use memcached and notice that every few days this crashes. PID found but not running. Is this messing up MPS? Any suggestions what this cause is or how to troubleshoot?

Thom

[Joshua Marantz]

Not exactly: mod_pagespeed makes your HTML caching more restrictive (unless you have downstream caching enabled), but it tries to increase the TTL of your resources (with CoreFilters).

It makes your resources cached for a year, which it can do while still honoring your origin TTL by renaming the URL to include the md5 sum of the content.  It will only do this if you have more than a minute of TTL set at your origin.  What pagespeed options are you using?  Can you paste your config?  What Apache snippet are you using to set your origin TTL for resources?  Could you supply the URL to your site?

If memcached crashes, that won't be a functional problem for mod_pagespeed, but it will force it to start over in terms of optimizations.  I've never seen memcached crash; which version are you using?

-Josh

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/110a4bc5-39c9-4109-80fd-6474a6437960%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Thom Dunaway]

Hi,

Here is my MPS config with comments removed

<IfModule !mod_version.c>

  LoadModule version_module /usr/lib64/httpd/modules/mod_version.so

</IfModule>

<IfVersion < 2.4>

  LoadModule pagespeed_module /usr/lib64/httpd/modules/mod_pagespeed.so

</IfVersion>

<IfVersion >= 2.4.2>

  LoadModule pagespeed_module /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

</IfVersion>

<IfModule !mod_deflate.c>

 LoadModule deflate_module /usr/lib64/httpd/modules/mod_deflate.so

</IfModule>

<IfModule pagespeed_module>

    ModPagespeed on

    ModPagespeedInheritVHostConfig on

    AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html

    ModPagespeedFileCachePath            "/var/cache/mod_pagespeed/"

    ModPagespeedLogDir "/var/log/pagespeed"

    ModPagespeedSslCertDirectory "/etc/httpd/conf/ssl.crt"

    ModPagespeedSslCertFile /etc/httpd/conf/ssl.crt/my.crt 

    ModPagespeedMemcachedServers localhost:11211

    ModPagespeedRewriteLevel CoreFilters

ModPagespeedFetchHttps enable

ModPagespeedDomain https://www.wheretostay.com

ModPagespeedDomain https://static.wheretostay.com

ModPagespeedMapOriginDomain http://localhost https://www.wheretostay.com

ModPagespeedMapOriginDomain http://127.0.0.1 https://static.wheretostay.com,https://10.186.38.98

    ModPagespeedFileCacheSizeKb          1024000

    ModPagespeedFileCacheCleanIntervalMs 3600000

    ModPagespeedFileCacheInodeLimit        5000000

ModPagespeedLoadFromFile "https://static.wheretostay.com/prop_photos/" "/web/public_html/prop_p

hotos/"

ModPagespeedLoadFromFile "https://static.wheretostay.com/images/" "/web/public_html/images/"

ModPagespeedLoadFromFile "https://static.wheretostay.com/photos/" "/web/public_html/photos/"

ModPagespeedLoadFromFile "https://static.wheretostay.com/lib/" "/web/public_html/lib/"

ModPagespeedLoadFromFile "https://10.186.38.98/prop_photos/" "/web/public_html/prop_photos/"

ModPagespeedLoadFromFile "https://10.186.38.98/images/" "/web/public_html/images/"

ModPagespeedLoadFromFile "https://10.186.38.98/photos/" "/web/public_html/photos/"

ModPagespeedLoadFromFile "https://10.186.38.98/lib/" "/web/public_html/lib/"

    ModPagespeedStatisticsLogging on

    ModPagespeedMessageBufferSize 1000000

In http.conf I have this:

    ExpiresActive On

    ExpiresByType image/gif "access plus 2 weeks"

    ExpiresByType image/jpg "access plus 2 weeks"

    ExpiresByType image/jpeg "access plus 2 weeks"

    ExpiresByType image/png "access plus 2 weeks"

    ExpiresByType image/x-icon "access plus 2 weeks"

    ExpiresByType image/ico "access plus 2 weeks"

    ExpiresByType application/javascript A86400

    ExpiresByType application/x-javascript A86400

    ExpiresByType text/javascript A86400

    ExpiresByType text/css A86400

    ExpiresByType text/html A86400

    ExpiresDefault A86400

https://www.wheretostay.com

My memcached is 1.4.15. Every so often I look at the phpMemcachedAdmin and it shows nothing so I check the server and PID found but not running.

Thanks for your help!!!

Thom

[Thom Dunaway]

Did my config reveal anything? I also noticed that I am getting lots of these errors:

[Mon Oct 19 14:34:19 2015] [error] [mod_pagespeed 1.9.32.4-7251 @16180] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key where.com/https://www.where.com/images/maps/down.png, value-size 1147

[Mon Oct 19 14:34:19 2015] [error] [mod_pagespeed 1.9.32.4-7251 @16343] AprMemCache::Get error: Connection refused (111) on key rname/aj_PAiZn-u5BeKs53Kg7zpB/https://www.where.com/images/maps/map_factsheet_info.png@@_

[Mon Oct 19 14:34:19 2015] [error] [mod_pagespeed 1.9.32.4-7251 @16403] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key where.com/https://www.where.com/images/maps/expandIcon.png, value-size 1765

[Jeff Kaufman]

Those errors are consistent with your earlier problem of memcached not
being stable. PageSpeed needs a cache to perform well, and if
memcached is often not running PageSpeed won't be able to do most of
it's optimizations. You could set up memcached as a daemon and have
your OS restart it if it fails, or you could switch PageSpeed to use a
file cache.

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/a4bdade4-9681-4c08-9a76-54895f9ab84e%40googlegroups.com.

[Joshua Marantz]

Underscoring this point: we do not see memcached crash often.  Is PageSpeed the only memcached client you have?  If you have a recent memcached version and it's crashing, I wonder whether there are memory problems on your hardware.

And just to make sure we've closed out the question that started this thread: are you all set now with understanding the cache-control used by PageSpeed on HTML and resources?  Now we are just trying to figure out why your memcached installation crashes, right?

-Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUv-MfdcV_9mFz2B8rkgyLB6%3DDunDpUiouSxo%2BiwRzk-UA%40mail.gmail.com.

[Thom Dunaway]

This seems to be the issue. I found it not running again this morning and checked logs and found this:

messages-20151018:Oct 15 19:24:37 www kernel: [36629892.390698] [ 7250]   498  7250   620363   526610    1050        0             0 memcached

messages-20151018:Oct 15 19:24:37 www kernel: [36629892.390977] Out of memory: Kill process 7250 (memcached) score 549 or sacrifice child

messages-20151018:Oct 15 19:24:37 www kernel: [36629892.390983] Killed process 7250 (memcached) total-vm:2481452kB, anon-rss:2106440kB, file-rss:0kB

Have been trying to research this morning. I have 3.75 RAM on my AWS instance and memcached is set to use 2. I'm a little at a loss on what to do to resolve this.

Thom

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

I confess I don't note how to interpret the display you pasted, but it seems like it night be worth seeing if memcached would be more stable at 1gig on your hardware (I assume you are talking about gigs).

This might be a good question to bring over to memcached mailing lists as well.

Josh

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/a4bdade4-9681-4c08-9a76-54895f9ab84e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/dfb2ab17-6c8e-4ed0-a21f-da8217aaeb94%40googlegroups.com.