Cache questions

Rob

unread,

Nov 24, 2017, 10:51:05 AM11/24/17

to mod-pagespeed-discuss

I have mod_pagespeed running on CentOS 7 & Apache. Viewing the Console graphs after reaching steady-state (12 hours of running) I get the following:

Cache misses: 30.08%
Cache lookups that were expired: 19.67%
Resources not rewritten because domain wasn't authorized: 6.77%
Resources not rewritten because of restrictive Cache-Control headers: 2.07%
Image rewrite failures: 0.28%

and the rest is 0%...

It's that "cache misses" I'm concerned about. It seems rather high! When running Google's PageSpeed it will typically see a fairly low score due to images that are no optimized, rerunning it a minute later will see a much higher score with all images optimized (because the previous run caused those images to get optimized). I.e. indicative of cache-churn. Web URL is https://www.solacity.com/

This machine has a large amount of RAM, and mod_pagespeed has its disk cache (/var/cache/mod_pagespeed/) mapped to a tmpfs on RAM. Setup is as follows:

Version: 14: on

Filters

ah	Add Head

ai	Add Instrumentation

ij	Canonicalize Javascript library URLs

cw	Collapse Whitespace

cc	Combine Css

ch	Combine Heads

jc	Combine Javascript

gp	Convert Gif to Png

jp	Convert Jpeg to Progressive

jw	Convert Jpeg To Webp

mc	Convert Meta Tags

pj	Convert Png to Jpeg

wa	Convert animated images to WebP

ws	When converting images to WebP, prefer lossless conversions

dd	Dedup Inlined Images

di	Delay Images

ea	Elide Attributes

ec	Cache Extend Css

ei	Cache Extend Images

ep	Cache Extend PDFs

es	Cache Extend Scripts

fc	Fallback Rewrite Css

if	Flatten CSS Imports

hpsr	Hint Preload of Subresources

hw	Flushes html

ci	Inline Css

ii	Inline Images

il	Inline @import to Link

ji	Inline Javascript

idp	Insert DNS Prefetch

id	Insert Image Dimensions

js	Jpeg Subsampling

ll	Lazyload Images

tu	Left Trim Urls

ga	Make Google Analytics Async

cj	Move Css Above Scripts

cm	Move Css To Head

co	Outline Css

pr	Prioritize Critical Css

rj	Recompress Jpeg

rp	Recompress Png

rw	Recompress Webp

rc	Remove Comments

rq	Remove Quotes

ri	Resize Images

rm	Resize Mobile Images

ir	Resize to Rendered Image Dimensions

rx	Responsive Images

rz	Responsive Images Zoom

cf	Rewrite Css

jm	Rewrite External Javascript

jj	Rewrite Inline Javascript

cs	Rewrite Style Attributes

cu	Rewrite Style Attributes With Url

is	Sprite Images

cp	Strip Image Color Profiles

md	Strip Image Meta Data

Options

  AvoidRenamingIntrospectiveJavascript (aris)     False

  CssFlattenMaxBytes (cf)                         204800

  CssImageInlineMaxBytes (cii)                    3072

  CssInlineMaxBytes (ci)                          3072

  CssOutlineMinBytes (co)                         4000

  DefaultSharedMemoryCacheKB (dsmc)               500000

  DisableRewriteOnNoTransform (drnt)              False

  EnableCachePurge (euci)                         True

  EnableRewriting (e)                             1

  FetchHttps (fhs)                                enable

  FileCacheCleanIntervalMs (afcci)                3600000

  FileCacheInodeLimit (afcl)                      1000000

  FileCachePath (afcp)                            /var/cache/mod_pagespeed/

  FileCacheSizeKb (afc)                           5120000

  GoogleFontCssInlineMaxBytes (gfci)              1000

  ImageInlineMaxBytes (ii)                        3072

  ImageLimitOptimizedPercent (ip)                 97

  ImageLimitResizeAreaPercent (ia)                100

  ImageMaxRewritesAtOnce (im)                     -1

  ImageRecompressionQuality (irq)                 80

  ImplicitCacheTtlMs (ict)                        1209600000

  InPlaceResourceOptimization (ipro)              True

  JpegRecompressionQuality (iq)                   80

  JpegRecompressionQualityForSmallScreens (iqss)  70

  JsInlineMaxBytes (ji)                           3072

  JsOutlineMinBytes (jo)                          4000

  LogDir (ald)                                    /var/log/pagespeed

  LRUCacheByteLimit (alcb)                        16384

  LRUCacheKbPerProcess (alcp)                     2048

  MaxCacheableContentLength (rcl)                 33554432

  MaxCombinedCssBytes (xcc)                       -1

  MaxCombinedJsBytes (xcj)                        256000

  ReportUnloadTime (rut)                          True

  RewriteDeadlinePerFlushMs (rdm)                 200

  RewriteLevel (l)                                Pass Through

  SslCertDirectory (assld)                        /etc/letsencrypt/live/solacity.com

  SslCertFile (asslf)                             /etc/letsencrypt/live/solacity.com/cert.pem

  StatisticsLogging (asle)                        True

  SupportNoScriptEnabled (snse)                   False

  WebpRecompressionQuality (iw)                   80

  WebpRecompressionQualityForSmallScreens (iwss)  70

Domain Lawyer

  http://localhost/

  http://www.solacity.com/ Auth

  https://s1.solacity.com/ Auth RewriteDomain:https://www.solacity.com/ OriginDomain:http://localhost/

  https://s2.solacity.com/ Auth RewriteDomain:https://www.solacity.com/ OriginDomain:http://localhost/

  https://s3.solacity.com/ Auth RewriteDomain:https://www.solacity.com/ OriginDomain:http://localhost/

  https://www.solacity.com/ Auth OriginDomain:http://localhost/ Shards:{https://s1.solacity.com/, https://s2.solacity.com/, https://s3.solacity.com/}

Invalidation Timestamp: (none)

I have ModPagespeedFileCacheSizeKb set to 5120000 (ie. 5 GB), while ModPagespeedDefaultSharedMemoryCacheKB is set at 500000. I've tried double those amounts, no difference. The cache isn't taking up nearly that amount of space anyway:

# df

Filesystem          1K-blocks     Used Available Use% Mounted on

/dev/mapper/cl-root  49259008 24698760  24560248  51% /

devtmpfs             32893856        0  32893856   0% /dev

tmpfs                32904836        0  32904836   0% /dev/shm

tmpfs                32904836     8708  32896128   1% /run

tmpfs                32904836        0  32904836   0% /sys/fs/cgroup

tmpfs                32904836        0  32904836   0% /tmp

tmpfs                32904836   510972  32393864   2% /var/cache/mod_pagespeed

tmpfs                32904836        0  32904836   0% /var/tmp

/dev/vda1             1038336   338920    699416  33% /boot

tmpfs                 6580968        0   6580968   0% /run/user/0

# free

total used free shared buff/cache available

Mem: 65809672 9958084 51590696 680616 4260892 54567240

Swap: 2093052 0 2093052

Another oddity is the output from 'caches - physical cache' in mod_pagespeed:

Shared memory metadata cache '/var/cache/mod_pagespeed/' statistics:

Total put operations: 45024

  updating an existing key: 1999

  replace/conflict miss: 3

  simultaneous same-key insert: 0

  dropped since all of associativity set locked: 0

  spinning sleeps performed by writers: 0

Total get operations: 201402

  hits: 193483 (96.07%)

Entries used: 37646 (13.21%)

Blocks used: 566397 (99.39%)

Why is "Blocks used" at 99%??

Any help in resolving this would be very much appreciated!

Thank you!

-Rob-

Otto van der Schaaf

unread,

Nov 25, 2017, 2:54:23 PM11/25/17

to mod-pagesp...@googlegroups.com

I am not sure about the 99% blocks used in the shared memory cache report, though the cache rate seems pretty reasonable at first sight so I'm not sure this is something to worry about. It suspect that this cache/stat will always converge to 100% because of how the underlying design works.

The low hit ratio on the other caches may be caused by a lot of expiring resources. Bumping the original http expiry may help mod-pagespeed keeping the derived optimized versions in cache longer.

Alternatively there is also LoadFromFile, which comes with some nice additional benefits if you are able to use it: https://www.modpagespeed.com/doc/domains#ModPagespeedLoadFromFile

Otto

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/915b0672-d2ef-4f55-a47f-7fa2562567e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rob

unread,

Nov 26, 2017, 2:13:33 PM11/26/17

to mod-pagespeed-discuss

Thanks for the feedback Otto!

I assume the shared memory cache is controlled by DefaultSharedMemoryCacheKB? Is there a way to see how much is actually in use, so I can figure out what cache size makes sense? Similar question for the disk cache, though I suppose for that one I can guestimate from the 'df' output what's being used.

LoadFromFile is already set up for all physical resources. I have:

ModPagespeedLoadFromFile "https://www.solacity.com/docs/" "/var/www/html/solacity/docs/"

ModPagespeedLoadFromFile "http://www.solacity.com/docs/" "/var/www/html/solacity/docs/"

ModPagespeedLoadFromFile "https://www.solacity.com/wp-content/uploads/" "/var/www/html/solacity/wp-content/uploads/"

ModPagespeedLoadFromFile "http://www.solacity.com/wp-content/uploads/" "/var/www/html/solacity/wp-content/uploads/"

This takes care of all the images. The rest of the pages is dynamically generated.

Current HTTP expiry times are set to at least 14 days for anything that's not dynamic, and ModPagespeedImplicitCacheTtlMs =1209600000.

My understanding is that mod_pagespeed doesn't check for changes until 3/4 of the expiry time is reached, so upping that would run the risk of stale pages?

-Rob-

Rob

unread,

Nov 27, 2017, 3:27:43 PM11/27/17

to mod-pagespeed-discuss

Otto, to be more specific, I'm trying to resolve the situation where Pagespeed will give a low score one moment, mainly because images are served unoptimized. Refresh the page (causing optimizing) and rerun Pagespeed and the score will be much higher. Come back an hour later, and the same low score will show again, with the same or other images on the page being served without optimization:

Needs Work
73 / 100
This page is missing some common performance optimizations that may result in a slow user experience. Please investigate the recommendations below.
Possible Optimizations
Optimize images
Properly formatting and compressing images can save many bytes of data.
Optimize the following images to reduce their size by 236.3KiB (51% reduction).

Compressing and resizing https://www.solacity.com/…inlong-Solis-WiFi-monitoring-600x600.png could save 83.3KiB (75% reduction).
Compressing and resizing https://www.solacity.com/…ent/uploads/2015/11/P1060437-600x401.jpg could save 59.3KiB (76% reduction).
Compressing https://www.solacity.com/…ontent/uploads/2016/05/Flower-Meadow.jpg could save 38.5KiB (36% reduction).
Compressing https://www.solacity.com/…2/library-488678_1280-e1490035781133.jpg could save 24.4KiB (36% reduction).
Compressing and resizing https://www.solacity.com/…2016/03/Fronius-IG-PLus-Advanced-3-7.jpg could save 14.2KiB (75% reduction).
Compressing https://www.solacity.com/…s/2016/12/Light-bulbs-e1490035805133.jpg could save 10.1KiB (30% reduction).
Compressing https://www.solacity.com/…r/Homepagerevolution/wood-background.jpg could save 6.7KiB (17% reduction).

Hit refresh on that page and rerun Pagespeed:

Good
94 / 100
Great job! This page applies most performance best practices and should deliver a good user experience.
Possible Optimizations
Leverage browser caching
Reduce server response time

Somehow Mod_Pagespeed is kicking those images out of its cache. However, the default cache time for JPG and WEBP is set to 1 month (ExpiresByType image/jpeg "access plus 1 month"), on top of that, those images are static resources that are accessed through ModPagespeedLoadFromFile, it doesn't even have to go through HTTP.

Cache size seems to be way more than enough, it shouldn't throw out these images due to space constraints.

Why is it loosing these cached images?

Thank you for any leads!

-Rob-

Otto van der Schaaf

unread,

Nov 27, 2017, 3:36:55 PM11/27/17

to mod-pagesp...@googlegroups.com

Thanks for bumping the thread, this fell off my radar.

Perhaps tuning the beacon expiry to your needs may be helpful:

https://www.modpagespeed.com/doc/config_filters#FinderPropertiesCacheExpirationTimeMs

When beacon data expires for a page, some filters may back out or not apply fully until fresh beacons are received, causing a temporary partially optimized state. Fresh beacon data is required for different (classes of) user agents, including the one used by Insights.

Bumping FinderPropertiesCacheExpirationTimeMs may help getting a more steady state optimization state for PageSpeed Insights.

(For actual production traffic this may be less of an issue, as the beacon data should be kept fresh naturally)

Otto

--

You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/7e334826-a66b-4560-adb6-9bcaea0c830c%40googlegroups.com.

Longinos

unread,

Nov 28, 2017, 11:49:21 AM11/28/17

to mod-pagespeed-discuss

...And if you ae using LoadFromFile you can set an implicit ttl for cache with

ModPagespeedImplicitCacheTtlMs

In LoadFromFile, the headers set for http don´t work as far as the file is not served by http.

Rob

unread,

Nov 29, 2017, 7:12:35 AM11/29/17

to mod-pagespeed-discuss

Longinos & Otto thank you!

ImplicitCacheTtlMs was already set to 1209600000 (14 days).

I've upped the beacon time with a few hours as well.

I'm still missing a basic understanding when and why cached images get flushed by ModPagespeed, as that seems to be the main issue (one moment they're optimized, a few hours later they're not and the server sends originals, despite very long TTLs).

I'll continue chipping away at it. Appreciate all the help!

-Rob-

John M

unread,

Feb 17, 2018, 9:11:37 PM2/17/18

to mod-pagespeed-discuss

Hey Rob, I saw that you are using wordpress on your website.

I am too using it for my website but i have a problem every 4-5 hours i got lots of 404 errors.

I was thinking if you could tell me what plugins are you using on your website.

I saw on your page source something with autoptimize as cache.

Waiting for your answer. All the best!

Rob

unread,

Feb 18, 2018, 10:00:13 AM2/18/18

to mod-pagespeed-discuss

Hi John,

We do indeed use Wordpress, with a whole lot of plugins and some custom code (and lots of custom CSS). It's showing 61 active plugins at this time! For the things that could affect mod-pagespeed; I'm running "WP Fastest Cache" to cache Wordpress pages (most of its features are switched off, it's mostly just to cache pages), "Autoptimize" to massage CSS and Javascript for better Pagespeed scores, REDIS with "WP Redis" for database caching, and "Lazy Load" to defer loading of images.

Mod-pagespeed itself is set up to do just about every type of optimization it provides. CSS & Javascript minifying/optimizing, image optimization, sharding of static resources to Amazon's Cloudfront servers, and caching of course. I have it about 85% of what I would like, some things are resistant to working (at least in the way I would expect them), there are errors in the mod-pagespeed log that I can't seem to fix (not affecting the site though). It's a work in progress...

The server also uses PHP's ZEND for caching PHP code. In all it is a pretty complex interplay of various mechanisms with lots of possibilities of unexpected interactions and problems. Our setup is very stable as it is now, and I'm pretty careful to test changes before doing more changes.

My advice for your particular case, if mod-pagespeed seems at the root of the 404s, is to switch off most options in there and start with just the benign basics. Let it run a day, see how things go, and slowly add. Then when things do go wrong plow through all the logs: the mod-pagespeed log, the PHP log file, HTTP server regular logs and error logs. Somewhere in there it should give you a clue why things are going wrong. In case of 404s there should be URLs (generated by mod-pagespeed I assume?) that don't exist or no longer exist. What resources are they for? What are the settings affecting those resources? What does the mod-pagespeed log say when it optimized those resources?

At its basic, you have to get your site to be stable before even trying to add mod-pagespeed to the mix. Wordpress with all its plugins really is a house-of-cards, it's somewhat of a miracle (in my opinion) that it works as well as it does. Just too much that interacts with each other in weird and unexpected ways...

Good luck!

-Rob-

Reply all

Reply to author

Forward