inconsistent Pagespeed Insights score, Images not compressing, optimization lost very quickly

[Bastian Schwarz]

Hi everyone,

we are currently using mod_pagespeed on a lot of sites and so far, everything worked out pretty well. But recently, MPS started to give me some headaches for a single site.

On http://www.seefeld.com we're having the issue that images are not compressed ... most of the time. I'm using Pagespeed Insights to see what Google complains about, and most of the time, the images are not compressed and MPS sets the Browser Caching to 5 Minutes. Sometimes, the images are compressed, the Cache Headers seem to be set correctly (Insights shows a 85-90 Score), and 2 Minutes later when checking again, everything seems to be reset and the score is down to the low 30-40s

The same setup and configuration works for other site on the same server, e.g. https://developers.google.com/speed/pagespeed/insights/?url=www.foehr.de&tab=mobile

What irritates me:

• Sometimes it works, sometimes it doesn't
• other pages on the same server working constantly
• All progress seems to be dropped after a short amount of time (within 1-2 Minutes)

Our Configuration and Server setup:

<IfModule pagespeed_module>

  ModPagespeed off #MPS is activated per vHost

  ModPagespeedDisallow "*/admin/*"

  ModPagespeedDisallow "*/imxsearch/*"

  ModPagespeedDisallow "*/imxtools2/*"

  ModPagespeedDisallow "*/imxplatformj/*"

  ModPagespeedDisallow "*/imxeventmanager2/*"

  ModPagespeedDisallow "*/vds_snowreport/*"

  ModPagespeedDownstreamCacheRebeaconingKey ***

  ModPagespeedEnableFilters prioritize_critical_css,move_css_above_scripts,inline_preview_images,fallback_rewrite_css_urls

  ModPagespeedEnableFilters elide_attributes,sprite_images,dedup_inlined_images,lazyload_images,inline_google_font_css,rewrite_images,extend_cache

  ModPagespeedMaxCombinedJsBytes 460800

  ModPagespeedUseExperimentalJsMinifier on

  ModPagespeedFileCacheSizeKb          8388608

  ModPagespeedMemcachedServers localhost

  ModPagespeedStatistics on

  ModPagespeedStatisticsLogging on

  ModPagespeedLogDir /var/log/pagespeed

  ModPagespeedEnableCachePurge on

  ModPagespeedFetchHttps enable

  ModPagespeedRespectXForwardedProto on

#Desperation attemps, usually not part of the default config

  ModPagespeedRewriteDeadlinePerFlushMs 5000000

  ModPagespeedRewriteRandomDropPercentage 0

  ModPagespeedImageResolutionLimitBytes 128000000

  ModPagespeedImageMaxRewritesAtOnce 50

  ModPagespeedImplicitCacheTtlMs 31536000000

</IFModule>

• Debian jessie
• Apache 2.4
• Multiple vHosts
• local Varnish and MPS (Varnish currently deactivated for testing, but works pretty good on other servers)

Out page setup: We're using our own framework across all our sites. Our sites are mostly the same when it comes down to the code and apache/http setup (headers etc).

In the logs, I'm seeing a lot of warnings like

[Fri Jul 01 09:28:53.050868 2016] [pagespeed:warn] [pid 25282] [mod_pagespeed 1.11.33.2-0 @25282] Rewrite http://www.seefeld.com/images/7wfcl7lfjlg-/langlaufen-beim-seekirchl-in-seefeld.jpeg failed while f

etching http://www.seefeld.com/images/7wfcl7lfjlg-/xlanglaufen-beim-seekirchl-in-seefeld.jpeg.pagespeed.ic.xVTvY2axPC.jpg

[Fri Jul 01 09:28:53.053894 2016] [pagespeed:warn] [pid 24027] [mod_pagespeed 1.11.33.2-0 @24027] Rewrite http://www.seefeld.com/images/uynag2o1tou-/wandern-im-karwendelgebirge.jpeg failed while fetching h

ttp://www.seefeld.com/images/uynag2o1tou-/xwandern-im-karwendelgebirge.jpeg.pagespeed.ic.4XfI4mzPfO.jpg

[Fri Jul 01 09:28:53.058389 2016] [pagespeed:warn] [pid 23808] [mod_pagespeed 1.11.33.2-0 @23808] Rewrite http://www.seefeld.com/images/_iccqtt7fzg-/wandern-in-der-olympiaregion-seefeld.jpeg failed while f

etching http://www.seefeld.com/images/_iccqtt7fzg-/xwandern-in-der-olympiaregion-seefeld.jpeg.pagespeed.ic.MgfovGk8Z6.jpg

[Fri Jul 01 09:28:53.936902 2016] [pagespeed:warn] [pid 25139] [mod_pagespeed 1.11.33.2-0 @25139] Rewrite http://www.seefeld.com/images/8w!oo!mtmu8-/casino.jpg failed while fetching http://www.seefeld.com/

images/8w!oo!mtmu8-/xcasino.jpg.pagespeed.ic.LH81tWaMAX.jpg

[Fri Jul 01 09:28:53.945111 2016] [pagespeed:warn] [pid 24896] [mod_pagespeed 1.11.33.2-0 @24896] Rewrite http://www.seefeld.com/images/jwvqcxliamw-/schneefest.jpg failed while fetching http://www.seefeld.

com/images/jwvqcxliamw-/xschneefest.jpg.pagespeed.ic.K8QOYqEArC.jpg

[Fri Jul 01 09:28:58.031711 2016] [pagespeed:warn] [pid 25254] [mod_pagespeed 1.11.33.2-0 @25254] Rewrite http://www.seefeld.com/images/kjynuwnc3aa-/iml.jpg failed while fetching http://www.seefeld.com/images/kjynuwnc3aa-/ximl.jpg.pagespeed.ic.HVrwnlO2rM.jpg

[Fri Jul 01 09:28:58.037803 2016] [pagespeed:warn] [pid 25145] [mod_pagespeed 1.11.33.2-0 @25145] Rewrite http://www.seefeld.com/images/7gbi1njxiqc-/markttage.jpg failed while fetching http://www.seefeld.com/images/7gbi1njxiqc-/xmarkttage.jpg.pagespeed.ic.tEvOlPc3SS.jpg

[Fri Jul 01 09:28:58.039108 2016] [pagespeed:warn] [pid 25139] [mod_pagespeed 1.11.33.2-0 @25139] Rewrite http://www.seefeld.com/images/bujan2obz6e-/ghtrail.jpg failed while fetching http://www.seefeld.com/images/bujan2obz6e-/xghtrail.jpg.pagespeed.ic.vFEZuggBTV.jpg

But again, I don't understand why. 

I also looked at the individual images, they seem to be all right in terms of size and HTTP headers. And as I said, they sometimes DO get compressed.

I also thought about server load, but the load isn't too high and the other sites on the server should have the same issue.

I'm officially out of ideas now so any help is greatly appreciated. Please let me know if you need any more information.

Thanks in advance

Bastian

[Maksim Orlovich]

That is pretty weird --- it looks like when it gets request for
resource back, it somehow fails to produce an optimized version,
yet clearly it succeeds when looking at the webpage. Are there
multiple backends involved by any chance?

(ModPagespeedRewriteDeadlinePerFlushMs 5000000 was indeed a sensible
thing to try here, since its default value could
potentially cause your symptoms if things were somehow always
falling out of cache)

A few things that may help understand what's wrong if you have
pagespeed_admin/ handler available:
1. By default it will show a stats page. Filter it by image_, tell to
auto-refresh, and see if any of the image_rewrites_
numbers start going up when the problem happens. That's likely to be
too vague to see what's wrong, but maybe
there'll be something revealing there.

2. Go to caches tab, and paste in the URL that failed to optimized
just now, e.g.:
http://www.seefeld.com/images/kjynuwnc3aa-/ximl.jpg.pagespeed.ic.HVrwnlO2rM.jpg
or the like. That will dump the information we cached on the results
of the optimization attempt.

3. The message history tab may also include some [info] debug
messages, which may be enlightening

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/6386ee8b-7c99-4084-baa7-400b4f970c29%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bastian Schwarz]

Thanks for your reply!

No, there's just a single backend on the same physical machine.

I just had a look at the stats and the logs:

image_rewrite_latency_total_ms 2317124 1

image_rewrite_total_bytes_saved 968313878 1

image_rewrite_total_original_bytes 4769736877 1

image_rewrite_uses 314425 1

image_rewrites 27050 1

image_rewrites_dropped_due_to_load 1953 1

image_rewrites_dropped_intentionally 1000 1

image_rewrites_dropped_mime_type_unknown 240 1

image_rewrites_dropped_nosaving_noresize 760 1

About 2 Minutes later:

image_rewrite_uses 323296 37

image_rewrite_latency_total_ms 2363220 35

image_rewrite_total_bytes_saved 990185935 35

image_rewrite_total_original_bytes 4889598017 35

image_rewrites 27894 35

image_rewrites_dropped_intentionally 1020 12

image_rewrites_dropped_due_to_load 1979 11

image_rewrites_dropped_nosaving_noresize 779 11

image_ongoing_rewrites 0 8

image_rewrites_dropped_mime_type_unknown 241 2

The number of dropped images seems pretty high, though I thought the "desperation config" should at least have eliminated the "due_to_load" factor?

This is definately different to other servers, though the load is lower compared to servers where this problem does not exist.

Cache entries:

Metadata cache key:rname/aj_2y2NkQkV2VZshO8q3DOo/http://www.seefeld.com/images/a1rliagvkna-/weihnachtsmarkt.jpg@@_

cache_ok:false

can_revalidate:false

partitions:

Metadata cache key:rname/ic_2y2NkQkV2VZshO8q3DOo/http://www.seefeld.com/images/azwiciftxje-/gartenfest.jpg@x@a_

cache_ok:false

can_revalidate:false

partitions:

Metadata cache key:rname/aj_2y2NkQkV2VZshO8q3DOo/http://www.seefeld.com/images/jwvqcxliamw-/schneefest.jpg@@_

cache_ok:false

can_revalidate:false

partitions:

cache ok: false doesn't sound too good.

Memcached is running with 24GB and the filecache right now should have 8GB of which 4,6GB are used.

The logs don't show too much image related, there are some resize message which look like the rewrite was successful (they are ipro though, but they should be optimized and cached in the background for further use right?)

[Tue, 05 Jul 2016 07:38:53 GMT] [Info] [29556] ipro: Shrinking image `http://www.seefeld.com/images/ppppvc6bqlu-/sk.jpg' (24861 bytes) to `http://www.seefeld.com/images/ppppvc6bqlu-/xsk.jpg.pagespeed.ic.kUipKU4PPR.jpg' (21615 bytes)

[Tue, 05 Jul 2016 07:38:53 GMT] [Info] [29698] ipro: Shrinking image `http://www.seefeld.com/images/k1gciakoyrw-/flowers-ice.jpg' (38944 bytes) to `http://www.seefeld.com/images/k1gciakoyrw-/xflowers-ice.jpg.pagespeed.ic.6CLbjeB3zQ.jpg' (35249 bytes)

 There are some none ipro too:

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29694] http://www.seefeld.com/en/service/arrival:587: Shrinking image `http://www.seefeld.com/images/tnbodvtagum-/anreise-in-die-olympiaregion-seefeld.jpeg' (54626 bytes) to `http://www.seefeld.com/images/tnbodvtagum-/xanreise-in-die-olympiaregion-seefeld.jpeg.pagespeed.ic.VzJymcF92D.jpg' (46687 bytes)

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29694] http://www.seefeld.com/en/service/arrival:652: Shrinking image `http://www.seefeld.com/images/f2pnwfoj_hy-/anreise-flugzeug.jpeg' (28003 bytes) to `http://www.seefeld.com/images/f2pnwfoj_hy-/xanreise-flugzeug.jpeg.pagespeed.ic.JLyXfh70Q6.jpg' (25877 bytes)

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29694] http://www.seefeld.com/en/service/arrival:667: Shrinking image `http://www.seefeld.com/images/_75onwxtzow-/anreise-zug.jpeg' (32646 bytes) to `http://www.seefeld.com/images/_75onwxtzow-/xanreise-zug.jpeg.pagespeed.ic._6voY6qqJD.jpg' (30076 bytes)

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29694] http://www.seefeld.com/en/service/arrival:682: Shrinking image `http://www.seefeld.com/images/uxppmspcglo-/anreise.jpeg' (32938 bytes) to `http://www.seefeld.com/images/uxppmspcglo-/xanreise.jpeg.pagespeed.ic.0B_6lKapuq.jpg' (30600 bytes)

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29694] http://www.seefeld.com/en/service/arrival:697: Shrinking image `http://www.seefeld.com/images/itpq0tzobno-/anreise-bus.jpeg' (24110 bytes) to `http://www.seefeld.com/images/itpq0tzobno-/xanreise-bus.jpeg.pagespeed.ic.bw5iySFMTj.jpg' (22591 bytes)

[Tue, 05 Jul 2016 07:38:47 GMT] [Info] [29678] http://www.eifelsteig.de/a-eifelsteig-etappe-9:8: Shrinking image `http://www.eifelsteig.de/images/h/8/z/g/t/f/7/v/j/h/i/-/etappe-9-fernblick.jpg' (318877 bytes) to `http://www.eifelsteig.de/images/h/8/z/g/t/f/7/v/j/h/i/-/xetappe-9-fernblick.jpg.pagespeed.ic.XUDl-4CGHo.jpg' (285496 bytes)

What I forgot to mention: When I remove the explicit cache time MPS sets the browser caching to 5 Minutes which I think also means something went wrong since all our resources have a Cache-Control and Expires header.

Today when I first started Pagespeed Insights, the score was again 83, running it 30 Seconds later, the Score dropped to 43 as if the request caused the cache to drop.

Does any of this help? Could it really be due to server load? But the images are resized, so once they are cached, the problem should go away?

Greetings

Bastian

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Otto van der Schaaf]

It looks to me like PSI is currently giving more or less consistent results after taking a couple of samples. Can you confirm that?

One possible explanation for persisting fluctuating results would be if you have originally set short expirations on the images/. Pagespeed will respect the specified cache-TTL and refetch the input images to check if they have changed and need to be re-optimized. I believe it may be possible that a less optimized version of the html gets served during revalidation in some cases. Short Expiry times and relatively little traffic are factors that won't help with a stable PSI score.

You could either set longer cache expiration times on the original images, or consider using LoadFromFile: https://developers.google.com/speed/pagespeed/module/domains#ModPagespeedLoadFromFile to see if that helps stabilizing the PSI reports if you still experience fluctuations.

Setting up LoadFromFile allows pagespeed to detect changes to files immediately and eliminates the need to revalidate via http(s), reducing load on the http fetcher and possibly the system in general by avoiding revalidation. 

Otto

Thanks for your reply!

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/6386ee8b-7c99-4084-baa7-400b4f970c29%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/f7f2169d-4051-4044-8df1-5b5346e2cbf7%40googlegroups.com.

[Bastian Schwarz]

No, it's still inconsistent. It regularly drops back to 40-50. Important side note: The score itself doesn't concern me as much (although the customers ask a lot about it, but as long as I can give valid reasons, it's ok), but the inconsistency that worries me.

The Cache Headers are set to 

1. Cache-Control:
   max-age=2592000, public
   
   
   for all images.
   
   
   I will try LoadByFile to see if it changes anything. But since this involves additional configuration (Domain Mapping) that will be forgotten when additional Domains are added to the project, I'd rather not use it as default config, especially since it works for other projects.

I will try some diffrent things over the next days and report back.

Thanks for your reply!

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/6386ee8b-7c99-4084-baa7-400b4f970c29%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Bastian Schwarz]

I just remembered this post ..

The problems are more or less "solved", a lot of the missing score is due to srcset image that aren't optimized yet which should be fixed with the next release.