inconsistent number of webp images

136 views
Skip to first unread message

Ungov

unread,
Mar 17, 2022, 7:25:21 PMMar 17
to mod-pagespeed-discuss
When I look at the source code of the homepage of my domain and CTRL+F "webp" to find how many webp images are on the page, the number is always changing

For example right now there is 31 results for webp images in the source code
Now I just force refresh with CTRL+F5 and the number went up to 41
I refreshed again and now it's 50
After many refreshes I see over 100 webp images
But if I come back after a while it will start from 0 again

Why is this happening? The page receives a lot of visitors, the cache headers are set correctly, and there is plenty of disk space for caching

Additionally, I tried logging in the pagespeed admin section but I always get a 403 Forbidden error using this code:
<Location /pagespeed_admin>
        Order allow,deny
        Allow from localhost
        Allow from *
        SetHandler pagespeed_global_admin
</Location>

Longinos

unread,
Mar 18, 2022, 5:13:37 AMMar 18
to mod-pagespeed-discuss
Hi
Pagespeed by default don´t optimize ALL the images but 8. You need to set
ModPagespeedImageMaxRewritesAtOnce NumImages
The value -1 try to optimize ALL images, but if you have too many this has a cost in time waiting for the optimization.
Well... why the optimization restart after some time?
2 main reasons:
1.- cache-control: max-age value
2.- cache storage too short in size.

For 1, I can´t check the original cache-control header, I see wen the image is not optimized a cache-control: max-age 1 year, but is the original header or is set by Clodflare?
Have you set Cloudflare to respect original headers?
I see in the main request 2 cache-headers, f.e.

For 2, How have you set cache size?

ModPagespeedFileCachePath "/var/cache/pagespeed/"
ModPagespeedFileCacheSizeKb 102400
ModPagespeedFileCacheCleanIntervalMs 3600000
ModPagespeedFileCacheInodeLimit 500000

What values have you here?

Pagespeed admin access is configured wrong:

Allow from localhost only allow from localhost acces
Allow from * This is not a valid statement.
Allow from admit hostname (like localhost or www.myhost.com), ip ( like 123.123.123) or the statement "all"
So for permit acces from any site must be "Allow from all"

Ungov

unread,
Mar 18, 2022, 3:50:56 PMMar 18
to mod-pagespeed-discuss
Hi,

I understand that Pagespeed doesn't optimize all of the images, but after refreshing the page tens of times and getting over 100 webp images in the source code, I would expect that if I refresh the page 1 hour later there will still be at least 100 webp images optimized

Cloudflare is set to respect original headers. Image cache expiry is set to 1 year in the htaccess file.

I have already tried to increase the cache size many times, it is probably overkill as is right now:
ModPagespeedFileCachePath            "/var/cache/mod_pagespeed/"
ModPagespeedFileCacheSizeKb          50000000
ModPagespeedFileCacheCleanIntervalMs 86400000
ModPagespeedFileCacheInodeLimit      5000000
ModPagespeedLoadFromFileCacheTtlMs 2419200000
ModPagespeedDownstreamCachePurgeLocationPrefix http://localhost:8443

Thanks for your help, I am now able to access the admin section. I'll let it publicly accessible for a couple of days if you need to check. I see a lot of errors "Too busy to rewrite image" and "served with only 90% rewriting done"

Longinos

unread,
Mar 19, 2022, 9:07:43 AMMar 19
to mod-pagespeed-discuss
Hi
Why do you have this

ModPagespeedDownstreamCachePurgeLocationPrefix?

Have you a ngnix proxy cache or a varnish cache in front of your site?

These messages are not error but info:
Too busy to rewrite image: self explanatory, the server is busy rewriten other images (rewrite images, conver it to webp,etc,etc is too cpu expensive)
served with only 90% rewriting done well... the resource is not totally optimized when served.

In pagespeed_admin -> statistics:

file_cache_evictions
775620
file_cache_hits 146154
file_cache_inserts 807026

I think this will say the cache is not so big for the number of assets stored:
evictions are deletions to made site to new assets not by expiration.

In pagespeed_admin -> Configuration:
ImagePreserveURLs (ipu) True This makes image url don´t change, some filters are disabled, images not converted to webp..... ModifyCachingHeaders (mch) False This make pagespeed s unable to change the cache headers.

For now I can´t test much more as far as I go out of house...

Ungov

unread,
Mar 19, 2022, 4:15:08 PMMar 19
to mod-pagespeed-discuss
Hello,

I'm not sure about  ModPagespeedDownstreamCachePurgeLocationPrefix. This is an old config. I used nginx cache previously but right now it is disabled, but I still use nginx proxy mode without caching. I tested both with and without "Serve static files directly by nginx" but it didn't change anything https://prnt.sc/hCjFn7DKzKXV

How can I see how much space is used by pagespeed cache? I tried doing "du -sh" on /var/cache/mod_pagespeed/ but it always fail with the error "Aborted (core dumped)" maybe because there are too many files in the folder. For now I raised cache size from 50GB to 100GB but the total disk space used by this domain in Plesk is only 10GB so the cache size should be enough

ImagePreserveURLs doesn't seem to be working because the webp filenames keep changing

Caching is set to 1 year through htaccess, what should I do with ModifyCachingHeaders ?

Longinos

unread,
Mar 20, 2022, 7:56:57 AMMar 20
to mod-pagespeed-discuss
Hi
ModPagespeedDownstreamCachePurgeLocationPrefix is used when a cache (nginx or varnish) is in front of the web server, and in the pagespeed admint there are cache purge request (remember the messages "served with only 90% rewriting done"? these messages come from purge request, delete a cached page when only 90% is rewrited), so I think the proxy cache still here and working.

Should purge "https://www.no-gods-no-masters.com/women-tshirt-zad-partout-D01001235777P0347/" which was served with only 90% rewriting done.[Sun, 20 Mar 2022 11:49:40 GMT] [Info] [1905276] Purge url is http://localhost:8443/women-tshirt-zad-partout-D01001235777P0347/



Seem that in /var/cache/mod_pagespeed/  are stored non only 1 domain but all, in pagespeed_admin I can se request for more that 1 domain.
Pagespeed cache need to be 5x - 7x the size of files (f.e. 1 image can have many "versions" for diferent sizes, devices....)

ImagePreserveURLs is here... is all most strange it not works..... maybe there are diferent config files?

Well... caching headers, how it works (whe you fecht the site directly, no Cloudflare, no cache of any type)

1.- The original cache-control: max-age (the headers used if pagespeed is off) tell pagespeed how much time the optimized resource is stored in their cache.
If this header is st to 600, the optimized resource (these have pagespeed in the url) lives in the pagespeed cahe 600 (5minutes)

2.- If ModifyCachingHeaders is true, then pagespeed changes the cache-control: max-age to 1 year for the optimized resource, if the resource is not fully optimized
then it serves the original resiurce, with the original cache-control and add a s-max-age=10 to tell intermediates cahe (Cloudflare f.e) to not stored the unoptimized resource more that 10 sec.

3.- If you use LoadFromFile (and you do, allmost is in the config) then no http headers are in the resource request because is not http requested but loaded directly from disk.
In this case you must use LoadFromFileCacheTtlMs (you have it in the config set to 28 days) to tell pagespeed how much time store the optimized resource in their cache.
ModPagespeedImageMaxRewritesAtOnce NumImages

You need to increase this parameter, is not set and the default is 8.
Your pages have more than 50 images each, so rewriten 8 each request it takes some hits to have the images rewrited This makes the "too busy..." mesages.
Increase it progressively and take a look at CPU use and time to the page be served.
Try a start of 25, this takes 3-4 hits to get all images rewited, lest than the actual 12-13

Longinos

unread,
Mar 20, 2022, 9:09:36 AMMar 20
to mod-pagespeed-discuss
Hi again

I think some config about the pagespeed config is wrong


file_cache_evictions
1042581
file_cache_hits 19467
file_cache_inserts 989448
file_cache_misses 544898

seems like each item inserted in the config is evicted.
Whitout the whole config, apache, nginx, pagespeed,.... I can´t say what is wrong here

Ungov

unread,
Mar 20, 2022, 8:42:45 PMMar 20
to mod-pagespeed-discuss
Hello,

Yes there are 5 domains storing cache in /var/cache/mod_pagespeed/
I just updated the config to use  /var/cache/mod_pagespeed_ngnm/ for the main domain I'm testing with. So there should be plenty of dedicated space (100GB cache for 10GB of files)

Also added:
ModPagespeedLoadFromFileCacheTtlMs 31556952000 (1 year)
ModPagespeedImageMaxRewritesAtOnce 50

Full pagespeed config: https://pastebin.com/CAnX77py

I have not touched nginx/apache config. I'm not even sure where the config files are located. If you give me the exact path I can copy the config for you. I use CentOS 8.5 with Plesk

/etc/nginx/nginx.conf  ----> https://pastebin.com/kPFKV0JH
/etc/httpd/conf/httpd.conf  ----> https://pastebin.com/TK5nEnK3

Please note that I don't put my pagespeed config in the nginx/apache config file, I put it directly in the "additional directives" section of Plesk Apache & Nginx Settings
like this screenshot: https://ibb.co/2njgtJj

Longinos

unread,
Mar 21, 2022, 6:28:45 AMMar 21
to mod-pagespeed-discuss
Hi
Now you have a new problem:
[Mon, 21 Mar 2022 10:26:13 GMT] [Error] [2531937] Failed to make directory /var/cache/mod_pagespeed_ngnm/v3: Permission denied
Check file owner and permisions on /var/cache/mod_pagespeed_ngnm

Longinos

unread,
Mar 21, 2022, 7:21:13 AMMar 21
to mod-pagespeed-discuss
Hi
I take a look at the config files. Tell me if i´m wrong.

1.- You have apache+pagespeed running in localhost:8443
2.- You have nginx running main domain:443
3.-  apache+pagespeed load files from disk
4.-  nginx serve statics files

I that is rigth I don´t know why pagespeed is working with images as far as images are served by nginx with no pagespeed module.
So I don´t know why but seems some images are served directly from nginx (and no pagesped rewrited) and some images are served by apage and are rewrited by paespeed.
I think you need to deselect "Serve static  file directly from nginx" in the nginx section of plesk.

What about files here /etc/nginx/conf.d/*.conf;? Have you 1 file for each server nginx section?

For the pagespeed config:
Here is the
ModPagespeedImagePreserveURLs on
comment it put off.

About the pagespeed cache size:
There are 2 parameters that control the cache size:
ModPagespeedFileCacheSizeKb  # This control the size in Kb
ModPagespeedFileCacheInodeLimit# Ths control the inode number used.
So cache can be considered "full" used when the size of files are 95Gb OR when the number of files are 5.000.000 even if no 95Gb of space is occuped.
For example you may have 95 files of 1gb each and the cache is full,  or have 5.000.000 files of 1kb each (4Gb) and the cache is full.

Ungov

unread,
Mar 21, 2022, 4:12:34 PMMar 21
to mod-pagespeed-discuss
Ok now I set the same file owner for "mod_pagespeed_ngnm" as "mod_pagespeed"  (apache:apache owner instead of root:root)

I have disabled "Serve static files directly by nginx"
Should I also disable "Smart static files processing" ? The description says "If turned off, Apache will process all requests for static files. nginx will only pass requests and responses without modification. Turn off this option only for troubleshooting."

Here are the files in /etc/nginx/conf.d/

[root@ns508279 cache]# cd /etc/nginx/conf.d/
[root@ns508279 conf.d]# ls
brotli.conf  ssl.conf  zz010_psa_nginx.conf  zz030_ext-monitoring.conf

Do you want me to disable ModPagespeedImagePreserveURLs?

Is there a way to know the current disk space usage of pagespeed cache and the current inode usage? I have changed it to:
ModPagespeedFileCacheInodeLimit      50000000

Ungov

unread,
Mar 21, 2022, 4:22:00 PMMar 21
to mod-pagespeed-discuss
I found how to get the cache size and inode usage

[root@ns508279 /]# cd /var/cache
[root@ns508279 cache]# du -sh *
0       app-info
0       bpf
105M    dnf
0       httpd
0       krb5rcache
36K     ldconfig
0       libX11
1.8M    man
37G     mod_pagespeed
291M    mod_pagespeed_ngnm
0       nginx
0       PackageKit
13M     parallels_installer
0       private
0       yum
[root@ns508279 cache]# ls -id
3773759 .
[root@ns508279 cache]#

5 domains storing cache in /mod_pagespeed/ are using only 37G total and 3773759 inode
So the cache limit is definitely not exceeded

I can also see the size of /mod_pagespeed_ngnm/ is growing so pagespeed seems to be correctly saving the cached files now

Longinos

unread,
Mar 21, 2022, 8:46:49 PMMar 21
to mod-pagespeed-discuss
Hi
Yes, try to unset "Smart static files processing" and see whats happen.
At a first glance nginx don´t need to serve statics files as far as pagespeed is not running here.
I see you restarted about 9:35p.m. CET and until now, 1:01 a.m CET file evictions are 0
Much more images get converted to webp in 2-3 hits.
And yes, disable ModPagespeedImagePreserveURLs
Whats the content of zz010_psa_nginx.conf?

Ungov

unread,
Mar 22, 2022, 2:30:27 AMMar 22
to mod-pagespeed-discuss
Hi,
I disabled Smart Static Files Processing and ModPagespeedImagePreserveURLs

The webp images count on the homepage is rising much faster now. I think ModPagespeedImageMaxRewritesAtOnce helped a lot. I could try raising it higher, it raised a little bit the CPU usage and load average but not significantly https://prnt.sc/vSgiIaGV9M5D
But I would rather prefer to fix the root cause and not the symptoms

[root@ns508279 ~]# cd /etc/nginx/conf.d/
[root@ns508279 conf.d]# nano zz010_psa_nginx.conf
#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.
include /etc/nginx/plesk.conf.d/server.conf;
include /etc/nginx/plesk.conf.d/webmails/*.conf;
include /etc/nginx/plesk.conf.d/vhosts/*.conf;
include /etc/nginx/plesk.conf.d/forwarding/*.conf;
include /etc/nginx/plesk.conf.d/wildcards/*.conf;


Maybe this can be useful?

/var/www/vhosts/system/domain.com/conf/httpd.conf :

/var/www/vhosts/system/domain.com/conf/vhost_ssl.conf :
^ this one is the pagespeed config

Longinos

unread,
Mar 22, 2022, 8:31:04 AMMar 22
to mod-pagespeed-discuss
I still getting some strange issue
Some webp converted images have
content-type: image/jpeg
This is nor normal behaviuor, must be
content-type: image/webp

See these 2 images

Ungov

unread,
Mar 22, 2022, 9:10:53 PMMar 22
to mod-pagespeed-discuss
The content-type bug seems to be due to Cloudflare. I paused Cloudflare and now the image you linked is showing webp content-type. I will leave Cloudflare disabled to continue the investigation.
https://prnt.sc/Ic49DBJYIyMt (on the left is jpg content-type before disabling CF, and on the right is webp content-type after disabling CF)

What can I do to make my config less complex? I didn't edit any files manually. I put all the config in Plesk Web Admin interface. This is a standard Plesk installation.

Longinos

unread,
Mar 23, 2022, 4:24:44 AMMar 23
to mod-pagespeed-discuss
Hi

I think complexity comes from plesk.
As far as I can see you only need a web server without proxy.
The idea of using nginx+apache is not so bad, nginx serves statics files and manage user conections, in both is better than apache.
But using pagespeed introduces changes that these combination is useless if no are a server cluster o load balancers in place.
May be you need to re-tink the whole installation and use only apache or nginx. Pagespeed nginx module need to be compiled with nginx not as easy than apache
but is better serving statics files and managing connections, this not so importan as far as you use CF.

Well... now we are requesting to the original server :
Take a look at the main request (https://www.no-gods-no-masters.com/), there are 2 cache-control headers:
cache-control: no-store, no-cache, must-revalidate, max-age=0, post-check=0, pre-check=0
cache-control: max-age=864000
You need to find from where comes the second header and unset it

Take a look at this file
https://www.no-gods-no-masters.com/images/homepage/trending_refugees_welcome.jpg from the headers look at is not served by apache but from nginx and then get not rewrited, never.
It preserves the original headers as if pagespeed don´t care about this file.

Despite no file evictions in the pagespeed cache, optimizations are lost after a bit of time, maybe not so fast as before, but they lost it.
Other strange thing, despite cache is storing assets, and not evicting, the hits are low, I think
file_cache_hits 78238
file_cache_inserts 816818

About retinking the site, are you able to set a stage site with only apache + pagespeed and a copy of the main domain?


Ungov

unread,
Mar 23, 2022, 5:25:04 PMMar 23
to mod-pagespeed-discuss
Hello,

I found the cause of the double cache-control header, it was in the htaccess:
ExpiresActive On
ExpiresDefault "access plus 10 days"
I removed it and now the cache-control header with max-age=864000 is gone

The website is not ready to upgrade to nginx-only and it seems impossible to completely disable nginx proxy mode with Plesk according to this article: https://support.plesk.com/hc/en-us/articles/360006364914-How-to-make-a-website-be-processed-only-by-Apache-when-nginx-is-enabled-on-a-Plesk-server
So I followed the instructions in the link above and set the domain to use "Dedicated FPM application served by Apache" instead of served by nginx

However, as the article notes:
"After applying this solution, the website will still be shown as served by nginx in a web browser. This is due to the fact that nginx is working as the front-end proxy. The website will be processed by Apache in fact"

Thanks for the support, you are very helpful I appreciate it a lot!

Longinos

unread,
Mar 24, 2022, 4:43:14 AMMar 24
to mod-pagespeed-discuss
Hi
I´m glad to help from a little union

Ungov

unread,
Mar 24, 2022, 5:21:46 AMMar 24
to mod-pagespeed-discuss
However, the issue with the webp images is still not fixed :( Is there anything else I can investigate?

Ungov

unread,
Mar 25, 2022, 5:04:09 PMMar 25
to mod-pagespeed-discuss
If the problem is related to Plesk, then I could contact Plesk and ask for support, but I'm not sure what should I ask them?

Longinos

unread,
Mar 26, 2022, 10:09:38 AMMar 26
to mod-pagespeed-discuss
Hi
No, the problem is not with Plesk.
They do a good decision, serving statics assets from nginx and managing conections with it, and let apache to the background process.
But this a  web "normal" configuration. When you put pagespeed in the equation then things changes.
If you put apache + pagespeed, then statics assets need to be served by apache because are rewrited and these rewrited static contend served from the pagespeed cache.... from apache.
This makes the initial config netx to useless.
Pagespeed need hits on the web server it run t work, so if you put a cache in front then you need a shot time for the cache to mage request go to the backend and make pagespeed wor.
This config ( nginx proxy cache -> apache + pagespeed) makes sense in a very high used web server.
And Plesk make it a plus confuse thing creating a lot of config files, but is how it work. And you cn´t taylar these config files because Plesk overwrite it when you makes changes to the files.
So my tougths in this cases is making the whole proccess step by step:
1.- Create the apache web server and do it work well.
2.- Add the pagespeed module, and try the pagespeed directives snf configurations, see what they do and what the web need.
3.- Add the nginx proxy cache, see how it works and play with the config
4.-  Add CF to the whole thing.
But this a production installation, so changes can have non desired effects.

And this web have a heavy work to do because you have 100 images in each page (or near it)
So, what I think you need to do:
1.- Make sure all statics asset that pagespeed processes have a public cacheable header / cache-contol: max-age=XXXX, public)
the XXX not need to be 1 year because when pagespeed rewrites it then they have the 1 year. Try 2-3 days. 172800 - 259200, If you set it to 1 year this tell pagespeed to store the optimized resources for 1 year in the cache , why do you need to store in cache an asset not accessed by 2-3 days?
2.- Make sure ALL request go to the apache, nginx (seems that in plesk nginx can´t be bypassed) must be a transparent proxy
3.- For some time and as a "experiment" put ModPagespeedImageMaxRewritesAtOnce -1, and see if the times for rewrite all images are acceptable.
4.- For help in 4, put  RewriteDeadlinePerFlushMs 20 (the defaul is 10). This the time pagespeed used to to optimizations in a page before it flush the page to the client
5.- Take a look on how cache is working. Remember you have 2 caches in place SHMcache and filecache. Shmcache is a in-memory thing, so increase the size consumes RAM memory
6.- Consider intrduce redis in the config, this a ram database to store the cache.

Ungov

unread,
Mar 26, 2022, 5:20:17 PMMar 26
to mod-pagespeed-discuss
Hi,

2- what is pagespeed directives snf configurations?
3- nginx cache is still disabled, I will enable it back only after I fixed the issues with pagespeed
4- Cloudflare is disabled but it didn't change anything
5- The double cache-control header was fixed, right now there is only 1 cache-control header. 1 year is fine with me because I want to always serve webp images and I have a lot of disk space I can use. Anyway I don't think it will change anything about the issue with inconsistent number of webp served
6- With ModPagespeedImageMaxRewritesAtOnce -1 I can see 287 webp images on the homepage after 3 page reload. Then if I wait 5 minutes and force refresh again, the webp images go down to zero again. It does help but as mentioned previously it seems like treating symptoms instead of fixing the underlying issue. Webp images will still constantly gets regenerated, which is what I'm trying to avoid/fix
7- Can Redis really help with the issue of webp images being constantly regenerated? Seems like it would only introduce another variable in my config and make it even more complex
8- How do I enable SHMcache? I get this error when adding it to my config through Plesk interface:
[mod_pagespeed 1.13.35.2-0 @706229] 'CreateSharedMemoryMetadataCache' is global and is ignored at this scope AH00526: Syntax error on line 15 of /var/www/vhosts/system/no-gods-no-masters.com/conf/vhost_ssl.conf: <VirtualHost> cannot occur within <VirtualHost> section

Longinos

unread,
Mar 27, 2022, 7:13:54 AMMar 27
to mod-pagespeed-discuss
HI

2- what is pagespeed directives snf configurations?
snf mus be "and", my bad... a mistake

8:- Becacse this directive can´t be set in a <VirtualHost> section. This must be set like this:

ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/pagespeed/" 51200
ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/pagespeed-new/" 51200
<VirtualHost www.example.com:80>
 ModPagespeedFileCachePath "/var/cache/pagespeed/"
</VirtualHost>
 <VirtualHost alt.example.com:80>
ModPagespeedFileCachePath "/var/cache/pagespeed-new/"
 </VirtualHost>
This creates 2 SHMCache and is used in each <VirtualHost>. Yes, is a bit confising, you create a  CreateSharedMemoryMetadataCache but is used with  ModPagespeedFileCachePath)

Let me explain how pagespped cache is structured:
Basically there are 2 cache, metadata and html (no, not the html code but rewrited resources, webp images for example)
The metadata cache info about a resource, for example :
this image is converted in this webp
whit a size 400x400, is read from file, has a size of 8432, can be inlined as " RIFFD\r\000\000WEBPVP8 8......"
These info is stored in the SHMCache.
The webp himself is stored in the file cache.
If the metadata is not here, pagespeed don´t know what webp belongs to whats jpg file, even if the webp is in the file cache, so the optimization process restart to recreate the metadata cahe.
Where the metadata cache is stored? Well.... small piezes bellow 64Kb in the SHMCache (in the RAM memory) and pizes over 64Kb in the file cache (you can see an rname folfer under the
/var/cache/mod_pagespeed_ngnm/)
The most of the metadata are below the 64Kb limit, so are stored in the SHMCache in Ram, when evicted, the optimization process restar despite the optimized resource is in the file cache.

Your case:
You have now 2 files cache, 1 for the main domain and 1 for all others, if I remember:
/var/cache/mod_pagespeed/
/var/cache/mod_pagespeed_ngnm/
rigth?
then you can create a SHMCache for each
ModPagespeedCreateSharedMemoryMetadataCache " /var/cache/mod_pagespeed/ " 51200
ModPagespeedCreateSharedMemoryMetadataCache "  /var/cache/mod_pagespeed_ngnm/ " 51200
and in the
<VirtualHost main-domain>
ModPagespeedFileCachePath " /var/cache/mod_pagespeed_ngnm/"
  </VirtualHost>
<VirtualHost other-domain>
ModPagespeedFileCachePath " /var/cache/mod_pagespeed_ngnm/"
  </VirtualHost>

If you don´t have any ModPagespeedFileCachePath directive set in you config, there is a default one, with a 50Mb size.
You can modify the size with "ModPagespeedDefaultSharedMemoryCacheKB size-in-kb"

When you put redis (or memcached) in place, then you can make ALL the caches get stored in the redis (or memcached9 database in RAM memory

Longinos

unread,
Mar 27, 2022, 7:16:09 AMMar 27
to mod-pagespeed-discuss
Hi
My bad , again

<VirtualHost other-domain>
ModPagespeedFileCachePath " /var/cache/mod_pagespeed/"
  </VirtualHost>

This is the rigth 2nd <VirtualHost>

Ungov

unread,
Mar 27, 2022, 6:24:51 PMMar 27
to mod-pagespeed-discuss
This one took me a while to figure out where to put the  ModPagespeedCreateSharedMemoryMetadataCache directive because I can't put it with the rest of my config in Plesk Additional Directives section
So I tried to google where to put GLOBAL apache directives for Plesk but I get conflicting answers
Currently I put the SharedMemory directive in  /etc/httpd/conf/httpd.conf .... is it correct?


and yes, that is right, I use /var/cache/mod_pagespeed_ngnm/ for the main domain and /var/cache/mod_pagespeed/ for all other domains

The changes didn't fix the issues with the webp images. After some time all webp images are still lost on the homepage

I have also tested turning ModPagespeedImagePreserveURLs back on, but it's still not working

Ungov

unread,
Mar 27, 2022, 6:27:51 PMMar 27
to mod-pagespeed-discuss
As for the ModPagespeedFileCachePath directive, I already have it in Plesk Apache Additional Directives GUI

So I didn't put it in  /etc/httpd/conf/httpd.conf because it's already in /var/www/vhosts/system/no-gods-no-masters.com/conf/vhost_ssl.conf

Longinos

unread,
Mar 28, 2022, 4:02:33 AMMar 28
to mod-pagespeed-discuss
Hi
The http.conf is the main config file for apache, all other config files must be included here to work.
ModPagespeedCreateSharedMemoryMetadataCache must be before all <Vhost> are included so, I think, the rigth place is
in the http.conf before IncludeOptional conf.d/*.conf /Think this line includes the files where <Vhost> are defined)

For now I see that in the main domain it takes more time to lost the webp conversion.
Can you compare how much time take webp to be lost in the main domain and in others domains?

There is a parameter you can try:
ModPagespeedShmMetadataCacheCheckpointIntervalSec 600
This must be in the same site of ModPagespeedCreateSharedMemoryMetadataCache
This creates a disk copy of SHMCache then the optimizations survive server restart.
600 are 5min.

Longinos

unread,
Mar 28, 2022, 7:53:56 AMMar 28
to mod-pagespeed-discuss
Hi

There is another image:
as you can see, is not webp converted but:
Metadata cache key:rname/aj_GtCzyBf1YlXRfF0-5CW8/https://www.no-gods-no-masters.com/images/texturebg.jpg@@a_
cache_ok:true
can_revalidate:false
partitions:partition {
optimizable: true
input { index: 0 type: FILE_BASED last_modified_time_ms: 1596189146000 filename: "/var/www/vhosts/no-gods-no-masters.com/public_html/images/texturebg.jpg" } optimized_image_type: 4
}
debug_message: "Image https://www.no-gods-no-masters.com/images/texturebg.jpg does not appear to need resizing."

There is a webp image in the metadata cache but is not served. Maybe the no rewrite images url is working here?

In other hand this image
cache_ok:true
can_revalidate:false
partitions:partition {
optimizable: false
 input { index: 0 type: FILE_BASED last_modified_time_ms: 1645526592000 filename: "/var/www/vhosts/no-gods-no-masters.com/public_html/images/homepage/trending_refugees_welcome.jpg" } }
other_dependency { type: FILE_BASED last_modified_time_ms: 1645526592000 filename: "/var/www/vhosts/no-gods-no-masters.com/public_html/images/homepage/trending_refugees_welcome.jpg" }
debug_message: "Image https://www.no-gods-no-masters.com/images/homepage/trending_refugees_welcome.jpg does not appear to need resizing."

As you can see no webp version is stored in the metadata cahe.
Why? well.... maybe the webp version is less than 10% in size that the jpg version, in such cases pagespeed don´t rewrite it.

Ungov

unread,
Mar 28, 2022, 5:32:07 PMMar 28
to mod-pagespeed-discuss
I don't see any difference... After only a couple of minutes, webp images are still lost and regenerated :(

I added ModPagespeedShmMetadataCacheCheckpointIntervalSec 600 to my config

I also opened a ticket with Plesk support to confirm where to put directives in the global config and they told me to put it in /etc/httpd/conf.d/pagespeed.conf

Longinos

unread,
Mar 29, 2022, 3:32:28 AMMar 29
to mod-pagespeed-discuss
Yes, webp is lost, but I think in the main menu takes more time than in the others domains.
No matter where the global directives are set, but out of <Vhost> declarations. In some way put it in a *.conf file that is included in the http.conf , but make sense use of pagespeed.conf to maintain all pagespeed directives toghether.

What the use of RAM in the server?
Can you try to increase SHMCache size?
Some like:
ModPagespeedCreateSharedMemoryMetadataCache "  /var/cache/mod_pagespeed_ngnm/ " 102400
My bet in this issue is the metadata cache is small for all the assets it need to store, so evict entries and this makes the lost of webp optimizations.


In other hand I see this:


This make me thinking that apache+pagespeed is running at http://localhost:7080 and then this is rewrited to  https://www.no-gods-no-masters.com/
But why the others domains aren´t?

Ungov

unread,
Mar 29, 2022, 5:53:53 PMMar 29
to mod-pagespeed-discuss
Hello,

We have 64GB ram so we can definitely increase SHMCache. https://prnt.sc/D-50iqikvHdr
I just set it to 1000000 (1GB)
It seems to help, but further testing is required to confirm

I am confused about the last part of your message about the other domains. 

I'm not sure if apache+pagespeed is running at localhost:7086 because I don't see the website when I log in to the server and do a CURL
[root@ns508279 ~]# curl http://localhost:7080
<!doctype html>
<html lang="en">
<body>
    You see this page because there is no Web site at this address.
</body>
</html>

Ungov

unread,
Mar 29, 2022, 7:16:07 PMMar 29
to mod-pagespeed-discuss
Now I can confirm increasing SHMCache size really made a BIG difference. The webp are not getting evicted anymore. Hope I don't jinx it but I THINK it fixed the bug. Every time I reload the page there are 287 webp images in the source code and the number is stable. I am so happy we finally found the solution!

The only thing left to figure out is why PreserveURLs is not working

Longinos

unread,
Mar 30, 2022, 4:20:34 AMMar 30
to mod-pagespeed-discuss
Hi
If there is no http://localhost:7080  then you don´t need


Not sure where you put the the PreserveURL´s directive but in the admin pages -> Configuration, you can see is set to false.

ImagePreserveURLs (ipu) False

But if set to true, then you don´t have webp images. Preserving images url deactivate some image filters.

Ungov

unread,
Apr 10, 2022, 6:43:57 PMApr 10
to mod-pagespeed-discuss
Pardon me for bumping this thread again, but we're having issues with webp and cache header again

There are a lot of errors like this:
<!-- Resource headers are preventing rewriting of https://www.keinegotterkeinemeister.de/template/ngnm4/logo_ngnm4_de.png-->
But when looking at the image, the cache-control header seems fine

Longinos

unread,
Apr 11, 2022, 5:28:52 AMApr 11
to mod-pagespeed-discuss
Hi
This message is ot "uncacheable content"
Looking at the code is more related to don´t have a status in the pagespeed cache.
Or even any other header but not the cache-control.
Have you tried to clear the pagespeed cache?

In other hand: You have enabled the lazyload filter, disable it.
This filter interfere with the lazyload javascript provided by lazysizes. And have an old implementation that load all images on pageload event.
You have
ModifyCachingHeaders (mch) False
Modify it and let pagespeed manipulate the cache.headers
ModPagespeedModifyCachingHeaders on

Ungov

unread,
Apr 11, 2022, 3:22:41 PMApr 11
to mod-pagespeed-discuss
I flushed the cache and added ModPagespeedModifyCachingHeaders

However I am still getting the same warning (Resource headers are preventing rewriting)
There are 538 warnings like that on the page, and even after reloading the page many many times it doesn't go away and the images aren't getting converted to webp

Ungov

unread,
Apr 11, 2022, 4:20:36 PMApr 11
to mod-pagespeed-discuss
Something is weird with the content-type of the images returning the error "Resource headers are preventing rewriting"

On the first page load, content-type is image/jpeg
If you reload the page again, content-type is image/webp

When disabling ModPageSpeed, the image content-type is always image/jpeg

Longinos

unread,
Apr 11, 2022, 6:04:05 PMApr 11
to mod-pagespeed-discuss
Hi
About content-type
If there is a header etag: W/PSA..... is a normal behaviuor.
Is a image optimized by IPRO, the on-fly optimizer
This do some image optimizations but don´t change the url, so no web file extension.

in the german page:
I see all images converted to webp but 5, these have Vary:user-agent header and this prevent the optimization
Maybe are images that changes between mobil/desktop?
logo_ngnm4_de.png
footer-ethics-8.png
and 3 imahes that have img_proxy in the url

No messages in the debug saying Resource headers are preventing rewriting

Ungov

unread,
Apr 11, 2022, 9:19:21 PMApr 11
to mod-pagespeed-discuss
Sorry, we were doing some testing and we had re-activated Cloudflare, that's why you didn't see the messages in debug mode (Cloudflare is removing all HTML comments)
We disabled Cloudflare again and the warnings show up (173 messages of "Resource headers are preventing rewriting")

Example:
<!--Resource headers are preventing rewriting of https://www.keinegotterkeinemeister.de/img/mini-design/13082415.jpg-->

Longinos IntersindicalRM

unread,
Apr 12, 2022, 8:28:04 AMApr 12
to mod-pagesp...@googlegroups.com
Hi
No, when tested the server was nginx, the messages are not in the html and ALL the images get webp converted unless 5 that have Vary: user-agent in theirs headers.

--
You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/nhZInLUODnk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/3ca0c594-1fbb-48b2-9d32-84ee8bcb2db1n%40googlegroups.com.

Ungov

unread,
Apr 12, 2022, 5:12:24 PMApr 12
to mod-pagespeed-discuss
There are 147 images not converted to webp and returning the error "Resource headers are preventing rewriting"

Longinos

unread,
Apr 13, 2022, 4:16:17 PMApr 13
to mod-pagespeed-discuss
Hi
Just now at 22:03 CEST
I get only 67 images, scrolling the page at bottom to load lazyloaded images.
62 webp converted, 5 not (the 5 I before mentioned they have a hader Vary: user-agent)
No message in the html code and served throug CF.
In these numbers data:image embebed url images don´t count.
But cade the messages are deleted by CF, the webp images are here....

Ungov

unread,
Apr 13, 2022, 6:49:03 PMApr 13
to mod-pagespeed-discuss
Hi,
When looking at the source code I still see the same warnings after disabling Cloudflare https://prnt.sc/4WSisaAeXrLV  
For example this image:  /mini-design/11068713.jpg
I also see it when inspecting the element: https://prnt.sc/7t7OnFx_IZiG
But maybe it is not a problem because those images are lazyloaded and not above the fold?

Some of the webp images that were converted are also showing JPEG as file type
https://prnt.sc/-YUyUhyByrJe

Longinos

unread,
Apr 14, 2022, 4:36:24 AMApr 14
to mod-pagespeed-discuss
Hi
Yes I can see these messages now.
Some are from images with the header Vary: user-agent
And as far as I can see the images are from the user uploaded images, those that where displayed when the user click a button.
They show 800 images aprox.... Can you try to shoe less user images?
These images are loaded by jquery so can´t be converted to webp. Resources loaded by javascript or by a xhr subrequest are not modified by pagespeed, only by IPRO
(you can see a header like etag: ·W/PSAXXXXX")
But others have the message and are loaded "normaly".... What happen here?
Maybe this, take a look on this one:


<div style='display:inline-block'>


<a href="https://www.keinegotterkeinemeister.de/tshirt-cnt-ait-la-lucha-sigue-D011192181P0210/">


<img data-src="https://www.keinegotterkeinemeister.de/img/mini-design/11192181.jpg" class="lazyload suggestion-img suggestion-img-design" id="2389" style="padding:2px;max-width:180px" width="200" height="200" alt="CNT-AIT la lucha sigue" title="CNT-AIT la lucha sigue"/><!--Resource headers are preventing rewriting of https://www.keinegotterkeinemeister.de/img/mini-design/11192181.jpg--><!--Resource headers are preventing rewriting of https://www.keinegotterkeinemeister.de/img/mini-design/11192181.jpg-->


</a>


</div>

There is no src attribute, the pagespeed version you use can rewrite data-src (but not data-srcset) , in fact others images with the same html are rewrited and the same image get rewrited in a page refresh so is an  intermitent issue. But the absence of the src attribute breaks the html spec.
Can you try to put a src attribute whit a placeholder image?
Some like:
<img data-src="https://www.keinegotterkeinemeister.de/img/mini-design/11192181.jpgsrc="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==class="lazyload suggestion-img suggestion-img-desig.....

This put a 1x1 px gif image, and lazisizes moves the data-src value to the src one when needed.
Reply all
Reply to author
Forward
0 new messages