ModPagespeedDisallow & ModPagespeedAllow not working as expected

[Rob]

I have an odd issue that I can't explain or find a solution for. Maybe someone here has dealt with this before.

Our site uses sharding to distribute static resources over Amazon Cloudfront. That worked fine, until search engines indexed dynamic resources through those shard servers as well, and customers were entering our site through these snapshots from Cloudfront.

So, with our site at https://www.solacity.com/ and shard servers on Cloudfront at s1.solacity.com, s2.solacity.com, s3.solacity.com it turned out that users could pull up https://s2.solacity.com/ and see a static version of our site. Any dynamic behaviour, buttons etc., would of course fail because it pointed to Cloudfront instead of our server.

To solve that I set up 'behaviors' on Cloudfront for all static file types (css, jpg, js etc.), and wrote a lambda function for the default behavior ('*' at the end of the list of file types so it would execute if none of the other types matched) to do a permanent redirect back to the same URL on our actual, dynamic, site.

This threw up the next potential issue; if modpagespeed sharded a file type that was not in my list of pre-defined behaviors on Cloudfront this could set up an endless loop (Cloudfront redirects back to our actual site, modpagespeed rewrites the URL back to Cloudfront, rinse and repeat!). To head this off I tried to limit modpagespeed to only the set of predefined file types using ModPagespeedDisallow & ModPagespeedAllow. The effect that had was to completely switch off modpagespeed with the following in my pagespeed.conf:

# Tell mod_pagespeed what file types are allowed for rewriting, and which ones should be left alone

ModPagespeedDisallow "*"

ModPagespeedAllow "*.css*"

ModPagespeedAllow "*.js*"

ModPagespeedAllow "*.gif*"

ModPagespeedAllow "*.png*"

ModPagespeedAllow "*.jpg*"

ModPagespeedAllow "*.jpeg*"

ModPagespeedAllow "*.webp*"

ModPagespeedAllow "*.svg*"

ModPagespeedAllow "*.html*"

ModPagespeedAllow "*.txt*"

ModPagespeedAllow "*.ico*"

    

# Specific disallowed files come last

ModPagespeedDisallow    "*/wp-content/plugins/akismet/*"

ModPagespeedDisallow    "*/CRM/*"

ModPagespeedDisallow "*/docs/*"

ModPagespeedDisallow "*/wp-content/themes/Avada/includes/lib/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/revslider/public/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/woocommerce/assets/fonts/*"

So, with the above modpagespeed is completely gone. Even "curl -Il https://www.solacity.com/" won't show the X-Mod-Pagespeed header any more, no sharding happens etc. Oddly the loggin at the modpagespeed admin Message History shows it is still happily working away at optimizing images etc. No URL gets rewritten though, so none of the optimized resources gets used and no sharding happens.

If I take away the line with ModPagespeedDisallow "*" everything works again (but there is the risk of rewriting a resource that ends up in that endless redirect loop).

From reading the modpagespeed documentation it seems that disallowing everything ("*") and then selectively allowing URL matches should rewrite just those resources. So why is that not what is happening, and is there a fix or workaround for this?

Thank you for any help you can give with this!

-Rob-

[Otto van der Schaaf]

re: It turned out that users could pull up https://s2.solacity.com/ and see a static version of our site

Maybe dynamic content should detect if it's requested via a shard, and issue a redirect to www.?

Wouldn't that solve the problem?

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/dad3a8e6-1cdf-46bd-a60d-8aa96aedc0c7%40googlegroups.com.

[Longinos]

Hi Rob

Can you post the whole pagespeed conf?

I have tried 3 options and take a look to the image files:

1.- https://www.solacity.com/

2.- https://www.solacity.com/?PageSpeedFilters=+debug

3.- https://www.solacity.com/?PageSpeed=off

 

With #1 I get image files that come from s1/s2.solcity.com and from www.solacity.com. From s1/s2 got rewrited but from www.solacity.com not but have a ETAG header from pagespeed IPRO (etag:

W/"PSA-aj-M3QNpfbzRI").

With #2 get all images rewrited unless 1. All images come from s1/s2/s3 unless the 1 not rewrited

With #3 none of the images got rewrited and all images come from www

The image blue-seal-200-42-solacityinc-41078.png has a strange behaviour in #2 their domain (https://seal-ottawa.bbb.org/seals/) got rewrited to s1 (https://s1.solacity.com/.pscache/sealottawabbborg/) and throw a 403 error code and don´t show.

None of these images (s1/s2/s3 nor www) have the x-mod-pagespeed header so maybe the header is stripes by the cache server.

[Rob]

Otto, re. detecting dynamic content via a shard and redirecting: That's a tough one to do, because it does not hit our Web site. It hits the shard, so I can't use code on my server side to detect this and take action. On the shards I only have Amazon's Cloudfront Lamba functions to take action, and I'll be the first to admit that I'm no whiz when it comes to writing those for Cloudfront. That's why I thought that restricting (and detecting) shards to static file types only would be the easier way to do about it. It's pretty straight forward to get Cloudfront to redirect everything that's not on a list of file types.

[Rob]

Longinos, here is the whole pagespeed.conf file (pardon the messed up layout, tabs don't get expanded the same way as they do in my editor):

# Attempt to load mod_version if it wasn't loaded or compiled in (eg on Debian)

<IfModule !mod_version.c>

  LoadModule version_module /usr/lib64/httpd/modules/mod_version.so

</IfModule>

<IfVersion < 2.4>

  LoadModule pagespeed_module /usr/lib64/httpd/modules/mod_pagespeed.so

</IfVersion>

<IfVersion >= 2.4.2>

  # As default pagespeed.conf configuration uses old-style ACLs using

  # 'allow' and 'deny', we load mod_access_compat in Apache 2.4. If that's a

  # problem, the blocks using that can be converted to the newer 'Require'

  # syntax.

  <IfModule !access_compat_module>

    LoadModule access_compat_module /usr/lib64/httpd/modules/mod_access_compat.so

  </IfModule>

  LoadModule pagespeed_module /usr/lib64/httpd/modules/mod_pagespeed_ap24.so

</IfVersion>

# Only attempt to load mod_deflate if it hasn't been loaded already.

<IfModule !mod_deflate.c>

 LoadModule deflate_module /usr/lib64/httpd/modules/mod_deflate.so

</IfModule>

<IfModule pagespeed_module>

    ModPagespeed on

ModPagespeedDomain www.solacity.com

    AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html

    AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER application/xhtml+xml

    # enable mod_deflate for html,xml,js,css,php files 

    ModPagespeedFetchWithGzip on

    SetOutputFilter DEFLATE

    AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml

    AddOutputFilterByType DEFLATE application/xhtml+xml text/javascript

    AddOutputFilterByType DEFLATE text/css application/x-javascript

    AddOutputFilterByType DEFLATE application/x-httpd-php

    DeflateCompressionLevel 9

    

# for old (not compatbile) browsers

    BrowserMatch ^Mozilla/4 gzip-only-text/html

    BrowserMatch ^Mozilla/4\.0[678] no-gzip

    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

    

# Set env var for images

    SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip

    # The ModPagespeedFileCachePath directory must exist and be writable

    # by the apache user (as specified by the User directive).

    ModPagespeedFileCachePath            "/var/cache/mod_pagespeed/"

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 51200

    # LogDir is needed to store various logs, including the statistics log

    # required for the console.

    ModPagespeedLogDir "/var/log/pagespeed"

    ModPagespeedMessageBufferSize 200000

# add extra error information (switch off for regular use)

# ModPagespeedListOutstandingUrlsOnError on

    # If you want, you can use one or more memcached servers as the store for

    # the mod_pagespeed cache.

    # ModPagespeedMemcachedServers localhost:11211

    # A portion of the cache can be kept in memory only, to reduce load on disk

    # (or memcached) from many small files.

    ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed_mem/" 51200

# explicitly enable only the filters we want

    ModPagespeedRewriteLevel PassThrough

ModPagespeedEnableFilters add_head,convert_meta_tags,rewrite_style_attributes_with_url

ModPagespeedEnableFilters rewrite_style_attributes,insert_dns_prefetch,rewrite_css

ModPagespeedEnableFilters combine_heads,fallback_rewrite_css_urls,move_css_to_head

    ModPagespeedEnableFilters elide_attributes,hint_preload_subresources

ModPagespeedEnableFilters rewrite_javascript

ModPagespeedEnableFilters collapse_whitespace,prioritize_critical_css,combine_javascript

ModPagespeedEnableFilters flatten_css_imports,remove_comments,remove_quotes,combine_css

ModPagespeedEnableFilters move_css_above_scripts,outline_css

ModPagespeedEnableFilters responsive_images,rewrite_images,responsive_images_zoom

ModPagespeedEnableFilters resize_images,resize_mobile_images,dedup_inlined_images,insert_image_dimensions 

ModPagespeedEnableFilters inline_css

ModPagespeedEnableFilters inline_javascript,inline_import_to_link

ModPagespeedEnableFilters sprite_images,recompress_images,resize_rendered_image_dimensions

ModPagespeedEnableFilters convert_png_to_jpeg,convert_jpeg_to_webp,convert_gif_to_png,convert_to_webp_lossless

ModPagespeedEnableFilters convert_to_webp_animated,convert_jpeg_to_progressive,recompress_webp,recompress_png

ModPagespeedEnableFilters make_google_analytics_async,recompress_jpeg

ModPagespeedEnableFilters lazyload_images,inline_images,inline_preview_images

ModPagespeedEnableFilters extend_cache

ModPagespeedEnableFilters trim_urls,canonicalize_javascript_libraries

ModPagespeedEnableFilters rewrite_domains

# ModPagespeedEnableFilters inline_google_font_css,add_instrumentation

ModPagespeedGoogleFontCssInlineMaxBytes 1000

ModPagespeedSupportNoScriptEnabled false

ModPagespeedForceCaching on

ModPagespeedInPlaceResourceOptimization on

ModPagespeedDisableRewriteOnNoTransform off

ModPagespeedEnableCachePurge on

# Defering Javascript scores high in PageSpeed, but is dangerous!!!

# ModPagespeedEnableFilters defer_javascript

# switch on SSL/TLS for Serf fetching

    # The locations of SSL Certificates is distribution-dependent.

    ModPagespeedSslCertDirectory "/etc/letsencrypt/live/solacity.com"

    ModPagespeedSslCertFile "cert.pem"

ModPagespeedFetchHttps enable,allow_self_signed,allow_unknown_certificate_authority,allow_certificate_not_yet_valid

ModPagespeedMapOriginDomain "http://www.solacity.com" "https://www.solacity.com"

# additional domains we want mod_pagespeed to optimize for

ModPagespeedDomain https://s0.wp.com

ModPagespeedDomain https://s1.wp.com

ModPagespeedDomain https://stats.wp.com

ModPagespeedDomain https://secure.gravatar.com

ModPagespeedDomain https://seal-ottawa.bbb.org

ModPagespeedDomain https://fonts.googleapis.com

ModPagespeedDomain https://platform.twitter.com

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/s0wpcom/ https://s0.wp.com/wp-content/js/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/s1wpcom/ https://s1.wp.com/wp-content/js/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/statswpcom/ https://stats.wp.com/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/securegravatarcomjs/ https://secure.gravatar.com/js/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/securegravatarcomimg/ https://secure.gravatar.com/images/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/sealottawabbborg/ https://seal-ottawa.bbb.org/seals/

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/fontsgoogleapiscom/ https://fonts.googleapis.com/css

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/platformtwittercom/ https://platform.twitter.com/

ModPagespeedInlineResourcesWithoutExplicitAuthorization Script,Stylesheet

# shard files over three domains hosted at Amazon CloudFront

ModPagespeedShardDomain https://www.solacity.com https://s1.solacity.com,https://s2.solacity.com,https://s3.solacity.com

    # Other defaults (cache sizes and thresholds):

ModPagespeedRewriteDeadlinePerFlushMs 200

    ModPagespeedFileCacheSizeKb          5120000

    ModPagespeedFileCacheCleanIntervalMs 3600000

    ModPagespeedFileCacheInodeLimit        1000000

ModPagespeedDefaultSharedMemoryCacheKB 500000

    ModPagespeedLRUCacheKbPerProcess      2048

    ModPagespeedLRUCacheByteLimit        16384

    ModPagespeedCssFlattenMaxBytes        204800

    ModPagespeedCssInlineMaxBytes        5096

    ModPagespeedCssImageInlineMaxBytes    5096

    ModPagespeedImageInlineMaxBytes      5096

    ModPagespeedJsInlineMaxBytes          5096

    ModPagespeedCssOutlineMinBytes        4000

    ModPagespeedJsOutlineMinBytes        4000

    ModPagespeedMaxCombinedCssBytes      -1

    ModPagespeedMaxCombinedJsBytes        -1

ModPagespeedMaxCacheableContentLength 33554432

ModPagespeedPermitIdsForCssCombining *

ModPagespeedImageMaxRewritesAtOnce 0

ModPagespeedImplicitCacheTtlMs        1209600000

# ModPagespeedFinderPropertiesCacheExpirationTimeMs 86400000

    # Limit the number of inodes in the file cache. Set to 0 for no limit.

    # The default value if this paramater is not specified is 0 (no limit).

    # customize the number of threads per Apache process

ModPagespeedNumRewriteThreads 4

    ModPagespeedNumExpensiveRewriteThreads 4

    # Many filters modify the URLs of resources in HTML files. This is typically

    # harmless but pages whose Javascript expects to read or modify the original

    # URLs may break. The following parameters prevent filters from modifying

    # URLs of their respective types.

    #

    # ModPagespeedJsPreserveURLs on

    # ModPagespeedImagePreserveURLs on

    # ModPagespeedCssPreserveURLs on

    # Settings for image optimization:

    ModPagespeedImageRecompressionQuality 80

    ModPagespeedJpegRecompressionQuality 80

    ModPagespeedJpegRecompressionQualityForSmallScreens 70

    ModPagespeedWebpRecompressionQuality 80

    ModPagespeedWebpRecompressionQualityForSmallScreens 70

    ModPagespeedImageLimitOptimizedPercent 97

    ModPagespeedImageLimitResizeAreaPercent 100

    # Renaming JavaScript URLs can sometimes break them.  With this

    # option enabled, mod_pagespeed uses a simple heuristic to decide

    # not to rename JavaScript that it thinks is introspective.

    ModPagespeedAvoidRenamingIntrospectiveJavascript off

    # The following line configures a library that is recognized by

    # canonicalize_javascript_libraries.  This will have no effect unless you

    # enable this filter (generally by uncommenting the last line in the

    # previous stanza).  The format is:

    #    ModPagespeedLibrary bytes md5 canonical_url

    # Where bytes and md5 are with respect to the *minified* JS; use

    # js_minify --print_size_and_hash to obtain this data.

    # Note that we can register multiple hashes for the same canonical url;

    # we do this if there are versions available that have already been minified

    # with more sophisticated tools.

    #

    # Additional library configuration can be found in

    # pagespeed_libraries.conf included in the distribution.  You should add

    # new entries here, though, so that file can be automatically upgraded.

    # ModPagespeedLibrary 43 1o978_K0_LNE5_ystNklf http://www.modpagespeed.com/rewrite_javascript.js

    # Explicitly tell mod_pagespeed to load some resources from disk.

    ModPagespeedLoadFromFile "https://www.solacity.com/" "/var/www/html/solacity/"

    ModPagespeedLoadFromFile "http://www.solacity.com/"  "/var/www/html/solacity/"

    ModPagespeedLoadFromFileRuleMatch disallow .*

    ModPagespeedLoadFromFileRuleMatch allow \.css$

    ModPagespeedLoadFromFileRuleMatch allow \.js$

    ModPagespeedLoadFromFileRuleMatch allow \.gif$

    ModPagespeedLoadFromFileRuleMatch allow \.png$

    ModPagespeedLoadFromFileRuleMatch allow \.jpg$

    ModPagespeedLoadFromFileRuleMatch allow \.jpeg$

    ModPagespeedLoadFromFileRuleMatch allow \.webp$

    ModPagespeedLoadFromFileRuleMatch allow \.svg$

# Tell mod_pagespeed what file types are allowed for rewriting, and which ones should be left alone

# ModPagespeedDisallow "*"

ModPagespeedAllow "*.css*"

ModPagespeedAllow "*.js*"

ModPagespeedAllow "*.gif*"

ModPagespeedAllow "*.png*"

ModPagespeedAllow "*.jpg*"

ModPagespeedAllow "*.jpeg*"

ModPagespeedAllow "*.webp*"

ModPagespeedAllow "*.svg*"

ModPagespeedAllow "*.html*"

ModPagespeedAllow "*.txt*"

ModPagespeedAllow "*.ico*"

    

# Specific disallowed files come last

ModPagespeedDisallow    "*/wp-content/plugins/akismet/*"

ModPagespeedDisallow    "*/CRM/*"

ModPagespeedDisallow "*/docs/*"

ModPagespeedDisallow "*/wp-content/themes/Avada/includes/lib/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/revslider/public/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/woocommerce/assets/fonts/*"

    # Enables server-side instrumentation and statistics.  If this rewriter is

    # enabled, then each rewritten HTML page will have instrumentation javacript

    # added that sends latency beacons to /mod_pagespeed_beacon.  These

    # statistics can be accessed at /mod_pagespeed_statistics.  You must also

    # enable the mod_pagespeed_statistics and mod_pagespeed_beacon handlers

    # below.

    #

    # ModPagespeedEnableFilters add_instrumentation

    # The add_instrumentation filter sends a beacon after the page onload

    # handler is called. The user might navigate to a new URL before this. If

    # you enable the following directive, the beacon is sent as part of an

    # onbeforeunload handler, for pages where navigation happens before the

    # onload event.

    #

    # ModPagespeedReportUnloadTime on

    # Uncomment the following line so that ModPagespeed will not cache or

    # rewrite resources with Vary: in the header, e.g. Vary: User-Agent.

    # Note that ModPagespeed always respects Vary: headers on html content.

    # ModPagespeedRespectVary on

    # Uncomment the following line if you want to disable statistics entirely.

    #

    # ModPagespeedStatistics off

    # These handlers are central entry-points into the admin pages.

    <Location /pagespeed_admin>

        <IfModule mod_rewrite.c>

            RewriteEngine Off

        </IfModule>

       Order allow,deny

        Allow from localhost

        Allow from 127.0.0.1

Allow from 192.168

        SetHandler pagespeed_admin

    </Location>

    <Location /pagespeed_global_admin>

        <IfModule mod_rewrite.c>

            RewriteEngine Off

        </IfModule>

        Order allow,deny

        Allow from localhost

        Allow from 127.0.0.1

Allow from 192.168

        SetHandler pagespeed_global_admin

    </Location>

<Location /mod_pagespeed_beacon>

        <IfModule mod_rewrite.c>

            RewriteEngine Off

        </IfModule>

SetHandler mod_pagespeed_beacon

</Location>

<Location /mod_pagespeed_statistics>

        <IfModule mod_rewrite.c>

            RewriteEngine Off

        </IfModule>

Order allow,deny

Allow from localhost

        Allow from 127.0.0.1

Allow from 192.168

SetHandler mod_pagespeed_statistics

</Location>

    # Enable logging of mod_pagespeed statistics, needed for the console.

    ModPagespeedStatisticsLogging on

</IfModule>

[Longinos]

Hi Rob

These directives are not needed:

# additional domains we want mod_pagespeed to optimize for

ModPagespeedDomain https://s0.wp.com

ModPagespeedDomain https://s1.wp.com

ModPagespeedDomain https://stats.wp.com

ModPagespeedDomain https://secure.gravatar.com

ModPagespeedDomain https://seal-ottawa.bbb.org

ModPagespeedDomain https://fonts.googleapis.com

ModPagespeedDomain https://platform.twitter.com

Cause the directive ModPagespeedDomain is for domains that runs pagespeed and none of these domains runs it.

Do you need sharding domains? You are running http2, so sharding is not needed, you can search for "http2 sharding domains" and found a bunch of articles about it, for example https://love2dev.com/blog/domain-sharding-http-2/

In a "normal" hit (https://www.solacity.com/) the html is not the same that if you do a request with parameters (https://www.solacity.com/?PageSpeedFilters=%20debug)

When "normal" request, images have data-wpfc-original-src attribute (perhaps some type of lazyload set by WP fastest cache), these images can´t be rewrited, try:

ModPagespeedUrlValuedAttribute img data-wpfc-original-src image

Same with srcset, "normal" request have data-wpfc-original-srcset label, but at the moment pagespeed don´t have a UrlValuedAttrbute for srcset.

When request is with parameters, these attributes are not set.

If you are using WP Fastest Cache, this plugin had a CDN config, perhaps interfere with your CDN.

In "normal" request the image https://seal-ottawa.bbb.org/seals/blue-seal-200-42-solacityinc-41078.png is not proxied ( you have ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/sealottawabbborg/ 		https://seal-ottawa.bbb.org/seals/)
and the image show but when made with parameter, the image is proxied ( https://s1.solacity.com/.pscache/sealottawabbborg/blue-seal-200-42-solacityinc-41078.png) but returns a 403 Forbiden error,
if you try to hit the proxied image it show a message in the browser "Missing Content-Type required for proxied resource". So take a look at this, perhaps directory permissions issue.

Some images had a cors issue.



 

[Rob]

Longinos, thank you very much for looking into my configuration! Much appreciated!

I've removed the ModPageSpeedDomain items for those outside domains. My hope was that I could add directives to proxy some of the static content so it would be rewritten by modpagespeed and load from my domain (*.solacity.com) and speed things up that way. Things don't seem to work that way, or I have to look into that at a later date.

I've switched off lazy-load on the WP Fastest Cache, and left that entirely to modpagespeed.

Likewise, the only CDN is done by modpagespeed through sharding right now.

Regarding the CORS issues, I've been looking into that and ran into some modpagespeed weirdness: 

When I switch off modpagespeed ("ModPagespeed off") I see the the "Access-Control-Allow-Origin: *" header just fine:

[root@frodo ~]# curl -I https://www.solacity.com/wp-content/uploads/2015/11/P1060437-300x200.jpg

HTTP/1.1 200 OK

Date: Wed, 30 Oct 2019 13:56:29 GMT

Server: Apache

X-Content-Type-Options: nosniff

Strict-Transport-Security: max-age=63072000; includeSubdomains;

Upgrade: h2,h2c

Connection: Upgrade, close

Last-Modified: Tue, 16 Jul 2019 20:32:28 GMT

Accept-Ranges: none

Content-Length: 17847

Cache-Control: max-age=2592000

Expires: Fri, 29 Nov 2019 13:56:29 GMT

X-Content-Type-Options: nosniff

X-Frame-Options: sameorigin

X-XSS-Protection: 1; mode=block

Content-Security-Policy: default-src https:; font-src https: data:; img-src https: data: 'self' about:; script-src 'unsafe-inline' 'unsafe-eval' https: data:; style-src 'unsafe-inline' https:;

Vary: User-Agent

Access-Control-Allow-Origin: *

Content-Type: image/jpeg

The moment I switch on modpagespeed ("ModPagespeed on"), this header gets stripped, regardless if I use debug mode or not:

[root@frodo ~]# curl -I https://www.solacity.com/wp-content/uploads/2015/11/P1060437-300x200.jpg

HTTP/1.1 200 OK

Date: Wed, 30 Oct 2019 13:58:19 GMT

Server: Apache

Upgrade: h2,h2c

Connection: Upgrade, close

Content-Length: 16011

X-Original-Content-Length: 17847

Etag: W/"PSA-aj-YDnqjeQAmR"

Expires: Wed, 30 Oct 2019 14:03:19 GMT

Cache-Control: max-age=300

X-Content-Type-Options: nosniff

Content-Type: image/jpeg

[root@frodo ~]# curl -I https://www.solacity.com/wp-content/uploads/2015/11/P1060437-300x200.jpg?PageSpeedFilters=%20debug

HTTP/1.1 200 OK

Date: Wed, 30 Oct 2019 13:58:44 GMT

Server: Apache

Upgrade: h2,h2c

Connection: Upgrade, close

Expires: Wed, 13 Nov 2019 13:58:44 GMT

Last-Modified: Tue, 16 Jul 2019 20:32:28 GMT

Cache-Control: max-age=1209600, s-maxage=10

X-Content-Type-Options: nosniff

Content-Type: image/jpeg

Any idea what could be going no here? I would love to get CORS sorted out and working!

Lastly, the issue about restricting file types modpagespeed handles (as per my first post in this thread) remains as yet unsolved.

-Rob-

[Longinos]

I've removed the ModPageSpeedDomain items for those outside domains. My hope was that I could add directives to proxy some of the static content so it would be rewritten by modpagespeed and load from my domain (*.solacity.com) and speed things up that way. Things don't seem to work that way, or I have to look into that at a later date.

You can proxy external resources with  ModPagespeedMapProxyDomain and these directive was working but when /?PageSpeedFilters=+debug is set,so here is an issue on hoow the request is handled with or without parameters.

I've switched off lazy-load on the WP Fastest Cache, and left that entirely to modpagespeed.

Likewise, the only CDN is done by modpagespeed through sharding right now.

You can use the lazy-load thing, but need to tell to pagespeed what the label is. Pagespeed understand the labels src, srcset and data-src and have a category of image.

whit these category you can tell to pagespeed what other label you use for src (data-wpfc-original-src) but at the moment not for srcset (data-wpfc-original-srcset). In a next release that is solved.

Try to add ModpagespeedAddResourceHeader "Access-Control-Allow-Origin" "*" to the config. This directive dd headers to the optimized resources.

Lastly, the issue about restricting file types modpagespeed handles (as per my first post in this thread) remains as yet unsolved.

As far as I can see, in the home page font-awesome.css is untouched, this file comes from  /wp-content/plugins/revslider/public/assets/fonts/ or the file icomoon.woff that come from

/wp-content/themes/Avada/includes/lib/assets/fonts/icomoon/icomoon.woff, so the filters

ModPagespeedDisallow "*/wp-content/plugins/revslider/public/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/themes/Avada/includes/lib/assets/fonts/*"

are working.

I´m not sure, but may be related to the order that filters have. Try to put all the ModPagespeedAllow/Disallow before the ModPagespeedLoadFromFile directives. The order on what these directives are set matter.

# Tell mod_pagespeed what file types are allowed for rewriting, and which ones should be left alone

# ModPagespeedDisallow "*"

ModPagespeedAllow "*.css*"

ModPagespeedAllow "*.js*"

ModPagespeedAllow "*.gif*"

ModPagespeedAllow "*.png*"

ModPagespeedAllow "*.jpg*"

ModPagespeedAllow "*.jpeg*"

ModPagespeedAllow "*.webp*"

ModPagespeedAllow "*.svg*"

ModPagespeedAllow "*.html*"

ModPagespeedAllow "*.txt*"

ModPagespeedAllow "*.ico*"

    

# Specific disallowed files come last

ModPagespeedDisallow    "*/wp-content/plugins/akismet/*"

ModPagespeedDisallow    "*/CRM/*"

ModPagespeedDisallow "*/docs/*"

ModPagespeedDisallow "*/wp-content/themes/Avada/includes/lib/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/revslider/public/assets/fonts/*"

ModPagespeedDisallow "*/wp-content/plugins/woocommerce/assets/fonts/*"

[Rob]

Thank you Longinos!

I believe I've sorted out the CORS headers enough now to make it work. Using "ModpagespeedAddResourceHeader" does indeed work to add those headers:

# Add CORS header to optimized resources

ModpagespeedAddResourceHeader "Access-Control-Allow-Origin" "*"

ModPagespeedAddResourceHeader "Access-Control-Max-Age" "1000"

ModPagespeedAddResourceHeader "Access-Control-Allow-Methods" "POST, GET, OPTIONS, PUT, DELETE"

ModPagespeedAddResourceHeader "Access-Control-Allow-Headers" "X-Requested-With, Content-Type, X-Auth-Token, Origin, Authorization, Accept, Client-Security-Token, Accept-Encoding"

Using "Access-Control-Allow-Origin "*"" for everything completely defeats the purpose of CORS, but I've not found a way to return proper domain authorization when sharding is used (i.e. to actually enable CORS so it checks if a domain is allowed I would have to return "Access-Control-Allow-Origin "https://s1.solacity.com" if a resource request came from that shard). Anyway, it is what it is. CORS is half-arsed, but for now it at least makes things work with browsers that check.

One oddity remains, though this is probably of no consequence in real life: When modpagespeed is enabled it WILL strip out those "Access-Control-" headers even for resources that are directly requested from the server, without optimization or URL rewriting. For example:

[root@frodo ~]# curl -I https://www.solacity.com/wp-content/uploads/revslider/Homepagerevolution/Fronius-img.png

HTTP/1.1 200 OK

Date: Thu, 31 Oct 2019 15:19:27 GMT

Server: Apache

Upgrade: h2,h2c

Connection: Upgrade, close

Expires: Thu, 14 Nov 2019 15:19:27 GMT

Last-Modified: Wed, 10 Oct 2018 14:42:41 GMT

Cache-Control: max-age=1209600, s-maxage=10

X-Content-Type-Options: nosniff

Content-Type: image/png

No CORS headers in this reply, though Apache is set to explicitly add them (and will add them for file types that modpagespeed doesn't touch, I've tested that):

# ----------------------------------------------------------------------

# |CORS access control                                                 |

# ----------------------------------------------------------------------

# Always set these headers

<IfModule mod_headers.c>

Header always set Access-Control-Allow-Origin "*"

Header always set Access-Control-Max-Age "1000"

Header always set Access-Control-Allow-Methods "POST, GET, OPTIONS, PUT, DELETE"

Header always set Access-Control-Allow-Headers "X-Requested-With, Content-Type, X-Auth-Token, Origin, Authorization, Accept, Client-Security-Token, Accept-Encoding"

</IfModule>

Those headers do get added fine when "ModPagespeed off", but modpagespeed will strip them when on, for unoptimized resources as well.

Now, I suppose with modpagespeed enabled the situation where a resource is requested directly will simply never happen, and any resource that goes through modpagespeed gets CORS headers added via "ModpagespeedAddResourceHeader". So we'll let this be...

Moving on to restricting file types: I've tried moving all the ModPagespeedAllow and ModPagespeedDisallow lines before any ModPagespeedLoadFromFileRuleMatch lines. That did not make any difference. This sequence will completely disable any optimizing or sharding by modpagespeed:

ModPagespeedDisallow "*"

ModPagespeedAllow "*.css*"

ModPagespeedAllow "*.js*"

ModPagespeedAllow "*.gif*"

ModPagespeedAllow "*.png*"

ModPagespeedAllow "*.jpg*"

ModPagespeedAllow "*.jpeg*"

ModPagespeedAllow "*.webp*"

ModPagespeedAllow "*.svg*"

ModPagespeedAllow "*.html*"

ModPagespeedAllow "*.txt*"

ModPagespeedAllow "*.ico*"

 

At this point it looks like that may be a bug in modpagespeed. According to the docs it is legal to disable all file types and enable selective ones, as long as it is done in that order. Short of diving in the modpagespeed code I don't see a solution for this. I'll try and find a complete list of file types modpagespeed optimizes to make sure my Cloudflare cache handles all of those, and they don't fall through to the Lambda function that redirects to the original domain.

Lastly, I will try some more with ModPagespeedMapProxyDomain to proxy some static resources that are coming from other domains. It's not critical that this works, just would be nice as a way to speed up the site.

Thanks!

-Rob-

[Longinos]

Hi Rob

I have forgotten one thing about resource headers.

Apache set the headers when a web request is done, but... you have configured LoadFromFile, so no web request is done. When you read a file from disk they don´t have any header nor a cache-control.

For resources loaded from file, you have ModPagespeedLoadFromFileCacheTtlMs implicit_cache_ttl_in_milliseconds filter, if you don´t use this, you have a default cache-control of 300 for these resources.

In other hand, pagespeed work parsing the html and then optimize resources it found in the html code, so when you request a singular resource such an image, they can´t be optimized/rewrited, but IPRO (In-Place Resource Optimization) is working, so some optimization is done but without changing the url. This filter strip all the headers the resource has, and put a small set of headers in place. You can see whats resources are IPRO optimized with their etag header, it show PSA in it, like etag:

W/"PSA-aj-fVhg7vEsTR". I don´t know the reason for why this filter strip the headers....

Pagespeed can´t optimize resources that are loaded by javascript, so a image that is loaded by a javascript snipet is not rewrited, but if IPRO is in place then the image get some optimizations.

Maybe these 3 (load from file, parsing html and IPRO) things can explain the headers issues you see.

[Rob]

Longinos, indeed, I think that explains the header behavior! Thank you very much!

I've meanwhile been thinking through the CORS process (and need for it, or not). It seems to me that the only time CORS headers headers are needed is when my regular site (www.solacity.com) requests resources that were sharded (s1/2/3.solacity.com). Since it's modpagespeed doing the sharding, those always pass through there and "ModpagespeedAddResourceHeader" is applied, so those will have CORS headers. There is no need for me to add CORS headers to resources from www..., if they are requested, the request comes from the same URL anyway and CORS headers are not needed.

So, I have currently removed the "header set" statements from my httpd.conf and now only have modpagespeed add CORS headers to processed resources. Seems to work fine for the site; I've tested a number of pages and have yet to see any CORS errors. Safer too this way, since external sites can't request/link resources directly from my site without the client's browser refusing due to a CORS violation.

I will see if I can get proxy'ing with modpagespeed to work for some of the external resources my site loads. The work never ends! Always room for improvement! ;-)

-Rob-

[Longinos]

Hi Rob

About proxing resources, take care with it.

For example, when you have enabled it, you have:

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/sealottawabbborg/ https://seal-ottawa.bbb.org/seals/

What this does? If it work, you have ALL files from https.//seal-ottawa.bbb.org/seals/ stored in the directory. I say "if it work" cause s redirected to https://www.bbb.org/ and I don´t know if redirections are allowed. Or in the case of gravatar urls, these directories get a 403 forbidden.

Maybe you need a more concise configuration, like

ModPagespeedMapProxyDomain https://www.solacity.com/.pscache/sealottawabbborg/ https://seal-ottawa.bbb.org/seals/blue-seal-200-42-solacityinc-41078.png

I never used that directive so my experience is none, but I think that proxied resources must be public access, think that pagespeed need to access it like a browser. Maybe you need to configure pagespeed ssl to acces https sites.