How to add CDN with nginx_pagespeed properly?

[Saumya Majumder]

Hi Otto & others,

I have recently subscribed for KeyCDN Trial and wondering how can I incorporate it with my pagespeed so that all of my pagespeed optimized resources gets pushed using the cdn url instead of the normal url and at the same time all of my existing static file (css, js, img) gets auto rewritten from https://www.example.com/whatever.css to https://some123.keycdn.com/whatever.css

Now when I first installed pagespeed on my server Otto went above and beyond to help me and make sure my pagespeed.conf file is optimized to it's optimum level. So, without explaining further, here is my pagespeed.conf file:

pagespeed on;


#######################################################
# File cache settings
######################################
# needs to exist and be writable by nginx


pagespeed FileCacheSizeKb          1024000;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed FileCacheInodeLimit      5000000;
pagespeed LRUCacheKbPerProcess     81920;
pagespeed LRUCacheByteLimit        163840;


######################################################
# Mapping domain for HTTPS & Enabling HTTPS Fetch
######################################################


pagespeed LoadFromFileMatch "^https?://www.example.com/" "/home/nginx/domains/example.com/public/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.css$;
pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.gif$;
pagespeed LoadFromFileRuleMatch allow \.js$;
pagespeed Domain *.example.com/*;
pagespeed Disallow */amp/*;


#######################################################
# By default, ngx_pagespeed adds an X-PageSpeed header with a value of the version of
# ngx_pagespeed being used. This directive lets you specify the value to use instead:
# pagespeed XHeaderValue "ngx_pagespeed";


#######################################################
# let's speed up PageSpeed by storing it in the super duper fast memcached
# Ensure Memcached server installed http://centminmod.com/memcached.html
# default install for centmin mod is on port 11211, so localhost:11211 is correct
# uncomment - remove hash # in front of below 2 lines to enable
# timeout set at 100 milliseconds
#  pagespeed MemcachedThreads 1;
#  pagespeed MemcachedServers "localhost:11211";
#  pagespeed MemcachedTimeoutUs 100000;


#######################################################
## https://developers.google.com/speed/pagespeed/module/admin#virtual-hosts-and-stats
######################################
# pagespeed UsePerVhostStatistics on;


#######################################################
## 1.7.30.1 beta defaults
######################################
pagespeed PreserveUrlRelativity on;
pagespeed MaxCombinedCssBytes -1;
pagespeed ImageResolutionLimitBytes 16777216;
#pagespeed EnableFilters inline_google_font_css;


#######################################################
## 1.6.29.3 beta defaults
######################################
pagespeed AvoidRenamingIntrospectiveJavascript on;
pagespeed ImageInlineMaxBytes 3072;
pagespeed CssImageInlineMaxBytes 0;
pagespeed MaxInlinedPreviewImagesIndex -1;
pagespeed MinImageSizeLowResolutionBytes 3072;


#######################################################
## ngx_pagespeed filters settings below ##
######################################


# show half the users an optimized site, half the regular site
# change UA-XXXXXXXXXX-1 to your GA unique id
# uncomment - remove hash # in front of below 5 lines to enable
#  pagespeed RunExperiment on;
#  pagespeed AnalyticsID UA-XXXXXXXXXX-1;
#  pagespeed ExperimentVariable 1;
#  pagespeed ExperimentSpec "id=1;percent=50;level=CoreFilters;enabled=collapse_whitespace,remove_comments;";
#  pagespeed ExperimentSpec "id=2;percent=50";


# Filter settings
# https://developers.google.com/speed/pagespeed/module/filter-attribute-elide
pagespeed EnableFilters elide_attributes;


# https://developers.google.com/speed/pagespeed/module/filter-dedup-inlined-images
pagespeed EnableFilters dedup_inlined_images;


# filters outlined at http://ngxpagespeed.com/ngx_pagespeed_example/
pagespeed RewriteLevel CoreFilters;
pagespeed EnableFilters collapse_whitespace,remove_comments;


# make_google_analytics_async
# https://developers.google.com/speed/docs/mod_pagespeed/filter-make-google-analytics-async
pagespeed EnableFilters make_google_analytics_async;


# make_show_ads_async
# https://developers.google.com/speed/pagespeed/module/filter-make-show-ads-async
pagespeed EnableFilters make_show_ads_async;


# inline_google_font_css
# https://developers.google.com/speed/pagespeed/module/filter-css-inline-google-fonts
pagespeed EnableFilters inline_google_font_css;


# require FetchHttps if using https site
pagespeed FetchHttps enable;


# inline_css
# https://developers.google.com/speed/pagespeed/module/filter-css-inline
#   pagespeed EnableFilters inline_css;
#   pagespeed CssInlineMaxBytes 2048;


# prioritize_critical_css
# https://developers.google.com/speed/docs/mod_pagespeed/filter-prioritize-critical-css
pagespeed EnableFilters prioritize_critical_css;
#pagespeed CriticalImagesBeaconEnabled false;


# move_css_to_head
# https://developers.google.com/speed/docs/mod_pagespeed/filter-css-to-head
pagespeed EnableFilters move_css_to_head;


# move_css_above_scripts
# https://developers.google.com/speed/docs/mod_pagespeed/filter-css-above-scripts
pagespeed EnableFilters move_css_above_scripts;


# combine_css
# https://developers.google.com/speed/docs/mod_pagespeed/filter-css-combine
pagespeed EnableFilters combine_css;


# Uncomment this if you want to prevent mod_pagespeed from combining files
# (e.g. CSS files) across paths
#
#  pagespeed CombineAcrossPaths off;


# combine_javascript
# https://developers.google.com/speed/docs/mod_pagespeed/filter-js-combine
pagespeed EnableFilters combine_javascript;
pagespeed MaxCombinedJsBytes 140000;
#pagespeed EnableFilters defer_javascript;


# extend_cache
# https://developers.google.com/speed/docs/mod_pagespeed/filter-cache-extend
pagespeed EnableFilters extend_cache;
pagespeed ForceCaching on;


# rewrite_css
# https://developers.google.com/speed/docs/mod_pagespeed/filter-css-rewrite
pagespeed EnableFilters rewrite_css;


# rewrite_javascript
# https://developers.google.com/speed/docs/mod_pagespeed/filter-js-minify
pagespeed EnableFilters rewrite_javascript;


# inline_preview_images
# https://developers.google.com/speed/pagespeed/module/filter-inline-preview-images
pagespeed EnableFilters inline_preview_images;


# resize_mobile_images
# https://developers.google.com/speed/pagespeed/module/filter-inline-preview-images#resize_mobile_images
pagespeed EnableFilters resize_mobile_images;
pagespeed MaxInlinedPreviewImagesIndex 6;
pagespeed MinImageSizeLowResolutionBytes 3072;


# lazyload_images
# https://developers.google.com/speed/docs/mod_pagespeed/filter-lazyload-images
# pagespeed EnableFilters lazyload_images;
# pagespeed LazyloadImagesAfterOnload off;


# rewrite_images
# https://developers.google.com/speed/docs/mod_pagespeed/filter-image-optimize
pagespeed EnableFilters rewrite_images;
pagespeed EnableFilters inline_images;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters jpeg_subsampling;
pagespeed EnableFilters convert_png_to_jpeg;
pagespeed EnableFilters resize_images;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters convert_to_webp_lossless;


# convert_to_webp_animated
# https://developers.google.com/speed/pagespeed/module/filter-image-optimize#convert_to_webp_animated
pagespeed EnableFilters convert_to_webp_animated;


pagespeed EnableFilters insert_image_dimensions;


# responsive_images, responsive_images_zoom
# https://developers.google.com/speed/pagespeed/module/filter-image-responsive
pagespeed EnableFilters responsive_images;
pagespeed EnableFilters responsive_images_zoom;


#   pagespeed DisableFilters rewrite_images;
#   pagespeed DisableFilters recompress_images;
#   pagespeed DisableFilters convert_png_to_jpeg;
#   pagespeed DisableFilters extend_cache_images;


# sprite_images
# https://developers.google.com/speed/docs/mod_pagespeed/filter-image-sprite
pagespeed EnableFilters rewrite_css,sprite_images;


# insert_dns_prefetch
# https://developers.google.com/speed/docs/mod_pagespeed/filter-insert-dns-prefetch
pagespeed EnableFilters insert_dns_prefetch;


# InPlaceResourceOptimization
# https://developers.google.com/speed/pagespeed/module/system#ipro
# Enabled by default as of 1.9.32.1
pagespeed InPlaceResourceOptimization off;


#######################################################

Now as per the recommendation of Otto, I have added the following so make sure pagespeed can properly cache and optimize the static files:

######################################################
# Mapping domain for HTTPS & Enabling HTTPS Fetch
######################################################


pagespeed LoadFromFileMatch "^https?://www.example.com/" "/home/nginx/domains/example.com/public/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.css$;
pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.gif$;
pagespeed LoadFromFileRuleMatch allow \.js$;
pagespeed Domain *.example.com/*;
pagespeed Disallow */amp/*;

Now when I looked into the basic tutorial about adding cdn into my site, it made me extremely confused. So, should I change the above configs with something like this to make the cdn work?

Will this be the right way to do it?

######################################################
# Mapping domain for HTTPS & Enabling HTTPS Fetch
######################################################


pagespeed LoadFromFileMatch "^https?://www.example.com/" "/home/nginx/domains/example.com/public/";
pagespeed LoadFromFileRuleMatch disallow .*;
pagespeed LoadFromFileRuleMatch allow \.css$;
pagespeed LoadFromFileRuleMatch allow \.jpe?g$;
pagespeed LoadFromFileRuleMatch allow \.png$;
pagespeed LoadFromFileRuleMatch allow \.gif$;
pagespeed LoadFromFileRuleMatch allow \.js$;
pagespeed Domain *.example.com/*;
pagespeed Disallow */amp/*;


#################################
# Setting up CDN Things
#################################

pagespeed Domain *.keycdn.com/*;

pagespeed MapOriginDomain www.example.com origin123.keycdn.com;
pagespeed MapOriginDomain subdomain1.example.com origin345.keycdn.com;
pagespeed MapOriginDomain subdomain2.example.com origin678.keycdn.com;

pagespeed ShardDomain https://origin123.keycdn.com, https://origin456.keycdn.com, https://origin678.keycdn.com;

pagespeed MapRewriteDomain origin123.keycdn.com www.example.com;
pagespeed MapRewriteDomain origin345.keycdn.com subdomain1.example.com;
pagespeed MapRewriteDomain origin678.keycdn.com subdomain2.example.com;

Please let me know if I'm doing things in the right way? If I am not, please tell what config should I use? I am really sorry for this long post. I hope you guys will reply soon.

[Joshua Marantz]

What are you seeing so far with this config?  Or are you just vetting it before you install it?

Do you serve HTML directly from your example.com?  Or does your HTML get served from a domain that's mapped to the CDN? 

How are resources referenced from your HTML?  Via absolute URL to example.com?  Absolute to the CDN?  Or relative?

I have inlined some comments to your config:

On Sun, Jul 31, 2016 at 8:20 AM, Saumya Majumder <saumy...@gmail.com> wrote:

pagespeed LRUCacheKbPerProcess     81920;
pagespeed LRUCacheByteLimit        163840;

Our doc is not super clear on this and we need more tutorial.  We recommend you use the shared-memory metadata cache rather than the LRU Cache if possible.  I think if you comment this out, we'll provide a pretty well configured shared-memory metadata cache. 

I'm hoping we can deprecate the LRUCache implementation, once we finish some ongoing work to add checkpointing to the shared-memory metadata cache.  Did you try the SHM cache and have trouble with it?

pagespeed Disallow */amp/*;

In our next version this should not be necessary.  And there is some benefit to optimizing the images in an AMP document, particularly for users on a slow connection.

See https://github.com/pagespeed/mod_pagespeed/issues/1263 for details.  However for the current version, you will need this Disallow.

 

#######################################################
# let's speed up PageSpeed by storing it in the super duper fast memcached
# Ensure Memcached server installed http://centminmod.com/memcached.html
# default install for centmin mod is on port 11211, so localhost:11211 is correct
# uncomment - remove hash # in front of below 2 lines to enable
# timeout set at 100 milliseconds
#  pagespeed MemcachedThreads 1;
#  pagespeed MemcachedServers "localhost:11211";
#  pagespeed MemcachedTimeoutUs 100000;

Memcached can definitely speed things up, especially if you have multiple servers.  Any reason this is commented out?

# make_show_ads_async
# https://developers.google.com/speed/pagespeed/module/filter-make-show-ads-async
pagespeed EnableFilters make_show_ads_async;

I'm just curious -- we haven't heard feedback about this filter.  Is this helping you?  Do you use Google Adsense with the show_ads script and have PageSpeed convert it to the more modern (and async) adsbygoogle?   We'd love to hear about your experience with it.

Next you have a some CSS filter configuration:

#   pagespeed EnableFilters inline_css;
pagespeed EnableFilters prioritize_critical_css;
pagespeed EnableFilters move_css_to_head;
pagespeed EnableFilters move_css_above_scripts;

Given you have prioritize_critical_css, it makes sense to me to disable inline_css.  You definitely wouldn't want to inline the non-critical css, and the critical CSS will be inlined by prioritize_critical_css.  I think you should also turn off the move_css* filters as well, because I think they might interact badly with prioritize_critical_css.

Actually I think PageSpeed should do all of this automatically, because the priotiize_critical_css filter doesn't apply all the time (it needs fresh beacons), it should fall back to the other filters when they are enabled and prioritize_critical_css is not being applied.  I entered https://github.com/pagespeed/mod_pagespeed/issues/1368 to track.

But in the meantime, I think you should disable the move_css* filters.

pagespeed EnableFilters combine_css;

This is fine; I think this can help load the non-critical CSS more efficiently. 

pagespeed MaxCombinedJsBytes 140000;

I'm just curious where you got 140k from.   Did you do speed-tests with webpagetest at various settings of this?

 

pagespeed ForceCaching on;

I do not recommend ForceCaching.  It's really a debug/benchmarking tool.

 

# rewrite_images
# https://developers.google.com/speed/docs/mod_pagespeed/filter-image-optimize
pagespeed EnableFilters rewrite_images;
pagespeed EnableFilters inline_images;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters jpeg_subsampling;
pagespeed EnableFilters convert_png_to_jpeg;
pagespeed EnableFilters resize_images;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters convert_to_webp_lossless;

Given you have rewrite_images on, you don't need to enable the 7 below it; they are all auto-enabled when you specify rewrite_images.

 

Now as per the recommendation of Otto, I have added the following so make sure pagespeed can properly cache and optimize the static files:

pagespeed Disallow */amp/*;

You have this above, and unfortunately adding it a second time will do you no good, but will (infinitessimally) slow down PageSpeed. Might as well remove the second one :)  Moreover, when we do our next major release, you will be able to resume optimization of AMP pages, and you might forgot to comment out both of them...

Now when I looked into the basic tutorial about adding cdn into my site, it made me extremely confused. So, should I change the above configs with something like this to make the cdn work?

Will this be the right way to do it?

pagespeed Domain *.keycdn.com/*;

This is risky. If you wound up referencing an image from another site's keycdn domain, if that site was not also using ngx/mod_pagespeed.  You'd get 404s from rewriting the URLs on someothersite.keycdn.com.

For the subdomains of keycdn that reference back to your origin, it is neither necessary or sufficient to use "Domain" here.  Instead use MapOriginDomain, which will do the mapping and also authorize the domain.

 

pagespeed MapOriginDomain www.example.com origin123.keycdn.com;
pagespeed MapOriginDomain subdomain1.example.com origin345.keycdn.com;
pagespeed MapOriginDomain subdomain2.example.com origin678.keycdn.com;

You only need these if your HTML references resources directly on keycdn.

pagespeed ShardDomain https://origin123.keycdn.com, https://origin456.keycdn.com, https://origin678.keycdn.com;

pagespeed MapRewriteDomain origin123.keycdn.com www.example.com;
pagespeed MapRewriteDomain origin345.keycdn.com subdomain1.example.com;
pagespeed MapRewriteDomain origin678.keycdn.com subdomain2.example.com;

I'm a little confused by this.  Do you already have sharding to subdomain1 and subdomain2 in your HTML?  If you want to keep that sharding, then the 3 MapRewriteDomains are correct, but I wouldn't also put in ShardDomain.

If you want PageSpeed to do the resharding for you, then you can use MapRewriteDomain to pull together all the origin domains into one, and then use ShardDomains to redistribute them.  Does that make sense?

-Josh

[Saumya Majumder]

Hi Josh,

Thanks a lot for your in depth answer and I am really sorry my post was confusion to you. First of all I must say that the example.com here is actually www.isaumya.com and I haven't added the cdn thing on my site yet as I was worried if I do the config wrong it will break my site. My site always have active visitors. 

------------------------------------------------

Now coming to the LRUcache thing, actually it wasn't enabled before, but in my earlier threads Otto recommended me to add this in hope it will make my site more faster. So, what you are asking me to do is the following, right?

pagespeed FileCacheSizeKb          1024000;
pagespeed FileCacheCleanIntervalMs 3600000;
pagespeed FileCacheInodeLimit      5000000;

#pagespeed LRUCacheKbPerProcess     81920;
#pagespeed LRUCacheByteLimit        163840;

Let me know if this is what you wanted me to do.

-----------------------------------

I'm also glad to know that the amp issue is going to be resolved in the next version. I had to explicitly disable it as it was throwing too many errors in google webmaster.

---------------------------------

Regarding the MemCache thing, actually I am running my site on a single VPS. So, there is no multiple server working here. From what I read is memecache is highly efficient if you want using it in conjunction with several server, which is not the case for me. It is just a single VPS. Also as I run centminmod to setup my servers, it has a very easy option to enable redis on any server. But current the redis support is still a thing you guys working on (Github Issue #777) so after the redis support is enabled, I will enable redis on my server. That is why memcache is disabled for now.

--------------------------------

Now coming to the Google AdSense async question. Yes I do use AdSense on my site (only for the blog section) and I has seen this settings to works in a good way for old codes as well as the new codes. Though right now I use all the adsense new codes (async version) on my site but I still kept this settings enabled as I felt that it makes the async process even more good. It may be just my thought though :P

--------------------------------

Regarding the CSS thing, I am a bit confused with your reply. So do you want me to keep prioritize the critical css enabled while turning of the rest or turn on the res and disable the critical css feature? I mean do you want me to do this:

#   pagespeed EnableFilters inline_css;
pagespeed EnableFilters prioritize_critical_css;

# pagespeed EnableFilters move_css_to_head;
# pagespeed EnableFilters move_css_above_scripts;

or this:

  pagespeed EnableFilters inline_css;
# pagespeed EnableFilters prioritize_critical_css;

 pagespeed EnableFilters move_css_to_head;
 pagespeed EnableFilters move_css_above_scripts;

which one is more beneficial?

-------------------------------------------------------------------

Now about the following part:

pagespeed MaxCombinedJsBytes 140000;

it is a bit funny. The reason I added it because I was searching web for the best possible pagespeed config for my site and I saw someone has taken this approach, so I thought it might help my site as well. Hence I added it. Is it something bad? Should I remove it or increase the size? Please suggest.

------------------------------------------------------------------

OK. I will disable force caching but keep the extend cache enabled. I hope thats what you want me to do.

------------------------------------------------------------------

regarding the rewrite_image thing. I know that rewrite_image is the parent of all. But initially when I was configuring my pagespeed and just enabled rewrite_image but nothing else I saw that none of my image is getting converted to webp. Be it jpg or png or gif. So I added them all.

-----------------------------------------------------------------

Now coming to the most important thing that is the CDN part. Here are a few things you must need to know before I say thing further:

• As I already said that I use centminmod on my server whose current stable version is v0.8 But in v0.8 it does not support letsencrypt officially, it will come in upcoming v0.9. So for now I must need to use the shared ssl of KeyCDN. But after I upgrade to v0.9 of centminmod, I will be using proper cdn url for my main site as well as my subdomains. i.e. https://cdn.mydomain.com, https://subdomaincdn1.mydomain.com and so on for each sub domains on my site.
• Though www.isaumya.com is my main site, but I still run various expriments on my subdomains, each treated as seperate sites. Now as my main domain and sub-domain all are on the same VPS, so all of them take the benefit of pagespeed. That is why I wanted to know if I create a new wordpress site in one of my subdomain in future, how can I point the separate zone I have created for that subdomain to that subdomain only. So, my main site will have its own zone and subdomains will have its own. None conflicting with one another.
• I don't want to install any other plugin or something like that to my site just to change my static file urls from mydomain.com/something.pagespeed.css to something.keycdn.com/something.pagespeed.css I want pagespeed to handle all of this by its own.

Now I'm again a bit confused about which approach you suggest me to take here. Should I need to go like this:

pagespeed MapOriginDomain www.example.com origin123.keycdn.com;
pagespeed MapOriginDomain subdomain1.example.com origin345.keycdn.com;
pagespeed MapOriginDomain subdomain2.example.com origin678.keycdn.com;

to make sure all cdn works flawlessly for both my mainsite and subdomain.

or do I need to go like this:

pagespeed ShardDomain https://origin123.keycdn.com, https://origin456.keycdn.com, https://origin678.keycdn.com;

pagespeed MapRewriteDomain origin123.keycdn.com www.example.com;
pagespeed MapRewriteDomain origin345.keycdn.com subdomain1.example.com;
pagespeed MapRewriteDomain origin678.keycdn.com subdomain2.example.com;

Please suggest, I'm extremely confused. Also with this approach, when I make any changes to any of my static file will it automatically purge the cdn too, or I need to purge the cdn manually from keycdn dashboard after every change on the files (like css/js)?

--------------------------------------------------------

I want my site to perform as fast as possible and want pagespeed to work at it's extreme level to make my site super fast, so for that suggest me whatever needs to be done.

Also in case you have problem reading all these configs in these manner, you can also download my pagespeed.conf file from here: http://bit.ly/2anA4uz (Google Drive)

Thank you very much for taking your time to look into it. Looking forward to your fast reply. Hopefully you will reply soon.

[Otto van der Schaaf]

I'll try to answer some of the questions you have around the topic of cdn-rewriting:

For setting up PageSpeed and a CDN, I wrote a post a while ago: https://www.iispeed.com/blog/cdn-google-page-speed-optimization.

That domain uses the same CDN, and it works well there: automatic purging, and improved browser timings. 

Setting up the basics in ngx_pagespeed is simple!:

 

pagespeed MapRewriteDomain www-mysite.cdn.com www.yourdomain.com;

Reading up how you intend to use the subdomains, I think that you will probably have separate server{} blocks for those, and that you just have to repeat the one-liner above, using slightly different arguments in the other server{} block:

pagespeed MapRewriteDomain sub-mysite.cdn.com sub.yourdomain.com;

Optimized .pagespeed. urls constructed via LoadFromFile configuration will change their urls instantly when any of the inputs change, so you will not need to purge those. For those optimized urls that are not covered by LoadFromFile: these will change their url upon expiration of their input(s). You may have to purge the pagespeed side for these to force freshness, but there is no need to purge the CDN.

If you also want PageSpeed to rewrite non-optimized urls in html and css, you can add:

pagespeed EnableFilters rewrite_domains;

Any rewritten urls that do not have .pagespeed. in their urls will work as usual: you may have to purge the CDN to force freshness.

PageSpeed aside, note that you may have to take care of setting CORS headers, for example (review & modify as you see fit):

# Set CORS headers for .pagespeed. urls CORS
pagespeed AddResourceHeader "access-control-allow-origin" "*";
# non .pagespeed. CORS
add_header "access-control-allow-origin" "*";

You may also want to ensure canonical Link: header is set correctly. Later releases set this for .pagespeed. urls automatically.

As for origin mapping, if the cdn fetches from another cname, you may have to do that. For example:

pagespeed MapOriginDomain www.mydomain.com origin.mydomain.com; # alternatively you could use loopback fetching.

Hope that helps!

Otto

[Saumya Majumder]

Hi Otto, so in short you want me to add the following in my pagespeed.conf file right?

pagespeed AddResourceHeader "access-control-allow-origin" "*";
add_header "access-control-allow-origin" "*";
pagespeed EnableFilters rewrite_domains;

pagespeed ShardDomain https://origin123.keycdn.com, https://origin456.keycdn.com, https://origin678.keycdn.com;
pagespeed MapRewriteDomain origin123.keycdn.com www.example.com;
pagespeed MapRewriteDomain origin345.keycdn.com subdomain1.example.com;
pagespeed MapRewriteDomain origin678.keycdn.com subdomain2.example.com;

[Otto van der Schaaf]

Hi Otto, so in short you want me to add the following in my pagespeed.conf file right?

pagespeed AddResourceHeader "access-control-allow-origin" "*";
add_header "access-control-allow-origin" "*";
pagespeed EnableFilters rewrite_domains;
pagespeed ShardDomain https://origin123.keycdn.com, https://origin456.keycdn.com, https://origin678.keycdn.com;
pagespeed MapRewriteDomain origin123.keycdn.com www.example.com;
pagespeed MapRewriteDomain origin345.keycdn.com subdomain1.example.com;
pagespeed MapRewriteDomain origin678.keycdn.com subdomain2.example.com;

For CORS, that may be, but it would be good to:

- determine what you need (I don't know your requirements)

- read up so you know what you are doing if you have not already done so

For CDN-rewriting.

- Enabling rewrite_domains is up to you, see my earlier remarks on purging.

Some clarification would be helpful, because ShardDomain and multiple MapRewriteDomain lines still persist in the configuration you have in mind. 

- Why do you have ShardDomain in the configuration? 

- Why do you have multiple MapRewriteDomain lines (reusing the shard names) (in a single vhost?) 

Otto

[Saumya Majumder]

Hi actually I don't want to edit the nginx vhost config file. As I said earlier I have my main domain and sub domain for which I need to add the CDN.

I want the configuration needs to be done inside INR pagespeed config file.

I really don't understand what clarification you are asking to me. Sorry if I sound confused.

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/lteBtIQkb2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

--

Sincerely,

Saumya Majumder

Engineer, Author & Tech Reviewer

Web: www.isaumya.com

Email: saumy...@gmail.com

Confidentiality Notice: The information contained in this email and any accompanying attachment(s) is intended only for the use of the intended recipient and may be confidential and/or privileged.  If any reader of this communication is not the intended recipient, unauthorized use, disclosure or copying is strictly prohibited, and may be unlawful.  If you have received this communication in error, please immediately notify the sender by return email, and delete the original message and all copies from your system.  Thank you.

[Otto van der Schaaf]

Hi actually I don't want to edit the nginx vhost config file. As I said earlier I have my main domain and sub domain for which I need to add the CDN.

I want the configuration needs to be done inside INR pagespeed config file.

That made things clearer, thanks. Having both MapRewriteDomain and ShardDomain doing similar rewriting is what gets me confused (and I think it might get ngx_pagespeed confused as well). 

I'm trying to figure out why you have them both configured. Do you want ngx_pagespeed to perform sharding to parallize downloads across CDN hostnames? If not, you can leave ShardDomain out. If you do, my advice is to first get up and running without ShardDomain, and ask further questions here if you need information on (or have trouble with) moving from MapRewriteDomain to ShardDomain.

If you leave out ShardDomain, I think having multiple MapRewriteDomain lines (in a pagespeed configuration that is shared accross vhosts) will match your intent.

Otto

[Saumya Majumder]

Hi Otto,

thanks a lot for your reply. before I implement it, let me ask you one last question. In your message you have mentioned about sharding. Now my question is as my site is runs over HTTPS, I have enabled HTTP 2.0 technology on my server. So, in that case is sharding still going to help my site performance? As I'm already using HTTP 2?

[Joshua Marantz]

In general it would be great if you could set up a test.YOURSITE.COM vhost to experiment with new pagespeed configs before going live.

On Sun, Jul 31, 2016 at 11:31 AM, Saumya Majumder <saumy...@gmail.com> wrote:

Hi Josh,

Thanks a lot for your in depth answer and I am really sorry my post was confusion to you.

No worries!  Happy to help.

 

Let me know if this is what you wanted me to do.

Yes -- that was what I was suggesting.  It's independent of all the other questions about CDN setup and filter optimization.  What you have is OK, but you might get better cache hit rate if you change it.

I'm also glad to know that the amp issue is going to be resolved in the next version. I had to explicitly disable it as it was throwing too many errors in google webmaster.

Yes -- until you see a release with the AMP issue marked as solved in the release notes, please keep your Disallow -- but you just need one of them, not two :)

Our next release will *not* have that fix, as it will be a bug-fix release on the 1.11 branch. 

Regarding the MemCache thing, actually I am running my site on a single VPS. So, there is no multiple server working here. From what I read is memecache is highly efficient if you want using it in conjunction with several server, which is not the case for me. It is just a single VPS. Also as I run centminmod to setup my servers, it has a very easy option to enable redis on any server. But current the redis support is still a thing you guys working on (Github Issue #777) so after the redis support is enabled, I will enable redis on my server. That is why memcache is disabled for now.

Sounds good, staying with file-cache until we have Redis shipped makes sense. 

Now coming to the Google AdSense async question. Yes I do use AdSense on my site (only for the blog section) and I has seen this settings to works in a good way for old codes as well as the new codes.

That's great to hear.  When we put out new features, we rarely here from users whether they are useful and working.  We only here about them on this email list when there is a problem.

 

Though right now I use all the adsense new codes (async version) on my site but I still kept this settings enabled as I felt that it makes the async process even more good. It may be just my thought though :P

I think if you are using adsbygoogle.js on all the pages on your site, you don't need that filter anymore. 

--------------------------------

Regarding the CSS thing, I am a bit confused with your reply. So do you want me to keep prioritize the critical css enabled while turning of the rest or turn on the res and disable the critical css feature? I mean do you want me to do this:

#   pagespeed EnableFilters inline_css;
pagespeed EnableFilters prioritize_critical_css;
# pagespeed EnableFilters move_css_to_head;
# pagespeed EnableFilters move_css_above_scripts;

or this:

  pagespeed EnableFilters inline_css;
# pagespeed EnableFilters prioritize_critical_css;
 pagespeed EnableFilters move_css_to_head;
 pagespeed EnableFilters move_css_above_scripts;

which one is more beneficial?

The first one, with prioritize_critical_css.

-------------------------------------------------------------------

Now about the following part:

pagespeed MaxCombinedJsBytes 140000;

it is a bit funny. The reason I added it because I was searching web for the best possible pagespeed config for my site and I saw someone has taken this approach, so I thought it might help my site as well. Hence I added it. Is it something bad? Should I remove it or increase the size? Please suggest.

We set the default value based on tests we ran on a large number of sites.  There's no one right value for all sites and all clients over all networks and all devices.  If you have data that suggests the larger value is better, that's great.  Otherwise I'd suggest leaving the default. 

------------------------------------------------------------------

OK. I will disable force caching but keep the extend cache enabled. I hope thats what you want me to do.

Yes.

 

------------------------------------------------------------------

regarding the rewrite_image thing. I know that rewrite_image is the parent of all. But initially when I was configuring my pagespeed and just enabled rewrite_image but nothing else I saw that none of my image is getting converted to webp. Be it jpg or png or gif. So I added them all.

Remember that the first time a web page is viewed in your system, it will not be fully optimized.  That doesn't mean you need to enable more filters.  Always refresh a few times to see the fully optimized version.

Looks like Otto is helping you on the CDN issue, so I won't comment on that part.

-Josh

[Saumya Majumder]

Thanks a ton Josh for your in-depth replies. I will definitely fix my config file as per your recommendation. :) Thanks again. 

[Otto van der Schaaf]

On Mon, Aug 1, 2016 at 2:17 PM, Saumya Majumder <saumy...@gmail.com> wrote:

Now my question is as my site is runs over HTTPS, I have enabled HTTP 2.0 technology on my server. So, in that case is sharding still going to help my site performance? As I'm already using HTTP 2?

If the CDN you are intending to use runs HTTP/2, it probably doesn't make sense to shard in responses to clients that also support HTTP/2.

For HTTP1.1 clients, however, it may still make sense.

You can add some logic in configuration to decide upon the strategy [1], example:

http {
  pagespeed ProcessScriptVariables on;
  server {
    set $disable_filters "";
    set $domain_shards "shard1,shard2,shard3..."

    if ($http2) {
      set $disable_filters "filter1,filter2";
      set $domain_shards "";
    }
    pagespeed DisableFilters "$disable_filters";
    pagespeed ShardDomain domain_to_shard "$domain_shards";
  }
}

Alternatively, under certain conditions I believe connections may be re-used when on HTTP/2 [2] but I wouldn't know how that works out with your CDN. So deciding on the strategy on your webserver would probably be the most straightforward.

[1] https://developers.google.com/speed/pagespeed/module/https_support#h2_configuration_nginx

[2] https://tools.ietf.org/html/rfc7540#section-9.1.1

Otto

 

[Saumya Majumder]

Hi Otto, the CDN that I am going to use i.e. KeyCDN does support HTTP2. Thanks for the config you have provided above. I will let you know if I need more help.

[Saumya Majumder]

Hey Guys,

I have done everything as you recommended added the following line in in my pagespeed conf:

pagespeed EnableFilters rewrite_domains;
pagespeed MapRewriteDomain cdn.isaumya.com www.isaumya.com;

but if you visit my site and check the source code you will see that all the static resources are still using the https://www.isaumya.com and not https://cdn.isaumya.com

Why is this happening? Is this because this is a wordpress site?

Also as you suggested I tried adding :

pagespeed AddResourceHeader "access-control-allow-origin" "*";

but it throws error everytime I run npreload.

Do you have any idea why is this happening? In case you need here is my current updated pagespeed.conf file: https://docs.google.com/document/d/1RqvpRVUlHicsqNBTtc04kF_3YJWDKVz6TCPU_rQ9fRw/edit?usp=sharing

Please help.

[Otto van der Schaaf]

pagespeed EnableFilters rewrite_domains;
pagespeed MapRewriteDomain cdn.isaumya.com www.isaumya.com;

but if you visit my site and check the source code you will see that all the static resources are still using the https://www.isaumya.com and not https://cdn.isaumya.com

Why is this happening? Is this because this is a wordpress site?

The MapRewriteDomain directive should reflect that you are running on https:

pagespeed MapRewriteDomain https://cdn.isaumya.com https://www.isaumya.com;

 

Also as you suggested I tried adding :

pagespeed AddResourceHeader "access-control-allow-origin" "*";

but it throws error everytime I run npreload.

 

Do you have any idea why is this happening? 

Can you explain about what npreload is? And the specifics of what gets thrown?

Otto

[Saumya Majumder]

npreload is the command to restart the nginx after modifying the pagespeed config. Currently I'm sleeping bit I'll definately try the https thing tomorrow morning and post the error I'm getting when I add that directive.

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/lteBtIQkb2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Saumya Majumder]

Hi Otto,

after changing the pagespeed.conf file when I run npreload, it generally gives output like this when everything is OK:

[username@server ~]# nano /usr/local/nginx/conf/pagespeed.conf
[username@server ~]# npreload
Reloading nginx configuration (via systemctl):             [  OK  ]
Reload service php-fpm  done

but when I add 

pagespeed AddResourceHeader "access-control-allow-origin" "*";

and then run npreload, it gives me this:

[username@server ~]# nano /usr/local/nginx/conf/pagespeed.conf
[username@server ~]# npreload
Reloading nginx configuration (via systemctl):  Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
                                                           [FAILED]
Reload service php-fpm  done

Hope this helps you to clear your confusion.

Also adding the https:// in the MapRewriteDomain direct worked perfectly. But I just saw one minor issue (not sure if this is an issue with pagespeed itself or not). The problem is pagespeed is not doing it's magic or adding the cdn for the image tag inside srcset. For example, take a look at this image tag:

<img width="150" height="150" src="https://cdn.isaumya.com/wp-content/uploads/2015/09/xpremade-image-08.png.pagespeed.ic.anj2_W-qmt.webp" class="et_bloom_image_slideup et_bloom_image" alt="premade-image-08" srcset="https://www.isaumya.com/wp-content/uploads/2015/09/premade-image-08.png 150w, https://www.isaumya.com/wp-content/uploads/2015/09/premade-image-08-75x75.png 75w, https://www.isaumya.com/wp-content/uploads/2015/09/premade-image-08-65x65.png 65w" sizes="(max-width: 150px) 100vw, 150px">

as you can see the image within the src tag of image is properly compressed and cdn added by pagespeed. But for the image tags inside the srcset, they left untouched by pagespeed.

Is this a pagespeed bug? is there a way to fix it now or is it going to be a part of next pagespeed release and I have to wait for that?

Thanks again for all of your helps. Without you this never would have been possible. :)

[Saumya Majumder]

Hi Otto,

I'm also facing another problem after enabling cdn on my site. I think I'm facing this problem as I'm running a wordpress site. You see in the frontend of the site everything is running fine. But in the backend (inside wp-admin) wordpress uses load-script.php to load most of its JS files. Now the problem is that this load-script.php is getting treated as a static file and getting parsed through CDN and breaking the wp-admin functionality.

Is there a way to mention inside pagespeed.conf that it doesn't treat load-script.php or any other php file (though it has static behaviour) as static file and hence do not rewrite domain? Is it possible?

Take a look at the screenshot:

[Otto van der Schaaf]

On Wed, Aug 3, 2016 at 9:18 AM, Saumya Majumder <saumy...@gmail.com> wrote:

pagespeed AddResourceHeader "access-control-allow-origin" "*";

....

Hope this helps you to clear your confusion.

I thought that this feature was already released, but it looks like it wasn't yet. So you can't configure this yet. I'm sorry about that!

 

Also adding the https:// in the MapRewriteDomain direct worked perfectly. But I just saw one minor issue (not sure if this is an issue with pagespeed itself or not). The problem is pagespeed is not doing it's magic or adding the cdn for the image tag inside srcset. 

.... 

Is this a pagespeed bug? is there a way to fix it now or is it going to be a part of next pagespeed release and I have to wait for that?

The current version doesn't parse and optimize srcset attributes. It looks like a change for supporting that was already committed via https://github.com/pagespeed/mod_pagespeed/commit/428b7e044e753169724e7512804d41cca9c11651 but I do not know when the feature will be released.

Otto

[Otto van der Schaaf]

On Wed, Aug 3, 2016 at 11:19 AM, Saumya Majumder <saumy...@gmail.com> wrote:

But in the backend (inside wp-admin) wordpress uses load-script.php to load most of its JS files. Now the problem is that this load-script.php is getting treated as a static file and getting parsed through CDN and breaking the wp-admin functionality.

Is there a way to mention inside pagespeed.conf that it doesn't treat load-script.php or any other php file (though it has static behaviour) as static file and hence do not rewrite domain? Is it possible?

You could try disallowing:

pagespeed Disallow "*/load-script.php*";

Otto

[Saumya Majumder]

Thank you Otto for you unbelievable help on this matter. so as you said that

pagespeed AddResourceHeader "access-control-allow-origin" "*";

This feature is not available now, so I think I better wait for it to be published. Can you please keep me posted on this thread when this feature gets released?

Also the 

pagespeed Disallow "*/load-scripts.php*";

this trick works flawlessly. Thanks again. Now wp-admin is back to its normal stage, thanks to you. :)

Regarding the SRCSET thing, I have seen your link: https://github.com/pagespeed/mod_pagespeed/commit/428b7e044e753169724e7512804d41cca9c11651

But I have a question regarding this. As you said there is no ETA about when this feature is going to be introduced inside the ngx_pagespeed, not in the next major release. May be the major release after the upcoming major release.

So, for that time being do you suggest me to manually edit my pagespeed file and include this patch or it is unstable and I should wait for the official release?

If you tell me I can add this patch now manually then I will forward the github patch link to my server admin so that he can add it to the server. But waiting for your recommendation first. 

[Jeff Kaufman]

On Wed, Aug 3, 2016 at 7:59 AM, Saumya Majumder <saumy...@gmail.com> wrote:
> Regarding the SRCSET thing, I have seen your link:
> https://github.com/pagespeed/mod_pagespeed/commit/428b7e044e753169724e7512804d41cca9c11651
>
> But I have a question regarding this. As you said there is no ETA about when
> this feature is going to be introduced inside the ngx_pagespeed, not in the
> next major release. May be the major release after the upcoming major
> release.

This won't be in the upcoming bugfix release (1.11.33.3), because it's
a new feature, but it should be in the next major release (1.12.x.x).

Jeff

[Saumya Majumder]

So should I add this patch manually to my pagespeed now?

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/lteBtIQkb2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Otto van der Schaaf]

On Wed, Aug 3, 2016 at 8:06 PM, Saumya Majumder <saumy...@gmail.com> wrote:

So should I add this patch manually to my pagespeed now?

You would be running code from the bleeding edge in production. That has a certain risk associated to it - which you need to weigh against how much having the feature is worth to you.

[Saumya Majumder]

WOW! What risks are we talking about with this srcset patch? If this srcset patch has risks associated with it then I'll wait for the final build. Please let me know.

[Otto van der Schaaf]

The risk I mentioned is not specific to the srcset patch. The risk is in early adopting code in general.

Otto

On Wed, Aug 3, 2016 at 9:38 PM Saumya Majumder <saumy...@gmail.com> wrote:

WOW! What risks are we talking about with this srcset patch? If this srcset patch has risks associated with it then I'll wait for the final build. Please let me know.

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Saumya Majumder]

Oh! OK. I got your point. :)

[Saumya Majumder]

Hi Jeff,

Sorry to disturb you again man with this thread. Just wanted to ask you one quick question. As you have seen in my pagespeed.conf that I use 

pagespeed EnableFilters rewrite_css;

So, I was wondering Should I add

pagespeed EnableFilters fallback_rewrite_css_urls;

pagespeed EnableFilters fallback_rewrite_css_urls;

pagespeed EnableFilters fallback_rewrite_css_urls;

too or just use rewrite_css? Looking forward to your reply.

[Otto van der Schaaf]

You can add that line (probably better to do it just once instead of three times though), if you want to rewrite urls in css that failed to parse correctly.

Looking at the sources of the optimization examples, you can see that without enabling it, urls will not get rewritten, even with rewrite_css enabled:

https://www.modpagespeed.com/fallback_rewrite_css_urls.html?PageSpeed=on&PageSpeedFilters=debug,rewrite_css

But when you enable fallback_rewrite_css_urls, the unparseable css does get rewritten:

https://www.modpagespeed.com/fallback_rewrite_css_urls.html?PageSpeed=on&PageSpeedFilters=debug,rewrite_css,fallback_rewrite_css_urls

Otto

--

You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Saumya Majumder]

Thanks. I mistakenly pasted 3 times here. Didn't intended to do so. Thanks for your reply on this.