Confused on how to setup pagespeed with CDN.

[Jason Kim]

Hello, I manage a high traffic site with about 500TByte transfer per month using nginx caching proxy for serving static/image resources.

The origin website is hosted on an nginx server with three domains on ip ORIGINIP

www.domain.com - PHP/html requests, not cached, user browsers connect to this

static.domain.com - js/css requests, only for CDN

image.domain.com - media request, only for CDN

The nginx caching proxy is hosted on an nginx server with two domains on ip CACHINGIP

static.domain.com - js/css requests, user browsers connect to this

image.domain.com - media request, user browsers connect to this

www.domain.com points to ORIGINIP

static.domain.com points to CACHINGIP

image.domain.com points to CACHINGIP

When a browser requests something from static.domain.com or image.domain.com, the nginx caching proxy requests it from ORIGINIP and then caches it, afterwards sends the cached response for the same requests.

User browser -> www.domain.com at ORIGINIP html ->  static.domain.com or image.domain.com resources at CACHINGIP -> CACHINGIP server requests static.domain.com or image.domain.com resources from ORIGINIP and saves request for caching -> CACHINGIP  sends static.domain.com or image.domain.com resources to user browser

I wish to save bandwidth by optimizing the images automatically using pagespeed.

However, all my tests have failed to do so.

A few questions

I wish to only install pagespeed in ORIGINIP server, as it is much more powerful.

1)

What settings should I look at when doing this?

I have tried many setting I looked up on google but failed.

2)

ORIGINIP has about 4 TB of images stored on it with about 64 GB of ram.

What would be an appropriate starting point for

pagespeed FileCacheSizeKb

pagespeed FileCacheCleanIntervalMs

pagespeed FileCacheInodeLimit

Thank you.

Have a nice day.

[Jason Kim]

Some additional information:

I tried options

        pagespeed RewriteLevel OptimizeForBandwidth;

        pagespeed EnableFilters remove_comments,collapse_whitespace;

My tests showed the following problems

css/js files were processed with "remove_comments,collapse_whitespace" when deleting the cache multiple times at CACHINGIP, and the css/js URL was the same.

I thought they would be renamed with .pagespeed automatically which would have made the new modified files cached automatically.

None of the image files were also optimized.

[Otto van der Schaaf]

In OptimizeForBandwidth mode, you won't see any renaming of urls to the .pagespeed. variants.

(From https://modpagespeed.com/doc/optimize-for-bandwidth: "This option is suitable for use in a root configuration at a hosting service, CDN, or multi-site setup, in conjunction with InheritVhostConfig. In this mode, PageSpeed does not alter HTML at all.

Enabling rewrite_images and extend_cache should give you optimized images with long cache TTL's, rewritten to .pagespeed. versions which include a content hash in the url:

https://modpagespeed.com/doc/filter-image-optimize

https://modpagespeed.com/doc/filter-cache-extend

Cache requirements are going to depend on the configuration you set, and wether you need to optimize a hot set or everything.

Ballpark I think you may need to assign 6TB to 8TB of storage space (for the originals + the optimized versions, possibly in multiple variants for user-agents with different capabilities) to enable the FileSystemCache to store the complete set.

Otto

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[highclass99]

Thank you for your reply.

I forgot but I did already change OptimizeForBandwidth, after I realized it was for in place optimization.

The following is my current config:

pagespeed On;

pagespeed FileCachePath "/home/pagespeed";

pagespeed FileCacheSizeKb 10000000000;

pagespeed FileCacheCleanIntervalMs 3600000;

pagespeed FileCacheInodeLimit 500000000;

pagespeed PreserveUrlRelativity on;

#pagespeed AddResourceHeader "Access-Control-Allow-Origin" "*";

pagespeed RewriteLevel PassThrough;

pagespeed EnableFilters remove_comments,collapse_whitespace,recompress_images,convert_to_webp_lossless,convert_to_webp_animated;

pagespeed Domain mydomain.com;

pagespeed Domain m.mydomain.com;

pagespeed Domain www.mydomain.com;

pagespeed Domain static.mydomain.com;

pagespeed Domain images.mydomain.com;

pagespeed LoadFromFile "https://image.mydomain.com/files" "/home/mydomain/www/files";

pagespeed LoadFromFile "https://static.mydomain.com/files" "/home/mydomain/www/files";

pagespeed LoadFromFileRuleMatch Disallow \.php$;

pagespeed FetchWithGzip on;

pagespeed UseNativeFetcher on;

pagespeed LRUCacheKbPerProcess     20480;

pagespeed LRUCacheByteLimit        81920;

pagespeed CreateSharedMemoryMetadataCache "/dev/shm/ngx_shared" 640000;

pagespeed CssInlineMaxBytes 2048;

pagespeed JsInlineMaxBytes 2048;

No images/js/css seem to be rewriten.

What should I try out?

And are there any other suggestions for optimizing performance?

Thank you.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/ix5qU9fVMMo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[highclass99]

Note:

I also added "extend_cache" option to the above "pagespeed EnableFilters" after reading the above reply.

[highclass99]

Additional notes

I changed to following options, because curl -H "PageSpeedFilters: +debug" showed that domain wasn't authorized properly in the original config:

        pagespeed Domain "https://static.mydomain.com";

        pagespeed Domain "https://image.mydomain.com";

        pagespeed Domain "http://static.mydomain.com";

        pagespeed Domain "http://image.mydomain.com";

        pagespeed LoadFromFile "https://image.mydomain.com/files" "/home/mydomain/www/files";

        pagespeed LoadFromFile "https://static.mydomain.com/files" "/home/mydomain/www/files";

        pagespeed LoadFromFile "http://image.mydomain.com/files" "/home/mydomain/www/files";

        pagespeed LoadFromFile "http://static.mydomain.com/files" "/home/mydomain/www/files";

However no urls are rewritten and I am seeing no results.

[highclass99]

Sorry about the multitude of posts.

I noticed after adding http and https to pagespeed Domain + LoadFromFile

The images ARE being rewritten.

It just took a while for it to work.

My css/js files are still not being rewritten. I hope that is also a time issue.

A few additional questions:

1.

I am curious about whether settings such as

pagespeed FileCacheSizeKb 10000000000;

pagespeed FileCacheCleanIntervalMs 3600000;

pagespeed FileCacheInodeLimit 500000000;

are viable performance wise, and also whether

2.

How much hard drive IO load would

        pagespeed FileCacheCleanIntervalMs 3600000;

cost me?

3.

What problems, how much space would occur if I set FileCacheCleanIntervalMs  -1

4.

Also whether settings such as

pagespeed UseNativeFetcher on;

pagespeed LRUCacheKbPerProcess     20480;

pagespeed CreateSharedMemoryMetadataCache "/home/pagespeed_shared" 640000;

is viable performance wise.

5.

Lastly would adding a memcache layer be helpful?

How large would a memcache layer being needed?

Thank you.

I'll post my bandwidth graph usage after this test. Right now because of the new replaced images, bandwidth usage is going through the roof on the origin server :(

Hope it decreases.