Vary-Agent

[AnInterestedPerson]

Based on documentation, I am not ure I grasped handling of Vary-headers correctly.

Basics:
pagespeed performs some optimizations, which apply to a few browsers only respectively handles different browsers (webp, but also which font is requested by which browser based on the css,..)

Questions:
1. Does pagespeed save the handling of different agents without RespectVary (aka pagespeed RespectVary off;) set? How long is it saved?

2. In the documentation it says, that with RespectVary on: "Please note that turning on this option will disable optimization of any resources with Vary headers, apart from Vary: Accept-Encoding." Lookign at context, this sentence is wrong? Wouldnt pagespeed just optimize all thsoe ressources nonetheless, but save a lot more info to cache to serve the different agents?? (in case of vary agent)?

3. Right now I have the impression, that pagespeed either discards optimzation strategies for an agent "quite fast" or the "strategies" of the last visitor are a viable start?Its plain irritating to see a fully otimized site and suddenly all optimizations are gone within 10 mins.

TIA for the answers!

Martin

[AnInterestedPerson]

PS: any additional advice on how to handle different User-Agents? a. with nginx fastcgi cache, b. with a seperate nginx-edge-proxy with ssl terminators?

[Joshua Marantz]

PageSpeed doesn't optimize resources that it can't cache; there's no such thing as "optimize but don't cache".  Usually the first optimization is done in the background and stored in cache, and only already-cached resources are served to clients.

Also note that RespectVary is superseded by the more general AllowVaryOn: https://www.modpagespeed.com/doc/reference-image-optimize#AllowVaryOn

Maybe the explanation there can shed some light, especially with regard to how PageSpeed deals with browsers that have Accept:image/webp and those that don't.  Specifically, it includes the Accept header in its internal cache keys, so it can cache and store optimized webp for Chrome and still optimize images without webp for browsers that do not accept webp.

On Fri, Feb 9, 2018 at 5:53 AM, AnInterestedPerson <AnInteres...@kunden-besser-informieren.de> wrote:

PS: any additional advice on how to handle different User-Agents? a. with nginx fastcgi cache, b. with a seperate nginx-edge-proxy with ssl terminators?

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.
Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[AnInterestedPerson]

I have to rephrase my issue.

I am right now through 3 days of testing with pagespeed - actually more but nm. Several points keep me puzzled, where it seems pagepeed behaves almost unpredictable:

When testing I am going for filters and options step by step. Three pagespeed features are most important to me.

1. Resize to rendered image dimensions: When testing, pagespeed performs just fine and suddenly upon a later reload the resized images are no longer shown. 10-20 reloads later those resized images are taken again. So basically I have one fully optimized load of 20. So far I assumed its maybe by another browser (Agent) and that pagespeed somehow "forgot" to use the resized images.
   
2. Combine JS and CSS: pagespeed has them all combined and some reloads later (testing another filter and option - something completly different from js and css) I again have them not combined. Takes a lot of reloads to have them combined again. Again somethign like 1 perfectly optimized view in 20.
3. Apart from 1. the sites behave better *speedwise* when all images have been processed by pagespeed, so "ImageLimitOptimizedPercent 100" is cool, but... not all images are proessed by pagespeed. Sometimes it takes almost an hour on a machine without load that those images are processed? Also it seems pagespeed doesnt process images with a blank in the URL (which is a pita since its not on me to define URLs but a third party).

There are much more similar things happening, where a perfectly fine configuration has one filter added (which really shouldnt interfere) and all optimizations are reset.

--- I attach a pagespeed config from current tests. Maybe I am just too much into it, so I dont see my error... ---

#
# General
#
        pagespeed On;
        pagespeed RewriteLevel CoreFilters;
        pagespeed FileCachePath "/var/cache/ngx_pagespeed/";
        pagespeed FileCacheSizeKb            204800;
        pagespeed FileCacheCleanIntervalMs   3600000;
        pagespeed FileCacheInodeLimit        500000;
        pagespeed LRUCacheKbPerProcess     8192;
        pagespeed LRUCacheByteLimit        16384;
        pagespeed CreateSharedMemoryMetadataCache "/var/cache/pagespeed/" 51200;
        pagespeed ShmMetadataCacheCheckpointIntervalSec 300;
        pagespeed EnableCachePurge on;
        pagespeed PurgeMethod PURGE;

#
# Basics
#
        pagespeed LogDir "/var/log/pagespeed";
        pagespeed XHeaderValue "Comanche";
        pagespeed PreserveUrlRelativity on;
        pagespeed StaticAssetPrefix /CDN/;
        pagespeed MaxSegmentLength 250;
        pagespeed EnableFilters remove_comments;
        pagespeed EnableFilters extend_cache;

#
# Admin Path
#
        pagespeed AdminPath /pagespeed_admin;
        pagespeed GlobalAdminPath /pagespeed_global_admin;

#
# Images
#
        pagespeed EnableFilters lazyload_images;
        pagespeed EnableFilters convert_to_webp_lossless;
        pagespeed EnableFilters recompress_images;
        pagespeed EnableFilters resize_rendered_image_dimensions;
        pagespeed EnableFilters rewrite_images;
        pagespeed EnableFilters sprite_images;

        pagespeed AllowVaryOn Auto;
        pagespeed CssImageInlineMaxBytes 5000;
        pagespeed ImageInlineMaxBytes 5000;
        pagespeed ImageJpegNumProgressiveScans 10;
        pagespeed ImageLimitOptimizedPercent 100;
        pagespeed ImageLimitResizeAreaPercent 100;
        pagespeed ImageResolutionLimitBytes 128000000;
        pagespeed WebpAnimatedRecompressionQuality 70;
        pagespeed WebpRecompressionQuality 70;

Am Freitag, 9. Februar 2018 14:33:33 UTC+1 schrieb Joshua Marantz:PageSpeed doesn't optimize resources that it can't cache; there's no such thing as "optimize but don't cache".  Usually the first optimization is done in the background and stored in cache, and only already-cached resources are served to clients.

[Otto van der Schaaf]

If may be worth checking https://www.modpagespeed.com/doc/faq#varying-results

With regard to spaces in the url, I'd be surprised if that is why these aren't optimized.. 

Did you see https://www.modpagespeed.com/doc/faq#not-rewriting

Otto

--

You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[AnInterestedPerson]

Thank you for the heads up Otto and Joshua. I actually know the documentation. Yet, I was not that aware on how mod_pagespeed applies filters. Esspecially when optimization stops or filters disabling another.

Bottom line for me: it becomes more complex for my case than I anticipated.

Suggestion: Four additional fully commented master files, two for mod_pagespeed and two for ngx_pagespeed containing all viable options and filters would help a lot (for each server one file for "preferably" server-wide configurations and one for "preferably" vhost-specific configuration). Am aware that there is no clear difference for each option/filter - still it would be a starting point. Imho after these tests the Apache pagespeed.conf alone doesnt cut it.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

resize_rendered_image_dimensions is a particularly complex filter; it requires accurate, possibly conflicting beacon results coming from a variety of clients.  If CSS sizes images based on a percentage of the screen width, the rendered dimensions coming from different clients will all conflict and the filter will do a lot of work but not be able to use any of it.  So it must be applied with care.

And, as with all features relying on client-side beacons (notably prioritize_critical_css), a page which is rarely visited will not be optimized well, as the previous beacon results will expire from cache before they are used.

At one point we had brainstormed on how to rebuild it much more robustly, so that it didn't require beacons to work at all, but used client-side JavaScript to determine the correct size image based on the current page layout, and request that from the server.  But this was never pursued.

Your suggestion is a good one, but in large part that's the role of the RewriteLevel directive:

  pagespeed RewriteLevel CoreFilters;                       -- what most people should use, that provides good speedup with minimal risk of issues

  pagespeed RewriteLevel OptimizeForBandwidth:    -- super conservative setting for proxies where there has to be near-zero risk of issues

Most issues in CoreFilters arise from JavaScript on the page being sensitive to URL syntax on the page.  A user that is willing to test their page can still get value from CoreFilters by disallowing troublesome URLs with "pagespeed Disallow <wildcard-pattern>;"  A user that cannot tolerate any breakage should go to OptimizeForBandwidth.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsubscri...@googlegroups.com.

Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[AnInterestedPerson]

With AllowVaryOn User-Agent, the resize_rendered_image_dimensions filter would use all its work done so far - in case I understood you correctly? Similar with other filters. So to make it more "robust",.. wouldtn it be sufficient to just store the results of the beacon for a much longer time?

My needs/wishlsit for pagespeed atm:
1. I am actually putting together those 2 (a general and the vhost specific one) of those files for nginx. Yet right now only for my specific needs. Still having those files in general for Apache and nginx would be imho a blast. At least Id love it. (And for christmas please the DIN A0 poster outlining pagespeed option/Filter logic and dependancy).
2.  On a dual stack environment it would be cool to synchronize pagespeed cache with nginx proxy_cache.
3. I believe you are too cautious with some filters. Esspecially css and js rewrites. A "force filter" would help in some cases, thereby ignoring most stop-filter conditions.
4. I miss simple filters. Something where i can plain define what is above the fold. Just by its file name and URL. Plain old school. Or where I can define image resize by class, maybe even id.Or modifying max-age of omptimized ressources. While setting max-age 0 makes overall sense on optimized html, it doesnt make sense on microcached sites, which expect drastic burst usage. Yes I can configure it in nginx nonetheless, yet at a certain amount of conditins to do so is just a pita. Simpyl since someone having to handle one small machine can do it, but.. how does that someone" handle 1,000 machines and more?

[Joshua Marantz]

My general sense is that in the universe of nginx and apache users, there are a wide variety of specific needs.  We've tried to capture two patterns we think are very common with the RewriteLevel concept (as opposed to a file enumerating the filters in those levels).

It sounds like you are on the aggressive side; so start with CoreFilters and keep adding from there.

In terms of more advanced features, there's no definitive set I can think of that would be particularly common.  One common pattern is that some aspects of mod_pagespeed's optimizations can be incorporated into a build flow, and others are more difficult.  So users may wish to cherrypick which advanced optimizations to use.

I'm not sure what you mean by "too cautious"?  Are you talking about something like abstaining from combining css elements with IDs?

RE html caching: you can override that with 'ModifyCachingHeaders off' -- just be sure you are not doing user-agent-specific optimizations.  OptimizeForBandwidth is good for this.

[AnInterestedPerson]

commented your answer:

Am Montag, 12. Februar 2018 03:46:09 UTC+1 schrieb Joshua Marantz:

My general sense is that in the universe of nginx and apache users, there are a wide variety of specific needs.

So true. Add an unwillingness of a large part to dive deeper into it to get it right..
 

We've tried to capture two patterns we think are very common with the RewriteLevel concept (as opposed to a file enumerating the filters in those levels).

Not sure if I got your reference right. I had several occassions where css wasnt rewritten by pagespeed. Apparently due to a "stop" condition by pagespeed, where it is considered too "risky" to rewrite with that specific css-statement inside. Yet I was sure it is safe in that specific case to still rewrite nonetheless and ignore those STOP-conditions for this file. Thats what I meant with "too cautious". Just that I would like it personally to have a "force filter" switch, basically ignoring stop-conditions, but instead having e.g. a Warning in Log (preferably only once daily ;) )
 

It sounds like you are on the aggressive side; so start with CoreFilters and keep adding from there.

Exactly what I do, except I grabbed an Excel sheet noting results down. And it grows faster than I thought.
 

In terms of more advanced features, there's no definitive set I can think of that would be particularly common.  One common pattern is that some aspects of mod_pagespeed's optimizations can be incorporated into a build flow, and others are more difficult.  So users may wish to cherrypick which advanced optimizations to use.

 resize_rendered_image_dimensions to me is a godsend. Basically I pay with cache disc storage and some RAM for speed. Imho one of the coolest filters you have. To me its common and imho should be for many, many more.

Similar if you could do a "font-filter": E.g in the case of Font Awesome making a subset containing only the characters used would reduce Font Awesome files on regular by 90%+. 68KB savings on A LOT of pages for selecting only the characters used. There are more options with fonts in general where pagespeed could drop 20-40% of the font file content without impacting the site. Three (five) custom fonts (e.g. Font Awesome + (heading font + body text font) * 2 variants = 5 font files)  would save every web page in such a common setting app. 120KB+. Any solution for that I looked for myself can not really be automated for a large number of sites. So thatll be another potential common yet complex filter, where pagespeed could excel.

 

I'm not sure what you mean by "too cautious"?  Are you talking about something like abstaining from combining css elements with IDs?

Elaborated about it above.
 

RE html caching: you can override that with 'ModifyCachingHeaders off' -- just be sure you are not doing user-agent-specific optimizations.  OptimizeForBandwidth is good for this.

Just to be sure, maybe I am not that fit in documentation:
1. I always thought CoreFilters, Optimize for Bandwidth,.. are just filter sets to start off, which have a different aggressiveness to start with by means of the filter choice only.
2. Or is it a difference if I have two filter sets, but configure them to have the same filters and options? Do they work exactly the same then or is there a difference? Or in other words: two sites, which show the exact same filter sets and option settings except one is based on CoreFilters, the other on OptimizeForBandwidth - Do these sites behave identically or not?

TIA

[AnInterestedPerson]

see below

Am Sonntag, 11. Februar 2018 17:04:40 UTC+1 schrieb Otto van der Schaaf:

If may be worth checking https://www.modpagespeed.com/doc/faq#varying-results

thx

With regard to spaces in the url, I'd be surprised if that is why these aren't optimized.. 

Did you see https://www.modpagespeed.com/doc/faq#not-rewriting

Yes saw it. Right now I got an example with a picture (well and some JS/css-files on that site too). Pagespeed works on that site. Recompress image is enabled, ImageLimitOptimizedPercent is 100. Still that image is untouched.
 

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Otto van der Schaaf]

So that image with a space in the url also has an Expires: header which implicates it expires right away.

What happens when you change that?

Also, you may want to check out https://www.modpagespeed.com/doc/domains#url-valued-attributes,

perhaps adding li/data-thumb here makes sense.

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

Visit this group at https://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.