optimising mod_pagespeed for a site with a focus on image quality (and speed)

[Luke Kenneth Casson Leighton]

hi folks i've recently discovered mod_pagespeed and the fact that it
can handle all of the complexity of optimisation which would otherwise
*massively* complicate managing this site is... just... well, thank
you doesn't quite cover it strongly enough :)

i am managing a site which is very very simple, and there are a few
things i could do with some assistance and advice. the site is
horseboxseller com and without mod_pagespeed it's a 4.5 second load
time; with the settings below it's between ONE and three seconds for
most pages - i'm continually changing various settings in order to
experiment, webmaster tools "speed suggestions" is currently reporting
today's statistics as a bewildering 0.61 seconds but i don't quite
believe it (small dataset?) - we'll see later in the day when there's
more data.

anyway it's quite important that images be presented in the best
possible light, but it's also important that the site be kept very
very simple. this is horseboxes being sold, so a no-nonsense approach
is important so that customers feel comfortable.

now, one of the problems that i have is that say on the 7.5 ton page
the 241x160 image of the red horsebox in the banner is sometimes
"inlined" to be included in the page... all good... except it looks
really really blurry. the original is at
/images/example-horsebox-7.jpg and it's already been resized to
241x160: further compression (and inlining) is probably what's causing
the artefacts.

i'm just going to try an experiment by substituting the 640x480 gif in
its place (so you may see a change by the time you read this), but the
jpg will remain if people would like to look at it.

also checking right now with pagespeed insights it says "Compressing
and resizing example-horsebox-H-xxxx-pagespeed.jpg could save XX%" -
that's on the /horseboxes-for-sale/hgv/ page. so that one's not
getting properly compressed with the size of the image in it... *but*
when i view the page manually it has also been "inlined" to such an
extent that the quality is questionable: blurring to the right of the
word "SCANIA" and you can see artefacts down the right-hand rear side
just where it turns to sky.

then there is /horseboxes-for-sale/detail/1735/ where the black lines
on the side clearly have artefacts on the red paint, yet some of the
other ones (interior, rear) look ok... yet all of them are compressed.
and aren't webp (despite convert_jpeg_to_webp, below).

so i'm... confuuUused :) i'm doing too much in the way of arbitrary
modifications and it's a live site. i could really use some advice on
how to get this right first time, as well as some outside perspective
on what looks "acceptable" image-wise... yet still meets the pagespeed
insights criteria.

all help greatly appreciated with this fantastic time-saving module.

l.


<IfModule pagespeed_module>
ModPagespeed on
ModPagespeedInheritVHostConfig on
AddOutputFilterByType MOD_PAGESPEED_OUTPUT_FILTER text/html
ModPagespeedFileCachePath "/var/cache/mod_pagespeed/"
ModPagespeedLogDir "/var/log/pagespeed"
ModPagespeedSslCertDirectory "/etc/ssl/certs"
ModPagespeedMemcachedServers localhost:11211
ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 6400
ModPagespeedEnableFilters prioritize_critical_css
ModPagespeedEnableFilters collapse_whitespace
ModPagespeedEnableFilters rewrite_images
ModPagespeedEnableFilters extend_cache
ModPagespeedEnableFilters local_storage_cache
ModPagespeedEnableFilters defer_javascript
ModPagespeedEnableFilters sprite_images
ModPagespeedEnableFilters convert_png_to_jpeg,convert_jpeg_to_webp
ModPagespeedEnableFilters remove_comments
ModPagespeedEnableFilters resize_rendered_image_dimensions
ModPagespeedRewriteDeadlinePerFlushMs 100
ModPagespeedFileCacheInodeLimit 500000
ModPagespeedRewriteRandomDropPercentage 90
ModPagespeedImageRecompressionQuality 25
ModPagespeedMinImageSizeLowResolutionBytes 4096
ModPagespeedEnableFilters canonicalize_javascript_libraries
<Location /pagespeed_admin>
Order allow,deny
Allow from localhost
Allow from 127.0.0.1
SetHandler pagespeed_admin
</Location>
<Location /pagespeed_global_admin>
Order allow,deny
Allow from localhost
Allow from 127.0.0.1
SetHandler pagespeed_global_admin
</Location>
ModPagespeedStatisticsLogging on
ModPagespeedMessageBufferSize 100000
ModPagespeedEnableFilters add_instrumentation
<Location /mod_pagespeed_beacon>
SetHandler mod_pagespeed_beacon
</Location>
</IfModule>

[Joshua Marantz]

ModPagespeedImageRecompressionQuality 25 is very low.  Can you leave this at the default for now?  Our defaults are designed for your use-case: reasonable looking images that display reasonably fast.

FYI, inlining is a lossless operation.  It mainly trades off network round-trip time vs the potential for caching the image bytes.  For very small images it's usually a win to inline them because the network request overhead is non-zero.  For now just leave the default inlining thresholds.

Of course you will get more inlining with aggressive compression -- because the images will shrink below the byte threshold, but the images will look like crap.

You can tweak compression level a little bit but 25 is way too low.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDwv182JuagzmdDrtRoQTbwkRRwqrfZm%3DWN_7FndRAN%2BnA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Kenneth Casson Leighton]

hiya joshua, thanks for replying, really appreciated.

On Mon, Jul 31, 2017 at 2:13 PM, 'Joshua Marantz' via
mod-pagespeed-discuss <mod-pagesp...@googlegroups.com> wrote:

> ModPagespeedImageRecompressionQuality 25 is very low. Can you leave this at
> the default for now?

sure - i can do that... however i know it has an adverse affect on
the pagespeed insights score. that's been my current "benchmark for
taking (often random) action".

> Our defaults are designed for your use-case:
> reasonable looking images that display reasonably fast.

understood.

> FYI, inlining is a lossless operation. It mainly trades off network
> round-trip time vs the potential for caching the image bytes. For very
> small images it's usually a win to inline them because the network request
> overhead is non-zero. For now just leave the default inlining thresholds.

ok done.

> Of course you will get more inlining with aggressive compression -- because
> the images will shrink below the byte threshold, but the images will look
> like crap.

:) oh ok so wait.... ModPagespeedMinImageSizeLowResolutionBytes is
actually a *Max* parameter? i am probably thinking of this the wrong
way round... :) regardless, the logic goes, "if image size after
compression falls below MinImageSizeLowResolutionBytes then inline
it", but you're saying that it's not the *inlining* that has an effect
on the image, it's the compression.

ok. i get it.

> You can tweak compression level a little bit but 25 is way too low.

a stack of images on the page come up and drop the pagespeed score,
i'll put it back to default then see what crops up, and come back here
with specific examples, if that's ok?

l.

[Luke Kenneth Casson Leighton]

ok so using the default compression resulted in:

"Compressing horseboxseller/images/horsebox_banner3.jpg could save
4.2KiB (14% reduction)".

... which i've now renamed to banner4 and the inline css is still in
the cache, doh :) so it's not being picked up yet by either of the
two browser i'm using, or pagespeed... yet. give it some time.

l.

[Joshua Marantz]

If you are using mod_pagespeed's default image settings, and PSI is making suggestions for further reduction, there are a few things that might be causing it.  Off the top of my head

1. PSI might be seeing a not-fully-optimized view of the page because mod_pagespeed optimizes in the background, and the image optimizations are specific to properties of the browser.
2. PSI can be a little aggressive in its expectation of how resizing works.  mod_pagespeed can resize image based on simple inline size attributes, but cannot resize based on CSS classes or attributes that compute image-width as a percentage of viewport width.
3. There may possibly be a bug in PSI around this, judging by the amount of traffic on this top in pagespeed-insights-discuss

For now I'd focus on getting good performance measured using webpagetest.org.  If you are using mod_pagespeed your images are likely being served reasonably well.  You might still want to look at other suggestions made in PSI.  But keep in mind they are suggestions and may not be appropriate or workable on all sites.

Two other filters in mod_pagespeed you may want to look at are inline_preview_images and lazyload_images.  See if they work well for your site.  ModPagespeedMinImageSizeLowResolutionBytes is actually designed for inline_preview_images, so it will only take effect if you enable that filter.

l.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDyM%2BGdoLBOH80PkOHg1tfNLY1fotOMR%2Be6yyaMNp%3DNg7g%40mail.gmail.com.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Mon, Jul 31, 2017 at 3:47 PM, 'Joshua Marantz' via
mod-pagespeed-discuss <mod-pagesp...@googlegroups.com> wrote:
> If you are using mod_pagespeed's default image settings, and PSI is making
> suggestions for further reduction, there are a few things that might be
> causing it. Off the top of my head
>
> PSI might be seeing a not-fully-optimized view of the page because
> mod_pagespeed optimizes in the background, and the image optimizations are
> specific to properties of the browser.
> PSI can be a little aggressive in its expectation of how resizing works.
> mod_pagespeed can resize image based on simple inline size attributes, but
> cannot resize based on CSS classes or attributes that compute image-width as
> a percentage of viewport width.

horsebox_banner4.jpg is a css background image with "width: 960px"
and "min-height: 220px". there is nothing out-of-the-ordinary yet it
is still not being optimised. i'm going to try renaming it to banner5
just to make absolutely sure.

> There may possibly be a bug in PSI around this, judging by the amount of
> traffic on this top in pagespeed-insights-discuss

:)

> For now I'd focus on getting good performance measured using
> webpagetest.org.

ok that's a really useful resource, thank you.

> If you are using mod_pagespeed your images are likely
> being served reasonably well. You might still want to look at other
> suggestions made in PSI. But keep in mind they are suggestions and may not
> be appropriate or workable on all sites.

yehyeh, such as twitter and even google-analytics javascript being
reported as requiring compression *and* cacheing....

> Two other filters in mod_pagespeed you may want to look at are
> inline_preview_images and lazyload_images. See if they work well for your
> site. ModPagespeedMinImageSizeLowResolutionBytes is actually designed for
> inline_preview_images, so it will only take effect if you enable that
> filter.

ahh ok. ok so i enabled them both yesterday. interestingly, if i
have local_storage_cache enabled absolutely nothing works. only when
switching off local_storage_cache does the lazyload start to work (i
haven't worked out yet how to check inline_preview_images).

i left local_storage_cache on overnight, checked with view-source
this morning, no sign of any 1-pixel images on any below-the-fold
content. i'll re-disable local_storage_cache and recheck it.

l.

[Luke Kenneth Casson Leighton]

yep confirmed, /horseboxes-for-sale/detail/1769/ within 2 minutes and
8 or 9 page-refreshes, after disabling local_storage_cache,
lazyload_images kicked in.

one very interesting thing, when i *re-enabled* local_storage_cache
last night before retiring, i noticed that the source code did
actually have signs of using both features. however by this morning
all signs of local_storage_cache usage had been erased from what
mod_pagespeed outputs (exactly the same page being tested).

l.

[Luke Kenneth Casson Leighton]

ooo veryverybad: i apologise but i had to act very quickly: i was
getting consistent *four second* delays on google chrome but not
firefox, on certain arbitrary but always the same specific pages. if
it was both browsers i would have gone "huh this must be apache2 or
might be networking or something". initially it started happening to
/horseboxes-for-sale/7.5-ton/ but when it hit the home page as well
that was it, goodbye metadata store in /var/cache/mod_pagespeed,
restart memcached and the problem's "gone".

merely restarting apache2 did *not* make this problem go away, hence
the drastic action of restarting memcached as well as blowing away the
metadata store.

version info:

dpkg -l | grep pagespeed
ii mod-pagespeed-stable 1.12.34.2-r0 i386
Apache 2 module to optimize web content.

joshua i'll keep an eye on things but i apologise i had to act fast.
any suggestions on what data to record if it happens again?

l.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Wed, Aug 2, 2017 at 4:28 PM, Luke Kenneth Casson Leighton
<lk...@lkcl.net> wrote:
> ooo veryverybad: i apologise but i had to act very quickly: i was
> getting consistent *four second* delays on google chrome but not
> firefox, on certain arbitrary but always the same specific pages. if

> joshua i'll keep an eye on things but i apologise i had to act fast.
> any suggestions on what data to record if it happens again?

ah ha!

ok i might have a repro case. basically you don't wait for the page
to finish loading, you hit F5 or Ctrl-F5 repeatedly in the middle of
loading, including before the browser has had a chance to even start
loading the page let alone display it.

this seems to be enough to do something horrible to the cache...
let's check that....

ctrl-f5, f5, ctrl-f5, f5, ctrl-f5, f5, f5.... ok successfully screwed
up one page... :) 4 second delay is back....

let's remove the shared memory cache from pagespeed.conf.... restart
apache2....nope, no effect.

ok let's just try restarting memcached... ah ha! okaaay.... so it's
memcached getting its knickers in a twist..

dpkg -l | grep memcache
ii memcached 1.4.21-1.1+deb8u1 i386
high-performance memory object caching system

so let's do that again... ctrl-f5, f5, ctrl-f5 ... screwed... :)

now let's check another page... nope, other pages all fine.

go back to the screwed page... still screwed. *SLOWLY* do
ctrl-f5.... still screwed. let page load.... f5... still screwed.
restart memcached.... speed's back!

ok so this is definitely memcached-related.

i'll see if i can repro this locally as i can't mess with this live
server. also i'll try at some point just... disabling the use of
memcached.

l.

[Joshua Marantz]

Can you try disabling local_storage_cache?  I have two concerns about it, reinforced given this thread:

• I don't think our cross-testing between that feature and others is sufficient. It wouldn't surprise me if there were cross-compatibility issues with the lazy-load filter.
  
• local_storage implementation in browsers may cause delays loading a page as it loads in all the local_storage data associated with a domain.

Part of the issue, generally, is that the Web Platform people have focused much more on ServiceWorker as a technology to push forward, and have de-emphasized local_storage relative to that.

l.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDyD23r%2BL98TyoKKxTKr1zzOF%3DYm5r5%3Dug1Vdun64xJgwg%40mail.gmail.com.

[Luke Kenneth Casson Leighton]

On Wed, Aug 2, 2017 at 5:10 PM, 'Joshua Marantz' via
mod-pagespeed-discuss <mod-pagesp...@googlegroups.com> wrote:
> Can you try disabling local_storage_cache?

already done, back a few days ago after experimenting and noticing
that it was interfering with lazy-load. left it off.

> I have two concerns about it,
> reinforced given this thread:

yehyeh.

> I don't think our cross-testing between that feature and others is
> sufficient. It wouldn't surprise me if there were cross-compatibility issues
> with the lazy-load filter.

i'm slightly surprised by that. i would have expected there to be
like a chain between the filters, pretty much transparent, just using
the output from one as the input to the next.

thus, we can surmise: if a developer implements local storage
cacheing *themselves*... what then happens?? does their website
*break* if they enable the lazy-load filter? because that would be
bad :)

> local_storage implementation in browsers may cause delays loading a page as
> it loads in all the local_storage data associated with a domain.

hmmm.... that defeats the object of the exercise, ehn? :) but i
wasn't seeing anything large when i was using it (and am not using it
any more).

anyway: so: no to local_storage_cache. of much bigger concern is the
*massive* delays (4 seconds) which are related to memcached. bear in
mind that's with local_storage_cache DISABLED.

btw i did try the lmdb version of memcached: it was... interesting.
mod_pagespeed *completely* failed to be able to use it. as in, when i
enabled debug info pagespeed was showing timeouts to socket 11211 (!)
mind you i am on a 32-bit server and lmdb on 32-bit machines is no
longer supported. i have yet to try on a 64-bit machine, but it's a
datapoint point to "problems with pagespeed interacting with
memcached" nonetheless.

i wasn't able to replicate the setup on my laptop by pressing ctrl-f5
and f5 on an ongoing basis, i suspect that there's more going on here:
several incoming connections required etc. etc.

l.

[Luke Kenneth Casson Leighton]

On Fri, Aug 4, 2017 at 8:15 AM, Luke Kenneth Casson Leighton
<lk...@lkcl.net> wrote:

> i wasn't able to replicate the setup on my laptop by pressing ctrl-f5
> and f5 on an ongoing basis, i suspect that there's more going on here:
> several incoming connections required etc. etc.

ok i have another datapoint in triaging this issue: squid proxy cache
was being used on the google chrome browser that arbitrarily (but
consistently) adds a four second delay to page-load times.

obviously when trying to replicate this on my laptop (127.0.0.1) i
did not think to add squid proxy cache in between so replication was
unsuccessful.

however when BYPASSING the squid proxy cache, google chrome ALSO DOES
NOT HAVE THE SAME FOUR SECOND DELAY.

remember, that delay goes away for a few minutes if i kill and
restart memcached.

so, just to reiterate, the scenario is as follows:

* start memcached
* start mod_pagespeed
* start apache2, configured to use "persistent connections"
* start squid proxy cache, configured in the stock (default) configuration.
* start google chrome configured to use squid proxy cache
* go to any pagespeed-optimised page
* repeatedly press F5 and CTRL-F5 on a random constant basis for at
least one minute
* stop doing that. wait fifteen seconds
* press F5 and time the page: it will have a 4 second delay
* go to ANY OTHER PAGE. it will NOT have a 4 second delay
* stop memcached
* start memcached
* go back to the FIRST page. it will NO LONGER HAVE A 4 SECOND DELAY
* wait for a few minutes for other users to visit the page
* press F5. the page will ALMOST CERTAINLY have its 4 second delay back
* repeat from step "stop memcached" until this is true.
* configure google chrome to NO LONGER use squid
* go back to the FIRST page. it will NO LONGER HAVE A 4 SECOND DELAY

this is a consistently 100% reproducible scenario.

l.

[Otto van der Schaaf]

I think a capture of the issue with:

- chrome://net-internals/ of a delayed response may be helpful.

- Plus verbose logs from squid for the same (if possible), it would be good to see what is going on between squid and the pagespeed enabled server in terms of timings and request/response headers.

- Plus a log of memcached in verbose mode

Maybe with those, we can figure out what happens.

Otto

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDwem1%3D%3DfG_FZ9%3D1rZE1HADAh3_ivz61sd6irtXg9jzvuQ%40mail.gmail.com.

[Luke Kenneth Casson Leighton]

On Sun, Aug 6, 2017 at 10:48 AM, Otto van der Schaaf <osc...@we-amp.com> wrote:

> I think a capture of the issue with:
> - chrome://net-internals/ of a delayed response may be helpful.
> - Plus verbose logs from squid for the same (if possible), it would be good
> to see what is going on between squid and the pagespeed enabled server in
> terms of timings and request/response headers.
> - Plus a log of memcached in verbose mode

good idea, otto.

> Maybe with those, we can figure out what happens.

yehyeh... hmmm.... chrome://net-internals shows a loootta info... i
have 50 tabs open and gmail seems to get stuff using AJAX... i'll have
to set up a separate chrome instance just to isolate the traffic...
before i do that, here's the relevant section of an appropriate
request - you can see a whopping 3 *seconds* in between request and
read response headers.


t=69785 [st= 303] +HTTP_TRANSACTION_SEND_REQUEST [dt=0]
t=69785 [st= 303] HTTP_TRANSACTION_SEND_REQUEST_HEADERS
--> GET http://horseboxseller.com/ HTTP/1.1
Host: horseboxseller.com
Proxy-Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76
Safari/537.36
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer:
http://horseboxseller.com/horseboxes-for-sale/detail/1772/3.5-ton/2006-model-55-vauxhaul-movano-coach-built-by-chaighley-76-000-miles-stalled-for-2-rear-facing-large-external-tack-locker/
Accept-Encoding: gzip, deflate, sdch
Accept-Language:
en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,pl;q=0.2
Cookie: [312 bytes were stripped]
t=69785 [st= 303] -HTTP_TRANSACTION_SEND_REQUEST
t=69785 [st= 303] +HTTP_TRANSACTION_READ_HEADERS [dt=3132]
t=69785 [st= 303] HTTP_STREAM_PARSER_READ_HEADERS [dt=3132]
t=72917 [st=3435] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 200 OK
Date: Sun, 06 Aug 2017 15:42:33 GMT
Server: Apache/2.4.10 (Debian)
Vary: Accept-Encoding
X-Mod-Pagespeed: 1.12.34.2-0
Cache-Control: max-age=0, no-cache
Content-Length: 33836
Content-Type: text/html; charset=UTF-8
Proxy-Connection: keep-alive
t=72917 [st=3435] -HTTP_TRANSACTION_READ_HEADERS

then here's one for a different page, you can see the time delay is...
tiny by comparison: st=280... st=779 i presume those are in
milliseconds, so around half a second (not three!!)

btw i'm in taiwan, server's in the UK, ping round-trips are
around...270ms over the VPN i'm running the proxy over. also i know
from webmaster tools that the response time of the server is around
200ms, so overall 500ms sounds about right.

more when i've got it set up.

t=4235 [st= 280] -HTTP_STREAM_REQUEST
t=4235 [st= 280] +HTTP_TRANSACTION_SEND_REQUEST [dt=0]
t=4235 [st= 280] HTTP_TRANSACTION_SEND_REQUEST_HEADERS
--> GET
http://horseboxseller.com/horseboxes-for-sale/ HTTP/1.1
Host: horseboxseller.com
Proxy-Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76
Safari/537.36
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://horseboxseller.com/
Accept-Encoding: gzip, deflate, sdch
Accept-Language:
en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,pl;q=0.2
Cookie: [544 bytes were stripped]
t=4235 [st= 280] -HTTP_TRANSACTION_SEND_REQUEST
t=4235 [st= 280] +HTTP_TRANSACTION_READ_HEADERS [dt=499]
t=4235 [st= 280] HTTP_STREAM_PARSER_READ_HEADERS [dt=499]
t=4734 [st= 779] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 200 OK
Date: Sun, 06 Aug 2017 15:57:35 GMT
Server: Apache/2.4.10 (Debian)
Vary: Accept-Encoding
X-Mod-Pagespeed: 1.12.34.2-0
Cache-Control: max-age=0, no-cache
Content-Length: 46140
Content-Type: text/html; charset=UTF-8
Proxy-Connection: keep-alive
t=4734 [st= 779] -HTTP_TRANSACTION_READ_HEADERS
t=4734 [st= 779] HTTP_CACHE_WRITE_INFO [dt=0]

[Luke Kenneth Casson Leighton]

http://lkcl.net/mod_pagespeed/privoxy_log_delays/

still filling in some of the data (memcache logs and so on)

l.

[Luke Kenneth Casson Leighton]

damn. i just switched *off* memcached, and the bug is still present.
except because memcached is no longer in the picture i don't have a
trigger mechanism to "fix" it. not that restarting (thus destroying)
the memcache is a good "fix".

but it at least demonstrates that memcached isn't the actual root
cause of the problem.

i suspect some sort of locking contention involving the cache (caches
in general, not memcached specifically), where "stopping memcached if
it's configured" *happens* to cause an error which also either breaks
the internal deadlock *or* causes the cache entry to be invalidated.

does anyone know: is it possible to limit the number of processes /
threads used by mod_pagespeed, down to say... two or even one? i know
performance will be adversely affected but i can do some testing
during off-peak hours (tomorrow morning for example) as i am on a
different timezone from the main customers.

l.

p.s. apologies to the developers - much respect for what you're doing,
i've had one *second* average response time out of mod_pagespeed at
times - but as a software libre developer and advocate, you should be
aware that if you weren't using a proprietary bugtracker i would put
this information where it really belongs: in a bugtracker.

[Joshua Marantz]

We're using github's bug tracker, which is proprietary to github, I guess, but pretty open :)

https://github.com/pagespeed/mod_pagespeed/issues

For thread-count control, see https://modpagespeed.com/doc/system#tune_thread

For process-count control, see https://httpd.apache.org/docs/2.4/mod/worker.html#page-header

I've lost track of what bug you are trying to isolate by turning off memcached (how did you do that?) and reducing process/thread-count.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDzCVhheSGxzaus0m%2BNc%3DPFGn_sArDQpRjmuTtwheyH9dA%40mail.gmail.com.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Tue, Aug 8, 2017 at 7:12 PM, 'Joshua Marantz' via
mod-pagespeed-discuss <mod-pagesp...@googlegroups.com> wrote:
> We're using github's bug tracker, which is proprietary to github, I guess,

indeed.

> but pretty open :)

open in the way that facebook, slack.com and other proprietary
systems are "open" [ever read slack's terms and conditions?]. in FSF
terminology they're known as "Service as a Software Substitute", but
also - worse than that - come with user-tracking. long story, short
version: no way i'm using it. closed the account several years ago.

> https://github.com/pagespeed/mod_pagespeed/issues

yehyeh - very useful [read-only] resource. i'm just relieved that
you have a public mailing list. pain in the neck finding bugs or
wanting to contribute, and there's absolutely *ZERO* method [outside
of having my privacy and my principles totally violated] of contacting
the user....

welcome to the world of peer pressure.... which is precisely why i do
not have a github account. i don't bow to peer pressure. ever.

> For thread-count control, see
> https://modpagespeed.com/doc/system#tune_thread

ah good stuff.

> For process-count control, see
> https://httpd.apache.org/docs/2.4/mod/worker.html#page-header

ok... ahh... btw mpm_worker caused segfaults - had to switch it off.
as in, it caused the *ENTIRE* apache2 process tree - right back to and
*including* the top-level parent process - to terminate [leaving the
client with no web site for about an hour during peak times...
baaaad]. never seen anything happen like that, not in 20 years of
running apache2. pretty damn serious bug. anyway, apparently this is
a known problem with mpm_worker, so out it went.

i'm currently using mpm_event, but the delays also occurred when
mpm_prefork was being deployed.

>
> I've lost track of what bug you are trying to isolate

the exact same one that's been under this subject-line, with no other
topics being discussed under this subject line, all messages solely
and exclusively under this subject line being about this one bug and
this one bug only, for about... a week.
so you don't have to look them up, here's the summary: absolutely
unnecessary delays (around 3 seconds) being introduced on arbitrary
(random) pages but consistently 100% repeatable (once they occur),
when using google chrome, mod_pagespeed, and a proxy (in this case
privoxy).

> by turning off memcached (how did you do that?)

ModPagespeedMemcachedServers localhost:11211

-->

#ModPagespeedMemcachedServers localhost:11211

followed by /etc/init.d/apache2 restart

btw as disabling memcached had the side-effect of dramatically
decreasing the effectiveness of mod_pagespeed (average times gone up
from 1.2, 1.8 seconds to a whopping 3 to 4 seconds) i've since
re-enabled it. [live site: can't mess with it too badly, can't
replicate the problem locally...]

> and reducing process/thread-count.

haven't investigated that yet as i had no idea what options to
deploy. the main ones appear to be NumRewriteThreads and
NumExpensiveRewriteThreads. i'll look up the equivalent of the number
of threads (etc.) for mpm_event, thanks for reminding me that it's
important to try that out as well.

l.

[Luke Kenneth Casson Leighton]

btw joshua the reason why this is pretty important to find and fix is
because if there happens to be upstream *transparent* proxies (ISPs
etc.) outside of the user's control, there's absolutely nothing anyone
can do about it, and the effectiveness of mod_pagespeed is completely
nullified.

l.

[Joshua Marantz]

On Wed, Aug 9, 2017 at 1:26 PM, Luke Kenneth Casson Leighton <lk...@lkcl.net> wrote:

> For process-count control, see
> https://httpd.apache.org/docs/2.4/mod/worker.html#page-header

 ok... ahh...  btw mpm_worker caused segfaults - had to switch it off.

See also:

https://httpd.apache.org/docs/2.4/mod/prefork.html#page-header

https://httpd.apache.org/docs/2.4/mod/event.html#page-header

BTW if 'worker' causes segfaults it's usually because some module you are using is not thread-safe.  I think mod_php is a classic example but I admit I know as little as possible about PHP and could be mis-remembering. I know we test mod_pagespeed with the worker MPM.

>> I don't think our cross-testing between that feature and others is
>> sufficient. It wouldn't surprise me if there were cross-compatibility issues
> with the lazy-load filter.

> i'm slightly surprised by that.  i would have expected there to be
> like a chain between the filters, pretty much transparent, just using
> the output from one as the input to the next.

There is.  However it's not the integrity of the C++ structures in mod_pagespeed that concerns me; it's the interaction between the effects of the filters on the dom and JS when there are two filters that inject JS hacking on the same DOM element.

Are there other specific questions?  Apologies if I've missed some in the thread history.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Wed, Aug 9, 2017 at 8:20 PM, 'Joshua Marantz' via
mod-pagespeed-discuss <mod-pagesp...@googlegroups.com> wrote:
> On Wed, Aug 9, 2017 at 1:26 PM, Luke Kenneth Casson Leighton <lk...@lkcl.net>
> wrote:
>>
>> > For process-count control, see
>> > https://httpd.apache.org/docs/2.4/mod/worker.html#page-header
>>
>> ok... ahh... btw mpm_worker caused segfaults - had to switch it off.
>
>
> See also:
>
> https://httpd.apache.org/docs/2.4/mod/prefork.html#page-header
> https://httpd.apache.org/docs/2.4/mod/event.html#page-header
>
> BTW if 'worker' causes segfaults it's usually because some module you are
> using is not thread-safe. I think mod_php is a classic example but I admit
> I know as little as possible about PHP and could be mis-remembering. I know
> we test mod_pagespeed with the worker MPM.

the main application's fastcgi (wsgi) in python, and i spent a bit of
time setting up straight cgi for php. the office "backend" is in php
(which i really can't stand... inherited project...) but yes mod_php
is not thread-safe and i wanted it gone, and the php processing
completely isolated, out of the apache2 main application. simplest
way i could see to do that: CGI-bin it. it's an unusual approach but
speed's not a high priority for the backoffice, and i wanted to try
mpm_worker (and mpm_event).

so whilst it would not be outside the realm of possibility that
mod_cgi and mpm_worker have not been properly stabilised (not even
involving mod_pagespeed), i would be hugely surprised to learn that
mod_cgi - which is supposed to completely isolate the actual
page-generation from apache2 - was responsible for causing the
segfault.

likewise, the main application is a wsgi application, it's a pure
python2,7 application (not even a django or any other framework), uses
standard python modules like the cgi module, uses python-mysqldb
*directly* (not even any kind of sql OO framework like sqlobject or
sqlalchemy)- so i am pretty confident that there's absolutely zero
threading involved there.

bottom line is: it all - vaguely - points towards mod_pagespeed as
being the cause of the segfault.

>>> I don't think our cross-testing between that feature and others is
>>> sufficient. It wouldn't surprise me if there were cross-compatibility
>>> issues
>> with the lazy-load filter.
>
>> i'm slightly surprised by that. i would have expected there to be
>> like a chain between the filters, pretty much transparent, just using
>> the output from one as the input to the next.
>
> There is. However it's not the integrity of the C++ structures in
> mod_pagespeed that concerns me; it's the interaction between the effects of
> the filters on the dom and JS when there are two filters that inject JS
> hacking on the same DOM element.

yehyeh that makes a lot of sense. well, local cache is off, now.

> Are there other specific questions? Apologies if I've missed some in the
> thread history.

nono, not a problem - i've been doing quite a bit of experimentation
but have backed away from that, now, and am leaving things alone to
keep an eye on them more long-term, my primary concern and focus for
now is that proxy (privoxy) interaction. as i said in the last
message, it's not so much *my* proxy that's the problem, it's what
happens if *ISPs* add in *transparent* proxies downstream - massive
ones - that users have absolutely no control over.

that would completely screw mod_pagespeed for... hundreds of
thousands of users worldwide, adding in several seconds of delay on
arbitrary pages without any reason and there's *nothing anyone can do
to fix it*. you can't switch off an ISP's transparent proxy, not in
the fascist UK for example where BT and other ISPs are demanded by the
stupid ignorant UK govt to do real-time fascist privacy-violating deep
packet inspection.(and associated long-term storage of people's
browsing history). yes i really did say the UK, not e.g. China.

sorry... had to get that off my chest... but you get the point i'm
sure: there are entire countries with transparent HTTP proxies so it
would be kinda important to replicate this scenario.

l.

[Otto van der Schaaf]

In the logs you posted earlier, I see this:

Aug 07 07:08:29.773 afafdb40 Header: A HTTP/1.1 response without Connection header implies keep-alive.

Aug 07 07:08:29.773 afafdb40 Header: Adding: Proxy-Connection: keep-alive

Aug 07 07:08:32.562 afafdb40 Header: Adjusted Content-Length to 33669

Aug 07 07:08:32.562 afafdb40 Header: Crunching: Content-Encoding: gzip

Is that an example of the problem you are describing?

It looks like the headers arrive timely, but receiving the body bytes from the pagespeed-enabled server takes a couple of seconds to complete.

A slow response may become more apparent to the end-user as it looks like the proxy is fully buffering the response before forwarding it. 

(that isn't something most forward proxies do)

A capture of the problem with wireshark on both privoxy and the mod_pagespeed enabled server may be helpful to pinpoint exactly where the delay is generated and if the delay is on the first byte of the response body or somewhere in the transfer the body.

Otto

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAPweEDxQr2us_Jrx2s41RotUMHvXe3H7gkAAvtcrEeK4rmGCVg%40mail.gmail.com.

[Luke Kenneth Casson Leighton]

---
crowd-funded eco-conscious hardware: https://www.crowdsupply.com/eoma68

On Thu, Aug 10, 2017 at 9:53 AM, Otto van der Schaaf <osc...@we-amp.com> wrote:
> In the logs you posted earlier, I see this:
>
> Aug 07 07:08:29.773 afafdb40 Header: A HTTP/1.1 response without Connection
> header implies keep-alive.
> Aug 07 07:08:29.773 afafdb40 Header: Adding: Proxy-Connection: keep-alive
> Aug 07 07:08:32.562 afafdb40 Header: Adjusted Content-Length to 33669
> Aug 07 07:08:32.562 afafdb40 Header: Crunching: Content-Encoding: gzip
>
> Is that an example of the problem you are describing?

yes! perfect example. the bizarre thing is: it only occurs
arbitrarily and selectively (not on *all* pages), it can be triggered
by doing a lot of "Ctrl-F5, F5, Ctrl-F5, Ctrl-F5, F5" fast/slow
pressing, and once triggered the page *stays* that way [until
memcached is restarted].

> It looks like the headers arrive timely, but receiving the body bytes from
> the pagespeed-enabled server takes a couple of seconds to complete.

oof. ouch. *thinks*.... i'm returning the length (correctly) on the
wsgi app.... i'm not returning a file object i'm returning a string,
but there's no reason why there should be a huge delay....

... BUT...

... i do need to check that, don't i? :) i'll add in some logging to
txt files, just to eliminate the WSGI app as the root source of the
delays...

> A slow response may become more apparent to the end-user as it looks like
> the proxy is fully buffering the response before forwarding it.
> (that isn't something most forward proxies do)

hmmmm... well... persistent HTTP 1.1 connections _is_ enabled...

AH! damnit!! https://github.com/trailofbits/algo/issues/93

of course... *smack*... privoxy is a filtering service, it's most
likely finding something it doesn't like, and cuts it. but it would
need to pull the entire page into memory to do it.... damn.

so let's check.... aaand yes, disabling "default.filter" in the
privoxy config "fixes" the problem. switch it on, delay is back.
switch it off, delay goes away.

good prompting, otto. the clue was the "buffering". there didn't
seem to be any reason for that... until i remembered this is privoxy.

joshua: panic over :)

thanks for your help otto.

l.