image subrequests when all image rewriting disabled and forbidden

[Michael Johnson]

Hi,

I am running mod_pagespeed in an environment where the webservers are diskless and we are utilizing a ramdisk for the pagespeed cache.  Because of this, we have forbidden rewrite_images.  However, when a page is requested, mod_pagespeed is still making sub requests for all of the images in the page and caching those images in the pagespeed cache.  As a result, I went through and explicitly disable and forbid every individual filter related to images as listed at https://developers.google.com/speed/pagespeed/module/filter-image-optimize

However, this did not stop the subrequests from happening nor did it prevent the files from showing up in the pagespeed cache.  Am I missing something here or is this the expected behavior.  I am running 1.2.24.1-2581.

If there is no way to prevent this from happening, I am afraid I may have to disable mod_pagespeed entirely.  :(

[Joshua Marantz]

Did you disable extend_cache?  You can also disable extend_cache_images and allow cache extension of javascript and css files.

Out of curiosity, what filters *are* you enabling?

Note that 1.2.* is pretty old.  The latest stable is 1.4, and the latest beta is 1.6.

-Josh

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/790298ae-2f33-4722-ae84-d9f428130580%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael Johnson]

We are not explicitly enabling any filters by default, nor are we setting the rewrite level, which as I understand is the same as setting the rewrite level to 'CoreFilters'.

We then explicitly forbid the following filters:

rewrite_images,make_google_analytics_async,flatten_css_imports,combine_javascript,canonicalize_javascript_libraries,local_storage_cache,insert_ga,inline_preview_images,resize_mobile_images,sprite_images,rewrite_domains,add_instrumentation,defer_javascript

I'm not sure we really need to forbid all of these things, but it is what we have currently in place.

As you can see, we do not disable extend_cache nor extend_cache_images.  It would seem that disabling extend_cache_images should not have an impact to mod_pagespeed copying the images into it's cache as that should only impact the headers.  However, I went ahead and tried with the following:

    ModPagespeedDisableFilters rewrite_images,make_google_analytics_async,flatten_css_imports,combine_javascript,canonicalize_javascript_libraries,local_storage_cache,insert_ga,inline_preview_images,resize_mobile_images,sprite_images,rewrite_domains,add_instrumentation,defer_javascript,extend_cache_images

    ModPagespeedForbidFilters rewrite_images,make_google_analytics_async,flatten_css_imports,combine_javascript,canonicalize_javascript_libraries,local_storage_cache,insert_ga,inline_preview_images,resize_mobile_images,sprite_images,rewrite_domains,add_instrumentation,defer_javascript,extend_cache_images

This did not prevent the images from being copied into the mod_pagespeend file cache path.  It will take some time, but can test with the current stable and beta versions.  Although I don't expect a different result as I've not found any reports or documentation that would indicate this problem exists, much less has been fixed.  I'm not terribly surprised though as our usage probably is not terribly common.

On Monday, September 23, 2013 5:40:44 AM UTC-7, jmarantz wrote:

Did you disable extend_cache?  You can also disable extend_cache_images and allow cache extension of javascript and css files.

Out of curiosity, what filters *are* you enabling?

Note that 1.2.* is pretty old.  The latest stable is 1.4, and the latest beta is 1.6.

-Josh

On Mon, Sep 23, 2013 at 8:30 AM, Michael Johnson <ther...@gmail.com> wrote:

Hi,

I am running mod_pagespeed in an environment where the webservers are diskless and we are utilizing a ramdisk for the pagespeed cache.  Because of this, we have forbidden rewrite_images.  However, when a page is requested, mod_pagespeed is still making sub requests for all of the images in the page and caching those images in the pagespeed cache.  As a result, I went through and explicitly disable and forbid every individual filter related to images as listed at https://developers.google.com/speed/pagespeed/module/filter-image-optimize

However, this did not stop the subrequests from happening nor did it prevent the files from showing up in the pagespeed cache.  Am I missing something here or is this the expected behavior.  I am running 1.2.24.1-2581.

If there is no way to prevent this from happening, I am afraid I may have to disable mod_pagespeed entirely.  :(

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

As far as I know, turning off extend_cache_images and rewrite_images should prevent images from being cached.  Note that setting those options won't delete existing cache entries, so there may still be old image files in your ramdisk cache directory.

It might might be easier just to use pass-through mode and explicitly enable the features you want, and then add:

  ModPagespeedForbidAllDisabledFilters true

I am curious why you turned off flatten_css_imports.  Did you find a problem with it?  We think it's pretty impactful and low risk, although I'd definitely upgrade to 1.4 at least.  None of the other ones you disabled are in the core set anyway.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/52408ff4-9899-4a9b-ab94-01589e578428%40googlegroups.com.

[Michael Johnson]

I considered the possibility of old cache files, so removed them, shut down apache, made sure they were still gone, started apache, made sure they were still gone, and then curled a URL, and saw them all appear.

I was not the person who decided initially what we would enable/disable/forbid by default so I'm not sure the reasoning behind the flatten css bits.  I suspect it was because https://developers.google.com/speed/pagespeed/module/filter-flatten-css-imports lists it as "moderate risk".  However, that seems to call for maybe disabling it by default rather than forbiding it's use.

I'll do some more playing with switching to passthrough mode and toggling some differing things including trying out the newer versions and report back.

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsubscri...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/790298ae-2f33-4722-ae84-d9f428130580%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

I think we should investigate the unexpected fetches as a bug, though I'd prefer not to investigate 1.2.  It might have been fixed already.

However you should be able to workaround using Disallow:

   ModPagespeedDisallow *.jpg

   ModPagespeedDisallow *.png

   ModPagespeedDisallow *.gif

This will work independent of the filters.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/4c6f2727-eee7-4e62-b066-8767e391fb90%40googlegroups.com.

[Michael Johnson]

The following (minimal forbids) results in the desired behavior for all version (1.2, 1.4 stable, and 1.6 beta):

ModPagespeedForbidFilters rewrite_images,extend_cache_images,convert_png_to_jpeg

I suppose I would chaulk this up to a "documentation unclear" or something similar.  It is documented that "Some image recompression options are not included in rewrite_images, the group filter, and so must be enabled specifically."  There are a number of items below this point in the documentation that are coverd including convert_png_to_jpeg.  The inverse also applies if using CoreFilters as convert_png_to_jpeg is part of core and disabling rewrite_images does not disable convert_png_to_jpeg.

I missed this in my earlier testing as my first pass at disabling all the image related filters did not include the extend_cache_images as it didn't seem like it would apply.  When you later suggested it, I added it to the original line which disabled rewrite_images but not convert_png_to_jpeg.

Thanks for your help with this!

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/52408ff4-9899-4a9b-ab94-01589e578428%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Yeah -- that was a bug in 1.2 I'm pretty sure.  convert_png_to_jpeg will be turned off by disabling rewrite_images in later releases -- I'm not remembering whether that got fixed in time for 1.4 or not but I'm pretty sure it's fixed in 1.6.  In any case you worked around it.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/27eac01c-85b4-4f31-af6a-a31c31cf92f3%40googlegroups.com.

[Joshua Marantz]

One other point I want to make: if you are disabling rewrite_images then you are losing a lot of the value provided by mod_pagespeed.  You might consider using memcached so that you can pay for that cache memory once across cluster of machines.

-Josh

[Michael Johnson]

Based on my testing, the behavior in all versions was that forbidding rewrite_images does not disable convert_png_to_jpeg.  I did not try disabling separately from forbidding.  But yeah, the important thing is we have a solution.  :)

Thanks for the pointers.  We are running clusters of 10+ nodes each.  We have considered memcached and are tentatively planning on running it on the nodes themselves.  The concern with this is the ModPagespeedMemcachedServers will get pretty unwieldy with that many servers.  The fact that memcached is restricted to 1MB values is a bit annoying, but it probably still handles the majority of the cases.

One of the reasons we noticed the image rewrites is we were seeing some load issues and there seemed to be a fair amount of correlation spikes in load and the Serf requests.  Given there was no rewriting of the page links to the cache files generated by these requests appear to (in our current configuration) be costing us resources and not providing in benefit.  Looking through the logs, these request are 2.75% of all requests which is a lot of resources getting burned.  Granted if we were caching all of this in a shared location such a memcached, in a 10 node cluster that overhead in a perfect world would drop to 0.275% which is far less of an issue.

We have a long way to go to get the full benefit from mod_pagespeed, but we are on our way.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/4c6f2727-eee7-4e62-b066-8767e391fb90%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

Cool.  Taking it one step at a time makes sense to me.

I'll take a look at why disabling or forbidding rewrite_images is not enough; it should be.  Must still be a bug at least in 1.6.

I'm also curious about load spikes correlated with fetches.  What is the Cache-Control setting on your images?  Maybe we are having an issue that the images are all simultaneously expiring, and this happens frequently enough that it's an issue.  We have an experimental rate-limiter for fetching, but evidence from other users in a similar situation suggests it needs further tuning to be effective.  We are going to try and tweak it soon to try to address that.

However it seems likely that the load-spikes are due to image optimization CPU, rather than fetching overhead.  This can be addressed by a setting: https://developers.google.com/speed/pagespeed/module/filter-image-optimize#ImageMaxRewritesAtOnce

What kind of hardware are you running on?  How many CPUs, running at what speed?

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/1c6a2294-3b77-4ff6-beea-47b62dba1c99%40googlegroups.com.

[Michael Johnson]

The cache-control is set to max-age=31536000 (although this is configurable by the end user).  The load spikes seems to be partly related to the fact that the subrequests are happening very rapidly and in some cases there are well over 100 images.  Not great site design in my opinion, but it's not something I have control over.

Another aspect is that I was able to determine that 2.75% or our requests to apache were in fact these subrequests.  Given these now cached images were never utilized, the clients would still make requests for each of these images as well.  The end result being that effectively 2.5% of all resources were beind spent handling this requests.  If we were actually utilizing the results, it could very well negate that overhead, but for the time being, it is just a waste of resources.

A rate limiter for the fetching sounds like a good idea.  Not sure how much that concept is flushed out, but it seems like a feature that might be a great idea to build into the serf library and give the option to grab robots.txt and parse out crawl-delay or something similar.  But that might be overkill.  Maybe limit fetching to X subrequests per request and build up the cache slowly.  Either way, rate limiting the requests in some way will be a very useful feature.  

I'll keep the ImageMaxRewritesAtOnce directive in mind as if/when we get a shared cached for all of our nodes together, it might be something I'll need.

The hardware we are running on is dual quadcore cpus @ 2.6GHz.  Not top of the line, but fairly robust.  We do keep them pretty busy though.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/27eac01c-85b4-4f31-af6a-a31c31cf92f3%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

I have a naive question.  Don't you run into the same trouble with bursts of 100 requests for image resources every time a client navigates to the site?  Granted that's a little less effort on your server compared to MPS's loopback fetches because you are only serving the images and not also acting as a client.  To address that we have ModPagespeedLoadFromFile, which should bring the fetch-count to zero for static resources.

I'm also not understanding why you say the cached results are never used.  It's true that mod_pagespeed caches both origin resources and optimized resources: this is perhaps not necessary in some cases.  But in general we may optimize an image in several different ways (webp vs jpeg, different resolutions) and in some deployments it's faster to look it up from cache than it is to fetch it from its HTTP origin.  Is that what you meant?

One situation I can think of is that the images are in the markup, especially the CSS, but are not actually put into visible state in the DOM, and the browser might not fetch them until they are actually exposed.  Is that what's happening on your site?

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/04b4d32a-cd95-4b3d-906d-222b535ad2dd%40googlegroups.com.

[Michael Johnson]

I wouldn't consider that a naive question at all, rather a very reasonable question.  :)

We don't run into quite the same problem becuase those 100+ requests are more spread out rather that the near instantaneous burst that occurs with the mod_pagespeed fetches.  I was seeing them all log at same second for the most part.  I don't have any further insight at to the actual concurrency happening, but based on the fact that mod_pagespeed would seem to want those files processed for the current request, it would make sense that there is a large amount of parallelization going on.  Based on http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser, browsers won't open more that 10 concurrent connections.  Given that the cached results aren't used, that means we still get the 100+ requests from the browser as well.

Which bring me to your second question...

It is my (possibly incorrect) understanding, that any resource that will be served from the pagespeed cache will be via a modified URL in the page that linked to the original.  For example, if my page includes this:

<img src="/path/to/image.jpg">

It would get modified to be:

<img src="/path/to/image.jpg.pagespeed.XXXXXXXX.jpg>

or something similar to that.  In this case, the requested page did not have the modified URLs.  This indicates to me that the pagespeed cache, while being generated, was not actually getting utilized for these image resources.

In the cases I looked at, the images were all in the html of the response.  There may have been some in CSS, but at minimum, the lions share was in the html via <img /> tags.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/1c6a2294-3b77-4ff6-beea-47b62dba1c99%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Joshua Marantz]

That's a very good answer: the browsers themselves all rate-limit, usually to 6 concurrent requests per domain or less.  This argues we should employ similar limits in our loopback fetches so we are no more impactful to the server load than a single browser HTML view.

I'm not sure why your 'img' tags are not successfully rewritten.  I agree: in the short term it makes no sense to fetch those images if you are not getting value, but I'd like to explore why that is.  It usually isn't hard for us to nail that down to a cause by turning on apache "loglevel info" and looking at the logs.

One thing that would make that easier is for you to avoid the 'forbid' directives, and just use 'disable'.  That would allow us to experiment with ?ModPagespeedFilters=+rewrite_images, which might cause a burst of activity on your server but only for that one request.  If you want us to look directly you'd have to share a URL or course, or we could guide you via email.  Feel free to share with me privately if you'd prefer not to broadcast.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/93b97788-d723-41b7-83d1-df07f6f05ce0%40googlegroups.com.

[Michael Johnson]

The system in question has many users with many sites and as we are very restricted on cache, so we want to make certain that the end users are not able to do things that could potentially cause problems which is why are are using forbid.  While it is unlikely anyone would enable image_rewrites while we were testing, if they did and caused a problem, I would have a hard time explaining why I allowed something when I already know it was a potential problem.  :) However, I pretty confident this is not unique to this environment, and I could probably find the time to try to reproduce the issue on a system that I could give you full access to.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/04b4d32a-cd95-4b3d-906d-222b535ad2dd%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Michael Johnson]

I've set up a vm running ubuntu 13.04 with pagespeed 1.6.29.5-r3346 and a very vanilla apache configuration.  Once the config was in place, I started apache and then checked to cache directory and saw nothing in it:

# find /var/cache/mod_pagespeed/

/var/cache/mod_pagespeed/

Then I curled for the index page I put in place with one image:

# curl -D - localhost

HTTP/1.1 200 OK

Date: Wed, 25 Sep 2013 20:37:24 GMT

Server: Apache/2.2.22 (Ubuntu)

Accept-Ranges: bytes

X-Mod-Pagespeed: 1.6.29.5-3346

Vary: Accept-Encoding

Cache-Control: max-age=0, no-cache

Content-Length: 66

Content-Type: text/html

<html><head/><body>

<img src="darth-vader-2.jpg"/>

</body></html>

Notice that the image URL is unchanged.

But when I check the cache directory again:

# find /var/cache/mod_pagespeed/ -ls

  1417    4 drwxr-xr-x   4 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/

   530    4 drwxr-xr-x   3 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/http,3A

  1543    4 drwxr-xr-x   2 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/http,3A/,2Flocalhost

  3433  228 -rw-------   1 www-data www-data   230483 Sep 25 17:00 /var/cache/mod_pagespeed/http,3A/,2Flocalhost/darth-vader-2.jpg,

  3436    4 drwxr-xr-x   3 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/rname

  3442    4 drwxr-xr-x   3 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/rname/ic_e1ybVIr4QFQPj-KdUZUi

  3444    4 drwxr-xr-x   3 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/rname/ic_e1ybVIr4QFQPj-KdUZUi/http,3A

  3454    4 drwxr-xr-x   2 www-data www-data     4096 Sep 25 17:00 /var/cache/mod_pagespeed/rname/ic_e1ybVIr4QFQPj-KdUZUi/http,3A/,2Flocalhost

  3464    4 -rw-------   1 www-data www-data       62 Sep 25 17:00 /var/cache/mod_pagespeed/rname/ic_e1ybVIr4QFQPj-KdUZUi/http,3A/,2Flocalhost/darth-vader-2.jpg,40x,40_,

I do have the log level set to 'info', and there is nothing of interest in the error.log file:

[Wed Sep 25 17:00:03 2013] [notice] caught SIGTERM, shutting down

[Wed Sep 25 17:00:04 2013] [notice] Apache/2.2.22 (Ubuntu) configured -- resuming normal operations

Nothing else.

The config is not setting a rewrite level and disabling the following:

ModPagespeedDisableFilters rewrite_images,extend_cache_images

I specifically did not disable convert_png_to_jpeg in order to demonstrate this issue.

I'm happy to create an ssh login for you on this box and give you root access as it is a test machine that I will wipe later.  Do you use last pass?  If so, that would be an easy way to do this.  If you want to email me and SSH key, we could go that route as well.  Let me know what you think.

Thanks again.

[Joshua Marantz]

Thanks for the testcase.  Tracking this as https://code.google.com/p/modpagespeed/issues/detail?id=787.

I'm still wondering why this happens but we'll sort that out when we fix the bug.  By inspection the code looks OK but I believe you.  Fortunately you have a workaround.

I'm more concerned actually about what happens when you enable rewrite_images, but say that doesn't help rewrite <img> links to reference the .pagespeed. URL.  Can you repro that?

And with respect to fetch throttling, can you try turning on rate-limiting?

  ModPagespeedRateLimitBackgroundFetches on

Based on the current parameters, we think that should work too, so we'd like to have you try it if possible.  It should already be the case that a single refresh should generate only 1 conccurrent fetch per domain when using Apache Prefork MPM.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/327ff03a-615c-49a8-bb37-407b11e6d0fe%40googlegroups.com.

[Michael Johnson]

I can confirm that in this test case, enabling rewrite_images does cause the URL in the resulting html to be modified. Sorry if it came across as otherwise.  The wasted work only come into play when rewirte_images is disabled and a filter that would cause image fetching to occur is enabled as the images are still fetched and processed, just never used as the html is not modified. 

The apache config I used was the mpm-prefork, start servers = 1, max spare = 1, min spare = 1 max clients = 150.

I created a page with 2000 images on it and did a single curl.  The load went up to about 0.17 and the number of apache processes hit max_connections and then immediately started dropping.  329 files were fetched.  Lots of this in the error log:

Wed Sep 25 18:57:36 2013] [error] [mod_pagespeed 1.6.29.5-3346 @24162] http://205.186.154.59/1979.jpg:0: serf_context_run error status=105 (No buffer space available)

[Wed Sep 25 18:57:36 2013] [warn] [mod_pagespeed 1.6.29.5-3346 @24162] [0925/185736:WARNING:serf_url_async_fetcher.cc(1173)] Fetch failed to start: http://205.186.154.59/1979.jpg

[Wed Sep 25 18:57:36 2013] [error] [mod_pagespeed 1.6.29.5-3346 @24162] http://205.186.154.59/1980.jpg:0: serf_context_run error status=105 (No buffer space available)

[Wed Sep 25 18:57:36 2013] [warn] [mod_pagespeed 1.6.29.5-3346 @24162] [0925/185736:WARNING:serf_url_async_fetcher.cc(1173)] Fetch failed to start: http://205.186.154.59/1980.jpg

[Wed Sep 25 18:57:36 2013] [error] [mod_pagespeed 1.6.29.5-3346 @24162] http://205.186.154.59/1981.jpg:0: serf_context_run error status=105 (No buffer space available)

[Wed Sep 25 18:57:36 2013] [warn] [mod_pagespeed 1.6.29.5-3346 @24162] [0925/185736:WARNING:serf_url_async_fetcher.cc(1173)] Fetch failed to start: http://205.186.154.59/1981.jpg

I then cleared the cache, disabled mod_pagespeed and made the request from a web browser.  It was slow, but the load never went over 0.3.  There were 6 apache children spawned for this.

I then cleared the cache, set ModPagespeedRateLimitBackgroundFetches on, and tried again.  The number of apache processes hit max_connections quickly and stayed there.  Load continually went up.  I killed apache when the load was at 140.  283 files were successfully fetched.  Lots of this in the error log:

[Wed Sep 25 19:01:53 2013] [warn] [mod_pagespeed  @24162] [0925/190153:WARNING:serf_url_async_fetcher.cc(1155)] Aborting fetch of http://205.186.154.59/1869.jpg

[Wed Sep 25 19:01:53 2013] [warn] [mod_pagespeed  @24162] [0925/190153:WARNING:serf_url_async_fetcher.cc(1155)] Aborting fetch of http://205.186.154.59/1870.jpg

[Wed Sep 25 19:01:53 2013] [warn] [mod_pagespeed  @24162] [0925/190153:WARNING:serf_url_async_fetcher.cc(1155)] Aborting fetch of http://205.186.154.59/1871.jpg

[Wed Sep 25 19:01:53 2013] [warn] [mod_pagespeed  @24162] [0925/190153:WARNING:serf_url_async_fetcher.cc(1155)] Aborting fetch of http://205.186.154.59/1872.jpg

 

Hope that all makes sense.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/93b97788-d723-41b7-83d1-df07f6f05ce0%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Michael Johnson]

I actually reversed the results here.  To clarify, the ModPagespeedRateLimitBackgroundFetches setting did not seem to limit the number of concurrent fetches under the prefork mpm.  Regardless of if it was on or off, it still made enough subrequests to hit the MaxClients setting of apache.  The one difference is that with it set 'on', once MaxClients was reached, no new subrequests were made.  With it set to 'off', subrequests continued to be made keeping it at 'maxclients' and driving the load on the box continually higher.

[Joshua Marantz]

I've captured this as https://code.google.com/p/modpagespeed/issues/detail?id=788.  I corrected the reversed results in the bug description.  Thanks for the detailed repro instructions.  We will try to get this fixed.

-Josh

To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/e91d291d-551d-4716-a311-929086566e28%40googlegroups.com.