Re: limit serf connections to backend

[Maksim Orlovich]

ModPagespeedRateLimitBackgroundFetches on
might help.

On Tue, May 14, 2013 at 11:59 PM, Alexandre Biancalana
<bianc...@gmail.com> wrote:
> Hi list !
>
> I'm using apache 2.4 (mpm_event) with mod_pagespeed1.3.25.4 as a reverse
> proxy and planning to use different configuration foreach of our
> virtualhosts.
>
> Apparently everything is working fine, but I see a lot of serf erros in
> error.log:
>
> [Wed May 15 00:38:34.375460 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 40 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.376316 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 39 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.376940 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 38 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.377562 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 37 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.378188 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 36 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.378758 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 35 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.379393 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 34 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.380054 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 33 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.381269 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 32 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.382163 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 31 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.383111 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 30 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.383776 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 29 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.384405 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 28 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.384897 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 27 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.385433 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 26 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.386065 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 25 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.386663 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 24 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.387288 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 23 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.387832 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 22 threaded fetches for 0.05 seconds
> [Wed May 15 00:38:34.388348 2013] [pagespeed:error] [pid 20292:tid
> 140214747961088] [mod_pagespeed 1.3.25.4-2941 @20292] Serf status
> 111(Connection refused) polling for 21 threaded fetches for 0.05 seconds
>
> Troubleshooting that I've discovered that during my first request to the
> domain.com site (when mod_pagespeed fetch all site components to background
> optimisation), that mod_pagespeed is opening +70 connections to the website
> backend (10.10.254.183 in this case) and is getting dropped by some filter.
>
> My question is if there is any option to control/limit how many connections
> mod_pagespeed can open with each backend server ? I already tried to control
> that thought ProxyPass max directive without success....
>
> Follow my pagespeed and virtualhost config:
>
> I've a pagespeed.conf where I define the "global config":
>
> LoadModule pagespeed_module
> /opt/apps/mod_pagespeed/modules/mod_pagespeed_ap24.so
> <IfModule pagespeed_module>
>
> ModPagespeed off
> ModPagespeedInheritVHostConfig on
> ModPagespeedFetcherTimeoutMs 60000
> ....
> </IfModule>
>
>
> Insite each virtualhost file I turn mod_pagespeed on and make some specific
> config:
>
> <VirtualHost *:80>
> ServerName www.domain.com
> ServerAlias domain.com
>
> ProxyPreserveHost On
> ProxyPass / http://10.10.254.183/ max=5 connectiontimeout=10 timeout=15
> ProxyPassReverse / http://10.10.254.183/
>
> ErrorLog "logs/error_log"
> CustomLog "logs/access_log" combinedio
>
> ModPagespeed on
>
> ModPagespeedFileCachePath "/opt/apps/mod_pagespeed/cache"
>
> ModPagespeedDomain http://domain.com
> ModPagespeedDomain http://www.domain.com
>
> ModPagespeedRewriteLevel CoreFilters
>
> ModPagespeedEnableFilters rewrite_javascript,combine_javascript
> ModPagespeedEnableFilters rewrite_css,combine_css,inline_css
> ModPagespeedEnableFilters collapse_whitespace
> ModPagespeedEnableFilters move_css_to_head
> #ModPagespeedEnableFilters
> elide_attributes,combine_javascript,remove_comments
> ModPagespeedEnableFilters rewrite_images,recompress_images
> ModPagespeedEnableFilters defer_javascript
> ModPagespeedEnableFilters rewrite_domains
>
> ModPagespeedEnableFilters make_google_analytics_async
>
> ModPagespeedCssInlineMaxBytes 2048
>
> ModPagespeedImageMaxRewritesAtOnce 0
>
> <Location /mod_pagespeed_statistics>
> Order allow,deny
> Allow from localhost
> Allow from 127.0.0.1
> Allow from 192.168.100.20
> SetHandler mod_pagespeed_statistics
> </Location>
>
> </VirtualHost>
>
> Regards,
> Alexandre
>
> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Alexandre Biancalana]

Hi Maksim!

 I wasn't aware of this config, this is not mentioned in any documentation... Do you know if work in version 1.3.25.4 ?

 I'll give it a try.

Regards,

Alexandre

[Maksim Orlovich]

On Wed, May 15, 2013 at 11:06 AM, Alexandre Biancalana
<bianc...@gmail.com> wrote:
> Hi Maksim!
>
> I wasn't aware of this config, this is not mentioned in any
> documentation... Do you know if work in version 1.3.25.4 ?

It's supported in 1.3.25.4. The reason it's not documented is that we don't
have enough data on its effectiveness at this point to do so usefully ---
if it helps you, that would certainly go a long way :)

Hope this helps,
Maks

[Alexandre Biancalana]

What´s is the rate limit applied when it´s turned on ?

Is there any other parameter to control the connection opening rate or the maximum number of opened connections ?

Thinking about that and sniffing the traffic between pagespeed and backend server I had came across with another question:

 Even Serf have been using HTTP/1.1 to issue the backend requests, I can see only one request per connection, Is there any configuration to control this behavior enabling issuing multiple requests in the same connection ?

Thank you for your time,

Alexandre

[Alexandre Biancalana]

Sorry for the top post.

I´ve enabled ModPagespeedRateLimitBackgroundFetches, but the behavior is the same pagespeed still open too much connections...

[Alexandre Biancalana]

On Wed, May 15, 2013 at 4:02 PM, Alexandre Biancalana <bianc...@gmail.com> wrote:

What´s is the rate limit applied when it´s turned on ?

Is there any other parameter to control the connection opening rate or the maximum number of opened connections ?

Thinking about that and sniffing the traffic between pagespeed and backend server I had came across with another question:

 Even Serf have been using HTTP/1.1 to issue the backend requests, I can see only one request per connection, Is there any configuration to control this behavior enabling issuing multiple requests in the same connection ?

Thank you for your time,

Alexandre

Someone knows if that kind of limits exists or can be configured ? 

[Maksim Orlovich]

On Thu, May 23, 2013 at 1:42 PM, Alexandre Biancalana

<bianc...@gmail.com> wrote:
> On Wed, May 15, 2013 at 4:02 PM, Alexandre Biancalana <bianc...@gmail.com>
> wrote:
>>
>> What´s is the rate limit applied when it´s turned on ?

So the rate limit is supposed to be 1 background request per domain in single
threaded MPM, and 4 for multi-threaded ones, but it sounds like it's
not working for you..

>> Is there any other parameter to control the connection opening rate or the
>> maximum number of opened connections ?
>>
>> Thinking about that and sniffing the traffic between pagespeed and backend
>> server I had came across with another question:
>>
>> Even Serf have been using HTTP/1.1 to issue the backend requests, I can
>> see only one request per connection, Is there any configuration to control
>> this behavior enabling issuing multiple requests in the same connection ?

Unfortunately not.