Re: pagespeed behind load balancer problems

[Jeremy Heffner]

Jonathan --

Have you used the origin domain mapping option?

https://developers.google.com/speed/docs/mod_pagespeed/domains

Jeremy

--
Jeremy Heffner

Azavea  |  340 N 12th St, Suite 402, Philadelphia, PA
jhef...@azavea.com  |  T 215.701.7712  |  F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  |  Twitter @azavea

On Thu, Jun 28, 2012 at 4:38 PM, jone <jona...@bertieandbean.com> wrote:

pagespeed rewrites "/css/file.css" to "http://domainurl/css/file.pagespeedstuff.css"

thus this can attempt to pull the css file from a server behind the load balancer that hasn't created the pagespeed css file

i want each server to pull the css from the local pagespeed cache and not through the load balancer

[Matt Atterbury]

On Thu, Jun 28, 2012 at 4:38 PM, jone <jona...@bertieandbean.com> wrote:

pagespeed rewrites "/css/file.css" to "http://domainurl/css/file.pagespeedstuff.css"

thus this can attempt to pull the css file from a server behind the load balancer that hasn't created the pagespeed css file

i want each server to pull the css from the local pagespeed cache and not through the load balancer

I presume you're using the default set of core filters? And I'm guessing this is a new install?

The latest release removed trim_urls from the core filter set:

This release includes the following miscellanea:

• The handling of HTML escapes in element attributes has been made more robust
• We are removing “trim_urls” from the Core set to better support Ajax loading of HTML from a different path (Issue 393).
• Improved memory efficiency during HTML parsing and rewriting of large documents
• Many improvements to quality and performance of the rewriting engine

 

Could you please enable trim_urls and see if the helps:

ModPagespeedEnableFilters trim_urls

(https://developers.google.com/speed/docs/mod_pagespeed/filter-trim-urls)

m.

[Jeremy Heffner]

Jonathan --

I haven't tried it with replacing ip addresses from the html.

Jeremy

--
Jeremy Heffner

Azavea  |  340 N 12th St, Suite 402, Philadelphia, PA
jhef...@azavea.com  |  T 215.701.7712  |  F 215.925.2663
Web azavea.com  |  Blog azavea.com/blogs  |  Twitter @azavea

On Thu, Jun 28, 2012 at 6:28 PM, jone <jona...@bertieandbean.com> wrote:

Hi Jeremy

Yes I tried this before I posted but couldn't get it to change anything. Not sure whether the fact that I'm testing this using an ip address makes any difference, but I tried:

ModPagespeedMapOriginDomain localhost http://ipaddress
ModPagespeedMapOriginDomain localhost *.ipaddress
ModPagespeedMapOriginDomain localhost *ipaddress
ModPagespeedMapOriginDomain localhost ipaddress
ModPagespeedMapOriginDomain http://localhost http://ipaddress
ModPagespeedMapOriginDomain localhost / (caused a problem in google_url.cc)

Jonathan

[Matt Atterbury]

Jonathan,

Taking a step back, I would like to ask why the current behavior is a problem.

Yes, the server that is asked for the rewritten resource will have to re-fetch and re-rewrite it, but it will then cache it and now we'll have 2 backend servers that have it.

If yet another request comes in and goes to yet another backend server, we'll end up with 3 with it.

Eventually we might end up with all of them having it in which case it will definitely be served from cache.

Now that sounds bad in theory, but in our experience it's not so bad in practice, since the re-fetching and re-rewriting and re-caching don't take too long.

It means that initial requests take longer than desired but that generally fixes itself reasonably quickly.

Do you find that it isn't so good in practice, or did you notice this behavior and think it wrong?

Thanks, m.

--
"Klaatu barada nikto"                          (754) 444-6288

[Matt Atterbury]

Matt, to answer your question, if it was just a case of slow requests that would be not only acceptable, but preferable behaviour as you suggest. Unfortunately, missing images and badly formatted pages aren't acceptable for the users.

So the servers behind the load balancer aren't Apache servers running mod_pagespeed? If they were, they *would* rewrite the resource if its name matched the pagespeed pattern _and_ if it could fetch the decoded name OK.

m.

[Matt Atterbury]

OK, so have a look in your Apache logs on a server that 404s the pagespeed resource.

There should be some indication in there of what went wrong.

My guess is that it can't fetch the original resource for some reason.

cheers, m.

[b...@messagebird.com]

Hi,

This is kind of an old post, but I am having the same issue at this moment - and I can't seem to find a solution.

Our setup:

Loadbalancer

 -> Webserver1 with pagespeed (shared memcached)

 -> Webserver2 with pagespeed (shared memcached)

 -> Webserver3 with pagespeed (shared memcached)

On top of this, we have a cloudfront CDN that requests the file at our side if it is not cached yet. 

Now at this moment, when visitor A visits webserver1, pagespeed creates some css, js and image files and stores them in the shared memcached. All URL's are rewritten to de CDN (MapRewriteDomain) and the CDN then requests the files with us, but the CDN can request this file at Webserver2 (because of our loadbalancer). Webserver2 then says: I don't know this file - and cloudfront throws a 404. 

Is there a way to really share the cache (apparently memcached doesnt really do this in the way I was hoping), without using a shared file system?

--- Bob 

[Jeff Kaufman]

Do all of your webservers have exactly the same pagespeed
configuration? What you're describing should work; that's the
standard use case we added memcached support for.

Part of what's surprising here, is that even if you weren't using
memcached and so didn't have a shared cache at all, you still
shouldn't be getting 404s because pagespeed should be able to
interpret and reconstruct a pagespeed-created url even if it was
generated on another server.

Can you share a link to your site?

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mod-pagespeed-di...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mod-pagespeed-discuss/ff1696d3-26ca-4432-b6aa-68e1bf58ff3d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[b...@messagebird.com]

Hi Jeff,

Thanks for your reply!

The pagespeed configuration is indeed the same on all webservers. I added the current configuration below.

I did turn off the CDN at this moment, because we cannot run it in production when 404 error's on images, css and JS occur, but maybe you see something on the config that could give this problem.

As you can see we currently use the memcached on the local servers, but it had the same problems with a shared memcached.

I just added the CDN part for a couple of seconds, and immediately see 404 requests from cloudfront coming in:

12/Mar/2015:10:19:06 +0100: 192.168.1.1 (ip.address.here.wh, 216.137.58.43): messagebird.com/frontend-assets/images/cases/gif/234x174xamberalert-500x500.gif.pagespeed.ic.jYKOBa9lq-.webp -> - [404] [u: - r: 5.015]

12/Mar/2015:10:19:06 +0100: 192.168.1.1 (ip.address.here.wh, 216.137.58.43): messagebird.com/frontend-assets/images/references/xarjandejong.png.pagespeed.ic.nq0lJAzpOQ.webp -> - [404] [u: - r: 5.062]

12/Mar/2015:10:19:06 +0100: 192.168.1.1 (ip.address.here.wh, 216.137.58.43): messagebird.com/frontend-assets/images/references/xjaapvantriet.png.pagespeed.ic.eFczgC8zaK.webp -> - [404] [u: - r: 5.052]

12/Mar/2015:10:19:06 +0100: 192.168.1.1 (ip.address.here.wh, 216.137.58.43): messagebird.com/frontend-assets/images/cases/gif/234x174xtnw-500x500.gif.pagespeed.ic.gTxXxWxbNj.webp -> - [404] [u: - r: 5.026]

12/Mar/2015:10:19:07 +0100: 192.168.1.1 (ip.address.here.wh, 216.137.58.43): messagebird.com/frontend-assets/images/cases/gif/234x174xdominos-500x500.gif.pagespeed.ic.PVDwWKYSWd.webp

When I go to messagebird.com/frontend-assets/images/cases/gif/234x174xdominos-500x500.gif.pagespeed.ic.PVDwWKYSWd.webp it seems to work at this moment, but that's because I am probably not send to the same webserver.

Is there a pagespeed log that can maybe tell me what is going wrong?

Thanks!

pagespeed on;

pagespeed RewriteLevel PassThrough;

pagespeed MemcachedServers "127.0.0.1:11211";

pagespeed FileCachePath /var/ngx_pagespeed_cache;

# Ensure requests for pagespeed optimized resources go to the pagespeed handler

# and no extraneous headers get set.

location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {

add_header "" "";

}

location ~ "^/  /" { }

location ~ "^/ngx_pagespeed_beacon$" { }

# pagespeed MapRewriteDomain cdn.messagebird.com *messagebird.com;

# pagespeed RespectXForwardedProto on;

# pagespeed FetchHttps enable;

# pagespeed MapRewriteDomain "https://cdn.messagebird.com" "https://www.messagebird.com"

pagespeed EnableFilters make_google_analytics_async;

pagespeed EnableFilters rewrite_images;

pagespeed EnableFilters convert_jpeg_to_progressive;

pagespeed EnableFilters convert_png_to_jpeg;

pagespeed EnableFilters convert_jpeg_to_webp;

pagespeed EnableFilters convert_to_webp_lossless;

pagespeed EnableFilters inline_images;

pagespeed EnableFilters recompress_images;

pagespeed EnableFilters recompress_png;

pagespeed EnableFilters recompress_webp;

pagespeed EnableFilters resize_images;

pagespeed EnableFilters resize_rendered_image_dimensions;

pagespeed EnableFilters resize_mobile_images;

pagespeed EnableFilters collapse_whitespace;

pagespeed EnableFilters in_place_optimize_for_browser;

pagespeed EnableFilters pedantic;

pagespeed EnableFilters insert_dns_prefetch;

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[b...@messagebird.com]

I'm Sorry, I just noticed that this is the mod_pagespeed group, and not the pagespeed in general! So this is probably something specific to Nginx, shall I post this in the Nginx group or is this something that you can also help with?

On Tuesday, March 10, 2015 at 12:27:32 PM UTC+1, Jeff Kaufman wrote:

> email to mod-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

It's ok, we can keep the discussion here; I think this is probably not
nginx-specific.

PageSpeed uses the regular server error log, though you might need to
turn it up to make it more sensitive. Do you see anything in
error_log when getting one of these 404s?

>> > email to mod-pagespeed-di...@googlegroups.com.

>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/mod-pagespeed-discuss/ff1696d3-26ca-4432-b6aa-68e1bf58ff3d%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mod-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to mod-pagespeed-di...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/88610a96-d407-461f-b6f4-0988f4892da0%40googlegroups.com.