ngx_pagespeed on haproxy load balanced server backends using CDNs for CSS/JS url links ?

[Centmin Mod George]

I've moved my ngx_pagespeed testing from local virtualbox CentOS 6.4 instance, to my live site for testing. But have encountered an interesting issue with haproxy load balanced setup and CDN usage and was wondering how and what ngx_pagespeed settings and options can be used to work around this.

Background of live site/server setup:

* haproxy 1.5.x front end on port 80 load balancer

* 4x 512MB memory based VPS each running nginx 1.4.0 backend on port 8888

* 1 of the VPS with ngx_pagespeed 1.5.27.2 beta for testing first before deploying to the other 3x VPSes

* All run CentOS 6.4 32bit

VPS ip addresses and haproxy sets a cookie to ensure persistance on same backend

* vps1 ip address = 1.2.3.4 no ngx_pagespeed installed yet (cookie name vps1)

* vps2 ip address = 1.2.3.5 no ngx_pagespeed installed yet (cookie name vps2)

* vps3 ip address = 1.2.3.6 no ngx_pagespeed installed yet (cookie name vps3)

* vps4 ip address = 1.2.3.7 only this VPS with ngx_pagespeed installed (cookie name vps4)

For CSS and JS files, I'm using pull based CDN service from cdn77.com with CNAME cdn.mydomain.com looking to mydomain.com as the origin. 

So have following settings in ngx

pagespeed EnableFilters rewrite_domains;

pagespeed Domain cdn.mydomain.com;

pagespeed MapRewriteDomain cdn.mydomain.com/css mydomain.com/css; 

pagespeed MapRewriteDomain cdn.mydomain.com/js mydomain.com/js;

So when visitor hits mydomain.com, haproxy will direct them to 1 of 4 ip addresses 1.2.3.4 or 1.2.3.5 or 1.2.3.6 or 1.2.3.7. When it hits 1.2.3.7 ip VPS server, ngx_pagespeed is in play. So CSS files get rewritten to format such as http://cdn.mydomain.com/css/bootstrap.css+bootstrap-responsive.css+style.css.pagespeed.cc.SXpmva6V8K.css

** The Problem **

Now this is all fine so far... well that's what I thought until I started testing mydomain.com on various page speed test sites such as loads.in, webpagetest.org and gtmetrix.com. Sometimes, these tests report bad requests where the ngx_pagespeed rewritten CSS/JS url link gives 404/410 errors. 

When I check the waterfall output and look at response header for the haproxy set cookie on these bad request tests, I notice that mydomain.com html has cookie name vps4 suggesting it was served from vps4 on 1.2.3.7. But the bad requested CSS or JS file has response header cookie set to vps1 or vps2 or vps3 for the non ngx_pagespeed based Nginx backend servers.

I guess this is because the MapRewriteDomain is placing cdn.mydomain.com into html and that cdn.mydomain CNAME is set to mydomain.com origin on port 80 via cdn77.com CDN service which hits haproxy first and haproxy is directing the request to a VPS that doesn't have ngx_pagespeed enabled. So what should I be doing ngx_pagespeed settings wise ?

If I install ngx_pagespeed on vps1, vps2 and vps3 as well, won't the rewritten CSS/JS url links differ between each VPS as well ? So hitting cdn.mydomain.com CSS/JS links won't necessarily call the right VPS ?

** Possible solution ??? **

Would I need to map the origin domain to each VPS's specific ipaddress instead for each VPS ?

i.e. for vps4 set pagespeed option 

pagespeed MapOriginDomain 1.2.3.7:8888 cdn.mydomain.com;

vps4 1.2.3.7 server's Nginx vhost would contain

server {

            listen   8888;

  server_name mydomain.com.com cdn.mydomain.com;

Would that work ? 

[Jeff Kaufman]

On Sat, May 4, 2013 at 8:26 AM, Centmin Mod George <ng...@centminmod.com> wrote:
> when visitor hits mydomain.com, haproxy will direct them to 1
> of 4 ip addresses 1.2.3.4 or 1.2.3.5 or 1.2.3.6 or 1.2.3.7. When
> it hits 1.2.3.7 ip VPS server, ngx_pagespeed is in play

Do I understand this properly: every request to your website will get
ngx_pagespeed 1/4 of the time? This isn't going to work. Imagine
this:

1. User requests /index.html
2. The request hits 1.2.3.7, gets ngx_pagespeed
3. ngx_pagespeed rewrites foo.css to /foo.css.pagespeed.ce.0123456789.css
4. User's browser requests /foo.css.pagespeed.ce.0123456789.css
5. The request hits 1.2.3.4, doesn't get ngx_pagespeed
6. The server on 1.2.3.4 doesn't know what to do with
/foo.css.pagespeed.ce.0123456789.css, issues a 404.

If ngx_pagespeed is running on the server that processes the html, it
also needs to be running on any server that might get the request for
the rewritten resources.

Jeff

[Centmin Mod George]

Yes for now only rolled out ngx_pagespeed on 1 of 4 VPS servers. But due to haproxy cookie persistence it's okay if CSS/JS rewritten resource is served from mydomain.com (non-CDN) as mydomain.com/css resources stick to same haproxy set cookie server throughout the user's session on the site. This works right now on the live site and no 404 errors.

The problem comes when i set ngx_pagespeed to rewrite domain to CDN url cdn.mydomain.com 

i.e. cdn.mydomain/css. 

As cdn.mydomain.com at cdn77.com service is pull based it looks to mydomain.com as the source but mydomain.com is haproxy load balancer based, so there is a chance cdn.mydomain.com CSS/JS file lookup at mydomain.com get's one of the non ngx-pagespeed setup servers. 

I think i would still have the problem if all 4 VPS had ngx_pagespeed when using cdn.mydomain.com rewritten CSS/JS url links, as there's no guarantee each VPS server's ngx_pagespeed rewrites the CSS/JS url link exact the same ? I guess problem is at cdn.mydomain.com needing to keep the same haproxy cookie for each VPS ?

Just curious if there's anything in ngx_pagespeed config/settings to tackle this ?

Or could tackle this from Nginx config end by adding a cookie/haproxy cookie to CSS/JS extension files rewritten by page speed ? a Nginx regex match on pagespeed *.css or *.js that adds a cookie ?

[Jeff Kaufman]

To get this to work, requests from the browser to cdn.mydomain.com
would need to go out with a haproxy cookie, but I don't see how you
would assign that cookie.

> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ngx-pagespeed-di...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/ngx-pagespeed-discuss?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Centmin Mod George]

Yeah, essentially, the mydomain.com page call becomes 

mydomain.com/index tagged with vps4 cookie which is ngx_pagespeed nginx backend based 

cdn.mydomain.com/css/style.pagespeed.css tagged with vps1 cookie which isn't ngx_pagespeed based so style.css gets 404 not found error 

I think haproxy sees 2 user sessions here, the first being from actual user hitting mydomain.com and 2nd user it considers the cdn.mydomain.com calling mydomain.com original for the css file so it has a different cookie. 

Guess I need to figure out a way of having load balanced ngx_pagespeed based backends and still use a pull based CDN or just live without the CDN part.

[Nico Andrade]

Hello

I'm facing a similar situation, and my approach will be to get another CDN property and set it's Origin server to pagespeed.static.mydomain.com

Nginx in the PageSpeed-powered node will listen to both mydomain.com and pagespeed.origin.mydomain.com

In that way I make sure the CDN pulls the file from a server actually having it.

Does this make sense?

Nic