pagespeed and fastcgi cache

2,252 views
Skip to first unread message

Jan-Philip Gehrcke

unread,
Mar 11, 2013, 1:03:58 PM3/11/13
to ngx-pagesp...@googlegroups.com
Hello,

I have a production system running nginx in front of a PHP-FPM backend. Nginx' fastcgi_cache is set up to cache backend responses in the file system. This works perfectly fine by itself.

Now, I am interested in adding pagespeed optimizations. Naive addition of pagespeed to that stack seems to bypass the fastcgi_cache. That's not what I want. It would make sense to apply the pagespeed optimizations to a backend response right before letting fastcgi_cache caching it. Can we do that? Or are there other approaches for the combination of both, fastcgi_cache and pagespeed?

Thanks for valuable insights,

Jan-Philip

 

Jan-Philip Gehrcke

unread,
Mar 11, 2013, 1:31:59 PM3/11/13
to ngx-pagesp...@googlegroups.com
"Naive addition of pagespeed to that stack seems to bypass the fastcgi_cache" was false alarm, sorry. I got them now running together. The fastcgi cache contains the non-optimized code. Before being delivered to the client, pagespeed optimizes it.

Is that a proper setup? Is there an easy way to add pagespeed between nginx and the FastCGI backend so that the FastCGI cache actually stores pagespeed-optimized responses?

Thanks,

Jan-Philip

Jeff Kaufman

unread,
Mar 11, 2013, 2:39:21 PM3/11/13
to ngx-pagesp...@googlegroups.com
On Mon, Mar 11, 2013 at 1:31 PM, Jan-Philip Gehrcke
<jgeh...@googlemail.com> wrote:
> "Naive addition of pagespeed to that stack seems to bypass the
> fastcgi_cache" was false alarm, sorry. I got them now running together. The
> fastcgi cache contains the non-optimized code. Before being delivered to the
> client, pagespeed optimizes it.
>

That's what I would expect.

>
> Is there an easy way to add pagespeed between nginx and the
> FastCGI backend so that the FastCGI cache actually stores
> pagespeed-optimized responses?
>

Summary: html that comes out of pagespeed is intentionally non-cacheable.

Long version. Pagespeed doesn't generally manage to rewrite
everything the first time. While the html is passing through it needs
to go get the referenced resources, rewrite them, and put them back in
the html. That process can take a while, and when it takes more than
a few ms it just sends on the html unmodified or partially modified.
If you cached the output of pagespeed immediately you'd be caching
only a partly rewritten page.

Even worse, imagine you have a page like:

<html>
...
<link href="style.css">
<img src="todays-date.png">

This runs through pagespeed and becomes:

<html>
...
<link href="V.style.css.pagespeed.cf.KM5K8SbHQL.css">
<link href="vtodays-date.png.pagespeed.ic.2FihTuVNs_.png">

Caching headers have changed too. Imagine before pagespeed we had:

html:
Cache-Control: public,max-age=10000

style.css:
Cache-Control: public,max-age=10000

todays-date.png:
Cache-Control: public,max-age=300

This is intentional. The site owner is ok with changes to the html
and style.css taking a little while to propagate, balancing speed and
freshness, but they want todays-date.png to be always current.
(Because when tomorrow comes todays-date.png needs to be updated.)

Now pagespeed comes and uses the following headers:

<html>
...
<link href="V.style.css.pagespeed.cf.KM5K8SbHQL.css">
<link href="vtodays-date.png.pagespeed.ic.2FihTuVNs_.png">

html:
Cache-Control: no-cache,max-age=0

V.style.css.pagespeed.cf.KM5K8SbHQL.css:
Cache-Control: max-age=31536000

vtodays-date.png.pagespeed.ic.2FihTuVNs_.png:
Cache-Control: max-age=31536000

Now the site owner no longer has to balance speed and freshness. When
they make a change to style.css, or when tomorrow rolls around and
todays-date.png is automatically updated, pagespeed will change the
html to link to the new versions of the files. But this only works
when the html isn't cached.

You might think that a smarter version of pagespeed could take into
account cache control headers for every resource linked from the html
and use the shortest one for the html's caching header. Plus it could
track whether it has fully rewritten the page, and only mark the html
as cacheable if it has finished. Except that pagespeed needs to send
out headers before it's seen the whole html. The best solution we
have so far is to keep the html non-cacheable, even though it's kind
of a cop-out.

Jeff

Ovidiu Pacuraru

unread,
Mar 14, 2013, 6:08:58 PM3/14/13
to ngx-pagesp...@googlegroups.com
same setup here, my fast_cgi cache does only cache stuff for non-logged-in visitors or visitors who don't have a cookie. 
BUT those are the majority of my visitors hence it makes sense.

So I was asking myself if I put ngx_pagespeed in front of the fastcgi_cache I do understand that it still uses the cache and then optimizes it but is the effort to optimize that cache before sending it to the client worth what I save by using fast_cgi cache? I mean does it not make fastcgi_cache redundant?

And then I rememebred: #pagespeed FileCachePath /var/cache/ngx_pagespeed_cache; 

does that mean that basically ngx_pagespeed DOES cache its own output? 

Can someone elaborate the flow a little bit more to help clarify this issue?

Jeff Kaufman

unread,
Mar 15, 2013, 9:28:21 AM3/15/13
to ngx-pagesp...@googlegroups.com
I'm having trouble understanding what you're asking.

ngx_pagespeed doesn't cache it's html output, but it does cache the
results of optimizations like image compression.

If your request flow goes:

user -> ngx_pagespeed -> fastcgi_cache -> fastcgi

then fastcgi_cache is not redundant because it will still reduce the
work for fastcgi.
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ngx-pagespeed-di...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/ngx-pagespeed-discuss?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Pacuraru, Ovidiu

unread,
Mar 15, 2013, 11:21:09 AM3/15/13
to ngx-pagesp...@googlegroups.com
I managed to confuse myself with my last post, I guess I was just thinking out loud :-) 

Did I get this right?

 user -> ngx_pagespeed (cached optimizations) -> ngx_pagespeed -> fastcgi_cache -> fastcgi 

So basically if there is a cached optimized version of ngx_pagespeed available that is delivered, if not HTMl is fetched from fastcgi_cache and optimized and then delivered and if the cache is empty only then the query hits fastcgi ?



--
The information in this email and attachments hereto may contain legally privileged, proprietary or confidential information that is intended for a particular recipient. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, retention or use of the contents of this e-mail information is prohibited and may be unlawful. When addressed to customers or vendors, any information contained in this e-mail is subject to the terms and conditions in the governing contract, if applicable. If you have received this communication in error, please immediately notify us by return e-mail, permanently delete any electronic copies of this communication and destroy any paper copies.

Jeff Kaufman

unread,
Mar 15, 2013, 11:26:42 AM3/15/13
to ngx-pagesp...@googlegroups.com
On Fri, Mar 15, 2013 at 11:21 AM, Pacuraru, Ovidiu <ovi...@pacura.ru> wrote:
>
> user -> ngx_pagespeed (cached optimizations) -> ngx_pagespeed ->
> fastcgi_cache -> fastcgi
>
> So basically if there is a cached optimized version of ngx_pagespeed
> available that is delivered, if not HTMl is fetched from fastcgi_cache and
> optimized and then delivered and if the cache is empty only then the query
> hits fastcgi ?
>

No, it's more like:

User requests html with cold cache:

user -> ngx_pagespeed -> fastcgi_cache -> fastcgi

User requests html, fastcgi_cache has it in cache:

user -> ngx_pagespeed -> fastcgi_cache

User requests .pagespeed. optimized resource (css, js, img):

user -> ngx_pagesepeed

In other words, ngx_pagespeed caches resources, but it doesn't cache
html. Every html request will be passed back to fastcgi_cache.

Jeff

Pacuraru, Ovidiu

unread,
Mar 15, 2013, 11:34:59 AM3/15/13
to ngx-pagesp...@googlegroups.com
Ok, lets see if I got it this time:

The data travels backwards after that request:

 user <- ngx_pagespeed <- fastcgi_cache <- fastcgi 

and ngx_pagespeed replaces css, jpg, img with optimized versions since obviously fastcgi/fastcgi_cache forwards "un-optimized" resources?
ngx_pagespeed would also store the optimized css/jpg/js files but NOT the optimized HTML so it does optimize the HTML it gets back from fastcgi/fastcgi_cache on every pass?

Jan-Philip Gehrcke

unread,
Mar 15, 2013, 11:42:18 AM3/15/13
to ngx-pagesp...@googlegroups.com
That's what Jeff is saying. I guess this on-the-fly HTML processing is
extremely fast, so nothing to worry about performance-wise.

>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "ngx-pagespeed-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to

Jeff Kaufman

unread,
Mar 15, 2013, 1:20:16 PM3/15/13
to ngx-pagesp...@googlegroups.com
Yes.

Dominique Dutra

unread,
Oct 18, 2013, 8:13:51 AM10/18/13
to ngx-pagesp...@googlegroups.com
Hello guys

How did you solved this?

I am having the same problem. I run nginx + fastcgi_cache. Trying to add some aggressive pagespeed optimization on the mix and have static asset urls rewrited to origin pull cdn. This is working just fine.
Problem is the html output of php pages are processed by ngx_pagespeed on every request, thus my poor throughput.

I would like fastcgi_cache to cache OPTIMIZED OUTPUT to tmpfs. Why does ngx_pagespeed has to do its thing on the html everytime?

If I disable ngx_pagespeed I get extreme performance.

Best regards.

Dominique Dutra

unread,
Oct 18, 2013, 9:28:13 AM10/18/13
to ngx-pagesp...@googlegroups.com
Well I missed jeffkaufman explanation before. Sorry.

I am looking into Varnish in front of nginx, dropping fastcgi_cache and using https://developers.google.com/speed/pagespeed/module/downstream-caching.

Maybe thats worth a try.

best regards.

Ovidiu Pacuraru

unread,
Apr 11, 2014, 4:40:34 PM4/11/14
to ngx-pagesp...@googlegroups.com
Has anyone done some realistic speed testing to see the actual difference between using fastcgi_cache WITH  and WITHOUT ngx_pagespeed? Really curious to figure out if adding ngx_pagespeed to the mix leads to loss of speed or not...

Jeff Kaufman

unread,
Apr 11, 2014, 5:07:35 PM4/11/14
to ngx-pagesp...@googlegroups.com
One way to do this test would be with ngx_pagespeed's A/B test
framework: https://developers.google.com/speed/pagespeed/module/module-run-experiment
Give half your visitors a pagespeed-optimized site and the other half
the regular site, and compare page load times.
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ngx-pagespeed-di...@googlegroups.com.
> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
> For more options, visit https://groups.google.com/d/optout.

Ovidiu Pacuraru

unread,
Apr 12, 2014, 5:57:41 PM4/12/14
to ngx-pagesp...@googlegroups.com
WOW! I didn't even know stuff like that existed for pagespeed! Slowly reading my way through that stuff...

Rahul Bansal

unread,
Apr 16, 2015, 4:36:12 PM4/16/15
to ngx-pagesp...@googlegroups.com
Me and some other users did test this - https://github.com/rtCamp/easyengine/issues/497

My results:
It turns out that fastcgi-cache without pagespeed is much better!

Now I am also trying to figure out how to use pagespeed optimizations without compromising power of fastcgi-cache.

Otto van der Schaaf

unread,
Apr 16, 2015, 4:50:31 PM4/16/15
to ngx-pagesp...@googlegroups.com
What did you exactly benchmark? Html throughput only?
How large are the html pages? The hit rate seems with ngx_pagespeed seems far too low.
Are you by chance serializing access to fastcgi (fastcgi_lock)? That would cause only one page to be generated by fastcgi at a time at the speed of fastcgi, as ngx_pagespeed disallows caching for html in the default setup.

I'd recommend comparing against a downstream caching setup for a fairer comparison:

Otto


--

Rahul Bansal

unread,
Apr 16, 2015, 4:54:28 PM4/16/15
to ngx-pagesp...@googlegroups.com
Hi Otto,

Thanks for your attention. :-)

I did not touch fastcgi_lock value. So whatever value is used is nginx default value.

I will be trying downstream caching soon as based on description it seems it will solve the problem.

HTML page is here - http://rtcamp.cloudapp.net/ 
Its wordpress site but homepage was already cached before I ran test. So the test did not account anything related to php/mysql.

Thanks again.

--
Rahul Bansal | Founder & CEO 
rtCamp Solutions Pvt. Ltd. | https://rtcamp.com/

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Vidyut

unread,
Apr 16, 2015, 8:26:52 PM4/16/15
to ngx-pagesp...@googlegroups.com
I am interested in your results, Rahul. I am trying to get the proxy-cache to work as advertized. Right now I'm getting terrible 1 second waits on static CACHED resources - images and such (https://aamjanata.com). I'm sure it is something to do with my configuration, since Varnish never gave any problems (but now I am ssl enabled everywhere). I've posted some changes I've made to the recommended downstream configuration on another thread here - which may help in speeding it up some - I'd like your inputs on that too. I've read your pieces on Nginx configurations and they have saved my skin in other instances :D

So basically I'm hoping to see if I can copy anything from settings you use :p

Otto van der Schaaf

unread,
Apr 17, 2015, 4:04:28 AM4/17/15
to ngx-pagesp...@googlegroups.com
Getting back to this, I ran a quick test with fastcgi_cache with and without ngx_pagespeed,
to see if there's anything wrong. I just wanted to get the ballpark numbers, so I'm not going into
too much detail here.

The relevant nginx configuration I used: https://gist.github.com/oschaaf/5c4504877db5a132e531
I was benchmarking  a php script that sleeps one second before outputting a small 100 byte page.
After warming up the cache, the numbers looked like this:

- With ngx_pagespeed disabled, the server was able to serve up the page around 15000 times/second
- With ngx_pagespeed enabled, the server was able to serve up the page around 2350 times/second

With ngx_pagespeed enabled, things do run a bit slower, but in your test the numbers look more like php
execution speed instead of the fastcgi_cache+ngx_pagespeed execution speed (averaging around 21 requests per second).
Enabling ngx_pagespeed causes a drop in rps, as now each response is transformed and optimized on the
fly as it is served by the fastcgi_cache, which has a cost in terms of cpu.

My experience with setups similar to the fastcgi_cache setup here is that it will be able to handle a substantial amount of 
traffic, as html represents only a fraction of all the responses that a webserver serves. The simplicity of this setup is an upside.
However, if you really need all the performance you can get, a correct downstream caching setup should help.

That setup relies on proxy_cache which I expect to run at more or less the same speed as fastcgi_cache. 
Note that you'd have to test with `Accept-Encoding: gzip` in the requests in the downstream cache scenario, or else the 
configuration will bypass the cache if you set it up according to the docs. 
With the change proposed by Vidyut, without specifying the Accept-Encoding, nginx will have to decompress on each request.

Otto

On Thu, Apr 16, 2015 at 10:36 PM, Rahul Bansal <rahul....@rtcamp.com> wrote:

--

Jeff Kaufman

unread,
Apr 17, 2015, 8:20:30 AM4/17/15
to ngx-pagesp...@googlegroups.com
Looking at your tests, it seems you ran:

GET http://rtcamp.cloudapp.net:80

Because this will only load the html, it will captures all of the
downside of PageSpeed and none of the upside. PageSpeed parses and
optimizes the html as it goes through so it can make changes for it to
load faster for browsers. For example, it might inline a small css
file or cache-extend a larger one. If you load the whole page this is
a speed gain, because it cuts out round trips and improves cache
performance, but if you load only the html you don't get that benefit.

This is a problem with running this kind of a test, but as Otto
noticed that's not why you're getting such low performance with
ngx_pagespeed. The issue is that with fastcgi_cache your html ends up
cached while with ngx_pagespeed it does not. In general it's not safe
to cache html, so PageSpeed doesn't do it by default, but if you don't
have any user-specific dynamic code then enabling it by turning on
downstream caching should take a lot of load off php.

Vidyut

unread,
Apr 17, 2015, 8:36:58 AM4/17/15
to ngx-pagesp...@googlegroups.com
On a tangent - I think I have brought this up before - why shouldn't html be cached if it can be planned for? There should be a decent option to do exactly that (and not just leaving caching headers be).

Some examples:
  • Static html pages - unlikely to change in a hurry like CMSs.
  • A blog or CMS that isn't updated like a news site should be able to safely cache for whatever duration is appropriate for their update frequency - including expiry time. 
  • Anyone who is fine with their homepage not showing their latest post for half an hour for people who have visited just before, in return for their website whizzing on the internet.
It ought to be a choice.

These days, the entry point to a website is usually interior pages to specific pieces (at least for my sites) - delays on archive type pages - that too around an infrequent publish - is not the end of the world if it means the browsing experience will be much faster.

I understand the reasons why you don't want to do it. But it is a legitimate trade-off to be able to set say.... html pages cacheable for 10 minutes or 10 hours or till midnight tonight (by which time most of the rush is over, and I'll publish in morning and reset the expires to tomorrow night. Or even hey... pages in this folder will never change - it is an archive (or version that will change into separate document if/when it changes or whatever) optimize the daylights out of them and cache them to the end of time!

Of all these scenarios, I'm most likely to be the "cache for 30 minutes and I'll deal with any heartbreak if people don't see fresh posts on homepage instantly".

And sure, there will be people who will be "don't you dare show these pages stale in the slightest!" and that is fine too.

Vidyut

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Jeff Kaufman

unread,
Apr 17, 2015, 9:03:00 AM4/17/15
to ngx-pagesp...@googlegroups.com
Caching your pages should definitely be a choice! That's why we
support downstream caching.

A big reason people don't like caching of html, though, is that a lot
of pages are customized per visitor. It's really important not to
take a page generated just for one visitor and show it to someone
else, which is why PageSpeed needs to be very careful around full-page
caching.

Vidyut

unread,
Apr 17, 2015, 9:31:59 AM4/17/15
to ngx-pagesp...@googlegroups.com
The downstream cache code strikes fear into the hearts of us mere mortals wishing for other than the max-age zero with the strong recommendation against changing it :D

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


Jeff Kaufman

unread,
Apr 17, 2015, 9:48:13 AM4/17/15
to ngx-pagesp...@googlegroups.com
Yes, you're right, it's much more complex than we'd like.

I'll take a pass at simplifying it today.

Rahul Bansal

unread,
Apr 20, 2015, 2:56:27 PM4/20/15
to ngx-pagesp...@googlegroups.com
@Vidyut

Can you share downstream cache config you got working?

I am digging into this. 

At first downstream cache seems overly complicated to setup.
I am wondering if we can get it working right away with fastcgi_cache rather than having to setup a proxy_cache and a proxy site.

Rahul Bansal

unread,
Apr 20, 2015, 3:04:57 PM4/20/15
to ngx-pagesp...@googlegroups.com
@Otto

In our case, web server need to handle only html response. Fonts, images, css & js are all served from CDN.
Most of the traffic is non-logged in users. (We deal with WordPress sites only)

So idea is to compress/minify/combine static stuff => rewrite static URL with CDN => rewrite html => cache it (preferably inf fastcgi). 

HTML won't change most of the time unless some admin/editor adds content.

proxy_cache and fastcgi_cache runs on same speed. I am just experimenting with downstream cache and so far it seems a lot of work. 

IMHO there should have been a flag only to turn on/off pagespeed default behavior, where it anticipates that HTML is changing all the time. 

Regards.

Rahul Bansal

unread,
Apr 20, 2015, 3:08:30 PM4/20/15
to ngx-pagesp...@googlegroups.com
Yes, you're right, it's much more complex than we'd like.

Just wondering, if there is any plan to simplify this?
 

Vidyut

unread,
Apr 20, 2015, 3:38:01 PM4/20/15
to ngx-pagesp...@googlegroups.com
Rahul,

Config lines in nginx.conf relevant to downstream that I'm using 

pagespeed DownstreamCachePurgeLocationPrefix http://128.199.212.123:443;
pagespeed DownstreamCachePurgeMethod GET;
pagespeed DownstreamCacheRewrittenPercentageThreshold 95;


to set up the cache and I have replaced the if statements with mapping since Nginx says if is evil.

map $http_user_agent $ps_capability_list {
        default "LargeScreen.SkipUADependentOptimizations";
        ~*"Chrome/|Firefox/|Gecko|Trident/6\.|Safari|Wget" "ll,ii,dj:";
        ~*"Chrome/[2][3-9]+\.|Chrome/[[3-9][0-9]+\.|Chrome/[0-9]\{3,\}\." "ll,ii,dj,jw,ws:";
        ~*"Firefox/[1-2]\.|bot|Yahoo!|Ruby|RPT-HTTPClient|Google.+\(\+https\:\/\/developers\.google\.com\/\+\/web\/snippet\/\)|Android|iPad|TouchPad|Silk-Accelerated|Kindle%20Fire" "LargeScreen.SkipUADependentOptimizations";
        ~*"Mozilla.*Android.*Mobile*|iPhone|BlackBerry|Opera.+Mobi|Opera.+Mini|SymbianOS|UP.Browser|J-PHONE|Profile/MIDP|portalmmm|DoCoMo|Obigo|Galaxy.+Nexus|GT-I9300|GT-N7100|HTC.+One|Nexus.+[4|7|S]|Xoom|XT907" "TinyScreen.SkipUADependentOptimizations";
}

In addition to this,

I'm using this in proxy-cache.conf instead of passing non-gzip to backend (Block #3b) - so only gzip versions are cached and gunzipped for browsers that won't accept gzip.

proxy_set_header Accept-Encoding gzip;
gzip_static on; #probably unnecessary as explained to me, but didn't get time to test
gunzip on;

I am no Nginx expert (I often copy stuff from your site :p) but this seems to be working fine. Do tell me if you spot a problem.

I have used pagespeed and fastcgi cache before (config from your site) - it works, somewhat. Pagespeed optimizations don't get cached, so the cache is kind of useless, though slightly faster than non-cached. It works fastest with ngx_pagespeed on backend and Varnish on front (could get load times around 1.5 seconds consistently till I switched to ssl and that story ended). I have tried with Nginx+ngx_pagespeed - Varnish - Nginx-proxy (without cache) as well as Nginx+ngx_pagespeed - Nginx-proxy-cache with similar load times (between 2.5s and 3.5s and rarely upto 4.5s) - only advantage is that Varnish manages beaconing out of the box by importing std, while none of the available Nginx packages have the misc module, so it is a compulsory compiling thing to enable passing only some beacons to backend - not got around to doing it yet. One of these days. If you figure out an alternative way to set up the beaconing, do let me know. Otherwise adding Varnish doesn't give much of a speed boost with Nginx as the ssl terminator in front as compared with Nginx for ssl termination and proxy-cache.

Vidyut

unread,
Apr 20, 2015, 3:41:23 PM4/20/15
to ngx-pagesp...@googlegroups.com
Rahul,

Missed one more thing. proxy-cache.conf should also contain

proxy_set_header PS-CapabilityList $ps_capability_list;

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


Jeff Kaufman

unread,
Apr 20, 2015, 3:43:18 PM4/20/15
to ngx-pagesp...@googlegroups.com
On Mon, Apr 20, 2015 at 3:08 PM, Rahul Bansal <rahul....@rtcamp.com> wrote:
>> Yes, you're right, it's much more complex than we'd like.
>
> Just wondering, if there is any plan to simplify this?
>

Yes! In fact, I'm working on this right now.

Vidyut

unread,
Apr 20, 2015, 3:46:21 PM4/20/15
to ngx-pagesp...@googlegroups.com
I have not mentioned any of the headers and stuff normal for proxy_cache, since your defaults will probably be way better than anything I can come up with. I'm still struggling to get my site faster.

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


Rahul Bansal

unread,
Apr 20, 2015, 3:54:36 PM4/20/15
to ngx-pagesp...@googlegroups.com
Wow. I will wait for update then. :-)

I am not sure about complexities but from user perspective, I wish following:
  1. There should not be any need to create a proxy site or alter ports. (will get messy on some servers where nginx runs many sites)
  2. Ideally the behavior can be turned on/off by adding one or few lines. 
Technically, pagespeed without downstream caching works exactly most users in this thread want __except__ it "disables" nginx fastcgi_cache.
So everything aside, if this behavior can be changed, that will work nicely for most static sites. Cache purge can be left to the caching year.
For example, on wordpress we have plugins to purge homepage/edited page(s) from fastcgi_cache as and when we (editor) makes changes to the site content.

Please let me know if I can provide any more feedback/use-cases. 

--
Rahul Bansal | Founder & CEO 
rtCamp Solutions Pvt. Ltd. | https://rtcamp.com/

Rahul Bansal

unread,
Apr 20, 2015, 3:58:24 PM4/20/15
to ngx-pagesp...@googlegroups.com
@Vidyut

Thanks for sharing your config. :-)

At the moment, I am playing with Jeff's config I found in other thread.




--
Rahul Bansal | Founder & CEO 
rtCamp Solutions Pvt. Ltd. | https://rtcamp.com/

Jeff Kaufman

unread,
Apr 21, 2015, 8:12:58 AM4/21/15
to ngx-pagesp...@googlegroups.com
On Mon, Apr 20, 2015 at 3:58 PM, Rahul Bansal <rahul....@rtcamp.com> wrote:
>
> At the moment, I am playing with Jeff's config I found in other thread:
> Link: view-source:http://www.jefftk.com/2014-01-03--nginx.conf

That's an early draft of the downstream caching configuration, and I
don't have a lot of confidence in it. In particular, I suspect the
regexps for user agent matching aren't quite right for PageSpeed 1.9.

Rahul Bansal

unread,
Apr 21, 2015, 8:28:10 AM4/21/15
to ngx-pagesp...@googlegroups.com
On Tue, Apr 21, 2015 at 5:42 PM, 'Jeff Kaufman' via ngx-pagespeed-discuss <ngx-pagesp...@googlegroups.com> wrote:

That's an early draft of the downstream caching configuration, and I
don't have a lot of confidence in it.  In particular, I suspect the
regexps for user agent matching aren't quite right for PageSpeed 1.9.

Proxy part worked for me but proxy cache wasn't hitting/working as expected.
I will try it again soon. But even if it works, complexity will limit its usage.

May I know if changes you are working on will still require proxy site creation?


Jeff Kaufman

unread,
Apr 21, 2015, 8:36:08 AM4/21/15
to ngx-pagesp...@googlegroups.com
The changes I'm currently working on are so people who are already
caching their html in a proxy cache like varnish can set caching up
more easily.

A better setup for people using fastcgi cache is something I'll need
to look into separately; I think it's probably a pretty different fix.
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

Rahul Bansal

unread,
Apr 21, 2015, 8:53:46 AM4/21/15
to ngx-pagesp...@googlegroups.com
Oh! I wish there could have been easier way to solve fastcgi_cache issues, considering fastcgi_cache and proxy_cache have almost similar API (from Nginx perspective).

Thanks for details though. I will try to use proxy workaround on some very high traffic sites. :-)

--
Rahul Bansal | Founder & CEO 
rtCamp Solutions Pvt. Ltd. | https://rtcamp.com/

You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Jeff Kaufman

unread,
Apr 21, 2015, 8:54:21 AM4/21/15
to ngx-pagesp...@googlegroups.com
There may be for fastcgi_cache, I just need to look into it.

Vidyut

unread,
Apr 21, 2015, 9:00:10 AM4/21/15
to ngx-pagesp...@googlegroups.com
Rahul:

Proxy part worked for me but proxy cache wasn't hitting/working as expected.

I had an issue getting proxy-cache to hit as well. Much googling led me to ignore cache and cookie headers for non-admin and "pagespeed ModifyCachingHeaders off;" before it started hitting cache - but I am not so happy with this. The default proxy config gives "MISS" for me.

I thought it was something wrong with how I've set up my server (I am not passing static files through to pagespeed, for example even if it means double caching of them - too slow for some reason). 

Not sure whether it is ngx_pagespeed setting unnecessary controls preventing caching (even for static files) or some issue with proxy_cache not caching if some headers are present. I intend to troubleshoot and narrow down exactly what line fixed it (I just copy-pasted a suggested fix in a hurry and it worked). Varnish handles this way more gracefully.


===============================
Jeff,

Would fastcgi handle anything other than php backend output? Probably won't cache html, etc? Static assets, images, etc will use pagespeed cache, I guess without an option to cache them separate from pagespeed (which sometimes gets slow for me). I have never used fastcgi for anything other than php generated pages - maybe I am wrong. Rahul will probably know this better.

Vidyut

unread,
Apr 21, 2015, 9:23:39 AM4/21/15
to ngx-pagesp...@googlegroups.com
Update: I quickly tested some changes on my server and this is the line needed in the proxy-cache config to let it cache the pagespeed output 

        proxy_ignore_headers "Expires" "Cache-Control";

I was able to take out the cookies from the ignored headers and still get proxy-cache to cache the output. Taking this line out results in proxy-cache not caching the pages.

I have "pagespeed ModifyCachingHeaders off;" in the backend config as well, but still, the pages are not getting cached without the line to ignore headers. Not even static assets. I'm stuck for time right now, but investigating further tonight (India - in another 2-3 hours).

Current line of thought is testing without both the ignored headers and the line "pagespeed ModifyCachingHeaders off;" to see if the problem gets solved or persists. I will update if I can find a more elegant solution or identify what problematic configuration causes it. Suggestions welcome.

Jeff Kaufman

unread,
Apr 21, 2015, 3:42:29 PM4/21/15
to ngx-pagesp...@googlegroups.com
Here's the first draft of the rewritten doc:
http://www.jefftk.com/downstream-caching-draft

Would anyone using pagespeed with varnish (3 or 4) or proxy_cache be
up for testing it?
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

bml13

unread,
Apr 21, 2015, 5:57:56 PM4/21/15
to ngx-pagesp...@googlegroups.com
proxy_ignore_headers "Expires" "Cache-Control";
This has caused the proxy_cache to cache the pages of logged in users and show them to not logged in users,

bml13

unread,
Apr 21, 2015, 6:03:03 PM4/21/15
to ngx-pagesp...@googlegroups.com
I have managed to get pagespeed working with proxy_cache following the guide here:

But I did not have success to make proxy_cache cache the optimized html from pagespeed, it is always MISS. 

On Tuesday, April 21, 2015 at 8:42:29 PM UTC+1, Jeff Kaufman wrote:
Here's the first draft of the rewritten doc:
http://www.jefftk.com/downstream-caching-draft

Would anyone using pagespeed with varnish (3 or 4) or proxy_cache be
up for testing it?

On Tue, Apr 21, 2015 at 9:23 AM, Vidyut <wide....@gmail.com> wrote:
> Update: I quickly tested some changes on my server and this is the line
> needed in the proxy-cache config to let it cache the pagespeed output
>
>         proxy_ignore_headers "Expires" "Cache-Control";
>
> I was able to take out the cookies from the ignored headers and still get
> proxy-cache to cache the output. Taking this line out results in proxy-cache
> not caching the pages.
>
> I have "pagespeed ModifyCachingHeaders off;" in the backend config as well,
> but still, the pages are not getting cached without the line to ignore
> headers. Not even static assets. I'm stuck for time right now, but
> investigating further tonight (India - in another 2-3 hours).
>
> Current line of thought is testing without both the ignored headers and the
> line "pagespeed ModifyCachingHeaders off;" to see if the problem gets solved
> or persists. I will update if I can find a more elegant solution or identify
> what problematic configuration causes it. Suggestions welcome.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

Vidyut

unread,
Apr 22, 2015, 12:54:14 AM4/22/15
to ngx-pagesp...@googlegroups.com
bml13

I am so sorry! I should have stressed that I used it for non-admin users only. I am bypassing the cache for logged in users.

Try


a few times from terminal and then 


Admin pages don't get cached at all, so no question of serving from cache.

If you are using wordpress, try this in your location / block. Or adapt for whatever you are using. (Hint: The line for cookies for almost anything can be copy pasted from abundant fastcgi configs abundant on the internet)

                        if ($http_cookie ~* "comment_author_|wordpress_(?!test_cookie)|wp-postpass_" ) {
                                set $bypass_cache 1;
                        }

Or something like this, but this won't prevent your admin bar appearing on cached pages in some set ups.
        location ~* wp\-.*\.php|wp\-admin {
                                 proxy_pass http://127.0.0.1:8080;
                                 proxy_cache off;
        }


I am using SSL, SNI and a somewhat complicated weave of includes in my badly styled configuration, so it is difficult to paste the whole config at this stage, but once I have tweaked everything to satisfaction, I will tidy everything up and post my configs.

Vidyut

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Vidyut

unread,
Apr 22, 2015, 12:57:33 AM4/22/15
to ngx-pagesp...@googlegroups.com
bml13

you can try 

curl -X GET -I https://aamjanata.com/wp-admin/post-new.php

and

curl -X GET -I https://aamjanata.com

from terminal to get headers.

Vidyut

Lahcen Bensaad

unread,
Apr 22, 2015, 4:41:58 AM4/22/15
to ngx-pagesp...@googlegroups.com
Thanks you Vidyut
I already use the variable $bypass_cache and it is set to 1 if the user is logged in.
But how to use proxy_ignore_headers with it?
I mean to make it ignore the Cache-Control header only if the $bypass_cache = 0 and cause it to cache the html

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

Vidyut

unread,
Apr 22, 2015, 9:12:04 AM4/22/15
to ngx-pagesp...@googlegroups.com
Lahcen, 

It works for me like that. It ignores headers for what it caches. If you aren't caching admin, then well, your admin shouldn't get cached to get served. You should check your settings for bypass again. If your admin pages are getting cached (and thus served to others), then they aren't getting bypassed - you will have to fix those rules, IMO.

V

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


Thanks you Vidyut

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Vidyut

unread,
Apr 22, 2015, 9:13:49 AM4/22/15
to ngx-pagesp...@googlegroups.com
I am not so good at explaining this caching. I'll tidy my working configs and post them up in a day or two.I am able to get hits on non-admin html pages with proxy-cache with my config.

Vidyut

Social Media: Twitter Facebook Google+ Diaspora


Lahcen Bensaad

unread,
Apr 22, 2015, 12:53:32 PM4/22/15
to ngx-pagesp...@googlegroups.com
Thanks Vidyut,
it will be nice to have post your config files, i ma check mine again while waiting yours,
it also need to make it work with https.
Thanks you Vidyut

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

Rahul Bansal

unread,
Jul 13, 2015, 6:53:22 AM7/13/15
to ngx-pagesp...@googlegroups.com
Hi Jeff,

Just checking, if there is any update about this?

Regards.

Jeff Kaufman

unread,
Jul 13, 2015, 7:17:29 AM7/13/15
to ngx-pagesp...@googlegroups.com
The new downstream caching doc will be
http://www.jefftk.com/downstream-caching-draft , but I haven't got it
up on developers.google.com yet.
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ngx-pagespeed-di...@googlegroups.com.

Rahul Bansal

unread,
Jul 13, 2015, 7:31:03 AM7/13/15
to ngx-pagesp...@googlegroups.com
Thanks for link. :-)

Looks like nginx-proxy will remain requirement for long time.
Direct solution with fastcgi-upstream would have been better.

Anyway, thanks again for link and workaround.
I am not less concerned about proxy, and more concerned about performance. 
I will benchmark new config soon.

--
Rahul Bansal | Founder & CEO 
rtCamp Solutions Pvt. Ltd. | https://rtcamp.com/

You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Ovidiu Pacuraru

unread,
Jun 13, 2016, 7:48:05 AM6/13/16
to ngx-pagespeed-discuss
Just curios if anything has changed in the last 3 years regarding how pagespeed works together with fastcgi_cache?
I have ditched fastcgi_cache as I considered pagespeed to be more important but would love to find out how to use both.

Hans van Eijsden

unread,
Jun 13, 2016, 8:44:48 AM6/13/16
to ngx-pagespeed-discuss
Unfortunately not: still the same problems with FastCGI Cache and Redis Cache.
So, I don't use ngx_pagespeed anymore because everything is HTTP/2 here now (all the sites on my 119 servers) HTTP/2 is faster without pagespeed filters like sharing, combining, inlining.
Redis Cache without Pagespeed gives me the fastest results nowadays.

Op maandag 13 juni 2016 13:48:05 UTC+2 schreef Ovidiu Pacuraru:

Otto van der Schaaf

unread,
Jun 13, 2016, 3:53:31 PM6/13/16
to ngx-pagesp...@googlegroups.com
I just tested with FastCGI Cache, and I had no problem getting it to cache the unoptimized html generated by a simple php script.
ngx_pagespeed runs after FastCGI Cache, and optimized the html it receives from FastCGI Cache before forwarding it.

Ovidiu, Hans, is this the flow you would expect to see when enabling fastcgi_cache? Or would you expect ngx_pagespeed to integrate it's downstream caching feature with FastCGI Cache when it is enabled?

Otto

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

Vidyut

unread,
Jun 13, 2016, 5:22:47 PM6/13/16
to ngx-pagesp...@googlegroups.com
Otto, what I think is expected is for fastcgi_cache to function like a downstream cache for ngx_pagespeed as well. I don't see how that can happen unless pagespeed intercepts before the cache, which to my limited knowledge is sounding complicated. Alternative is what we have. the proxy_cache. Fastcgi would probably be faster, but so far not seen anything that makes it sound possible.


Social Media: Twitter Facebook Google+ Diaspora


--
You received this message because you are subscribed to a topic in the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ngx-pagespeed-discuss/vzSq8XQPGmM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ngx-pagespeed-di...@googlegroups.com.

Hans van Eijsden

unread,
Jun 13, 2016, 6:42:23 PM6/13/16
to ngx-pagespeed-discuss
Otto,

I would love to have ngx_pagespeed integrating its downstream caching feature with the ultra-fast Redis. This way it becomes scalable, finally.

I don't use the repo's of rtCamp though, I compiled everything from source.

When I enable ngx_pagespeed, the X-Cache and X-Cache-2 headers both say "MISS". When I disable ngx_pagespeed, the X-Cache header says "HIT" (on the second access) and the X-Cache-2 header says "STORE" (on the first access), like they should.

Without Redis cache, the average system load goes up immediately to 35.00 - 40.00, with Redis cache the system load stays almost idle, 0.01 - 0.03 (I have a LOT of requests a second everywhere, during peak hours around 4000 req/s).
So that's why I migrated everything (almost everything) away from ngx_pagespeed and moved all the sites to HTTP/2 with fast efficient ECC (ECDSA) SSL certificates.

If ngx_pagespeed could do downstream caching into Redis, working together with the downstream caching config of rtCamp I would love to test it and eventually deploy it again. 😃 

There they say: "you can't use ngx_pagespeed with redis cache at nginx level as ngx_pagespeed sets cache control to private and max-age=0".

- Hans


Op maandag 13 juni 2016 21:53:31 UTC+2 schreef Otto van der Schaaf:
I just tested with FastCGI Cache, and I had no problem getting it to cache the unoptimized html generated by a simple php script.
ngx_pagespeed runs after FastCGI Cache, and optimized the html it receives from FastCGI Cache before forwarding it.

Ovidiu, Hans, is this the flow you would expect to see when enabling fastcgi_cache? Or would you expect ngx_pagespeed to integrate it's downstream caching feature with FastCGI Cache when it is enabled?

Otto
On Mon, Jun 13, 2016 at 2:44 PM, Hans van Eijsden <hans...@gmail.com> wrote:
Unfortunately not: still the same problems with FastCGI Cache and Redis Cache.
So, I don't use ngx_pagespeed anymore because everything is HTTP/2 here now (all the sites on my 119 servers) HTTP/2 is faster without pagespeed filters like sharing, combining, inlining.
Redis Cache without Pagespeed gives me the fastest results nowadays.

Op maandag 13 juni 2016 13:48:05 UTC+2 schreef Ovidiu Pacuraru:
Just curios if anything has changed in the last 3 years regarding how pagespeed works together with fastcgi_cache?
I have ditched fastcgi_cache as I considered pagespeed to be more important but would love to find out how to use both.

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages