Re: [ngx-pagespeed-discuss] No expire date for CSS when using Varnish/ngx_pagespeed

[Otto van der Schaaf]

I'm not really a varnish expert - but it seems to me that varnish will miss the files because of this in the varnish configuration ( vcl_recv ):

1. # Block 5a: Bypass the cache for .pagespeed. resource. PageSpeed has its own
2.   # cache for these, and these could bloat up the caching layer.
3.   if (req.url ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
4.     # Skip the cache for .pagespeed. resource.  PageSpeed has its own
5.     # cache for these, and these could bloat up the caching layer.
6.     return (pass);
7.   }

I haven't been able to figure out why your cache-control and expiry headers are dropped.

Are you able to check if they are served properly when your request a .pagespeed. resource directly on the webserver?

Otto

On Wed, Mar 26, 2014 at 4:33 PM, Andreas Lehr <ma...@andreas-lehr.com> wrote:

We are using Pagespeed on Nginx with Downstream Caching to Varnish for a Site. (URL: http://holi-gaudy.com )

Unfortunately it seems as if varnish is somehow deleting the expire settings of pagespeed in the setup.

At least Pagespeed Insights tells us, that there are a lot of css/js/webp files without expiration date set.

In addition the Files are missed by varnish and move always through pagespeed.

Example File:

http://holi-gaudy.com/wp-content/plugins/revslider/rs-plugin/css/A.captions.css,qver=3.8.1.pagespeed.cf.BNdMsjCAMw.css

Response Headers:

1. Accept-Ranges:
   bytes
2. Age:
   0
3. Age:
   0
4. Connection:
   keep-alive
5. Content-Encoding:
   gzip
6. Content-Length:
   813
7. Content-Type:
   text/css
8. Date:
   Wed, 26 Mar 2014 15:12:29 GMT
9. Server:
   nginx
10. Vary:
    Accept-Encoding
11. Via:
    1.1 varnish
12. Via:
    1.1 varnish
13. X-Cache:
    MISS
14. X-Page-Speed:
    1.7.30.4-3847
15. X-Varnish:
    1323764441
16. X-Varnish:
    1287915516

The URL to the page: http://holi-gaudy.com

Pagespeed statistics & config are accessible: http://holi-gaudy.com/ngx_pagespeed_statistics?config

nginx config: http://pastebin.com/tBxDfi13

pagespeed include: http://pastebin.com/7PZ0YrYH

varnish config: http://pastebin.com/yuf01WEJ

We are using the recommended varnish config from the pagespeed documentation.

Would it be better to use the caching feature of nginx instead of varnish?

Main problems are:

- no expire date for a lot of the css/js files

- varnish misses the files (tried with same browser)

Any help would be very appreciated!
Thank you very much!

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Andreas Lehr]

Am Samstag, 29. März 2014 21:40:59 UTC+1 schrieb Otto van der Schaaf:

I'm not really a varnish expert - but it seems to me that varnish will miss the files because of this in the varnish configuration ( vcl_recv ):

1. # Block 5a: Bypass the cache for .pagespeed. resource. PageSpeed has its own
2.   # cache for these, and these could bloat up the caching layer.
3.   if (req.url ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
4.     # Skip the cache for .pagespeed. resource.  PageSpeed has its own
5.     # cache for these, and these could bloat up the caching layer.
6.     return (pass);
7.   }

 

Yes, you are right. This is the recommended configuration when Downstream Caching is active: https://developers.google.com/speed/pagespeed/module/downstream-cachin

As far as i understood the files are stored per User-Agent in Varnish after the initial pagespeed handshake for the next requests in the session.

 

I haven't been able to figure out why your cache-control and expiry headers are dropped.

Are you able to check if they are served properly when your request a .pagespeed. resource directly on the webserver?

If i deactivate varnish and serve the requests only through nginx the headers are set correctly.

 

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Otto van der Schaaf]

As far as i understood the files are stored per User-Agent in Varnish after the initial pagespeed handshake for the next requests in the session.

Varnish will cache the html, but it looks like pagespeed resources are always fetched from the origin by design. The is served from the cache when I look - so that works as far as I can tell.

If i deactivate varnish and serve the requests only through nginx the headers are set correctly.

If the headers are served ok from nginx, it looks like somehow they are changed when passing through Varnish right? There are also some duplicated headers, which I think is caused by PageSpeed fetching resources via Varnish. You might be able to avoid that by using MapOriginDomain (or LoadFromFile) in ngx_pagespeed [1]. Some urls respond with 3 "Accept-Ranges: bytes" headers, for example: 

http://holi-gaudy.com/wpcontent/uploads/2014/02/xholi_logo_retina.png.pagespeed.ic.0qTM40Z3Db.webp

I'm not sure why that is. Is the vcl you linked to your full configuration? 

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Andreas Lehr]

Am Montag, 31. März 2014 21:09:58 UTC+2 schrieb Otto van der Schaaf:

As far as i understood the files are stored per User-Agent in Varnish after the initial pagespeed handshake for the next requests in the session.

Varnish will cache the html, but it looks like pagespeed resources are always fetched from the origin by design. The is served from the cache when I look - so that works as far as I can tell.

yes this works. But initially i thought the Hit Ratio would be higher... Server through nginx + ngx_pagespeed/memcache is still a lot slower than using varnish.

 

If i deactivate varnish and serve the requests only through nginx the headers are set correctly.

If the headers are served ok from nginx, it looks like somehow they are changed when passing through Varnish right? There are also some duplicated headers, which I think is caused by PageSpeed fetching resources via Varnish. You might be able to avoid that by using MapOriginDomain (or LoadFromFile) in ngx_pagespeed [1]. Some urls respond with 3 "Accept-Ranges: bytes" headers, for example: 

http://holi-gaudy.com/wpcontent/uploads/2014/02/xholi_logo_retina.png.pagespeed.ic.0qTM40Z3Db.webp

I'm not sure why that is. Is the vcl you linked to your full configuration? 

Thanks for the hint with the duplicate headers. Could it be that the request goes two times through varnish because of:

        pagespeed MapOriginDomain localhost holi-gaudy.com;

i changed that now to:

        pagespeed MapOriginDomain http://127.0.0.1:8080 http://holi-gaudy.com;

The configs at pastebin are the full configuration. 

Is this the right usage of MapOriginDomain when varnish is running at :80 and using 127.0.0.1:8080 as backend?

config links again:

nginx: http://pastebin.com/tBxDfi13

ngx_pagespeed: http://pastebin.com/7PZ0YrYH

varnish: http://pastebin.com/yuf01WEJ

 

 

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsubscri...@googlegroups.com.

Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Otto van der Schaaf]

yes this works. But initially i thought the Hit Ratio would be higher... Server through nginx + ngx_pagespeed/memcache is still a lot slower than using varnish.

If you have enough cache space available in Varnish, I think you could try to cache the optimized .pagespeed. urls as well.

As far as I can tell, you would only need to comment block 5a in the vcl configuration.

Doing so would cause ngx_pagespeed to see less traffic, which will influence /ngx_pagespeed_statistics though.

Apart from the skewed statistics and needing more cache space, I can’t see any trouble with caching .pagespeed. resources in varnish - but perhaps

someone from Google can add to this.

Is this the right usage of MapOriginDomain when varnish is running at :80 and using 127.0.0.1:8080 as backend?

As you are serving the website from both holi-gaudy.com and www.holi-gaudy.com, I think that next to this:

pagespeed MapOriginDomain http://127.0.0.1:8080 http://holi-gaudy.com;

You also want 

pagespeed MapOriginDomain http://127.0.0.1:8080 http://www.holi-gaudy.com;

We lost almost all header duplication - only Accept-Ranges went down from three to two headers. I’m still not sure 

why an extra one gets added.

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Otto van der Schaaf]

Also - I’m curious if commenting block 5a fixes the dropped Expiry/Cache-Control headers.

Other urls on your website seem to emit those headers properly, and this is the only vcl section

I can find that might influence these headers.

Otto

[Andreas Lehr]

I commented block 5a now.

Could you please inspect from your side?
From my side (behind proxy in corp web) i can see that some of the css files are now directly served by varnish withough going through pagespeed...

Caching hit ratio already increased although.

thank you very much for your help!

But i would like to know what other impacts this configuration change could have..

[Otto van der Schaaf]

It looks like pagespeed resources are now cached and served from Varnish.

The Accept-Ranges response header is down to 1 as well for cache hits.

Still no Expires/Cache-Control headers though - I'll try if I can reproduce that later today with Varnish. 

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Anupama Dutta]

Sorry about jumping in very late on this thread.

If I remember correctly, the reasons for not recommending caching .pagespeed resources in the varnish layer are:

1) these are already cached in your pagespeed server cache

2) there could be an explosion in the cache storage required for these pagespeed resources because you will end up with multiple rewritten URLs per User-Agent class. For instance, for the same image resource, you might have one webp version, and another non-webp version. And, I think when some resources in say a CSS file expires, you would see a partially rewritten version of the CSS resource and later a fully rewritten version of the resource. Since, all of these variations are already going into the pagespeed server cache, it seems unnecessary for you to store them in varnish all over again. Of course, the cache itself will eventually drop the less used variation, but deciding on the optimal cache size for this maybe tricky. If the latency between fetches from varnish and your pagespeed server is high enough, you may still want to cache it. Is that the primary reason you want to do this? 

Regarding Expires header, I am wondering if https://github.com/pagespeed/ngx_pagespeed/pull/555/files could have anything to do with this. This was intended to be applied for HTML only, but I don't know if it happened to somehow get rid of Expires (and other headers) for other resources as well. Could you take a look?

--
Anupama

[Otto van der Schaaf]

Sounds like it would be a good idea to monitor the cache hit rate even more closely when enabling caching of .pagespeed. resources, right?

Regarding Expires header, I am wondering if https://github.com/pagespeed/ngx_pagespeed/pull/555/files could have anything to do with this. This was intended to be applied for HTML only, but I don't know if it happened to somehow get rid of Expires (and other headers) for other resources as well. Could you take a look?

I was working with the assumption that ngx_pagespeed is serving the headers OK -- earlier in this thread:

"If i deactivate varnish and serve the requests only through nginx the headers are set correctly."

I also don't see ModifyCachingHeaders anywhere in the mentioned configurations.

Nevertheless, I'll double check to see if this is working correctly for resources.

[Anupama Dutta]

On Tue, Apr 1, 2014 at 8:48 AM, Otto van der Schaaf <osc...@we-amp.com> wrote:

Sounds like it would be a good idea to monitor the cache hit rate even more closely when enabling caching of .pagespeed. resources, right?

Regarding Expires header, I am wondering if https://github.com/pagespeed/ngx_pagespeed/pull/555/files could have anything to do with this. This was intended to be applied for HTML only, but I don't know if it happened to somehow get rid of Expires (and other headers) for other resources as well. Could you take a look?

I was working with the assumption that ngx_pagespeed is serving the headers OK -- earlier in this thread:

"If i deactivate varnish and serve the requests only through nginx the headers are set correctly."

I also don't see ModifyCachingHeaders anywhere in the mentioned configurations.

Nevertheless, I'll double check to see if this is working correctly for resources.

ModifyCachingHeaders should be on by default right? However, if the test you mention above (deactivating varnish) worked, then I too have no clue as to what might be going on.

One extreme way to test this might be to replace the varnish config with the default config (without the downstream-caching-related blocks), and see if this problem still persists. Otto, do you think that might be useful for Andreas to try?

A random query: What does the following block in your pagespeed config do?

1. # Ensure requests for pagespeed optimized resources go to the pagespeed handler
2. # and no extraneous headers get set.
3. location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
4.   add_header "" "";
5. }

--
Anupama

[Andreas Lehr]

Regarding Expires header, I am wondering if https://github.com/pagespeed/ngx_pagespeed/pull/555/files could have anything to do with this. This was intended to be applied for HTML only, but I don't know if it happened to somehow get rid of Expires (and other headers) for other resources as well. Could you take a look?

Should i test with an earlier version of ngx_pagespeed? or what exactly to look at this link? (unfortunately my developer skills are limited :) )

 

I was working with the assumption that ngx_pagespeed is serving the headers OK -- earlier in this thread:

"If i deactivate varnish and serve the requests only through nginx the headers are set correctly."

I also don't see ModifyCachingHeaders anywhere in the mentioned configurations.

Nevertheless, I'll double check to see if this is working correctly for resources.

ModifyCachingHeaders should be on by default right? However, if the test you mention above (deactivating varnish) worked, then I too have no clue as to what might be going on.

One extreme way to test this might be to replace the varnish config with the default config (without the downstream-caching-related blocks), and see if this problem still persists. Otto, do you think that might be useful for Andreas to try?

As far as i understood ModifyCachingHeaders is enabled per default. As mentioned before if i disable varnish and use plain nginx the expire headers are set correctly.

Should i test with a default varnish config? Or deactivate the 3 lines in the pagespeed config considering DownstreamCaching? Both?

 

A random query: What does the following block in your pagespeed config do?

1. # Ensure requests for pagespeed optimized resources go to the pagespeed handler
2. # and no extraneous headers get set.
3. location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
4.   add_header "" "";
5. }

These are the recommendations for nginx configuration from over here: https://github.com/pagespeed/ngx_pagespeed/blob/master/README.md 

 

--
Anupama

--
Anupama

[Anupama Dutta]

On Tue, Apr 1, 2014 at 10:53 AM, Andreas Lehr <ma...@andreas-lehr.com> wrote:

Regarding Expires header, I am wondering if https://github.com/pagespeed/ngx_pagespeed/pull/555/files could have anything to do with this. This was intended to be applied for HTML only, but I don't know if it happened to somehow get rid of Expires (and other headers) for other resources as well. Could you take a look?

Should i test with an earlier version of ngx_pagespeed? or what exactly to look at this link? (unfortunately my developer skills are limited :) )

Sorry, I should have been more clear :) I wanted Otto to take a look at this code change, to see if it was doing the right thing. You can ignore it for now. 

 

I was working with the assumption that ngx_pagespeed is serving the headers OK -- earlier in this thread:

"If i deactivate varnish and serve the requests only through nginx the headers are set correctly."

I also don't see ModifyCachingHeaders anywhere in the mentioned configurations.

Nevertheless, I'll double check to see if this is working correctly for resources.

ModifyCachingHeaders should be on by default right? However, if the test you mention above (deactivating varnish) worked, then I too have no clue as to what might be going on.

One extreme way to test this might be to replace the varnish config with the default config (without the downstream-caching-related blocks), and see if this problem still persists. Otto, do you think that might be useful for Andreas to try?

As far as i understood ModifyCachingHeaders is enabled per default. As mentioned before if i disable varnish and use plain nginx the expire headers are set correctly.

Should i test with a default varnish config? Or deactivate the 3 lines in the pagespeed config considering DownstreamCaching? Both?

If it is possible to test in all modes, i.e. (a) with default varnish config and DownstreamCaching enabled, (b) with current config and with DownstreamCaching disabled in pagespeed end and (c) both a and b, it will help debug the situation better. I think (a) would be the most useful one 

--
Anupama

[Andreas Lehr]

As far as i understood ModifyCachingHeaders is enabled per default. As mentioned before if i disable varnish and use plain nginx the expire headers are set correctly.

Should i test with a default varnish config? Or deactivate the 3 lines in the pagespeed config considering DownstreamCaching? Both?

If it is possible to test in all modes, i.e. (a) with default varnish config and DownstreamCaching enabled, (b) with current config and with DownstreamCaching disabled in pagespeed end and (c) both a and b, it will help debug the situation better. I think (a) would be the most useful one 

I just deployed a very basic varnish configuration, see here: http://pastebin.com/EFymXXL8

Please have a look.

 

...

[Otto van der Schaaf]

Anapuma: seems your suspicion was right -- adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources:

pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;

I'll have a look at how to fix this.

Andreas - are your sure that direct requests on the origin return cacheable responses?

It seems that with the basic varnish configuration, the Expiry/Cache-Control headers still are omitted.

Otto

[Andreas Lehr]

Am Dienstag, 1. April 2014 19:11:59 UTC+2 schrieb Otto van der Schaaf:

Anapuma: seems your suspicion was right -- adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources:

pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;

I'll have a look at how to fix this.

Andreas - are your sure that direct requests on the origin return cacheable responses?

It seems that with the basic varnish configuration, the Expiry/Cache-Control headers still are omitted.

in the nginx only setup i commented the 3 lines regarding DownstreamCaching in nginx, cause i thought it won't work this way in any case.

I am sorry to confuse you on this issue. 

This was the only configuration change i did in the nginx only setup.

Should i reactivate it for testing?

[Anupama Dutta]

On Tue, Apr 1, 2014 at 2:05 PM, Andreas Lehr <ma...@andreas-lehr.com> wrote:

Am Dienstag, 1. April 2014 19:11:59 UTC+2 schrieb Otto van der Schaaf:

Anapuma: seems your suspicion was right -- adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources:

pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;

I'll have a look at how to fix this.

Andreas - are your sure that direct requests on the origin return cacheable responses?

It seems that with the basic varnish configuration, the Expiry/Cache-Control headers still are omitted.

in the nginx only setup i commented the 3 lines regarding DownstreamCaching in nginx, cause i thought it won't work this way in any case.

I am sorry to confuse you on this issue. 

This was the only configuration change i did in the nginx only setup.

Should i reactivate it for testing?

Andreas, are you saying that if you retain the new varnish config (with downstream-caching specific stuff included) and use the pagespeed conf that does not use the 3 DownstreamCaching lines, you are able to get Expires and Cache-Control headers properly for your css files. If yes, this seems to tally well with what Otto just said "adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources: pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;".

--
Anupama

[Andreas Lehr]

in the nginx only setup i commented the 3 lines regarding DownstreamCaching in nginx, cause i thought it won't work this way in any case.

I am sorry to confuse you on this issue. 

This was the only configuration change i did in the nginx only setup.

Should i reactivate it for testing?

Andreas, are you saying that if you retain the new varnish config (with downstream-caching specific stuff included) and use the pagespeed conf that does not use the 3 DownstreamCaching lines, you are able to get Expires and Cache-Control headers properly for your css files. If yes, this seems to tally well with what Otto just said "adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources: pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;".

no, by "Nginx Only" Setup i mean a setup where Nginx is listening on the external IP and varnish is completely deactivated.

So, should i test with "nginx only" setup and enable DownstreamCache in order to check if the headers aren't existing there?

 

Otto

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsubscribe@googlegroups.

...

[Anupama Dutta]

On Tue, Apr 1, 2014 at 3:52 PM, Andreas Lehr <ma...@andreas-lehr.com> wrote:

in the nginx only setup i commented the 3 lines regarding DownstreamCaching in nginx, cause i thought it won't work this way in any case.

I am sorry to confuse you on this issue. 

This was the only configuration change i did in the nginx only setup.

Should i reactivate it for testing?

Andreas, are you saying that if you retain the new varnish config (with downstream-caching specific stuff included) and use the pagespeed conf that does not use the 3 DownstreamCaching lines, you are able to get Expires and Cache-Control headers properly for your css files. If yes, this seems to tally well with what Otto just said "adding this line to nginx.conf makes the cache headers disappear for .pagespeed. resources: pagespeed DownstreamCachePurgeLocationPrefix http://localhost:80/;".

no, by "Nginx Only" Setup i mean a setup where Nginx is listening on the external IP and varnish is completely deactivated.

So, should i test with "nginx only" setup and enable DownstreamCache in order to check if the headers aren't existing there?

Since Otto is already able to repro this locally in some form, I suggest that you wait for his updates (hopefully a fix that you can maybe patch in and try).  Thanks for helping us identify this issue!

--
Anupama

[Otto van der Schaaf]

I'm think I have a fix ready at https://github.com/pagespeed/ngx_pagespeed/compare/morlovich-trunk-tracking-upto-r3895...oschaaf-downstream-caching-headers

It yet needs to be reviewed - but do you want to try it out?

If so, I can look into a patch that will apply cleanly to your version.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Andreas Lehr]

Am Mittwoch, 2. April 2014 00:14:27 UTC+2 schrieb Otto van der Schaaf:

I'm think I have a fix ready at https://github.com/pagespeed/ngx_pagespeed/compare/morlovich-trunk-tracking-upto-r3895...oschaaf-downstream-caching-headers

It yet needs to be reviewed - but do you want to try it out?

If so, I can look into a patch that will apply cleanly to your version.

Sure, i'd like to test it. My current version is 1.7.30.4.

Thanks Otto!

 

...

[Otto van der Schaaf]

I looked into to backporting the fix to 1.7.30.4, but doing so is not as straightforward as I hoped as it depends on other changes - sorry! So testing it requires building PSOL r3895 from source[1] and building https://github.com/pagespeed/ngx_pagespeed/tree/oschaaf-downstream-caching-headers against that. This would switch you to the "bleeding edge", which is something you might or might not want.

[1] https://github.com/pagespeed/ngx_pagespeed/wiki/Building-PSOL-From-Source

Otto

--

[Andreas Lehr]

i tried building PSOL from trunk (cause i didn't find the syntax for gconfig usage with a specific svn revision)

The build stopped with the following error:

make[1]: Entering directory `/root/src/net/instaweb/automatic'

/usr/bin/g++ -I/root/src -I/root/src/out/Release/obj/gen/protoc_out/instaweb -I/root/src/third_party/chromium/src -I/root/src/third_party/gflags/gen/arch/linux/x64/include -I/root/src/third_party/google-sparsehash/src -I/root/src/third_party/google-sparsehash/gen/arch/linux/x64/include -I/root/src/third_party/protobuf/src -I/root/src/third_party/re2/src -DSERF_HTTPS_FETCHING=0  -c static_rewriter_main.cc -o /root/src/net/instaweb/automatic/static_rewriter_main.o

In file included from /root/src/pagespeed/kernel/base/statistics.h:29:0,

                 from /root/src/pagespeed/kernel/base/null_statistics.h:23,

                 from /root/src/net/instaweb/util/public/null_statistics.h:20,

                 from /root/src/net/instaweb/rewriter/public/rewrite_driver_factory.h:27,

                 from /root/src/net/instaweb/automatic/public/static_rewriter.h:22,

                 from static_rewriter_main.cc:21:

/root/src/pagespeed/kernel/base/string_util.h:32:33: fatal error: base/stringprintf.h: No such file or directory

 #  include "base/stringprintf.h"

                                 ^

compilation terminated.

make[1]: *** [/root/src/net/instaweb/automatic/static_rewriter_main.o] Error 1

make[1]: Leaving directory `/root/src/net/instaweb/automatic'

make: *** [all] Error 2

What i did till then:

gclient config http://modpagespeed.googlecode.com/svn/trunk/src

gclient sync --force --jobs=1

make AR.host="$PWD/build/wrappers/ar.sh"        AR.target="$PWD/build/wrappers/ar.sh"        BUILDTYPE=Release        mod_pagespeed_test pagespeed_automatic_test

 cd net/instaweb/automatic/
make CXXFLAGS="-DSERF_HTTPS_FETCHING=0"        BUILDTYPE=Release        AR.host="$PWD/../../../build/wrappers/ar.sh"        AR.target="$PWD/../../../build/wrappers/ar.sh"        all

...

[Otto van der Schaaf]

PSOL trunk won't work because of API changes. To get the revision that would work you'd need this:

gclient sync --force --jobs=1 --revision=3895

Otto

--

[Anupama Dutta]

In case it helps, here is the documentation that tells you how to build ngx_pagespeed synced to PSOL trunk: https://github.com/pagespeed/ngx_pagespeed/wiki/Building-trunk-tracking-ngx_pagespeed-with-PSOL-synced-to-trunk

--
Anupama

[Andreas Lehr]

Am Mittwoch, 2. April 2014 15:37:46 UTC+2 schrieb Anupama Dutta:

In case it helps, here is the documentation that tells you how to build ngx_pagespeed synced to PSOL trunk: https://github.com/pagespeed/ngx_pagespeed/wiki/Building-trunk-tracking-ngx_pagespeed-with-PSOL-synced-to-trunk

ok, thanks!
I managed to build PSOL with the specific revision.

Furthermore i downloaded the ngx_pagespeed version linked by Otto.

I am trying to build on Debian Jessie with dpkg-buildpackage (http://www.howtoforge.com/using-ngx_pagespeed-with-nginx-on-debian-jessie-testing). This worked like a charm before.

If i follow the instructions there i extracted the special build of psol directly into ngx_pagespeed dir of Otto.

But the directory structure from the former extracted 1.7.30.4 Release looks completely different from what the build produced:

Former Version:

ll ngx_pagespeed-release-1.7.30.4-beta

total 90532

-rw-r--r-- 1 root   root 92652779 Mar 13 14:00 1.7.30.4.tar.gz

-rw-r--r-- 1 root   root     8133 Mar 13 15:37 config

-rw-r--r-- 1 root   root     2881 Mar 13 15:37 cpp_feature

-rw-r--r-- 1 root   root    11342 Mar 13 15:37 LICENSE

drwxr-x--- 4 182960 5000     4096 Mar 12 20:55 psol

-rw-r--r-- 1 root   root     4833 Mar 13 15:37 README.md

drwxr-xr-x 2 root   root     4096 Mar 13 15:37 scripts

drwxr-xr-x 2 root   root     4096 Mar 13 15:37 src

drwxr-xr-x 2 root   root     4096 Mar 13 15:37 test

root@web101:/usr/src/pagespeed/nginx-1.4.7/debian/modules# ll ngx_pagespeed-release-1.7.30.4-beta/psol/

total 12

drwxr-x--- 9 182960 5000 4096 Mar 12 20:00 include

-rw-r----- 1 182960 5000   62 Mar 12 20:55 include_history.txt

drwxr-x--- 4 182960 5000 4096 Mar 12 20:55 lib

psol build:

ll

total 78

drwxr-xr-x  4 root root  1024 Apr  2 15:58 base

drwxr-xr-x 10 root root  1024 Apr  2 15:58 build

-rw-r--r--  1 root root  9353 Apr  2 15:54 DEPS

drwxr-xr-x  7 root root  1024 Apr  2 15:55 googleurl

drwxr-xr-x  4 root root  1024 Apr  2 15:54 googleurl_noconv

drwxr-xr-x  8 root root  1024 Apr  2 15:54 install

-rw-r--r--  1 root root 57743 Apr  2 15:58 Makefile

drwxr-xr-x  3 root root  1024 Apr  2 15:54 net

drwxr-xr-x  3 root root  1024 Apr  2 15:58 out

drwxr-xr-x  3 root root  1024 Apr  2 15:58 pagespeed

drwxr-xr-x  8 root root  1024 Apr  2 15:58 testing

drwxr-xr-x 29 root root  1024 Apr  2 15:57 third_party

drwxr-xr-x  4 root root  1024 Apr  2 15:58 tools

What did i mess up here?

Thanks for your patience :)

[Otto van der Schaaf]

The directory structure of your psol build looks ok to me. 

The build is different from the 1.7.30.4 release, because it isn’t stripped down to contain only things

needed to build ngx_pagespeed.

You should be able to proceed with “Download and build nginx:” at

https://github.com/pagespeed/ngx_pagespeed/wiki/Building-PSOL-From-Source

Otto

[Andreas Lehr]

Ok, i tried 2 to 3 times yesterday and just now. I had to rebuild psol but the sync won't complete:

gclient sync --force --jobs=1 --revision=3895

....
...

______ running '/usr/bin/python src/build/gyp_chromium -Dchromium_revision=161115' in '/root/mod_pagespeed'

Updating projects from gyp files...

Traceback (most recent call last):

  File "../build/version.py", line 25, in <module>

    execfile(os.path.join(chrome_src, 'chrome', 'tools', 'build', 'version.py'))

IOError: [Errno 2] No such file or directory: '../third_party/chromium/src/chrome/tools/build/version.py'

gyp: Call to 'python ../build/version.py -f ../net/instaweb/public/VERSION -t "@MAJOR@.@MINOR@.@BUILD@.@PATCH@"' returned exit status 1. while loading dependencies of /root/mod_pagespeed/src/build/all.gyp while trying to load /root/mod_pagespeed/src/build/all.gyp

Error: Command /usr/bin/python src/build/gyp_chromium -Dchromium_revision=161115 returned non-zero exit status 1 in /root/mod_pagespeed

[Otto van der Schaaf]

I'm not sure I understand -- why did you have to rebuild PSOL?

Did you execute the same steps that previously led to a successful build?

Otto

[Andreas Lehr]

I messed something up and had to do it again.

A manual build of nginx seems to work, but not with the preferred way via "dpkg-buildpackage -b"

this ends up with:

/usr/src/pagespeed/nginx-1.4.7/debian/modules/nginx-upstream-fair/ngx_http_upstream_fair_module.c

cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -Wno-unused-local-typedefs -Wno-error  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /root/mod_pagespeed/src -I /root/mod_pagespeed/src/third_party/chromium/src -I /root/mod_pagespeed/src/third_party/google-sparsehash/src -I /root/mod_pagespeed/src/third_party/google-sparsehash/gen/arch/linux/x64/include -I /root/mod_pagespeed/src/third_party/protobuf/src -I /root/mod_pagespeed/src/third_party/re2/src -I /root/mod_pagespeed/src/out/Debug/obj/gen -I /root/mod_pagespeed/src/out/Debug/obj/gen/protoc_out/instaweb -I /root/mod_pagespeed/src/third_party/apr/src/include -I /root/mod_pagespeed/src/third_party/aprutil/src/include -I /root/mod_pagespeed/src/third_party/apr/gen/arch/linux/x64/include -I /root/mod_pagespeed/src/third_party/aprutil/gen/arch/linux/x64/include -I /usr/include/libxml2 -I objs -I src/http -I src/http/modules -I src/mail \

-o objs/addon/ngx_http_substitutions_filter_module/ngx_http_subs_filter_module.o \

/usr/src/pagespeed/nginx-1.4.7/debian/modules/ngx_http_substitutions_filter_module/ngx_http_subs_filter_module.c

cc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -Wno-unused-local-typedefs -Wno-error  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /root/mod_pagespeed/src -I /root/mod_pagespeed/src/third_party/chromium/src -I /root/mod_pagespeed/src/third_party/google-sparsehash/src -I /root/mod_pagespeed/src/third_party/google-sparsehash/gen/arch/linux/x64/include -I /root/mod_pagespeed/src/third_party/protobuf/src -I /root/mod_pagespeed/src/third_party/re2/src -I /root/mod_pagespeed/src/out/Debug/obj/gen -I /root/mod_pagespeed/src/out/Debug/obj/gen/protoc_out/instaweb -I /root/mod_pagespeed/src/third_party/apr/src/include -I /root/mod_pagespeed/src/third_party/aprutil/src/include -I /root/mod_pagespeed/src/third_party/apr/gen/arch/linux/x64/include -I /root/mod_pagespeed/src/third_party/aprutil/gen/arch/linux/x64/include -I /usr/include/libxml2 -I objs -I src/http -I src/http/modules -I src/mail \

-o objs/addon/src/log_message_handler.o \

/root/ngx_pagespeed/src/log_message_handler.cc

/root/ngx_pagespeed/src/log_message_handler.cc:31:41: fatal error: net/instaweb/public/version.h: No such file or directory

 #include "net/instaweb/public/version.h"

is it searching in the wrong directory?

 find . | grep -i version.h

./mod_pagespeed/src/out/Release/.deps/out/Release/obj.target/build/mod_pagespeed_version_header.stamp.d

./mod_pagespeed/src/out/Release/.deps/out/Release/obj/gen/net/instaweb/public/version.h.d

./mod_pagespeed/src/out/Release/obj.target/build/mod_pagespeed_version_header.stamp

./mod_pagespeed/src/out/Release/obj/gen/net/instaweb/public/version.h

./mod_pagespeed/src/build/mod_pagespeed_version_header.target.mk

./mod_pagespeed/src/net/instaweb/public/version.h.in

./mod_pagespeed/src/third_party/aprutil/src/include/apu_version.h

./mod_pagespeed/src/third_party/apr/src/include/apr_version.h

./mod_pagespeed/src/third_party/chromium/src/base/win/windows_version.h

./mod_pagespeed/src/third_party/chromium/src/base/i18n/case_conversion.h

./mod_pagespeed/src/third_party/chromium/src/base/version.h

./mod_pagespeed/src/third_party/chromium/src/base/chromeos/chromeos_version.h

./mod_pagespeed/src/third_party/icu/source/common/unicode/uversion.h

./mod_pagespeed/src/third_party/libjpeg_turbo/src/jversion.h

./mod_pagespeed/src/third_party/libjpeg_turbo/yasm/genversion.host.mk

./mod_pagespeed/src/third_party/mod_spdy/src/mod_spdy/common/version.h.in

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Andreas Lehr]

the corresponding configure line for the nginx-full build looks like this:

cd /usr/src/pagespeed/nginx-1.4.7/debian/build-full && MOD_PAGESPEED_DIR="/root/mod_pagespeed/src" ./configure  \

   --with-cc-opt="-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2" --with-ld-opt="-Wl,-z,relro" --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module  \

   --with-http_addition_module \

   --with-http_dav_module \

   --with-http_geoip_module \

   --with-http_gzip_static_module \

   --with-http_image_filter_module \

   --with-http_spdy_module \

   --with-http_sub_module \

   --with-http_xslt_module \

   --with-mail \

   --with-mail_ssl_module \

   --add-module=/usr/src/pagespeed/nginx-1.4.7/debian/modules/nginx-auth-pam \

   --add-module=/usr/src/pagespeed/nginx-1.4.7/debian/modules/nginx-dav-ext-module \

   --add-module=/usr/src/pagespeed/nginx-1.4.7/debian/modules/nginx-echo \

   --add-module=/usr/src/pagespeed/nginx-1.4.7/debian/modules/nginx-upstream-fair \

   --add-module=/usr/src/pagespeed/nginx-1.4.7/debian/modules/ngx_http_substitutions_filter_module \

   --add-module=/root/ngx_pagespeed \

            >config.status.full

i used the same methodology for building previous nginx packages with downloaded psol/pagespeed version. Worked without any problems.

[Otto van der Schaaf]

is it searching in the wrong directory?

I think removing  --with-debug will make the build succeed. 

If you want a debug build - you'll also need to build the PSOL debug libraries.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Andreas Lehr]

narf, thanks again Otto.

The build succeded without setting "--with-debug".

Install was fine, and i have expire headers now!
Great work, thank you very much!!

i will try and test it with this live site!

Thanks for your patience!

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsubscri...@googlegroups.com.

Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Otto van der Schaaf]

The fix was merged on the ngx_pagespeed trunk-tracking branch via https://github.com/pagespeed/ngx_pagespeed/pull/653

Otto

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.