PageSpeed Module often does not combinde JS, sometimes sets expiration header of images to 5 min

[Anton Telle]

I just installed NGINX (1.8.0) with PageSpeed Module (1.9.32.10). We use it as a proxy for our backend server.
I configured it with CoreFilters and additionally defer_javascript,lazyload_images,insert_image_dimensions.

Somehow I do not get consistent results. Often the JavaScript files are not combined. Or the expiration date header for images is suddenly set back to only 5 minutes.
I gather from the documentation (https://developers.google.com/speed/pagespeed/module/filter-cache-extend#operation) that occasionally PageSpeed will change the TTL to 5 minutes to check if new content is being served. But in my case this happens quite often.

1. Can there be other reasons why the JS is often not combined and image expiration is set to only 5 minutes?
2. Can I configure how often PageSpeed rechecks the cached items, i.e. sets its expiration to only 5 minutes?

[Joshua Marantz]

Generally PS sets TTL to 5 minutes when the content-hash of what is being served does not match the content-hash in the request.  That can happen in a multi-server setup with inconsistent content on each server.  It can also happen -- usually for images -- if the PageSpeed settings are not the same between the servers, but I think that's less likely for JavaScript because there are not so many settings that affect out JS is minified.

Is this inconsistent-content theory a possibility in your environment?

-Josh

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.
Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
For more options, visit https://groups.google.com/d/optout.

[Anton Telle]

Hm, no, this does not match our infrastructure.
It is just one CentOS 6 proxy server running NGINX with PageSpeed. There is only one backend server to which requests are passed. (The backend server is a Windows Server 2008 R2 with IIS 7.5. The application itself is build upon Orchard CMS.)

-Anton

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

Could you paste your pagespeed directives?

>>> email to ngx-pagespeed-di...@googlegroups.com.

>>> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

[Anton Telle]

Of course. Here are the directives:

========
    pagespeed on;
    pagespeed RewriteLevel CoreFilters;
    pagespeed EnableFilters defer_javascript,lazyload_images,insert_image_dimensions;

    # using tmpfs
    pagespeed FileCachePath /var/ngx_pagespeed_cache;

    location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
      add_header "" "";
    }
    location ~ "^/pagespeed_static/" { }
    location ~ "^/ngx_pagespeed_beacon$" { }
========

>>> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>>> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Otto van der Schaaf]

Does IIS send out the responses for the js, css and images with expiry headers set?

>>> email to ngx-pagespeed-di...@googlegroups.com.

>>> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ngx-pagespeed-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ngx-pagespeed-di...@googlegroups.com.

[Jeff Kaufman]

Those pagespeed directives are very reasonable, and don't seem like
they should be causing behavior like this.

Could you pick a js file that's getting this 5-min cache expiration
problem and post:

- the original headers for that file
curl -D- -o/dev/null http://example.com/foo.js?PageSpeed=off

- any error messages in the log for it
grep foo.js /path/to/error_log

>> >>> email to ngx-pagespeed-di...@googlegroups.com.

>> >>> Visit this group at
>> >>> http://groups.google.com/group/ngx-pagespeed-discuss.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ngx-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ngx-pagespeed-di...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

[Anton Telle]

If I go directly to the IIS I get this response for the request http://yogastudios.discoverize.com/Modules/Teamaton.Search/scripts/searchform.min.js:
HTTP/1.1 200 OK
Content-Type: application/x-javascript
Content-Encoding: gzip
Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "803a892a5d1bd11:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Thu, 12 Nov 2015 11:16:30 GMT
Content-Length: 2284

IIS does not set any expiration headers.

==================

For this request via the proxy NGINX server (PageSpeed off):
http://yogastudios.discoverize.com/Modules/Teamaton.Search/scripts/searchform.min.js?PageSpeed=off

gives me the following headers:
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 12 Nov 2015 10:44:13 GMT
Content-Type: application/x-javascript
Content-Length: 6093
Connection: keep-alive
Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "5cb92a5d1bd11:0"
Vary: Accept-Encoding
X-Powered-By: ASP.NET

no errors were logged for this request

>> >>> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>> >>> Visit this group at
>> >>> http://groups.google.com/group/ngx-pagespeed-discuss.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ngx-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Anton Telle]

The strange thing is, I get different responses for the same requests to http://yogastudios.discoverize.com/yogasuche (over NGINX with PageSpeed), and I cannot explain them. These screenshots are taken off fiddler.
First request:

response headers for GET /Modules/Teamaton.Search/scripts/searchform.min.js HTTP/1.1

HTTP/1.1 200 OK
Server: nginx

Date: Thu, 12 Nov 2015 11:32:43 GMT
Content-Type: application/x-javascript
Content-Length: 2284
Connection: keep-alive

Content-Encoding: gzip
Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "803a892a5d1bd11:0"
Vary: Accept-Encoding

X-Powered-By: ASP.NET

Next request:

Here the scripts are combined, as expected. But why does the browser send a request if it has the combined scripts cached?
Response for GET /Modules/Teamaton.Search,_scripts,_jquery.pinme.min.js+Teamaton.Discoverize,_scripts,_jquery.unobtrusive-ajax.min.js+Teamaton.Discoverize,_scripts,_jquery.tn.loading.min.js+Teamaton.Discoverize,_scripts,_jquery.columnizelists.min.js+Teamaton.Discoverize,_scripts,_jquery.tn.tooltip.min.js+Teamaton.Search,_scripts,_searchform.min.js
HTTP/1.1 304 Not Modified
Server: nginx
Date: Thu, 12 Nov 2015 11:37:30 GMT
Connection: keep-alive
X-Page-Speed: 1.9.32.10-7423

5 minutes later the requests look like this:

Strangely, compared to the first request, here the expiration header is set, but only to 5 minutes.
Response for GET /Modules/Teamaton.Search/scripts/searchform.min.js HTTP/1.1:

HTTP/1.1 200 OK
Content-Type: application/x-javascript

Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Server: nginx
Vary: Accept-Encoding

Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "5cb92a5d1bd11:0"

X-Powered-By: ASP.NET
Cache-Control: max-age=300
Date: Thu, 12 Nov 2015 11:45:16 GMT
Expires: Thu, 12 Nov 2015 11:50:16 GMT
Content-Encoding: gzip

Two questions arise:
1. Why are there so many requests which have a 304 response?
2. Why does PageSpeed "unbundle" the JavaScript files every 5 minutes?

[Jeff Kaufman]

On Thu, Nov 12, 2015 at 7:07 AM, Anton Telle <anton...@gmail.com> wrote:

1. Why are there so many requests which have a 304 response?

When your browser receives a cached file with an "etag" or "last-modified" header that's something it can use on later requests to check if a resource is still valid.  So if you download searchform.min.js and see the etag 5cb92a5d1bd11, then the next time you need that resource if it's already expired you can send a conditional request to the server.  The server will check if that etag is still current, and if so send back a 304 response with 0 bytes of body.  So all those 304s look normal to me, and represent significant bandwidth savings for your users.

 

2. Why does PageSpeed "unbundle" the JavaScript files every 5 minutes?

Are you getting other requests to your site within that 5 minute period?  PageSpeed only refreshes its input resources when it's getting traffic that needs them, so if you're not sending much traffic to this page then on the first request after 5min the input files will be stale in the cache and it will serve them uncombined instead of delaying the request while it combines them.

[Anton Telle]

Am Donnerstag, 12. November 2015 15:20:01 UTC+1 schrieb Jeff Kaufman:

On Thu, Nov 12, 2015 at 7:07 AM, Anton Telle <anton...@gmail.com> wrote:

1. Why are there so many requests which have a 304 response?

When your browser receives a cached file with an "etag" or "last-modified" header that's something it can use on later requests to check if a resource is still valid.  So if you download searchform.min.js and see the etag 5cb92a5d1bd11, then the next time you need that resource if it's already expired you can send a conditional request to the server.  The server will check if that etag is still current, and if so send back a 304 response with 0 bytes of body.  So all those 304s look normal to me, and represent significant bandwidth savings for your users.

Does that mean, the client will only send a conditional request to the server, if the resource has expired in the client?
Then I don't understand why these requests are send every time, and get a 304 response.
 

 

2. Why does PageSpeed "unbundle" the JavaScript files every 5 minutes?

Are you getting other requests to your site within that 5 minute period?  PageSpeed only refreshes its input resources when it's getting traffic that needs them, so if you're not sending much traffic to this page then on the first request after 5min the input files will be stale in the cache and it will serve them uncombined instead of delaying the request while it combines them.

It does not matter whether there are other requests within the 5 minute period. Every 5 minutes the first request will respond with the JavaScript files unbundled. If I send the same request inbetween these requests the JavaScript files are bundled (always requested, always getting a 304).
Strangely the requests which get the JavaScript files unbundled differ in their response - always alternating these are the responses to http://yogastudios.discoverize.com/Modules/Teamaton.Search/scripts/searchform.min.js:

HTTP/1.1 200 OK
Content-Type: application/x-javascript

Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Server: nginx
Vary: Accept-Encoding
Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "5cb92a5d1bd11:0"
X-Powered-By: ASP.NET
Cache-Control: max-age=300

Date: Thu, 12 Nov 2015 16:25:55 GMT
Expires: Thu, 12 Nov 2015 16:30:55 GMT
Content-Encoding: gzip

HTTP/1.1 200 OK
Server: nginx

Date: Thu, 12 Nov 2015 16:32:04 GMT

Content-Type: application/x-javascript
Content-Length: 2284
Connection: keep-alive
Content-Encoding: gzip
Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "803a892a5d1bd11:0"
Vary: Accept-Encoding
X-Powered-By: ASP.NET

They have different ETags (alternating always between these two ETags). One has Expires header set, the other not. Both requests seem untouched by PageSpeed even though they are routed through the nginx proxy server with PageSpeed on.
What could be the cause of this behavior?

[Jeff Kaufman]

If you turn off pagespeed do you still get the alternating etags?

> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

[Anton Telle]

If I turn off pagespeed I do not get alternating etags. This would be one response:

HTTP/1.1 304 Not Modified
Server: nginx

Date: Fri, 13 Nov 2015 15:06:56 GMT
Connection: keep-alive

Last-Modified: Tue, 10 Nov 2015 02:11:53 GMT
Accept-Ranges: bytes
ETag: "5cb92a5d1bd11:0"
X-Powered-By: ASP.NET

Strange. I will test it with a different site.
If you see any other possible solutions to this problems - I am very open to experiment.

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Anton Telle]

I tested it with a different site: caravanmarkt.info
Same configuration, same setup - yet the described peculiarities do not appear. That is good :).
I will see whether I can fix it for the other site.