inconsistency rewriting images

[Bryan Berry]

dear mod_ps,

I am seeing some strange inconsistencies w/ the rewriting of images w/
the current_stable_x86_64.rpm package. I am using apache 2.2.15 on
CentOS 6.3.

mod_ps only seems to consistently rewrite my images when I have the
following configuration for my domains where my domain is example.org
and my site is data.example.org

ModPagespeedDomain *.example.org
ModPagespeedDomain mynode1.int.example.org # fqdn for my server
ModPagespeedDomain localhost

the following does not work or at least not consistently

ModPagespeedDomain *.example.org

and this only rewrites some of the images

ModPagespeedDomain *.example.org
ModPagespeedDomain mynode1.int.example.org # fqdn for my server

the "localhost" seems to be necessary. Is this true?

It also seems that the rewriting of images varies minute by minute

Here is what I see in my logs for the resource
http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0
(hideous URL I know)

It is alternately shrunk and then later mod_ps fetches the non-shrunk
version from cache.

[Wed Jan 16 16:55:53 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25054]
http://data.example.org/:252: Shrinking image
`http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0'
(129242 bytes) to
`http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/xphoto,qauthKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66,aversion=1.0.pagespeed.ic.p4oNbIsFdo.jpg'
(118809 bytes)
[Wed Jan 16 16:56:11 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25056]
Fetching resource
http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/xphoto,qauthKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66,aversion=1.0.pagespeed.ic.p4oNbIsFdo.jpg...
[Wed Jan 16 16:56:11 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25056]
Fetch succeeded for
http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/xphoto,qauthKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66,aversion=1.0.pagespeed.ic.p4oNbIsFdo.jpg,
status=200

[Wed Jan 16 17:04:37 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25054]
Fetch complete:
http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0
[Wed Jan 16 17:04:37 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25054]
HTTPCache key=http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0:
remembering not-cacheable status for 299 seconds.
[Wed Jan 16 17:04:37 2013] [info] [mod_pagespeed 1.1.23.2-2258 @25054]
HTTPCache key=http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0:
remembering not-cacheable status for 299 seconds.


I hope to very soon expose my site to the Internet so you guys can
interact w/ it directly. Going through the security dance now. tks
again to the mod_ps folks for their help.

[Shawn Ligocki]

I'm not sure wildcards are allowed in ModPagespeedDomain, we don't document that they are: https://developers.google.com/speed/docs/mod_pagespeed/domains

However, you should not need "ModPagespeedDomain localhost" unless you have URLs on your page that have localhost (which seems like a bad idea).

This line looks notable:

HTTPCache key=http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0:
remembering not-cacheable status for 299 seconds.

What headers do you have on the original resource? Are they changing? mod_pagespeed seems to think they are uncacheable here, while cacheable before.

In general, inconsistencies are somewhat expected (especially for a low-traffic site, say where you are the only one visiting it). We rewrite each resource (image, css, js) as we load it, so it will usually take a few loads to have all the resources rewritten. Then we will need to recheck them them every time the original resource expired (I think by default that is every 10 minutes). If you have higher traffic, those resources should be kept warm in cache, so all your resources should be rewritten all the time, but if you are the only user and you only visit it sporadically ~ once per 5minutes or so, you can expect many resources to have fallen out of cache.

[Joshua Marantz]

Wildcards are OK in 'ModPagespeedDomain'.  They are useful when experimenting with mod_pagespeed in a forward proxy.

-Josh

[Shawn Ligocki]

Oh, of course, we should make sure to document that.

[Bryan Berry]

Shawn, it is in the documents, including the comments in pagespeed.conf.

however, I have been seeing such erratic behavior that I had to ask if
wildcards are indeed used

[Bryan Berry]

thanks Shawn

On Wed, Jan 16, 2013 at 5:33 PM, Shawn Ligocki <slig...@google.com> wrote:
> HTTPCache
> key=http://data.example.org/ref/5dc91595-6ce4-4d16-a608-81128220e2f1/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0:
> remembering not-cacheable status for 299 seconds.
>
> What headers do you have on the original resource? Are they changing?
> mod_pagespeed seems to think they are uncacheable here, while cacheable
> before.

I don't believe they are changing. When I look at the HTTP headers
accessing the resource directly there aren't any "Cache-Control"
headers. Are there any other headers that could cause mod_ps to decide
the resource is uncacheable?

> In general, inconsistencies are somewhat expected (especially for a
> low-traffic site, say where you are the only one visiting it).

this explains a lot. I am testing mod_ps w/ the QA version of our
site, which is rarely accessed.

thanks again team mod_ps!

[Shawn Ligocki]

On Thu, Jan 17, 2013 at 3:33 AM, Bryan Berry <bryan...@gmail.com> wrote:

I don't believe they are changing. When I look at the HTTP headers
accessing the resource directly there aren't any "Cache-Control"
headers. Are there any other headers that could cause mod_ps to decide
the resource is uncacheable?

Yeah, a lot of things can influence our cacheability algorithm. For example, requests with query params are default not cacheable, so that could explain why we aren't caching this (although it doesn't explain why we did cache it originally).

If you can set explicit Cache-Control headers for these resources that might help. Otherwise, not really sure what's going on.

>> In general, inconsistencies are somewhat expected (especially for a

>> low-traffic site, say where you are the only one visiting it).

> this explains a lot. I am testing mod_ps w/ the QA version of our

> site, which is rarely accessed.

This may really be the issue, just try refreshing the page several times withing a minute or so, by the last time it ought to have most resources rewritten.

[Shawn Ligocki]

On Thu, Jan 17, 2013 at 10:56 AM, Shawn Ligocki <slig...@google.com> wrote:

On Thu, Jan 17, 2013 at 3:33 AM, Bryan Berry <bryan...@gmail.com> wrote:

I don't believe they are changing. When I look at the HTTP headers
accessing the resource directly there aren't any "Cache-Control"
headers. Are there any other headers that could cause mod_ps to decide
the resource is uncacheable?

Yeah, a lot of things can influence our cacheability algorithm. For example, requests with query params are default not cacheable, so that could explain why we aren't caching this (although it doesn't explain why we did cache it originally).

Actually, I misspoke, query params shouldn't be the problem, maybe something else in your headers though, missing Date header, Authorization header in the request headers, etc. Could you post the response headers?

[Bryan Berry]

On Thu, Jan 17, 2013 at 5:19 PM, Shawn Ligocki <slig...@google.com> wrote:
> On Thu, Jan 17, 2013 at 10:56 AM, Shawn Ligocki <slig...@google.com> wrote:
> Actually, I misspoke, query params shouldn't be the problem, maybe something
> else in your headers though, missing Date header, Authorization header in
> the request headers, etc. Could you post the response headers?
>>

Hey Shawn, here are the request and response headers for one of the photos

Request URL:http://data.example.org/ref/a42c4da3-9195-443c-a5b8-e1bb0081dc51/photo?authKey=d30aebf0-ab2a-11e1-afa6-0800200c9a66&version=1.0
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:__utma=1.1906986528.1335292732.1335292732.1335292732.1;
__utmz=1.1335292732.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
JSESSIONID=CC917FAA98700F917F041306C102CABF; GUEST_LANGUAGE_ID=en_US;
COOKIE_SUPPORT=true; SRVPRCATALB=1|UPkSX; hellobar_1_shown=1;
SRVPRAPIFE=1|UPkSX; SRVPRLRAYFE=3|UPkSa;
__utma=135863972.1358978218.1358500441.1358500441.1358500441.1;
__utmb=135863972.2.10.1358500441; __utmc=135863972;
__utmz=135863972.1358500441.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
hellobar_current=1
Host:data.example.org
Referer:http://data.example.org/
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML,
like Gecko) Chrome/23.0.1271.64 Safari/537.11
Query String Parametersview URL encoded
authKey:d30aebf0-ab2a-11e1-afa6-0800200c9a66
version:1.0
Response Headersview source
Content-Disposition:inline; filename=photo
Content-Language:unspecified
Content-Type:image/jpg;charset=UTF-8
Date:Fri, 18 Jan 2013 09:14:17 GMT
esb-msg-id:2e111b3b-8084-484e-a235-d6161b237507
faodata-request-id:03dd86d2-747a-4f00-935b-745bb4403563
Server:Apache-Coyote/1.1
Server:Apache-Coyote/1.1
Set-Cookie:SRVPRAPIFE=1|UPkSb; path=/
Transfer-Encoding:chunked
Transfer-Encoding:chunked

[Shawn Ligocki]

That should be cacheable.