proxy_pass images not optimized

[Matthew Jacobi]

Hi,

Trying to get pagespeed to optimize some images that we proxy_pass but it's not doing anything with them.

I knocked up a test page with a normal image and a proxy pass image.

https://www.pitchup.com/pagespeed_proxy_pass/

We have a 3rd party image hosting company for our uploaded images which we then proxy pass so it seems to come from our domain and also we can cache it our varnish (and not use up our 3rd party hosting bandwith).

So we have this in our location block:

    location ~ ^/cloudinary/(\d+)/(.*)$ {
        expires max;
        # ideally we should be serving the file from varnish cache, but in case it gets to here, go request it from cloudinary
        rewrite ^/cloudinary/(\d+)/(.*)$ /pitchup-com/$2 break;
        proxy_pass http://a$1.res.cloudinary.com;
        proxy_pass_request_headers off;
    }

so images for our domain pitchup.com/cloudianry/1/file.jpg get proxied to a1.res.cloudianry.com/pitchup-com/file.jpg

I've tried adding cloudinary's domain to the authorized list.

Some of the configuration settings we have

http block:

    # Prevent mobile network providers from modifying your site
    add_header "Cache-Control" "no-transform";


    # https://github.com/pagespeed/ngx_pagespeed
    pagespeed on;
    pagespeed FileCachePath /mnt/pagespeed;  # Use tmpfs for best results.
    pagespeed FileCacheSizeKb            12582912;
    pagespeed FileCacheCleanIntervalMs   3600000;
    pagespeed FileCacheInodeLimit        12374945;  # 0.75 * max

    pagespeed DownstreamCacheRebeaconingKey "<KEYREMOVED>";

    pagespeed UseNativeFetcher on;
    resolver 8.8.8.8;

    pagespeed RespectXForwardedProto on;

    pagespeed Disallow "*/page_view/";

    pagespeed EnableFilters convert_to_webp_lossless;
    pagespeed EnableFilters convert_jpeg_to_progressive;
    pagespeed EnableFilters convert_jpeg_to_webp;
    pagespeed EnableFilters insert_image_dimensions;
    pagespeed EnableFilters remove_comments;
    pagespeed EnableFilters collapse_whitespace;
    pagespeed EnableFilters elide_attributes;
    pagespeed EnableFilters rewrite_css;
    pagespeed EnableFilters fallback_rewrite_css_urls;
    pagespeed EnableFilters lazyload_images;
    pagespeed EnableFilters insert_dns_prefetch;
    pagespeed EnableFilters prioritize_critical_css;
    pagespeed EnableFilters extend_cache;
    pagespeed EnableFilters rewrite_javascript;
    pagespeed EnableFilters rewrite_images;
    pagespeed EnableFilters dedup_inlined_images;
    pagespeed EnableFilters defer_javascript;
    pagespeed EnableFilters inline_preview_images;
    pagespeed EnableFilters resize_mobile_images;
    pagespeed EnableFilters rewrite_domains;

    pagespeed InlineResourcesWithoutExplicitAuthorization Script,Stylesheet;

    include pagespeed_libraries.conf;
    pagespeed EnableFilters canonicalize_javascript_libraries;

    # yimg support
    pagespeed UrlValuedAttribute yimg src image;

    pagespeed FetchHttps enable,allow_self_signed;

and then from the server block:

    pagespeed Domain www.pitchup.com;
    pagespeed Domain m.pitchup.com;
    pagespeed Domain *.pitchup.co.uk;
    pagespeed Domain https://www.pitchup.com;
    pagespeed Domain https://m.pitchup.com;
    pagespeed Domain https://*.pitchup.co.uk;


    pagespeed ShardDomain media.pitchup.co.uk media1.pitchup.co.uk,media2.pitchup.co.uk;


    pagespeed MapOriginDomain "http://127.0.0.1" "https://m.pitchup.com" m.pitchup.com;
    pagespeed MapOriginDomain "http://127.0.0.1" "https://www.pitchup.com" www.pitchup.com;
    pagespeed MapOriginDomain "http://127.0.0.1" "https://media.pitchup.co.uk" media.pitchup.co.uk;


    pagespeed LoadFromFile "https://www.pitchup.com/static/" "/mnt/ssd/django/pitchup_prod/pitchup/static/";
    pagespeed LoadFromFile "https://m.pitchup.com/static/" "/mnt/ssd/django/pitchup_prod/pitchup/static/";
    pagespeed LoadFromFile "https://media.pitchup.co.uk/" "/mnt/ssd/django/pitchup_prod/pitchup/static/";


    pagespeed LoadFromFileRule Disallow "/mnt/ssd/django/pitchup_prod/pitchup/static/cloudinary";
    pagespeed LoadFromFileRuleMatch Disallow "/mnt/ssd/django/pitchup_prod/pitchup/static/v.*";

note that we've just tried moving the static media to media.pitchup.co.uk to be the cookieless domain, and thats why the config is there. It has the very same config. Tried to get it to shard but it didn't.

I did have cloudianry images proxied before but that was if we left the images pointing to cloudianry's domain, but then used MapProxyDomain. Problem with that was that if the image wasn't optimized by pagespeed, it would point to cloudianry and use up bandwith there.

I saw an issue on github that's now closed about Accept-Encoding, tried setting that to blank but still no luck.

Any help would be much appreciated.

Thanks,

Matthew Jacobi

Pitchup.com

[Jeff Kaufman]

Could you try removing the '''add_header "Cache-Control"
"no-transform";''' directive? PageSpeed respects "Cache-Contro:
no-transform".

> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ngx-pagespeed-di...@googlegroups.com.
> Visit this group at http://groups.google.com/group/ngx-pagespeed-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Matthew Jacobi]

Took it off (since it's not needed anyway, the instructions for the upstream caching has no-tranform in it)

curl headers going direct and bypassing the varnish cache:

< HTTP/1.1 200 OK

< Server: nginx/1.6.0

< Content-Type: text/html; charset=utf-8

< Transfer-Encoding: chunked

< Connection: keep-alive

< Vary: Accept-Encoding

< Vary: Accept-Language, User-Agent

< Content-Language: en-gb

< X-Upstream: 127.0.0.1:8001

< Date: Thu, 15 May 2014 14:49:00 GMT

< X-Page-Speed: 1.8.31.2-3973

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

Could you paste the headers for an image downloaded via the location
block "location ~ ^/cloudinary/(\d+)/(.*)$ {}" ?

>> > email to ngx-pagespeed-di...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

[Matthew Jacobi]

Managed to get it sharding by changing the static links to be relative to the first domain, and then having it rewrite the domain to be the media domain and then shard that. Can confirm it sees the image and rewrites it (probably because the filter rewrite urls is on).

 

So I end up with something like this:

<img src="https://media1.staging.pitchup.co.uk/cloudinary/2/image/upload/t_primary_v2/v1368799724/corriefodly-holiday-park/corriefodly-holiday-park-two-bedroom-holiday-home-glen-clova-caravan-home-interior.jpg" pagespeed_url_hash="1957969592" data-pagespeed-onload="pagespeed.CriticalImages.checkImageForCriticality(this);" onload="var elem=this;if (this==window) elem=document.body;elem.setAttribute('data-pagespeed-loaded', 1)" data-pagespeed-loaded="1">

[Matthew Jacobi]

Sure:

< HTTP/1.1 200 OK

< Server: nginx/1.6.0

< Date: Thu, 15 May 2014 16:15:24 GMT

< Content-Type: image/jpeg

< Content-Length: 12902

< Connection: keep-alive

< Accept-Ranges: bytes

< Access-Control-Allow-Headers: X-Requested-With

< Access-Control-Allow-Origin: *

< ETag: "4db612879b319ac94ee825a50522c678"

< Last-Modified: Thu, 14 Nov 2013 19:57:46 GMT

< Cache-Control: max-age=315360000

< Expires: Thu, 31 Dec 2037 23:55:55 GMT

>> > email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

Those image headers look right; cache-control is permissive enough to
allow optimization.

Did you say you have it working now, or does something still need fixing?

>> >> > email to ngx-pagespeed-di...@googlegroups.com.

>> >> > Visit this group at
>> >> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ngx-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ngx-pagespeed-di...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.

[Matthew Jacobi]

Only managed to get it sharding (by changing my source to not do the media domain, and have pagespeed rewrite the domain). But that's a minor issue.

Still trying to get the proxy_pass images to pagespeed.

What's the best way to configure a media cookie less domain server block in nginx? Should it have pagespeed on/off?  

>> >> > email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>> >> > Visit this group at
>> >> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ngx-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-discuss+unsub...@googlegroups.com.

[Jeff Kaufman]

The normal way to do a cookieless domain with without ngx_pagespeed would be:

http {
server {
server_name www.example.com;
root /var/www;
}
server {
server_name examplestaticcontent.com;
root /var/www;
}
}

And you'd have to make the html reference resources on examplestaticcontent.com.

Add in ngx_pagespeed. Now we can leave everything as www.example.com
in the html and have it rewrite resources onto
examplestaticcontent.com for us with MapRewriteDomain. We also should
set CacheFragment, now that it exists, to improve cache performance.
(You wouldn't have to do this if it was www.example.co.uk and
static.example.co.uk because those would both use example.co.uk as the
default CacheFragment.) This looks like:

http {
... set up pagespeed ...
server {
server_name www.example.com;
root /var/www;

pagespeed MapRewriteDomain examplestaticcontent.com www.example.com;
pagespeed CacheFragment example;
}
server {
server_name examplestaticcontent.com;
root /var/www;
pagespeed CacheFragment example;

>> >> >> > email to ngx-pagespeed-di...@googlegroups.com.

>> >> >> > Visit this group at
>> >> >> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> >> >> > For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "ngx-pagespeed-discuss" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an

>> >> > email to ngx-pagespeed-di...@googlegroups.com.

>> >> > Visit this group at
>> >> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "ngx-pagespeed-discuss" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to ngx-pagespeed-di...@googlegroups.com.

>> > Visit this group at
>> > http://groups.google.com/group/ngx-pagespeed-discuss.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "ngx-pagespeed-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ngx-pagespeed-di...@googlegroups.com.