Cache static html version?

[Brian G]

Is there a specific config needed to get pagespeed to generate an optimized html version of a site and cache it, serve it to varnish for caching?

On Wordpress test sites, the only way we get this working is to enable the W3 plugin and check the "static html cache" option. We'd like to make this work in pagespeed without a WP plugin if possible.

Same issue on a php website (non WP), we're not getting a static html cache version.

Thanks,

B

[Joshua Marantz]

Yes.  Please check out https://developers.google.com/speed/pagespeed/module/downstream-caching

-Josh

--
You received this message because you are subscribed to the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/71d5f5ea-0b46-48d6-a6f1-1818324c164c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian G]

Thanks for the response, we already have the Varnish specific stuff from that page setup on our system. We're getting caching on just about everything but like I said, it just refuses to cache a static HTML compiled version of the site.

W3 is able to do it with one check, I think there has to be a setting or variable in pagespeed we're missing to be able to do the same thing without W3.

Thanks,

B

--
You received this message because you are subscribed to a topic in the Google Groups "mod-pagespeed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mod-pagespeed-discuss/v0WiX_XzByI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mod-pagespeed-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAGKR%2BEDT0wQ248R4CJCejmvNdEm%2BjC6HFMaytxLcHHyE1G6voQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Brian

[Jeff Kaufman]

If you have pagespeed set up for downstream caching with varnish
following that doc then this is something that is supported and should
work.

What do you have DownstreamCachePurgeMethod and
DownstreamCachePurgeLocationPrefix set to?

If you fetch your site and look at the headers do you see X-Cache headers?

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAJKQafNjjNwm27JKUyJYMjuT7RWMw-qS%2BSyZ2DXBry4eQe4pvA%40mail.gmail.com.

[Brian G]

From the apache config, directory of webroot section:

ModPagespeedDownstreamCachePurgeLocationPrefix http://localhost:80

ModPagespeedDownstreamCachePurgeMethod PURGE

ModPagespeedDownstreamCacheRewrittenPercentageThreshold 95

Headers from the non WP test site index page:

HTTP/1.1 200 OK

Server: Apache/2.2.15 (CentOS)

X-Powered-By: PHP/5.5.21

X-Mod-Pagespeed: 1.9.32.3-4448

Vary: Accept-Encoding,User-Agent

Content-Type: text/html; charset=UTF-8

Content-Length: 32779

Accept-Ranges: bytes

Date: Fri, 30 Jan 2015 16:19:15 GMT

X-Varnish: 923755076

Age: 0

Via: 1.1 varnish

Connection: keep-alive

X-Cache: MISS

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUs6A%3DMYqECX2OBL9tKYVy31JkE46qbesuzcT-2iFWU9UQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Brian

[Jeff Kaufman]

If you request it multiple times do you start getting "X-Cache: HIT"?

If you bypass Varnish and request directly from the Apache server,
what Cache-Control header do you see?

If you look at Varnish's access log do you see any PURGEs coming from
mod_pagespeed?

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAJKQafOnG6kjv22Hm5TyZV%2BnaRBhRpaZy327psyoDmSq3OQ7Yg%40mail.gmail.com.

[Brian G]

No, X:cache is showing MISS on everything I check headers for manually.

Access log does not show any PURGE entries.

I can't access apache direct, it's setup to run on 127.0.0.1:8080 w/ Varnish on 80 in same server.

Wondering if I need to move the cache statements from Apache conf to the virtualhost block directory statements?

default.vcl:

# This is a basic VCL configuration file for varnish.  See the vcl(7)

# man page for details on VCL syntax and semantics.

# 

# Default backend definition.  Set this to point to your content

# server.

# 

# Server 1.2.3.4 is where the websites are hosted.

backend default {

  .host = "127.0.0.1";

  .port = "8080";

  }

# 

# Below is a commented-out copy of the default VCL logic.  If you

# redefine any of these subroutines, the built-in logic will be

# appended to your code.

# sub vcl_recv {

#     if (req.restarts == 0) {

# if (req.http.x-forwarded-for) {

#    set req.http.X-Forwarded-For =

# req.http.X-Forwarded-For + ", " + client.ip;

# } else {

#    set req.http.X-Forwarded-For = client.ip;

# }

#     }

#     if (req.request != "GET" &&

#       req.request != "HEAD" &&

#       req.request != "PUT" &&

#       req.request != "POST" &&

#       req.request != "TRACE" &&

#       req.request != "OPTIONS" &&

#       req.request != "DELETE") {

#         /* Non-RFC2616 or CONNECT which is weird. */

#         return (pipe);

#     }

#     if (req.request != "GET" && req.request != "HEAD") {

#         /* We only deal with GET and HEAD by default */

#         return (pass);

#     }

#     if (req.http.Authorization || req.http.Cookie) {

#         /* Not cacheable by default */

#         return (pass);

#     }

#     return (lookup);

# }

# 

# sub vcl_pipe {

#     # Note that only the first request to the backend will have

#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to

#     # have it set for all requests, make sure to have:

#     # set bereq.http.connection = "close";

#     # here.  It is not set by default as it might break some broken web

#     # applications, like IIS with NTLM authentication.

#     return (pipe);

# }

# 

# sub vcl_pass {

#     return (pass);

# }

# 

# sub vcl_hash {

#     hash_data(req.url);

#     if (req.http.host) {

#         hash_data(req.http.host);

#     } else {

#         hash_data(server.ip);

#     }

#     return (hash);

# }

# 

# sub vcl_hit {

#     return (deliver);

# }

# 

# sub vcl_miss {

#     return (fetch);

# }

# 

# sub vcl_fetch {

#     if (beresp.ttl <= 0s ||

#         beresp.http.Set-Cookie ||

#         beresp.http.Vary == "*") {

# /*

# * Mark as "Hit-For-Pass" for the next 2 minutes

# */

# set beresp.ttl = 120 s;

# return (hit_for_pass);

#     }

#     return (deliver);

# }

# 

# sub vcl_deliver {

#     return (deliver);

# }

# 

# sub vcl_error {

#     set obj.http.Content-Type = "text/html; charset=utf-8";

#     set obj.http.Retry-After = "5";

#     synthetic {"

# <?xml version="1.0" encoding="utf-8"?>

# <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

#  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

# <html>

#   <head>

#     <title>"} + obj.status + " " + obj.response + {"</title>

#   </head>

#   <body>

#     <h1>Error "} + obj.status + " " + obj.response + {"</h1>

#     <p>"} + obj.response + {"</p>

#     <h3>Guru Meditation:</h3>

#     <p>XID: "} + req.xid + {"</p>

#     <hr>

#     <p>Varnish cache server</p>

#   </body>

# </html>

# "};

#     return (deliver);

# }

# 

# sub vcl_init {

# return (ok);

# }

# 

# sub vcl_fini {

# return (ok);

# }

# Block 2: Define a key based on the User-Agent which can be used for hashing.

# Also set the PS-CapabilityList header for PageSpeed server to respect.

sub generate_user_agent_based_key {

    # Define placeholder PS-CapabilityList header values for large and small

    # screens with no UA dependent optimizations. Note that these placeholder

    # values should not contain any of ll, ii, dj, jw or ws, since these

    # codes will end up representing optimizations to be supported for the

    # request.

    set req.http.default_ps_capability_list_for_large_screens = "LargeScreen.SkipUADependentOptimizations:";

    set req.http.default_ps_capability_list_for_small_screens = "TinyScreen.SkipUADependentOptimizations:";

    # As a fallback, the PS-CapabilityList header that is sent to the upstream

    # PageSpeed server should be for a large screen device with no browser

    # specific optimizations.

    set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_large_screens;

    # Cache-fragment 1: Desktop User-Agents that support lazyload_images (ll),

    # inline_images (ii) and defer_javascript (dj).

    # Note: Wget is added for testing purposes only.

    if (req.http.User-Agent ~ "(?i)Chrome/|Firefox/|Gecko|Trident/6\.|Safari|Wget") {

      set req.http.PS-CapabilityList = "ll,ii,dj:";

    }

    # Cache-fragment 2: Desktop User-Agents that support lazyload_images (ll),

    # inline_images (ii), defer_javascript (dj), webp (jw) and lossless_webp

    # (ws).

    if (req.http.User-Agent ~

        "(?i)Chrome/[2][3-9]+\.|Chrome/[[3-9][0-9]+\.|Chrome/[0-9]{3,}\.") {

      set req.http.PS-CapabilityList = "ll,ii,dj,jw,ws:";

    }

    # Cache-fragment 3: This fragment contains (a) Desktop User-Agents that

    # match fragments 1 or 2 but should not because they represent older

    # versions of certain browsers or bots and (b) Tablet User-Agents that

    # on all browsers and use image compression qualities applicable to large

    # screens. Note that even Tablets that are capable of supporting inline or

    # WebP images, e.g. Android 4.1.2, will not get these advanced

    # optimizations.

    if (req.http.User-Agent ~ "(?i)Firefox/[1-2]\.|bot|Yahoo!|Ruby|RPT-HTTPClient|(Google \(\+https\:\/\/developers\.google\.com\/\+\/web\/snippet\/\))|Android|iPad|TouchPad|Silk-Accelerated|Kindle Fire") {

      set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_large_screens;

    }

    # Cache-fragment 4: Mobiles and small screen Tablets will use image compression

    # qualities applicable to small screens, but all other optimizations will be

    # those that work on all browsers.

    if (req.http.User-Agent ~ "(?i)Mozilla.*Android.*Mobile*|iPhone|BlackBerry|Opera Mobi|Opera Mini|SymbianOS|UP.Browser|J-PHONE|Profile/MIDP|portalmmm|DoCoMo|Obigo|Galaxy Nexus|GT-I9300|GT-N7100|HTC One|Nexus [4|7|S]|Xoom|XT907") {

      set req.http.PS-CapabilityList = req.http.default_ps_capability_list_for_small_screens;

    }

    # Remove placeholder header values.

    # Varnish 4.x uses "unset" instead of "remove".

    remove req.http.default_ps_capability_list_for_large_screens;

    remove req.http.default_ps_capability_list_for_small_screens;

}

sub vcl_hash {

  # Block 3: Use the PS-CapabilityList value for computing the hash.

  hash_data(req.http.PS-CapabilityList);

}

# Block 3a: Define ACL for purge requests.

acl purge {

  # Purge requests are only allowed from localhost.

  "localhost";

  "127.0.0.1";

  "204.77.2.162";

}

# Blocks 3b and 3c are only for Varnish 3.x, so if you're using 4.x don't

# include them.  In Varnish 4.x purging when you get a purge request is what

# happens by default.

# Block 3b: Issue purge when there is a cache hit for the purge request.

sub vcl_hit {

  if (req.request == "PURGE") {

    purge;

    error 200 "Purged.";

  }

}

# Block 3c: Issue a no-op purge when there is a cache miss for the purge

# request.

sub vcl_miss {

  if (req.request == "PURGE") {

     purge;

     error 200 "Purged.";

  }

}

# Block 4: In vcl_recv, on receiving a request, call the method responsible for

# generating the User-Agent based key for hashing into the cache.

sub vcl_recv {

  call generate_user_agent_based_key;

  # Block 3d: Verify the ACL for an incoming purge request and handle it.

  if (req.request == "PURGE") {

    if (!client.ip ~ purge) {

      error 405 "Not allowed.";

    }

    return (lookup);

  }

  # Blocks which decide whether cache should be bypassed or not go here.

}

# Block 5a: Bypass the cache for .pagespeed. resource. PageSpeed has its own

  # cache for these, and these could bloat up the caching layer.

  if (req.url ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {

    # Skip the cache for .pagespeed. resource.  PageSpeed has its own

    # cache for these, and these could bloat up the caching layer.

    return (pass);

  }

  

  # Block 5b: Only cache responses to clients that support gzip.  Most clients

  # do, and the cache holds much more if it stores gzipped responses.

  if (req.http.Accept-Encoding !~ "gzip") {

    return (pass);

  }

  

  # Block 6: Mark HTML uncacheable by caches beyond our control.

# Varnish 4.x calls this "vcl_backend_response" instead of "vcl_fetch".

sub vcl_fetch {

   if (beresp.http.Content-Type ~ "text/html") {

     # Hide the upstream cache control header.

     # Varnish 4.x uses "unset" instead of "remove".

     remove beresp.http.Cache-Control;

     # Add no-cache Cache-Control header for html.

     set beresp.http.Cache-Control = "no-cache, max-age=0";

   }

   return (deliver);

}

# Block 7: Add a header for identifying cache hits/misses.

sub vcl_deliver {

  if (obj.hits > 0) {

    set resp.http.X-Cache = "HIT";

  } else {

    set resp.http.X-Cache = "MISS";

  }

}

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUvN93bK859XR4fJb6M55yfn4bRk5ZxSh6FTarpn4ak0AA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Brian

[Brian G]

Tried moving the apache statements into the virtual host directory blocks, 200 PURGE is showing up in varnish access now but X:Cache MISS still showing on headers for everything.

--

Thanks,

Brian

[Jeff Kaufman]

What happens if you send a request to 127.0.0.1:8080 (your apache
server) -- what's the CacheControl header being set to?

> https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAJKQafMXA_ygJ2g0ME9yGTFtUfu5zE3d%3DwWn9O49vaMtY2U8Kg%40mail.gmail.com.

[Brian G]

I have bigger issues to work on, the pagespeed specific stuff in default.vcl suddenly stopped working and it refused to start. Debug kept throwing errors about the entries one after another so I finally had to remove them all to get it going again.

My boss wants Varnish to handle all the caching instead of pagespeed or apache which I don't know is even really possible. I'm honestly not seeing any load time difference between varnish and pagespeed w/ memcached.

My google searches indicate the X-Cache: MISS issue might be that no cache headers are being set either by apache or pagespeed for the original issue, unfortunately I have the bigger problem now to resolve.

I plan to re-configure apache to use the same external IP, different port in place of loop back so it can be reached from outside.

If anyone has a varnish 3 default.vcl w/ pagespeed entries that work and would share it I'd greatly appreciate it.

Thanks,

B

To view this discussion on the web visit https://groups.google.com/d/msgid/mod-pagespeed-discuss/CAMJ6YUsnKTjiBK79o0-AyPBZxmHk%2BWB2JE6z4WY%2BT%2B%3D2HRO1CA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Brian