Re: [page-speed-discuss] vary: user-agent and CDN caching issues

[Bryan McQuade]

Many HTTP caches decide that Vary:User-Agent is effectively Vary:*
since the number of user-agents in the wild is so large. By asking to
vary on User-Agent you are asking your CDN to store many copies of
your resource which is not very efficient for them, hence their
turning off caching in this case.

What are you varying on user-agent for specifically? Can you solve the
problem in some other way?

On Tue, Oct 16, 2012 at 4:47 PM, Jody <odonne...@gmail.com> wrote:
> We have an issue where adding the "Vary: User-Agent" HTTP header to webpages
> attached to Akamai CDN. After adding this, the Akamai servers stopped
> caching hardly anything and drove the traffic back to the originating
> webserver increasing the load very significantly. Does anyone know how to
> configure the akamai servers to appropriately cache the pages on the
> user-agent string and not have this happen?
>
> --
> You received this message because you are subscribed to the Google Groups
> "page-speed-discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/page-speed-discuss/-/p5XrsHFnoksJ.
> To post to this group, send email to page-spee...@googlegroups.com.
> To unsubscribe from this group, send email to
> page-speed-disc...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/page-speed-discuss?hl=en.

[Bryan McQuade]

Thanks Jody. The documentation you link to is managed by a different
team at Google. I'll touch base with them over the next few days and
see if they are aware of the CDN caching issues and if there are
alternate mechanisms we can use for this instead. I'll follow up
shortly when I have more information.

On Tue, Oct 16, 2012 at 5:57 PM, Jody <odonne...@gmail.com> wrote:
> Part of the Google documentation for Smartphones suggests using an HTTP
> Vary: User Agent header to send a potential signal to not cache the mobile
> and desktop versions when they are the same URL. This came out in a June
> announcement from Pierre Far.
>
>> The Vary HTTP header
>> The Vary HTTP header has two important and useful implications:
>>
>> It signals to caching servers used in ISPs and elsewhere that they should
>> consider the user agent when deciding whether to serve the page from cache
>> or not. Without the Vary HTTP header, a cache may mistakenly serve mobile
>> users the cache of the desktop HTML page or vice versa.
>>
>> It helps Googlebot discover your mobile-optimized content faster, as a
>> valid Vary HTTP header is one of the signals we may use to crawl URLs that
>> serve mobile-optimized content.

> https://groups.google.com/d/msg/page-speed-discuss/-/xdk8Bua8KQYJ.

[Bryan McQuade]

Thanks Igal. It is very helpful to understand how additional CDNs process this header.

Unfortunately, using a different hostname for mobile has undesirable consequences, namely that it requires an HTTP redirect that incurs an additional DNS lookup, TCP connection, and request/response, substantially slowing down users on mobile. Since round trips on mobile are very expensive, this is not a great solution either.

Igal and Jody, what other mechanisms would be useful for you to communicate that your content varies depending on whether the client has a mobile vs desktop UA? Would additional information in a sitemap xml file indicating that the content of a URL varies depending on whether the client has a mobile vs desktop UA be an ok way to communicate this to crawlers?

On Tue, Oct 23, 2012 at 10:54 AM, Igal Zeifman - Incapsula <ig...@incapsula.com> wrote:

Very interesting observation.

This motivated me to do some research (I work for CDN provider myself).

From what I can tell, "Vary:User Agent" will prevent Caching (as it basically marks the content as "personalized" - even if for a group of visitor)

How about creating a different URL structure for mobile users (i.e. identifying the client and sending all mobile traffic to mobile.mydomain.com)?
After reading Google Documentation I understand your reasoning but I still think that combining Vary:User-Agent and Caching will almost certainly lead to false-positives.

GL

On Tuesday, October 16, 2012 11:57:14 PM UTC+2, Jody wrote:

Part of the Google documentation for Smartphones suggests using an HTTP Vary: User Agent header to send a potential signal to not cache the mobile and desktop versions when they are the same URL.  This came out in a June announcement from Pierre Far.

The Vary HTTP header
The Vary HTTP header has two important and useful implications:

1. It signals to caching servers used in ISPs and elsewhere that they should consider the user agent when deciding whether to serve the page from cache or not. Without the Vary HTTP header, a cache may mistakenly serve mobile users the cache of the desktop HTML page or vice versa.

1. It helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals we may use to crawl URLs that serve mobile-optimized content.     

On Tuesday, October 16, 2012 2:11:14 PM UTC-7, Bryan McQuade wrote:

> page-speed-discuss+unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/page-speed-discuss?hl=en.

--

You received this message because you are subscribed to the Google Groups "page-speed-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/page-speed-discuss/-/XhEMDT8GcUoJ.

[Bryan McQuade]

Jody,

In the short term I don't have any good solutions for you to work around this issue. First, I assume that you must have some way to communicate to Akamai which is your mobile and which is your desktop response, so they can properly serve the desktop version to desktop clients, and the mobile version to mobile clients. Is that right? If so I'd be curious to know how that works. Is there a custom HTTP response header you send to Akamai to tag a response as mobile vs desktop? Or does Akamai send something to your origin indicating they want the mobile vs desktop version via e.g. a query parameter?

If possible, can you configure your Akamai serving to add a Vary:User-Agent header when Akamai serves responses to your users. without you having to include Vary:User-Agent in your responses from the origin? This would cause Akamai to serve the response to Google with this header, which would allow them to more efficiently discover that you have a mobile version. I know this is not an ideal solution, but it may be the best option in the short term anyway.

On Wed, Oct 24, 2012 at 10:57 AM, Igal Zeifman - Incapsula <ig...@incapsula.com> wrote:

Jody - thanks for sharing this.

Bryan I see your point about using different URL but I don't think a redirect is a "must" here (how about a providing a popup for mobile visitor that can direct them to the mob. subdomain, if they so choose)
As to sitemap xml file... I guess this could be done, but honestly it will probably be simpler to do the same thing with dynamic parameters and then explain the redundancy to Google via Webmaster Tools... (or am I wrong about this?)

"Canonical" tag will also work and as far as I know it can be used cross-domain. 

Finally, I think that even "noidexing" the redundant mobile version will not hurt SEO or effect Google SERP results.

Anyway, Caching wise, I think that any solution that can differentiate between the 2 versions of the same object (i.e. providing different and uniqe URL for regular and mobile content) is a good one. 

Just as long as it can be combined with relevant HTTP header - something that can allow Caching... (i.e. Max-Age:XXXX)

The company I work for uses a patent pending "behavior learning" algorithm to deal with dynamic Caching (quite successfully I might add)...

This is a gist of it: Dynamic Caching Algorithm and I think it maybe useful to look at.
As I've said, the current heuristics  will not help Cache any User-Agent:Vary objects (just like Akamai wrote - this can create issues when dealing with personalized content).
Maybe, by introducing additional steps into the process (like checking sitemap xml for preferment signals or recognizing universally accepted parameters) the algorithm can be improved?

  

On Tuesday, October 23, 2012 10:44:17 PM UTC+2, Jody wrote:

Thanks everyone for these replies and looking into this.  The XML sitemap and <head> section of the HTML document offer an opportunity to put a link "alternate" pointing to the mobile URL.  In conjunction, the mobile page would have a link "canonincal" pointing to the desktop page.  I am not sure how this would help with the caching aspects of the pages, but maybe you know more?  These don't help with the caching piece, but might it be better to forego this part with CDN's?

Just to update as well, I got some of the akamai documentation which outlines how they look at this:

Why don't Akamai servers cache objects if the origin server's Vary header contains anything other than Accept-Encoding?

This is necessary to prevent different versions of content from being cached and then later served to end users for which they are not intended. For instance, this prevents pages in one language from being cached and served to users of another language, or pages designed specifically for a certain browser from being cached and then served to users on another browser.

The HTTP Vary header is used by servers to indicate that the object being served will vary (the content will be different) based on some attribute of the incoming request, such as the requesting client's specified user-agent or language. The Akamaiservers cannot cache different versions of the content based on the values of the Vary header. As a result, objects received with a Vary header that contains any value(s) other than "Accept-Encoding" will not be cached. To do so might result in some users receiving the incorrect version of the content (wrong language, etc.)

An "Accept-Encoding" value to the Vary header is the exception to this rule only when it relates to serving the object compressed. Since compression does not change the content of the object (only its size for transfer), an object that varies only by its compression can be safely cached.

To summarize, Akamai servers will cache the object, based on the configuration settings, in either of the following cases:

• If "Vary: Accept-Encoding" is received and the content is served compressed ("Content-Encoding: gzip").
• If no Vary header at all is received.

More detailed description of Vary and Accept-Encoding header interaction:
"Vary: Accept-Encoding" from the origin server indicates that the object served varies based on values received in the Accept-Encoding header of the incoming client request. For instance, the Akamai servers send the header "Accept-Encoding: gzip" in their forward requests to tell the origin server that gzip compression is supported. It is expected that the origin server will respond with the "Vary: Accept-Encoding" and "Content-Encoding: gzip" headers if serving the object compressed, or with no Vary header at all if serving the object uncompressed. Any other combination, such as other values in the Vary header or no "Content-Encoding: gzip" header in the response, indicates that the object varies in some way other than compression (that its actual content may be different), and therefore should not be cached.

> page-speed-discuss+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/page-speed-discuss?hl=en.

--
You received this message because you are subscribed to the Google Groups "page-speed-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/page-speed-discuss/-/XhEMDT8GcUoJ.

To post to this group, send email to page-spee...@googlegroups.com.
To unsubscribe from this group, send email to page-speed-discuss+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/page-speed-discuss?hl=en.

--
You received this message because you are subscribed to the Google Groups "page-speed-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/page-speed-discuss/-/EG5PonyC88gJ.

[Christian Oliveira]

Thank you very much for your detailed explanation!

I'll forward it to our Sysadmins so they can check that with the Akami support team

Christian

2013/5/9 Guy Podjarny <guy...@gmail.com>

Hi Christian, 

The key point to understand is that the response Akamai sees and the response Google (or browsers) see is not the same one. 

Akamai, and other CDNs, are effectively extensions of your infrastructure. You control the behavior of the Akamai Edge when it receives a request, and can therefore dictate how to handle caching of a response on the edge, and manipulate the headers passed on from the Akamai edge to the client. 

Google, on the other hand, is just a client, and more specifically it's outside your control. As a result, the only way you can communicate something to Google - like the fact you serve different content to a mobile and a desktop website - is through HTTP. There are also caching forward proxies, like those often used by ISPs, which are also outside of your control, so again you have to use the HTTP Vary header to tell them not to serve content cached during a desktop client request to a mobile client.

The way to bridge this gap is through Akamai configuration. Assuming your origin indeed returns different content for mobile and desktop content, and assuming your content is cacheable in the first place, you'd want to do this:

1) Return a "Vary: User-Agent" from your origin

2) Configure Akamai to ignore the Vary: User-Agent, but not remove it

3) Configure Akamai to cache based on the "is mobile" characteristic of the incoming device, using Aqua Ion Mobile's Device Characteristics feature. When the next request arrives, if it has the same "is mobile" flag, the response will be returned from cache. If not, it'll be routed through to the origin, and then its response will be cached for the new "is mobile" value.

Note that 1-2 can be replaced with not returning a Vary: User-Agent and adding that header on the edge.

I'd suggest contacting Akamai support (or the services person you work with) to walk you through doing so in the UI.

Cheers,

Guypo

On Thursday, May 9, 2013 8:32:28 AM UTC-4, Christian Oliveira wrote:

Hi Guy,

Thanks for adding some light to this problem :)

Google specifically recommends to use "Vary: User-Agent" header for non-responsive mobile websites, that's why started to use it and discover the problem with Akamai.

I recently send a question to Google WMC to know if they still recommend using it after knowing that CDNs like Akamai won't cache the page if that header is there, and they say they do: http://www.youtube.com/watch?v=va6qtaiZRHg

Could you explain further your solution proposed in  1) ?

Thanks in advance!

Christian

El jueves, 11 de abril de 2013 00:50:38 UTC+2, Guy Podjarny escribió:

Hi, 

Sorry for the late response, I only stumbled onto this thread now.I work for Akamai and just wanted to point out a few things. 

1) If your goal is mostly to send a "Vary: User-Agent" header to the user (not to Akamai), then you can simply configure Akamai to ignore the Vary header (it should be a simple checkbox under the "Optional Features") and then return "Vary: User-Agent" from your servers. There are also ways of configuring Akamai to add headers to the response being sent to the client, but in this case it'll be easier to just have Akamai ignore Vary and send it back from your origin.

2) If you indeed have different content for different clients, for example use the same URL for mobile & desktop but serve different content, there are also ways to configure Akamai to have a more tuned cache key, for example caching based on whether the device is a smartphone or not. Many of the mobile specific features are packaged as part of Aqua Ion Mobile (http://www.akamai.com/dl/brochures/aqua_ion_mobile_pb.pdf). 

3) I agree Vary is broken, which is why we currently need to rely on configuration for the edge instead of Vary. At some point in the future maybe "Key" (http://tools.ietf.org/html/draft-fielding-http-key-00) would help us out here, though Key has some pretty major impact on cache lookup time so probably can't be implemented quite as-is. Various parties are experimenting with Key right now, if we manage to find the right way to make it work, we may not need to have this conversation in the future...

Cheers,

Guypo

--
You received this message because you are subscribed to a topic in the Google Groups "page-speed-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/page-speed-discuss/kk-_1xslHrE/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to page-speed-disc...@googlegroups.com.

To post to this group, send email to page-spee...@googlegroups.com.

Visit this group at http://groups.google.com/group/page-speed-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Christian

@christian_wilde