HTTP Vary:User-Agent Server or Page Level?

[KeriMorgret]

Thomas, I think the voice recognition software botched some of your reply. Could you go through and edit it a little? There are some words that seem to be missing. Thanks!

[BlueprintMarketing]

Hi Keri,

Thank you for the heads up on that. I definitely was having some technical issues. I have cleaned it up let me know if you think it is a need any more work.

Thank you for letting me know.

Sincerely,

Thomas

[burnseo]

Hey Thomas, thank you for your interest in answering my question. However, the question isn't really about using a CDN. It is more around how using the Vary HTTP Header can affect the CDN performance. In addition, I wanted to find guidance on where to implement the Vary HTTP Header as it was brought to my attention that search engines don't like it when this is implemented site wide even on pages that don't have a mobile version.

[BlueprintMarketing]

• The cache-control header is the primary mechanism for an HTTP server to tell a caching proxy the "freshness" of a response. (i.e., how/if long to store the response in the cache)

• In some situations, cache-controldirectives are insufficient. A discussion from the HTTP working group is archived here, describing a page that changes only with language. This is not the correct use case for the vary header, but the context is valuable for our discussion. (Although I believe the Vary header would solve the problem in that case, there is a Better Way.) From that page:

Vary is strictly for those cases where it's hopeless or excessively complicated for a proxy to replicate what the server would do.

• This page describes the header usage from the server perspective, this one from a caching proxy perspective. It's intended to specify a set of HTTP request headers that determine uniqueness of a request.

A contrived example:

Your HTTP server has a large landing page. You have two slightly different pages with the same URL, depending if the user has been there before. You distinguish between requests and a user's "visit count" based on Cookies. But -- since your server's landing page is so large, you want intermediary proxies to cache the response if possible.

The URL, Last-Modified and Cache-Control headers are insufficient to give this insight to a caching proxy, but if you add Vary: Cookie, the cache engine will add the Cookie header to it's caching decisions.

Finally, for small traffic, dynamic web sites -- I have always found the simpleCache-Control: no-cache, no-storeand Pragma: no-cache sufficient.

Edit -- to more precisely answer your question: the HTTP request header 'Accept' defines the Content-Types a client can process. If you have two copies of the same content at the same URL, differing only in Content-Type, then using Vary: Accept could be appropriate.

Update 11 Sep 12:

I'm including a couple links that have appeared in the comments since this comment was originally posted. They're both excellent resources for real-world examples (and problems) with Vary: Accept; Iif you're reading this answer you need to read those links as well.

The first, from the outstanding EricLaw, on Internet Explorer's behavior with the Vary header and some of the challenges it presents to developers: Vary Header Prevents Caching in IE. In short, IE (pre IE9) does not cache any content that uses the Vary header because the request cache does not include HTTP Request headers. EricLaw (Eric Lawrence in the real world) is a Program Manager on the IE team.

The second is from Eran Medan, and is an on-going discussion of Vary-related unexpected behavior in Chrome:Backing doesn't handle Vary header correctly. It's related to IE's behavior, except the Chrome devs took a different approach -- though it doesn't appear to have been a deliberate choice.

[KeriMorgret]

Thomas, it appears that this is taken from http://stackoverflow.com/questions/1975416/trying-to-understand-the-vary-http-header. Q&A is for original answers; if you are referring to another blog post, it's best to just put a link into the blog post and let people go there rather than copy work (that may be copyright) and use that as your answer. Thanks for understanding!

[BlueprintMarketing]

Very true I shoud have compleated it woun't use a cell phone to Q&A

[burnseo]

I found some information  that suggests that it is recommended to avoid using the Vary HTTP Header by User-Agent site-wide because search engines and (and this is Google) would assume the other version simply hadn't yet been discovered and perhaps keep looking for it. There is also a recommendation to implement the Vary Header on a page-level only when there is a mobile version. This only applies to sites that are serving mobile HTML content dynamically based in the user-agent.  Additionally, there is some controversy around using the header when a CDN network like Akamai is in place because it can overload the site. Despite this controversy Google still recommends using the header. These seem to be two important points to consider before implementing the Vary HTTP Header.

[BlueprintMarketing]

I would go by what Google recommends I cannot imagine Akamai being something bad for website or overwhelming it anyway. You may try using a C name with your www. straight to the CDN & if you're using a mobile subdomain like m. also having that go directly into your content delivery network.

I hope this is better help.

sincerely,

Thomas

[P-C-A]

This post is deleted!

[P-C-A]

hey burnseo - if you're still getting notifications from this thread, would you happen to recall where you ended up finding info. that google recommends placing the vary header at page level?  running into the same question myself.  if you have links you could post to where you found the answer, that'd be great.  thanks!