Popular websites are often not present in cache?

[aura...@eng.ucsd.edu]

Popular websites such as google.com, wikipedia.com, and youtube.com do not appear to be present in the resolver's cache for large periods of time (at least a day). I noticed this by running the command "dig @8.8.8.8 domain.com +norecurse" on multiple websites, for a few months, once a second. As far as I can tell, Google DNS returns a SERVFAIL response if a DNS request is made with the Recursion Desired bit unset. Google.com, wikipedia.com, and youtube.com consistently return SERVFAIL. However, many websites of similar popularity according to the Alexa top websites ARE cached. I am hoping someone can tell me why these websites are not present in cache when their popularity would intuitively imply that they should be. Does Google DNS deliberately not cache certain websites? Is my measurement technique flawed? Any assistance you can give me would be very welcome.

For context, I am a PhD student at University of California San Diego studying Internet measurement, and a three-time Google intern. One of my current projects involves measuring the behavior of different DNS resolvers, which is why I'm interested in this. This is a cross-post from the Google DNS Issue Tracker, because I posted there two weeks ago but haven't gotten a response yet.

Has anyone else noticed this behavior?

Thank you.

[Alex Dupuy]

aurandal wrote:

Popular websites such as google.com, wikipedia.com, and youtube.com do not appear to be present in the resolver's cache for large periods of time (at least a day). I noticed this by running the command "dig @8.8.8.8 domain.com +norecurse" on multiple websites, for a few months, once a second. As far as I can tell, Google DNS returns a SERVFAIL response if a DNS request is made with the Recursion Desired bit unset. Google.com, wikipedia.com, and youtube.com consistently return SERVFAIL. However, many websites of similar popularity according to the Alexa top websites ARE cached. I am hoping someone can tell me why these websites are not present in cache when their popularity would intuitively imply that they should be. Does Google DNS deliberately not cache certain websites? Is my measurement technique flawed? Any assistance you can give me would be very welcome.

Your analysis of RD flag is correct.

You are missing the fact that google.com, wikipedia.org, and youtube.com all use DNS name servers that support EDNS Client Subnet (ECS), which fragments the cached based on the /24 or /56 subnet of the client address. While these are popular domains, unless someone from a network in the same range queries Google Public DNS without RD=0, our resolvers will not have any cached data. The other "similarly popular" domains probably do not support ECS, so the cache is not fragmented and you will see returned cache records.

[aura...@eng.ucsd.edu]

Thank you so much, Alex! We've been wondering about this for a while now.