May you preemptively whitelist LinkedIn.com NSes for EDNS-Client Subnet?

220 views
Skip to first unread message

pub...@samir.ca

unread,
Aug 2, 2016, 4:39:05 AM8/2/16
to public-dns-discuss
We currently have EDNS-CS enabled on 4 of our 8 NS and want to enable it on the remaining 4 this Wednesday. As we expect a traffic shift, we'd like to know how long the automatic whitelisting takes and what thresholds have to be met? 

Is there any way you may pre-whitelist our NSes before our change so that the traffic shift happens exactly during our change window?

The zone we're changing is linkedin.com.

These 4 NS already have EDNS-CS:


These are the 4 that we're adding it to on Wednesday:


My corporate Google account doesn't support Google Groups so I'm using my personal account to post this. My next post will be TXT record validating this message.

Any help would be much appreciated. We can move the change window around if you need more time. Thanks.

-Samir Jafferali
DNS SRE for LinkedIn

Samir Jafferali

unread,
Aug 2, 2016, 11:51:50 AM8/2/16
to public-dns-discuss
I've created TXT record NESK_5zCKc0.linkedin.com to validate this request is from LinkedIn. NESK_5zCKc0 is the url identifier of this thread. Thank you. 

-Samir

Alex Dupuy

unread,
Aug 3, 2016, 3:46:53 PM8/3/16
to public-dns-discuss
Hi Samir,


You wrote:
We currently have EDNS-CS enabled on 4 of our 8 NS and want to enable it on the remaining 4 this Wednesday. As we expect a traffic shift, we'd like to know how long the automatic whitelisting takes and what thresholds have to be met? 

Is there any way you may pre-whitelist our NSes before our change so that the traffic shift happens exactly during our change window?

The zone we're changing is linkedin.com.

We really don't do whitelisting any more.

The Dyn resolvers are well known to us (as they serve many other domains) so there were effectively pre-whitelisted automatically, simply because we have sent them millions of DNS queries with ECS, so even several dropped responses won't change that.

One thing I was wondering about, because it has been an issue with some of the other domains that Dyn serves, is that although all their nameservers support ECS, unless you are using their Traffic Director product, your domain is actually not ECS-enabled (i.e. results are returned with global (/0) scope and are cached and returned to anyone regardless of their IP subnet). In particular, their Traffic Controller depends on anycast IPs and BGP routing to send queries to a nearby location, which then sends response based on its own location. This doesn't always work as well as one might hope, because BGP (enough said).

I was concerned that you might be using ECS and returning scoped data on your "self-branded" ns[1234].linkedin.com (really Neustar) nameservers but would be using Dyn's Traffic Controller rather than Traffic Director - mixing the two approaches would not be as effective. However, it appears that both your Dyn and self-branded Neustar nameservers are returning zero-scoped responses, which is fine, although in that case the only benefit of enabling ECS is visibility into where the name lookups are really coming from.

If at some point in the future, you do plan to enable ECS, you may want to consider the following guidelines for ECS deployment (I will be adding this to our official Google Public DNS documentation at some point, this is just a draft).



The recent RFC 7871 – EDNS Client Subnet (ECS) – provides extra IP address information to authoritative DNS name servers for Content Delivery Networks (CDNs) or other latency-sensitive services, to help them return accurate geo-located responses when responding to name lookups coming through public DNS resolvers. The RFC describes some features of ECS that authoritative name servers must implement; but implementors don’t always follow those requirements. There are also operational and deployment aspects of ECS that are not addressed in the RFC, but which can cause problems unless you are careful. If you are deploying ECS on your authoritative name servers and DNS zones, follow the guidelines below, or Google Public DNS may not send ECS data to your name servers, and may be unable to resolve names in your DNS zones.


Definitions of requirements


We use the following terminology for describing EDNS Client Subnet protocol operation in the guidelines in the next section, and note that if these requirements are not met, ECS is very likely to work poorly or not at all in some situations.


A name server implements (or supports) ECS if its response to a query with ECS option data is a reply with ECS option data (even if that reply has a global (/0) scope). If the reply does not have ECS option data that matches the query’s ECS subnet family, address, and source prefix length,1 the name server does not correctly implement ECS, and a recursive resolver that probes for ECS implementations may not detect it.

A zone is ECS-enabled if its name servers' responses to queries with ECS option data that has a non-zero prefix are replies with ECS option data that has non-zero scope.


Guidelines

  1. All authoritative name servers for an ECS-enabled zone must implement ECS and enable it for the zone.

    • Even just one of many servers that does not implement ECS or enable it for the zone quickly becomes the source of most cached data. Its responses have global scope and are used (until their TTL expires) as the response for all queries for the same name (regardless of client subnet). Responses from servers that do implement ECS are only used

  2. Authoritative name servers that implement ECS must do so for all zones being served from an IP address (or name server hostname) even if the zones are not ECS-enabled.

    • Intermediate resolvers that probe for ECS implementations may detect by IP address rather than name server hostname (let alone DNS zone) simply because the number of addresses is smaller than the number of name server hostnames and much smaller than the number of DNS zones. If an authoritative name server does not always send replies with ECS data in response to queries with ECS data (even for zones that are not ECS-enabled), a probing resolver may not send further queries with ECS data.

  3. Authoritative name servers that implement ECS must answer all ECS queries with ECS, including negative and referral (delegation) responses.

    • Negative responses should use global scope (/0) for compatibility and caching.2

  4. Authoritative name servers returning ECS-enabled CNAME responses SHOULD3 only include the first CNAME in the chain, and furthermore the final target of the CNAME chain should be ECS-enabled to the same granularity. Because of ambiguity in the ECS specification, some recursive resolvers (notably Unbound4) may return a response with the scope of the final non-CNAME domain (/0 if it is not ECS-enabled). The practical effect of this is usually limited, however, since Unbound is most often deployed as a local stub or forwarding resolver, and all of its clients would be likely to get the same response.

  5. ECS data may contain IPv6 addresses even if your name servers are IPv4-only (and vice-versa, although IPv6-only name servers are rare).

    • You need to handle this correctly, with a valid ECS response (/0 scope is okay).

    • ECS for a zone can be enabled separately for IPv4 and IPv6 addresses.

  6. When implementing ECS support on name servers that previously did not implement it, use new IP addresses (and name server hostnames) for ECS enabled zones.

    • Authoritative name servers that already implemented ECS but returned global scope results can start returning ECS enabled answers for a zone as soon as the TTLs for previous replies expire.

    • Intermediate resolvers that probe for ECS implementation very rarely try ECS queries for an IP address (or name server hostname) that they have learned does not implement ECS (repeatedly returning FORMERR, BADVERS, or simply failing to respond at all). New ECS implementations on those IP addresses (or name server hostnames) will be auto-detected very slowly, or not at all.

  7. Make sure your network connections are reliable and that any response rate limiting is

  8. Make sure your responses are timely (ideally within 1 second under reasonable load).

    • Using online Geo-IP lookup services to handle DNS ECS queries will not work reliably.


  1. https://tools.ietf.org/html/rfc7871#section-7.2.1

  2. https://tools.ietf.org/html/rfc7871#section-7.4

  3. https://tools.ietf.org/html/rfc7871#section-7.2.1

  4. https://unbound.net/pipermail/unbound-users/2015-May/003875.html

Samir Jafferali

unread,
Aug 3, 2016, 5:05:53 PM8/3/16
to public-dns-discuss
Hi Alex,

>We really don't do whitelisting any more.
>The Dyn resolvers are well known to us

Okay.  Our NSes are dedicated and I realized this morning you auto-whitelisted our NSes when we enabled ECS on a test zone a month ago.

>unless you are using their Traffic Director product,
>both your Dyn and self-branded Neustar nameservers are returning zero-scoped responses

  We are indeed using Traffic Director only for www.linkedin.com and will ramp more records this sprint. Were you checking for scope on www.linkedin.com or another record?  

ECS should be enabled across all our NSes as of 10:15am and we're seeing traffic shifts inline with our expectations.  We're also synthetically monitoring the Google DNS response time for www.linkedin.com and it increased sharply at 10:15am indicating that you're now fetching answers for each synthetic agent rather than handing out a zero-scoped cached answer.  Thanks.

-Samir

Message has been deleted

Alex Dupuy

unread,
Aug 4, 2016, 4:58:06 AM8/4/16
to public-dns-discuss
I was checking the apex domain linkedin.com, and not the www.linkedin.com subdomain.  I get /0 scope for linkedin.com,  /24 (and sometimes /31 scope from Dyn) for www.linkedin.com:

; <<>> DiG 9.10.2 <<>> -4 www.linkedin.com +subnet=1.2.3.0/24 @ns1.linkedin.com.
; CLIENT-SUBNET: 1.2.3.0/24/24
;; SERVER: 156.154.64.23#53(156.154.64.23)
; <<>> DiG 9.10.2 <<>> -6 www.linkedin.com +subnet=1.2.3.0/24 @ns1.linkedin.com.
; CLIENT-SUBNET: 1.2.3.0/24/24
;; SERVER: 2001:502:f3ff::23#53(2001:502:f3ff::23)
; <<>> DiG 9.10.2 <<>> -4 www.linkedin.com +subnet=1.2.3.0/24 @ns1.p43.dynect.net.
; CLIENT-SUBNET: 1.2.3.0/24/31
;; SERVER: 208.78.70.43#53(208.78.70.43)
; <<>> DiG 9.10.2 <<>> -6 www.linkedin.com +subnet=1.2.3.0/24 @ns1.p43.dynect.net.
; CLIENT-SUBNET: 1.2.3.0/24/31
;; SERVER: 2001:500:90:1::43#53(2001:500:90:1::43)

but only for some client subnets:

; <<>> DiG 9.10.2 <<>> -4 www.linkedin.com +subnet=104.132.34.0/24 @ns1.linkedin.com.
; CLIENT-SUBNET: 104.132.34.0/24/24
;; SERVER: 156.154.64.23#53(156.154.64.23)
; <<>> DiG 9.10.2 <<>> -6 www.linkedin.com +subnet=104.132.34.0/24 @ns1.linkedin.com.
; CLIENT-SUBNET: 104.132.34.0/24/24
;; SERVER: 2001:502:f3ff::23#53(2001:502:f3ff::23)
; <<>> DiG 9.10.2 <<>> -4 www.linkedin.com +subnet=104.132.34.0/24 @ns1.p43.dynect.net.
; CLIENT-SUBNET: 104.132.34.0/24/24
;; SERVER: 208.78.70.43#53(208.78.70.43)
; <<>> DiG 9.10.2 <<>> -6 www.linkedin.com +subnet=104.132.34.0/24 @ns1.p43.dynect.net.
; CLIENT-SUBNET: 104.132.34.0/24/24
;; SERVER: 2001:500:90:1::43#53(2001:500:90:1::43)
Reply all
Reply to author
Forward
0 new messages