Ask for solution recommendation to setup DNS resolver with EDNS-Client-Subnet

[iuc...@gmail.com]

Hi,

I am working in a French multinational company, in charge of Greater China region, sit in Shanghai, China.

We are using Google Apps (G-Mail, Contacts, Calendar, Drive, Sites, etc.) for office application.

As we all know the GFW (Greater Firewall) in place, which blocks access to many Internet resources (Google, Facebook, Twitter, SalesForce, GitHub…). So we implemented tunnels to Hong Kong/Taiwan to cross it.

 

The network layer topology is simple:

• We setup 1 VPN tunnel to Hong Kong (with 1 other tunnel to Taiwan as redundancy).
  
• We created routing rules based on China’s country IP range. If destination is China, traffic go out through local link directly; if destination is Non-China, traffic go out though VPN tunnel.
  
• 2 tunnels will be switched in case 1 is down.
  

 

On DNS part, since China DNS providers are doing DNS poisoning, we choose a France DNS provider as resolver.

The solution works ok, except France DNS always reply record according to the breakout IP. This causes slowness when we access China website, e.g. open www.taobao.com, but in fact access a Taobao’s CDN node in Iceland.

 

My idea is to change to Google DNS (or maybe OpenDNS) and utilize EDNS-Client-Subnet.

• When local client doing DNS query, the on premise Recursive Name Server (currently using Windows Server 2008 R2’ DNS role) should pass the query to 8.8.8.8 with location indication of Shanghai (in reality, the traffic will go through Hong Kong tunnel and expose Hong Kong IP to 8.8.8.8.)
  
• 8.8.8.8 should reply record with a node most optimized for Shanghai.
  
• Client then access the node with local Internet link, with best speed.
  

 

After my research, I don’t think Windows Server support EDNS-Client-Subnet yet. Another stable Name Server should be adopted to replace it for local DNS service.

 

So I am asking is there any recommended solution to achieve such goal? Any special considerations?

 

Thanks in advance,

[Alex Dupuy]

On DNS part, since China DNS providers are doing DNS poisoning, we choose a France DNS provider as resolver.

The solution works ok, except France DNS always reply record according to the breakout IP. This causes slowness when we access China website, e.g. open www.taobao.com, but in fact access a Taobao’s CDN node in Iceland.

 

My idea is to change to Google DNS (or maybe OpenDNS) and utilize EDNS-Client-Subnet.

Sending EDNS Client Subnet (ECS) options to Google Public DNS may or may not do what you want. I just checked OpenDNS, and their name servers appear to completely ignore ECS data that you send to them - they replace it with new ECS data using your client IP address:

$ dig +noall +answer +nocl TXT o-o.myaddr.l.google.com @208.67.220.220 +subnet=1.2.3.0/24

o-o.myaddr.l.google.com. 60 TXT "208.67.217.17"

o-o.myaddr.l.google.com. 60 TXT "edns0-client-subnet 192.0.2.0/24"

### your actual IP address (or part of it) on the second line means "no luck, your ECS data was replaced"

You can try the same experiment with the Google Public DNS servers - depending on where your queries are coming out in France, and what IPv4 or IPv6 address you use, you may have better or worse luck (and this is not guaranteed to continue to work in the same way indefinitely). Note that (for this experiment using o-o.myaddr) if you get a TTL with a value other than 59 or 60 you are getting cached data from another query, so ignore the result and try again.

$ dig +noall +answer +nocl TXT o-o.myaddr.l.google.com @8.8.8.8 +subnet=1.2.3.0/24

o-o.myaddr.l.google.com. 59 TXT "74.125.176.202"

o-o.myaddr.l.google.com. 59 TXT "edns0-client-subnet 1.2.3.0/24"

### provided subnet IP address on the second line means "success, your ECS data was sent to the authoritative server"

If you are interested in using the DNS-over-HTTPS API (https://developers.google.com/speed/public-dns/docs/dns-over-https) we do propagate supplied ECS data to the authoritative servers except for the (very non-hypothetical) case where we are contractually obligated not to do so (the experiment won't reveal this, since it is querying our own authoritative server, where we can do so). There are some tools out there (search Google) that will proxy regular DNS to DNS-over-HTTPS, so this might work for you.

After my research, I don’t think Windows Server support EDNS-Client-Subnet yet. Another stable Name Server should be adopted to replace it for local DNS service.

You don't want to use Windows DNS Server for this, and they don't support ECS. Unbound (edns-subnet branch) and PowerDNS (https://github.com/PowerDNS/pdns/issues/573) seem to be the only standard recursive servers that support it.

[iuc...@gmail.com]

Hi Alex,

Thank you for the comment!

If you are interested in using the DNS-over-HTTPS API (https://developers.google.com/speed/public-dns/docs/dns-over-https) we do propagate supplied ECS data to the authoritative servers except for the (very non-hypothetical) case where we are contractually obligated not to do so (the experiment won't reveal this, since it is querying our own authoritative server, where we can do so). There are some tools out there (search Google) that will proxy regular DNS to DNS-over-HTTPS, so this might work for you.

I did the same test and confirm only 8.8.8.8 respond with ECS sent by user specified, not respond with the client IP as OpenDNS, Google is stronger.

DNS to DNS-over-HTTPS proxy is a good option, I searched around but can't identify an outstanding candidate. Do you have recommendation for me to study deeper into?

You don't want to use Windows DNS Server for this, and they don't support ECS. Unbound (edns-subnet branch) and PowerDNS (https://github.com/PowerDNS/pdns/issues/573) seem to be the only standard recursive servers that support it.

 

Indeed, Windows DNS doesn't support ECS, but I have to keep it as it runs on a AD DC and tightly integrated for AD hosts and Intranet hosts.

My idea is use the AD DC+Windows DNS to serve all internal queries, and for all external queries, forward them to a recursor that support ECS to 8.8.8.8.

I tried PowerDNS Recursor these days (it claims support ECS), and set "forward-zones-recurse=8.8.8.8".

But when I dig it with +subnet option, it always return same IP for both China and Kong Kong subnet. Meanwhile 8.8.8.8 respond 2 different IP as desired. It looks like PowerDNS either doesn't respond to ECS, or doesn't chain the ECS info to 8.8.8.8? Not sure it needs further configuration of not.

[Alex Dupuy]

DNS to DNS-over-HTTPS proxy is a good option, I searched around but can't identify an outstanding candidate. Do you have recommendation for me to study deeper into?

Either https://github.com/wrouesnel/dns-over-https-proxy (Go) or  https://github.com/aarond10/https_dns_proxy (C) look plausible, you should probably try them both and use the one that works best for you (or if they both have problems, the one where you are most comfortable debugging the code)

I tried PowerDNS Recursor these days (it claims support ECS), and set "forward-zones-recurse=8.8.8.8".

But when I dig it with +subnet option, it always return same IP for both China and Kong Kong subnet. Meanwhile 8.8.8.8 respond 2 different IP as desired. It looks like PowerDNS either doesn't respond to ECS, or doesn't chain the ECS info to 8.8.8.8? Not sure it needs further configuration of not.

Without checking, I would expect that PowerDNS is ignoring the ECS data it receives. If you run the PowerDNS Recursor on a different network from your client, you can use o-o.myaddr.l.google.com and see whether the original client IP is passed to Google as ECS from PowerDNS. (Note that if your original client IP is an RFC1918 private address—10.*.*.*, 172.{16-31}.*.*, 192.168.*.*—passing that to Google is useless and may well be ignored, so you need to use a public routable IP address for your clients (which may make things a bit tricky to get it passed through the tunnel).  There might be a PowerDNS configuration option to map client subnet IPs, I don't know. Using (public) IPv6 (addresses) for the clients to contact PowerDNS Recursor may give better results (it is possible that PowerDNS doesn't handle IPv6 addresses in ECS data properly, but Google can handle it, as can most ECS-enabled authoritative servers).

[iuc...@gmail.com]

Either https://github.com/wrouesnel/dns-over-https-proxy (Go) or  https://github.com/aarond10/https_dns_proxy (C) look plausible, you should probably try them both and use the one that works best for you (or if they both have problems, the one where you are most comfortable debugging the code)

I looked at both projects, but they looks still very primitive, and i would like wait some time for a more mature proxy to try.

Without checking, I would expect that PowerDNS is ignoring the ECS data it receives. If you run the PowerDNS Recursor on a different network from your client, you can use o-o.myaddr.l.google.com and see whether the original client IP is passed to Google as ECS from PowerDNS. (Note that if your original client IP is an RFC1918 private address—10.*.*.*, 172.{16-31}.*.*, 192.168.*.*—passing that to Google is useless and may well be ignored, so you need to use a public routable IP address for your clients (which may make things a bit tricky to get it passed through the tunnel).  There might be a PowerDNS configuration option to map client subnet IPs, I don't know. Using (public) IPv6 (addresses) for the clients to contact PowerDNS Recursor may give better results (it is possible that PowerDNS doesn't handle IPv6 addresses in ECS data properly, but Google can handle it, as can most ECS-enabled authoritative servers).

I did more test on pdns-recursor and finally make it forward query to 8.8.8.8 with ECS option appended. (Through a very tricky setting after several days of source code reading and recompiling! I am checking with PowerDNS team is it a bug or intentional design).

However, the pdns-recursor by now only take source IP as ECS data, and ignored the original ECS data set in dig command. I am also discussing with PowerDNS team to see whether it is possible to add support of the ECS data in query, like 8.8.8.8 is now doing.

Anyway, since Windows DNS don't support ECS in query, so current pdns-recursor behavior can work. Except must do something on the DNS packet routing.

Current:

Windows DNS connect to pdns-recursor through local LAN, which source IP is a private IP, which make no sense to 8.8.8.8. So it still respond with a node for Hong Kong.

TODO:

- Set pdns-recursor on another site than Windows DNS located. 

- Let Window DNS send query through local public IP (still a Shanghai IP though) to remote public IP of pdns-recusor. 

- pdns-recursor will take the valid public source IP of Windows DNS, then generate ECS option to 8.8.8.8

- After get result (should be optimal node for Shanghai, rather than for Hong Kong) from 8.8.8.8, pdns-recursor will send back to Windows DNS.

Will share result of such test.

[iuc...@gmail.com]

I did more test on pdns-recursor and finally make it forward query to 8.8.8.8 with ECS option appended. (Through a very tricky setting after several days of source code reading and recompiling! I am checking with PowerDNS team is it a bug or intentional design).

However, the pdns-recursor by now only take source IP as ECS data, and ignored the original ECS data set in dig command. I am also discussing with PowerDNS team to see whether it is possible to add support of the ECS data in query, like 8.8.8.8 is now doing.

Anyway, since Windows DNS don't support ECS in query, so current pdns-recursor behavior can work. Except must do something on the DNS packet routing.

Current:

Windows DNS connect to pdns-recursor through local LAN, which source IP is a private IP, which make no sense to 8.8.8.8. So it still respond with a node for Hong Kong.

TODO:

- Set pdns-recursor on another site than Windows DNS located. 

- Let Window DNS send query through local public IP (still a Shanghai IP though) to remote public IP of pdns-recusor. 

- pdns-recursor will take the valid public source IP of Windows DNS, then generate ECS option to 8.8.8.8

- After get result (should be optimal node for Shanghai, rather than for Hong Kong) from 8.8.8.8, pdns-recursor will send back to Windows DNS.

Will share result of such test.

Confirm it works as what I want. 

[Alex Dupuy]

Confirm it works as what I want. 

If you can share any details of your PowerDNS Recursor configuration (even generally) I imagine there may be others who find it useful.

 

[iuc...@gmail.com]

If you can share any details of your PowerDNS Recursor configuration (even generally) I imagine there may be others who find it useful.

Agree, I was stuck on other business the whole week. I will write a full article to explain how I do it and post to the group.

However, when I test the PDS (Power DNS Recursor), for most websites, 8.8.8.8 can resolve per the ECS OPT forwarded by PDS.

But for a few websites, e.g. www.sohu.com, www.cctv.com, 8.8.8.8 still reply according to Source IP (Hong Kong). Seems 8.8.8.8 can't find a optimal China node for ECS OPT subnet (Shanghai), therefore fall down to use Source IP (Hong Kong)? sohu and cctv are definitely China websites with many CDN nodes.

I verified with Web DNS https://dns.google.com/query?name=www.sohu.com&type=A&dnssec=true&ecs=180.166.129.0 and it resolve to 175.100.207.205 (Hong Kong) while 180.166.129.0 is Shanghai IP.

I wonder why 8.8.8.8 can't find optimal answer for a request to www.sohu.com from 180.166.129.0?

P.S. More weird, it seems for several chances, 8.8.8.8 did resolved www.sohu.com to a Shanghai IP, but can't reproduce. I suspect it because I query 8.8.8.8 too often during the test? Get no clue...

[Alex Dupuy]

But for a few websites, e.g. www.sohu.com, www.cctv.com, 8.8.8.8 still reply according to Source IP (Hong Kong). Seems 8.8.8.8 can't find a optimal China node for ECS OPT subnet (Shanghai), therefore fall down to use Source IP (Hong Kong)? sohu and cctv are definitely China websites with many CDN nodes.

The authoritative name servers for those domains do not support ECS, so passing ECS to Google Public DNS will have no effect; authoritative server GeoIP resolution will be based on our resolvers' IP addresses.

[Shen Wan]

Sometimes sohu's nameservers return CNAME fusa.a.sohu.com. for www.sohu.com and sometimes they return fbjuni.a.sohu.com, even for queries we send out from the same location. Maybe they are doing some kind of load balancing using DNS but I do not see a pattern though. I think that's the reason you get a Shanghai IP occasionally. Hopefully they can do a better job geo-locating our resolvers' egress IP. Unfortunately there is nothing we can help when their nameservers do not support EDNS client subnet.