getaddrinfo breaks round robin DNS

[Patrick Feliciano]

Hello,

I was implementing a test consul cluster and testing round robin DNS when I made a startling discovery about the default behaviour of many apps that have switched to using getaddrinfo() for name resolution rather than gethostbyname(). getaddrinfo does an internal sort on returned addresses.  This means that when provided a randomized list in round robin it will sort it into a nearly deterministic list. This means that only one or 2 hosts receive almost all of the traffic while some hosts are never hit.  I encountered this while testing a service with 3 members running curl in a loop on an Ubuntu trusty host.  This issue is widespread as it comes from the upstream glibc implementing rule 9 of RFC3484. This has been an issue for over a decade but is becoming more or a problem as more and more applications and libraries switch to getaddrinfo for ipv6 capability believing it is a drop in replacement for gethostbyname.  This creates a serious limit in the usefulness of consul in load balancing internal services for legacy applications.

Patching all of our hosts glibc is impractical given the many distros in consideration.  It occurs to me that the best work around is a consul configurable that simply breaks the for loop after a single entry.  This gives getaddrinfo nothing to sort and returns effective round robin behaviour to the applications.  The only downside is that it doesn't give applications will fail thru capability a list to work through.  This should be mitigated by consul health checks and is the unfortunate price to get our broken round robin back.  Below is the local diff.  Let me know if I should do a pull request. 

diff --git a/command/agent/config.go b/command/agent/config.go
index c036591..664010c 100644
--- a/command/agent/config.go
+++ b/command/agent/config.go
@@ -76,6 +76,14 @@ type DNSConfig struct {
        // returned by default for UDP.
        EnableTruncate bool `mapstructure:"enable_truncate"`
 
+       // EnableSingleton is used to override default behavior
+       // of DNS and return only a single host in a round robin
+       // DNS service request rather than all healthy service
+       // members. This is a work around for systems using
+       // getaddrinfo using rule 9 sorting from RFC3484 which
+       // breaks round robin DNS.
+       EnableSingleton bool `mapstructure:"enable_singleton"`
+
        // MaxStale is used to bound how stale of a result is
        // accepted for a DNS lookup. This can be used with
        // AllowStale to limit how old of a value is served up.
@@ -1034,6 +1042,9 @@ func MergeConfig(a, b *Config) *Config {
        if b.DNSConfig.EnableTruncate {
                result.DNSConfig.EnableTruncate = true
        }
+       if b.DNSConfig.EnableSingleton {
+               result.DNSConfig.EnableSingleton = true
+       }
        if b.DNSConfig.MaxStale != 0 {
                result.DNSConfig.MaxStale = b.DNSConfig.MaxStale
        }
diff --git a/command/agent/dns.go b/command/agent/dns.go
index 33db8ba..1cee5c7 100644
--- a/command/agent/dns.go
+++ b/command/agent/dns.go
@@ -665,6 +665,9 @@ func (d *DNSServer) serviceNodeRecords(nodes structs.CheckServiceNodes, req, res
                if records != nil {
                        resp.Answer = append(resp.Answer, records...)
                }
+               if d.config.EnableSingleton {
+                       break
+               }
        }
 }

[Sean Chittenden]

btw, if people are interested, here is the quick fix for this *IF* you can run without IPv6 (the patch is important for those that can't).  We run the following when we install an OS:

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

cat<<EOF > /etc/gai.conf

scopev4 ::ffff:169.254.0.0/112 2

scopev4 ::ffff:127.0.0.0/104 2

scopev4 ::ffff:0.0.0.0 14

EOF

chmod 644 /etc/gai.conf

And it works immediately on CentOS hosts. I wish glibc followed suit with the rest of the *NIXes and ignored implementing rule 9 (I think it's been patched out in recent versions of glibc). -sc

[Patrick Feliciano]

FYI, I tested disabling IPv6 on Ubuntu Trusty,Precise, and Debian Squeeze.  Only disabling on Trusty turned off IPv4 sorting.  On the older hosts IPv4 sorting remained active. 

Based on the last link in the list below I think this will be true of CentOS versions that are slightly older as well.  I hope that future versions of glibc disable it completely.  Below are some links to more discussion about it.  The debian list conversation makes for a long and interesting read.

http://daniel.haxx.se/blog/2012/01/03/getaddrinfo-with-round-robin-dns-and-happy-eyeballs/
https://lists.debian.org/debian-glibc/2007/09/msg00347.html
http://ssms.cs2c.com.cn/otrs/pc.pl?Action=PublicFAQZoom;CategoryID=14;ItemID=2996

[Armon Dadgar]

Hey Patrick,

This is a great point. I’ve created a ticket for this here: https://github.com/hashicorp/consul/issues/1481

I think it should be pretty straightforward to make this a configurable option for the DNS server. 

This is certainly a subtle gotcha.

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/207c0bdb-0f98-43f2-bbeb-5fcdf23a3695%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.