[OT] Global DNS Failures February 1 2019

129 views
Skip to first unread message

Tony Gravagno

unread,
Jan 25, 2019, 6:02:49 PM1/25/19
to Pick and MultiValue Databases

I faced a dilemma with regard to posting on this topic or not. The topic of DNS is OT for this forum. But people here do rely on network communications to keep their systems going. I would feel bad Not bringing up this topic to this group and then watching people suffer here and elsewhere. So here goes...


"The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019. This change affects only sites which operate software which is not following published standards."


That is a quote from https://dnsflagday.net/


I need to crawl out from under my rock - I thought I was somewhat savvy about what's happening on the interwebs but this one blindsided me.


On February 1st, major DNS resolver software vendors and DNS public resolvers will stop doing workarounds to avoid misconfigurations in authoritative DNS servers and will start failing on errors found (especially regarding EDNS configuration and filtering).

The main change is that DNS software from vendors [ ... named on https://dnsflagday.net ] will interpret timeouts as sign of a network or server problem. Starting February 1st, 2019 there will be no attempt to disable EDNS as reaction to a DNS query timeout. This effectivelly means that all DNS servers which do not respond at all to EDNS queries are going to be treated as dead.

You can get more info about EDNS. To be clear about exactly what this is: When DNS servers exchange protocol info about "what features do you support", one server will ask another, "do you support the new EDNS protocol?". If the response is No, I do not support it, that's fine. Everything will proceed as normal. But if the server responds "I don't know what that is", then the calling server will no longer fail back to "well, that server is stupid so I'll dumb-down to basic DNS exchanges". Now, it's action will be "that server is too old, it must be unmaintained, I can't resolve the domain I need, so that domain must not exist."


What if it's Your domain that this other server, like Google, is trying resolve? What will Google display in search results if it thinks your site no longer exists? What if someone is trying to place an order using their GMail account, and the Google server says your domain doesn't exist because of the above DNS exchanges? How will that person get their order or other request to you?


You can check any domain name to see if it may be affected on or around February 1st. If you don't get an ALL OK kind of response then you need to look into it. Examples of ALL OK domains include Google.com, Siteground. An example of high profile sites that fail are Slack.com, WordPress.org. If you rely on those sites for any reason, like communications with your business partners or for your own blog/site, you may be affected. The authoritative servers, for some common hosts that many of us here use, also don’t seem to be properly configured for this. DreamHost is an example, where I host my servers and DNS.


I'm sure this thread could result in a lot of "I pass" and "I fail" responses. Your results for one site with your provider could be subject to the specific nameservers you use, and not a definitive statement about their overall standing. I think it would be better if we just test our own environments and then act independently on the info we have.


Many people may think "I'm not a DNS provider, I can't do anything, why should I care?"


As I said above, I host at DreamHost. I checked my domains and I'm now terrified that all of my sites and email services will be unreachable in a week or so for both website and email. Note that I don't even host email at DreamHost, but because my domain is hosted there and my MX record points to my mail server, if their DNS fails then so does my email. So I'm trying to get DreamHost to respond to inquiries about what they intend to do about this. Is failure an option?


I think any company that has a SLA should care, where they have made commitments based on their upline providers, to (legally) provide some level of consistent service. Any company that provides services to others may find their customer support activity skyrocket around Feb1 because people mysteriously can't connect to some service anymore. That becomes a resource issue.


From the ISC:

Non-compliant domains may become unavailable
Domains served by DNS servers that are not compliant with the standard will not function reliably after February 1, 2019, and may become unavailable.
If your company’s DNS zones are served by non-compliant servers, your online presence will slowly degrade or disappear as ISPs and other organizations update their resolvers. When you update your own internal DNS resolvers to versions that don’t implement workarounds, some sites and email servers may become unreachable.
Test your domains


To verify that this is not an internet myth, I refer you to several other sources of information, in order of what seems to me to be less technical to more technical.

https://wordtothewise.com/2018/10/dns-flag-day/
https://www.tripwire.com/state-of-security/security-awareness/dns-flag-day-dns-doomsday/
https://www.itproportal.com/features/dns-flag-day-will-your-website-survive-the-domain-doomsday/
https://etherealmind.com/tech-notes-dns-flag-day-february-1-2019/
https://twitter.com/dnsflagday
https://www.isc.org/blogs/dns-flag-day/
https://blog.apnic.net/2018/10/11/dns-flag-day/


Now, all of that said. I've researched this and I've found surprisingly little buzz on the topic at all. Most people don't know about what's happening. Most casual observers who do know don't seem concerned. There's not enough buzz on this to truly warrant the kind of concern I'm expressing here. So personally I'm faced with announcing what seems like both, some degree of an impending apocalypse, and some degree of a non-issue that might go unnoticed. Unless you actually know details about this, it's unreasonable to form a "don't worry about it" opinion. We saw a lot of that with the Y2K issue, which was largely a non-issue only because people enmasse actually took action. In this case, this is potentially a man-made crisis, imposed on us, and if it does proceed as described I think this topic earns some degree of concern.


With that, all I can suggest is to check resources that are important to you. Perhaps extend that to your clients (some of mine may be affected) and to services on which you depend. If you aren't doing this kind of checking for your own interests, who is doing it for you? If you don't have an answer to that, just do a little asking around and see what kind of answers you get. Or wait it out and see what happens. :)


Good luck.

T

Peter McMurray

unread,
Jan 26, 2019, 5:15:20 PM1/26/19
to Pick and MultiValue Databases
Thanks for the heads up.

Brian Speirs

unread,
Jan 27, 2019, 2:16:01 PM1/27/19
to Pick and MultiValue Databases
Thanks, Tony.

I checked my webhost, and it shows MANY failures! I shall make enquiries about how to handle this.

Brian

Wols Lists

unread,
Jan 27, 2019, 4:25:58 PM1/27/19
to mvd...@googlegroups.com
On 27/01/19 19:16, Brian Speirs wrote:
> Thanks, Tony.
>
> I checked my webhost, and it shows MANY failures! I shall make enquiries
> about how to handle this.
>
> Brian

That's bad ...

I picked this up a little before Tony, and read up on curiosity.

Simply put, *ANY* DNS server that abides by the original DNS spec from
yonks ago should carry on working fine. So if you're getting failures,
it means that your DNS servers (or more likely, your ISP's) are not
getting the care and attention they deserve. Either nobody's bothered
upgrading them and they are ancient, or the people who wrote the
software don't care about doing it right.

Given my experience, the latter is quite likely - ages ago in the time
of Windows 3.1, we had Microsoft Mail. And when the Internet rolled out
8-bit SMTP, MS Mail failed pretty completely. Because where the RFC said
"must not", Mail did exactly that ...

For those who don't want to read the gory details, there is a protocol
called EDNS (presumably Extended DNS). And the RFC says you must reply
when asked "what extensions do you support?" An answer of "none" is
perfectly okay.

This flag day says "if you don't bother to reply at all, we will assume
you are dead".

Thing is, EDNS is used for things like suppressing DNS-amplification
DDoS attacks so these "ancient" servers are actually quite dangerous
things to have lying around.

Cheers,
Wol

Tony Gravagno

unread,
Jan 28, 2019, 5:37:43 PM1/28/19
to Pick and MultiValue Databases
I've been looking at this and on the fence for a couple weeks. The name Chicken Little keeps coming to mind and I figured I'd risk some backlash if this didn't turn out to be as bad as it seems.

Unfortunately we're not immune to that mentality in the MV industry, which is one of the reasons why I decided to post here."If it works, don't fix it" is fine for a while. But if you're sitting on train tracks, eventually a train is going to come by and run you over. That's what's happening here. Big players are forcing people who have not been attentive for over a decade to do their jobs.

To be clear, this was no surprise to anyone. DNS Flag Day has been recognized for a couple years but many companies that we rely upon still haven't acted. Maybe they're calling a bluff. I think they're going to lose this hand, and unfortunately we're the ones who suffer.
This reminds me of a recent government shutdown...

Anyway, you folks have seen that this is real. Please check with your providers.
Good luck.
T
Reply all
Reply to author
Forward
0 new messages