Warden handling DNS updates

[Brian McClain]

Hey all,

Yet another topic of "here's something I found but I don't know the best way to fix it" :)

In our development setup, in AWS, we're running an internal DNS server besides Route53 for various reasons, but I think the scenario still applies.

We had to recreate the DNS server for a couple of reasons, mainly because the instance being retired and we needed to do some upgrades on it anyways. We spun up the new DNS, did our testing, and updated our DHCP option sets in AWS. Everything was sunshine and rainbows until the old DNS actually terminated. All the DEAs were able to resolve hostnames if I did nslookups on them, but it seemed only the apps in the warden containers themselves were having issues doing lookups. We rebooted one of the apps in question and, voila, it was able to resolve the hostnames.

I had the hunch that the container grabbed the DNS settings at the time of creation, using whatever settings the DEA had at the time. This would explain why containers created prior to the DNS change did not work, but containers created after did.

A quick check, comparing the /tmp/rootfs/etc/resolv.conf files in both an old container and a new container confirmed this

Now the question is, is this a frequent enough situation to warrant a discussion on a fix? Or is the fix simply if you need to update your DNS, part of the process is bouncing all of the applications. I understand this isn't exactly a scenario that someone gets into every single day, and this may be the philosophy of how warden handles DNS settings. Was just curious :)

- Brian

@BrianMMcClain