Resolv.conf on GCE boxes now aggressively being replaced while instance is running

[Doug T]

Not sure if there is a new change for newly built instances or newly restarted instances, but i've seen GCE instances now aggressively reverting resolv.conf files. We use a custom resolv.conf in our domain, but it gets overridden by gce defaults seemingly on an new very frequent basis. I'm unable to find the source of the replacement, I've changed files in dhclient.conf and looked through many of the google scripts. Most appear to be startup scripts, which is not the issue in this case as it occurs on instances that have been up for some time. My 2 main questions:

1. What controls the replacement of the resolv.conf? 

2. How can I customize or replace the behavior that google is attempting to make? 

3. Is this a recent change that has occurred on gce side, and if so, how can we track these types of changes? 

Thanks

[George]

Hello Doug,

I will post the answers inline:

On Monday, February 22, 2016 at 11:15:02 AM UTC-5, Doug T wrote:

Not sure if there is a new change for newly built instances or newly restarted instances, but i've seen GCE instances now aggressively reverting resolv.conf files. We use a custom resolv.conf in our domain, but it gets overridden by gce defaults seemingly on an new very frequent basis. I'm unable to find the source of the replacement, I've changed files in dhclient.conf and looked through many of the google scripts. Most appear to be startup scripts, which is not the issue in this case as it occurs on instances that have been up for some time. My 2 main questions:

1. What controls the replacement of the resolv.conf? 

   "/etc/resolv.conf"  is not managed by a Google service. It respects whatever the DHCP server is giving it.

2. How can I customize or replace the behavior that google is attempting to make? 

Depending on the distro, telling the service or daemon responsible for managing resolv.conf to add in whatever other DNS entries you need is the way to do this. That is likely either resolvconf or Network Manager. Both have their own ways of doing this.

 

3. Is this a recent change that has occurred on gce side, and if so, how can we track these types of changes? 

 

The GCE change in question is that DHCP leases now refresh every 24 hours.

Some relevant docs from Ubuntu: http://askubuntu.com/questions/130452/how-do-i-add-a-dns-server-via-resolv-conf

Some other relevant docs from Debian: https://wiki.debian.org/NetworkConfiguration#Defining_the_.28DNS.29_Nameservers

I hope this helps,

Sincerely,

George

Thanks

[Manish Dalwadi]

Hello,

Compute Engine's DHCP behavior is documented here:

https://cloud.google.com/compute/docs/networking#internal_dns_and_resolvconf

Thanks,

-Manish

[Matt Lesko]

Hi Doug,

If you're running CentOS7, as I am, you'll find that you need to override the /etc/resolv.conf fix through NetworkManager. Other cloud providers, and our internal CentOS7 servers, do not use NetworkManager, which gave me a bit of a chase before figuring out the solution.

Add:

[main]
dns=none

to either /etc/NetworkManager/NetworkManager.conf or a file in /etc/NetworkManager/conf.d/.

Of course, if your instance does not use NetworkManager, you'll need to override it in a different way - typically on other CentOS images in /etc/dhcp/dhclient-enter-hooks, with the contents:

#!/bin/sh
make_resolv_conf () { : }

-- Matt

[Jon Anslow]

Hi Matt,

I am also using CentOS 7 with NetworkManager. I added the dns=none entry to /etc/NetworkManager/NetworkManager.conf

I then rebooted the box and I am no longer able to ssh to it. Is there something I have missed? another setting I need to make?

Regards,

Jon