eth1 loses internet connection after google_network_daemon restart

[William Sharlow]

I'm running Google Cloud, Centos 7, and in order to get two separate external IP addresses, I need to configure two interfaces (eth0 and eth1) as outlined here:

https://cloud.google.com/vpc/docs/create-use-multiple-interfaces

In the Configuring Policy Routing section, they state to run these commands:

sudo ifconfig eth1 192.168.0.2 netmask 255.255.255.255 broadcast 192.168.0.2 mtu 1430
sudo echo "1 rt1" | sudo tee -a /etc/iproute2/rt_tables # (sudo su - first if permission denied)
sudo ip route add 192.168.0.1 src 192.168.0.2 dev eth1
sudo ip route add default via 192.168.0.1 dev eth1 table rt1
sudo ip rule add from 192.168.0.2/32 table rt1
sudo ip rule add to 192.168.0.2/32 table rt1

Which for my configuration works out to:

sudo ifconfig eth1 10.140.0.2 netmask 255.255.255.255 broadcast 10.140.0.2 mtu 1430
sudo echo "1 rt1" | sudo tee -a /etc/iproute2/rt_tables # (sudo su - first if permission denied)
sudo ip route add 10.140.0.1 src 10.140.0.2 dev eth1
sudo ip route add default via 10.140.0.1 dev eth1 table rt1
sudo ip rule add from 10.140.0.2/20 table rt1
sudo ip rule add to 10.140.0.2/20 table rt1

I've managed to make this permanent too by configuring route-eth1,  so eth1 comes up properly upon reboot. However, when google_network_daemon is restarted, I need to run the following manually.

 ip route add default via 10.140.0.1 dev eth1 table rt1

Any way to solve this so when google_network_daemon is restarted, eth1 is configured properly?

[dcava...@google.com]

The current possibility is that the NetworkManager is taking down the secondary interfaces and routes even though we are not using it to manage them. Restarting works because the network setup that we do happens again on boot. NetworkManager is the default system in RHEL/CentOS 7.

What is supposed to happen is that we set up the DHCP client to bring up the other interfaces (we write a dhclient config file for most distros) and then the DHCP client maintains the interface over time. So far this works correctly in other cases.

Running these commands and pay attention to the BOOTPROTO line:

$ cat /etc/sysconfig/network-scripts/ifcfg-eth1

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

As a workaround, updating ifcfg-eth1 to set BOOTPROTO=dhcp should fix the issue.

A permanent solution is being considered by our Network Engineers, but there is no date for going on production.

[William Sharlow]

I'm been searching for answers to this problem off and on for months now and one post suggested changing BOOTPROTO and I did, Here is my current configuration:

# Added by Google.
#BOOTPROTO=none
#DEFROUTE=no
#DEVICE=eth1
#IPV6INIT=no
#NM_CONTROLLED=no
#NOZEROCONF=yes
#TYPE=Ethernet
#MTU=1460
# From ifcfg-eth0
BOOTPROTO=dhcp
DEVICE=eth1
ONBOOT=yes
TYPE=Ethernet
#DEFROUTE=yes
DEFROUTE=no
#PEERDNS=yes
#PEERROUTES=yes
#DHCP_HOSTNAME=localhost
IPV4_FAILURE_FATAL=no
NAME="System eth1"
MTU=1460
PERSISTENT_DHCLIENT="y"
ZONE=trusted
NM_CONTROLLED=no

[Larbi (Google Cloud Support)]

So, did the BOOTPROTO=dhcp fix your issue?

LetBOOTPROTO when set to dhcp than the netwoking will us ethe interface and assign it an IP

[William Sharlow]

The eth1 comes up as expected during a reboot, but then again, I run a script to load the roots. The problem come into play when google's network daemon restarted as in

# systemctl restart google-network-daemon

In this case eth1 fails to come up properly and I have to run:

# ip route add default via 10.140.0.1 dev eth1 table rt1

I found this and it appears to be related:

https://github.com/GoogleCloudPlatform/compute-image-packages/issues/475

While I'm slowly becoming familiar with network interfaces, I dont want to keep trying solutions - given the amount of time I've already tried to solve this problem. Certainly, if someone knows how to solve it, I will give it a try but nothing's worked to date.

[Larbi (Google Cloud Support)]

Hi,


As already updated, a permanent solution is being considered by the Network Engineers, they are adding the systemd dependency to restart the network daemon when networking is restart, But unfortunately , we don't have an ETA when it will be available on the production.


The only temporary solution is to use the workaround as suggested.

[William Sharlow]

Unfortunately, the suggested workaround did not work in my case. eth1 comes back after a reboot, but I believe that is because I call a script that loads the routes and rules indicated in my original post. It doe not come back when google-network-daemon is restarted - even after trying the suggested BOOTPROTO=dhcp 

Bill

[William Sharlow]

Again, early this morning the server runs yum, resulting in...

Oct 25 04:09:46 host network-setup: INFO Ethernet interfaces: ['eth1'].

Oct 25 04:09:46 host network-setup: INFO Enabling the Ethernet interfaces ['eth1'].

This disconnects eth1 from the internet until I wake up and run:

ip route add default via 10.140.0.1 dev eth1 table rt1

I have tried adding 

default 10.140.0.1 dev eth1

10.140.0.0/20 via dev eth1

default via 10.140.0.1 dev eth1 table rt1

to route-eth1

Can someone please explain how I can get eth1 to work after /etc/cron.daily/0yum-daily.cron runs? I cannot imagine I am the only one that has this problem.

[Digil (Google Cloud Platform Support)]

Hello William,

This seems like a defect in the product. For further investigation, I would recommend you to open a issue-report using this link (while opening the report try to include all the necessary details including the repo steps and error logs ).

In future, if you came across with such defects(or if you want to request for a new feature) with Google Cloud platform and/or its connected resources you could use this help center article for reporting it.