GCE: network down every day

[Gandalf Corvotempesta]

Hi guys

I have a VM in europe-west3-c that every day, goes down. Digging it further i've found that the network is down, as I can see this in syslog:

GCEGuestAgent[486]: 2021-03-18T13:07:03.4198+01:00 GCEGuestAgent Error main.go:180: Network error when requesting metadata, make sure your instance has an active network and can reach the metadata server.

GCEGuestAgent[486]: 2021-03-18T13:07:03.4212+01:00 GCEGuestAgent Error main.go:183: Error watching metadata: Get http://metadata.google.internal/computeMetadata/v1//?recursive=true&alt=json&wait_for_change=true&timeout_sec=60&last_etag=53bbd693e157547c: dial tcp 169.254.169.254:80: connect: network is unreachable

the only solution is to stop and start the VM, this will fix the network issue up to the next day.

any idea ? I have other VMs hosted on google and this is the first time that something similar happens. It's an official Debian 10 

[mbazzi]

Hello,

I recommend you to reinstall/update guest environment packages in the VM as it could be the root cause of the issue.

Before that I recommend you to create a snapshot [1] first of your VM instance before proceeding with the reinstalling process.Then clean out Debian cache and add the correct repositories. SSH to the instance and reinstall guest environment packages by following these steps [2]. After reinstalling services you need to restart the instance .

[1] https://cloud.google.com/compute/docs/disks/create-snapshots#create_your_snapshot

[2] https://cloud.google.com/compute/docs/images/install-guest-environment#update-guest  

[Gandalf Corvotempesta]

Hello

guest packages are already up-to-date

Why they could be the culprit ?

[Gandalf Corvotempesta]

it happened again. every day, from around the 11:00 to around 13:00

[Md Mahboob Sayeed Khan]

Hello,

As per the doc [1] please run the below command to see that all the services are enabled or not and share us the output. Sometime it has been observed as per our internal database if this services are disabled then unexpected issue might occurred. 

sudo systemctl list-unit-files \ | grep google | grep enabled

[1] https://cloud.google.com/compute/docs/images/install-guest-environment?hl=en#wgei_services

[Gandalf Corvotempesta]

Hello

# systemctl list-unit-files | grep google | grep enabled

google-disk-expand.service             enabled  

google-guest-agent.service             enabled  

google-osconfig-agent.service          enabled  

google-shutdown-scripts.service        enabled  

google-startup-scripts.service         enabled  

google-oslogin-cache.timer             enabled  

[Ruben (Google Cloud Support)]

Hello,

These are the expected services for a Debian instance, so they should be working as expected. Which leaves me with a couple of questions: does any script in the instance starts to run during or before the downtime? What could you see in both the Cloud logging logs and the  serial logs?

Have a nice day!