GCE: network down every day

2,283 views
Skip to first unread message

Gandalf Corvotempesta

unread,
Mar 18, 2021, 9:15:53 AM3/18/21
to gce-discussion
Hi guys
I have a VM in europe-west3-c that every day, goes down. Digging it further i've found that the network is down, as I can see this in syslog:

GCEGuestAgent[486]: 2021-03-18T13:07:03.4198+01:00 GCEGuestAgent Error main.go:180: Network error when requesting metadata, make sure your instance has an active network and can reach the metadata server.

GCEGuestAgent[486]: 2021-03-18T13:07:03.4212+01:00 GCEGuestAgent Error main.go:183: Error watching metadata: Get http://metadata.google.internal/computeMetadata/v1//?recursive=true&alt=json&wait_for_change=true&timeout_sec=60&last_etag=53bbd693e157547c: dial tcp 169.254.169.254:80: connect: network is unreachable

the only solution is to stop and start the VM, this will fix the network issue up to the next day.

any idea ? I have other VMs hosted on google and this is the first time that something similar happens. It's an official Debian 10 

mbazzi

unread,
Mar 18, 2021, 7:48:44 PM3/18/21
to gce-discussion
Hello,

I recommend you to reinstall/update guest environment packages in the VM as it could be the root cause of the issue.
Before that I recommend you to create a snapshot [1] first of your VM instance before proceeding with the reinstalling process.Then clean out Debian cache and add the correct repositories. SSH to the instance and reinstall guest environment packages by following these steps [2]. After reinstalling services you need to restart the instance .

Gandalf Corvotempesta

unread,
Mar 19, 2021, 6:15:50 AM3/19/21
to gce-discussion
Hello
guest packages are already up-to-date

Why they could be the culprit ?

Gandalf Corvotempesta

unread,
Mar 19, 2021, 9:39:20 AM3/19/21
to gce-discussion
it happened again. every day, from around the 11:00 to around 13:00

Md Mahboob Sayeed Khan

unread,
Mar 19, 2021, 2:00:22 PM3/19/21
to gce-discussion
Hello,

As per the doc [1] please run the below command to see that all the services are enabled or not and share us the output. Sometime it has been observed as per our internal database if this services are disabled then unexpected issue might occurred. 

sudo systemctl list-unit-files \ | grep google | grep enabled

Gandalf Corvotempesta

unread,
Mar 20, 2021, 6:56:03 AM3/20/21
to gce-discussion
Hello

# systemctl list-unit-files | grep google | grep enabled
google-disk-expand.service             enabled  
google-guest-agent.service             enabled  
google-osconfig-agent.service          enabled  
google-shutdown-scripts.service        enabled  
google-startup-scripts.service         enabled  
google-oslogin-cache.timer             enabled  


Ruben (Google Cloud Support)

unread,
Mar 25, 2021, 8:24:14 AM3/25/21
to gce-discussion
Hello,

These are the expected services for a Debian instance, so they should be working as expected. Which leaves me with a couple of questions: does any script in the instance starts to run during or before the downtime? What could you see in both the Cloud logging logs and the  serial logs?

Have a nice day!

Reply all
Reply to author
Forward
0 new messages