Couldn't resolve host 'metadata.google.internal'

[Alex Radinsky]

Hi

My VM was properly running few years until some days ago I couldn't reach the web site I hosted there.

I tried ssh but it failed connecting.

I restarted the VM but it not booting and on serial console I see a lot of prints on  accounts-from-metadata, attaching the log. 

What it can be?

Thanks!

[ [36minfo [39;49m] Checking for tables which need an upgrade, are corrupt or were 
not closed cleanly..
curl: (6) Couldn't resolve host 'metadata.google.internal'
Nov 27 13:21:54 (none) google: 
Nov 27 13:21:54 (none) google: No startup script found in metadata.
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: Looking for 'mysql' as: /usr/bin/mysql
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: This installation of MySQL is already upgraded to 5.5.52, use --force if you still need to run mysql_upgrade
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2377]: Checking for insecure root accounts.
Nov 27 13:21:54 (none) /etc/mysql/debian-start[2382]: Triggering myisam-recover for all MyISAM tables
Nov 27 13:21:56 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 27 13:22:01 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 27 13:22:06 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 27 13:22:11 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 27 13:22:16 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

[Alex Radinsky]

Hi

My VM was running for 2 uptime years or even more with no problems.

Several days ago I could reach the web site it hosts.

I tried ssh but it fail connecting, so I restarted the VM and it fails to boot.

Attaching the log from serial console.

I see problems resolving metadata server.

What can be wrong?

Thanks!

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2326]: Upgrading MySQL tables if necessary.

[ [36minfo [39;49m] Checking for tables which need an upgrade, are corrupt or were 

not closed cleanly..

curl: (6) Couldn't resolve host 'metadata.google.internal'

Nov 27 13:21:54 (none) google: 

Nov 27 13:21:54 (none) google: No startup script found in metadata.

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: Looking for 'mysql' as: /usr/bin/mysql

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2330]: This installation of MySQL is already upgraded to 5.5.52, use --force if you still need to run mysql_upgrade

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2377]: Checking for insecure root accounts.

Nov 27 13:21:54 (none) /etc/mysql/debian-start[2382]: Triggering myisam-recover for all MyISAM tables

Nov 27 13:21:56 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:01 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:06 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:11 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:16 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:21 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:26 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:31 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:37 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:42 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

Nov 27 13:22:47 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

[Kamran (Google Cloud Support)]

Hello Alex,

You can enable interactive access to the serial console so you can more easily troubleshoot instances that are not booting properly or that are otherwise inaccessible. For more information please visit Interacting with the Serial Console article.

[Alex Radinsky]

Hi Kamran, 

I can access the serial console, but I can't do nothing, it stuck with Network is unreachable messages, cntl+c doesn't help, can't interrupt this:

Nov 28 10:29:52 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 28 10:29:57 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 28 10:30:02 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 28 10:30:07 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 28 10:30:12 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
Nov 28 10:30:17 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>
^C^C^C^C
Nov 28 10:30:22 (none) accounts-from-metadata: WARNING error while trying to update accounts: <urlopen error [Errno 101] Network is unreachable>

[Karthick (Cloud Platform Support)]

Hello Alex,

I am wondering if the issue could be due to network configuration on the instance. Could you able to create a new instance, mount the disk of affected VM, and see if you can connect?

You can refer to Validate that your disk has a valid file system on how to perform this. However, I would also suggest you to take snapshot of the disk beforehand.