Google Daemon not automatically starting on Compute Engine Instance

[Reporting User]

Hey there!

I have a 6 core 26GB memory VM running Ubuntu 16.04.4 LTS. It's been chugging along fine for months--I do regular updates/maintenance. However, some scripts got out of control and had it running at 100% CPU usage for almost two entire weeks. Sometime during that time I lost my ability to connect to it through Filezilla/SFTP. No other changes were made to the firewall/networking, ssh/metadata keys, etc. SSH using the same key that Filezilla used (although Filezilla converts it to a .ppk) still worked normally. I got the scripts under control, and reset the instance. It's run fine for a few days since (besides still not being able to connect with Filezilla).

I tried to troubleshoot the Filezilla issue today, and ended up creating a new SSH key to try to start from scratch. I added it to the instances metadata > SSH keys just like I did the last one, but it never picked it up. I know this b/c I was never able to connect to it using the new ssh key, but when I tried with the old one (that had been deleted from metadata > ssh keys...and therefore should no longer work) I was still able to connect. To verify I checked $HOME/.ssh/authorized_keys and sure enough the only one in there was the old one.





After trying to connect through SSH in my browser from GCE console (unsuccessfully as well) I check out the help at https://cloud.google.com/compute/docs/ssh-in-browser#ssherror.




After reading through that I checked my serial port logs and couldn't find accounts-from-metadata: anywhere, so I'm pretty sure my Google Daemon is not starting up (which would explain why my instance never updates the metadata/ssh keys). I reset it again for good measure, but am still having the same issue.




Is there a way I can manually start up/repair the Google Daemon?




Thanks!


[Reporting User]

Some additional data:

After comparing running processes to a different VM that is working normally, I do not see these processes running on the VM with issues:

ssh-agent -s

/usr/bin/python3 /user/bin/google_ip_forwarding_daemon

/usr/bin/python3 /user/bin/google_clock_skew_daemon

/usr/bin/python3 /user/bin/google_accounts_daemon

[Reporting User]

Also - Disk space is not an issue. I have plenty of room.

On Wednesday, July 18, 2018 at 7:52:39 AM UTC-5, Reporting User wrote:

[Reporting User]

Additionally, the following scripts are missing from /user/bin/

google_accounts_daemon
google_authorized_keys
google_clock_skew_daemon
google_instance_setup
google_ip_forwarding_daemon
google_metadata_script_runner
google_network_setup
google_oslogin_control

On Wednesday, July 18, 2018 at 7:52:39 AM UTC-5, Reporting User wrote:

[Reporting User]

Also this service is missing (not running):

google-network-setup.service

On Wednesday, July 18, 2018 at 7:52:39 AM UTC-5, Reporting User wrote:

[Reporting User]

After some more research it appears that the Guest Environment is not installed/working. So I reinstalled it according to https://cloud.google.com/compute/docs/instances/linux-guest-environment#in_place.

Fixed some of the issues. It picked up the new ssh_key from my console's metadata page, but Filezilla is still not able to connect, and a couple pieces of the Guest Environment don't appear to have been initialized/running.

From here: https://cloud.google.com/compute/docs/instances/linux-guest-environment#wgei_console_logs

I'm missing these from the console logs:

Started Google Compute Engine IP Forwarding Daemon
Started Google Compute Engine Startup Scripts

And the following loaded services:

google-ip-forwarding-daemon.service enabled

Why are these scripts/services not starting/running? I'm suspicious they are the reason (or caused by the same thing) that I cannot SFTP to the VM.

Thanks

Started Google Compute Engine Startup Scripts

On Wednesday, July 18, 2018 at 7:52:39 AM UTC-5, Reporting User wrote:

[Fady (Google Cloud Platform)]

In Ubuntu 16.04 LTS both “Google Compute Engine Network Setup” ( seems only on RHEL and CentOS), and “Google Compute Engine IP Forwarding Daemon” are not part of the guest environment. However, “Google Compute Engine Network Daemon” is. You may verify all enabled services by running the following command:

$ sudo systemctl list-unit-files \
> | grep google | grep enabled

google-accounts-daemon.service             enabled
google-clock-skew-daemon.service           enabled
google-instance-setup.service              enabled
google-network-daemon.service              enabled
google-shutdown-scripts.service            enabled
google-startup-scripts.service             enabled

As for the startup script service, it does not keep a running a process in the background after booting up. The script’s sole purpose is to run startup scripts attached to the metadata when booting up, and not related to SSH. Hence, if it is missing in the list above, you could try enabling it as to automatically start on boot-up :

sudo systemctl enable google-startup-scripts.service

Furthermore, if you have a startup script that needs to run immediately you can use:

sudo systemctl start google-startup-scripts.service

On the other hand, to check for processes needed to SSH, you may verify if  “network daemon” and “accounts daemon” are running:

sudo ps aux | grep google

That said, and from your description (after reinstalling the environment), and assuming you are able to SSH from console or using gcloud commands, the managed key pair are being propagated which indicates that the guest environment is working fine.

Getting back to your current issue, I suggest first eliminating the possibility of having a local issue (SFTP client or your computer’s firewall, etc.) by testing your SFTP client with a different or new instance (e.g f1-micro).

If that is not the case, you may try (with the existing instance) recreating the key pair for a new user, making sure that the same user with the public key are attached to the metadata (project level or instance level). Then, and before using an SFTP client, verify through a third party SSH client (e.g putty) that you are able to SSH using the private key. If that fails, and for further help, please open a private issue tracker, including your project ID, instance name, and all SSH verbosity logs.

[Reporting User]

How embarrassing. Posting my resolution here in case anyone with the same issue stumbles across this.

While I did have a legitimate issue (google daemon not running/collecting metadata updates) that I resolved by reinstalling the guest environment...the issue with Filezilla was something else. I won't go through the countless things I tried b/c it's just way too long and embarrassing. But the problem was this:

Apparently the Host field in Filezilla's Site Manager allows newline characters...

And I guess whatever element I copied my VM's external IP from on the web console pasted with a newline in there, or pasted twice, or somehow didn't delete an old one when it had an ephemeral IP...whatever happened it turns out I had 2 IPs in there separated by a newline.

And you guessed it...that gives you these weird connection issues with Filezilla. Deleted that whole field, put the correct one in there, and we're up and running. Thanks for watching everyone.

On Wednesday, July 18, 2018 at 7:52:39 AM UTC-5, Reporting User wrote: