ssh keys disappear from the instance

6,465 views
Skip to first unread message

Manuele Simi

unread,
Apr 4, 2016, 5:04:04 PM4/4/16
to gce-discussion
Hi,

I am new to google cloud engine. I have created a new VM instance in my account successfully. Now, I want to grant access to the instance to users who don’t have a google account. For that, I add their public ssh keys to my frontend node (in .ssh/authorized_keys) and this way they are able to submit jobs to my GE cluster (installed on the instance) through SSH & SFTP. However, after a while (typically a few hours), it seems that their keys are removed from the node and they can’t connect to the instance anymore. Only my credentials (also stored at project level) are still valid.
My VM is created with an ubuntu distribution, if that matters.

Please advise if this is an expected behavior or am I doing something wrong.

Thanks!
manuele

Faizan (Google Cloud Support)

unread,
Apr 5, 2016, 5:44:12 PM4/5/16
to gce-discussion
Hello Manuele,

I was not able to reproduce the behavior you have mentioned on my test Ubuntu GCE instance (created with ubuntu-1510-wily-v20160329 image). The ssh keys manually added to authorized_keys file were not removed. With that said, can you provide me with your steps including the image used to create your Ubuntu instance to try reproduce the issue again.

You also can manage the ssh access through project or instance metadata. You can refer to this link for more information and steps.

I hope that helps.

Faizan

Manuele Simi

unread,
Apr 5, 2016, 8:56:46 PM4/5/16
to gce-discussion
Hi Faizan,

Thanks for taking the time to investigate the issue. I'm adding here more details to see if we can spot the problem.

First, I'm creating my GCE instance with elasticluster. It should not be the reason of the problem because the instance is successfully created and it works very well. I'm using the ubuntu-1204-precise-v20160114 image. The command I use is:

elasticluster start --name $CLUSTER_NAME $NAME


Once the GE instance is up, I append an ssh key to /home/ubuntu/.ssh/authorized_keys either manually or with a bash fragment like this:

KEY="public key goes here"

elasticluster ssh $CLUSTER_NAME <<  EOF

echo "$KEY" >>  ~/.ssh/authorized_keys

EOF 

So far, everything is OK. The owner of the private key can submit jobs to the GE on the image via SSH/SCP/SFP. Then, without doing anything beside connecting via SSH, the key disappears. If I log into the frontend node, the authorized_keys has only keys preceded by the comment #Added by Google.  All the other ones I appended have been wiped out. Also, the file's timestamp says that it was modified few minutes before.

I'm aware that I can add/remove ssh access from the console, but that's not what we need for our project.

Thanks again!
manuele

Faizan (Google Cloud Support)

unread,
Apr 7, 2016, 5:12:52 PM4/7/16
to gce-dis...@googlegroups.com
Hello Manuele,

Thank you for providing additional information. I was able to reproduce this issue.

The behavior you are seeing on the instance is due to the Google Daemons which are responsible for managing user accounts and ssh public keys using the metadata server. Due to these daemons user accounts and ssh keys are synced with metadata server and the VM instance. As such, the recommended method to manage ssh keys is by using metadata server. You can use instance metadata to add ssh keys which only will be applied to that instance and will not be propagated to all the instances in the project. The process is documented on this link.

In case you wish to manually manage the ssh keys you need to disable these daemons. This practice is not recommend as this might break other functionalities of GCE instance. For more information on these Google daemons you can refer to this link.

Manuele Simi

unread,
Apr 7, 2016, 9:25:57 PM4/7/16
to gce-discussion
Hi Faizan,

What you describe is exactly what we observed and it matches with our hypothesis that there is some background tasks that syncs the keys. We will think about how to work around this, but at least the mystery of the missing key is solved! 

Many thanks.
manuele

Gaurav Vij

unread,
Dec 26, 2019, 9:34:35 AM12/26/19
to gce-discussion
Hi Manuele, 

Were you able to figure out a workaround on this issue? I have the same issue at my end and can't possibly seem to get around this.. I need to programmatically add keys. Not manually through the console..

Alexandre Duval-Cid

unread,
Dec 26, 2019, 10:43:56 AM12/26/19
to gce-discussion
Hey,

you can disable the Google Account  Deamon [1], Although it's not recommended, the ramifications of doing so are somewhat unknown. I would recommend using OS login as a possible alternative for SSH access management [2], here are some instructions on the topic [3]

Kevin Parker

unread,
Mar 25, 2020, 5:21:21 PM3/25/20
to gce-discussion
Yes, we have the same issue. Google cloud doesn't seem to be the solution for us. It's way too new, too much of a headache. AWS is laid out much better and though despite its initial learning curve overall is a better product. Apply a firewall on GCM? 18 seconds. Can't delete a nic. Yes, you heard that. It's just they built out so much expecting people to use all these interesting features and they didn't get the basics right.

 For me even though I add the keys in the GCM web console, they still disappear once I login and authroized_keys is wiped blank. 

Justin Reiners

unread,
Mar 25, 2020, 5:31:51 PM3/25/20
to Kevin Parker, gce-discussion
Thats strange, we use custom keys and users across our fleet, and i've yet to have a key deleted. I do not use OS login though, I've had ~140 machines for a few years in GCP now.

On Wed, Mar 25, 2020 at 4:21 PM Kevin Parker <kpa...@near.shop> wrote:
Yes, we have the same issue. Google cloud doesn't seem to be the solution for us. It's way too new, too much of a headache. AWS is laid out much better and though despite its initial learning curve overall is a better product. Apply a firewall on GCM? 18 seconds. Can't delete a nic. Yes, you heard that. It's just they built out so much expecting people to use all these interesting features and they didn't get the basics right.

 For me even though I add the keys in the GCM web console, they still disappear once I login and authroized_keys is wiped blank. 

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/3b874316-bf69-4d68-8a14-87991678bd96%40googlegroups.com.

Kevin Parker

unread,
Mar 25, 2020, 5:50:06 PM3/25/20
to Justin Reiners, gce-discussion
Well other people on here have the same issue. It's frustrating to spend hours with gcm erasing vm files and expect me to pay for this kind of experience...

Justin Reiners

unread,
Mar 25, 2020, 6:01:23 PM3/25/20
to Kevin Parker, gce-discussion
I'm not saying it's not happening, just haven't had that issue myself (and wanted to point of my lack of OS Login use just in case), are they deleted immediately after login?

anarayanaswamy

unread,
Mar 25, 2020, 9:50:32 PM3/25/20
to gce-discussion
Please note Enabling OS Login on instances disables metadata-based SSH key configurations on those instances [1]. Disabling OS Login restores SSH keys that you have configured in project or instance metadata.[2]


On Wednesday, March 25, 2020 at 6:01:23 PM UTC-4, Justin Reiners wrote:
I'm not saying it's not happening, just haven't had that issue myself (and wanted to point of my lack of OS Login use just in case), are they deleted immediately after login?

On Wed, Mar 25, 2020 at 4:49 PM Kevin Parker <kpa...@near.shop> wrote:
Well other people on here have the same issue. It's frustrating to spend hours with gcm erasing vm files and expect me to pay for this kind of experience...

On 25 March 2020 5:31:43 pm Justin Reiners <jus...@hotlinesinc.com> wrote:

Thats strange, we use custom keys and users across our fleet, and i've yet to have a key deleted. I do not use OS login though, I've had ~140 machines for a few years in GCP now.

On Wed, Mar 25, 2020 at 4:21 PM Kevin Parker <kpa...@near.shop> wrote:
Yes, we have the same issue. Google cloud doesn't seem to be the solution for us. It's way too new, too much of a headache. AWS is laid out much better and though despite its initial learning curve overall is a better product. Apply a firewall on GCM? 18 seconds. Can't delete a nic. Yes, you heard that. It's just they built out so much expecting people to use all these interesting features and they didn't get the basics right.

 For me even though I add the keys in the GCM web console, they still disappear once I login and authroized_keys is wiped blank. 

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.

Max Illfelder

unread,
Mar 26, 2020, 7:45:09 PM3/26/20
to gce-discussion
Providing some information about how pieces of the guest environment works in the Google Compute Engine guest environment.

Without OS Login enabled users can specify SSH keys in metadata. When a user attempts to log into a VM using a Google provided tool (gcloud, ssh from the browser), a username and SSH key is added to metadata. The guest environment configures a local user account for the user if one does not already exist, and the authorized keys file is populated with the SSH keys. At this point, the user is configured Google managed in the VM. When all SSH keys associated with the user are removed from metadata, the guest environment does its best to ensure the user can no longer log into the VM unless they still have permission; this is done by deleting the authorized keys file. I suspect this is what caused your authorized keys file to be removed. If you are managing your own authorized keys file, it is not recommended that you also use metadata for managing your local user's SSH keys.


When OS Login is enabled, metadata is no longer used to serve SSH keys. OS Login users SSH keys are never stored in the VM (and authorized keys files are not used for these users). Instead, the VM will remotely pull the SSH keys associated with the identity of the user trying to log in (this does not require external internet access). In addition, the login flow will perform an IAM authorization check before allowing the user to log in. This means that with OS Login enabled, you can prevent a user from logging into a VM by removing their IAM permission without needing to change the keys stored in the authorized keys file. This also means that a user with permission can log into any VM using their same SSH key (no authorized keys file config needed).

On Wednesday, March 25, 2020 at 6:50:32 PM UTC-7 anarayanaswamy wrote:
Please note Enabling OS Login on instances disables metadata-based SSH key configurations on those instances [1]. Disabling OS Login restores SSH keys that you have configured in project or instance metadata.[2]


On Wednesday, March 25, 2020 at 6:01:23 PM UTC-4, Justin Reiners wrote:
I'm not saying it's not happening, just haven't had that issue myself (and wanted to point of my lack of OS Login use just in case), are they deleted immediately after login?

On Wed, Mar 25, 2020 at 4:49 PM Kevin Parker <kpa...@near.shop> wrote:
Well other people on here have the same issue. It's frustrating to spend hours with gcm erasing vm files and expect me to pay for this kind of experience...

On 25 March 2020 5:31:43 pm Justin Reiners <jus...@hotlinesinc.com> wrote:

Thats strange, we use custom keys and users across our fleet, and i've yet to have a key deleted. I do not use OS login though, I've had ~140 machines for a few years in GCP now.

On Wed, Mar 25, 2020 at 4:21 PM Kevin Parker <kpa...@near.shop> wrote:
Yes, we have the same issue. Google cloud doesn't seem to be the solution for us. It's way too new, too much of a headache. AWS is laid out much better and though despite its initial learning curve overall is a better product. Apply a firewall on GCM? 18 seconds. Can't delete a nic. Yes, you heard that. It's just they built out so much expecting people to use all these interesting features and they didn't get the basics right.

 For me even though I add the keys in the GCM web console, they still disappear once I login and authroized_keys is wiped blank. 

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages