"Remote host identification has changed" warning when ssh into GCP VM.

[Gang Chen]

When I tried to SSH into one of our GCP VM, it suddenly gave me the following warning:

I am now able to connect by deleting the known_hosts entry. 

But should I worry about this message? What should I do to make sure that our machine is still safe? 

Sorry, I am not a DevOp person, if there is any other information you need to know to answer the questions, please let me know. 

[Justin Reiners]

I also had a machine do this to me a few weeks ago myself, I saw no logins or strange things on the instance itself, I'm not sure what the heck caused it.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/96b47a84-7f94-4323-af62-e1895b44fec6%40googlegroups.com.

[Justin Reiners]

I should at least mention it happened to a single machine, of about 200 instances we run on Google between a few accounts.

I also just removed the known_hosts entry, and wasn't able to find anything else strange, I'm not sure what would cause the host key to change, what operating system are you using? For me it's Centos 7

[Declan McArdle]

Live Migration event?

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/CAG9nfHxXzo4Vmc2zqpJeY2sm4-huCaX14Tb_Haw1kw9t_zb7NA%40mail.gmail.com.

[Gang Chen]

Justin,

Thanks for the reply. 

We also have several instances ourselves, and this is the only one has this problem happened, and it is Centos 7. 

To unsubscribe from this group and stop receiving emails from it, send an email to gce-dis...@googlegroups.com.

[Gang Chen]

Declan, 

No, it was holiday period, no one touched the servers. 

However, I do find a HOST ERROR in the log, and the server was restarted by the system. Could this be the reason?

On Thursday, January 9, 2020 at 1:09:01 PM UTC-6, Declan McArdle wrote:

Live Migration event?

On Thu, 9 Jan 2020 at 18:40, Justin Reiners <jus...@hotlinesinc.com> wrote:

I should at least mention it happened to a single machine, of about 200 instances we run on Google between a few accounts.

I also just removed the known_hosts entry, and wasn't able to find anything else strange, I'm not sure what would cause the host key to change, what operating system are you using? For me it's Centos 7

On Thu, Jan 9, 2020 at 11:37 AM Justin Reiners <jus...@hotlinesinc.com> wrote:

I also had a machine do this to me a few weeks ago myself, I saw no logins or strange things on the instance itself, I'm not sure what the heck caused it.

On Thu, Jan 9, 2020 at 11:29 AM Gang Chen <gc...@americandatanetwork.com> wrote:

When I tried to SSH into one of our GCP VM, it suddenly gave me the following warning:

I am now able to connect by deleting the known_hosts entry. 

But should I worry about this message? What should I do to make sure that our machine is still safe? 

Sorry, I am not a DevOp person, if there is any other information you need to know to answer the questions, please let me know. 

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/96b47a84-7f94-4323-af62-e1895b44fec6%40googlegroups.com.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-dis...@googlegroups.com.

[Justin Reiners]

Declan,

A live migration event should never change the hostkey, at least I wouldn't think it would, as it would break a lot of things, 

I did find it strange, maybe some update was installed which regenerated the hostkey, I chalked it up to a dev running a command to regenerate it for some reason but nobody would admit to it lol.

Looking through the logs I don't even remember seeing a reboot. the DC I use is US-CENTRAL1-A

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/8aa57470-75d0-4bc1-bf91-4ef63329907d%40googlegroups.com.

[Gang Chen]

As described here :

"Yes actually, Google Does change your hostkey in some cases.

For example, hostkey will change during host maintenance migrations if that is enabled.

The key will change if any changes are made requiring recreating the instance are needed: something as dumb as setting a scope, adding a service account, or even the lovely setting a description on an instance will force a replacement of a VM and the host key will change."

Can someone confirm this statement? We have host maintenance migration enabled, but that didn't change hostkey previously. 

BTW: we are also on US-CENTRAL1-A.

[Justin Reiners]

All I can say is I've been a $20,000/mo customer for over 3 years now and it's only happened a single time. I've had host migrations many many times over the years and it's never changed the key, except in this instance if that was actually the case. @GOOGLE can you please comment on this if you see it? I think we're genuinely curious what might have happened.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/faeaf996-2a8f-45f2-bf09-fee980ae8f79%40googlegroups.com.

[Gautham (Google Cloud Support)]

 

Hi,

 

The issue appears to be at the OS level not really within the scope of this board, Google groups is meant for architectural and outages based discussions. However, I’ve found some public threads[1][2][3] which explains the same issue that might be helpful.

 

[1] https://stackoverflow.com/questions/20840012/ssh-remote-host-identification-has-changed

[2] https://support.rackspace.com/how-to/ssh-remote-host-identification-has-changed/

[3] https://stackabuse.com/how-to-fix-warning-remote-host-identification-has-changed-on-mac-and-linux/

On Thursday, January 9, 2020 at 4:07:58 PM UTC-5, Justin Reiners wrote:

All I can say is I've been a $20,000/mo customer for over 3 years now and it's only happened a single time. I've had host migrations many many times over the years and it's never changed the key, except in this instance if that was actually the case. @GOOGLE can you please comment on this if you see it? I think we're genuinely curious what might have happened.

On Thu, Jan 9, 2020 at 2:51 PM Gang Chen <gc...@americandatanetwork.com> wrote:

As described here :

"Yes actually, Google Does change your hostkey in some cases.

For example, hostkey will change during host maintenance migrations if that is enabled.

The key will change if any changes are made requiring recreating the instance are needed: something as dumb as setting a scope, adding a service account, or even the lovely setting a description on an instance will force a replacement of a VM and the host key will change."

Can someone confirm this statement? We have host maintenance migration enabled, but that didn't change hostkey previously. 

BTW: we are also on US-CENTRAL1-A.

On Thursday, January 9, 2020 at 11:29:40 AM UTC-6, Gang Chen wrote:

When I tried to SSH into one of our GCP VM, it suddenly gave me the following warning:

I am now able to connect by deleting the known_hosts entry. 

But should I worry about this message? What should I do to make sure that our machine is still safe? 

Sorry, I am not a DevOp person, if there is any other information you need to know to answer the questions, please let me know. 

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.