Google deletes preemptible instances freeely

[dmitriy...@gmail.com]

Hello guys,

As I understand, preemptible instances are shut down every 24 hours and can also be shut down at any time before it. 

I'm just making sure that I understand the service correctly.

Now... what happened. 

I had a 3 instances group running fine, then another minute I had my app send me dead letters, so I jumped in to investigate.

All 3 instances were up and running but their hard drives were wiped completly, hard drives were set to be deleted with the instance. 

So what happened is seems like that all 3 of my instances were deleted and recreated from scratch, that's why the disks were reset to their virgin state. 

Okay... I didn't know that there's such a slight difference between shutting down and a complete wipe... but okay. 

Let's go on. 

I've recovered that 3 node cluster, and this time I've set hard drives to not get deleted when the instance is deleted. 

And guess what? 1 hour after it was up and running... I've got a node deleted completly :) The whole instance. 

Glad that this time I had my disk saved, so I was able to bootstrap a new node right away. 

Is this acceptable behavior for such a service and for such an instances?

[dmitriy...@gmail.com]

Update, 2 more nodes got removed completly. 

That bumps it to 6 total. 

[Faizan (Google Cloud Support)]

Hello Dimitriy,

The regular behavior of preemptible instances is that they can last for 24 hours however, they may be terminate anytime due to system events in different zones. Once these instances are shutdown they will show in your project as "Terminated" state. The disk attached to these instances will be accessible in order to retrieve your data from them. This information is documented on this link.

With that said, have you tried creating instances in different zones to see if you experience the same behavior ?

Faizan

[dmitriy...@gmail.com]

пятница, 26 июня 2015 г., 22:20:02 UTC+3 пользователь Faizan (Google Cloud Support) написал:

Problem is that they aren't terminated anymore, they are completely removed from my project and of course they don't show there (cause well... they are removed). And it isn't possible to recreate them using the same instance group because... again... they are removed. 

That started like 2-3 days ago and it was ok before that. 

[Paul Nash]

Hello Dimitriy,

Are you using Managed Instance Groups to create these instances? The behavior of Managed Instance Groups is that a preemption will be viewed as an instance termination just as if the machine had "crashed". It assumes the instance is "unhealthy" and so it deletes it and creates a brand new instance to replace it, in order to maintain the group at the programmed size. This behavior is rooted in the fact that Managed Instance Groups currently assume that the instances are stateless and replaceable, as they would be for web services, for example.

At present, the only way to avoid this behavior would be to create the instances yourself through the command line or web console.

However, note that the fundamental reason behind this is most likely that your instances were preempted earlier than the 24h maximum duration. As to why this would change, it may be that you simply weren't getting preempted very often until recently. As customer usage patterns and capacity changes over time, preemption behavior also varies. So if you create the instances without instance group manager, you will still need to handle the case where your instances are preempted.

You can inspect the sequence of events related to an instance by querying the operations log in the GCE API (for example, "gcloud compute operations list", and you probably want to filter by the instance name you're interested in).

HTH,

-Paul

[Franc.J.]

Hello,

On a related tangent, my Server 2012 R2 instances created through the Developers Console are failing almost immediately.

All programs installed on my instance were able to be installed on Google Compute Engine Compute Instances, however now that the instance is nearly complete, VM Compute instances created through the Developers Console either fail within an hour and even fail within ten minutes.


Naturally, public cloud is not private cloud, what surprises me is that I've been able to build a Compute Image completely inside GCE that is now unuseable.


It doesn't appear completely clear to me that this would have happened with other Public Cloud compute vendors.


Thanks,

Franc.J.

[Alex Gaysinsky]

Hi Franc,

I understand you are having issues with Windows 2012R2. Could you please reply privately to my email gaysinsky at google.com

with the project and instance names.

Thanks,

--Alex

--
© 2014 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/38658101-ac90-479f-a1fc-2c762743384e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex Gaysinsky | Product Manager | gays...@google.com | 310-406-4298