Persistent disks and backups

[Gandalf Corvotempesta]

Hi to all
I would like to create an instance as much similiar as possible to a standard virtual machine so that no data loss happens in case of shutdown, stop, poweroff or anything else

Should i add a persistent disk to the instance?
Is this disk mounted somewhere in the instance or the whole disk, including the operating system, will be placed on the persistent disk?

Is traffic to and from the persistent disk billed?

How can i create a daily backup for the whole instance?

[Carlos (Cloud Platform Support)]

Hi Gandalf, 

Every GCE instance must have at least one root disk.  This disk will contain the OS that allows the VM  to boot. You can choose to add additional disks to the same VM.  Most VMs types will allow you to have up to 16 PDs and 64 TB of total space. It is your choice  whether to have you data and OS in different disks, you must configure your application to write/read from the appropiate disk. 

Traffic to your PD counts only against the egress throughput cap. You will be billed for the cost of your PD but not for the traffic that goes or comes from it. To attach a PD to your VM, both resources must be in the same zone.  Egress to the same zone using the internal IP  (which is the case of PD attached to a VM) is not charged.

To create backups of your disks you can use snapshots. Ideally you would stop your VM to ensure that the snapshot is consistent with the disk state.  If this is not possible you can follow the recommendations posted here. 

You can certainly create your own script to automate the snapshots. To do that you can call Cloud SDK from a script that can be triggered by a cron job.  

[Gandalf Corvotempesta]

Il giorno martedì 29 novembre 2016 17:52:42 UTC+1, Carlos (Cloud Platform Support) ha scritto:

Hi Gandalf, 

Hi and thank you for the response

 

Every GCE instance must have at least one root disk.  This disk will contain the OS that allows the VM  to boot. You can choose to add additional disks to the same VM.  Most VMs types will allow you to have up to 16 PDs and 64 TB of total space. It is your choice  whether to have you data and OS in different disks, you must configure your application to write/read from the appropiate disk. 

That's ok, but If I understood properly, the only way to preserve data after stopping the VM is by using a persistent disks. When adding the persistent disks, the root disk would be moved over it or should I consider the persistent disk as an extra disk to mount somewhere in my VM ?

 

Traffic to your PD counts only against the egress throughput cap. You will be billed for the cost of your PD but not for the traffic that goes or comes from it. To attach a PD to your VM, both resources must be in the same zone.  Egress to the same zone using the internal IP  (which is the case of PD attached to a VM) is not charged.

Ok, so, to keep it simply, traffic to/from the persistent disks is not charged.

 

What I would like to create is a VM similiar to any other (non cloud, like DigitalOcean or similiar) VM offer, where my data are safe even after shutting down the VM. Is external disk the solution ?

[Carlos (Cloud Platform Support)]

Hi Gandalf, 

In a very simplistic way you can think of your VM as a regular computer. It will have RAM which is a form of volatile storage. Information in RAM will be lost when your VM is stopped.  Therefore applications that have information in RAM need to write it to a hard drive (in your VM will be the Persistent Disk)  when shutting down. In that way the application state and data is preserved after your VM is stopped or rebooted.  

Your VM, as a regular computer, will always have RAM and at least one PD containing the operating system (also called root disk). This is not an option instead is a must have. One of the big differences in the Cloud Environment is that the PD might not be physically connected to the VM . The fact that the PD is not physically attached to the hardware is transparent to the server.  The logic in the cloud infrastructure will take care of it. The disk just need to be attached in order to be recognized by the server. The VM does not know that it is not physically connected to a disk.

As a summary, PD will preserve your data when the VM is stopped.  There are other durable options to preserve your data like Google Cloud Storage. In that case, you must configure your applications to access their data from a bucket. Now, even if you decide to use GCS, you will always need to have root PD on your VM. Finally,  traffic to the PD is never billed. Traffic from GCE VMs to Cloud Storage might be charged.

I hope this helps.

[Gandalf Corvotempesta]

I know what a VPS is.

I've read somewhere on Google Docs (i don't remember exactly where) that root disk is lost when the instance is stopped, like for local SSD.

Is this true ? If I don't need extra space, can I safely only use the root disk coming with the VM or i'm at risk to loose data ?

In example, a very simple DNS server only require 500MB for MySQL database. This can be placed on the VM root disk, right ? Are my data safe ?

About pricing: let's assume a simple infrastructure made by:

- Google Cloud DNS (one zone with 1.000.000 queries)

- 1 Compute

- 1 Google SQL (replicated)

I'll pay for 1 zone with 1.000.000 queries, 1 compute instance for each running hours, 1 (or 2?) Google SQL based on stored data, plus egress network traffic coming out from the compute, right?

Anything else ?

[Carlos (Cloud Platform Support)]

Since your root disk is a persistent disk, information stored on it is durable. That means root disks and in general PD will keep your data even after you have stopped the VM. The size of your root disk is defined when you create the VM, but you can always resize it. Persistent disks have built-in redundancy to protect your data against equipment failure. In any case, it is very important that you define a backup policy that serves your business needs.

In regards to the billing part I suggest using the online calculator to estimate your costs. Communication between a GCE VM and a Cloud SQL Server will take place via the external IP and therefore generate egress traffic. 

[Gandalf Corvotempesta]

Il giorno venerdì 2 dicembre 2016 18:28:17 UTC+1, Carlos (Cloud Platform Support) ha scritto:

Since your root disk is a persistent disk, information stored on it is durable. That means root disks and in general PD will keep your data even after you have stopped the VM. The size of your root disk is defined when you create the VM, but you can always resize it. Persistent disks have built-in redundancy to protect your data against equipment failure. In any case, it is very important that you define a backup policy that serves your business needs.

Seems that resizing the root disk is the way to go to archieve a VPS-like machine. 

[Paul Nash]

Gandalf, you may have heard that a PD can be *deleted* IF the instance itself is DELETED. This is not the same as simply stopping it. Since we're doing analogies here, it's like the difference between powering down a real computer and throwing it in the landfill. :)

By default in our API, a PD is deleted when the Instance is deleted, because otherwise people tend to forget they have those disks there and they go on paying to store them, whether they needed that data or not. When the instance is just shutdown ("stopped" in our terminology) the PD is still there.

I think Carlos answered the rest of your items, I just wanted to point out the possible source of the rumor you mentioned about losing the data.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/575800e8-9df1-4ad7-a721-b5b04e1c68b0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Paul R. Nash |

Product Manager, Compute Engine |

paul...@google.com |

206-876-1620

[Gandalf Corvotempesta]

Il giorno martedì 6 dicembre 2016 01:02:36 UTC+1, Paul Nash ha scritto:

Gandalf, you may have heard that a PD can be *deleted* IF the instance itself is DELETED. This is not the same as simply stopping it. Since we're doing analogies here, it's like the difference between powering down a real computer and throwing it in the landfill. :)

By default in our API, a PD is deleted when the Instance is deleted, because otherwise people tend to forget they have those disks there and they go on paying to store them, whether they needed that data or not. When the instance is just shutdown ("stopped" in our terminology) the PD is still there.

This is clear.

So, only a local SSD is lost when stopping the image, right ?

Root disks (are these SSD?) are persistent.

Any differences, in data availability, between root disks and additional persistent disks or are the same ?

[Scott Van Woudenberg]

Hi Gandalf,

Local SSD would be lost if you stop a VM that has Local SSD, which is why we try to prevent users from doing that; the UI, CLI, and API will all return an error, but if the host machine beneath your VM crashes or you do a 'shutdown -P' from within the guest you'll lose the contents of your Local SSD.

Boot/root disks are always either PD HDD or PD SSD and thus persist across stop/start, VM/host crashes, etc.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/e37e78fa-0a75-4d57-97cc-a35da8a1b18d%40googlegroups.com.

[Gandalf Corvotempesta]

Il giorno martedì 6 dicembre 2016 18:05:48 UTC+1, Scott Van Woudenberg ha scritto:

Boot/root disks are always either PD HDD or PD SSD and thus persist across stop/start, VM/host crashes, etc.

Perfect. So a GCE instance can be used like any other non-cloud VPS. Just sping up a new instance and I'm ready to go

[Paul Nash]

Exactly! Good luck. :)