Snapshot creation time; after how much time after start create snapshot it is safe to unfreeze fs

[Eric Aarts]

We host ERP applications on GCE on Ubuntu 14.04.

The underlaying database is MySQL.

Slaves in other CGE regions and/or EC2 real-time replicate the databases.

With a script start from cron at 2AM we create snapshots.

The primary script is a Perl scripts, that:

-first issues "flush tables with read lock" to MySQL,

-then calls a bash-script for sync and fsfreeze, issues the command “timeout 5m /u/gce/google-cloud-sdk/bin/gcloud compute disks snapshot …” and unfreezes the fs and 

-at the end issues "unlock tables" to MySQL.

We want the duration of executing this script as short as possible.

Creating a snapshot most of the times takes minutes even for 50GB.

Is it safe enough to set the bash-timeout command for creating the snapshot back to e.g. 30 seconds?

From ‘Compute Engine Persistent Disk Backups using Snapshots’, Posted: Thursday, October 31, 2013: 

“In physical terms, it’s like stopping the disk head while the disk is in use and copying all disk sectors out, except that, thanks to Persistent Disk design, it takes only a fraction of a second to create the snapshot. The rest of the time is spent uploading the bits to Google Cloud Storage and there is no performance impact to your disks.”

[Alex Gaysinsky]

Hi Eric,

The safest thing is to monitor snapshot status and validate that snapshotting is complete. You can do this by using gcloud command.

gcloud compute snapshots describe <snapshot> | grep status

should be

status: READY

--Alex

--
© 2014 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/5db45427-2aaf-4396-8edd-db3046838c51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex Gaysinsky | Product Manager | gays...@google.com | 310-406-4298

[Leif Pedersen]

I might suggest (with a grain of salt) that a true snapshot shouldn't require freezing the system. In that case, when you restore the snapshot, ext3/4 and MySQL should see this the same way as a power-loss recovery.

If the system can't handle this type of recovery, I'd be concerned about its resilience in the face of other crash recovery scenarios. I actually like to exercise these recoveries whenever possible to expose problems at times when I'm not frantic to recover a system. (I recently developed a habit which seems to frighten my coworkers of simply pulling the power plug instead of shutting down properly.)

My understanding is that GCE snapshots are true atomic snapshots. Perhaps a GCE engineer would like to chime in if there's anything inadequate about my idea.

[Eric Aarts]

Hi Alex and Leif,

Thanks for your replies.

@Leif: I’m almost sure you need to freeze the system for a while, strongly suggested by the GCE team in documentation and blogs. I copied at the bottom more text from the October 31, 2013 GCE Blog.

@Alex: I understand safest is waiting till status is complete.

But I think I understand reading the October 31, 2013 Blog (but I’m not sure!) as soon as the status is CREATING, it would be safe to restart IO on the file system (we us a separate mounted disk for the MySQL databases).

We could easy change our script not to wait anymore after 30 seconds from the start of the snapshot-creation end then check whether the status is CREATING. If not, the script could loop for several times, and exit after e.g. 2 minutes if case of still no status change (happens 1 in 10 attempts…).

A second script could be started 1 hour after the first attempt, checking whether we have a successful snapshot. And in case of negative, try one time again the first script.

In my opinion a downtime of 5 minutes – even at night – already is too long, as internationally operating companies use our ERP systems.

I know for Amazon EC2 instances, it is safe to unfreeze a disk seconds after a snapshot actually starts being created and it is not necessary to wait till the snapshot finishes.

Regards, Eric

“Snapshots are guaranteed to be consistent at the block level (block size=4KiB). This means that snapshots never contain a partially written block (no torn writes). However, if application is making a write operation that spans multiple blocks, it’s possible that only the first portion of the write operation is stored in the snapshot. In addition, writes made to the file system cache will not be present in the snapshot, meaning a snapshot is equivalent to a disk that underwent an unclean shutdown. While most file systems and server applications can handle this just fine (due to journaled file systems, redo logs, etc.), it may require time-consuming steps to restore the disk back to a consistent state. Certain applications may not be able to handle this situation gracefully.

There are a few simple steps you can take to ensure your snapshot is consistent. First, we strongly recommend using mounted volumes for application data (not boot disk). Among other things, it’s easier and safer to take snapshots in a running instance. These are the steps you need to follow before snapshots are taken:

Pause IOs coming from the application if possible. Certain applications have explicit commands to pause and resume IOs, e.g. Oracle, MySQL. If your application doesn’t support pause/resume, make sure that your application can survive unclean shutdowns (e.g. MongoDB).

Flush the file system cache (sync), and

Pause IOs coming from file system. You can use fsfreeze command for Ext3/4, ReiserFS, JFS, and XFS.

Once there are no more IOs being sent to disk, it’s safe to start the snapshot. Note that IOs need to be paused for just the fraction of second that it takes to create the snapshot. IO operations can then be resumed while snapshot data is copied to Cloud Storage in the background.”

[Alex Gaysinsky]

Hi Eric,

Here is the link to the original post: http://googlecloudplatform.blogspot.com/2013/10/persistent-disk-backups-using-snapshots.html

I believe you can unfreeze after snapshot status changes to UPLOADING or READY.

Let me check with the persistent disk engineering team if anything has changed during the last year.

--Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/2cd7edf0-81db-48cb-a122-865e07771acb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fabricio Voznika]

That is correct. As soon as the snapshot state transitions to UPLOADING, it's safe to unfreeze the file system.

Thanks,

Fabricio.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/CAGKVHNrXnKgXD8n3jJm0naK7L%2B5tA4_Q5%3DOaitJj7yRsZb9f%2BA%40mail.gmail.com.

[Fabricio Voznika]

Leif, you're correct stating that snapshot of a disk is similar to a power-loss event, and that freezing the file system isn't required because modern file systems can recover. However, to avoid the need to run file system recovery and potentially long database recovery, it's best to freeze applications and file system before taking a snapshot. This causes minimal impact to the applications and provides an extra level of safety to your snapshots.

Thanks,

Fabricio.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/CAK-wPOhsiqfO1GLvmaNWNP3XcG%3DYpUwLpuN_oFNOCcJ-oSY4rA%40mail.gmail.com.

[Eric Aarts]

Alex, Fabricio,

All clear now for me, thanks for the prompt feedback.

A suggestion: it seems to me a good option, if with the command ‘gcloud compute disks snapshot’ we can add an extra flag asking the service to response as soon as the uploading starts (instead of only after reaching the ready status) and the script can restart disk-IO and may assume the snapshots ends with ready.

In our case, this would save extra commands asking for the snapshot status e.g. every 10 seconds and the downtime minimizes.

Regards, Eric

[Fabricio Voznika]

Yes, that would be handy. I filed a feature request internally with your suggestion. Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/b7fb174a-6cff-4c6c-831a-b9b541b369b0%40googlegroups.com.

[Leif Pedersen]

Perhaps in some cases that is the best solution, but not for mine. True that I have to wait for MyISAM/InnoDB checks when recovering (my filesystems don't need or have fsck), but the trade-off is that nominal operation would be more complex and flimsier and harder for other maintainers to examine because of the freeze/thaw process. It's hard to write this custom glue in a way that *guarantees* that if this node fails to unfreeze, the other nodes will not proceed with their snapshots, and that the freeze preemptively drains this node, etc. With enough work I could meet that guarantee, but I don't see how it would improve our service's availability in any predictable way. Even then, it limits how often we can take snapshots to do all that coordination; I like to snapshot very often to keep the window of potentially lost data small.

I don't really expect to convince anyone to use my method; all I'm really getting at is that it might be wise to make it clear in the documentation that the snapshots are atomic, so that the reader can make an informed decision whether to freeze/thaw while snapshotting. When I read the docs for the first time, I figured that the intended implication is that GCE's snapshots are not atomic, and I couldn't find a good statement that they are (so thank you for clarifying that).

--

As implied by email protocols, the information in this message is not confidential.  Any middle-man or recipient may inspect, modify, copy, forward, reply to, delete, or filter email for any purpose unless said parties are otherwise obligated.  As the sender, I acknowledge that I have a lower expectation of the control and privacy of this message than I would a post-card.  Further, nothing in this message is legally binding without cryptographic evidence of its integrity.

http://bilbo.hobbiton.org/wiki/Eat_My_Sig

[Fabricio Voznika]

The blog post mentions that snapshots are atomic (see excerpt below), but our docs online aren't clear about it indeed. I've updated the docs and it will shortly be available in https://cloud.google.com/compute/docs/disks#snapshots

"A snapshot is an exact copy of your disk at a point in time. A snapshot can be taken at the same time reads and writes are happening to the disk. In physical terms, it’s like stopping the disk head while the disk is in use and copying all disk sectors out, except that, thanks to Persistent Disk design, it takes only a fraction of a second to create the snapshot. The rest of the time is spent uploading the bits to Google Cloud Storage and there is no performance impact to your disks."

Thanks for your feedback,

Fabricio.