Alerts for disk snapshot failure

[Srihari Prabhakara]

I'm using disk snapshots for backing up MongoDB data. I'm taking a snapshot every 2 hours using a python script. But, the API calls return immediately and doesn't wait for the snapshot to complete. Is there any way I can wait in the script till snapshots complete. Or Is there a way to configure GCP to send alerts if snapshot operation fails?

[Justin Reiners]

Can you sanitize your script a bit and post it somewhere? I know there would be a way to use the gcloud api and not do --quiet whatever the flag was and it would wait until it got the OK before proceeding. I do this now in a couple of my python scripts.

On Wed, May 9, 2018 at 11:55 PM, Srihari Prabhakar <sriha...@gmail.com> wrote:

I'm using disk snapshots for backing up MongoDB data. I'm taking a snapshot every 2 hours using a python script. But, the API calls return immediately and doesn't wait for the snapshot to complete. Is there any way I can wait in the script till snapshots complete. Or Is there a way to configure GCP to send alerts if snapshot operation fails?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/b4ab4640-c44d-4342-b15a-d15bbc0490a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dinesh (Google Platform Support)]

Hi Srihari,

GCE API - zoneOperations.get could be used to monitor the status of ongoing operation (create a snapshot in your case) and you can use this API in your scripting to monitor disk snapshot status. This python-guide might help you to code for "waiting for operations".

These documentations from GitHub[1][2] might help further. 

Regarding your second question, have you tried stack driver monitoring for GCP? Unfortunately, I was unable to find a specific metrics for disk snapshot operation, but there are metrics for disk operations count, read, write[3] etc. Those metrics might not solve your purpose but worth to give a try. 

Let me know if above information helps?

[1]:https://github.com/emind-systems/GCE-Disk-Snapshot/blob/master/gce-disk-snapshot.py

[2]:https://gist.github.com/tsabat/5890733

[3]: https://cloud.google.com/monitoring/api/metrics_agent#agent-disk

[Srihari Prabhakara]

Below is the code snippet I am using. I am using the REST API not gcloud. I need to set a few labels while creating snapshots and AFAIK gcloud doesn't support that. 

for row in shard_to_disk_mappings:
    snapshot_name = "mongo-" + row.shard + "-" + str(int(start_time))
    zone = row.zone
    disk = row.disk
    snapshot_body = {
        "name" : snapshot_name,
        "labels" : {
            "shard" : row.shard,
            "type": snapshot_type # This is set to hourly/daily/weekly, value comes from command line.
        }
    }
    request = service.disks().createSnapshot(project=project, zone=zone, disk=disk, body=snapshot_body)
    response = request.execute()

But the problem with this is that it just returns a response, with operation id and status as pending. 
As Dinesh suggested, looks like zoneOperations.get will solve my problem.