Google Cloud Datastore Incremental Backups

[Vinuth Madinur]

Hi,

The official backup method only supports full backup as of now. I don't know when, if at all, it'll support incremental backup.  It'll be wasteful to take full backups every time.

What strategies are you using for taking incremental backups of your datastore? 


Thanks,

Vinuth.

[Ryan (Cloud Platform Support)]

Salutations Gubbi,

This Feature Request would best be posted in the Public Issue Tracker. When you fill out the report let me know on this thread and I will process it for you.

[Jason Collins]

I have an old feature request here if you want to star it:

  https://code.google.com/p/googleappengine/issues/detail?id=7428

[Vinuth Madinur]

I have created a new feature request here: https://code.google.com/p/googleappengine/issues/detail?id=12071

starred the old feature request as well.

Are there any strategies I can use now for incremental backups?

[Ryan (Cloud Platform Support)]

I will process your PIT later today. 

Unfortunately there is no way to do incremental backups at this time.

[Vinuth Madinur]

Thanks Ryan.

We were looking at the Mapreduce code that does the backup and also the Cloud Dataflow code for DatastoreIO. We can do our own custom backups today using a query via dataflow. Only restriction seems to be that the query can't have inequality operators.

However, we are not able to find a way to schedule Dataflow pipelines. Questions to this effect on StackOverflow have gone unanswered. The Dataflow FAQ however has this:

Is there a built-in scheduling mechanism to execute pipelines at given time or interval?

You can automate pipeline execution through Google App Engine or custom (CRON) job processes on GCE. Future releases of the SDK will support command line options for finer grained control over job management.

But there is no documentation on how to automate execution using Appengine or Cron. If you could please help me with this, I'll have a solution.


Thanks,
Vinuth.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06f79b85-be73-4dc4-a774-07dfc6cf9cb0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan (Cloud Platform Support)]

You would create a cron job to hit a URL in your app. That URL would then start the mapreduce job programmably.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

[Vinuth Madinur]

Hi Ryan,

Thanks for the pointers. But I was asking about scheduling a Dataflow pipeline not Mapreduce pipeline. The above FAQ section is from Dataflow documentation. Can you please help? 

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06f79b85-be73-4dc4-a774-07dfc6cf9cb0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/143a8860-8148-48a0-941c-aeb27c66063b%40googlegroups.com.

[Ryan (Cloud Platform Support)]

Salutations Gubbi,

It would the same idea. Create a handler to execute your pipeline then have your cron hit it.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06f79b85-be73-4dc4-a774-07dfc6cf9cb0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

[Vinuth Madinur]

Oh ok. I was thinking from python perspective. I get it now. Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/06f79b85-be73-4dc4-a774-07dfc6cf9cb0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/143a8860-8148-48a0-941c-aeb27c66063b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/90af1c98-5aff-4bf6-9192-462c6e4c1b1a%40googlegroups.com.