Slow Datastore Backups

[re...@el-tramo.be]

Hi,

We are using the process described in the App Engine Docs to perform a daily automated backup of all our datastore tables to GCS. We have about 1Gb of entity data (excluding indexes) according to the datastore statistics page (most of it text). However, a daily backup of our database takes over 3 hours. This sounds to us like it won't scale when our database starts growing further, so we were wondering whether this sounds like a normal time, or whether there could be something we are doing wrong?

thanks,

Remko

[Nick (Cloud Platform Support)]

There are several factors that might cause the process to run shorter or longer.

One possibility is that you store many small entities, and there's an abundance of metadata needed to copy, rather than fewer, larger entities with properties which can simply be sent along the wire in chunks. What effect if any this might have on performance really depends.

Another possibility is that your bucket is in a different location than where your app's datastore entities are being copied from. I highly suggest you check this using the methods mentioned in the docs.

This sounds more like a stackoverflow or serverfault question than a discussion on the platform generally, and should probably move there after this comment. I'd suggest putting more information as well, such as statistics about your entities, etc. If this is the case, you might want to look into regional buckets to possibly increase your performance. 

Best Regards,

Nick