download_data performance?

Dave Peck

unread,

Jun 5, 2012, 1:40:55 PM6/5/12

to Google App Engine

I want to download the full database for one of my apps. Entities
consume a relatively modest 2GB. (Indexes all told consume 19GB, but
my impression is that download_data won't download these?)

appcfg.py download_data has been running now for 12+ hours on a very
fast downstream connection.

Is there a faster way to download all data from a GAE app? Some
settings I can tweak that might help move things along?

Thanks,
Dave

c h

unread,

Jun 5, 2012, 1:53:23 PM6/5/12

to google-a...@googlegroups.com

by default download_data is throttled. read the docs for appcfg.py to see the settings and change them.

Dave Peck

unread,

Jun 5, 2012, 2:03:15 PM6/5/12

to Google App Engine

I'm not sure how I missed that! Thanks.

-Dave

Dave Peck

unread,

Jun 5, 2012, 2:08:15 PM6/5/12

to Google App Engine

Wait, hang on.

2GB of entities. Default bandwidth limit of 250,000 bytes/sec. So,
assuming we exactly saturate that limit, it should take ~2 hours 43
minutes. Now, we'll never saturate, so maybe estimate a 2x or even 3x
multiple of that time? That's still far less than the 14+ hours my
download job has been running.

-Dave

Barry Hunter

unread,

Jun 5, 2012, 2:33:34 PM6/5/12

to google-a...@googlegroups.com

How big is the data you have already downloaded?

You should be able to see the size of the file being written to.

(or find the temporally file its been written to)

Dave Peck

unread,

Jun 5, 2012, 3:02:05 PM6/5/12

to Google App Engine

~970MB right now.

Reply all

Reply to author

Forward