Screencast demonstrating the first Google App Engine data export solution (full backup and restore)

[Aral Balkan]

Hey everyone,

I've just recorded a short screencast demonstrating the data export
(backup/restore) that I've built for Google App Engine:

http://aralbalkan.com/1448

I'm going to be releasing it as a separate open source project in the
coming days. It will be a Django app that you can simply plug into
your own apps.

Woot! :)

Aral

[Ozgur Cem S.]

That looks sweet and exciting. Thumbs up and congrats.

Cem

[Fernando Correia]

2008/8/12 Aral Balkan <aralb...@gmail.com>

> I've just recorded a short screencast demonstrating the data export
> (backup/restore) that I've built for Google App Engine:

Congratulations, Aral! That's certainly a nice achievement.

How reliable do you consider this method to be? For instance, if
objects are being created and deleted as the process runs, is there a
change an object would be missed due to a paging scheme? Or if there
is some hiccup in the whole client html-reloading process?

[Bill]

Nice work, Aral. I look forward to seeing the code. Might be enough
to make me fully embrace Django through the helper :)
-Bill

On Aug 12, 8:56 am, Aral Balkan <aralbal...@gmail.com> wrote:

[Lee Olayvar]

Also, any word on an exporter from google? I know they planned on implementing a way to import/export data.. eventually.

--
Lee Olayvar

[Aral Balkan]

Hi Fernando,

> Congratulations, Aral! That's certainly a nice achievement.

Thanks :)

> How reliable do you consider this method to be? For instance, if
> objects are being created and deleted as the process runs, is there a
> change an object would be missed due to a paging scheme? Or if there
> is some hiccup in the whole client html-reloading process?

Currently, it's been reliable while backing up my app which has about
3000+ rows of data.

I'm currently checking for InternalErrors and retrying three times on
those. Apart from that I haven't had any issues.

One thing that Pete Koomen made me aware of is that the method I'm
using (which uses fetch with an offset) will not scale indefinitely.
This is because the datastore has to actually skip through the records
that you're skipping. What we need is a way to do inequality checks on
ids -- something that I believe the team at Google is working on.
Until that, this is the best/only solution. For me, it represents the
difference between not having backups and having them -- which is a
big one! :)

Once I've decoupled it from my app and released the source, I'll be
very interested in hearing your feedback and how well it works for you
and your apps.

Aral

[Aral Balkan]

Hey Bill,

Thanks! :) Re: Django, having started with webapp, I can attest that
there's a world of difference.

Aral

On Aug 13, 3:25 am, Bill <billk...@gmail.com> wrote:
> Nice work, Aral. I look forward to seeing the code. Might be enough
> to make me fully embrace Django through the helper :)
> -Bill

<snip>

[Chris Marasti-Georg]

Aral,

Any word on when you will be unleashing this on us?  I am in a beta stage right now, with quite a bit of user data in the datastore, and it is becoming more and more important that I test model updates with accurate data...

Thank you for the work on this - it looks to be a very useful tool!

[Valery]

Hi Aral,

let us know if you got some updates, please!
regards
Valery

[Garrett Davis]

For those of you who need just a simple 'data export' process,
I wrote a Bulk Data Download module back in July.
Here's a description: http://code.google.com/p/gawsh/wiki/BulkDownload
and the source code is here: http://code.google.com/p/gawsh
It complements the Google bulk upload program, and can download XML or
CSV files.
It has limitations, since it doesn't pickle or anything like that.
But it should scale: I tested it with 20,000 rows of data.

Aral, maybe you can use some of my ideas in your application, and
maybe you (or some other ambitious developer) can flesh out my design
(not coded yet) to handle Fernando's concerns about making a backup at
the same time others are adding/changing data.

Garrett Davis