South backend for App Engine Datastore/Django-nonrel
Obey Arthur Liu
People using Django-nonrel
(http://www.allbuttonspressed.com/projects/django-nonrel) lack a
schema migration tool.
Below is a post I made to the Django-nonrel mailing list outlining
some initial research I did on the feasibility of an App Engine
Datastore backend for South.
I'd be happy to contribute to writing such a backend.
Cheers
Arthur
---------- Forwarded message ----------
From: Obey Arthur Liu <art...@milliways.fr>
Date: Sun, Jan 23, 2011 at 11:24 PM
Subject: Re: How to Change the Database Structure Gracefully
To: django-non...@googlegroups.com
Hi all,
I am quite interested by this issue as proper migrations are
absolutely necessary for production usage.
I had a look at South with django-nonrel and unsurprisingly, it
doesn't work. It only supports pgsql, mysql, sqlite, mssql and oracle.
Each db backend actually needs to implement a common schema
manipulation interface:
db.add_column
db.alter_column
db.clear_table
db.commit_transaction
db.create_index
db.create_primary_key
db.create_table
db.create_unique
db.delete_column
db.delete_foreign_key
db.delete_primary_key
db.delete_table
db.delete_unique
db.execute
db.execute_many
db.rename_column
db.rename_table
db.rollback_transaction
db.send_create_signal
db.start_transaction
If you look at it, these are basically all column-wise actions, which
makes sense as we're talking about relational databases where the
schema is explicitly handled by the database. As explained in the link
provided (http://code.google.com/appengine/articles/update_schema.html),
App Engine "Schema migrations" involve running a mapreduce over each
entity in the datastore to massage it individually to fit in the new
schema by adding and removing properties. This makes it quite
different from how South operates (about 90° different).
A django-nonrel/datastore backend would implement each of these
operations by running over an entire table (with the necessary
breaking into <30 secs parts). Batching operations on the same table
using start/commit_transaction could help here.
Still, this needs some code writing. I don't have extensive experience
of the underlying datastore implementation in django-nonrel nor of
South database backends but I'm willing to contribute some time to
write one.
Cheers
Arthur
On Jul 21 2010 at 8:22 am, Mark Ellul <ma...@catalystic.com> wrote:
> From: Mark Ellul <m...@catalystic.com>
> Date: Jul 21 2010, 8:22 am
> Subject: How to Change the Database Structure Gracefully
> To: Django non-relational
>
>
> Basically, you want Data Migrations.
>
> On a Normal Django application South could be used (http://
> south.aeracode.org/)
>
> I have never tested django-nonrel and South, in theory the data
> migrations
> might work, using remote infront of the management commands.
>
> Its a good question though, as going forward I imagine a system with
> changes
> over time of live data, data migrations would be needed.
>
> In Salesforce's platform, they version the objects based on the API
> they are
> connected to. Maybe that is a good approach for systems that will be
> around
> for a while.
>
> Your thoughts?
>
> Regards
> Mark Ellul
> Cloud Solutions Engineer
> [image: logo.png]
> Google Talkm...@catalystic.com Skypemark-e-markSpain - Mobile (+34)
> 692 649
> 593 UK - Landline(+44) 020 8133 9754
>
> On 21 July 2010 00:47, killer barney <ajcha...@gmail.com> wrote:
>
>
>
>> still open to suggestions, but found a good article on how to do it.
>
>>http://code.google.com/appengine/articles/update_schema.html
>
>> such an awkward way to do these things, but hey, if this is how it
>> gots to be, then it's how it gots to be :)
>
>> On Jul 20, 12:58 pm, killer barney <ajcha...@gmail.com> wrote:
>> > Does anybody have any tips on how we can change the database structure
>> > gracefully? I have yet to move to a release, but everytime I change
>> > the structure of the database, ti seems I have to wipe everything
>> > out. This, obviously, won't be a reasonable thing to do once I go
>> > live. I find that a big reason for this is because I don't have a SQL
>> > database right on my hands so massaging the data to fit the new schema
>> > isn't easy.
>
>> > So some things I've thought of is maybe download the data then re-
>> > install the data with the new site. Problem with this is that I don't
>> > know how to download the data. appcfg.py has the download_data command
>> > but that downloads all of the data and puts it all back in the same
>> > exact form. Is there a way to just download the data you want to
>> > retain and then re-connect all of the other pieces?
>
> logo.png
> 4KViewDownload
>
Andrew Godwin
> Hi all,
>
> People using Django-nonrel
> (http://www.allbuttonspressed.com/projects/django-nonrel) lack a
> schema migration tool.
> Below is a post I made to the Django-nonrel mailing list outlining
> some initial research I did on the feasibility of an App Engine
> Datastore backend for South.
> I'd be happy to contribute to writing such a backend.
So, as you point out in your email, the schema change part of South
makes little sense applied to a nonrelational datastore (apart from
things like the index management, perhaps). There is, in fact, a
non-relational Django backend (https://github.com/batiste/mongodj) that
has South support, by just no-opping most of the schema change operations.
I'm not going to object to a contributed nonrel backend, but I question
its utility as part of South directly - most nonrelational applications
I've written don't have the concept of a single schema on the entire
dataset at once, but rather each individual record is checked and
upgraded as it is used (otherwise you have to map a transformation over
the entire dataset in a blocking manner, which isn't helpful).
This might be something more suited to being stand-alone, something
which will be greatly helped when I'll hopefully start adding generic
migration support into Django. Then again, things like the model
freezing, which you need for datamigrations (the arguably useful part
with nonrel databases), are really part of South, as opposed to anything
else...
Andrew