Schema Evolution code

71 views
Skip to first unread message

Victor Ng

unread,
Dec 10, 2006, 11:22:06 PM12/10/06
to django...@googlegroups.com
Hi Russ,

I've got a rough version of schema evolution working now.

The basic implementation is what the SoC project was trying to do.

I've added 1 new attribute to the Meta class of the model definition,
and 1 argument to the Field class - 'aka'.

So you do stuff like this:

--- old model def --

class OldFoo(models.Model):
blah = models.CharField(maxlength=20)

--- new model below ---

class Foo(models.Model):
new_blah = models.TextField(aka=['blah'])
class Meta:
aka = ['OldFoo']

I'll automatically 'upconvert' so that:
* increased lengths on CharFields is valid
* increased integer and/or decimal portion length on DecimalField
* upconvert CharField -> TextField
* upconvert boolean -> smallint (false/true == 0/1)
* upconvert boolean -> integer (false/true == 0/1)

null constraints, db_index and positive integer constraints can be
switched on or off.

Adding/dropping columns/tables works. Renaming tables/columns works.

M2M fields have a bug still with repointing tables and recreating all
the proper constraints.

The problem with the code right now is that it's pretty ugly and
suffers from cut/paste-itis as I didn't really understand
django.core.management when I started it.

I honestly think it's a *bad* solution to the problem. I've been
looking at sqlalchemy and the 'migrate' project :

http://erosson.com/migrate/trac/

and it seems like it's a better solution for a couple reasons.

Ways in which my schema evolution code sucks:

1) converting the database is done pretty much all automatically so
you don't really have a way to version control your schema changes

2) there are *lots* of data migrations that are non-trivial and you
really need to be able to express some kinds of migrations in terms of
SQL that you can't code generate simply by introspecting the database
schema and the current model. Even if you stored more meta-data in
the database about the old model definition, you can't possibly
automatically infer how to populate new columns that have NOT NULL
constraints specified.

3) This is related to 2) - for schema transformations that my schema
evolution can't figure out - i just fail out and do nothing. So it
basically makes "long, simple and tedious" really easy, but it utterly
fails to help out with difficult schema migration.

That said - if anyone is still interested in looking at my code, I'm
happy to release it. But it's crap. Really. It's really really bad.
The only saving grace is that i've got unit tests and I merged back
with Django trunk as of Dec 6 - so it's not completely out of date.

vic

On 12/3/06, Russell Keith-Magee <freakb...@gmail.com> wrote:
>
>
> On 12/4/06, marksibly <blit...@blitzbasic.com> wrote:
> >
> > I initially thought 'manage.py syncdb' would detect and handle this for
> > you, but it doesn't appear to.
>
> No - Syncdb will add any completely new model, but will not alter any
> existing model.
>
> The behaviour you are seeking is under development in the schema evolution
> branch. I don't know the status of this branch for certain; it was the
> subject of a Google 'summer of code' project, and I believe that the
> engineering work is complete, but requires feedback and review before it is
> committed to the trunk (I could be wrong on this - anybody in the loop on
> this stream care to comment?)
>
> Anyway - the wiki describes what was intended for this stream:
>
> http://code.djangoproject.com/wiki/SchemaEvolution
>
> > Currently, I'm using MySql to 'DROP TABLE' and then using 'manage.py
> > syncdb' again, but this seems a bit cumbersome.
>
> As an alternate to DROP TABLE, you can use ./manage.py reset [appname] -
> this will execute the required DROP TABLE calls on the database. Yes, it is
> cumbersome, but its the best we can offer at the moment.
> Yours,
> Russ Magee %-)
>
> >
>


--
"Never attribute to malice that which can be adequately explained by
stupidity." - Hanlon's Razor

Jeremy Dunck

unread,
Dec 11, 2006, 8:51:23 AM12/11/06
to django...@googlegroups.com
On 12/10/06, Victor Ng <crank...@gmail.com> wrote:
>
> Hi Russ,
>
> I've got a rough version of schema evolution working now.
...

> I honestly think it's a *bad* solution to the problem. I've been
> looking at sqlalchemy and the 'migrate' project :


The attached file is in no way a complete solution, but it's working
for us with PegasusNews.com. It doesn't handle rollback; we assume a
db restore of an old version is the likely recovery approach. :)

We're using postgresql and an old version of django, but with
adjustments, the script should be workable for lots of people. One
thing to note is that postgresql can do almost any schema changes
under a single transaction. This is an unusual feature, and the
script would be much more complicated for, say, mssql or oracle. It
gives nice error messages and generally tries hard to only fail in
clean ways.

db_upgrade.py is a general driver script. It expects a setting,
UPGRADE_STEPS_PATH, to define the path at which a separate
upgrade_steps module can be imported. See the attached
upgrade_steps.py for further usage notes. db_upgrade.py also expects
the target schema to have a table named db_version which defines the
current version.

texasgigs=> \d db_version
Table "public.db_version"
Column | Type | Modifiers
---------+-----------------------+-----------
branch | character varying(50) |
version | integer |

The general approach is to get the version from the db, the highest
version magically-named migration function in the upgrade_steps
module, backup the existing db, and then successively run the
migration functions until either all of them have been run or an error
occurs.

If DEBUG=True, commits are done between each migration function.
Also, note that the passed conn and cursor are Django's wrapped
conn/cursor, so you are free to use the DJ DB API in the migrations;
you aren't limited to SQL via conn/cursor.

I hope it's useful to someone, and let me know of any questions.

-Jeremy

upgrade_steps.py
db_upgrade.py

Steve Hutton

unread,
Dec 13, 2006, 2:24:09 AM12/13/06
to django...@googlegroups.com
On 2006-12-11, Victor Ng <crank...@gmail.com> wrote:
> Hi Russ,
>
> I've got a rough version of schema evolution working now.
>
> The basic implementation is what the SoC project was trying to do.
[...]

> On 12/3/06, Russell Keith-Magee <freakb...@gmail.com> wrote:
[...]

>> The behaviour you are seeking is under development in the schema
>> evolution
>> branch. I don't know the status of this branch for certain; it was
>> the
>> subject of a Google 'summer of code' project, and I believe that the
>> engineering work is complete, but requires feedback and review before
>> it is
>> committed to the trunk (I could be wrong on this - anybody in the
>> loop on
>> this stream care to comment?)
>>
>> Anyway - the wiki describes what was intended for this stream:
>>
>> http://code.djangoproject.com/wiki/SchemaEvolution

I get the impression that there's not a lot of momentum behind the
schema evolution branch at the moment. It seems it hasn't been brought
up to date with head in a while. My status query on the developer list
didn't result in any replies.

I'm kind of surprised at this, since I think its a pretty frequently
requested feature. Does anyone here who has tried the schema evolution
branch care to comment?

Does it have a realistic chance of being accepted into core if it's found
to be bug free? Is it fully documented? Is the design controversial or
does it follow a community consensus?

Thanks,
Steve

Russell Keith-Magee

unread,
Dec 14, 2006, 8:22:14 AM12/14/06
to django...@googlegroups.com
On 12/13/06, Steve Hutton <gm...@featurecomplete.com> wrote:

> Does it have a realistic chance of being accepted into core if it's found
> to be bug free? Is it fully documented? Is the design controversial or
> does it follow a community consensus?

There was discussion about the general problem of schema evolution
before the SOC project was started. The discussion was started by
Jacob, and other committers (Luke and Malcolm) weighed in at the time,
along with many other interested onlookers. The resulting design is on
the wiki.

Assuming that the implementation matches the proposal, I would say
there is a realistic chance of it getting accepted into core. However,
this would require that the implementation is up to date, and bug free
(including tests to validate this status that are integrated into the
Django system tests).

Yours,
Russ Magee %-)

Victor Ng

unread,
Dec 14, 2006, 10:55:36 AM12/14/06
to django...@googlegroups.com
If anyone wants to poke at our schema evolution code you should be
able to apply this patch attached.

It's mostly working. The bugs I know about are:

1) M2M fields can't be repointed at new tables properly
2) there's some weird quirk with modifying null and db_index at the
same time. i have to run the sqlevolve command twice to get the
column to get altered and to have the index added.

The patch also fixes a couple problems I discovered in Django's
existing syncdb command and the way it handles unique and db_index
arguments. Those bugs were:

1) syncdb sometimes seems to 'miss' setting up foreign key constraints
between applications.
2) foreign key constraints were specified inconsistently. Sometimes a
'REFERENCES' clause was added to the CREATE TABLE and sometimes a
CREATE CONSTRAINT was used. This caused problems with figuring out
the names of the constraints I needed to remove.
3) indexes were created multiple times sometimes in cases where a
primary key was set.
4) indexes were *not* created for unique columns. technically not a
bug, but i can't see why you wouldn't want an index. For small
tables, the cost of the inserting a record against an index is small -
it's a small table and there just aren't that many inserts. For large
tables, the lack of an index means you're going to incur a table scan.

Again - this code is ugly and needs to be cleaned up a lot, but it's
working well enough for us against postgresql+psycopg2. It doesn't
work on any other backend.

Test cases are in the tests/evolvedb/runtests.py script.

You'll need to symlink from
tests/evolvedbtests/db_settings/settings.py.postgresql script to
tests/evolvedbtests/backend_settings.py.

All the existing django tests seem to pass with my patches applied.

There's no documentation but there's ~40 odd tests checked into that
evolvedb tests directory.

Unfortunately, I'm pressed for time for the forseeable future and I
don't think I'll be able to spend a lot more time working on this
code.

vic

schema_evo.patch.gz

Todd O'Bryan

unread,
Dec 14, 2006, 3:22:25 PM12/14/06
to django...@googlegroups.com
I know the company line on the SOC Schema Evolution code is that it will
be integrated into the trunk after enough people have tested it, but I
think this creates a chicken and egg problem. People aren't going to use
it until it's in trunk and it won't be in trunk until enough people test
it.

Does it impact normal use without using schema evolution? In other
words, could it be integrated into trunk with the proviso that

"The Schema Evolution feature should be considered beta. If you use this
feature, please test the results carefully and report any bugs you find.
If you don't use Schema Evolution, you shouldn't be affected."

I'd really like to try it, but I don't have time to keep up to date with
two branches.

Todd

Tim Chase

unread,
Dec 14, 2006, 3:30:35 PM12/14/06
to django...@googlegroups.com
> If you don't use Schema Evolution, you shouldn't be affected."
>
> I'd really like to try it, but I don't have time to keep up to date with
> two branches.

Early on, I naively just did the generic install of Django which
put it in the usual system-wide place.

Now I'd like to toy with both the Schema Evolution and the
row-level user privs branches.

Are there any "best practices" tips for swapping among them for
testing/experimenting? Preferably without causing /too/ much
damage. :)

Thanks,

-tkc

James Bennett

unread,
Dec 14, 2006, 3:48:49 PM12/14/06
to django...@googlegroups.com
On 12/14/06, Tim Chase <django...@tim.thechases.com> wrote:
> Are there any "best practices" tips for swapping among them for
> testing/experimenting? Preferably without causing /too/ much
> damage. :)

I can only speak to Unix-based systems, but what I've done with lots
of things of this nature is write a little script which will rewrite
the .profile and .rc files for my shell to have various environment
variables (in this case, PYTHONPATH) point to different locations,
then use 'source' to force the files to reload.

--
"May the forces of evil become confused on the way to your house."
-- George Carlin

Adrian Holovaty

unread,
Dec 14, 2006, 4:03:36 PM12/14/06
to django...@googlegroups.com
On 12/14/06, Tim Chase <django...@tim.thechases.com> wrote:
> Now I'd like to toy with both the Schema Evolution and the
> row-level user privs branches.
>
> Are there any "best practices" tips for swapping among them for
> testing/experimenting? Preferably without causing /too/ much
> damage. :)

Hey Tim,

I've added some more instructions to the "Using branches" part of our
documentation. Let me know if this helps.

http://www.djangoproject.com/documentation/contributing/#using-branches

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com

Victor Ng

unread,
Dec 14, 2006, 4:16:07 PM12/14/06
to django...@googlegroups.com
The patch I previous sent in mostly adds a couple functions to the
psycopg2 backend in the introspection module. The only big changes
that affect the mainline django code are in django.core.management.

I'm using my patches, so that's been tested through 3 schema updates
in production. As previously mentioned -there are those 2 remaining
bugs in the schema-evo code, but they're not show stoppers for me.

I see no reason why people need to use the latest trunk to test beta
quality features. That just makes no sense - if you need beta
features - use a branch, or apply my patch to your own copy of trunk.
Is there a problem that I'm not seeing? I'm quite sure that all the
code that my patch modifies hasn't been touched in over a month, so it
should apply cleanly to trunk.

vic

Tim Chase

unread,
Dec 14, 2006, 4:35:17 PM12/14/06
to django...@googlegroups.com
>> Are there any "best practices" tips for swapping among them for
>> testing/experimenting? Preferably without causing /too/ much
>> damage. :)
>
> Hey Tim,
>
> I've added some more instructions to the "Using branches" part of our
> documentation. Let me know if this helps.
>
> http://www.djangoproject.com/documentation/contributing/#using-branches

Yes, thanks! I think with a combo of symlinks and the other
response regarding a personalized environment, I think my final
solution will be to

1) adjust my $PYTHONPATH to point to a ~/projects/django symlink
2) redirect that symlink to point to various branches of interest

My kudos to the Django team...a while back I had started my own
framework (what Python programmer hasn't? ;) with a number of
goals in mind. However, as I learned about Django, I found that
it already had implemented nearly all of what I wanted. There
are still the two aforementioned issues that are solved in
branches (row-level permissions and schema evolution), but it
looks like those will come down the pike and eventually merge
with the trunk. Thanks!

-tkc


Steve Hutton

unread,
Dec 15, 2006, 2:56:15 AM12/15/06
to django...@googlegroups.com
On 2006-12-14, Victor Ng <crank...@gmail.com> wrote:
>
> The patch I previous sent in mostly adds a couple functions to the
> psycopg2 backend in the introspection module. The only big changes
> that affect the mainline django code are in django.core.management.
>
> I'm using my patches, so that's been tested through 3 schema updates
> in production. As previously mentioned -there are those 2 remaining
> bugs in the schema-evo code, but they're not show stoppers for me.
>
> I see no reason why people need to use the latest trunk to test beta
> quality features. That just makes no sense - if you need beta
> features - use a branch, or apply my patch to your own copy of trunk.
> Is there a problem that I'm not seeing? I'm quite sure that all the
> code that my patch modifies hasn't been touched in over a month, so it
> should apply cleanly to trunk.

Victor, thanks very much for your efforts on this. I will be starting
a new project soon and would like to test schema evolution.

Can you explain the differences between your patch and the schema
evolution branch itself? Are you proposing that your changes be rolled
be rolled into the schema evolution branch?

Steve

Steve Hutton

unread,
Dec 15, 2006, 3:14:55 AM12/15/06
to django...@googlegroups.com
On 2006-12-14, Russell Keith-Magee <freakb...@gmail.com> wrote:
>
> On 12/13/06, Steve Hutton <gm...@featurecomplete.com> wrote:
>
>> Does it have a realistic chance of being accepted into core if it's found
>> to be bug free? Is it fully documented? Is the design controversial or
>> does it follow a community consensus?
>
> There was discussion about the general problem of schema evolution
> before the SOC project was started. The discussion was started by
> Jacob, and other committers (Luke and Malcolm) weighed in at the time,
> along with many other interested onlookers. The resulting design is on
> the wiki.

That's good to hear.

> Assuming that the implementation matches the proposal, I would say
> there is a realistic chance of it getting accepted into core. However,
> this would require that the implementation is up to date, and bug free
> (including tests to validate this status that are integrated into the
> Django system tests).

I see. It sounds like it may be a bit of moving target, due to changes
in core. Maybe a good goal would be to get the implementation up to
date (maybe that's what Victor has done?) and make sure the test
coverage is in place (I'm not sure the status of this?).

Then publicize it a bit and call for testers to essentially try what
would essentially be a release candidate.

Maybe someone is already thinking along these lines? It seems like the
branch could benefit from a "champion" who actively pushes it through.

Steve

Russell Keith-Magee

unread,
Dec 15, 2006, 4:56:48 AM12/15/06
to django...@googlegroups.com
On 12/15/06, Steve Hutton <gm...@featurecomplete.com> wrote:
>
> > Assuming that the implementation matches the proposal, I would say
> > there is a realistic chance of it getting accepted into core. However,
> > this would require that the implementation is up to date, and bug free
> > (including tests to validate this status that are integrated into the
> > Django system tests).
>
> I see. It sounds like it may be a bit of moving target, due to changes
> in core. Maybe a good goal would be to get the implementation up to
> date (maybe that's what Victor has done?) and make sure the test
> coverage is in place (I'm not sure the status of this?).

Keeping everything up to date is the role of the champion you mention below.

> Then publicize it a bit and call for testers to essentially try what
> would essentially be a release candidate.

The call for testing should be made by the champion; unless I've
missed a post, I don't think this has happened. The branch hasn't been
touched for 2 months, but I don't know if this is because the work is
done, or because Derek Anderson (the SOC participant running the
schema-evolution project) hasn't had time to work on it, or if he has
just lost interest.

Either way, the ultimate goal would be to convince Adrian that the
branch is stable, get him to take a look at it, and merge the branch
into the trunk. Adrian is pretty busy, so I'm guessing he would be
looking for some community consensus that the branch is complete and
stable before he spends any time looking at it.

> Maybe someone is already thinking along these lines? It seems like the
> branch could benefit from a "champion" who actively pushes it through.

This 'champion' role should be performed by Derek and/or Kenneth
Gonsalves (the SOC mentor for the project). If you want to help out,
check out the branch, and take it for a test drive. If what you find
is stable, let the community know; if it isn't, log bugs (and patches
if you can) until it is.

Yours,
Russ Magee %-)

Waylan Limberg

unread,
Dec 15, 2006, 12:37:36 PM12/15/06
to django...@googlegroups.com
On 12/14/06, Adrian Holovaty <holo...@gmail.com> wrote:
>
> I've added some more instructions to the "Using branches" part of our
> documentation. Let me know if this helps.
>
> http://www.djangoproject.com/documentation/contributing/#using-branches
>
> Adrian
>

No mention of the cross-platform path files? I would say they are
easier to edit (comment/uncomment a line in a file) than symlink and
as they are a python feature (not an OS feature) they work on any OS.

Just place the file `django.pth` in your site-packages dir (or
anywhere on your pythonpath). The file would look something like this:

/path/to/django_src
# /path/to/django_branch1
# /path/to/django_branch2

If you want to switch to branch2, just edit the file and comment line
one and uncomment line 3. It's that easy.

For more on path files see this page:
http://bob.pythonmac.org/archives/2005/02/06/using-pth-files-for-python-development/


--
----
Waylan Limberg
way...@gmail.com

Adrian Holovaty

unread,
Dec 15, 2006, 12:40:17 PM12/15/06
to django...@googlegroups.com
On 12/15/06, Waylan Limberg <way...@gmail.com> wrote:
> No mention of the cross-platform path files? I would say they are
> easier to edit (comment/uncomment a line in a file) than symlink and
> as they are a python feature (not an OS feature) they work on any OS.

Hey Waylan,

A patch to the documentation would be welcome. :)

Waylan Limberg

unread,
Dec 15, 2006, 4:49:41 PM12/15/06
to django...@googlegroups.com
On 12/15/06, Adrian Holovaty <holo...@gmail.com> wrote:
>
> On 12/15/06, Waylan Limberg <way...@gmail.com> wrote:
> > No mention of the cross-platform path files? I would say they are
> > easier to edit (comment/uncomment a line in a file) than symlink and
> > as they are a python feature (not an OS feature) they work on any OS.
>
> Hey Waylan,
>
> A patch to the documentation would be welcome. :)
>
Done: http://code.djangoproject.com/ticket/3147

I hope it is clear enough.

Victor Ng

unread,
Dec 16, 2006, 11:59:33 PM12/16/06
to django...@googlegroups.com
FYI - the schema evolution code submitted from the SoC project doesn't
work, so keeping it up to date is a moot point. There have been
several posts from people trying to use it where the SoC version of
the schema evolution code just halts.

The implementation in my patch is basically a complete rewrite. The
only things I've kept are the 'aka' attribute definitions in the Field
and the Model.Meta classes.

vic

On 12/15/06, Steve Hutton <gm...@featurecomplete.com> wrote:
>

> > There was discussion about the general problem of schema evolution
> > before the SOC project was started. The discussion was started by
> > Jacob, and other committers (Luke and Malcolm) weighed in at the time,
> > along with many other interested onlookers. The resulting design is on
> > the wiki.
>
> That's good to hear.
>
> > Assuming that the implementation matches the proposal, I would say
> > there is a realistic chance of it getting accepted into core. However,
> > this would require that the implementation is up to date, and bug free
> > (including tests to validate this status that are integrated into the
> > Django system tests).
>

The tests I've added to the schema evolution code are fully automated,
but they aren't currently integrated into the main Django test suite.
The main reason for this is that the schema evolution code relies on a
Postgresql backend. You can't run the tests against MySQL or SQLite.
MySQL doesn't work because I haven't looked at it yet, and SQLite is
probably going to be impossible to implement since sqlite doesn't
implement a proper ALTER TABLE statement.

> I see. It sounds like it may be a bit of moving target, due to changes
> in core. Maybe a good goal would be to get the implementation up to
> date (maybe that's what Victor has done?) and make sure the test
> coverage is in place (I'm not sure the status of this?).

Test coverage is ~44 test cases. There are currently 2 known problems
in the code from a 'correctness' standpoint, and there is a ton of
refactoring that needs to be done before it should be considered for
merging into the trunk.

The code is *messy*. Writing schema evolution went hand in hand with
me learning how Django's schema generation worked.

That said - the modifications are basically:

* decomposing a lot of functions in django.core.managemeent so that
they're more re-usabl
* removing all REFERENCES clauses in CREATE TABLE statements and
changed the code to just emit CREATE CONSTRAINT statements. This
makes everything more consistent and easier for me to figure out what
constraints to remove later if the schema is changed.
* rewriting the syncdb function since it missed creating some foreign
key relationships sometimes
* most of the heavy lifting is in django.contrib.schemaevo - again,
it's all Postgresql specific and I just ripped out the code that I
wrote into django.core.management and dropped it into
django.contrib.schemaevo. Minor changes were made to get my tests to
run again.

> Maybe someone is already thinking along these lines? It seems like the
> branch could benefit from a "champion" who actively pushes it through.

If someone wants to open up commit access on the django branch for
schema evo, I'll be happy to bring the schema evo code up to date with
the trunk.

I *do* need help in tidying it up to get it merged back to trunk
though. I'm hard pressed for time right now and I could use some
help.

vic

Steve Hutton

unread,
Dec 17, 2006, 1:11:20 AM12/17/06
to django...@googlegroups.com
On 2006-12-17, Victor Ng <crank...@gmail.com> wrote:
>
> FYI - the schema evolution code submitted from the SoC project doesn't
> work, so keeping it up to date is a moot point. There have been
> several posts from people trying to use it where the SoC version of
> the schema evolution code just halts.
>
> The implementation in my patch is basically a complete rewrite. The
> only things I've kept are the 'aka' attribute definitions in the Field
> and the Model.Meta classes.

Ah, I see. Your comments and the fact that the schema evolution branch
has not been actively updated lately or championed make me disinclined
to spend time testing it.

Perhaps your patches could form the basis of an alternative schema
evolution branch?

Steve

Steve Hutton

unread,
Dec 17, 2006, 1:26:29 AM12/17/06
to django...@googlegroups.com
On 2006-12-15, Russell Keith-Magee <freakb...@gmail.com> wrote:

> Either way, the ultimate goal would be to convince Adrian that the
> branch is stable, get him to take a look at it, and merge the branch
> into the trunk. Adrian is pretty busy, so I'm guessing he would be
> looking for some community consensus that the branch is complete and
> stable before he spends any time looking at it.

Absolutely makes sense to me.

>> Maybe someone is already thinking along these lines? It seems like the
>> branch could benefit from a "champion" who actively pushes it through.
>
> This 'champion' role should be performed by Derek and/or Kenneth
> Gonsalves (the SOC mentor for the project).

Right, also makes sense. It would be nice to get Derek and Kenneth's
thoughts on the status of the branch and what they think the remaining
tasks are to make it "merge ready".

> If you want to help out,
> check out the branch, and take it for a test drive. If what you find
> is stable, let the community know; if it isn't, log bugs (and patches
> if you can) until it is.

Thanks, I plan on doing just that in the coming weeks. However I'm not
sure yet if I'll be trying Victor's alternative implementation or the
existing branch.

Steve


Steve Hutton

unread,
Dec 17, 2006, 1:40:31 AM12/17/06
to django...@googlegroups.com
On 2006-12-17, Victor Ng <crank...@gmail.com> wrote:
>
> Test coverage is ~44 test cases. There are currently 2 known problems
> in the code from a 'correctness' standpoint, and there is a ton of
> refactoring that needs to be done before it should be considered for
> merging into the trunk.

I see, 44 test cases sound like a good start.

> The code is *messy*. Writing schema evolution went hand in hand with
> me learning how Django's schema generation worked.

[...]

> If someone wants to open up commit access on the django branch for
> schema evo, I'll be happy to bring the schema evo code up to date with
> the trunk.
>
> I *do* need help in tidying it up to get it merged back to trunk
> though. I'm hard pressed for time right now and I could use some
> help.

It sounds like your code has potential to be a full fledged alternative
implementation. I've never looked at the django internals, but I might
be able to help nevertheless. I will have some time starting about 10
days from now. At this point it would be a question of where I spend my
time - testing the existing branch code or helping test and/or clean up
and integrate your code.

The fact that you've already tried the existing schema evolution branch
and spent this much effort so far on an alternative implementation
certainly makes me lean towards your code. But it would be nice from
a logistical point of view if we could have access it to it from a
repository somewhere.

Steve

Steve Hutton

unread,
Dec 17, 2006, 2:42:07 AM12/17/06
to django...@googlegroups.com
On 2006-12-11, Victor Ng <crank...@gmail.com> wrote:
>
> I've got a rough version of schema evolution working now.
>
> The basic implementation is what the SoC project was trying to do.

[...]

> I honestly think it's a *bad* solution to the problem. I've been
> looking at sqlalchemy and the 'migrate' project :
>
> http://erosson.com/migrate/trac/
>
> and it seems like it's a better solution for a couple reasons.
>
> Ways in which my schema evolution code sucks:
>
> 1) converting the database is done pretty much all automatically so
> you don't really have a way to version control your schema changes

Is this a departure from the Proposal in the wiki[1]?
It says:

"Introspection + migration
This approach is a combination of Automatic db introspection and
Automatically applied migration code -- one command produces a migration
script, you tickle your version number, and the syncdb runs the
migrations"

Steve
[1] http://code.djangoproject.com/wiki/SchemaEvolution

Victor Ng

unread,
Dec 22, 2006, 3:50:48 PM12/22/06
to django...@googlegroups.com
Hi, sorry for the long delay in replying. Holiday season and work
craziness is getting in the way of writing free software - which is
really the fun part isn't it? ;)

On 12/17/06, Steve Hutton <gm...@featurecomplete.com> wrote:
>
> > Ways in which my schema evolution code sucks:
>
> > 1) converting the database is done pretty much all automatically so
> > you don't really have a way to version control your schema changes
> >
> Is this a departure from the Proposal in the wiki[1]?
> It says:
> "Introspection + migration
> This approach is a combination of Automatic db introspection and
> Automatically applied migration code -- one command produces a migration
> script, you tickle your version number, and the syncdb runs the
> migrations"

It's a little different, but barely.

The current codebase emits an SQL file with all the SQL statements to
evolve the database, There's no provision to save the version
anywhere, but that would be simple to do.

If nobody want's to champion the code, I can volunteer ~3-5 hours a
week to work on this stuff. My own team needs schema evolution in the
long run anyway - so it's well worth my time to make sure this
eventually gets back to trunk.

I won't be able to do too much until the new year though - so that
said - can somebody open up the schema evolution branch, or better -
open up a new branch from trunk so that we can start hacking away at
this problem?

victor "i hate parking lots at christmas" ng

Steve Hutton

unread,
Dec 27, 2006, 5:15:58 PM12/27/06
to django...@googlegroups.com
On 2006-12-22, Victor Ng <crank...@gmail.com> wrote:
>
> Hi, sorry for the long delay in replying. Holiday season and work
> craziness is getting in the way of writing free software - which is
> really the fun part isn't it? ;)

:-)

> It's a little different, but barely.
>
> The current codebase emits an SQL file with all the SQL statements to
> evolve the database, There's no provision to save the version
> anywhere, but that would be simple to do.
>
> If nobody want's to champion the code, I can volunteer ~3-5 hours a
> week to work on this stuff. My own team needs schema evolution in the
> long run anyway - so it's well worth my time to make sure this
> eventually gets back to trunk.
>
> I won't be able to do too much until the new year though - so that
> said - can somebody open up the schema evolution branch, or better -
> open up a new branch from trunk so that we can start hacking away at
> this problem?

Sounds great to me. Any core devs care to open a new branch for this?

Thanks,
Steve

Matthew Flanagan

unread,
Dec 28, 2006, 10:52:38 PM12/28/06
to django...@googlegroups.com

From my earlier involvement I know Victor's code is based on the
current schema evolution branch so I think it makes sense for his
fixes to be checked into it. Also it would be less confusing for the
rest of the django community if there was a single branch, considering
that there is atleast one enquiry a week about it.

Victor, have you got a ticket open with your patch? It would be worth
opening one so that other people could try it out while you are
waiting for commit access to be granted.

regards

matthew

Kenneth Gonsalves

unread,
Dec 28, 2006, 11:45:36 PM12/28/06
to django...@googlegroups.com

On 29-Dec-06, at 9:22 AM, Matthew Flanagan wrote:

> Victor, have you got a ticket open with your patch? It would be worth
> opening one so that other people could try it out while you are
> waiting for commit access to be granted.

victor i would suggest you directly mail jacob kaplan-moss with your
request for commit access

--

regards
kg
http://lawgon.livejournal.com
http://nrcfosshelpline.in/web/


Curtis Faith

unread,
Jan 7, 2007, 9:34:14 PM1/7/07
to Django users
> If nobody want's to champion the code, I can volunteer ~3-5 hours a
> week to work on this stuff. My own team needs schema evolution in the
> long run anyway - so it's well worth my time to make sure this
> eventually gets back to trunk.

I can devote about the same time to it perhaps a bit more in a few
weeks.

I have done schema migration before although not in python. My company
produced a development environment product for mobile computing in 1996
that supported arbitrary schema changes including versioning, and
automatic migration from development to test and production systems.

Our environment was a bit like Django in that we had a separate model
which was semantically richer and allowed us to automatically handle
99% of the database interactions (although in a GUI app).

I am not an expert on all the different issues among different
databases but I am familiar with many of them at least as they existed
a few years back.

I have also worked on the internals for SQL database parsers,
optimizers, indexing, and have played with the internals of PostgreSQL
a bit.

I don't know Django very well yet so I'll probably confine myself to
testing and bug fixing at first until I come up to speed.

- Curtis

Curtis Faith

unread,
Jan 8, 2007, 8:03:54 PM1/8/07
to Django users
I posted a message on django-developers since I think that is a more
common place for discussions to changes in django itself and Russ McGee
responded there:

http://groups.google.com/group/django-developers/browse_thread/thread/4cbc121bba0c134e

Here is the tail end of that response. I suggest following up the
discussion there unless I have completely missed the difference between
the users and developers list (my impressions synch with Russ's
comments so I think I'm right).

>
> There is no formal distinction - users are the set of people
> interested in using Django to develop websites, whereas developers are
> those interested in contributing to improve Django itself. Anyone is
> welcome to be a member of either group.
>
> Inside the developers, there are the core developers (such as myself)
> that have commit access to the repository, and branch developers that
> have commit access to a particular branch of Django. There is a subset
> of the core developers (Adrian and Jacob) that have administrator
> access to the repository and can grant commit access to various parts
> of the tree.
>
> Branch developer status will generally be granted to anyone that has a
> good idea, and can demonstrate that they will be able to deliver what
> they promise (a prototype implementation is the usual minimum
> benchmark). Core developer status is the next step, once a reputation
> for consistent contribution to the community is established.
>
> > Any idea what the status is on the potential for a branch, or for
> > Victor's changes getting rolled into the schema evolution branch? I've
> > volunteered to put some work into this on the other discussion but am
> > not sure what the next steps are or who will make those decisions.
>
> At the moment, the ball is in Victor's court - he has presented a
> prototype implementation, so if he wants to champion this cause, he
> needs to contact Jacob and request SVN access to the schema evolution
> branch. At that point, he will be the integration person for that
> branch, so if you want to contribute, he will be the person to
> coordinate with. If Victor doesn't want the champion role, then the
> floor is open for someone to step up, with some sample code to prove
> that they are capable of completing the task.
>
> Yours,
> Russ Magee %-)

Reply all
Reply to author
Forward
0 new messages