Hi Russ,

I've got a rough version of schema evolution working now.

The basic implementation is what the SoC project was trying to do.

I've added 1 new attribute to the Meta class of the model definition,
and 1 argument to the Field class - 'aka'.

So you do stuff like this:

--- old model def --

class OldFoo(models.Model):
  blah = models.CharField(maxlength=20)

--- new model below ---

class Foo(models.Model):
  new_blah = models.TextField(aka=['blah'])
  class Meta:
    aka = ['OldFoo']

I'll automatically 'upconvert' so that:
  * increased lengths on CharFields is valid
  * increased integer and/or decimal portion length on DecimalField
  * upconvert CharField -> TextField
  * upconvert boolean -> smallint (false/true == 0/1)
  * upconvert boolean -> integer (false/true == 0/1)

null constraints, db_index and positive integer constraints can be
switched on or off.

Adding/dropping columns/tables works.  Renaming tables/columns works.

M2M fields have a bug still with repointing tables and recreating all
the proper constraints.

The problem with the code right now is that it's pretty ugly and
suffers from cut/paste-itis as I didn't really understand
django.core.management when I started it.

I honestly think it's a *bad* solution to the problem.  I've been
looking at sqlalchemy and the 'migrate' project :

http://erosson.com/migrate/trac/

and it seems like it's a better solution for a couple reasons.

Ways in which my schema evolution code sucks:

1) converting the database is done pretty much all automatically so
you don't really have a way to version control your schema changes

2) there are *lots* of data migrations that are non-trivial and you
really need to be able to express some kinds of migrations in terms of
SQL that you can't code generate simply by introspecting the database
schema and the current model.  Even if you stored more meta-data in
the database about the old model definition, you can't possibly
automatically infer how to populate new columns that have NOT NULL
constraints specified.

3) This is related to 2) - for schema transformations that my schema
evolution can't figure out - i just fail out and do nothing.  So it
basically makes "long, simple and tedious" really easy, but it utterly
fails to help out with difficult schema migration.

That said - if anyone is still interested in looking at my code, I'm
happy to release it.  But it's crap. Really. It's really really bad.
The only saving grace is that i've got unit tests and I merged back
with Django trunk as of Dec 6 - so it's not completely out of date.

vic

On 12/3/06, Russell Keith-Magee <freakboy3...@gmail.com> wrote:

--
"Never attribute to malice that which can be adequately explained by
stupidity."  - Hanlon's Razor

On 12/10/06, Victor Ng <crankyco...@gmail.com> wrote:

The attached file is in no way a complete solution, but it's working
for us with PegasusNews.com.  It doesn't handle rollback; we assume a
db restore of an old version is the likely recovery approach.  :)

We're using postgresql and an old version of django, but with
adjustments, the script should be workable for lots of people.  One
thing to note is that postgresql can do almost any schema changes
under a single transaction.  This is an unusual feature, and the
script would be much more complicated for, say, mssql or oracle.  It
gives nice error messages and generally tries hard to only fail in
clean ways.

db_upgrade.py is a general driver script.  It expects a setting,
UPGRADE_STEPS_PATH, to define the path at which a separate
upgrade_steps module can be imported.  See the attached
upgrade_steps.py for further usage notes.  db_upgrade.py also expects
the target schema to have a table named db_version which defines the
current version.

texasgigs=> \d db_version
          Table "public.db_version"
 Column  |         Type          | Modifiers
---------+-----------------------+-----------
 branch  | character varying(50) |
 version | integer               |

The general approach is to get the version from the db, the highest
version magically-named migration function in the upgrade_steps
module, backup the existing db, and then successively run the
migration functions until either all of them have been run or an error
occurs.

If DEBUG=True, commits are done between each migration function.
Also, note that the passed conn and cursor are Django's wrapped
conn/cursor, so you are free to use the DJ DB API in the migrations;
you aren't limited to SQL via conn/cursor.

I hope it's useful to someone, and let me know of any questions.

  -Jeremy

upgrade_steps.py 1K Download

db_upgrade.py 6K Download

On 2006-12-11, Victor Ng <crankyco...@gmail.com> wrote:

I get the impression that there's not a lot of momentum behind the
schema evolution branch at the moment.  It seems it hasn't been brought
up to date with head in a while.  My status query on the developer list
didn't result in any replies.

I'm kind of surprised at this, since I think its a pretty frequently
requested feature.  Does anyone here who has tried the schema evolution
branch care to comment?  

Does it have a realistic chance of being accepted into core if it's found
to be bug free?  Is it fully documented?  Is the design controversial or
does it follow a community consensus?

Thanks,
Steve

On 12/13/06, Steve Hutton <gm...@featurecomplete.com> wrote:

There was discussion about the general problem of schema evolution
before the SOC project was started. The discussion was started by
Jacob, and other committers (Luke and Malcolm) weighed in at the time,
along with many other interested onlookers. The resulting design is on
the wiki.

Assuming that the implementation matches the proposal, I would say
there is a realistic chance of it getting accepted into core. However,
this would require that the implementation is up to date, and bug free
(including tests to validate this status that are integrated into the
Django system tests).

Yours,
Russ Magee %-)

If anyone wants to poke at our schema evolution code you should be
able to apply this patch attached.

It's mostly working.  The bugs I know about are:

1) M2M fields can't be repointed at new tables properly
2) there's some weird quirk with modifying null and db_index at the
same time.  i have to run the sqlevolve command twice to get the
column to get altered and to have the index added.

The patch also fixes a couple problems I discovered in Django's
existing syncdb command and the way it handles unique and db_index
arguments.  Those bugs were:

1) syncdb sometimes seems to 'miss' setting up foreign key constraints
between applications.
2) foreign key constraints were specified inconsistently.  Sometimes a
'REFERENCES' clause was added to the CREATE TABLE and sometimes a
CREATE CONSTRAINT was used.  This caused problems with figuring out
the names of the constraints I needed to remove.
3) indexes were created multiple times sometimes in cases where a
primary key was set.
4) indexes were *not* created for unique columns.  technically not a
bug, but i can't see why you wouldn't want an index.  For small
tables, the cost of the inserting a record against an index is small -
it's a small table and there just aren't that many inserts.  For large
tables, the lack of an index means you're going to incur a table scan.

Again - this code is ugly and needs to be cleaned up a lot, but it's
working well enough for us against postgresql+psycopg2.  It doesn't
work on any other backend.

Test cases are in the tests/evolvedb/runtests.py script.

You'll need to symlink from
tests/evolvedbtests/db_settings/settings.py.postgresql script to
tests/evolvedbtests/backend_settings.py.

All the existing django tests seem to pass with my patches applied.

There's no documentation but there's ~40 odd tests checked into that
evolvedb tests directory.

Unfortunately, I'm pressed for time for the forseeable future and I
don't think I'll be able to spend a lot more time working on this
code.

vic

On 12/14/06, Russell Keith-Magee <freakboy3...@gmail.com> wrote:

--
"Never attribute to malice that which can be adequately explained by
stupidity."  - Hanlon's Razor

schema_evo.patch.gz 53K Download

I know the company line on the SOC Schema Evolution code is that it will
be integrated into the trunk after enough people have tested it, but I
think this creates a chicken and egg problem. People aren't going to use
it until it's in trunk and it won't be in trunk until enough people test
it.

Does it impact normal use without using schema evolution? In other
words, could it be integrated into trunk with the proviso that

"The Schema Evolution feature should be considered beta. If you use this
feature, please test the results carefully and report any bugs you find.
If you don't use Schema Evolution, you shouldn't be affected."

I'd really like to try it, but I don't have time to keep up to date with
two branches.

Todd

Early on, I naively just did the generic install of Django which
put it in the usual system-wide place.

Now I'd like to toy with both the Schema Evolution and the
row-level user privs branches.

Are there any "best practices" tips for swapping among them for
testing/experimenting?  Preferably without causing /too/ much
damage. :)

Thanks,

-tkc

On 12/14/06, Tim Chase <django.us...@tim.thechases.com> wrote:

I can only speak to Unix-based systems, but what I've done with lots
of things of this nature is write a little script which will rewrite
the .profile and .rc files for my shell to have various environment
variables (in this case, PYTHONPATH) point to different locations,
then use 'source' to force the files to reload.

--
"May the forces of evil become confused on the way to your house."
  -- George Carlin

On 12/14/06, Tim Chase <django.us...@tim.thechases.com> wrote:

Hey Tim,

I've added some more instructions to the "Using branches" part of our
documentation. Let me know if this helps.

http://www.djangoproject.com/documentation/contributing/#using-branches

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com

The patch I previous sent in mostly adds a couple functions to the
psycopg2 backend in the introspection module.  The only big changes
that affect the mainline django code are in django.core.management.

I'm using my patches, so that's been tested through 3 schema updates
in production.  As previously mentioned -there are those 2 remaining
bugs in the schema-evo code, but they're not show stoppers for me.

I see no reason why people need to use the latest trunk to test beta
quality features.  That just makes no sense - if you need beta
features - use a branch, or apply my patch to your own copy of trunk.
Is there a problem that I'm not seeing?  I'm quite sure that all the
code that my patch modifies hasn't been touched in over a month, so it
should apply cleanly to trunk.

vic

On 12/14/06, Todd O'Bryan <toddobr...@mac.com> wrote:

--
"Never attribute to malice that which can be adequately explained by
stupidity."  - Hanlon's Razor

Yes, thanks!  I think with a combo of symlinks and the other
response regarding a personalized environment, I think my final
solution will be to

1) adjust my $PYTHONPATH to point to a ~/projects/django symlink
2) redirect that symlink to point to various branches of interest

My kudos to the Django team...a while back I had started my own
framework (what Python programmer hasn't? ;) with a number of
goals in mind.  However, as I learned about Django, I found that
it already had implemented nearly all of what I wanted.  There
are still the two aforementioned issues that are solved in
branches (row-level permissions and schema evolution), but it
looks like those will come down the pike and eventually merge
with the trunk.  Thanks!

-tkc

On 2006-12-14, Victor Ng <crankyco...@gmail.com> wrote:

Victor, thanks very much for your efforts on this.  I will be starting
a new project soon and would like to test schema evolution.

Can you explain the differences between your patch and the schema
evolution branch itself?  Are you proposing that your changes be rolled
be rolled into the schema evolution branch?

Steve

On 2006-12-14, Russell Keith-Magee <freakboy3...@gmail.com> wrote:

That's good to hear.

I see.  It sounds like it may be a bit of moving target, due to changes
in core.  Maybe a good goal would be to get the implementation up to
date (maybe that's what Victor has done?) and make sure the test
coverage is in place (I'm not sure the status of this?).  

Then publicize it a bit and call for testers to essentially try what
would essentially be a release candidate.  

Maybe someone is already thinking along these lines?  It seems like the
branch could benefit from a "champion" who actively pushes it through.

Steve

On 12/15/06, Steve Hutton <gm...@featurecomplete.com> wrote:

Keeping everything up to date is the role of the champion you mention below.

The call for testing should be made by the champion; unless I've
missed a post, I don't think this has happened. The branch hasn't been
touched for 2 months, but I don't know if this is because the work is
done, or because Derek Anderson (the SOC participant running the
schema-evolution project) hasn't had time to work on it, or if he has
just lost interest.

Either way, the ultimate goal would be to convince Adrian that the
branch is stable, get him to take a look at it, and merge the branch
into the trunk. Adrian is pretty busy, so I'm guessing he would be
looking for some community consensus that the branch is complete and
stable before he spends any time looking at it.

This 'champion' role should be performed by Derek and/or Kenneth
Gonsalves (the SOC mentor for the project). If you want to help out,
check out the branch, and take it for a test drive. If what you find
is stable, let the community know; if it isn't, log bugs (and patches
if you can) until it is.

Yours,
Russ Magee %-)

On 12/14/06, Adrian Holovaty <holov...@gmail.com> wrote:

No mention of the cross-platform path files? I would say they are
easier to edit (comment/uncomment a line in a file) than symlink and
as they are a python feature (not an OS feature) they work on any OS.

Just place the file `django.pth` in your site-packages dir (or
anywhere on your pythonpath). The file would look something like this:

    /path/to/django_src
    # /path/to/django_branch1
    # /path/to/django_branch2

If you want to switch to branch2, just edit the file and comment line
one and uncomment line 3. It's that easy.

For more on path files see this page:
http://bob.pythonmac.org/archives/2005/02/06/using-pth-files-for-pyth...

--
----
Waylan Limberg
way...@gmail.com

On 12/15/06, Waylan Limberg <way...@gmail.com> wrote:

Hey Waylan,

A patch to the documentation would be welcome. :)

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com

On 12/15/06, Adrian Holovaty <holov...@gmail.com> wrote:
Done: http://code.djangoproject.com/ticket/3147

I hope it is clear enough.

--
----
Waylan Limberg
way...@gmail.com

FYI - the schema evolution code submitted from the SoC project doesn't
work, so keeping it up to date is a moot point.  There have been
several posts from people trying to use it where the SoC version of
the schema evolution code just halts.

The implementation in my patch is basically a complete rewrite.  The
only things I've kept are the 'aka' attribute definitions in the Field
and the Model.Meta classes.

vic

On 12/15/06, Steve Hutton <gm...@featurecomplete.com> wrote:

The tests I've added to the schema evolution code are fully automated,
but they aren't currently integrated into the main Django test suite.
The main reason for this is that the schema evolution code relies on a
Postgresql backend.  You can't run the tests against MySQL or SQLite.
MySQL doesn't work because I haven't looked at it yet, and SQLite is
probably going to be impossible to implement since sqlite doesn't
implement a proper ALTER TABLE statement.

Test coverage is ~44 test cases.  There are currently 2 known problems
in the code from a 'correctness' standpoint, and there is a ton of
refactoring that needs to be done before it should be considered for
merging into the trunk.

The code is *messy*.  Writing schema evolution went hand in hand with
me learning how Django's schema generation worked.

That said - the modifications are basically:

 * decomposing a lot of functions in django.core.managemeent so that
they're more re-usabl
 * removing all REFERENCES clauses in CREATE TABLE statements and
changed the code to just emit CREATE CONSTRAINT statements.  This
makes everything more consistent and easier for me to figure out what
constraints to remove later if the schema is changed.
 * rewriting the syncdb function since it missed creating some foreign
key relationships sometimes
 * most of the heavy lifting is in django.contrib.schemaevo - again,
it's all Postgresql specific and I just ripped out the code that I
wrote into django.core.management and dropped it into
django.contrib.schemaevo.  Minor changes were made to get my tests to
run again.

If someone wants to open up commit access on the django branch for
schema evo, I'll be happy to bring the schema evo code up to date with
the trunk.

I *do* need help in tidying it up to get it merged back to trunk
though.  I'm hard pressed for time right now and I could use some
help.

vic

--
"Never attribute to malice that which can be adequately explained by
stupidity."  - Hanlon's Razor

On 2006-12-17, Victor Ng <crankyco...@gmail.com> wrote:

Ah, I see.  Your comments and the fact that the schema evolution branch
has not been actively updated lately or championed make me disinclined
to spend time testing it.

Perhaps your patches could form the basis of an alternative schema
evolution branch?

Steve

On 2006-12-15, Russell Keith-Magee <freakboy3...@gmail.com> wrote:

Absolutely makes sense to me.

Right, also makes sense.  It would be nice to get Derek and Kenneth's
thoughts on the status of the branch and what they think the remaining
tasks are to make it "merge ready".

Thanks, I plan on doing just that in the coming weeks.  However I'm not
sure yet if I'll be trying Victor's alternative implementation or the
existing branch.

Steve

On 2006-12-17, Victor Ng <crankyco...@gmail.com> wrote:

I see, 44 test cases sound like a good start.

[...]

It sounds like your code has potential to be a full fledged alternative
implementation.  I've never looked at the django internals, but I might
be able to help nevertheless.  I will have some time starting about 10
days from now.  At this point it would be a question of where I spend my
time - testing the existing branch code or helping test and/or clean up
and integrate your code.

The fact that you've already tried the existing schema evolution branch
and spent this much effort so far on an alternative implementation
certainly makes me lean towards your code.  But it would be nice from
a logistical point of view if we could have access it to it from a
repository somewhere.

Steve

On 2006-12-11, Victor Ng <crankyco...@gmail.com> wrote:

[...]

Is this a departure from the Proposal in the wiki[1]?
It says:

"Introspection + migration
This approach is a combination of Automatic db introspection and
Automatically applied migration code -- one command produces a migration
script, you tickle your version number, and the syncdb runs the
migrations"

Steve
[1] http://code.djangoproject.com/wiki/SchemaEvolution

Hi, sorry for the long delay in replying.   Holiday season and work
craziness is getting in the way of writing free software - which is
really the fun part isn't it? ;)

On 12/17/06, Steve Hutton <gm...@featurecomplete.com> wrote:

It's a little different, but barely.

The current codebase emits an SQL file with all the SQL statements to
evolve the database,   There's no provision to save the version
anywhere, but that would be simple to do.

If nobody want's to champion the code, I can volunteer ~3-5 hours a
week to work on this stuff.  My own team needs schema evolution in the
long run anyway - so it's well worth my time to make sure this
eventually gets back to trunk.

I won't be able to do too much until the new year though - so that
said - can somebody open up the schema evolution branch, or better -
open up a new branch from trunk so that we can start hacking away at
this problem?

victor "i hate parking lots at christmas" ng

--
"Never attribute to malice that which can be adequately explained by
stupidity."  - Hanlon's Razor

On 2006-12-22, Victor Ng <crankyco...@gmail.com> wrote:

:-)

Sounds great to me.  Any core devs care to open a new branch for this?

Thanks,
Steve

On 28/12/06, Steve Hutton <gm...@featurecomplete.com> wrote:

From my earlier involvement I know Victor's code is based on the
current schema evolution branch so I think it makes sense for his
fixes to be checked into it. Also it would be less confusing for the
rest of the django community if there was a single branch, considering
that there is atleast one enquiry a week about it.

Victor, have you got a ticket open with your patch? It would be worth
opening one so that other people could try it out while you are
waiting for commit access to be granted.

regards

matthew