Must a Django Database Support Migrations?

[Michael Manfre]

Must a Django Database Support Migrations?

Django 1.7 adds schema migration support and this behavior is currently required

to be able to effectively use Django and run the test suite. I opened ticket

#21841 [1], which requests adding a DatabaseFeature to disable schema

migrations. This ticket prompted an IRC discussion between Russell and myself

that ended with the question, must a database backend support schema migrations

to be compliant with Django 1.7? We're hoping to get opinions from others about

this to figure out how to proceed.

Database backend compliance is defined by its ability to pass Django's test

suite. A fully compliant database can pass 100% of tests dealing with the

database. If a database backend raises NotImplementedError for schema_editor, it

is currently 0% compliant because it cannot create the database tables.

If database backends are not required to support migrations, then there would

need to be some form of fallback behavior to provide a Django 1.6 syncdb-esque

behavior. It would create the schema without attempting alterations.

Why Disable Migrations?

1) Not all projects need schema migrations or want Django to manage the

migrations. My primary project has Django working against a legacy database and

relies upon an external tool (and a DBA) to manage schema changes in our

environment. This workflow uses syncdb to create the tables in to a temporary

database and then compares it will an external tool that generates an SQL diff.

This diff is tested by the DBA before being applied to the production database.

This basic workflow will continue even after upgrading to Django 1.7.

This scenario does not necessarily need to be handled as a database feature.

Russell's idea during the IRC discussion was a database configuration that acts

as a safety to prevent migrations from being applied; similar to managed models.

2) NoSQL databases don't necessarily have an enforced schema. What would

migration mean for NoSQL? Would it noop or attempt to update existing data? Do

we even care because NoSQL is not officially supported by Django?

3) The database backend or driver doesn't support altering the schema in a way

that would work with Django's schema migrations API. I can't think of any that

would be used with Django, but people still use mysql despite its many faults,

so anything is possible.

Potential Issues With Optional Support

1) If migration support is optional, a concern was raised that backends that

don't support it would give users a lesser experience and potentially make

Django look bad.

While this might be possible, the same concern holds true for any existing

database backend that is buggy, doesn't fully implement the existing API, or the

underlying database itself is slow or lacks basic functionality (e.g.

referential integrity, ACID compliance, etc).

2) Currently unknown level of effort or complexity to allow the migrations

to fallback to a Django 1.6 syncdb-esque behavior.

Feedback

Please share your questions and comments on the mailing list.

Regards,

Michael Manfre

[1] https://code.djangoproject.com/ticket/21841

[Russell Keith-Magee]

On Wed, Jan 22, 2014 at 1:51 PM, Michael Manfre <mma...@gmail.com> wrote:

Must a Django Database Support Migrations?

Django 1.7 adds schema migration support and this behavior is currently required

to be able to effectively use Django and run the test suite. I opened ticket

#21841 [1], which requests adding a DatabaseFeature to disable schema

migrations. This ticket prompted an IRC discussion between Russell and myself

that ended with the question, must a database backend support schema migrations

to be compliant with Django 1.7? We're hoping to get opinions from others about

this to figure out how to proceed.

I'm weighing in here to give my side of the argument :-)

 

Database backend compliance is defined by its ability to pass Django's test

suite. A fully compliant database can pass 100% of tests dealing with the

database. If a database backend raises NotImplementedError for schema_editor, it

is currently 0% compliant because it cannot create the database tables.

If database backends are not required to support migrations, then there would

need to be some form of fallback behavior to provide a Django 1.6 syncdb-esque

behavior. It would create the schema without attempting alterations.

Why Disable Migrations?

1) Not all projects need schema migrations or want Django to manage the

migrations. My primary project has Django working against a legacy database and

relies upon an external tool (and a DBA) to manage schema changes in our

environment. This workflow uses syncdb to create the tables in to a temporary

database and then compares it will an external tool that generates an SQL diff.

This diff is tested by the DBA before being applied to the production database.

This basic workflow will continue even after upgrading to Django 1.7.

This scenario does not necessarily need to be handled as a database feature.

Russell's idea during the IRC discussion was a database configuration that acts

as a safety to prevent migrations from being applied; similar to managed models.

I have no problem with this use case. My problem with the specific proposal in #21841 is that the database backend is the wrong place to be making this distinction. The decision as to whether the backend supports migrations is independent of whether the user wants to use them. The PostgreSQL supports migrations; that doesn't mean I want to use them.

If model-level Meta.managed isn't enough to disable migrations, then a settings.DATABASES flag may be called for, but that's different to what was originally proposed.

 

2) NoSQL databases don't necessarily have an enforced schema. What would

migration mean for NoSQL? Would it noop or attempt to update existing data? Do

we even care because NoSQL is not officially supported by Django?

This is a problem that we don't have (since we don't support NoSQL), so the question is completely hypothetical. However, I don't have any difficulty believing that the *concept* of migrations could exist on NoSQL databases - the implementation just won't look anything like "ALTER TABLE ADD COLUMN". 

This means we're really only left with:

 

3) The database backend or driver doesn't support altering the schema in a way

that would work with Django's schema migrations API. I can't think of any that

would be used with Django, but people still use mysql despite its many faults,

so anything is possible.

This one, which seems unlikely. SQL is a pretty fungible standard, and yet we've been able to coerce SQLite, MySQL, PostgreSQL and Oracle into the task. From what I *do* know about the migrations implementation, it's largely driven by high level primitives -- add a column, remove a column, alter a column etc. These high level primitives don't enforce anything significant on the underlying SQL -- and if it does, I'd argue that's a bug in the abstraction.

There's also option 4: The developer of a particular backend doesn't have the time or inclination to add support for migrations.

 

Potential Issues With Optional Support

1) If migration support is optional, a concern was raised that backends that

don't support it would give users a lesser experience and potentially make

Django look bad.

While this might be possible, the same concern holds true for any existing

database backend that is buggy, doesn't fully implement the existing API, or the

underlying database itself is slow or lacks basic functionality (e.g.

referential integrity, ACID compliance, etc).

 

This is true, but at that point we can point at the database and say "it's not our fault". MySQL *does* do dumb things with indexes (and subqueries, and referential integrity, and.… :-) Oracle *is* slow to do certain table setup operations. That's not our fault, and we can assign blame/point out limitations as appropriate.

If we explicitly add APIs that allow people to say "I've got a Django 1.7 compliant database API, but it doesn't support most of the API", then we've explicitly given permission for people to half-ass things, and that's something we *do* have to accept blame for. If we have this API, people will produce non-migrating 1.7 backends, and then we'll have ghettos in the community that can't use the banner features of 1.7.

Yes, adding migrations is a big thing, so writing a Django 1.7-compatible database backend will be a big task. But I don't see the value in providing a way to explicitly support a half-way solution.

Yours,

Russ Magee %-)

[Shai Berger]

Hi,

(quotations reordered a little)

On Wednesday 22 January 2014 14:48:25 Russell Keith-Magee wrote:
> On Wed, Jan 22, 2014 at 1:51 PM, Michael Manfre <mma...@gmail.com> wrote:

> > *Why Disable Migrations?*

> >
> > 1) Not all projects need schema migrations or want Django to manage the
> > migrations. My primary project has Django working against a legacy
> > database and relies upon an external tool (and a DBA) to manage schema
> > changes in our environment. This workflow uses syncdb to create the tables
> > in to a temporary database and then compares it will an external tool that
> > generates an SQL diff. This diff is tested by the DBA before being applied
> > to the production database. This basic workflow will continue even after
> > upgrading to Django 1.7.
> >
> > This scenario does not necessarily need to be handled as a database
> > feature. Russell's idea during the IRC discussion was a database
> > configuration that acts as a safety to prevent migrations from being
> > applied; similar to managed models.
>
> I have no problem with this use case. My problem with the specific proposal
> in #21841 is that the database backend is the wrong place to be making this
> distinction. The decision as to whether the backend supports migrations is
> independent of whether the user wants to use them. The PostgreSQL supports
> migrations; that doesn't mean I want to use them.
>
> If model-level Meta.managed isn't enough to disable migrations, then a
> settings.DATABASES flag may be called for, but that's different to what was
> originally proposed.
>

I was going to say that the argument confuses the backend level with the
project level. Russell beat me to it :)

> > 2) NoSQL databases don't necessarily have an enforced schema. What would
> > migration mean for NoSQL? Would it noop or attempt to update existing
> > data? Do we even care because NoSQL is not officially supported by Django?
>
> This is a problem that we don't have (since we don't support NoSQL), so the
> question is completely hypothetical. However, I don't have any difficulty
> believing that the *concept* of migrations could exist on NoSQL databases -
> the implementation just won't look anything like "ALTER TABLE ADD COLUMN".
>

+1 Russell, again.

> This means we're really only left with:
> > 3) The database backend or driver doesn't support altering the schema in a
> > way that would work with Django's schema migrations API. I can't think of
> > any that would be used with Django, but people still use mysql despite its
> > many faults, so anything is possible.
>
> This one, which seems unlikely.

Actually, SQLite "doesn't support altering the schema in a way that would work
with Django's schema migrations API". The SQLite backend bends over backwards
to comply; its implementation of "alter column" is "recreate the table,
copying the data". AFAIK it doesn't quite get everything right, even with the
somersaults it pulls; and yet, it's close enough for comfort.

So, in order to really count as "can't support", a backend needs to be
crippled to a level where implementing django-1.6-syncdb is probably
impossible as well.

>
> There's also option 4: The developer of a particular backend doesn't have
> the time or inclination to add support for migrations.
>

which brings me back to

> > Database backend compliance is defined by its ability to pass Django's
> > test suite. A fully compliant database can pass 100% of tests dealing with
> > the database. If a database backend raises NotImplementedError for
> > schema_editor, it is currently 0% compliant because it cannot create the
> > database tables.
> >
> > If database backends are not required to support migrations, then there
> > would need to be some form of fallback behavior to provide a Django 1.6
> > syncdb-esque behavior. It would create the schema without attempting
> > alterations.
> >

and

> > *Potential Issues With Optional Support*

> >
> > 2) Currently unknown level of effort or complexity to allow the migrations

> > to fallback to a Django 1.6 syncdb-esque behavior.

An alternative, which I think will have much lower costs, and send the right
message about such backends is:

A) Django only changes the schema through migrations; if you don't want
migrations in your backend or your project, than the schema changes need to
happen some other way, using an external tool.

B) Allow the test suite to run on an existing schema. The Oracle backend
already does this (badly) -- its six specific TEST_* parameters are awkwardly
named, but allow you to say exactly which schema to use for testing, and
whether or not you want it created. If this is made more general, then testing
can be made to not depend on migrations; with no migrations, you will need to
take care of the test database yourself -- just like you take care of the
production database.

For Django's own test-suite, tests which need migrations can even be
automatically skipped if getting a schema_editor raises a NotImplementedError;
allowing a backend to easily (well, relatively easily) pass the suite without
implementing migrations.

Shai.

[Russell Keith-Magee]

This last bit is the critical part. At present, we don't have any infrastructure to allow tests to be skipped because of a lack of migration support. All Django's officially supported backends support migrations, so there hasn't been any need to skip any tests. 

The question is whether we should allow for this for third party backends. 

Historically, the test for a "Django 1.X supported backend" has been "Passes the Django 1.X test suite without errors."; database features have been added to allow checking for specific known problems on specific backends (vendor-specific index name size limits, for example), but not for entire features. The proposal here is to provide a similar flag for migrations, enabled for all the builtin backends, so that you could have a "compliant" 1.7 database backend without supporting migrations. I'd argue this isn't something we want to encourage.

Yours,

Russ Magee %-)

[Aymeric Augustin]

2014/1/22 Michael Manfre <mma...@gmail.com>

Currently unknown level of effort or complexity to allow the migrations

to fallback to a Django 1.6 syncdb-esque behavior.

Assuming that cost isn't too high, another concern is our ability to

maintain this feature: since none of the core backends will use it,

it may fall behind.

For your production use case, you could get away with `./manage.py

sqlall` which is unchanged in 1.7. That will give you a new  schema

you can diff with the current schema.

If I understand your point regarding Django's test suite, it doesn't

even start because it uses the migration API to create the tables.

You could probably solve that with a custom test runner that

overrides table creation, basically to pipe the output of `sqlall` to

the database. That area of the code base isn't particularly complex.

Let's discuss on IRC if you need pointers.

Assuming this workaround provides a short term solution, it may be

bundled with the database backends that need it. I think that would

be good enough. I share Russell's concern about endorsing a partial

solution.

I hope this helps,

-- 

Aymeric.

[Shai Berger]

I agree -- it wasn't my intention to mark such backends as fully supported,
but as "supported with a caveat".

However, I do think the ability to say "I've got this half of the
functionality covered" is important. Further, since the DML part of SQL is
much more consistently implemented than the DDL part, the part without
migrations is a lot easier to implement than the migrations; the validation
should be pretty encouraging for 3rd-party backends.

So, I rephrase my suggestion for the Django tests:

For Django's own test-suite, tests which need migrations can be
automatically skipped if Russell's suggested "no migrations" DATABASES
setting is set; allowing a backend to easily validate its migration-less
part.

Shai.

[Andrew Godwin]

My thoughts in brief on this:

 - Database backends don't support migrations - they support schema alteration via SchemaEditor. This could be used separately from migrations if something wants it, and is meant to be an API on its own, so the backend is not the place to say if you want migrations or not.

 - There is some merit to the ability to turn off migrations on a per-backend basis, via a DATABASES setting, but bear in mind that routers already let you do this (with the allow_migrate method). The only extra thing it would achieve is not having Django record migrations in the django_migrations table, but it also looks like it would be useful for tests if you hadn't written that part yet.

 - I feel like a DB backend should at least implement SchemaEditor even if it returns NotImplementedError for most of the endpoints; even in the weirdest relational system, you can still manage create/delete model without too much difficulty.

 - Bear in mind that the plan is to remove DatabaseCreation entirely in favour of SchemaEditor in a few releases' time (and backends are more than welcome to make DatabaseCreation use SchemaEditor behind the scenes if they want), so at that point if you don't implement it the backend is only ever good for querying, which to me feels like an incomplete backend.

 - I'm not sure what the future of ./manage.py sqlall is, but it's going to have to be ported to SchemaEditor in future anyway, so it's only useful in the transition.

Looking at the discussion, I think the best thing to do is:

 - Make the schema and migrations test skip if connection.schema_editor() raises a NotImplementedError (not too hard, we can implement connection.has_schema_editor as a boolean to switch on)

 - Provide some way to skip the "creating models" part of test setup, so SchemaEditor is never needed during it

I still think all the current third-party backends should be able to provide a partial SchemaEditor implementation though, as at minimum they all have the DatabaseCreation code in place to make new models. Bear in mind that the ./manage.py sqlmigrate command lets you turn migrations into SQL scripts to send to your DBA (and many DBAs appreciate having some SQL they can work from - I know ours do), so having the ability to make that SQL is useful even if Django never runs it.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/33933764.nuWK3GA93E%40deblack.

For more options, visit https://groups.google.com/groups/opt_out.

[Michael Manfre]

On Wed, Jan 22, 2014 at 6:40 AM, Andrew Godwin <and...@aeracode.org> wrote:

My thoughts in brief on this:

 - Database backends don't support migrations - they support schema alteration via SchemaEditor. This could be used separately from migrations if something wants it, and is meant to be an API on its own, so the backend is not the place to say if you want migrations or not.

I've been using migrations to refer to migrations and the the schema alterations done by SchemaEditor since migrations depend on SchemaEditor being able to do the alterations it wants. I'll try to respect the distinction going forward to avoid confusion.

My request would probably be better read as "Must a database backend support schema alterations?", which also implies its ability to do a migration that alters the schema. Data only migrations would still technically be possible without the ability to alter the schema, but I'm not sure if that functionality would be as useful by itself.

 - There is some merit to the ability to turn off migrations on a per-backend basis, via a DATABASES setting, but bear in mind that routers already let you do this (with the allow_migrate method). The only extra thing it would achieve is not having Django record migrations in the django_migrations table, but it also looks like it would be useful for tests if you hadn't written that part yet.

 - I feel like a DB backend should at least implement SchemaEditor even if it returns NotImplementedError for most of the endpoints; even in the weirdest relational system, you can still manage create/delete model without too much difficulty.

I think we all agree that any database backend must continue to be able to create or delete schema objects, or provide something like Oracle's TEST_CREATE setting to allow that to be done as a non-Django step. I think it is reasonable to require any backend to support that portion of SchemaEditor.

I think we should have a property on DatabaseFeature that sniffs SchemaEditor to figure out whether it supports alterations. That would let us use the existing database feature decorators to skip tests.

 - Bear in mind that the plan is to remove DatabaseCreation entirely in favour of SchemaEditor in a few releases' time (and backends are more than welcome to make DatabaseCreation use SchemaEditor behind the scenes if they want), so at that point if you don't implement it the backend is only ever good for querying, which to me feels like an incomplete backend.

I'm not sure I like this generic plan, but I don't have all of the details and the discussion can be deferred until a later date. There is some value in having a distinction between something that creates just the databases (or files) and something that creates the schema inside of the databases.

Other than time constraints, is there a reason not to update DatabaseCreation in Django 1.7 so that is little more than a proxy to SchemaEditor for the create/delete sql? I think we all agree that any database backend must be able to create/delete the schema, so I don't think it makes much of a difference whether that happens in DatabaseCreation or SchemaEditor. All of the database backend API has been deemed internal API on several occasions, so it should be safe to make this sort of change and also not necessarily be subject to the alpha feature freeze. I'd be happy to work on this.

 

 - I'm not sure what the future of ./manage.py sqlall is, but it's going to have to be ported to SchemaEditor in future anyway, so it's only useful in the transition.

Looking at the discussion, I think the best thing to do is:

 - Make the schema and migrations test skip if connection.schema_editor() raises a NotImplementedError (not too hard, we can implement connection.has_schema_editor as a boolean to switch on) 

 - Provide some way to skip the "creating models" part of test setup, so SchemaEditor is never needed during it

I think it's okay to require SchemaEditor to exist with minimal functionality. It would be ideal if this minimal SchemaEditor was provided. Disabling the alterations with a DatabaseFeature would make the existing BaseDatabaseSchemaEditor on par with BaseDatabaseCreation's ability to create/delete schema objects.

 

I still think all the current third-party backends should be able to provide a partial SchemaEditor implementation though, as at minimum they all have the DatabaseCreation code in place to make new models. Bear in mind that the ./manage.py sqlmigrate command lets you turn migrations into SQL scripts to send to your DBA (and many DBAs appreciate having some SQL they can work from - I know ours do), so having the ability to make that SQL is useful even if Django never runs it.

Agreed. DBAs are happier to have a non-blank starting point.

Regards, 

Michael Manfre

[Shai Berger]

On Wednesday 22 January 2014 19:15:08 Michael Manfre wrote:
>
> I think we all agree that any database backend must continue to be able to
> create or delete schema objects, or provide something like Oracle's
> TEST_CREATE setting to allow that to be done as a non-Django step.

Actually, I don't; I think TEST_CREATE should be provided by the test runner,
so that it's available for all backends. It can be useful for a whole set of
situations, not just a dysfunctional backend.

(but that's a little off topic, I know).

Shai.

[Andrew Godwin]

On Wed, Jan 22, 2014 at 5:15 PM, Michael Manfre <mma...@gmail.com> wrote:

My request would probably be better read as "Must a database backend support schema alterations?", which also implies its ability to do a migration that alters the schema. Data only migrations would still technically be possible without the ability to alter the schema, but I'm not sure if that functionality would be as useful by itself.

I would say "yes", if something wants to claim to be a fully-supported Django backend. In this day and age being able to alter the schema is something that a framework/ORM should be providing, IMO.

 

I think we all agree that any database backend must continue to be able to create or delete schema objects, or provide something like Oracle's TEST_CREATE setting to allow that to be done as a non-Django step. I think it is reasonable to require any backend to support that portion of SchemaEditor.

I think we should have a property on DatabaseFeature that sniffs SchemaEditor to figure out whether it supports alterations. That would let us use the existing database feature decorators to skip tests.

I'm not sure if sniffing is the correct way, but I agree we should have either that or a direct attribute on Features that can say if a backend supports certain operations (we already have this to some extent with parts of SchemaEditor, with things like supports_foreign_keys), and then we can skip tests based on those.

 

I'm not sure I like this generic plan, but I don't have all of the details and the discussion can be deferred until a later date. There is some value in having a distinction between something that creates just the databases (or files) and something that creates the schema inside of the databases.

I wouldn't mind keeping actual database creation separate - it doesn't really belong in SchemaEditor and is mostly used for SQLite and testing anyway - but the bits of DatabaseCreation that make tables have to go.

 

Other than time constraints, is there a reason not to update DatabaseCreation in Django 1.7 so that is little more than a proxy to SchemaEditor for the create/delete sql? I think we all agree that any database backend must be able to create/delete the schema, so I don't think it makes much of a difference whether that happens in DatabaseCreation or SchemaEditor. All of the database backend API has been deemed internal API on several occasions, so it should be safe to make this sort of change and also not necessarily be subject to the alpha feature freeze. I'd be happy to work on this.

It's just time constraints. While that API is technically private, some things use it and I've done a lot of deep changes as it is without diving in there and possibly introducing backwards-incompatible code (we'd lose the syntax highlighted SQL too for command line output but that's not a massive loss, and should be done by e.g. pygments anyway). There's several bits in there and introspection that are only around because removing them was too much of an ask to do in the same release we add all the new stuff. I do need to stick a PendingDeprecationWarning on it though.

 

I think it's okay to require SchemaEditor to exist with minimal functionality. It would be ideal if this minimal SchemaEditor was provided. Disabling the alterations with a DatabaseFeature would make the existing BaseDatabaseSchemaEditor on par with BaseDatabaseCreation's ability to create/delete schema objects.

When you say "provided", what do you mean? The current base SchemaEditor provides a lot and once we've fixed up some of the issues you raised with the MSSQL patch it should be a lot better, certainly within reason to inherit it and just set the alteration stuff to NotImplementedError.

 

Andrew

[Michael Manfre]

On Wed, Jan 22, 2014 at 12:26 PM, Andrew Godwin <and...@aeracode.org> wrote:

On Wed, Jan 22, 2014 at 5:15 PM, Michael Manfre <mma...@gmail.com> wrote:

My request would probably be better read as "Must a database backend support schema alterations?", which also implies its ability to do a migration that alters the schema. Data only migrations would still technically be possible without the ability to alter the schema, but I'm not sure if that functionality would be as useful by itself.

I would say "yes", if something wants to claim to be a fully-supported Django backend. In this day and age being able to alter the schema is something that a framework/ORM should be providing, IMO.

The only backends that can ever claim "fully-supported" are those included in Django. All other backends rely upon Django internals that can, and do change in backwards incompatible ways.

There is a difference between the framework providing a feature and a feature being useful/required for a project/environment. E.g. GIS, template engine, or if we want to stick to just the ORM features, select related, distinct on, etc.

The follow up question is, if a backend doesn't support schema alterations, should Django stop it from working? This would go against the common mantra of "we won't include it in Django, but we'll make it possible."

I think it's okay to require SchemaEditor to exist with minimal functionality. It would be ideal if this minimal SchemaEditor was provided. Disabling the alterations with a DatabaseFeature would make the existing BaseDatabaseSchemaEditor on par with BaseDatabaseCreation's ability to create/delete schema objects.

When you say "provided", what do you mean? The current base SchemaEditor provides a lot and once we've fixed up some of the issues you raised with the MSSQL patch it should be a lot better, certainly within reason to inherit it and just set the alteration stuff to NotImplementedError.

I prefer having an explicit DatabaseFeature to disable alterations. With that feature, the base SchemaEditor could check that feature to either noop or raise NotImplementedError for the alterations. Requiring a backend to both set the feature and then noop/raise all of the methods controlled by the feature seems redundant.

Regards,

Michael Manfre

[Hannu Krosing]

On 01/22/2014 08:59 AM, Shai Berger wrote:
>> This means we're really only left with:
>>> 3) The database backend or driver doesn't support altering the schema in a
>>> way that would work with Django's schema migrations API. I can't think of
>>> any that would be used with Django, but people still use mysql despite its
>>> many faults, so anything is possible.
>> This one, which seems unlikely.
> Actually, SQLite "doesn't support altering the schema in a way that would work
> with Django's schema migrations API". The SQLite backend bends over backwards
> to comply; its implementation of "alter column" is "recreate the table,
> copying the data". AFAIK it doesn't quite get everything right, even with the
> somersaults it pulls; and yet, it's close enough for comfort.

What about moving this kind of migration support from database driver
to django core as an universal fallback ?

(btw, creating a new table and copying the data is what PostgreSQL
does in the background for ALTER TABLE ALTER COLUMN TYPE for many,
but not all column type changes)

My argument for this is that it is remarkably hard to do schema
migrations properly even for backends which "support it". You do not
want to end up with 3 days downtime "while the database migrates"
because you trusted django to do the right thing and it worked ok in
your (small) test database.

Automatic migrations are very useful at development time, but need
lots of TLC when done in production environment. They are hard enough
to orchestrate propery at pure SQL level. Adding an extra layer on top
with largely undefined performance characteristics makes things much
more complikated still.

Just my two 2 cents
Cheers

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

[Andrew Godwin]

On Wed, Jan 22, 2014 at 9:31 PM, Hannu Krosing <ha...@krosing.net> wrote:

What about moving this kind of migration support from database driver
to django core as an universal fallback ?

Definitely not! It's an awful way of doing migrations and only exists because people use SQLite a lot for development; I'd personally like to remove it someday, but that seems unlikely.

 

My argument for this is that it is remarkably hard to do schema
migrations properly even for backends which "support it". You do not
want to end up with 3 days downtime "while the database migrates"
because you trusted django to do the right thing and it worked ok in
your (small) test database.

Automatic migrations are very useful at development time, but need
lots of TLC when done in production environment. They are hard enough
to orchestrate propery at pure SQL level. Adding an extra layer on top
with largely undefined performance characteristics makes things much
more complikated still.

And this is why we have the sqlmigrate command so that you can get the SQL and tweak it if you like without having to write it manually in the first place.

South/django.db.migrations is actually relatively useful in production on the bottom 95% of websites provided you're using PostgreSQL; most operations it performs don't lock anything, and it's probably better than your average programmer at writing DDL SQL. If you're in that top 5% and have an actually sizeable database, then yes, you should look at the scripts it produces, but it's still very useful for developers to use as they make new features and the SQL it makes is usually an excellent start if you want to hand-run the migration (and if it was just an ADD COLUMN with NULL, then I'd let Django do it anyway as I know that's instantaneous and lock-free, and Django handles all the transaction management around migrations so it can roll back if things error).

Andrew

[Ivan VenOsdel]

From Andrew: "The only extra thing it would achieve is not having Django record migrations in the django_migrations table"

The Django Users thread on how to keep this table from being created seemed to result in the 'solution' being either to stay with 1.6 or comment the relevant lines in the Django code base. Is there really no other way?

I love the new migrations facilities in Django 1.7 and was a contributor to the kickstarter but please keep in mind that many legacy DB projects are not avoiding migrations because they want to. IMO it's usually because they are forced to for some (usually political) reason where they don't have control over the schema. Forcing a perpetually empty django_migrations table pretty much locks out those users.

I see that people are worried about encouraging the non-use of migrations but might I suggest following the Zen of Python and making migrations the "one obvious way to do it" and turning them off the not obvious way.

Ivan

[Andrew Godwin]

Hi Ivan,

I'm not sure what you're asking here - are you asking to have a way to not have Django create the migrations recording table? I was under the impression that it was only created when migrate was run (at least, that was my original intention) so if you're managing your own schema just don't run migrate. Is there something else that's not working right, or is that being made too optimistically?

Andrew

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e3bd21ad-6aec-4686-a037-69d486f969a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Markus Holtermann]

Hey,

as soon as the MigrationRecorder is used there is a call to "ensure_schema" that forces the creation of the migration table. The runserver command (among others?) checks for unapplied migrations and thus creates the migration table.

/Markus

[Andrew Godwin]

Aha! Then, I would suggest redesigning MigrationRecorder to only make the table when an actual recording operation is done, and have it swallow the table not existing as "no migrations applied" the rest of the time, if people think that seems sensible.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/81dcde2e-babb-4714-829e-b9308f3672cd%40googlegroups.com.

[Marcin Nowak]

Hi. 

Just give people possibility to disable migrations functionality (remove it from core and put as a contrib app, if needed).

/BR

Marcin

[Markus Holtermann]

Hey Marvin,

Unfortunately this won't work as a few migration components are tightly integrated into remaining parts of Django, e.g. the field deconstruction.

/Markus

><https://groups.google.com/d/msgid/django-developers/e3bd21ad-6aec-4686-a037-69d486f969a9%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>Groups
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>send an

>>> email to django-develop...@googlegroups.com <javascript:>.

>>> To post to this group, send email to django-d...@googlegroups.com

>>> <javascript:>.

>>> Visit this group at
>http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>>

>https://groups.google.com/d/msgid/django-developers/81dcde2e-babb-4714-829e-b9308f3672cd%40googlegroups.com
>
>>>
><https://groups.google.com/d/msgid/django-developers/81dcde2e-babb-4714-829e-b9308f3672cd%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .

[Marcin Nowak]

Hi. 

Before v1.7 there were no migrations and Django was quite useful (for example with South as an 3rd party app). What happened then?

I think that migrations can be decoupled from core and you just don't want to do this for some unknown reason. 

I'll prepare a pull request if it will not be dropped because of some kind of ethos.

BR

Marcin

[Michael Manfre]

On Sun, Apr 26, 2015 at 8:33 PM, Marcin Nowak <marcin....@gmail.com> wrote:

Hi. 

Before v1.7 there were no migrations and Django was quite useful (for example with South as an 3rd party app). What happened then?

I think that migrations can be decoupled from core and you just don't want to do this for some unknown reason. 

I'll prepare a pull request if it will not be dropped because of some kind of ethos.

As the person who started this thread over a year ago with the motivation of making migrations support optional, I can definitively say that migrations will not be removed from Django. Any pull request attempting to do so will be declined. There are certain portions of Django that can (and likely should) be decoupled, the database API (including migrations) is not one of those pieces. One of the many reasons for it to remain in core is that it's useful being able to modify the models that ship with Django. E.g. increasing the size of user name field.

Regards,

Michael Manfre

[Marten Kenbeek]

Hi Marcin,

I don't believe it's some sort of ethos that got Andrew to collaborate with Django and create the current migration framework: It's the fact that pretty much everyone who used Django also used South that convinced Andrew to recreate its behaviour as a core Django app.

I'm not saying that Liquibase isn't useful -- I haven't really investigated the project you're pitching. I'm simply saying that migrations are considered a necessary feature for any decently-sized project, and currently Django's way of supplying this is its core migration framework. If you are able to patch Django's behaviour to not depend on its core migration framework, but to depend on a pluggable migration backend, I'm sure everyone will be very happy to see that as a Django core feature. But as is currently stand, it's a lot of work to recreate the Django migration framework as a pluggable backend, and it's not worth it if there isn't a tightly integrated migration framework that picks up the changes to Django models. 

It should still be quite easy to use Liquibase for your database management: simply set your models as `managed=False`, and you'll have full control over your database. You'll simply have to keep your Django models up-to-date to use them. 

With king regards,

Marten

Op maandag 27 april 2015 02:33:44 UTC+2 schreef Marcin Nowak:

[Andrew Godwin]

Hi Marcin,

I originally started developing the new Django migrations as a contrib-style app, but for reasons of complexity and maintenance, they were best moved into the database core itself - I can describe them at length if you like, but trust me that I tried to keep things as cleanly separable as possible where it made sense.

In fact, even now the migrations framework is built as separable, clean components - the schema alteration backend is very much separate from the migration file reader and writer, and it wouldn't be too hard to build an alternate migrations framework that used the schema alteration backend (which is essentially the common shared hard work of abstracting how to make changes to databases across all backends we support) but completely ignored the migration file stuff and overrode the existing commands. 

You can't "turn off" the migration support in the same way you can't "turn off" syncdb in 1.6 and below - they're the core way Django ships with of making tables (and now, modifying them). What you can do, though, is entirely override the behaviour with something custom if you so wish - it wouldn't be too difficult to build something that overrode migrate to just make tables the old way (using the new schema alteration backend). If someone's willing to pay me some consulting time I'll write it for them, with the caveat that it's probably not what anyone wants.

If there's a place in the migrations right now that's hard to override (for example, I imagine the check in runserver for unmade migrations might be something like this) I'm happy to take those as bugs as fix them, but asking to remove migrations from core is like asking us to ship a broken product - they're not there in addition to syncdb/Creation classes, they replaced them with cleaner, more featureful code.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2907b0fa-e705-4208-8138-01c75bac02fe%40googlegroups.com.