Re: [Django] #470: Add Field.db_default for defining database defaults
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Tim Graham):
* status: closed => new
* resolution: wontfix =>
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:19>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Paul Tiplady):
Further justification for this feature -- it makes zero-downtime DB
migrations easier: https://code.djangoproject.com/ticket/29266
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:20>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Ryan Hiebert (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:21>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:22>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Ian Foote (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:23>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Ryan Moore (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:24>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Václav Řehák (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:25>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Hannes Ljungberg (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:26>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Marcin Nowak (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:27>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Marcin Nowak):
Hi.
Django 3.x generates sql with drop default:
```
ALTER TABLE "x" ADD COLUMN "y" integer DEFAULT 0 NOT NULL CHECK
("y" >= 0);
ALTER TABLE "x" ALTER COLUMN "y" DROP DEFAULT;
```
The whole thing is about not adding DROP DEFAULT. How about some kind of
option for database backend?
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:28>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Vaibhav Awachat):
Currently using this https://github.com/3YOURMIND/django-add-default-value
to run migrations without downtime using MySQL in strict mode.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:29>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Marcin Nowak):
Hi. 15 years have passed since this important bug report. Every time we're
uploading new version of the application, which includes new not nullable
fields in db tables, our service is failing. Db changes are applied first,
then app services are restarting on all servers (it takes few minutes).
During this time old version loaded into memory is failing due to missing
defaults:
{{{
IntegrityError
null value in column "X" violates not-null constraint DETAIL: Failing row
contains (...)
}}}
For us, default values must be set at the database layer. Please add
support of db_default attribute.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:30>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Simon Charette (removed)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:31>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Adrian Turjak (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:32>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Doug Harris (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:33>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Ian Foote
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:34>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Ian Foote):
* needs_docs: 0 => 1
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:35>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Leigh Brenecki (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:36>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_docs: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:37>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Johannes Maron (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:38>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:39>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Charlie Denton (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:40>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Anders Hovmöller):
I am getting bitten by this quite a lot.
I would suggest that long term it would be a good idea to have one field
for the default value set in the database schema and one that is dynamic
and set on the Django level. Maybe `default` and `default_db` to keep
backwards compatibility, or `default_dynamic` and `default` to improve the
default.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:41>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
> Apply this diff to django/core/mamagement.py
>
> Should work on any database.
>
> 75a76,77
> > if f.default <> meta.fields.NOT_PROVIDED:
> > field_output.append("DEFAULT '%s'" % (f.default,))
New description:
https://github.com/django/django/pull/13709
Apply this diff to django/core/mamagement.py
Should work on any database.
75a76,77
> if f.default <> meta.fields.NOT_PROVIDED:
> field_output.append("DEFAULT '%s'" % (f.default,))
--
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:42>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ian Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: robotmlg (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:43>
Django
-------------------------------------+-------------------------------------
| Joshi
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Ian Foote => Ayush Joshi
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:44>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ayush
| Joshi
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: raydeal (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:45>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ayush
| Joshi
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Todor Velichkov (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:46>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Ayush
| Joshi
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Alex Scott (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:47>
Django
-------------------------------------+-------------------------------------
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Ayush Joshi => John Whitlock
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:48>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by John Whitlock):
Hi all. I've struggled with the lack of this feature as well. Thanks to
Ayush Joshi for assigning me.
Ian Foote's previous work is in
[https://github.com/django/django/pull/13709 PR 13709]. I've been able to
rebase it and get tests passing, so it looks like a good approach. I'll
start from there, guided by the feedback to Ian's PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:49>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by John Whitlock):
I'm still in the code-reading and planning phase. This is a big one!
A lot of the changes for PR 13709 were implemented in the database
backend, such as quoting expressions and varying implementations. The
reviewers prefer a `DefaultExpression` class, analogous to the
`IndexExpression` class, and to use similar strategies as
`IndexExpression` for backend-specific implementations. They also note
that there is some support for database-level defaults in primary keys
(PK), and wonder if a generic database-default field could borrow from the
PK design, or if the PK feature could be built on top of a generic
database-default field.
Here's a rough idea of how this should work:
* Tests may be needed to verify feature support in different backends,
along with some new backend feature flags. For example:
* [https://www.sqlite.org/lang_altertable.html sqlite3] supports
`ALTER TABLE ADD COLUMN` for simple database defaults, but not for complex
expressions. These can be supported by recreating the table, which is a
more expensive option but commonly used for sqlite3 migrations.
* [https://dev.mysql.com/doc/refman/8.0/en/data-type-defaults.html
mysql 8.0.13] added support for default expressions
* [https://www.postgresql.org/docs/9.5/dml-returning.html postgres]
supports `RETURNING` to return database-created values, which may be
useful
* [https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf
/ALTER-TABLE.html oracle] does not support bind variables in DDL
statements, breaking a strategy in PR 13709.
* Fields grow a new `db_default` parameter. It is distinct from the
`default` parameter, and different combinations should be allowed. Simple
values and more complicated expressions are allowed. Some expressions,
such as a calculated value based on other fields, might be rejected at the
Django level, and some may be rejected by the chosen backend. There are
some checks that could be done at model initialization to verify a valid
`db_default` value, with an appropriate new error message. There may be
warnings for features supported on some but not all backends. The
`default` value is preferred for saving, creating, and bulk-updating
models - in other words, in every place where an application would pick an
unspecified value - leaving current behavior unchanged.
* If the `default` is not given but the `db_default` is, then a database
read may be needed to determine the database-created value (such as with
`refresh_from_db()`). If this is done anyway (for example, to determine
the database-created primary key), it would be nice to get the
`db_default`-initialized fields as well.
* The migrations system should notice changes in the `db_default` field,
and prepare a migration for it. There will be backend-specific strategies
to implement migrations that change these fields.
* Database introspecting should understand column defaults, and if
possible add them when creating models from introspection.
There may be more required features. I'm reading the Django database and
test code to get familiar with them and to discover other features, and
see if there is a way to break this into multiple PRs.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:50>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: David Sanders (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:51>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by John Whitlock):
I've reviewed the tests (so many tests!), the database code, and PR 13709.
While PR 13709 worked, I'm going to try a different approach. As Marcin
Nowak noted in May 2020, the SQL for adding a column looks like:
{{{
ALTER TABLE "x" ADD COLUMN "y" integer DEFAULT 0 NOT NULL CHECK ("y" >=
0);
ALTER TABLE "x" ALTER COLUMN "y" DROP DEFAULT;
}}}
That's the simple case. SQLite is a bit more complex, because it doesn't
support the `ALTER COLUMN ... DROP DEFAULT` clause, and re-creates the
table to get the same effect. A `DateTimeField` with `auto_add_now` looks
like this on PostgreSQL:
{{{
BEGIN;
ALTER TABLE "x" ADD COLUMN "y" timestamp with time zone DEFAULT
'2022-09-06T14:37:26.493371+00:00'::timestamptz NOT NULL;
ALTER TABLE "x" ALTER COLUMN "y" DROP DEFAULT;
COMMIT;
}}}
The current behaviour is:
1. Add the column with a `DEFAULT` of a static value of the field
`default` (for example, computing the current time), to populate existing
row
2. Drop the column `DEFAULT`
I want to keep this as the default behaviour, but allow a new behaviour
for step 2:
1. Add the column with a `DEFAULT` of a static value of the field
`default` (for example, computing the current time), to populate existing
row
2. Update the column `DEFAULT` to the value of the field `db_default`
value or expression. If it has the same value as step 1, do nothing
For the integer case, the SQL would just be the the first line
{{{
ALTER TABLE "x" ADD COLUMN "y" integer DEFAULT 0 NOT NULL CHECK ("y" >=
0);
}}}
For the datetime case, the SQL might be:
{{{
BEGIN;
ALTER TABLE "x" ADD COLUMN "y" timestamp with time zone DEFAULT
'2022-09-06T14:37:26.493371+00:00'::timestamptz NOT NULL;
ALTER TABLE "x" ALTER COLUMN "y" DEFAULT CURRENT_TIMESTAMP AT TIME ZONE
'utc';
COMMIT;
}}}
Some other features:
* `db_default` takes a new constant `NO_DEFAULT` for the current behaviour
of dropping the `DEFAULT` clause
* `db_default` takes a new constant `USE_DEFAULT` to copy the value of
`default`
* `db_default` allows static values or a class derived from a new
`DefaultExpression` class, a subclass of `Expression`
* `db_default` does not allow a generic callable, unlike `default`
* Classes like `DateTimeField` have logic to "do the right thing" when
`db_default=USE_DEFAULT` is combined with `auto_add_now` or `auto_now`.
* Classes like `AutoField` may have more restrictive rules for
`db_default`, since they maintain control of how the database chooses new
values.
I think there should be a way to set a general preference for `NO_DEFAULT`
or `USE_DEFAULT`, at the Model Options, AppConfig, and Settings level,
similar to how `DEFAULT_AUTO_FIELD` was implemented in Django 3.2
A few use cases:
* Developers that prefer the current behavior do nothing.
* Developers that use simple `default` values and want similar database
defaults use `USE_DEFAULT` at the Settings level. A migration is needed to
add database defaults across the application. They can add overrides at
the Field, Model Options, and AppConfig level to fix issues or avoid the
new behavior.
* Developers that want a database-default for an individual field, to
support a rolling deployment, can set `db_default=USE_DEFAULT` for added
columns, and remove it (if desired) in a future migration and deployment.
I'm open to feedback, here or on the django-developers group.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:52>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by John Whitlock):
I hit a dead-end with that approach. The field code can not access the
Model Options or App Config as initialization, so I would have to half-
initialize a field until it gets added to a model. That is awkward, and
would go against the existing code which fully initializes all fields
(with the exception of a primary key) before adding them to the Model.
I was also finding myself wondering what happened if `db_default` was set,
and `default` was not.
I'm attempting a new approach that does not use `db_default`. Instead, I
have a `BaseDefault` class that has more direct control of the default
policy. You can assign an instance of `BaseDefault` or a subclass to
`default`, and the `Field` class behaves differently, and eventually the
`Schema` class will as well.
My first goal is to replicate the `DateField` and `DateTimeField` with
`auto_now` and `auto_now_add`, but through the new class. Then I'll extend
the class and the migrations `Schema` code to tune the the pre-migration
and post-migration column defaults. The behaviors I've identified so far:
* Override `Field.__init__` parameters, such as `editable` and `blank`
(`auto_now`)
* Set the field value on `pre_save` (`auto_now`)
* Test if there is a Python-level default (`has_default`)
* Get a callable version of the Python-level default (`_get_default(),
cached`)
* Pick the `DEFAULT x` value when adding a column
and the new behavior:
* Pick the `DEFAULT x` value after adding a column
I _think_ this will allow a small changes to:
* Persist simple defaults (something like changing `default=0` to
`default=DatabaseDefault(0)`)
* Persist `DateTimeField` and `DateField` (something like changing
`auto_now_add=True` to `DateTimeDatabaseDefault(auto_now_add=True)`)
* Allow for per-database formatting of the `DEFAULT` clause, via
`Expression` or similar
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:53>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Lily Foote (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:54>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: John
| Whitlock
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by David Sanders):
Looks like Lily's opened a new PR with their newly rebased branch 🎉
Default + generated columns are in my top 10 list of wanted features for
Django.
I wanted to just add my 2¢ worth to anyone interested:
If we look at supporting returning values from inserts (with RETURNING in
pg and who knows what mechanism for other dbs) … could we also consider
leaving the design "open" to possibly accommodate generated columns from
both inserts **and** updates? 😊
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:55>
Django
-------------------------------------+-------------------------------------
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: John Whitlock => Lily Foote
Comment:
See https://github.com/django/django/pull/16092 - Good luck Lily Foote!
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:56>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: bcail (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:57>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Adrian Torres (added)
Comment:
Has anybody thought of implementing this feature without an extra `Field`
parameter? IMO it could work with `Field.default` already:
* If the value being passed is a Python literal, use a db default
* If the value is a `DefaultExpression` (or whatever it's actually
called), use a db default
* If the value is a callable, use a Python default
Or am I missing something?
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:58>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by bcail):
It may be useful to have a clear distinction between python and DB
defaults, and not switch between the two based on what the default value
is.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:59>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Adrian Torres):
I would argue that there is no legitimate reason to want to set e.g. a
literal value as default through python instead of through SQL.
Query expressions seem to be supported by `Fields.default`, I haven't seen
the actual implementation but I imagine it generates SQL that calculates a
value and is set for every row in python, so it would make sense that
these are always treated as a database default as well.
Callables are not supported in the db_default implementation since it only
makes sense to process them in python, calculating and setting the value
for every row.
The only case I can think of where having both `default` and `db_default`
in a single field definition might be useful is having a default based on
a python callable and having the db_default as a fallback in case the
callable returned `None` or some "undesirable" value, but that seems
complicated and far from the common case, IMO.
For me having both `default` and `db_default` seems like an easy trap to
fall into for beginners unaware of Django's implementation details, and
for more seasoned developers a hassle and an easy mistake to make that can
be costly.
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:60>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:61>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Lily Foote):
Replying to [comment:60 Adrian Torres]:
> I would argue that there is no legitimate reason to want to set e.g. a
literal value as default through python instead of through SQL.
The main reason I did it this way is backward compatibility, especially
with regards to migrations. If someone adds a field with a default in
Django 4.1, what should we do when they upgrade to 4.2?
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:62>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version:
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Michael Rosner (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:63>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* version: => dev
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:64>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:65>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Lily Foote):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:66>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Lily Foote):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:67>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:66>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
Keywords: sql schema | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:67>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: assigned
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:68>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Component: Database layer | Version: dev
(models, ORM) |
Keywords: sql schema | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"7414704e88d73dafbcfbb85f9bc54cb6111439d3" 7414704]:
{{{
#!CommitTicketReference repository=""
revision="7414704e88d73dafbcfbb85f9bc54cb6111439d3"
Fixed #470 -- Added support for database defaults on fields.
Special thanks to Hannes Ljungberg for finding multiple implementation
gaps.
Thanks also to Simon Charette, Adam Johnson, and Mariusz Felisiak for
reviews.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:69>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution: fixed
Keywords: sql schema | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
In [changeset:"352d860b9107adbcde0f1fe5d0fce8e9090a51e4" 352d860b]:
{{{#!CommitTicketReference repository=""
revision="352d860b9107adbcde0f1fe5d0fce8e9090a51e4"
Refs #470 -- Fixed field_defaults test failures due to year-end boundary
conditions.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:70>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution: fixed
Keywords: sql schema | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
In [changeset:"c85b68398b89bde944de87f528c7d2d1d9fb5dcb" c85b6839]:
{{{#!CommitTicketReference repository=""
revision="c85b68398b89bde944de87f528c7d2d1d9fb5dcb"
[5.2.x] Refs #470 -- Fixed field_defaults test failures due to year-end
boundary conditions.
Backport of 352d860b9107adbcde0f1fe5d0fce8e9090a51e4 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:71>
Django
-------------------------------------+-------------------------------------
Reporter: jws | Owner: Lily
| Foote
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: normal | Resolution: fixed
Keywords: sql schema | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
In [changeset:"836894f27a0e2c32dbbecbe8392ba7d4d0d17d8a" 836894f2]:
{{{#!CommitTicketReference repository=""
revision="836894f27a0e2c32dbbecbe8392ba7d4d0d17d8a"
Refs #470 -- Adjusted Field.db_returning to be backend agnostic.
Determining if a field is db_returning based on the default connection
feature availability prevents the usage of RETURNING for db_default
fields in setups where non-default backends do support RETURNING.
Whether or not the field should be attempted to be returned is already
checked at the compiler level which is backend aware.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/470#comment:72>