[Django] #34984: One-off default migrations fail on tables with GeneratedField (on SQLite)
Django
-----------------------------------------+------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
I added a non-nullable field (in example `new_field_temporary_default`) to
a model containing a GeneratedField and in my migration prompt I added a
one-off default. I am using SQLite.
Model
{{{
class Article(models.Model):
created_at = models.DateTimeField(db_default=Now())
title = models.CharField(max_length=300)
slug = models.GeneratedField(
expression=Lower(Replace(F("title"), Value(" "), Value("-"))),
db_persist=True,
unique=True,
)
new_field_temporary_default = models.TextField() # This field added
after table was created
}}}
Generated migration:
{{{
# Generated by Django 5.0b1 on 2023-11-21 12:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("blog", "0001_initial"),
]
operations = [
migrations.AddField(
model_name="article",
name="new_field_temporary_default",
field=models.TextField(default="Temp default"),
preserve_default=False,
),
]
}}}
This failed to migrate with the error
{{{
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: cannot INSERT into generated column
"slug"
}}}
Not certain if this is an issue. Thought I should raise.
--
Ticket URL: <https://code.djangoproject.com/ticket/34984>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Sarah Boyce | Owner: nobody
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Natalia Bidart):
* status: new => closed
* resolution: => worksforme
Comment:
Hey Sarah, thank you for the report!
Could you please re-test with latest 5.0 rc1? The `GeneratedField` was
changed so the `output_field` is required, and when using that, I can't
reproduce. I tried both with `5.0rc1` and latest main.
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:1>
Django
-------------------------------+--------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by Sarah Boyce):
I've upgraded to the release candidate and replicated.
Initially the model looks like this:
{{{
class Article(models.Model):
created_at = models.DateTimeField(db_default=Now())
title = models.CharField(max_length=300)
slug = models.GeneratedField(
expression=Lower(Replace(F("title"), Value(" "), Value("-"))),
db_persist=True,
unique=True,
output_field=models.TextField(),
)
}}}
First migration file:
{{{
# Generated by Django 5.0rc1 on 2023-11-21 13:58
import django.db.models.functions.datetime
import django.db.models.functions.text
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = []
operations = [
migrations.CreateModel(
name="Article",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created_at",
models.DateTimeField(
db_default=django.db.models.functions.datetime.Now()
),
),
("title", models.CharField(max_length=300)),
(
"slug",
models.GeneratedField(
db_persist=True,
expression=django.db.models.functions.text.Lower(
django.db.models.functions.text.Replace(
models.F("title"), models.Value(" "),
models.Value("-")
)
),
output_field=models.TextField(),
unique=True,
),
),
],
),
]
}}}
Then I add `new_field_temporary_default = models.TextField()` make
migrations, it prompts me that this field is non nullable and without a
default, I go for the option to add a temporary default
Second migration file
{{{
# Generated by Django 5.0rc1 on 2023-11-21 13:58
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("blog", "0001_initial"),
]
operations = [
migrations.AddField(
model_name="article",
name="new_field_temporary_default",
field=models.TextField(default="Temp value"),
preserve_default=False,
),
]
}}}
And migrate, same error:
{{{
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: cannot INSERT into generated column
"slug"
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:2>
Django
-------------------------------+--------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
Comment (by David Sanders):
Thanks Sarah 🏆. Confirmed on main, full traceback:
{{{
django-sample % dj migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, htc, query_json_in,
sessions, ticket_34984_one_off_default, union_issue
Running migrations:
Applying
ticket_34984_one_off_default.0002_article_new_field_temporary_default...Traceback
(most recent call last):
File "/path/to/django/django/db/backends/utils.py", line 105, in
_execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/backends/sqlite3/base.py", line 328, in
execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: cannot INSERT into generated column "slug"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/path/to/sample-project/./manage.py", line 22, in <module>
main()
File "/path/to/sample-project/./manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/path/to/django/django/core/management/__init__.py", line 442, in
execute_from_command_line
utility.execute()
File "/path/to/django/django/core/management/__init__.py", line 436, in
execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/path/to/django/django/core/management/base.py", line 412, in
run_from_argv
self.execute(*args, **cmd_options)
File "/path/to/django/django/core/management/base.py", line 458, in
execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/core/management/base.py", line 106, in
wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/core/management/commands/migrate.py", line
356, in handle
post_migrate_state = executor.migrate(
^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/migrations/executor.py", line 135, in
migrate
state = self._migrate_all_forwards(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/migrations/executor.py", line 167, in
_migrate_all_forwards
state = self.apply_migration(
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/migrations/executor.py", line 252, in
apply_migration
state = migration.apply(state, schema_editor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/migrations/migration.py", line 132, in
apply
operation.database_forwards(
File "/path/to/django/django/db/migrations/operations/fields.py", line
108, in database_forwards
schema_editor.add_field(
File "/path/to/django/django/db/backends/sqlite3/schema.py", line 303,
in add_field
self._remake_table(model, create_field=field)
File "/path/to/django/django/db/backends/sqlite3/schema.py", line 233,
in _remake_table
self.execute(
File "/path/to/django/django/db/backends/base/schema.py", line 201, in
execute
cursor.execute(sql, params)
File "/path/to/django/django/db/backends/utils.py", line 122, in execute
return super().execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/backends/utils.py", line 79, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/backends/utils.py", line 92, in
_execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/backends/utils.py", line 100, in
_execute
with self.db.wrap_database_errors:
File "/path/to/django/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/path/to/django/django/db/backends/utils.py", line 105, in
_execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/django/django/db/backends/sqlite3/base.py", line 328, in
execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: cannot INSERT into generated column
"slug"
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:3>
Django
-------------------------------+--------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Uncategorized | Status: closed
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution: worksforme
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:4>
Django
-------------------------------+------------------------------------
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* status: closed => new
* resolution: worksforme =>
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:5>
Django
-------------------------------+------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 5.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by Natalia Bidart):
Thank you Sarah, I tried again and I was able to reproduce, I must have
had some misconfiguration in my test project. Thank you!
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:6>
Django
Reporter: Sarah Boyce | Owner: nobody
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Natalia Bidart):
* type: Uncategorized => Bug
* component: Uncategorized => Database layer (models, ORM)
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Normal => Release blocker
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Release blocker => Normal
Comment:
SQL of the migration:
{{{#!sql
BEGIN;
--
-- Add field new_field_temporary_default to article
--
CREATE TABLE "new__ticket_34984_article" ("id" integer NOT NULL PRIMARY
KEY AUTOINCREMENT, "new_field_temporary_default" text NOT NULL,
"created_at" datetime DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) NOT
NULL, "title" varchar(300) NOT NULL, "slug" text GENERATED ALWAYS AS
(LOWER(REPLACE("title", ' ', '-'))) STORED UNIQUE);
INSERT INTO "new__ticket_34984_article" ("id", "created_at", "title",
"slug", "new_field_temporary_default") SELECT "id", "created_at", "title",
"slug", 'temp' FROM "ticket_34984_article";
DROP TABLE "ticket_34984_article";
ALTER TABLE "new__ticket_34984_article" RENAME TO "ticket_34984_article";
COMMIT;
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: Sarah Boyce | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):
* severity: Normal => Release blocker
--
Ticket URL: <https://code.djangoproject.com/ticket/34984#comment:10>