[Django] #29182: SQLite database migration breaks ForeignKey constraint, leaving <table_name>__old in db schema
Django
<table_name>__old in db schema
-------------------------------------+-------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: | Status: new
Uncategorized |
Component: | Version: 2.0
Migrations |
Severity: Normal | Keywords: sqlite migration
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
SQLite table alteration uses rename and copy method. When running a
database migration from Django command (via `call_command('migrate')`),
database is left with references to <table_name>__old table that has been
dropped after migration.
This happens only when running SQLite DB migrations from management
command.
Running migrations via `./manage.py migrate` does not cause any issues.
This is the demonstration:
https://github.com/ezaquarii/openvpn-at-home/tree/sqlite-migration-bug
Steps to reproduce the bug
1. build the project with `make devel`
2. enter Python virtualenv by `source backend/env/bin/activate` and enter
`backend` subdirectory
3. run `./manage.py bootstrap` to initialize the database
4. bootstrap command runs db migration by calling
`call_command('migrate')` in
`openvpnathome/apps/management/management/commands/bootstrap.py:37`:
- `openvpn_server` table is created
- `openvpn_client` table is created; it uses foreign key to
`openvpn_server`
- another `openvpn_server` migration is run, adding dummy field
- `openvpn_server` table is renamed to `openvpn_server__old`
- `openvpn_client` foreign key is re-linked to `openvpn_server__old`
- new table `openvpn_server` is created and data from
`openvpn_server__old` is copied with applied alterations
- `openvpn_server__old` is dropped
- `openvpn_client` still references `openvpn_sever__old` - ForeignKey
constraint is **not** updated and DB is left in unusable state
6. try creating `Client` object in django shell - SQL cannot be executed
due to missing table
Exact commands (available in provided `bug_repro.sh` script):
{{{
make devel
cd backend
source ./env/bin/activate
./manage.py bootstrap
echo '
Now type that:
./manage.py shell
In [1]: from openvpnathome.apps.openvpn.models import Server, Client
In [2]: Client.objects.create()
... snip ...
OperationalError: no such table: main.openvpn_server__old
'
}}}
Exported database schema clearly references deleted `openvpn_server__old`
table:
{{{
CREATE TABLE "openvpn_client" ("id" integer NOT NULL PRIMARY KEY
AUTOINCREMENT, "created" datetime NOT NULL, "name" varchar(64) NOT NULL,
"cert_id" integer NOT NULL REFERENCES "x509_cert" ("id") DEFERRABLE
INITIALLY DEFERRED, "owner_id" integer NOT NULL REFERENCES "accounts_user"
("id") DEFERRABLE INITIALLY DEFERRED, "server_id" integer NOT NULL
REFERENCES "openvpn_server__old" ("id") DEFERRABLE INITIALLY DEFERRED);
}}}
Please refer to `db_schema.sql` file placed in the `backend` subdirectory.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: sqlite migration | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ezaquarii):
* Attachment "bug_repro.sh" added.
Bug reproduction steps as shell script
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ezaquarii):
* Attachment "db_schema.sql" added.
Database schema after running migrations - see references to temporary
openvpn_server__old table
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by ezaquarii:
Old description:
New description:
echo '
Now type that:
In [2]: Client.objects.create()
... snip ...
Tested with Django versions: 2.0.1 and 2.0.2
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:1>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Could you provide a more minimal project that reproduces the issue? In
particular, the steps to reproduce shouldn't require installing a bunch of
NodeJS packages. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:2>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Old description:
> Tested with Django versions: 2.0.1 and 2.0.2
New description:
cd backend
make devel
source ./env/bin/activate
./manage.py bootstrap
echo '
Now type that:
./manage.py shell
In [1]: from openvpnathome.apps.openvpn.models import Server, Client
In [2]: Client.objects.create()
... snip ...
OperationalError: no such table: main.openvpn_server__old
'
}}}
Exported database schema clearly references deleted `openvpn_server__old`
table:
{{{
CREATE TABLE "openvpn_client" ("id" integer NOT NULL PRIMARY KEY
AUTOINCREMENT, "created" datetime NOT NULL, "name" varchar(64) NOT NULL,
"cert_id" integer NOT NULL REFERENCES "x509_cert" ("id") DEFERRABLE
INITIALLY DEFERRED, "owner_id" integer NOT NULL REFERENCES "accounts_user"
("id") DEFERRABLE INITIALLY DEFERRED, "server_id" integer NOT NULL
REFERENCES "openvpn_server__old" ("id") DEFERRABLE INITIALLY DEFERRED);
}}}
Please refer to `db_schema.sql` file placed in the `backend` subdirectory.
Tested with Django versions: 2.0.1 and 2.0.2
--
Comment (by ezaquarii):
Please check this repository:
https://github.com/ezaquarii/django-sqlite-migration-bug
{{{
with transaction.atomic():
call_command('migrate')
}}}
I think this is the issue. It works ok without transaction.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:3>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by ezaquarii:
Old description:
> SQLite table alteration uses rename and copy method. When running a
> cd backend
> make devel
> source ./env/bin/activate
> ./manage.py bootstrap
>
> echo '
> Now type that:
>
> ./manage.py shell
> In [1]: from openvpnathome.apps.openvpn.models import Server, Client
>
> In [2]: Client.objects.create()
>
> ... snip ...
>
> OperationalError: no such table: main.openvpn_server__old
> '
> }}}
>
> Exported database schema clearly references deleted `openvpn_server__old`
> table:
>
> {{{
> CREATE TABLE "openvpn_client" ("id" integer NOT NULL PRIMARY KEY
> AUTOINCREMENT, "created" datetime NOT NULL, "name" varchar(64) NOT NULL,
> "cert_id" integer NOT NULL REFERENCES "x509_cert" ("id") DEFERRABLE
> INITIALLY DEFERRED, "owner_id" integer NOT NULL REFERENCES
> "accounts_user" ("id") DEFERRABLE INITIALLY DEFERRED, "server_id" integer
> NOT NULL REFERENCES "openvpn_server__old" ("id") DEFERRABLE INITIALLY
> DEFERRED);
> }}}
>
> Please refer to `db_schema.sql` file placed in the `backend`
> subdirectory.
>
> Tested with Django versions: 2.0.1 and 2.0.2
New description:
SQLite table alteration uses rename and copy method. When running a
database migration from Django command (via `call_command('migrate')`),
database is left with references to <table_name>__old table that has been
dropped after migration.
This happens only when running SQLite DB migrations from management
command under transaction.
Running migrations via `./manage.py migrate` does not cause any issues.
This is the demonstration:
https://github.com/ezaquarii/django-sqlite-migration-bug
Steps to reproduce the bug
1. run ./bootstrap.sh
2. from import app.models import *
3. Bravo.objects.create()
4. SQL cannot be executed due to missing table `main.app_alpha__old`
- `app/management/commands/bootstrap.py:9`: run the migration
- `alpha` table is created
- `bravo` table is created; it uses foreign key to `alpha`
- another `alpha` migration is run, adding dummy field
- `alpha` table is renamed to `alpha__old`
- `bravo` foreign key is re-linked to `alpha__old`
- new table `alpha` is created and data from `alpha__old` is copied
with applied alterations
- `alpha__old` is dropped
- `bravo` still references `alpha__old` - ForeignKey constraint is
**not** updated and DB is left in unusable state
{{{
CREATE TABLE "app_bravo" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"dummy" varchar(10) NOT NULL, "alpha_id" integer NOT NULL REFERENCES
"app_alpha__old" ("id") DEFERRABLE INITIALLY DEFERRED);
}}}
Tested with Django versions: 2.0.1 and 2.0.2
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:4>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by ezaquarii:
Old description:
> SQLite table alteration uses rename and copy method. When running a
> database migration from Django command (via `call_command('migrate')`),
> database is left with references to <table_name>__old table that has been
> dropped after migration.
>
> This happens only when running SQLite DB migrations from management
> command under transaction.
> Running migrations via `./manage.py migrate` does not cause any issues.
>
> This is the demonstration:
> https://github.com/ezaquarii/django-sqlite-migration-bug
>
> Steps to reproduce the bug
> 1. run ./bootstrap.sh
> 2. from import app.models import *
> 3. Bravo.objects.create()
> 4. SQL cannot be executed due to missing table `main.app_alpha__old`
>
> - `app/management/commands/bootstrap.py:9`: run the migration
> - `alpha` table is created
> - `bravo` table is created; it uses foreign key to `alpha`
> - another `alpha` migration is run, adding dummy field
> - `alpha` table is renamed to `alpha__old`
> - `bravo` foreign key is re-linked to `alpha__old`
> - new table `alpha` is created and data from `alpha__old` is copied
> with applied alterations
> - `alpha__old` is dropped
> - `bravo` still references `alpha__old` - ForeignKey constraint is
> **not** updated and DB is left in unusable state
>
> {{{
> CREATE TABLE "app_bravo" ("id" integer NOT NULL PRIMARY KEY
> AUTOINCREMENT, "dummy" varchar(10) NOT NULL, "alpha_id" integer NOT NULL
> REFERENCES "app_alpha__old" ("id") DEFERRABLE INITIALLY DEFERRED);
> }}}
>
> Tested with Django versions: 2.0.1 and 2.0.2
New description:
SQLite table alteration uses rename and copy method. When running a
database migration from Django command (via `call_command('migrate')`),
database is left with references to `<table_name>__old` table that has
been dropped after migration.
This happens only when running SQLite DB migrations from management
command under transaction.
Running migrations via `./manage.py migrate` does not cause any issues.
This is the demonstration:
https://github.com/ezaquarii/django-sqlite-migration-bug
Steps to reproduce the bug
1. run ./bootstrap.sh
2. from import app.models import *
3. Bravo.objects.create()
4. SQL cannot be executed due to missing table `main.app_alpha__old`
- `app/management/commands/bootstrap.py:9`: run the migration
- `alpha` table is created
- `bravo` table is created; it uses foreign key to `alpha`
- another `alpha` migration is run, adding dummy field
- `alpha` table is renamed to `alpha__old`
- `bravo` foreign key is re-linked to `alpha__old`
- new table `alpha` is created and data from `alpha__old` is copied
with applied alterations
- `alpha__old` is dropped
- `bravo` still references `alpha__old` - ForeignKey constraint is
**not** updated and DB is left in unusable state
{{{
CREATE TABLE "app_bravo" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"dummy" varchar(10) NOT NULL, "alpha_id" integer NOT NULL REFERENCES
"app_alpha__old" ("id") DEFERRABLE INITIALLY DEFERRED);
}}}
Tested with Django versions: 2.0.1 and 2.0.2
--
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:5>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* cc: Simon Charette (added)
* stage: Unreviewed => Accepted
Comment:
Hey ezaquarii, thank you for your great report.
You can get a bit of context about why all the SQLite constraint
workarounds are required in #28849 but in this case I suspect odd things
happen because, as you might know, savepoints are used in nested
`atomic()` blocks.
The migration executor was not designed to be used within a transaction as
it does some pretty intense transaction management itself so I'm accepting
on the basis that we should do a better job at raising an error in this
case. The fact that this doesn't happen when `./manage.py migrate` is
called directly is a good indicator that other weird things might be
happening under the hood because this case is completely untested by the
suite right now. For example, do the backend that we know transactional
DDL support for all behave the same way with regards to savepoints?
I guess you were trying to come up with some kind of deploying script that
either fails or succeeds to apply multiple migrations?
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:6>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Uncategorized => Cleanup/optimization
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:7>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Florian Apolloner):
* type: Cleanup/optimization => Bug
* severity: Normal => Release blocker
Comment:
Changing this to a full blown bug and release blocker since I am able to
reproduce this with `./manage.py migrate` and very simple app.
Relevant versions:
Sqlite: 3.26.0 2018-12-01 12:34:55
bf8c1b2b7a5960c282e543b9c293686dccff272512d08865f4600fb58238alt1
python sqlite3 version: 2.6.0
Same test models:
{{{
#!python
from django.db import models
class File(models.Model):
content = models.ForeignKey("FileContent", on_delete=models.PROTECT)
class FileContent(models.Model):
pass
class PlayBook(models.Model):
file = models.ForeignKey("File", on_delete=models.PROTECT)
}}}
This generates the following initial migration (on latest master):
{{{
#!python
# Generated by Django 2.2.dev20181204152138 on 2018-12-04 16:49
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='File',
fields=[
('id', models.AutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
],
),
migrations.CreateModel(
name='FileContent',
fields=[
('id', models.AutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
],
),
migrations.CreateModel(
name='PlayBook',
fields=[
('id', models.AutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
('file',
models.ForeignKey(on_delete=django.db.models.deletion.PROTECT,
to='testing.File')),
],
),
migrations.AddField(
model_name='file',
name='content',
field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT,
to='testing.FileContent'),
),
]
}}}
Running this migration results in the following schema in SQLite:
{{{
#!sql
CREATE TABLE IF NOT EXISTS "django_migrations"(
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"app" varchar(255) NOT NULL,
"name" varchar(255) NOT NULL,
"applied" datetime NOT NULL
);
CREATE TABLE IF NOT EXISTS "testing_filecontent"(
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT
);
CREATE TABLE IF NOT EXISTS "testing_playbook"(
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"file_id" integer NOT NULL REFERENCES "testing_file__old"("id")
DEFERRABLE INITIALLY DEFERRED
);
CREATE TABLE IF NOT EXISTS "testing_file"(
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"content_id" integer NOT NULL REFERENCES "testing_filecontent"("id")
DEFERRABLE INITIALLY DEFERRED
);
CREATE INDEX "testing_playbook_file_id_debe0476" ON "testing_playbook"(
"file_id"
);
CREATE INDEX "testing_file_content_id_4682b86d" ON "testing_file"(
"content_id"
);
}}}
From the looks of it, we will need to do some similar to
https://github.com/django/django/blob/196b420fcb0cbdd82970e2b9aea80251bde82056/django/db/backends/sqlite3/schema.py#L108-L121
whenever we rename a table which has foreignkeys pointing to it.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:8>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Thanks for the extra details Florian. I'll try to have a look at it this
week.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:9>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Florian Apolloner):
* cc: Florian Apolloner (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:10>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Florian Apolloner):
Fwiw I think we will have to disable `can_rollback_ddl` on sqlite to get a
sane behavior, otherwise pretty much every migration out there will break.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:11>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Florian Apolloner):
* Attachment "initial_patch.diff" added.
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Florian Apolloner):
I have added an initial patch which fixes my issue -- it should serve only
as starting point for some ideas and also has a few questions left…
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:12>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Florian, if you still have a few minutes to dedicate to this issue I'd
appreciate if you could add a
[https://github.com/django/django/pull/9389/files#diff-
888b45a0a8f38ee67e8d22403cf994dbR1151 schema test] based on your
investigation reproducing the issue. It's surprising that this wasn't
caught by an existing foreign key addition test.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:13>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Florian Apolloner):
I'll try to do so tomorrow, I've never written a schema test :)
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:14>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
I did a bit of investigation on my side and there must be something else
at play here as I cannot reproduce against `stable/2.1.x` and `master`
with SQLite version 3.22.0 2018-01-22 18:45:57 and `sqlite3.version` 2.6.0
on Python 3.6.6.
{{{
$ sqlite3 project/db.sqlite3
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE IF NOT EXISTS "django_migrations" ("id" integer NOT NULL
PRIMARY KEY AUTOINCREMENT, "app" varchar(255) NOT NULL, "name"
varchar(255) NOT NULL, "applied" datetime NOT NULL);
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE IF NOT EXISTS "ticket_29182_filecontent" ("id" integer NOT
NULL PRIMARY KEY AUTOINCREMENT);
CREATE TABLE IF NOT EXISTS "ticket_29182_playbook" ("id" integer NOT NULL
PRIMARY KEY AUTOINCREMENT, "file_id" integer NOT NULL REFERENCES
"ticket_29182_file" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE TABLE IF NOT EXISTS "ticket_29182_file" ("id" integer NOT NULL
PRIMARY KEY AUTOINCREMENT, "content_id" integer NOT NULL REFERENCES
"ticket_29182_filecontent" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE INDEX "ticket_29182_playbook_file_id_f3bc0e50" ON
"ticket_29182_playbook" ("file_id");
CREATE INDEX "ticket_29182_file_content_id_7815674c" ON
"ticket_29182_file" ("content_id");
}}}
I also couldn't get a regression test to fail either. Florian could you
confirm one of these added tests fail for you on your local setup:
https://github.com/django/django/compare/master...charettes:ticket-29182
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:15>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Cannot reproduce with `SQLite 3.23.1` and `3.24.0 2018-06-04 14:10:15`
either, I'll try to get my hands on 3.26+ tomorrow. Could you provide the
output of
{{{#!python
import sqlite3
sqlite3.connect('file::memory:').cursor().execute('PRAGMA
compile_options').fetchall()
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:16>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Christoph Trassl):
It's broken with SQLite 3.26. SQLite 3.25.3 is fine, regardless of Django
version. Perhaps it is related to changes to the ALTER TABLE statement in
SQLite 3.26 (https://sqlite.org/pragma.html#pragma_legacy_alter_table).
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:17>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* version: 2.0 => 2.1
Comment:
Thanks for confirming Christoph, that explains why CI didn't break. In
this case the safest option is probably to turn this prama on for Django
2.1 and adjust the schema altering logic on 2.2+.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:18>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Florian Apolloner):
Simon, the following test is the best I could come up with -- not sure if
there is a better way aside from querying `sqlite_master`, but seems to be
okay for the purpose of the test.
{{{
#!python
@isolate_apps('schema')
@unittest.skipUnless(connection.vendor == 'sqlite', 'SQLite naively
remakes the table on field alteration.')
def test_table_remake_with_fks_pointing_to_it(self):
class File(Model):
content = ForeignKey("FileContent", on_delete=PROTECT)
class Meta:
app_label = 'schema'
class FileContent(Model):
class Meta:
app_label = 'schema'
class PlayBook(Model):
file = ForeignKey("File", on_delete=PROTECT)
class Meta:
app_label = 'schema'
new_field = ForeignKey(FileContent, on_delete=PROTECT)
new_field.set_attributes_from_name("filecontent")
with connection.schema_editor(atomic=True) as editor:
editor.create_model(File)
editor.create_model(FileContent)
editor.create_model(PlayBook)
editor.add_field(File, new_field)
with connection.cursor() as cursor:
cursor.execute("SELECT sql FROM sqlite_master WHERE
type='table' and tbl_name='schema_playbook'")
result = cursor.fetchone()[0]
self.assertNotIn("__old", result)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:19>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Adam Goode):
* cc: Adam Goode (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:20>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Changed ticket name to give it more exposure as there have been a few
duplicates so far (e.g. #30016).
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:21>
Django
<table_name>__old in db schema
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Lamb):
* Apologies for the dupe. I filed #30016after not reading the most-recent
entries here more carefully...
* I can reproduce with sqlite3 `3.26.0` (but not with `3.25.3`)
However, the patch in
https://code.djangoproject.com/ticket/29182#comment:12 does not work for
me:
{{{
test_testcase_ordering
(test_runner.test_discover_runner.DiscoverRunnerTest) ... ok
test_transaction_support (test_runner.tests.Sqlite3InMemoryTestDbs)
Ticket #16329: sqlite3 in-memory test databases ... ERROR
======================================================================
ERROR: test_transaction_support (test_runner.tests.Sqlite3InMemoryTestDbs)
Ticket #16329: sqlite3 in-memory test databases
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/lamby/git/debian/python-team/modules/python-
django/tests/test_runner/tests.py", line 226, in test_transaction_support
DiscoverRunner(verbosity=0).setup_databases()
File "/home/lamby/git/debian/python-team/modules/python-
django/django/test/runner.py", line 551, in setup_databases
self.parallel, **kwargs
File "/home/lamby/git/debian/python-team/modules/python-
django/django/test/utils.py", line 174, in setup_databases
serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE',
True),
File "/home/lamby/git/debian/python-team/modules/python-
django/django/db/backends/base/creation.py", line 68, in create_test_db
run_syncdb=True,
File "/home/lamby/git/debian/python-team/modules/python-
django/django/core/management/__init__.py", line 148, in call_command
return command.execute(*args, **defaults)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/core/management/commands/migrate.py", line 172, in handle
self.sync_apps(connection, executor.loader.unmigrated_apps)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/core/management/commands/migrate.py", line 306, in sync_apps
editor.create_model(model)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/db/backends/base/schema.py", line 312, in create_model
self.execute(sql, params or None)
File "/home/lamby/git/debian/python-team/modules/python-
django/django/db/backends/base/schema.py", line 118, in execute
"Executing DDL statements while in a transaction on databases "
django.db.transaction.TransactionManagementError: Executing DDL statements
while in a transaction on databases that can't perform a rollback is
prohibited.
----------------------------------------------------------------------
Ran 5020 tests in 138.176s
FAILED (errors=1, skipped=657, expected failures=3)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:22>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Simon Charette):
Hello Chris,
I don't think the approach of the patch
https://code.djangoproject.com/ticket/29182#comment:12 will work.
Could you give
https://github.com/django/django/compare/master...charettes:ticket-29182-pragma
a run against the test suite on SQLite 3.26.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:23>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Florian Apolloner):
Replying to [comment:22 Chris Lamb]:
> However, the patch in
https://code.djangoproject.com/ticket/29182#comment:12 does not work for
me:
As said this is just a starting point and we might come around to make
this actually the expected behavior. We cannot safely alter a SQLite
database inside a transaction from the looks of it (If you look at the
patch we rewrite internal tables and then need to issues a VACUUM -- not
sure if we can do that safely in transactions).
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:24>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Simon Charette):
> We cannot safely alter a SQLite database inside a transaction from the
looks of it...
It has been working just fine on SQLite < 3.26 and the tests prove it.
The current issue with 3.26 is that it changes the assumptions related to
foreign key repointing on table rename when constraint checks are
disabled. On SQLite < 3.26 renaming a table like we do in table remake
because of unsupported `ALTER table` doesn't automatically repoint the
foreign key constraints (e.g from `table_name` to `table_name__old`) but
it does on SQLite 3.26.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:25>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Florian Apolloner):
Replying to [comment:25 Simon Charette]:
> > We cannot safely alter a SQLite database inside a transaction from the
looks of it...
>
> It has been working just fine on SQLite < 3.26 and the tests prove it.
Yes, I was speaking with my 3.26 hat on (That is if we want to fix it
without the PRAGMA for the rename). The existing code in master absolutely
works fine and is stable for anything but the newest SQLite.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:26>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Chris Lamb):
This is being tracked upstream in sqlite3 here:
https://www.sqlite.org/src/info/f44bc7a8b3fac82a (hat-tip László
Böszörményi in https://bugs.debian.org/915626#27).
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:27>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Simon Charette):
Florian or Chris,
Could you give a try at running the suite with the following patch and
report if it helps
https://github.com/django/django/compare/master...charettes:ticket-29182-pragma
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:28>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Florian Apolloner):
Hrmpf, I must have forgot to hit send before. Yes your pragma patch fixes
it (with my testcase added).
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:29>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Simon Charette):
Awesome, thanks for confirming Florian. I'll turn it into a PR with a
release note for the next 2.1 release later today.
I feel like we'll want to stick to this solution until we drop support for
< 3.26 as the logic involved, as you've come to notice yourself, is
already quite convoluted and adding some version branching in there would
make it even worst. We've got a solution that allows us to keep atomic
migrations enabled on SQLite so I suggest we stick to it.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:30>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
* cc: Muflone (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:31>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.1
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
* has_patch: 0 => 1
Comment:
https://github.com/django/django/pull/10733
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:32>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Changes (by Simon Charette):
* version: 2.1 => 2.0
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:33>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Chris Lamb):
> https://github.com/django/django/pull/10733
Works for me.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:34>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution:
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Arthur Vuillard):
The patch worked for me on django 2.0 too
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:35>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"53b17d4734f06372b66e3ac4db7a1740c503b330" 53b17d4]:
{{{
#!CommitTicketReference repository=""
revision="53b17d4734f06372b66e3ac4db7a1740c503b330"
[2.0.x] Fixed #29182 -- Fixed schema table alteration on SQLite 3.26+.
SQLite 3.26 repoints foreign key constraints on table renames even when
foreign_keys pragma is off which breaks every operation that requires
a table rebuild to simulate unsupported ALTER TABLE statements.
The newly introduced legacy_alter_table pragma disables this behavior
and restores the previous schema editor assumptions.
Thanks Florian Apolloner, Christoph Trassl, Chris Lamb for the report and
troubleshooting assistance.
Backport of c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:36>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Tim Graham <timograham@…>):
In [changeset:"fc8c2284e9f9e8d5d17ef35f282917a6833fafed" fc8c2284]:
{{{
#!CommitTicketReference repository=""
revision="fc8c2284e9f9e8d5d17ef35f282917a6833fafed"
[2.1.x] Fixed #29182 -- Fixed schema table alteration on SQLite 3.26+.
SQLite 3.26 repoints foreign key constraints on table renames even when
foreign_keys pragma is off which breaks every operation that requires
a table rebuild to simulate unsupported ALTER TABLE statements.
The newly introduced legacy_alter_table pragma disables this behavior
and restores the previous schema editor assumptions.
Thanks Florian Apolloner, Christoph Trassl, Chris Lamb for the report and
troubleshooting assistance.
Backport of c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f from master.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:37>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Tim Graham <timograham@…>):
In [changeset:"c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f" c8ffdbe]:
{{{
#!CommitTicketReference repository=""
revision="c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f"
Fixed #29182 -- Fixed schema table alteration on SQLite 3.26+.
SQLite 3.26 repoints foreign key constraints on table renames even when
foreign_keys pragma is off which breaks every operation that requires
a table rebuild to simulate unsupported ALTER TABLE statements.
The newly introduced legacy_alter_table pragma disables this behavior
and restores the previous schema editor assumptions.
Thanks Florian Apolloner, Christoph Trassl, Chris Lamb for the report and
troubleshooting assistance.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:38>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Tim Graham):
The original issue reported in this ticket will be addressed in #30023.
The ticket was repurposed incorrectly for a different issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:39>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson <carlton.gibson@…>):
In [changeset:"7289874adceec46b5367ec3157cdd10c711253a0" 7289874a]:
{{{
#!CommitTicketReference repository=""
revision="7289874adceec46b5367ec3157cdd10c711253a0"
Fixed #30033 -- Conformed to the recommended table alterations procedure
on SQlite3.
Refs #29182.
The previous implementation was following a procedure explicitly
documented
as incorrect and was the origin of the breakage experienced on SQLite 3.26
release that were addressed by c8ffdbe514b55ff5c9a2b8cb8bbdf2d3978c188f.
Thanks to Richard Hipp for pointing out the usage of the incorrect
procedure.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:40>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Carlton Gibson <carlton.gibson@…>):
In [changeset:"894cb13779e6d092974c873bd2cf1452554d2e06" 894cb13]:
{{{
#!CommitTicketReference repository=""
revision="894cb13779e6d092974c873bd2cf1452554d2e06"
Refs #29182 -- Stopped relying on legacy alter table semantic on SQLite
3.26+.
SQLite 3.26 changed the behavior of table and column renaming operations
to
repoint foreign key references even if foreign key checks are disabled.
This makes the workarounds in place to simulate this behavior unnecessary
on
SQLite 3.26+. Refs #30033.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:41>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Tim Graham <timograham@…>):
In [changeset:"25a0781a1661e7329bc1ba9bce12dfddf7f2d433" 25a0781a]:
{{{
#!CommitTicketReference repository=""
revision="25a0781a1661e7329bc1ba9bce12dfddf7f2d433"
Refs #29182 -- Corrected SQLite's supports_atomic_references_rename
feature flag.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:42>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by johnthagen):
FWIW, I hit this bug too in Python 3.7.1, sqlite 3.26.0, and Django 2.1.4.
In my case, if I had models with `ForeignKey`s that related to each other
in reverse alphabetic order, I got errors on that looked like:
`django.db.utils.OperationalError: no such table: main.polls_cat__old`
In case it's helpful for another data point, I wrote a detailed report
here: https://github.com/beda-software/drf-writable-nested/issues/64
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:43>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Florian Apolloner):
The fix is committed but not yet in any released version afaik; so you'd
have to test against master…
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:44>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by johnthagen):
I just tried it out my minimal example using `pip install
git+https://github.com/django/django.git`
{{{
$ pip list
Package Version
--------------------- ---------------------
Django 2.2.dev20181230164339
...
}}}
And it seems to be fixed! I will double check with 2.1.5 as well after
it's released Jan 1. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:45>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by bennofs):
This is still broken in Django 1.11.25, shouldn't it be backported to that
version as well?
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:46>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Benno Fünfstück):
I created a PR to backport to 1.11:
https://github.com/django/django/pull/11986
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:47>
Django
<table_name>__old in db schema
----------------------------------+------------------------------------
Reporter: ezaquarii | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version: 2.0
Severity: Release blocker | Resolution: fixed
Keywords: sqlite migration | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Comment (by Tony S Yu):
The backport for the fix to Django 1.11 was closed. I commented on that
ticket, but for visibility, I'll comment here as well.
This issue makes it difficult to test libraries supporting both Django
1.11 and Django 2.2 on a single Centos server (without hacks to upgrade
supported sqlite versions):
- Centos 7 supports sqlite3 3.7.17 (see
[https://rpms.remirepo.net/rpmphp/zoom.php?rpm=sqlite EL-7 here])
- Centos 8 supports sqlite3 3.26.0 (see
[https://rpms.remirepo.net/rpmphp/zoom.php?rpm=sqlite EL-8 here])
Our team runs internal libraries on Centos servers using tox to test
against various versions of dependencies. The current version
incompatibilities mean that Django 1.11 tests of a library must run on
Centos 7 and tests for Django 2.2 must run on Centos 8.
--
Ticket URL: <https://code.djangoproject.com/ticket/29182#comment:48>