[Django] #30351: Migration auth.0011_update_proxy_permissions from Django 2.2 fails to apply
Django
-----------------------------------------+------------------------
Reporter: Jenselme | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 2.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
I am trying to update my project to Django 2.2. When I launch `python
manage.py migrate`, I get this error message when migration
`auth.0011_update_proxy_permissions` is applying (full stacktrace is
available [https://dpaste.de/5nJJ here]):
`django.db.utils.IntegrityError: duplicate key value violates unique
constraint "idx_18141_auth_permission_content_type_id_01ab375a_uniq"
DETAIL: Key (content_type_id, codename)=(12, add_agency) already exists`.
It looks like the migration is trying to re-create already existing
entries in the `auth_permission` table. At first I though it cloud because
we recently renamed a model. But after digging and deleting the entries
associated with the renamed model from our database in the
`auth_permission` table, the problem still occurs with other proxy models.
I tried to update directly from 2.0.13 and 2.1.8. The issues appeared each
time. I also deleted my venv and recreated it without an effect.
I searched for a ticket about this on the bug tracker but found nothing. I
also posted this on [https://groups.google.com/forum/#!topic/django-
users/KPnsJCgw8-U django-users] and was asked to report this here.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Jenselme | Owner: nobody
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: Uncategorized => contrib.auth
* type: Uncategorized => Bug
Comment:
Please provide a sample project or enough details to reproduce the issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:1>
Django
Reporter: Julien Enselme | Owner: nobody
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by kiahosseini):
Same problem for me.
If a `Permission` exists already with the new `content_type` and
`permission name`, `IntegrityError` is raised as it violates the
unique_key constraint on permission model or `content_type_id_code_name`
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:2>
Django
Reporter: Julien Enselme | Owner: nobody
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Sébastiaan Versteeg):
To get into the situation where you already have permissions with the
content type you should be able to do the following:
- Start on Django <2.2
- Create a model called 'TestModel'
- Migrate
- Delete the model called 'TestModel'
- Add a new **proxy** model called 'TestModel'
- Migrate
- Update to Django >=2.2
- Migrate
We think this is what happened in our case where we found this issue
(https://sentry.thalia.nu/share/issue/68be0f8c32764dec97855b3cbb3d8b55/).
We have a proxy model with the same name that a previous non-proxy model
once had. This changed during a refactor and the permissions for the
original model still exist.
Our solution will probably be removing the existing permissions from the
table, but that's really only a workaround.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:3>
Django
Reporter: Julien Enselme | Owner: nobody
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* cc: Simon Charette, Arthur Rio, Antoine Catton (added)
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Reproduced with steps from
[https://code.djangoproject.com/ticket/30351#comment:3 comment]. It's
probably regression in 181fb60159e54d442d3610f4afba6f066a6dac05.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:4>
Django
a proxy.
---------------------------------+------------------------------------
Reporter: Julien Enselme | Owner: nobody
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Description changed by felixxm:
Old description:
> I am trying to update my project to Django 2.2. When I launch `python
> manage.py migrate`, I get this error message when migration
> `auth.0011_update_proxy_permissions` is applying (full stacktrace is
> available [https://dpaste.de/5nJJ here]):
> `django.db.utils.IntegrityError: duplicate key value violates unique
> constraint "idx_18141_auth_permission_content_type_id_01ab375a_uniq"
> DETAIL: Key (content_type_id, codename)=(12, add_agency) already
> exists`.
>
> It looks like the migration is trying to re-create already existing
> entries in the `auth_permission` table. At first I though it cloud
> because we recently renamed a model. But after digging and deleting the
> entries associated with the renamed model from our database in the
> `auth_permission` table, the problem still occurs with other proxy
> models.
>
> I tried to update directly from 2.0.13 and 2.1.8. The issues appeared
> each time. I also deleted my venv and recreated it without an effect.
>
> I searched for a ticket about this on the bug tracker but found nothing.
> I also posted this on [https://groups.google.com/forum/#!topic/django-
> users/KPnsJCgw8-U django-users] and was asked to report this here.
New description:
I am trying to update my project to Django 2.2. When I launch `python
manage.py migrate`, I get this error message when migration
`auth.0011_update_proxy_permissions` is applying (full stacktrace is
available [https://dpaste.de/5nJJ here]):
`django.db.utils.IntegrityError: duplicate key value violates unique
constraint "idx_18141_auth_permission_content_type_id_01ab375a_uniq"
DETAIL: Key (co.ntent_type_id, codename)=(12, add_agency) already
exists`.
It looks like the migration is trying to re-create already existing
entries in the `auth_permission` table. At first I though it cloud because
we recently renamed a model. But after digging and deleting the entries
associated with the renamed model from our database in the
`auth_permission` table, the problem still occurs with other proxy models.
I tried to update directly from 2.0.13 and 2.1.8. The issues appeared each
time. I also deleted my venv and recreated it without an effect.
I searched for a ticket about this on the bug tracker but found nothing. I
also posted this on [https://groups.google.com/forum/#!topic/django-
users/KPnsJCgw8-U django-users] and was asked to report this here.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:5>
Django
a proxy.
Reporter: Julien Enselme | Owner: Arthur Rio
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Arthur Rio):
* owner: nobody => Arthur Rio
* status: new => assigned
Comment:
We do not automatically delete stale content types on purpose to avoid
data loss. When deleting a model, the `remove_stale_contenttypes.py`
management command
[https://github.com/django/django/blob/master/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py]
should be ran. We can do two things here:
1. Handle the `IntegrityError` with a helpful message to point to the
command
2. Re-use existing content types
I think 1. is safer but I am open to suggestions.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:6>
Django
a proxy.
---------------------------------+--------------------------------------
Reporter: Julien Enselme | Owner: Arthur Rio
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by B Martsberger):
It's also possible to get this kind of integrity error on the auth.0011
migration if another app is migrated first causing the auth
post_migrations hook to run. The auth post migrations hook runs
`django.contrib.auth.management.create_permissions`, which writes the new
form of the auth_permission records to the table. Then when the auth.0011
migration runs it tries to update things to the values that were just
written.
To reproduce this behavior:
1. pip install Django==2.1.7
2. Create an app, let's call it `app`, with two models,
`TestModel(models.Model)` and `ProxyModel(TestModel)` the second one with
proxy=True
3. `python manage.py makemigrations`
4. `python manage.py migrate`
5. pip install Django==2.2
6. Add another model to `app`
7. `python manage.py makemigrations`
8. migrate the app only, `python manage.py migrate app`. This does not run
the auth migrations, but does run the auth post_migrations hook
9. Note that new records have been added to auth_permission
10. `python manage.py migrate`, this causes an integrity error when the
auth.0011 migration tries to update records that are the same as the ones
already added in step 8.
This has the same exception as this bug report, I don't know if it's
considered a different bug, or the same one.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:7>
Django
a proxy.
---------------------------------+--------------------------------------
Reporter: Julien Enselme | Owner: Arthur Rio
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Arthur Rio):
Yes it is the same issue. My recommendation to let the users figure it out
with a helpful message still stands even if it may sound a bit painful,
because:
1. It prevents data loss (we don't do an automatic delete/create of
permissions)
2. It prevents security oversights (we don't re-use an existing
permission)
3. It shouldn't happen for most use cases
Again, I would love to hear some feedback or other alternatives.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:8>
Django
a proxy.
Reporter: Julien Enselme | Owner: (none)
Type: Bug | Status: new
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Arthur Rio):
* owner: Arthur Rio => (none)
* status: assigned => new
Comment:
I won’t have time to work on this for the next 2 weeks so I’m de-assigning
myself. I’ll pick it up again if nobody does and I’m available to discuss
feedback/suggestions.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:9>
Django
a proxy.
Reporter: Julien Enselme | Owner: Carlton
| Gibson
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* owner: (none) => Carlton Gibson
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:10>
Django
a proxy.
-------------------------------------+-------------------------------------
Reporter: Julien Enselme | Owner: Carlton
| Gibson
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
I'll make a patch for this.
I'll see about raising a suitable warning from the migration but we
already warn in the release notes for this to audit permissions: my
initial thought was that re-using the permission would be OK. (I see
Arthur's comment. Other thoughts?)
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:11>
Django
a proxy.
-------------------------------------+-------------------------------------
Reporter: Julien Enselme | Owner: Carlton
| Gibson
Type: Bug | Status: assigned
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Arthur Rio):
Being my first contribution I wanted to be super (super) careful with
security concerns, but given the existing warning in the release notes for
auditing prior to update, I agree that re-using the permission feels
pretty safe and would remove overhead for people running into this
scenario.
Thanks for taking this on Carlton, I'd be happy to review.
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:12>
Django
a proxy.
-------------------------------------+-------------------------------------
Reporter: Julien Enselme | Owner: Carlton
| Gibson
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"98296f86b340c8c9c968375d59f1d3a3479e60c2" 98296f86]:
{{{
#!CommitTicketReference repository=""
revision="98296f86b340c8c9c968375d59f1d3a3479e60c2"
Fixed #30351 -- Handled pre-existing permissions in proxy model
permissions data migration.
Regression in 181fb60159e54d442d3610f4afba6f066a6dac05.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:13>
Django
a proxy.
-------------------------------------+-------------------------------------
Reporter: Julien Enselme | Owner: Carlton
| Gibson
Type: Bug | Status: closed
Component: contrib.auth | Version: 2.2
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"4f8ebdd09835ad5c010c122ec669c1b224801302" 4f8ebdd0]:
{{{
#!CommitTicketReference repository=""
revision="4f8ebdd09835ad5c010c122ec669c1b224801302"
[2.2.x] Fixed #30351 -- Handled pre-existing permissions in proxy model
permissions data migration.
Regression in 181fb60159e54d442d3610f4afba6f066a6dac05.
Backport of 98296f86b340c8c9c968375d59f1d3a3479e60c2 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30351#comment:14>