[Django] #25313: Document how to migrate from a built-in User model to a custom User model
Django
---------------------------------------+------------------------
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Component: Migrations | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------+------------------------
(I'd have thought we might already have a ticket for this, but I couldn't
find it.)
So far our answer here has been "sorry, you can't do it, unless you're
very familiar with the depths of the migrations system and willing to put
a lot of time into it."
I don't believe that this is a tenable or defensible answer. It puts too
many of our users, too frequently, into an impossible quandary. I think we
need to clearly document how you can do it today, even if the process is
nasty and ugly. Hopefully seeing that nasty and ugly documentation might
clarify what could be improved to make the process less nasty and ugly.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* component: Migrations => Documentation
* stage: Unreviewed => Accepted
Old description:
> (I'd have thought we might already have a ticket for this, but I couldn't
> find it.)
>
> So far our answer here has been "sorry, you can't do it, unless you're
> very familiar with the depths of the migrations system and willing to put
> a lot of time into it."
>
> I don't believe that this is a tenable or defensible answer. It puts too
> many of our users, too frequently, into an impossible quandary. I think
> we need to clearly document how you can do it today, even if the process
> is nasty and ugly. Hopefully seeing that nasty and ugly documentation
> might clarify what could be improved to make the process less nasty and
> ugly.
New description:
So far our answer here has been "sorry, you can't do it, unless you're
very familiar with the depths of the migrations system and willing to put
a lot of time into it."
I don't believe that this is a tenable or defensible answer. It puts too
many of our users, too frequently, into an impossible quandary. I think we
need to clearly document how you can do it today, even if the process is
nasty and ugly. Hopefully seeing that nasty and ugly documentation might
clarify what could be improved to make the process less nasty and ugly.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:1>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
I did it at least twice. Unfortunately I don't remember all the details.
I think a reasonable procedure is:
1. Create a custom user model identical to `auth.User`, call it `User` (so
many-to-many tables keep the same name) and set `db_table='auth_user'` (so
it uses the same table)
2. Throw away all your migrations
3. Recreate a fresh set of migrations
4. Sacrifice a chicken, perhaps two if you're anxious; also make a backup
of your database
5. Truncate the `django_migrations` table
6. Fake-apply the new set of migrations
7. Unset `db_table`, make other changes to the custom model, generate
migrations, apply them
It is highly recommended to do this on a database that enforces foreign
key constraints. Don't try this on SQLite on your laptop and expect it to
work on Postgres on the servers!
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:2>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
Hmm. I thought I recalled you mentioning (at DUTH last year?) that you
achieved it using `SeparateDatabaseAndState`, without the need to wipe
migrations and start over. Maybe that was someone else.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:3>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
At some point I thought it was doable with `SeparateDatabaseAndState` but
eventually I realized that it isn't.
When you change `settings.AUTH_USER_MODEL`, suddenly Django has a
different view of the migration history. Piling hacks cannot hide this
fact. I don't think it's possible to salvage migration history when
changing `AUTH_USER_MODEL`.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:4>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
Hmm. Clearly it must be partially possible, otherwise you could never do
this at all if you use any third-party apps that link to `User` (since you
won't be changing their migration files). The whole reason we special-case
swappable models in migrations (instead of just treating them concretely)
is to allow for migrations to not be dependent on the value of
`AUTH_USER_MODEL`, so that reusable apps depending on `User` can still
generate workable migrations.
It's true that changing `AUTH_USER_MODEL` changes the _meaning_ of
historical migrations in some sense, but it still seems to me that if our
approach for reusable apps actually works, the same migration files ought
to be salvageable (presuming that when you switch `AUTH_USER_MODEL` you
point it to a new model that is initially exactly the same as the previous
one, and then only modify it in later, separate migrations).
But you've actually done this and I haven't, so I'm probably wrong...
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:5>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
Replying to [comment:5 carljm]:
> Hmm. Clearly it must be partially possible, otherwise you could never do
this at all if you use any third-party apps that link to `User` (since you
won't be changing their migration files).
Indeed you don't change the migration files. But you drop
`django_migrations`, change `settings.AUTH_USER_MODEL` and repopulate
`django_migrations`. While some migrations have the same name in
`django_migrations`, you don't have the same set of migrations and their
semantic has changed. Specifically:
- you add at least the migration that creates your custom user model
- the `auth.000_initial` migration has a different semantic because it
doesn't create a table for `auth.User`
> The whole reason we special-case swappable models in migrations (instead
of just treating them concretely) is to allow for migrations to not be
dependent on the value of `AUTH_USER_MODEL`, so that reusable apps
depending on `User` can still generate workable migrations.
That's the trick. A migration file that contains a migration that uses the
`swappable` option isn't a self-contained definition. Only the combination
of the migration file and the value of `getattr(django.conf.settings,
swappable)` is. This isn't reflected in the structure of
`django_migrations` because Django currently assumes `AUTH_USER_MODEL` to
be immutable.
> It's true that changing `AUTH_USER_MODEL` changes the _meaning_ of
historical migrations in some sense, but it still seems to me that if our
approach for reusable apps actually works, the same migration files ought
to be salvageable (presuming that when you switch `AUTH_USER_MODEL` you
point it to a new model that is initially exactly the same as the previous
one, and then only modify it in later, separate migrations).
I remember feeling smart, then making a huge mess of a SQLite database,
feeling dumb, editing the dump manually to fix broken FK constaints,
feeling lucky.
> But you've actually done this and I haven't, so I'm probably wrong...
Well, perhaps there's a way.
Even then I'd recommend the procedure I suggested above because:
- it's reasonably convenient: people have a fair chance to execute it
successfully
- it makes it clear that you're voiding your warranty (not that Django
comes with a warranty, but you get the point)
- it's possible to reason about why it works
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:6>
Django
Reporter: carljm | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shaib):
<brainstorming idea="halfbaked">
Perhaps it is possible to create a migration operation for changing a
swappable model.
Something like:
{{{
ChangeSwappableModel(
setting="AUTH_USER_MODEL",
old="auth.User",
new="my_auth.User"
)
}}}
This would be a migration in the `my_auth` app.
It would need to introspect the database, checking the constraints
corresponding to `my_auth.User`'s reverse relationships, and verifying
that they indeed point to the correct table (if they point to `auth_user`,
change them; if to `my_auth_user`, leave them be; otherwise, error out).
To change the swappable model, one would:
1) Create the set of migrations for creating the new user model, porting
existing data to it, and the `ChangeSwappableModel` operation;
2) Create a "squashing" migration replacing them by just creating the new
model, so new databases don't have to suffer
I think with careful migrations embargos in the right times this could be
made to work. I don't have it in me to work out all the details now, and
I'm probably missing something essential.
</brainstorming>
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:7>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by James Addison):
I think I've followed a very similar process in previous projects (older
Django versions) to what Aymeric mentions above in
https://code.djangoproject.com/ticket/25313#comment:2 - although I don't
think you can ever 'unset' `db_table` without having to do low level SQL
changes?
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:8>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Justin Smith):
Replying to [comment:2 Aymeric Augustin]:
> I did it at least twice. Unfortunately I don't remember all the details.
>
> I think a reasonable procedure is:
>
> 1. Create a custom user model identical to `auth.User`, call it `User`
(so many-to-many tables keep the same name) and set `db_table='auth_user'`
(so it uses the same table)
> 2. Throw away all your migrations
> 3. Recreate a fresh set of migrations
> 4. Sacrifice a chicken, perhaps two if you're anxious; also make a
backup of your database
> 5. Truncate the `django_migrations` table
> 6. Fake-apply the new set of migrations
> 7. Unset `db_table`, make other changes to the custom model, generate
migrations, apply them
>
> It is highly recommended to do this on a database that enforces foreign
key constraints. Don't try this on SQLite on your laptop and expect it to
work on Postgres on the servers!
Just recently had to go through this process using a Microsoft SQL Server
backend and used the steps above as my guideline. Just thought I'd drop in
and include some of my notes just in case they can help anyone in the
future.
Notes (by step):
1. Make sure the custom user model identical to auth.User is an
`AbstractUser` model. I originally made this mistake because I did an
`inspectdb auth_user` and just copy/pasted so I left it as `models.Model`
at first. Since I copied and pasted from `inspectdb` I went ahead and
removed `managed = False`
2. Quick shortcut to delete migrations for all apps in a project I used
was `find . -path "*/migrations/*.py" -not -name "__init__.py" -delete`.
3. No additional notes
4. Not kidding about the back up I had to start over a few times
5. No additional notes
6. I did `--fake-initial` first few times and not `--fake`
7. No notes
Thank you very much for posting this in the first place I am not sure I
would have figured this out on my own and saved us mid-project. I have
learned my lesson about starting a django project and not setting up
custom user model.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:9>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Luke Plant):
I created this project which was my attempt to automate the process:
https://bitbucket.org/spookylukey/django_custom_user_migration
However, I think Aymeric's solution looks better.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:10>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Pi Delport):
These are steps we took to switch our system to a custom
`AUTH_USER_MODEL`, for the record:
0. (Take full backups!)
1. Dump the database with: `django-admin dumpdata --natural-primary
--natural-foreign --exclude contenttypes.contenttype`
2. Run the JSON dump through a script that rewrites references from the
old user model to the new one. (See below.)
3. Define our new custom user model as a `AbstractUser` subclass, with no
other schema changes. Update `AUTH_USER_MODEL` to it, nuke all our app's
old migrations, and make fresh initial migrations.
4. Create and `django-admin migrate` a fresh new database, and load the
rewritten dump.
After this point, we can customise our user model with normal Django
migrations.
The script to rewrite the dump iterates through the list of objects, and
rewrites:
* The user's' `model` itself.
* The user's `user_permissions` field's references.
* The `auth.group` `permissions` field's references.
* The `auth.permission` and `admin.logentry` `content_type` fields.
* Any other references to the old `auth.User` type will need rewriting
too.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:11>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by dustingtorres):
Replying to [comment:2 Aymeric Augustin]:
> I did it at least twice. Unfortunately I don't remember all the details.
>
> I think a reasonable procedure is:
>
> 1. Create a custom user model identical to `auth.User`, call it `User`
(so many-to-many tables keep the same name) and set `db_table='auth_user'`
(so it uses the same table)
> 2. Throw away all your migrations
> 3. Recreate a fresh set of migrations
> 4. Sacrifice a chicken, perhaps two if you're anxious; also make a
backup of your database
> 5. Truncate the `django_migrations` table
> 6. Fake-apply the new set of migrations
> 7. Unset `db_table`, make other changes to the custom model, generate
migrations, apply them
>
> It is highly recommended to do this on a database that enforces foreign
key constraints. Don't try this on SQLite on your laptop and expect it to
work on Postgres on the servers!
Thanks for this, I followed your steps and Justin Smith's notes and want
to add one more:
6b. Any model that has a link to ContentType may be linked to the now
stale auth.User content type. I went through and updated all models that
were pointing to auth.User content type to my new User model. You can find
out what type of objects would be removed due to cascading delete of the
stale auth.User content type by running `manage.py
remove_stale_contenttypes` **and make sure to answer NO** when asked if
you want to proceed.
If you don't unset `db_table` the above step may be optional since the
content type lookup will use auth.User model which will still have a valid
database table (although I don't recommend relying on this).
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:12>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by mikob):
Thank you Aymeric et. al for these steps, they worked for me. I made one
silly mistake in 1. not creating the user properly. For other people who
might also get stuck on this, here is what your User model should look
like for step 1:
{{{
class User(AbstractUser):
class Meta:
db_table = 'auth_user'
}}}
Replying to [comment:12 Dustin Torres]:
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:13>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by David Sanders):
On the point of "Throw away all your migrations", I'd like to add a note
that some data migrations that setup fixtures may need to be kept. This
may involve extracting these operations from their respective migrations
and updating them to use the latest schema then appending them to the
resulting initial migrations generated.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:14>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam (Chainz) Johnson):
For anyone landing here, Tobias McNulty did half the work documenting it
in this blog post: https://www.caktusgroup.com/blog/2019/04/26/how-switch-
custom-django-user-model-mid-project/ .
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:15>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carsten Fuchs):
* cc: Carsten Fuchs (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:16>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jakub Dorňák):
I tried to address the mid-project migration in https://pypi.org/project
/django-user-unique-email/
It seems to work with SQLite and MySQL. I'll appreciate any feedback.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:17>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carsten Fuchs):
Based on
- Aymeric's [https://code.djangoproject.com/ticket/25313#comment:2
approach above],
- Tobias's blog post https://www.caktusgroup.com/blog/2019/04/26/how-
switch-custom-django-user-model-mid-project/,
- Vitor's blog post
https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-
migrations.html
I tried to write a summary. These steps worked well for me, although I'm
sure there is room for improvement:
== Assumptions
- Your project doesn't have a custom user model yet.
- All existing users must be kept.
- There are no pending migrations and all existing migrations are
applied.
- It is acceptable that all previous migrations are lost and can no
longer be unapplied, even if you use version control to checkout old
commits that still have the migration files. This is the relevant downside
of this approach.
== Preparations
- Make sure that ''any third party apps'' that refer to the Django User
model only use the
[https://docs.djangoproject.com/en/3.0/topics/auth/customizing
/#referencing-the-user-model generic referencing methods].
- Make sure that ''your own reusable apps'' (apps that are intended to
be used by others) use the generic reference methods.
- I suggest to ''not'' do the same with your project apps:
The switch to a custom user model is only done once per project and
never again.
It is easier (and in my opinion also clearer) to change `from
django.contrib.auth.models import User`
to something else (as detailed below) than replacing it with generic
references that are not needed in project code.
- Make sure that you have a backup of your code and database!
== Update the code
- You can create the new user model in any existing app or a newly
created one.
My preference is to create a new app:
{{{#!python
./manage.py startapp Accounts
}}}
I chose the name "Accounts", but any other name works as well.
- Aymeric: „Create a custom user model identical to `auth.User`,
call it `User` (so many-to-many tables keep the same name)
and set `db_table='auth_user'` (so it uses the same table).“
In `Accounts/models.py`:
{{{#!python
from django.contrib.auth.models import AbstractUser
from django.db import models
class User(AbstractUser):
class Meta:
db_table = 'auth_user'
}}}
- In `settings.py`, add the app to `INSTALLED_APPS` and update the
`AUTH_USER_MODEL` setting:
{{{#!python
INSTALLED_APPS = (
# ...
'Accounts',
)
AUTH_USER_MODEL = 'Accounts.User'
}}}
- In your project code, replace all imports of the Django user model:
{{{#!python
from django.contrib.auth.models import User
}}}
with the new, custom one:
{{{#!python
from Accounts.models import User
}}}
- Delete all old migrations. (Beforehand, see if
[https://code.djangoproject.com/ticket/25313#comment:14 comment 14] is
relevant to you!)
For example, in the project root:
{{{#!sh
find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete
}}}
- Create new migrations from scratch:
{{{#!sh
./manage.py makemigrations
}}}
- Make any changes to your `admin.py` files as required.
(I cannot give any solid information here, but this is not crucial for
the result and so the details can still be reviewed later.)
- Make sure that your testsuite completes successfully! (A fresh test
database must be used, it cannot be kept from previous runs.)
- At this point, the changes to the code are complete. This is a good
time for a commit.
Note that we're done – except that the new migration files mismatch the
contents of the `django_migrations` table.
(It may even be possible to serve your project at this point: It's easy to
back off before the database is actually changed. ''Only'' do this if you
understand that you ''cannot even touch'' the migrations system as long as
the steps below are not completed!)
== Update the database
- Truncate the `django_migrations` table. MySQL 8 example:
{{{#!sql
TRUNCATE TABLE django_migrations;
}}}
This is possibly different for other databases or verions of MySQL <
8.
- Fake-apply the new set of migrations
{{{#!sh
./manage.py migrate --fake
}}}
- Check the ContentTypes as described at
[https://code.djangoproject.com/ticket/25313#comment:12 comment 12]
== Conclusion
- The upgrade to the custom user model is now complete.
You can make changes to this model and generate and apply migrations
for it as with any other models.
- As a first step, you may wish to unset `db_table` and generate and
apply the resulting migrations.
- In my opinion, the `startproject` management command should anticipate
the introduction of a custom user model.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:18>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carsten Fuchs):
After the change, rewriting the ContentType for the admin log's LogEntry
instances was helpful:
{{{
#!python
./manage.py shell
>>>
>>> from django.contrib.admin.models import LogEntry
>>> from django.contrib.contenttypes.models import ContentType
>>>
>>> auth_user = ContentType.objects.get(app_label='auth', model='user')
>>> accouts_user = ContentType.objects.get(app_label='Accounts',
model='user')
>>>
>>> for le in LogEntry.objects.filter(content_type=auth_user):
... le.content_type = accouts_user
... le.save()
...
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:19>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Eric Theise):
Many thanks to Carsten Fuchs for integrating the various resources into a
single procedure that worked well for me in my simple situation. One
caveat: after running the `find` commands in the **Delete all old
migrations** step I received a `ModuleNotFoundError: No module named
'django.db.migrations.migration'` error and had to uninstall/reinstall
Django before proceeding.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:20>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by BrandonWoodward):
I believe the following to be the easiest method.
1. Follow the preparations in
[https://code.djangoproject.com/ticket/25313#comment:18 Carsten's
solution].
2. Create a new app with no existing migrations and migrate an empty
migration.
{{{#!python
./manage.py startapp user
./manage.py makemigrations user --empty
./manage.py migrate user
}}}
3. Update the migration to match the timestamp of the initial auth
migration.
{{{#!python
./manage.py shell
>>> from django.db.migrations.recorder import MigrationRecorder
>>> auth_initial = MigrationRecorder.Migration.objects.get(app='auth',
name='0001_initial')
>>> user_initial = MigrationRecorder.Migration.objects.get(app='user',
name='0001_initial')
>>> user_initial.applied = auth_initial.applied
>>> user_initial.save()
}}}
4. Delete `user 0001_initial.py`, set `AUTH_USER_MODEL = 'user.user'` in
your settings then run `./manage.py makemigrations` after creating the
following `user` model.
{{{#!python
from django.contrib.auth import models as auth_models
class User(auth_models.AbstractUser):
class Meta:
db_table = 'auth_user'
}}}
5. Update the content types
{{{#!python
./manage.py shell
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.filter(app_label='auth',
model='user').update(app_label='user')
}}}
6. Delete the model inherited from `auth_models.AbstractUser` and create
your custom user model. Run `makemigrations` then `migrate` after making
your changes and from this point on everything should be working properly.
{{{#!python
class User(auth_models.AbstractBaseUser,
auth_models.PermissionsMixin):
...
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:21>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Old description:
> So far our answer here has been "sorry, you can't do it, unless you're
> very familiar with the depths of the migrations system and willing to put
> a lot of time into it."
>
> I don't believe that this is a tenable or defensible answer. It puts too
> many of our users, too frequently, into an impossible quandary. I think
> we need to clearly document how you can do it today, even if the process
> is nasty and ugly. Hopefully seeing that nasty and ugly documentation
> might clarify what could be improved to make the process less nasty and
> ugly.
New description:
So far our answer here has been "sorry, you can't do it, unless you're
very familiar with the depths of the migrations system and willing to put
a lot of time into it."
I don't believe that this is a tenable or defensible answer. It puts too
many of our users, too frequently, into an impossible quandary. I think we
need to clearly document how you can do it today, even if the process is
nasty and ugly. Hopefully seeing that nasty and ugly documentation might
clarify what could be improved to make the process less nasty and ugly.
--
Comment (by Julien Palard):
I tried the procedure from
[https://code.djangoproject.com/ticket/25313#comment:18 comment 18] today
and it worked perfectly [https://framagit.org/hackinscience/hkis-
website/-/commit/3ec73d39af57394546a9499052fb2ebda2c3aff9 (see my
commit)].
P.S.: About [https://code.djangoproject.com/ticket/25313#comment:20
comment 20] I bet the find deleted things in the venv. Maybe add a
`-maxdepth 3`?
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:22>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by johanneswilm):
Is anyone working on fixing this in Django itself? The workarounds may
work in a lot of cases, but when you have an application set up to rely on
the Django migration system for any database change and have it deployed
on thousands of servers that are not under your direct control, it is
somewhat useless.
Would it not be possible to create one special type of migration for this
purpose? Maybe something as simple as let the migration that creates the
user model check wither `auth.User` has been in use, and if so, copy over
entries from there.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:23>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by johanneswilm):
The following approach has not been tested with all database backends
(only sqlite), but it seems to work by only using migrations, no raw SQL
and not destroying any existing migrations. This is based of Tobias
McNulty's walk-through [1], with some modifications to avoid doing
anything "nasty".
----
0. Assumptions:
* You have an existing project without a custom user model.
* You're using Django's migrations, and all migrations are up-to-date (and
have been applied to the production database).
* You have an existing set of users that you need to keep, and any number
of models that point to Django's built-in User model.
* **You have an existing django app with existing migrations to which you
want to add the custom user model.**
* **You need the same code to run both for new installations of your code
base and for existing ones.
1. First, assess any third party apps to make sure they either don't have
any references to the Django's User model, or if they do, that they use
Django's generic methods for referencing the user model.
2. Next, do the same thing for your own project. Go through the code
looking for any references you might have to the User model, and replace
them with the same generic references. In short, you can use the
get_user_model() method to get the model directly, or if you need to
create a ForeignKey or other database relationship to the user model, use
settings.AUTH_USER_MODEL (which is simply a string corresponding to the
appname.ModelName path to the user model).
Note that get_user_model() cannot be called at the module level in any
models.py file (and by extension any file that a models.py imports), since
you'll end up with a circular import. Generally, it's easier to keep calls
to get_user_model() inside a method whenever possible (so it's called at
run time rather than load time), and use settings.AUTH_USER_MODEL in all
other cases. This isn't always possible (e.g., when creating a ModelForm),
but the less you use it at the module level, the fewer circular imports
you'll have to stumble your way through.
3. In the existing app in which you want the custom User model to live (we
will call it "user" in this example), add the following to the models.py
file:
{{{
from django.db import models
from django.contrib.auth.models import AbstractUser
class User(AbstractUser):
class Meta:
db_table = 'auth_user'
}}}
4. Edit the migration with a name starting with "0001" of this app.
Under "operations" paste the following as the first item:
{{{
migrations.CreateModel(
name='User',
fields=[
('id', models.AutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
('password', models.CharField(max_length=128,
verbose_name='password')),
('last_login', models.DateTimeField(blank=True, null=True,
verbose_name='last login')),
('is_superuser', models.BooleanField(default=False,
help_text='Designates that this user has all permissions without
explicitly assigning them.', verbose_name='superuser status')),
('username', models.CharField(error_messages={'unique': 'A
user with that username already exists.'}, help_text='Required. 150
characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150,
unique=True,
validators=[django.contrib.auth.validators.UnicodeUsernameValidator()],
verbose_name='username')),
('first_name', models.CharField(blank=True,
max_length=150, verbose_name='first name')),
('last_name', models.CharField(blank=True, max_length=150,
verbose_name='last name')),
('email', models.EmailField(blank=True, max_length=254,
verbose_name='email address')),
('is_staff', models.BooleanField(default=False,
help_text='Designates whether the user can log into this admin site.',
verbose_name='staff status')),
('is_active', models.BooleanField(default=True,
help_text='Designates whether this user should be treated as active.
Unselect this instead of deleting accounts.', verbose_name='active')),
('date_joined',
models.DateTimeField(default=django.utils.timezone.now, verbose_name='date
joined')),
('groups', models.ManyToManyField(blank=True,
help_text='The groups this user belongs to. A user will get all
permissions granted to each of their groups.', related_name='user_set',
related_query_name='user', to='auth.Group', verbose_name='groups')),
('user_permissions', models.ManyToManyField(blank=True,
help_text='Specific permissions for this user.', related_name='user_set',
related_query_name='user', to='auth.Permission', verbose_name='user
permissions')),
],
options={
'db_table': 'auth_user',
},
managers=[
('objects', django.contrib.auth.models.UserManager()),
],
),
}}}
Under "dependencies" add:
{{{
('auth', '0012_alter_user_first_name_max_length'),
}}}
and, if present, remove
{{{
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
}}}
(The exact code to use here will likely change over time with newer
versions of Django. You can find the current code by creating a new app
temporarily, add the User model to it and then look at the migration file
`./manage.py makemigrations` produces.)
5. Create a new data migration to the user app by typing: `./manage.py
makemigrations --empty user`
6. Edit the newly created migration:
Under "operations" add `migrations.RunPython(change_user_type),`
Add this function to the top of the file:
{{{
def change_user_type(apps, schema_editor):
ContentType = apps.get_model('contenttypes', 'ContentType')
ct = ContentType.objects.filter(
app_label='auth',
model='user'
).first()
if ct:
ct.app_label = 'user'
ct.save()
}}}
7. You can now migrate to the custom user model by running `./manage.py
migrate`. This should always work, however, it will do so in two different
ways depending on whether it is run on an existing or a new instance:
* On a new instance, the new user model will be created in migration 0001.
The last migration will have no effect.
* On an existing instance, the migration 0001 will be ignored as the
system has already previously applied migration 0001 and will therefore
now ignore it. Instead the last migration will have the effect of change
the app of the user model.
8. Now, you should be able to make changes to your users.User model and
run makemigrations / migrate as needed. For example, as a first step, you
may wish to rename the auth_user table to something in your users app's
namespace. You can do so by removing db_table from your User model, so it
looks like this:
{{{
class User(AbstractUser):
pass
}}}
You'll also need to create and run a new migration to make this change in
the database:
`./manage.py makemigrations --name rename_user_table`
`./manage.py migrate`
----
Please let me know if this works for you - especially with other
databases.
[1] https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-
user-model-mid-project/ with mo
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:24>
Django
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Max F.):
The procedure in comment:24 worked perfectly for me on MySQL (ubuntu
16.04) and on SQLite (macOS 11.2.3). I had made a couple mistakes on my
first attempt and wanted to leave some feedback for others going through
the process:
1. In the `change_user_type` function I mistakenly didn't change
`ct.app_label = 'user'` to my apps name.. :(
2. I added `elidable=True` to the RunPython operation so that I could
squash the migrations later on
Quick side note, when renaming the `auth_user` table I did run into an
issue with my CI when using ubuntu 16.04. It was fixed when I upgraded to
20.04 LTS but for reference this was the message: ''Renaming the
'auth_user' table while in a transaction is not supported on SQLite < 3.26
because it would break referential integrity. Try adding `atomic = False`
to the Migration class.''
Thanks for the explanation Johannes!
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:25>
Django
-------------------------------+------------------------------------
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by Thomas Grainger):
I've used a custom management command to automate this - avoiding raw sql
by using MigrationRecorder
{{{
from __future__ import annotations
import argparse
from django.contrib.contenttypes.models import ContentType
from django.core.management import call_command
from django.core.management.base import BaseCommand
from django.db import DEFAULT_DB_ALIAS, connections
from django.db.migrations.recorder import MigrationRecorder
class Command(BaseCommand):
"""
Custom management command for forcing the django custom user migration
"""
help = "fake users.migrations.0001_initial and switch auth.User
content types to users.User if required"
def add_arguments(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument(
"--database",
default=DEFAULT_DB_ALIAS,
help="Nominates a database to apply migrations for. Defaults
to the 'default' database.",
)
def handle(self, *args: object, database: object = DEFAULT_DB_ALIAS,
**options: object) -> None:
"""
fake users.migrations.0001_initial and switch auth.User content
types to users.User if required"
"""
assert isinstance(database, str)
connection = connections[database]
recorder = MigrationRecorder(connection)
recorded_migrations = recorder.applied_migrations()
if (
# not faked yet
("users", "0001_initial") not in recorded_migrations
# django auth migrations have already been applied
and ("auth", "0001_initial") in recorded_migrations
):
# https://www.caktusgroup.com/blog/2019/04/26/how-switch-
custom-django-user-model-mid-project/
recorder.record_applied(app="users", name="0001_initial")
ContentType.objects.filter(app_label="auth",
model="user").update(app_label="users")
call_command("migrate", database=database)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:26>
Django
-------------------------------+------------------------------------
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by Tal500):
comment:24 was working well for me, together with a huge recommendation to
comment:25 ({{{elidable = True}}} and {{{atomic = False}}}).
I would like to add what you should do in case the "user" Django app has
no previous active migrations yet (which was my case), by still using the
proposal in comment:24.
Before stage 3, you should fake an initial migration, and apply it:
* Run {{{./manage.py makemigrations --empty user}}}. This will output an
empty migration file {{{0001_initial.py}}}.
* Run {{{./manage.py migrate user}}}.
Then, you should do the rest of the stages of comment:24, when stage 5
applies to the migration file {{{0001_initial.py}}} (after it has already
been done in the migration I mentioned).
BTW: After doing this process by my own both in local machine and on the
remote server, I recommend everyone to commit in the version control
system each step requires applying migration({{{./manage.py migrate
user}}}), so on the remove server you could checkout each step requires
applying migration and run {{{./manage.py migrate user}}} manually for
each such commit.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:27>
Django
-------------------------------+------------------------------------
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by pothiers):
At the start of our first real Django project we anticipated needing
authentication eventually, so followed instructions from
https://docs.djangoproject.com/en/4.0/topics/auth/customizing
/#customizing-authentication-in-django. (we expected to use an email
address instead of username for the identification token). We basically
stubbed it until we knew what we needed. For our 2nd project we thought we
would never need authentication so did nothing relating to users. We are
now a likely to nuke our databases and rebuild to avoid the migration
complications described in this ticket.
Given the impact of customizing authentication mid-project, why doesn't
the default setup simply create an appropriate stub? Is the problem
choosing an appropriate stub? Maybe that's easier than solving migration?
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:28>
Django
-------------------------------+------------------------------------
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by Tim Graham):
Using `AUTH_USER_MODEL` in the `startproject` template is proposed in
#27909.
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:29>
Django
-------------------------------+------------------------------------
Reporter: Carl Meyer | Owner: nobody
Type: New feature | Status: new
Component: Documentation | Version: 1.8
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by David Freiholtz):
With reservation for that I'm still pretty much a beginner and that I only
tried it on a small project.
I had great help to migrate from a built-in User model to a custom User
model from this page:
https://code.djangoproject.com/ticket/25313
And this one https://simpleisbetterthancomplex.com/article/2021/07/08
/what-you-should-know-about-the-django-user-model.html
Then I wrote up a short bullet list on how to do it on my django based
mastodon
https://freiholtz.net/@da...@freiholtz.net/posts/159869836921574712/
TL;DR
First, only use get_user_model in your code
1. Create a datadump python manage.py dumpdata > datadump.json
2. Create the new user model and set db_table = 'auth_user' in Meta
section.
3. Truncate the database
4. python manage.py makemigrations
5. python manage.py migrate
6. Edit {"model": "auth.user" to {"model": "myapp.customuser" in
datadump.json
7. python manage.py loaddata datadump.json
--
Ticket URL: <https://code.djangoproject.com/ticket/25313#comment:30>