Changing to a custom user model mid-project
133 views
Skip to first unread message
Mike Dewhirst
Dec 19, 2019, 11:13:09 PM12/19/19
to Django users
I'm documenting[*] the process I followed but have come to a gray area
and need some expert assistance.
Having achieved a working system with a new database table containing
all the existing data and now called ...
public.common_user
... I notice that it has a user_id sequence called ...
public.auth_user_id_seq
... along with similarly named sequences for similarly named tables
which now exist as common_... tables.
I can easily enough rename those sequences so they match the owning
tables - and I want to do so - but the question is should it be done via
raw SQL within the migration system?
Is there a proper way to align the names?
Another way (which I've tested) is to edit a database dump and reload that.
What is the correct approach? Is it even legal (ORM rules) to rename the
table?
Thanks for any advice
Mike
[*]
https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/
by Tobias McNulty as a variation of Django docs
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#changing-to-a-custom-user-model-mid-project
TL;DR
Custom user documentation (UNFINISHED DRAFT)
Based on Tobias McNulty's "How to Switch to a Custom Django User Model
Mid-Project"[1]
Assumptions
- Existing project without a custom user model
- All migrations are up to date and deployed in production
- Existing auth_user table has data which must be kept
- Relationships with other models exist and must be kept
Strategy
There are two strategies. One is to throw away history, delete all
migrations, empty (truncate) the migrations table and start again.[2]
Very attractive if the project repo is young and history is fresh and
therefore disposable.
The other strategy is to use the migration system to make the switch,
ensuring nothing breaks. That is the Tobias approach and the one
documented here.
This strategy is a genuine bottleneck. All pending changes must be
completed and fully deployed before starting and no planned changes are
commenced until after the switch is fully deployed.
Objective
- Completely align development, staging and production systems
- Series of new migrations
- Series of sql commands to adjust content_type records
- Series of scripts to execute migrations and sql commands
Process
1. Ensure all references to User everywhere (including 3rd party apps)
are indirect[3][4]. Ensure all code concerned with access control and
relying on users or user authentication is covered by unit tests as far
as possible and all tests are passing.
2. Make migrations and apply them. Ensure development, staging and
production systems are all synchronised and each database (structure) is
identical. This starts the bottleneck.
3. Start a new app or use an existing one which has no models.py. The
reason there needs to be initially no models is the migration which
creates the custom user must be '0001_initial.py' to persuade Django
there are no dependency issues. In this documentation I call the app
"common" but it can be anything eg "proj_user", "accounts" etc.
python manage.py startapp common
4. Write a new common/models.py ...
from django.db import models
from django.contrib.auth.models import AbstractUser
class User(AbstractUser):
""" Retain the model name 'User' to avoid unnecessary
refactoring during
the switchover process. Make no other changes here until after
complete
deployment to production.
"""
class Meta:
# use the existing Django users table for the initial migration
db_table = "auth_user"
5. Write a new common/admin.py
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from .models import User
admin.site.register(User, UserAdmin)
6. Include the new app in settings.py among other local apps and adjust
AUTH_USER_MODEL ...
INSTALLED_APPS = [
# ...
'common',
]
AUTH_USER_MODEL = 'common.User'
7. Make the initial migration to create the new User model ...
python manage.py makemigrations --> common/migrations/0001_initial.py
8. Write a script to deploy (rather than execute) the migration as
follows ... [5]
Windows 10 - PostgreSQL 10 ...
:: deploy_migration.bat
:: defeat Django's sanity check by manually entering that migration
in the database
:: and for good measure update content_types to avoid further
Django sanity checks
set host=dev_laptop
set dbowner=whoever
psql --username=%dbowner% --port=5432 --dbname=ssds --host=%host%
--command "INSERT INTO public.django_migrations (app, name, applied)
VALUES ('common', '0001_initial', CURRENT_TIMESTAMP)";
psql --username=%dbowner% --port=5432 --dbname=ssds --host=%host%
--command "UPDATE public.django_content_type SET app_label = 'common'
WHERE app_label = 'auth' and model = 'user'";
Linux (Ubuntu 18.04) - PostgreSQL 10 ...
# fetch_ssds.py [6]
# These next two psql command lines fake an initial migration to create
# a custom-user in a pre-existing project and adjust content_types to
# prevent Django from barfing if it automatically tried to add them
#
import os
host="dev_laptop"
dbowner="whoever"
cmd = "sudo psql --username=%s --port=5432 --dbname=ssds --host=%s
--command \"INSERT INTO public.django_migrations (app, name, applied)
VALUES ('common', '0001_initial', CURRENT_TIMESTAMP);\"" % (dbowner, host)
#
os.system(cmd)
#
cmd = "sudo psql --username=%s --port=5432 --dbname=ssds --host=%s
--command \"UPDATE public.django_content_type SET app_label = 'common'
WHERE app_label = 'auth' and model = 'user';\"" % (dbowner, host)
#
os.system(cmd)
9. After deploying with the above technique in development run all unit
tests and correct any errors or failures both in project code and in the
above scripts. Refresh the development database (structure) from
production (again) and repeat step 8 above and test again. All unit
tests must pass. Important - repeat until perfect.
10. Deploy to staging using one of the above scripts from step 8,
modified for the staging environment. When perfectly deployed on staging
and all testing is done, ensure production is backed up then deploy to
production in similar fashion. This ends the bottleneck.
The balance of this process is optional
11. To be written after resolving sequence naming questions
12. Edit common/models.py then makemigrations to rename the table of
existing users from auth_user to common_user. Finally migrate to execute
the rename to common_user
class User(AbstractUser):
""" Retain the model name 'User' to avoid unnecessary
refactoring during
the switchover process. Make no other changes here until after
complete
deployment to production.
Comment out Meta entirely to migrate to the default table name
"""
pass
#class Meta:
# # use the existing Django users table for the initial
migration
# db_table = "auth_user"
[1]
https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/
by Tobias McNulty as a variation of Django docs
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#changing-to-a-custom-user-model-mid-project
[2] https://code.djangoproject.com/ticket/25313#comment:2 by Aymeric
Augustin
[3]
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#referencing-the-user-model
[4] Note that get_user_model() cannot be called at the module level in
any models.py file (and by extension any file that a models.py imports),
since you'll end up with a circular import. Generally, it's easier to
keep calls to get_user_model() inside a method whenever possible (so
it's called at run time rather than load time), and use
settings.AUTH_USER_MODEL in all other cases. This isn't always possible
(e.g., when creating a ModelForm), but the less you use it at the module
level, the fewer circular imports you'll have to stumble your way
through. (From Tobias [1])
[5] Tobias notes that Django won't permit 'migrate common
--fake-initial' if there are other migrations which include
settings.AUTH_USER_MODEL
[6] fetch_ssds.py is a comprehensive auto-deployment script. Only the
relevant (and simplified) portion is shown.
and need some expert assistance.
Having achieved a working system with a new database table containing
all the existing data and now called ...
public.common_user
... I notice that it has a user_id sequence called ...
public.auth_user_id_seq
... along with similarly named sequences for similarly named tables
which now exist as common_... tables.
I can easily enough rename those sequences so they match the owning
tables - and I want to do so - but the question is should it be done via
raw SQL within the migration system?
Is there a proper way to align the names?
Another way (which I've tested) is to edit a database dump and reload that.
What is the correct approach? Is it even legal (ORM rules) to rename the
table?
Thanks for any advice
Mike
[*]
https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/
by Tobias McNulty as a variation of Django docs
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#changing-to-a-custom-user-model-mid-project
TL;DR
Custom user documentation (UNFINISHED DRAFT)
Based on Tobias McNulty's "How to Switch to a Custom Django User Model
Mid-Project"[1]
Assumptions
- Existing project without a custom user model
- All migrations are up to date and deployed in production
- Existing auth_user table has data which must be kept
- Relationships with other models exist and must be kept
Strategy
There are two strategies. One is to throw away history, delete all
migrations, empty (truncate) the migrations table and start again.[2]
Very attractive if the project repo is young and history is fresh and
therefore disposable.
The other strategy is to use the migration system to make the switch,
ensuring nothing breaks. That is the Tobias approach and the one
documented here.
This strategy is a genuine bottleneck. All pending changes must be
completed and fully deployed before starting and no planned changes are
commenced until after the switch is fully deployed.
Objective
- Completely align development, staging and production systems
- Series of new migrations
- Series of sql commands to adjust content_type records
- Series of scripts to execute migrations and sql commands
Process
1. Ensure all references to User everywhere (including 3rd party apps)
are indirect[3][4]. Ensure all code concerned with access control and
relying on users or user authentication is covered by unit tests as far
as possible and all tests are passing.
2. Make migrations and apply them. Ensure development, staging and
production systems are all synchronised and each database (structure) is
identical. This starts the bottleneck.
3. Start a new app or use an existing one which has no models.py. The
reason there needs to be initially no models is the migration which
creates the custom user must be '0001_initial.py' to persuade Django
there are no dependency issues. In this documentation I call the app
"common" but it can be anything eg "proj_user", "accounts" etc.
python manage.py startapp common
4. Write a new common/models.py ...
from django.db import models
from django.contrib.auth.models import AbstractUser
class User(AbstractUser):
""" Retain the model name 'User' to avoid unnecessary
refactoring during
the switchover process. Make no other changes here until after
complete
deployment to production.
"""
class Meta:
# use the existing Django users table for the initial migration
db_table = "auth_user"
5. Write a new common/admin.py
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from .models import User
admin.site.register(User, UserAdmin)
6. Include the new app in settings.py among other local apps and adjust
AUTH_USER_MODEL ...
INSTALLED_APPS = [
# ...
'common',
]
AUTH_USER_MODEL = 'common.User'
7. Make the initial migration to create the new User model ...
python manage.py makemigrations --> common/migrations/0001_initial.py
8. Write a script to deploy (rather than execute) the migration as
follows ... [5]
Windows 10 - PostgreSQL 10 ...
:: deploy_migration.bat
:: defeat Django's sanity check by manually entering that migration
in the database
:: and for good measure update content_types to avoid further
Django sanity checks
set host=dev_laptop
set dbowner=whoever
psql --username=%dbowner% --port=5432 --dbname=ssds --host=%host%
--command "INSERT INTO public.django_migrations (app, name, applied)
VALUES ('common', '0001_initial', CURRENT_TIMESTAMP)";
psql --username=%dbowner% --port=5432 --dbname=ssds --host=%host%
--command "UPDATE public.django_content_type SET app_label = 'common'
WHERE app_label = 'auth' and model = 'user'";
Linux (Ubuntu 18.04) - PostgreSQL 10 ...
# fetch_ssds.py [6]
# These next two psql command lines fake an initial migration to create
# a custom-user in a pre-existing project and adjust content_types to
# prevent Django from barfing if it automatically tried to add them
#
import os
host="dev_laptop"
dbowner="whoever"
cmd = "sudo psql --username=%s --port=5432 --dbname=ssds --host=%s
--command \"INSERT INTO public.django_migrations (app, name, applied)
VALUES ('common', '0001_initial', CURRENT_TIMESTAMP);\"" % (dbowner, host)
#
os.system(cmd)
#
cmd = "sudo psql --username=%s --port=5432 --dbname=ssds --host=%s
--command \"UPDATE public.django_content_type SET app_label = 'common'
WHERE app_label = 'auth' and model = 'user';\"" % (dbowner, host)
#
os.system(cmd)
9. After deploying with the above technique in development run all unit
tests and correct any errors or failures both in project code and in the
above scripts. Refresh the development database (structure) from
production (again) and repeat step 8 above and test again. All unit
tests must pass. Important - repeat until perfect.
10. Deploy to staging using one of the above scripts from step 8,
modified for the staging environment. When perfectly deployed on staging
and all testing is done, ensure production is backed up then deploy to
production in similar fashion. This ends the bottleneck.
The balance of this process is optional
11. To be written after resolving sequence naming questions
12. Edit common/models.py then makemigrations to rename the table of
existing users from auth_user to common_user. Finally migrate to execute
the rename to common_user
class User(AbstractUser):
""" Retain the model name 'User' to avoid unnecessary
refactoring during
the switchover process. Make no other changes here until after
complete
deployment to production.
Comment out Meta entirely to migrate to the default table name
"""
pass
#class Meta:
# # use the existing Django users table for the initial
migration
# db_table = "auth_user"
[1]
https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/
by Tobias McNulty as a variation of Django docs
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#changing-to-a-custom-user-model-mid-project
[2] https://code.djangoproject.com/ticket/25313#comment:2 by Aymeric
Augustin
[3]
https://docs.djangoproject.com/en/2.2/topics/auth/customizing/#referencing-the-user-model
[4] Note that get_user_model() cannot be called at the module level in
any models.py file (and by extension any file that a models.py imports),
since you'll end up with a circular import. Generally, it's easier to
keep calls to get_user_model() inside a method whenever possible (so
it's called at run time rather than load time), and use
settings.AUTH_USER_MODEL in all other cases. This isn't always possible
(e.g., when creating a ModelForm), but the less you use it at the module
level, the fewer circular imports you'll have to stumble your way
through. (From Tobias [1])
[5] Tobias notes that Django won't permit 'migrate common
--fake-initial' if there are other migrations which include
settings.AUTH_USER_MODEL
[6] fetch_ssds.py is a comprehensive auto-deployment script. Only the
relevant (and simplified) portion is shown.
Carsten Fuchs
Dec 26, 2019, 4:01:45 AM12/26/19
to django...@googlegroups.com
Hello Mike,
unfortunately I cannot answer your questions, but have you considered contacting Tobias directly?
I too am currently at the point of switching two of my projects to a custom user model. But honestly, Tobias' procedure looks complicated and error prone to me. Aymeric's description doesn't cover some of the details (some of your and Tobias' steps are needed there too), but at least for me it is easier to understand and therefore I feel more comfortable with it.
I was wondering though if I properly understand the downside of Aymeric's approach: All that it means is that once it is done, we cannot checkout an earlier version of the code across the breaking step and un-apply the migrations that were added since then?
Except in trivial circumstances, I've never done this anyway, and considering the lower complexity, it seems a small price to pay.
Can anyone confirm this?
Best regards,
Carsten
Am 20.12.19 um 05:12 schrieb Mike Dewhirst:
unfortunately I cannot answer your questions, but have you considered contacting Tobias directly?
I too am currently at the point of switching two of my projects to a custom user model. But honestly, Tobias' procedure looks complicated and error prone to me. Aymeric's description doesn't cover some of the details (some of your and Tobias' steps are needed there too), but at least for me it is easier to understand and therefore I feel more comfortable with it.
I was wondering though if I properly understand the downside of Aymeric's approach: All that it means is that once it is done, we cannot checkout an earlier version of the code across the breaking step and un-apply the migrations that were added since then?
Except in trivial circumstances, I've never done this anyway, and considering the lower complexity, it seems a small price to pay.
Can anyone confirm this?
Best regards,
Carsten
Am 20.12.19 um 05:12 schrieb Mike Dewhirst:
Mike Dewhirst
Dec 26, 2019, 7:00:27 AM12/26/19
to django...@googlegroups.com, mi...@dewhirst.com.au
Carsten
I have made some progress since documenting it and intend to refine it more.
Experimenting has boosted my confidence in Postgres and there is a bit of scripting involved to fake-apply migrations in the correct sequence.
At the moment I'm time poor and docco has to wait but can say I got both methods working.
I decided in the end to go with Aymeric's because it is simpler and importantly for me resulted in reducing the gross number of migration files - which of course get frequently scanned by Django during unit testing. So that is a continuing bonus.
More later ... but (ymmv) I am confirming :)
Cheers
Mike
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/0eb107a7-cefe-fa3f-e145-088225c084d9%40cafu.de.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/0eb107a7-cefe-fa3f-e145-088225c084d9%40cafu.de.
Reply all
Reply to author
Forward
0 new messages