[Django] #33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3

299 views
Skip to first unread message

Django

unread,
Aug 6, 2022, 4:33:43 PM8/6/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: DrChai | Owner: nobody
Type: Bug | Status: new
Component: | Version: 4.0
Migrations | Keywords: collation unique
Severity: Normal | citext
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Creating a non-deterministic collation used for case insensitive fields to
replace the old approach,
[https://docs.djangoproject.com/en/4.0/ref/contrib/postgres/fields
/#citext-fields CIText], would raise `django.db.utils.NotSupportedError:
nondeterministic collations are not supported for operator class
"varchar_pattern_ops"` when the `unique=True` is also set on the
field.Since unique implies the creation of an index despite setting
`db_index=False` explicitly.
here is a sample code to reproduce:
{{{
#!python
class Migration(migrations.Migration):

initial = True

dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]

operations = [
CreateCollation(
'ci',
provider='icu',
locale='und-u-ks-level2',
deterministic=False
),
migrations.CreateModel(
name='TestgUser',
fields=[
('id', models.BigAutoField(auto_created=True,
primary_key=True, serialize=False, verbose_name='ID')),
('password', models.CharField(max_length=128,
verbose_name='password')),
('username', models.CharField(db_collation='ci',
max_length=30, unique=True, verbose_name='username')),
],
options={
'verbose_name': 'Test User',
'verbose_name_plural': 'Test Users',
'abstract': False,
},
),
]
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33901>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Aug 6, 2022, 4:50:34 PM8/6/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: DrChai | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Accepted
citext |
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* version: 4.0 => dev
* stage: Unreviewed => Accepted


Comment:

Closely related if not a duplicate of #24082 but since `db_collation` is
the documented way off the now deprecated `contrib.postgres.CIText` I
think it's worth keeping the two issue distinct for now.

Could we have the Postgres schema editor not create these `opclass`
indexes when a custom `db_collation` is specified until we get a consensus
on #24082?

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:1>

Django

unread,
Aug 7, 2022, 12:01:26 AM8/7/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: nobody

Type: Bug | Status: new
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Accepted
citext |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ed Chai):

Replying to [comment:1 Simon Charette]: I found #24082 but filed the issue
for the same reason, `CIText` is deprecated and `db_collation` is
documented way to do this job without any conflicts. Removing the
duplicate index is an optimal solution and unique only for constraint
seems semantic to me. Although I know some others still need `LIKE` and
regex expression for regular case sensitive `CharFeild` and `Textfield`.


> Closely related if not a duplicate of #24082 but since `db_collation` is
the documented way off the now deprecated `contrib.postgres.CIText` I
think it's worth keeping the two issue distinct for now.
>
> Could we have the Postgres schema editor not create these `opclass`
indexes when a custom `db_collation` is specified until we get a consensus
on #24082?

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:2>

Django

unread,
Aug 10, 2022, 4:06:59 AM8/10/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Accepted
citext |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

Replying to [comment:1 Simon Charette]:

> Could we have the Postgres schema editor not create these `opclass`
indexes when a custom `db_collation` is specified until we get a consensus
on #24082?

Sounds good to me. Ed, would you like to prepare a patch?

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:3>

Django

unread,
Aug 10, 2022, 5:02:25 PM8/10/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Accepted
citext |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ed Chai):

Replying to [comment:3 Mariusz Felisiak]: Sure, I think i can make a pull
request this week.


> Replying to [comment:1 Simon Charette]:
> > Could we have the Postgres schema editor not create these `opclass`
indexes when a custom `db_collation` is specified until we get a consensus
on #24082?
>
> Sounds good to me. Ed, would you like to prepare a patch?

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:4>

Django

unread,
Aug 15, 2022, 12:29:01 AM8/15/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: assigned

Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* owner: nobody => Ed Chai
* status: new => assigned
* has_patch: 0 => 1
* stage: Accepted => Ready for checkin


Comment:

[https://github.com/django/django/pull/15959 PR]

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:5>

Django

unread,
Aug 15, 2022, 1:30:32 AM8/15/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed

Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed


Comment:

In [changeset:"f3f9d03edf17ccfa17263c7efa0b1350d1ac9278" f3f9d03]:
{{{
#!CommitTicketReference repository=""
revision="f3f9d03edf17ccfa17263c7efa0b1350d1ac9278"
Fixed #33901 -- Skipped varchar_pattern_ops/text_pattern_ops index
creation when db_collation is set.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:6>

Django

unread,
Oct 18, 2022, 9:42:24 PM10/18/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ismael ABBO):

Hello everyone!
First thing first, I am a newbie, using 3.2 LTS (latest 3.2.16) version
and Postgres 14.4!
While this has been closed, I am still facing the same issue unless it is
just a patch for Django 4.2?

Please help on how to fix it!

Here is the error:
{{{
...
Applying auth.0012_alter_user_first_name_max_length... OK
Applying person.0001_initial...Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.FeatureNotSupported: nondeterministic collations are not


supported for operator class "varchar_pattern_ops"


The above exception was the direct cause of the following exception:
...
File "/usr/local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)


django.db.utils.NotSupportedError: nondeterministic collations are not
supported for operator class "varchar_pattern_ops"
}}}

There is FeatureNotSupported error on psycopg2, I noticed but, if it is
the main reason, how was it managed before? Any workaround?

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:7>

Django

unread,
Oct 19, 2022, 6:08:11 AM10/19/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Tim Graham):

Yes, the fix will only be applied for Django 4.2. Since the bug isn't a
regression from a previous release or a major bug in a new feature it
won't be backported. See our
[https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions supported versions policy].

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:8>

Django

unread,
Oct 19, 2022, 10:42:40 AM10/19/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ismael ABBO):

Replying to [comment:8 Tim Graham]: Thanks for that information!
Then, is there any workaround for 3.2.x versions?


> Yes, the fix will only be applied for Django 4.2. Since the bug isn't a
regression from a previous release or a major bug in a new feature it
won't be backported. See our
[https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions supported versions policy].

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:9>

Django

unread,
Oct 19, 2022, 9:31:32 PM10/19/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Ismael, the only way to work around the issue on Django 3.2 would to be
subclass the postgres `SchemaEditor` to override its
`_create_like_index_sql` method and then use the adjusted backend in your
`DATABASES` entries.

You can refer to the [https://docs.djangoproject.com/en/4.1/ref/databases
/#subclassing-the-built-in-database-backends documentation on the
subject].

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:10>

Django

unread,
Oct 20, 2022, 10:02:47 AM10/20/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ed Chai):

Replying to [comment:9 Ismael ABBO]:


> Replying to [comment:8 Tim Graham]: Thanks for that information!
> Then, is there any workaround for 3.2.x versions?
> > Yes, the fix will only be applied for Django 4.2. Since the bug isn't
a regression from a previous release or a major bug in a new feature it
won't be backported. See our
[https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions supported versions policy].

You can keep using `CITEXT`, or set unique constraints directly via psql
terminal.

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:11>

Django

unread,
Oct 20, 2022, 10:22:21 AM10/20/22
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ismael ABBO):

Thanks for help !

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:12>

Django

unread,
Apr 20, 2023, 5:10:45 AM4/20/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Petter Friberg):

Not sure if there should be a new ticket here or not but I'm still seeing
this on 4.2 when having a relation that points to a unique field with a
non-deterministic collation.

For instance, consider if we run the following migration operation:

{{{
#!python
migrations.CreateModel(
name="ExtendedUser",


fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True,
serialize=False, verbose_name='ID')),

('user',
models.OneToOneField(on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL, to_field='username'))
]
)
}}}

Running `manage.py sqlmigrate` for that operation displays the generated
`_like` index

{{{
#!sql
CREATE INDEX "myapp_extendeduser_user_id_<chars>_like" ON
"myapp_extendeduser" ("user_id" varchar_pattern_ops);
}}}

(Not sure if there's more to it but) I think the
[https://code.djangoproject.com/changeset/f3f9d03edf17ccfa17263c7efa0b1350d1ac9278
provided patch] could be extended with something like below to resolve at
least `OneToOneField`

{{{
#!python
# Non-deterministic collations on Postgresql don't support
indexes
# for operator classes varchar_pattern_ops/text_pattern_ops.
if getattr(field, "db_collation", None) or (field.is_relation
and getattr(field.target_field, "db_collation", None)):
return None
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:13>

Django

unread,
Apr 20, 2023, 5:33:45 AM4/20/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

Petter, thanks for the report. If you believe it's an issue in Django,
then please create a [https://code.djangoproject.com/newticket new ticket
in Trac] and follow our
[https://docs.djangoproject.com/en/dev/internals/contributing/bugs-and-
features/#reporting-bugs bug reporting guidelines].

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:14>

Django

unread,
Apr 20, 2023, 1:12:51 PM4/20/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ed Chai):

Replying to [comment:13 Petter Friberg]:


> Not sure if there should be a new ticket here or not but I'm still
seeing this on 4.2 when having a relation that points to a unique field
with a non-deterministic collation.
>
> For instance, consider if we run the following migration operation:
>
> {{{
> #!python
> migrations.CreateModel(
> name="ExtendedUser",
> fields=[
> ('id', models.BigAutoField(auto_created=True, primary_key=True,
serialize=False, verbose_name='ID')),
> ('user',
models.OneToOneField(on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL, to_field='username'))
> ]
> )
> }}}
>
> Running `manage.py sqlmigrate` for that operation displays the generated
`_like` index
>
> {{{
> #!sql

> BEGIN;
> --
> -- Create model ExtendedUser
> --
> CREATE TABLE "myapp_extendeduser" ("id" bigint NOT NULL PRIMARY KEY
GENERATED BY DEFAULT AS IDENTITY, "user_id" varchar(30) COLLATE "ci" NOT
NULL UNIQUE);
> ALTER TABLE "myapp_extendeduser" ADD CONSTRAINT
"myapp_extendeduser_user_id_<chars>_fk_myapp_user_username" FOREIGN KEY
("user_id") REFERENCES "myapp_user" ("username") DEFERRABLE INITIALLY
DEFERRED;


> CREATE INDEX "myapp_extendeduser_user_id_<chars>_like" ON
"myapp_extendeduser" ("user_id" varchar_pattern_ops);

> COMMIT;


> }}}
>
> (Not sure if there's more to it but) I think the
[https://code.djangoproject.com/changeset/f3f9d03edf17ccfa17263c7efa0b1350d1ac9278
provided patch] could be extended with something like below to resolve at
least `OneToOneField`
>
> {{{
> #!python
> # Non-deterministic collations on Postgresql don't support
indexes
> # for operator classes varchar_pattern_ops/text_pattern_ops.
> if getattr(field, "db_collation", None) or
(field.is_relation and getattr(field.target_field, "db_collation", None)):
> return None
> }}}

I think your code is not related to this issue. This ticket only
addresses errors that occur when both db_collation and unique are applied
on Postgres. `CREATE INDEX` is the expected operation when you create
foreign keys

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:15>

Django

unread,
Jul 8, 2023, 10:41:47 AM7/8/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by alfonsrv):

Despite this ticket being closed, I still get an error when attempting to
run the SQL migration. Running Django 4.2.3 and Postgres 15.x

I created the collation required collation and attempt to run a migration
like this

{{{
#!python
operations = [
CreateCollation(
"case_insensitive",


provider="icu",
locale="und-u-ks-level2",

deterministic=False,
),
]
}}}

{{{
#!python
migrations.AlterField(
model_name='user',
name='email',
field=models.EmailField(db_collation='case_insensitive',
max_length=254, unique=True, verbose_name='email address'),
),
}}}

This results in the following SQL:

{{{
!#sql
BEGIN;
--
-- Alter field email on user
--
ALTER TABLE "core_user" ALTER COLUMN "email" TYPE varchar(254) COLLATE
"case_insensitive";
COMMIT;
}}}

And the same error when attempting to run `python manage.py migrate`:
`django.db.utils.NotSupportedError: nondeterministic collations are not
supported for operator class "varchar_pattern_ops"`

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:16>

Django

unread,
Jul 9, 2023, 4:17:26 AM7/9/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Ed Chai):

Replying to [comment:16 alfonsrv]:
The issue is likely caused by a "varchar_pattern_ops" that already existed
on your Postgres DB, you can run `select * from pg_indexes where
tablename='tablename';` in psql terminal to check if it was created. This
fix indeed only solves the problem when you first time creating model
fields with collation and prevents pattern ops index created. I'd be happy
to submit another PR for the migration.

> Despite this ticket being closed, I still get an error when attempting

to run the SQL migration. Running Django 4.2.3 and Postgres 15.x (also
tried 13.x)


>
> I created the collation required collation and attempt to run a
migration like this
>
> {{{
> #!python
> operations = [
> CreateCollation(
> "case_insensitive",
> provider="icu",
> locale="und-u-ks-level2",
> deterministic=False,
> ),
> ]
> }}}
>
> {{{
> #!python
> migrations.AlterField(
> model_name='user',
> name='email',
> field=models.EmailField(db_collation='case_insensitive',
max_length=254, unique=True, verbose_name='email address'),
> ),
> }}}
>
> This results in the following SQL:
>
> {{{

> #!sql


> BEGIN;
> --
> -- Alter field email on user
> --
> ALTER TABLE "core_user" ALTER COLUMN "email" TYPE varchar(254) COLLATE
"case_insensitive";
> COMMIT;
> }}}
>
> And the same error when attempting to run `python manage.py migrate`:
`django.db.utils.NotSupportedError: nondeterministic collations are not

supported for operator class "varchar_pattern_ops"`. I found that altering
the existing field doesn't work, but instead it has to be recreated
completely.

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:17>

Django

unread,
Oct 10, 2023, 3:02:51 PM10/10/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: new

Component: Migrations | Version: dev
Severity: Normal | Resolution:
Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by kenneweg):

* status: closed => new
* resolution: fixed =>


Comment:

the issues can be recreated as follows:

I have an old migration that sets the field with unique=true

I have a new migration that sets the field with unique=true,
db_collation="utf8_unicode_ci"

The old migration runs first, and creates a LIKE index for the field. The
new migration attempts to run:
{{{


django.db.utils.NotSupportedError: nondeterministic collations are not
supported for operator class "varchar_pattern_ops"
}}}


My work around is writing a DROP INDEX of the LIKE index that is created
by the first migration

{{{
migrations.RunSQL(
sql="DROP INDEX IF EXISTS
appname_modelname_fieldname_073c92ac_like;",
reverse_sql=migrations.RunSQL.noop,
),
migrations.AlterField(
model_name="modelname",
name="fieldname",
field=models.CharField(
blank=True, db_collation="utf8_unicode_ci",
max_length=255, null=True, unique=True
),
),
}}}

But this is quite messy.

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:18>

Django

unread,
Oct 10, 2023, 11:54:47 PM10/10/23
to django-...@googlegroups.com
#33901: non-deterministic collations doesn't work with Unique=True on Postgres13.3
-------------------------------------+-------------------------------------
Reporter: Ed Chai | Owner: Ed Chai
Type: Bug | Status: closed
Component: Migrations | Version: dev
Severity: Normal | Resolution: fixed

Keywords: collation unique | Triage Stage: Ready for
citext | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* status: new => closed
* resolution: => fixed


Comment:

Replying to [comment:18 Adam]:


> the issues can be recreated as follows:
>
> I have an old migration that sets the field with unique=true
>
> I have a new migration that sets the field with unique=true,
db_collation="utf8_unicode_ci"
>
> The old migration runs first, and creates a LIKE index for the field.
The new migration attempts to run:
> {{{
> django.db.utils.NotSupportedError: nondeterministic collations are not
supported for operator class "varchar_pattern_ops"
> }}}
>
>
> My work around is writing a DROP INDEX of the LIKE index that is created
by the first migration
>
> {{{
> migrations.RunSQL(
> sql="DROP INDEX IF EXISTS
appname_modelname_fieldname_073c92ac_like;",
> reverse_sql=migrations.RunSQL.noop,
> ),
> migrations.AlterField(
> model_name="modelname",
> name="fieldname",
> field=models.CharField(
> blank=True, db_collation="utf8_unicode_ci",
max_length=255, null=True, unique=True
> ),
> ),
> }}}
>
> But this is quite messy.

Thanks for the report. In the future, please open new tickets for linked
bug reports and don't reopen already closed. I've created #34898.

--
Ticket URL: <https://code.djangoproject.com/ticket/33901#comment:19>

Reply all
Reply to author
Forward
0 new messages