[Django] #36199: Redundant migrations generated with Coalesce Index in GinIndex with SearchVector
8 views
Skip to first unread message
Django
Feb 18, 2025, 4:48:41 PMFeb 18
to django-...@googlegroups.com
#36199: Redundant migrations generated with Coalesce Index in GinIndex with
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 5.1 | Severity: Normal
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Title: Redundant Migration Generation for Coalesce Index with TextField
Output in Django 5.1 (PostgreSQL)
Django Version: 5.1
Database Backend: PostgreSQL
'''Python Version:''' 3.12
Operating System: Observed on MacOS Sequoia, but likely to generalize
'''Description:'''
When running makemigrations on a Django 5.1 project using a PostgreSQL
backend, the migration autodetector redundantly generates a migration that
removes and then recreates a Coalesce index where the output_field is set
to TextField(). This occurs because the autodetector incorrectly registers
the TextField in the existing and new application state as distinct,
leading to continuous migration creation even when no schema changes have
occurred.
'''Steps to Reproduce:'''
Create a Django model with an index using Coalesce and explicitly set
output_field=TextField(), such as in the example below:
{{{
#!python
from django.contrib.postgres.indexes import GinIndex
from django.contrib.postgres.search import SearchVector
from django.db.models import Value
from django.db.models.functions import Coalesce
# ... model definition, etc.
class SummaryItem(models.Model):
class Meta:
indexes = [
GinIndex(
SearchVector(
Coalesce(
'item_text_by_user',
'item_text',
Value(''),
output_field=TextField()
),
'source',
config='english',
),
name='summary_item_search_index',
)
]
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
deleted_at = models.DateTimeField(null=True)
item_type = models.CharField(max_length=16, choices=ItemType)
item_text = models.TextField()
item_text_by_user = models.TextField(null=True)
}}}
Run makemigrations and apply the migration.
Run makemigrations again without making any changes.
Observe that a new migration is generated that removes and recreates the
same Coalesce index.
'''Expected Behavior:'''
Once the initial migration has been applied, subsequent runs of
makemigrations should not generate redundant migrations unless there is an
actual schema change.
'''Actual Behavior:'''
Each execution of makemigrations generates an unnecessary migration
removing and recreating the Coalesce index.
'''Potential Cause:'''
The migration autodetector may be treating the TextField instances in the
existing and new application state as distinct, even though they are
equivalent. This causes Django to incorrectly assume a change has occurred
and generate a new migration. It was observed while breaking down the
deconstructed indices from the autodetector's from_state and to_state (see
here:
https://github.com/django/django/blob/0bf412111be686b6b23e00863f5d449d63557dbf/django/db/migrations/autodetector.py#L1347)
that the deconstructed new and old indexes were identified as different
because the `creation_counter` attributes of the TextField() passed to
`output_field` in the indexes (which in turn occurred in the ' in the two
respective states were different, which is part of how the `__eq__`
operator is defined for objects inheriting from `Field`, and therefore the
index in the old and new states were identified as separate, even though
there was no change in the definition of the index.
'''Workaround:'''
As a temporary workaround, removing output_field=TextField() from Coalesce
may prevent redundant migrations if Django can infer the correct type
automatically. However, this might not be suitable in all cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/36199>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 5.1 | Severity: Normal
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Title: Redundant Migration Generation for Coalesce Index with TextField
Output in Django 5.1 (PostgreSQL)
Django Version: 5.1
Database Backend: PostgreSQL
'''Python Version:''' 3.12
Operating System: Observed on MacOS Sequoia, but likely to generalize
'''Description:'''
When running makemigrations on a Django 5.1 project using a PostgreSQL
backend, the migration autodetector redundantly generates a migration that
removes and then recreates a Coalesce index where the output_field is set
to TextField(). This occurs because the autodetector incorrectly registers
the TextField in the existing and new application state as distinct,
leading to continuous migration creation even when no schema changes have
occurred.
'''Steps to Reproduce:'''
Create a Django model with an index using Coalesce and explicitly set
output_field=TextField(), such as in the example below:
{{{
#!python
from django.contrib.postgres.indexes import GinIndex
from django.contrib.postgres.search import SearchVector
from django.db.models import Value
from django.db.models.functions import Coalesce
# ... model definition, etc.
class SummaryItem(models.Model):
class Meta:
indexes = [
GinIndex(
SearchVector(
Coalesce(
'item_text_by_user',
'item_text',
Value(''),
output_field=TextField()
),
'source',
config='english',
),
name='summary_item_search_index',
)
]
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
deleted_at = models.DateTimeField(null=True)
item_type = models.CharField(max_length=16, choices=ItemType)
item_text = models.TextField()
item_text_by_user = models.TextField(null=True)
}}}
Run makemigrations and apply the migration.
Run makemigrations again without making any changes.
Observe that a new migration is generated that removes and recreates the
same Coalesce index.
'''Expected Behavior:'''
Once the initial migration has been applied, subsequent runs of
makemigrations should not generate redundant migrations unless there is an
actual schema change.
'''Actual Behavior:'''
Each execution of makemigrations generates an unnecessary migration
removing and recreating the Coalesce index.
'''Potential Cause:'''
The migration autodetector may be treating the TextField instances in the
existing and new application state as distinct, even though they are
equivalent. This causes Django to incorrectly assume a change has occurred
and generate a new migration. It was observed while breaking down the
deconstructed indices from the autodetector's from_state and to_state (see
here:
https://github.com/django/django/blob/0bf412111be686b6b23e00863f5d449d63557dbf/django/db/migrations/autodetector.py#L1347)
that the deconstructed new and old indexes were identified as different
because the `creation_counter` attributes of the TextField() passed to
`output_field` in the indexes (which in turn occurred in the ' in the two
respective states were different, which is part of how the `__eq__`
operator is defined for objects inheriting from `Field`, and therefore the
index in the old and new states were identified as separate, even though
there was no change in the definition of the index.
'''Workaround:'''
As a temporary workaround, removing output_field=TextField() from Coalesce
may prevent redundant migrations if Django can infer the correct type
automatically. However, this might not be suitable in all cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/36199>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Feb 18, 2025, 5:00:35 PMFeb 18
to django-...@googlegroups.com
#36199: Redundant migrations generated with Coalesce Index in GinIndex with
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Owner: (none)
SearchVector
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Ian Bicket:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
Value('') (which defaults to a CharField rendering the output_field
necessary) parameter from Coalesce will prevent redundant migrations if
Django can infer the correct type automatically. However, this might not
be suitable in all cases.
--
--
be suitable in all cases.
--
Ticket URL: <https://code.djangoproject.com/ticket/36199#comment:1>
Django
Feb 18, 2025, 5:36:30 PMFeb 18
to django-...@googlegroups.com
> As a temporary workaround, removing output_field=TextField() and the
> Value('') (which defaults to a CharField rendering the output_field
> necessary) parameter from Coalesce will prevent redundant migrations if
> Django can infer the correct type automatically. However, this might not
> be suitable in all cases.
> Value('') (which defaults to a CharField rendering the output_field
> necessary) parameter from Coalesce will prevent redundant migrations if
> Django can infer the correct type automatically. However, this might not
> be suitable in all cases.
New description:
Title: Redundant Migration Generation for Coalesce Index with TextField
Output in Django 5.1 (PostgreSQL)
'''Django Version:''' 5.1
'''Database Backend:''' PostgreSQL (version 15)
Title: Redundant Migration Generation for Coalesce Index with TextField
Output in Django 5.1 (PostgreSQL)
'''Django Version:''' 5.1
Django
Feb 18, 2025, 5:38:39 PMFeb 18
to django-...@googlegroups.com
#36199: Redundant migrations generated with Coalesce Index in GinIndex with
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Ian Bicket:
Old description:
> Title: Redundant Migration Generation for Coalesce Index with TextField
> Output in Django 5.1 (PostgreSQL)
>
> '''Django Version:''' 5.1
>
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Ian Bicket:
Old description:
> Title: Redundant Migration Generation for Coalesce Index with TextField
> Output in Django 5.1 (PostgreSQL)
>
> '''Django Version:''' 5.1
>
field where the output_field parameter is set to TextField(). Furthermore,
makemigrations can be run an arbitrary number of times, each execution re-
creating what is essentially the same migration, removing and recreating
the index.
Django
Feb 18, 2025, 5:39:03 PMFeb 18
to django-...@googlegroups.com
> that removes and then recreates a GinIndex with a SearchVector over a
> Coalesced field where the output_field parameter is set to TextField().
> Furthermore, makemigrations can be run an arbitrary number of times, each
> execution re-creating what is essentially the same migration, removing
> Coalesced field where the output_field parameter is set to TextField().
> Furthermore, makemigrations can be run an arbitrary number of times, each
Django
Feb 18, 2025, 6:58:58 PMFeb 18
to django-...@googlegroups.com
#36199: Redundant migrations generated with Coalesce Index in GinIndex with
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Owner: (none)
Type: Bug | Status: closed
SearchVector
-------------------------------------+-------------------------------------
Reporter: Ian Bicket | Owner: (none)
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution: duplicate
(models, ORM) |
Keywords: migrations, search, | Triage Stage:
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):
coalesce, index | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: => duplicate
* status: new => closed
Comment:
Duplicate of #36173 (fixed by df2c4952df6d93c575fb8a3c853dc9d4c2449f36)
which will be part of Django 5.2b1 meant to be released tomorrow.
--
Ticket URL: <https://code.djangoproject.com/ticket/36199#comment:5>
Reply all
Reply to author
Forward
0 new messages