[Django] #36198: Unique Constraints defined with __date lookups on F expressions fail validation but create database constraints sucessfully

22 views
Skip to first unread message

Django

unread,
Feb 18, 2025, 5:15:12 AMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: 5.1 | Severity: Normal
Keywords: constraint lookup | Triage Stage:
unique validation | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Unique constraints defined with date lookups on F expressions create the
correct index and work as expected at the database/ORM level but raise
false positives during validation.

Initially discovered this for date lookups but it would seem to apply to
the other date/time lookups, at this point I have explicitly confirmed the
same behaviour for year, iso_year and minute.

Minimal reproducable example:

{{{#!python
# models.py
from django.db import models

class DateLookup(models.Model):
timestamp = models.DateTimeField()

class Meta:
constraints = [
# triggering validation errors for objects that can actually
be saved
models.UniqueConstraint(
models.F("timestamp__date"),
name="%(app_label)s_%(class)s_unique_date_F_expr",
),
]

# tests.py
import datetime as dt

from django.core.exceptions import ValidationError
from django.db import IntegrityError
from django.utils import timezone
from django.test import TestCase

from .models import DateLookup

TIMESTAMP = timezone.make_aware(dt.datetime(2025, 2, 18, 8, 18, 6))

class TestDateLookupConstraint(TestCase):
def test_demonstrate_constraint_fails(self):
"""Demonstrate constraint validation failure at python level when
db constraint is not triggered"""
DateLookup.objects.create(timestamp=TIMESTAMP)
new = DateLookup(timestamp=TIMESTAMP - dt.timedelta(days=1))
new.save() #success
self.assertTrue(new.pk)
try:
new.full_clean() #ValidationError
except ValidationError:
self.fail("False Positive")

def test_constraint_db(self):
"""Demonstrate DB constraint is triggered sucessfully."""
DateLookup.objects.create(timestamp=TIMESTAMP)
new = DateLookup(timestamp=TIMESTAMP)
with self.assertRaises(ValidationError):
new.full_clean()
with self.assertRaises(IntegrityError):
new.save()
self.assertFalse(new.pk)
}}}

Output of sqlmigrate command:

{{{#!sql
BEGIN;
--
-- Create model DateLookup
--
CREATE TABLE "minimal_example_datelookup" ("id" integer NOT NULL PRIMARY
KEY AUTOINCREMENT, "timestamp" datetime NOT NULL);
CREATE UNIQUE INDEX "minimal_example_datelookup_unique_date_F_expr" ON
"minimal_example_datelookup" ((django_datetime_cast_date("timestamp",
'UTC', 'UTC')));
COMMIT;
}}}

The workaround is to explicitly use the corresponding database functions
in the constraint definition; this version works correctly.

{{{#!python
class TruncDateExpression(models.Model):
timestamp = models.DateTimeField()

class Meta:
constraints = [
models.UniqueConstraint(
models.functions.TruncDate("timestamp"),
name="%(app_label)s_%(class)s_unique_date",
),
]
}}}

The
[https://docs.djangoproject.com/en/5.1/ref/models/constraints/#expressions
docs for constraint expressions] don't show any examples of lookups being
used in this way but also don't explicitly disallow them either, and given
that the correct constraint is in fact being generated on the database, it
implies this ''could'' be fixed to work as expected. Otherwise, at a
minimum the documentation should be updated to forbid the use of F
expression lookups, and the constraint should fail to apply during
migration.


Initially discovered this and investigated on the
[https://forum.djangoproject.com/t/unique-constraint-on-datetime-date-
unexpectedly-fails-validation-but-database-allows-saving/38889 django
forum], see there for additional context and my efforts at debugging the
validation step.
--
Ticket URL: <https://code.djangoproject.com/ticket/36198>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Feb 18, 2025, 5:18:41 AMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage:
unique validation | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Tom Hall:

Old description:
New description:
Output of sqlmigrate command for SQLite backend:
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:1>

Django

unread,
Feb 18, 2025, 8:28:31 AMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):

* stage: Unreviewed => Accepted

Comment:

Thank you for the ticket!

I can replicate and I agree that if this is not allowed it is not clear in
the docs
Here is a possible regression test
{{{#!diff
--- a/tests/constraints/models.py
+++ b/tests/constraints/models.py
@@ -73,6 +73,7 @@ class UniqueConstraintProduct(models.Model):
name = models.CharField(max_length=255)
color = models.CharField(max_length=32, null=True)
age = models.IntegerField(null=True)
+ updated = models.DateTimeField(null=True)

class Meta:
constraints = [
diff --git a/tests/constraints/tests.py b/tests/constraints/tests.py
index 51f09f2937..a42cf9b369 100644
--- a/tests/constraints/tests.py
+++ b/tests/constraints/tests.py
@@ -1,3 +1,4 @@
+from datetime import datetime, timedelta
from unittest import mock

from django.core.exceptions import ValidationError
@@ -1030,6 +1031,21 @@ class UniqueConstraintTests(TestCase):
exclude={"name"},
)

+ def test_validate_expression_date(self):
+ updated_date = datetime(2005, 7, 26)
+ UniqueConstraintProduct.objects.create(name="p1",
updated=updated_date)
+ constraint = models.UniqueConstraint(models.F("updated__date"),
name="date_created_unique")
+ msg = "Constraint “date_created_unique” is violated."
+ with self.assertRaisesMessage(ValidationError, msg):
+ constraint.validate(
+ UniqueConstraintProduct,
+ UniqueConstraintProduct(updated=updated_date),
+ )
+ constraint.validate(
+ UniqueConstraintProduct,
+ UniqueConstraintProduct(updated=updated_date +
timedelta(days=1)),
+ )
+
def test_validate_ordered_expression(self):
constraint = models.UniqueConstraint(
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:2>

Django

unread,
Feb 18, 2025, 10:43:25 AMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):

To me this is a duplicate or at least closely related to #34871 and is due
to the fact that `models.F("timestamp__date")` is not turned into
something alike `UnresolvedTransform(F("timestamp"), "date")` which would
allow proper replacement of `F("timestamp")` by `Value(TIMESTAMP -
dt.timedelta(days=1), DateLookup._meta.get_field("timestamp"))`.

The could be ways to resolve this particular issue by adapting
`F.replace_expressions` when `LOOKUP_SEP in self.name` and the replacement
(a `Value` instance in the case of constraint validation) has an
`output_field` though which might be a less invasive solution.
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:3>

Django

unread,
Feb 18, 2025, 12:50:16 PMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):

Submitted [https://github.com/django/django/pull/19190 a PR] demonstrating
how this can be implemented for `F.resolve_expression` without going fully
unboard with the `Unresolved(Lookup|Transform)` approach that might be
necessary to fully resolve #34871.
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:4>

Django

unread,
Feb 18, 2025, 12:50:21 PMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:5>

Django

unread,
Feb 18, 2025, 2:48:26 PMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* owner: (none) => Simon Charette
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:6>

Django

unread,
Feb 18, 2025, 5:45:01 PMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tom Hall):

Replying to [comment:3 Simon Charette]:
> To me this is a duplicate or at least closely related to #34871 and is
due to the fact that `models.F("timestamp__date")` is not turned into
something alike `UnresolvedTransform(F("timestamp"), "date")` which would
allow proper replacement of `F("timestamp")` by `Value(TIMESTAMP -
dt.timedelta(days=1), DateLookup._meta.get_field("timestamp"))`.

I did spot that one and meant to link to it in the description as maybe
related but wasn't familiar ~~enough~~ at all with the underlying
mechanics to be confident. And then I forgot. First time here and getting
used to the system. Nice work on fixing it up, I suspected something in
`F.replace_expressions` was needed.

Can try writing something up for the docs to demonstrate this being used
if that would be helpful?
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:7>

Django

unread,
Feb 18, 2025, 6:37:05 PMFeb 18
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Accepted
unique validation |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):

> Can try writing something up for the docs to demonstrate this being used
if that would be helpful?

Thanks for the offer but we don't normally document bug fixing so in this
case I don't think any documentation update is necessary.

I'll adjust the PR to include Sarah's test as well.
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:8>

Django

unread,
Mar 1, 2025, 2:00:05 PMMar 1
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: assigned
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: constraint lookup | Triage Stage: Ready for
unique validation | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:9>

Django

unread,
Mar 1, 2025, 3:01:22 PMMar 1
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: closed
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: constraint lookup | Triage Stage: Ready for
unique validation | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"fc303551077c3e023fe4f9d01fc1b3026c816fa4" fc30355]:
{{{#!CommitTicketReference repository=""
revision="fc303551077c3e023fe4f9d01fc1b3026c816fa4"
Fixed #36198 -- Implemented unresolved transform expression replacement.

This allows the proper resolving of F("field__transform") when
performing constraint validation.

Thanks Tom Hall for the report and Sarah for the test.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:10>

Django

unread,
Aug 4, 2025, 3:44:31 AMAug 4
to django-...@googlegroups.com
#36198: Unique Constraints defined with __date lookups on F expressions fail
validation but create database constraints sucessfully
-------------------------------------+-------------------------------------
Reporter: Tom Hall | Owner: Simon
| Charette
Type: Bug | Status: closed
Component: Database layer | Version: 5.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: constraint lookup | Triage Stage: Ready for
unique validation | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):

In [changeset:"e5ccb69bc3da407ab2b0477c0cc5db27c7207225" e5ccb69b]:
{{{#!CommitTicketReference repository=""
revision="e5ccb69bc3da407ab2b0477c0cc5db27c7207225"
[5.2.x] Fixed #36198 -- Implemented unresolved transform expression
replacement.

This allows the proper resolving of F("field__transform") when
performing constraint validation.

Thanks Tom Hall for the report and Sarah for the test.

Prerequisite for #36518.

Backport of fc303551077c3e023fe4f9d01fc1b3026c816fa4 from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36198#comment:11>
Reply all
Reply to author
Forward
0 new messages