[Django] #35223: Fields with db_default raise ValidationErrors when full_clean() called

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
quinceleaf |
Type: Bug | Status: new
Component: Database | Version: 5.0
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Starting to migrate models in my (large) project to use Django 5’s new
db_default attribute for fields (using Django 5.0.2), encountering
behavior contrary to my expectations.

A field with {{{db_default}}} raises a {{{ValidationError}}} when
{{{full_clean()}}} called, if that field has been omitted from the
{{{create()}}} call.

This is (to me) unexpected behavior. Would expect that no error would be
raised, the instance would be saved successfully, with the specified
{{{db_default}}} value set at the database level.

Has been explained to me in the Django forums that this is correct, that I
should instead either

(1) explicitly choose to {{{exclude}}} the missing fields from
{{full_clean()}}} call,
(2) write a custom clean method for each field, or
(3) simply save the instance rather than calling {{{full_clean()}}}

Had always been impressed on me that it is best practice to always call
{{{full_clean()}}} on instance creation and/or update.

In either case, having to determine the missing fields and annotate the
{{{full_clean()}}} call or write a custom clean method per field seem to
work against the usual conception of a default value, which should
propagate... well, ''by default'' and allow for simpler operation where
possible.

I would suggest having fields with {{{db_default}}} be excluded by default
from {{{full_clean()}}}

If the current behavior is re-affirmed, would suggest incorporating the
suggested 3 workaround steps into the Django documentation, as I suspect
most users would have similar expectations as myself when implementing
this in future code.
--
Ticket URL: <https://code.djangoproject.com/ticket/35223>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------

Reporter: Brian Ibbotson | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Brian Ibbotson:

Old description:

New description:

--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:1>

[Django]

Has been explained to me in the Django forums that the
{{{ValidationError}}} is correct, that I should instead either

(1) explicitly choose to {{{exclude}}} the missing fields from
{{{full_clean()}}} call,
(2) write a custom clean method for each field, or
(3) simply save the instance rather than calling {{{full_clean()}}}

Had always been impressed on me that it is best practice to always call
{{{full_clean()}}} on instance creation and/or update.

In either case, having to determine the missing fields and annotate the
{{{full_clean()}}} call or write a custom clean method per field seem to
work against the usual conception of a default value, which should
propagate... well, ''by default'' and allow for simpler operation where
possible.

I would suggest having fields with {{{db_default}}} be excluded by default
from {{{full_clean()}}}

If the current behavior is re-affirmed, would suggest incorporating the
suggested 3 workaround steps into the Django documentation, as I suspect
most users would have similar expectations as myself when implementing
this in future code.

--
--

Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:2>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

> I would suggest having fields with db_default be excluded by default
from full_clean()

`db_default` is just a default value, we cannot always exclude it from
`full_clean()`. What about not valid values provided explicitly in
`create()`? What about `None` when field is non-nullable? etc.
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:3>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: nobody
Type: Bug | Status: new
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

I believe that we should find a way to have `db_default` behave the same
way as generated fields when no value has been assigned to the instance
when `full_clean` is called (#35127). Requiring users to explicitly pass
it to `exclude` is simply bad ergonomics IMO.

One thing that the solution should take into account is that the value
should only be skipped when the value is not assigned. In other words

{{{#!python
class Article(models.Model):
upvotes = models.PositiveIntegerField(db_default=0)

a1 = Article(upvotes=-1)
a1.full_clean() # should raise a ValidationError

a2 = Article(upvotes=42)
a2.full_clean() # should *not* raise a ValidationError

a1 = Article()
a1.full_clean() # should not raise a ValidationError
}}}

In other words, when a value is present it should be validated otherwise
if `db_default` is provided it should be assumed to be valid just like
it's the case with generated fields.

A workaround in the mean time is to have both `default` and `db_default`
assigned but obviously that kind of defeats the purpose of using
`db_default` in the first place.
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:4>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* stage: Unreviewed => Accepted
* type: Bug => Cleanup/optimization

Comment:

Replying to [comment:4 Simon Charette]:

> I believe that we should find a way to have `db_default` behave the same
way as generated fields when no value has been assigned to the instance
when `full_clean` is called (#35127). Requiring users to explicitly pass
it to `exclude` is simply bad ergonomics IMO.

Agreed, I think we should treat this as a cleanup not bug.
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:5>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------

Reporter: Brian Ibbotson | Owner: Damir
Type: | Nafikov
Cleanup/optimization | Status: assigned

Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Damir Nafikov):

* owner: nobody => Damir Nafikov
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:6>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: Damir
Type: | Nafikov
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by bcail):

* cc: bcail (added)

Comment:

@Damir Nafikov, are you still working on this ticket?
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:7>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: Damir
Type: | Nafikov
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by bcail):

* has_patch: 0 => 1

Comment:

I opened [https://github.com/django/django/pull/17939 a PR].
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:8>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: Damir
Type: | Nafikov
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 5.0
(models, ORM) |

Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* severity: Normal => Release blocker

Comment:

Bumping to a release blocker as `Model.full_clean()` crashes on some
expressions, e.g.
{{{
File "/django/tests/field_defaults/tests.py", line 175, in
test_full_clean
obj.full_clean()
File "/django/django/db/models/base.py", line 1586, in full_clean
self.clean_fields(exclude=exclude)
File "/django/django/db/models/base.py", line 1641, in clean_fields
setattr(self, f.attname, f.clean(raw_value, self))
File "/django/django/db/models/fields/__init__.py", line 830, in clean
value = self.to_python(value)
File "/django/django/db/models/fields/__init__.py", line 1620, in
to_python
parsed = parse_datetime(value)
File "/django/django/utils/dateparse.py", line 114, in parse_datetime
return datetime.datetime.fromisoformat(value)
TypeError: fromisoformat: argument must be str
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:9>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: Damir
Type: | Nafikov
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak):

Bug in 414704e88d73dafbcfbb85f9bc54cb6111439d3.
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:10>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------

Reporter: Brian Ibbotson | Owner: bcail
Type: | Status: assigned
Cleanup/optimization |

Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Release blocker | Resolution:

Keywords: | Triage Stage: Ready for
| checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak):

* owner: Damir Nafikov => bcail
* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:11>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: bcail

Type: | Status: closed

Cleanup/optimization |
Component: Database layer | Version: 5.0
(models, ORM) |

Severity: Release blocker | Resolution: fixed

Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"1570ef02f34037d32218d463342592debccf915c" 1570ef0]:
{{{#!CommitTicketReference repository=""
revision="1570ef02f34037d32218d463342592debccf915c"
Fixed #35223 -- Made Model.full_clean() ignore fields with db_default when
validating empty values.

Thanks Brian Ibbotson for the report.

Regression in 7414704e88d73dafbcfbb85f9bc54cb6111439d3.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:12>

[Django]

#35223: Fields with db_default raise ValidationErrors when full_clean() called
-------------------------------------+-------------------------------------
Reporter: Brian Ibbotson | Owner: bcail
Type: | Status: closed
Cleanup/optimization |
Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"5f07460a67825bfc3f0129feec94a24bbf6d2a5f" 5f07460a]:
{{{#!CommitTicketReference repository=""
revision="5f07460a67825bfc3f0129feec94a24bbf6d2a5f"
[5.0.x] Fixed #35223 -- Made Model.full_clean() ignore fields with

db_default when validating empty values.

Thanks Brian Ibbotson for the report.

Regression in 7414704e88d73dafbcfbb85f9bc54cb6111439d3.

Backport of 1570ef02f34037d32218d463342592debccf915c from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35223#comment:13>