[Django] #36484: Add optional error-on-integer-overflow setting for QuerySet filters

[Django]

#36484: Add optional error-on-integer-overflow setting for QuerySet filters
-------------------------------------+-------------------------------------
Reporter: Yosuke Takeuchi | Type: New
| feature
Status: new | Component: Database
| layer (models, ORM)
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
== Overview
In Django 5.0 the behaviour for arithmetic that overflows an integer
column changed: a filter that would overflow now always returns an empty
QuerySet instead of being transparently cast to a wider integer type (the
historical behaviour on most back-ends). While this keeps runtime
semantics simple, it makes migrations from Django 4.x harder to
verify—tests may “pass” yet silently return fewer (or zero) rows. It is
also difficult for developers on 5.x to notice when an overflow is
happening because the condition is swallowed inside the ORM.
(see:
https://docs.djangoproject.com/en/5.2/releases/5.0/#:~:text=Filtering%20querysets%20against%20overflowing%20integer%20values%20now%20always%20returns%20an%20empty%20queryset.%20As%20a%20consequence%2C%20you%20may%20need%20to%20use%20ExpressionWrapper()%20to%20explicitly%20wrap%20arithmetic%20against%20integer%20fields%20in%20such%20cases)

To improve debuggability and give teams a migration aid, I propose an opt-
in setting that raises a specific exception the moment an overflow is
detected.

== Proposal

=== Add new parameter.
{{{
# settings.py
ERROR_IF_INTEGER_OVERFLOW_IN_FILTER_QUERYSET = True # default: False
}}}

=== Implementation sketch

* Add new exception
{{{
class EmptyResultSetDueToIntegerOverflow(Exception):
"""Raised when an integer overflow is detected while building a WHERE
clause. Subclasses plain Exception so it is *not* caught by existing
`EmptyResultSet` handlers."""
}}}

* Raise it at the source
In django.db.models.lookups.IntegerFieldOverflow (or equivalent part of
the lookup machinery), raise EmptyResultSetDueToIntegerOverflow instead of
EmptyResultSet.

* Handle it centrally
In django.db.models.sql.where.WhereNode.as_sql() wrap the call to
compiler.compile(child)

{{{
try:
...
except EmptyResultSetDueToIntegerOverflow:
if settings.ERROR_IF_INTEGER_OVERFLOW_IN_FILTER_QUERYSET:
raise
empty_needed -= 1 # current silent-empty logic
}}}

* Default behaviour unchanged
The setting defaults to False, so existing applications keep the current
silent-empty semantics unless they explicitly enable strict mode.

=== Benefits

* Safe migrations – Projects upgrading from 4.x can enable the flag in CI
to surface every place where arithmetic now truncates results.

* Debuggability – Developers on 5.x who run into an unexpected empty
queryset get an immediate stack-trace pointing at the offending
expression.

* Backwards-compatible – Opt-in toggle preserves the 5.0 behaviour by
default; no existing code breaks unless the project explicitly asks for
stricter checking.

* Path for future tightening – If the community agrees, the flag could
default to True in a future major release after a full deprecation cycle.


I am happy to prepare a pull request with tests and documentation if this
approach is acceptable. Thank you for considering this enhancement.
--
Ticket URL: <https://code.djangoproject.com/ticket/36484>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36484: Add optional error-on-integer-overflow setting for QuerySet filters
-------------------------------------+-------------------------------------

Reporter: Yosuke Takeuchi | Owner: (none)

Type: New feature | Status: new

Component: Database layer | Version: 5.0
(models, ORM) |
Severity: Normal | Resolution:

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Description changed by Yosuke Takeuchi:

Old description:

> == Overview
> In Django 5.0 the behaviour for arithmetic that overflows an integer
> column changed: a filter that would overflow now always returns an empty
> QuerySet instead of being transparently cast to a wider integer type (the
> historical behaviour on most back-ends). While this keeps runtime
> semantics simple, it makes migrations from Django 4.x harder to
> verify—tests may “pass” yet silently return fewer (or zero) rows. It is
> also difficult for developers on 5.x to notice when an overflow is
> happening because the condition is swallowed inside the ORM.
> (see:
> https://docs.djangoproject.com/en/5.2/releases/5.0/#:~:text=Filtering%20querysets%20against%20overflowing%20integer%20values%20now%20always%20returns%20an%20empty%20queryset.%20As%20a%20consequence%2C%20you%20may%20need%20to%20use%20ExpressionWrapper()%20to%20explicitly%20wrap%20arithmetic%20against%20integer%20fields%20in%20such%20cases)
>
> To improve debuggability and give teams a migration aid, I propose an

> opt-in setting that raises a specific exception the moment an overflow is

New description:

= Overview
In Django 5.0 the behaviour for arithmetic that overflows an integer
column changed: a filter that would overflow now always returns an empty
QuerySet instead of being transparently cast to a wider integer type (the
historical behaviour on most back-ends). While this keeps runtime
semantics simple, it makes migrations from Django 4.x harder to
verify—tests may “pass” yet silently return fewer (or zero) rows. It is
also difficult for developers on 5.x to notice when an overflow is
happening because the condition is swallowed inside the ORM.
(see:
https://docs.djangoproject.com/en/5.2/releases/5.0/#:~:text=Filtering%20querysets%20against%20overflowing%20integer%20values%20now%20always%20returns%20an%20empty%20queryset.%20As%20a%20consequence%2C%20you%20may%20need%20to%20use%20ExpressionWrapper()%20to%20explicitly%20wrap%20arithmetic%20against%20integer%20fields%20in%20such%20cases)

To improve debuggability and give teams a migration aid, I propose an opt-
in setting that raises a specific exception the moment an overflow is
detected.

= Proposal

--
Ticket URL: <https://code.djangoproject.com/ticket/36484#comment:1>

[Django]

#36484: Add optional error-on-integer-overflow setting for QuerySet filters
-------------------------------------+-------------------------------------

Reporter: bubusuke | Owner: (none)
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution: wontfix

Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* resolution: => wontfix
* status: new => closed
* version: 5.0 => dev

Comment:

Hello bubusuke! Thank you for the detailed and thoughtful proposal.

Considering that this is a new feature request, and per our
[https://docs.djangoproject.com/en/5.2/internals/contributing/bugs-and-
features/#requesting-features current documented process for proposing new
features], it should be submitted to the
[https://github.com/orgs/django/projects/24/ new feature ideas GitHub
project], rather than this ticket tracker. That space is used for
community discussion and Steering Council review prior to any decision and
ultimately implementation work.

A small note: adding new settings is generally considered a high bar in
Django and often controversial, so any proposal involving one will need
strong justification and community support. Also, if you used any AI tools
to assist in drafting this proposal, now and in the future we'd appreciate
a brief note to that effect, just to help us assess how best to review and
engage with the content.

Following standard practice, I'm closing this ticket as `wontfix` for now.
If the idea gains support through the feature ideas process, it can be
revisited.
--
Ticket URL: <https://code.djangoproject.com/ticket/36484#comment:2>

[Django]

#36484: Add optional error-on-integer-overflow setting for QuerySet filters
-------------------------------------+-------------------------------------
Reporter: bubusuke | Owner: (none)
Type: New feature | Status: closed
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by bubusuke):

Hi Natalia,

Thank you for the clear guidance.

I’m still getting used to the contribution workflow and didn’t realize the
proper submission path, but I understand now. I’ll draft the proposal in
the GitHub new-feature ideas project.

For the record, I didn’t use any AI tools when preparing the idea.

Thanks again.
--
Ticket URL: <https://code.djangoproject.com/ticket/36484#comment:3>