[Django] #34370: PositiveIntegerField validators don't work if the database is SQLite, leading to overflow errors

Django

unread,

Feb 24, 2023, 11:54:22 AM2/24/23

to django-...@googlegroups.com

#34370: PositiveIntegerField validators don't work if the database is SQLite,
leading to overflow errors
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
alexurbanowicz |
Type: Bug | Status: new
Component: Database | Version: 4.1
layer (models, ORM) | Keywords: IntegerField SQLite
Severity: Normal | MaxValueValidator
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
-------------------------------------+-------------------------------------
The [https://docs.djangoproject.com/en/4.1/ref/models/fields/#integerfield
documentation for IntegerField] states that:

''It uses MinValueValidator and MaxValueValidator to validate the input
based on the values that the default database supports.''

The
[https://docs.djangoproject.com/en/4.1/ref/models/fields/#positiveintegerfield
PositiveIntegerField] states that:

"Like an IntegerField, but must be either positive or zero (0). Values
from 0 to 2147483647 are safe in all databases supported by Django. The
value 0 is accepted for backward compatibility reasons."

So it is simple to assume that PositiveIntergerField uses
MaxValueValidator that applies the limit of the default database.

Yet it is possible, while using SQLite to attempt to store the value of
positive integer not supported by the DB, which leads to OverflowError:

**Python int too large to convert to SQLite INTEGER**

So it is either the IntegerField's MaxValueValidator is not applied,
contrary to what the documentation suggests, or the documentation has an
error for lack of specifying that you must use an explicit validator to
validate to the safe value given (why not default then?).

In my opinion it should be limited to the safe value by default, or have
the MaxValueValidator check it by default against the database limit.

Should I prepare the patch to limit it to the safe value?

This is also related to #27397.

--
Ticket URL: <https://code.djangoproject.com/ticket/34370>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Feb 24, 2023, 1:23:47 PM2/24/23

to django-...@googlegroups.com

#34370: PositiveIntegerField validators don't work if the database is SQLite,
leading to overflow errors
-------------------------------------+-------------------------------------

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak):

* cc: Simon Charette (added)
* easy: 1 => 0

Comment:

Thanks for the report.

SQLite support values up to `2 ** 63 - 1` we could add limits to the
`DatabaseOperations.integer_field_range()` on SQLite. What do you think
[https://code.djangoproject.com/ticket/12030#comment:14 Simon]?

--
Ticket URL: <https://code.djangoproject.com/ticket/34370#comment:1>

Django

unread,

Feb 24, 2023, 2:33:57 PM2/24/23

to django-...@googlegroups.com

#34370: PositiveIntegerField validators don't work if the database is SQLite,
leading to overflow errors
-------------------------------------+-------------------------------------

Reporter: Alex Urbanowicz | Owner: nobody

Comment (by Simon Charette):

Not opposed to adding the value to `integer_field_range` for SQLite. It
should be mentioned in the release notes though as it's slightly backward
incompatible as users of SQLite were previously able to store overflowing
values (e.g. 64 bit integers in 32 bit columns) without issues.

This issue is really just #27397 on save instead of reads because
`sqlite3` raises an `OverflowError` on when dealing with
[https://github.com/python/cpython/blob/090819ec5faea2d0c2cdf6bcfdbbc9df144a8aad/Modules/_sqlite/util.c#L143-L167
longs that overflow 64 bits].

In order to properly address #27397 by raising `(Empty|Full)ResultSet`
when provided overflowing values we need `integer_field_range` to be
defined anyway. Once it's the case we could adapt lookups such as `Exact`,
`GreaterThan(Eq)?` to special case literal right-hand-sides that overflow
their left hand side ranges to raise `EmptyResultSet` and the
`LowerThen(Eq)?` to raise `FullResultSet` and not only address #27397 for
SQLite but also deal protect against
[https://code.jeremyevans.net/2022-11-01-forcing-sequential-scans-on-
postgresql.html a form of DDoS vector on PostgreSQL] that is trivial to
exploit (think of any view that accepts `\d+` or `<int:pk>` to perform a
`Model.objects.get(pk=pk)`).

--
Ticket URL: <https://code.djangoproject.com/ticket/34370#comment:2>

Django

unread,

Feb 25, 2023, 6:09:57 AM2/25/23

to django-...@googlegroups.com

#34370: PositiveIntegerField validators don't work if the database is SQLite,
leading to overflow errors
-------------------------------------+-------------------------------------

Reporter: Alex Urbanowicz | Owner: Mohamed
| Nabil Rady
Type: Bug | Status: assigned