[Django] #36233: Specific DecimalField with max_digits > 15 can be stored but not retrieved on SQLite

[Django]

#36233: Specific DecimalField with max_digits > 15 can be stored but not retrieved
on SQLite
-------------------------------------+-------------------------------------
Reporter: Orazio | Type: Bug
Status: new | Component: Database
| layer (models, ORM)
Version: dev | Severity: Normal
Keywords: sqlite, orm, | Triage Stage:
decimalfield, invalidoperation | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Similar to https://code.djangoproject.com/ticket/33954 but different
exception:

Given this model

{{{
class Apple(models.Model):
weight = models.DecimalField(max_digits=16, decimal_places=0)
}}}

Trying to store 9999999999999999 (16 digits)

{{{
$ python manage.py shell
7 objects imported automatically (use -v 2 for details).

Python 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> test = Apple(weight="9999999999999999")
>>> test.full_clean()
>>> test.save()
>>> Apple.objects.last()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/user/test_project/django/django/db/models/manager.py", line
87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/test_project/django/django/db/models/query.py", line
1105, in last
for obj in queryset[:1]:
File "/home/user/test_project/django/django/db/models/query.py", line
383, in __iter__
self._fetch_all()
File "/home/user/test_project/django/django/db/models/query.py", line
1923, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/test_project/django/django/db/models/query.py", line
122, in __iter__
for row in compiler.results_iter(results):
File "/home/user/test_project/django/django/db/models/sql/compiler.py",
line 1540, in apply_converters
value = converter(value, expression, connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/user/test_project/django/django/db/backends/sqlite3/operations.py",
line 346, in converter
return create_decimal(value).quantize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
>>>
}}}

I made sure to test with the latest commit


{{{
$ pip list
Package Version Editable project location
-------- --------------------- -------------------------
asgiref 3.8.1
Django 6.0.dev20250306010223 /home/user/test_project/django
pip 24.0
sqlparse 0.5.3
}}}

Exception is raised here:
https://github.com/django/django/blob/bad1a18ff28a671f2fdfd447bdf8f43602f882c2/django/db/backends/sqlite3/operations.py#L346

As you might imagine this also fails when say max_digits is 20 and storing
20 nines, and so on. Shouldn't this raise an error when running
full_clean() on the object before storing?
--
Ticket URL: <https://code.djangoproject.com/ticket/36233>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36233: Specific DecimalField with max_digits > 15 can be stored but not retrieved
on SQLite
-------------------------------------+-------------------------------------

Reporter: Orazio | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite, orm, | Triage Stage: Accepted
decimalfield, invalidoperation |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Hridesh MG):

* stage: Unreviewed => Accepted

Comment:

Was able to reproduce this issue, also noticed that it triggers on a
specific number - i.e. test = Apple(weight=9999999999999994) seems to work
fine but the moment weight becomes 9999999999999995 it starts to throw the
error
--
Ticket URL: <https://code.djangoproject.com/ticket/36233#comment:1>

[Django]

#36233: Specific DecimalField with max_digits > 15 can be stored but not retrieved
on SQLite
-------------------------------------+-------------------------------------
Reporter: Orazio | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite, orm, | Triage Stage: Accepted
decimalfield, invalidoperation |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

I'm not sure if this relates to a particular version of Python or SQLite
(I tested with Python 3.12.3 and `sqlite3.sqlite_version` of 3.46) and if
it now implements the `decimal` data type differently but it appears to be
returning an `int` instance when there are no floating values and in this
case the quantization is completely unnecessary. Here's a demonstration of
how the code could be adapted to take that into consideration.

{{{#!diff
diff --git a/django/db/backends/sqlite3/operations.py
b/django/db/backends/sqlite3/operations.py
index 08de246d70..9ad0bf3833 100644
--- a/django/db/backends/sqlite3/operations.py
+++ b/django/db/backends/sqlite3/operations.py
@@ -342,7 +342,9 @@ def get_decimalfield_converter(self, expression):
)

def converter(value, expression, connection):
- if value is not None:
+ if isinstance(value, int):
+ return decimal.Decimal(value)
+ elif value is not None:
return create_decimal(value).quantize(
quantize_value,
context=expression.output_field.context
)
@@ -350,7 +352,9 @@ def converter(value, expression, connection):
else:

def converter(value, expression, connection):
- if value is not None:
+ if isinstance(value, int):
+ return decimal.Decimal(value)
+ elif value is not None:
return create_decimal(value)

return converter
}}}

Note that I haven't put much thoughts into it but it seemed adequate to
only apply float quantization when provided `float` instances?
--
Ticket URL: <https://code.djangoproject.com/ticket/36233#comment:2>

[Django]

#36233: Specific DecimalField with max_digits > 15 can be stored but not retrieved
on SQLite
-------------------------------------+-------------------------------------
Reporter: Orazio | Owner: (none)
Type: Bug | Status: new
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite, orm, | Triage Stage: Accepted
decimalfield, invalidoperation |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Orazio):

I omitted the decimal part because it didn’t seem relevant but it actually
occurs when the integer part has more than 15 digits, such as in the
following model as well:

{{{
class Apple(models.Model):
weight = models.DecimalField(max_digits=21, decimal_places=5)
}}}

While storing the same number 9999999999999999
--
Ticket URL: <https://code.djangoproject.com/ticket/36233#comment:3>