[Django] #25626: Storing big strings with special chars fails when using PostgreSQL with SQL_ASCII encoding

2 views

Skip to first unread message

Django

unread,

Oct 28, 2015, 3:28:26 PM10/28/15

to django-...@googlegroups.com

#25626: Storing big strings with special chars fails when using PostgreSQL with
SQL_ASCII encoding
----------------------------------------------+--------------------
Reporter: JoseTomasTocino | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer (models, ORM) | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------------------+--------------------
If a project uses a PostgreSQL database with SQL_ASCII encoding, a model
with a CharField may raise a {{{DataError: value too long}}} error when
storing strings with special characters.

Supposing a simple model like this:

{{{
#!python
class TestModel(models.Model):
title = models.CharField(max_length=5)
}}}

And a PostgreSQL database called `testing` with SQL_ASCII encoding, that
can be created using:
{{{
#!sql
CREATE DATABASE testing WITH TEMPLATE = template0 ENCODING = 'SQL_ASCII';
}}}

If you try to add an instance of the model with a title with special
characters it fails, even being under the length limit:

{{{
#!python
TestModel.objects.create(title="ñññ")
}}}

{{{
#!pytb
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/query.py", line 348, in create
obj.save(force_insert=True, using=self.db)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/base.py", line 734, in save
force_update=force_update, update_fields=update_fields)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/base.py", line 762, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/base.py", line 846, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk,
raw)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/base.py", line 885, in _do_insert
using=using, raw=raw)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/query.py", line 920, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 974, in execute_sql
cursor.execute(sql, params)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/vagrant/ENV/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
DataError: value too long for type character varying(5)
}}}

This is particularly relevant in the admin app, where the {{{LogEntry}}}
model has a {{{object_repr}}} field with `max_length=200` that's prone to
fail whenever a user does something on a model with a potentially big
`repr` string (which is exactly how I came across this bug).

IMHO this has to do with how non-ascii characters are represented in a
ASCII database.

--
Ticket URL: <https://code.djangoproject.com/ticket/25626>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,

Oct 28, 2015, 3:36:53 PM10/28/15

to django-...@googlegroups.com

#25626: Storing big strings with special chars fails when using PostgreSQL with
SQL_ASCII encoding

Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by timgraham):

* status: new => closed
* needs_better_patch: => 0
* resolution: => invalid
* needs_tests: => 0
* needs_docs: => 0

Comment:

[https://docs.djangoproject.com/en/1.8/ref/databases/#encoding As
documented], "Django assumes that all databases use UTF-8 encoding. Using
other encodings may result in unexpected behavior such as “value too long”
errors from your database for data that is valid in Django."