[Django] #31221: HStoreField returns str instead of dict during tests
Django
----------------------------------------------+------------------------
Reporter: Michael Mulholland | Owner: (none)
Type: Uncategorized | Status: new
Component: contrib.postgres | Version: 2.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
----------------------------------------------+------------------------
It appears that while running tests on my app, the behavior of HStoreField
is not consistent. The TestCase I am trying to run is to test a script
(which normally runs fine as a management command) that crashes upon
trying to use metadata in an HStoreField. Model instances for testing are
loaded from a fixture.
While debugging the test, I've found that accessing an HStoreField on a
model instance will return a str type instead of the expected dict. What's
stranger is that this doesn't occur when the test database already exists
and --keepdb is used (the first run with --keepdb will fail, but the
second doesn't). This leads me to believe that the way fixtures are loaded
during testing is breaking HStoreField's to_python conversion or
something.
I found a similar issue #25454, where the same problem was solved, but it
looks like the problem is either occurring under different circumstances
or has been reintroduced.
My test runner is from this gist, where the test db is created with hstore
extension installed:
[https://gist.github.com/smcoll/bb2533e4b53ae570e11eb2ab011b887b]
Currently my knowledge of the django code base isn't extensive enough to
suggest a fix, but I have been able to reproduce this problem with the
following models/tests:
{{{
# models.py
class MyModel(models.Model):
name = models.CharField(max_length=66)
metadata = HStoreField(blank=True)
}}}
{{{
# tests.py
class TestMyApp(TestCase):
fixtures = ['i_myapp']
def test_mymodel(self):
for mymodel_instance in MyModel.objects.all():
self.assertEqual(type(mymodel_instance.metadata), dict)
}}}
{{{
# i_myapp.json
[
{
"model": "myapp.mymodel",
"pk": 4,
"fields": {
"name": "First",
"metadata": "{}"
}
},
{
"model": "myapp.mymodel",
"pk": 5,
"fields": {
"name": "Second",
"metadata": "{\"details\": \"this one isn't blank\"}"
}
}
]
}}}
The test test_mymodel fails with:
{{{
AssertionError: <class 'str'> != <class 'dict'>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31221>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Michael Mulholland | Owner: (none)
Type: Uncategorized | Status: new
Component: contrib.postgres | Version: 2.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Simon Charette):
Did you make sure to add `django.contrib.postgres` to your
`INSTALLED_APPS`
[https://docs.djangoproject.com/en/3.0/ref/contrib/postgres/fields/#hstorefield
as documented]?
The latter is in charge of
[https://github.com/django/django/blob/b9cf764be62e77b4777b3a75ec256f6209a57671/django/contrib/postgres/apps.py#L11-L28
connecting a signal receiver] that
[https://github.com/django/django/blob/b9cf764be62e77b4777b3a75ec256f6209a57671/django/contrib/postgres/signals.py#L37-L43
registers] the necessary `psycopg2` adapters on connection creation.
If that's the origin of your issue I suggest we re-purpose this ticket to
add system checks to all `django.contrib.postgres.fields` that errors when
a field is used but `self.model._meta.apps.get('postgres')` is missing.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:1>
Django
Reporter: Michael Mulholland | Owner: (none)
Type: Uncategorized | Status: new
Component: contrib.postgres | Version: 2.2
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Michael Mulholland):
Replying to [comment:1 Simon Charette]:
> Did you make sure to add `django.contrib.postgres` to your
`INSTALLED_APPS`
[https://docs.djangoproject.com/en/3.0/ref/contrib/postgres/fields/#hstorefield
as documented]?
Yes, that was already in the installed apps when I encountered the
problem.
When I debug the test program, it does appear to be running
register_type_handlers as expected during database setup. It also seems to
be performing the deserialization to a python dict when the fixtures are
loaded (using to_python). I'm not sure what it's doing with the models
after that though, since there's nothing in the test db during testing,
and accessing model instance values from the queryset gives the hstore
formatted strings.
So the test data I gave above turns into: {{{[(4, 'First', ''), (5,
'Second', '"details"=>"this one isn\'t blank"')]}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:2>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* status: new => closed
* type: Uncategorized => Bug
* version: 2.2 => master
* resolution: => needsinfo
Comment:
Thanks for this ticket, however it works for me. This can be an issue in
your custom `TestRunner`, is there any reason to not use
`HStoreExtension()` in migrations instead of a custom `TestRunner`.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:3>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: needsinfo
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Michael Mulholland):
Replying to [comment:3 felixxm]:
> is there any reason to not use `HStoreExtension()` in migrations instead
of a custom `TestRunner`.
Looks like that was my problem. The migrations for the app I work on were
created with {{{makemigrations}}} and the original db user wasn't a
superuser so they couldn't run {{{HStoreExtension}}} (the extension was
being created externally). Upon closer inspection, the only other
operation being run by {{{HStoreExtension}}} is clearing cached hstore
oids. Adding {{{get_hstore_oids.cache_clear()}}} to my testrunner fixes
the issue, though I'll probably just scrap the testrunner and add
{{{HStoreExtension}}}.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:4>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: master
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by felixxm):
* resolution: needsinfo => invalid
Comment:
Thanks for sharing the results of your investigation.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:5>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: master
Severity: Normal | Resolution:
Keywords: hstore | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Ryan C. Schwiebert):
* keywords: => hstore
* status: closed => new
* resolution: invalid =>
Comment:
This has been happening in a web server of mine. It is not clear at all
what causes it to happen, but when starts happening it persists.
{{{
Django==2.2.13
python==3.5.2
psycopg2-binary==2.8.3
django-tastypie==0.14.3
}}}
The model in question has this field:
`entities = HStoreField(null=True, default=None)`
The input in question is `{"1": None}`
The traceback looks like this, triggered by an assertion that checks the
type of the field to make sure it is a dict:
{{{
returned a value for entities that was not a dict. It returned this:
"1"=>NULL.
Jun 22 14:09:22 uwsgi[2563]: Traceback (most recent call last):
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/tastypie/resources.py", line 227, in wrapper
Jun 22 14:09:22 uwsgi[2563]: response = callback(request, *args,
**kwargs)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/tastypie/resources.py", line 476, in dispatch_detail
Jun 22 14:09:22 uwsgi[2563]: return self.dispatch('detail', request,
**kwargs)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/tastypie/resources.py", line 499, in dispatch
Jun 22 14:09:22 uwsgi[2563]: response = method(request, **kwargs)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/tastypie/resources.py", line 1383, in get_detail
Jun 22 14:09:22 uwsgi[2563]: obj =
self.cached_obj_get(bundle=basic_bundle,
**self.remove_api_resource_names(kwargs))
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/tastypie/resources.py", line 1202, in cached_obj_get
Jun 22 14:09:22 uwsgi[2563]: cached_bundle =
self.obj_get(bundle=bundle, **kwargs)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/testpackage/frontend/api.py", line 2997, in obj_get
Jun 22 14:09:22 uwsgi[2563]: ruleData = self.getData(pk)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/testpackage/frontend/api.py", line 2941, in getData
Jun 22 14:09:22 uwsgi[2563]: update = buildUpdate(entry, Display)
Jun 22 14:09:22 uwsgi[2563]: File "/opt/venvs/test/lib/python3.5/site-
packages/eventLogger.py", line 133, in buildUpdate
Jun 22 14:09:22 uwsgi[2563]: raise TypeError('Wrong entities data
type.')
}}}
I tested using the ORM in a manage.py shell session that recovering this
object, I was able to access the `entities` attribute and indeed get
`{"1": None}`
Restarting the wsgi server sometimes fixes it.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:6>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: master
Severity: Normal | Resolution: invalid
Keywords: hstore | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Simon Charette):
* status: new => closed
* resolution: => invalid
Comment:
Unless you can provide a test project to reproduce the issue there isn't
much that can be done on our side as nothing proves that Django is at
fault.
Every time a similar issue was reported it was due to misconfiguration
issue.
If you can provide a test project that follows the aforementioned
documented guidelines and reproduces the issue we'll certainly look into
it.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:7>
Django
------------------------------------+--------------------------------------
Reporter: Michael Mulholland | Owner: (none)
Component: contrib.postgres | Version: dev
Severity: Normal | Resolution: invalid
Keywords: hstore | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Dylan Young):
FYI for those still encountering this issue, this seems to be connected to
the caching in `django.contrib.postgres.signals.get_hstore_oids`.
Basically it's caching the oids from the production DB before it starts
running tests, so the registration call when the testing DB is created is
registering the wrong OIDs as hstore fields.
Still probably a configuration issue (haven't tracked down the cause yet),
but hopefully this helps narrow it down for those coming to this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/31221#comment:8>