Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

[Shaheed Haque]

Hi,

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

===================
    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)

        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'

===================

I am testing the update to Django 3.1.2 and hit an error in executing this line:

    jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

==================

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
===============================

As you can perhaps see from the bolded part (for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74), the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

[Jason]

this is pretty interesting, and may be something to report to the devs at https://groups.google.com/g/django-developers, or open up a ticket at https://code.djangoproject.com/  I don't see any tickets with the query jsonfield foreign data that are similar with your experience, so you may have found an edge case.

[shahee...@gmail.com]

Thanks Jason, I filed https://code.djangoproject.com/ticket/32111#ticket.

[Jason]

looks like you got a response back, and there's more context in the linked ticket too.  tl;dr json is not jsonb, and django only works with the latter

[Shaheed Haque]

Indeed. And to close out the discussion I can confirm that my FDW implementations were using JSON columns, and the issue was resolved by switching to JSONB instead. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f5e441c5-f38a-4260-8188-d002fbe83c99n%40googlegroups.com.