Django models and multiprocessing: "no results to fetch"

[Rainer Hihn]

Hi all,

in my project, I'm traversing through a huge directory with millions of files, parse them and write the results into a Postgres database.
Unfortunately, I'm facing erros when I'm using multiprocessing.Pool with more than one processes.

This is how I'm inserting new values into the database:

obj, created = MyModel.objects.get_or_create(foo=foo, defaults={'foo': foo,}) # foo is unique


And this is the error message I'm getting: 


[ERROR] no results to fetch
Traceback (most recent call last):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
psycopg2.ProgrammingError: no results to fetch

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/develop/myApp/web/service/ais.py", line 94, in save
    db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
  File "~/develop/myApp/web/utils.py", line 85, in create_foo
    obj = MyModel.objects.get(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
    num = len(clone)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
    self._fetch_all()
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1177, in execute_sql
    return list(result)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1576, in cursor_iter
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1576, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
django.db.utils.ProgrammingError: no results to fetch
[ERROR] error with status PGRES_TUPLES_OK and no message from the libpq
Traceback (most recent call last):
  File "~/develop/myApp/web/utils.py", line 85, in create_foo
    obj = MyModel.objects.get(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get
    raise self.model.DoesNotExist(
web.models.MyModel.DoesNotExist: MyModel matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/develop/myApp/web/service/ais.py", line 94, in save
    db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
  File "~/develop/myApp/web/utils.py", line 89, in create_foo
    obj = MyModel.objects.create(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "~/develop/myApp/web/models.py", line 34, in save
    return super().save(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 782, in save_base
    updated = self._save_table(
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 924, in _do_insert
    return manager._insert(
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1384, in execute_sql
    cursor.execute(sql, params)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

---

Some additional info:

• This does not happen with sqlite
• I'm pretty sure it has something to do with multithreading, since I don't get this error with Pool(processes=1) and the higher the processes-parameter is, the more often does this error occur.
• Versions:
• Django==3.0.3
• psycopg2-binary==2.8.4

Is there any chance I can get this working with ~8 threads? Is there something wrong with how I insert the data?
Unfortunately, I have several models that depend on each other (although I get this error with just inserting the first - didn't even get to the other models), so bulk-insert is not an option.