[Django] #32424: in-memory test database does not work with multiprocessing.apply_async
Django
-------------------------------------+-------------------------------------
Reporter: Tata | Owner: nobody
Krushna Chaitanya |
Type: | Status: new
Uncategorized |
Component: Database | Version: 3.0
layer (models, ORM) | Keywords: sqlite3, threading,
Severity: Normal | tests
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Hi,
I am using Django 3.0 (On Ubuntu 18.04 installed using pip3). The below
example works only with threading but not with
multiprocessing.apply_async. Here is my sample, Looking at the Django code
I have made sure that all requirements are met, I am using py3.6 and the
sqlite3 version is also met.
{{{
$ python3 -c "from sqlite3 import dbapi2 as Database; print
(Database.__name__, Database.sqlite_version_info)"
sqlite3.dbapi2 (3, 22, 0)
}}}
{{{
import os
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"
import datetime, threading
from multiprocessing import Pool
# django stuff
import django
django.setup()
from polls.models import *
from django.core.mail import mail_admins
from django.test.utils import *
from django.db import connection
def create_object():
print("Creating Poll")
p = Question()
p.question_text = "What's up doc ?"
p.pub_date = datetime.date.today()
p.save()
print("Poll object saved. Id: %d" % p.id)
MODULE = 1
if __name__ == "__main__":
setup_test_environment()
old_db_name = "db.sqlite3"
new_db_name = connection.creation.create_test_db(verbosity=1)
print("New DATABASE:", new_db_name)
if MODULE == 0:
t = threading.Thread(target=create_object)
t.start()
t.join()
elif MODULE == 1:
with Pool(2) as p:
p.apply_async(create_object)
else:
create_object()
obj = Question.objects.get()
print(obj.question_text)
teardown_test_environment()
connection.creation.destroy_test_db(old_db_name)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32424>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Chaitanya |
Type: Uncategorized | Status: new
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite3, threading, | Triage Stage:
tests | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tata Krushna Chaitanya):
Similar to https://code.djangoproject.com/ticket/12118 but with
multiprocess.apply_async instead of normal threads.
--
Ticket URL: <https://code.djangoproject.com/ticket/32424#comment:1>
Django
-------------------------------------+-------------------------------------
Chaitanya | ArkaprabhaChakraborty
Type: Uncategorized | Status: assigned
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: sqlite3, threading, | Triage Stage:
tests | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => ArkaprabhaChakraborty
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/32424#comment:2>
Django
-------------------------------------+-------------------------------------
Chaitanya | Arkaprabha Chakraborty
Type: Uncategorized | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Keywords: sqlite3, threading, | Triage Stage:
tests | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => wontfix
Comment:
I fear like this won't be possible to do per the in-memory nature of such
databases.
It's one thing to serialize queries between multiple threads in a single
process but it's another one to coordinate it across multiple processes
that need to share the same memory.
Closing as ''wontfix'' as nothing seems to indicate this is even possible
and Django's core doesn't have any use for it.
--
Ticket URL: <https://code.djangoproject.com/ticket/32424#comment:3>
Django
-------------------------------------+-------------------------------------
Chaitanya | Arkaprabha Chakraborty
Type: Uncategorized | Status: closed
Component: Database layer | Version: 3.0
(models, ORM) |
Severity: Normal | Resolution: wontfix
Keywords: sqlite3, threading, | Triage Stage:
tests | Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tata Krushna Chaitanya):
Thanks, Simon. It's a bit obscure use case (Django + multiprocessing) but
albiet an important one as the test will fail, currently, I am working
around with making the DB updates manually from the tests, we can revisit
if there are more users for this.
--
Ticket URL: <https://code.djangoproject.com/ticket/32424#comment:4>