Sqlite 3.12 breaks the Django test suite

[Raphael Hertzog]

[ CCing an upstream developer of SQlite too ]

Hello,

I did not want to open a ticket as I'm not sure if the problem
is in SQLite or in Django but the Django test suite fails really badly
with SQLite 3.12.0 that got recently released (and which is already
in Debian Unstable).

When using the default --parallel run, I get many weird
errors and sometimes even a deadlock. When I run it with --parallel=1,
I got a segfault.

You can see a log of the failure here (this is a parallel run):
https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz

There's a backtrace of the segfault (with --parallel=1):
https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1

Cheers,

PS: Tracked here in Debian too:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

[Aymeric Augustin]

This other ticket about a SQLite segfault may or may not be related:
https://code.djangoproject.com/ticket/24080

--
Aymeric.

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20160407110012.GA18876%40home.ouaza.com.
> For more options, visit https://groups.google.com/d/optout.

[Florian Apolloner]

Perfect, I can reproduce it. This also happens if the default/other databases are not in memory -- I'll see what happens if I switch to postgres for the second database (though I am getting an error there currently -- might have to fix master :D)

[Raphael Hertzog]

Le jeudi 07 avril 2016, Aymeric Augustin a écrit :
> This other ticket about a SQLite segfault may or may not be related:
> https://code.djangoproject.com/ticket/24080

Given how old it is, I was tempted to say it's entirely unrelated. But
maybe not. I discovered faulthandler thanks to this and the stacktrace
appears in the same place (rollback handling):

$ cd tests
$ LANG=C PYTHONPATH=.. python3 -X faulthandler ./runtests.py --verbosity=2 --parallel=1 admin_views.tests.AdminViewBasicTest
[...]
test_popup_dismiss_related (admin_views.tests.AdminViewBasicTest) ... ok
Fatal Python error: Segmentation fault

Current thread 0x00007fab40575700 (most recent call first):
File "/home/rhertzog/deb/pkg/python-django/django/db/backends/sqlite3/base.py", line 321 in execute
File "/home/rhertzog/deb/pkg/python-django/django/db/backends/utils.py", line 62 in execute
File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 288 in _savepoint_rollback
File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", line 328 in savepoint_rollback
File "/home/rhertzog/deb/pkg/python-django/django/db/transaction.py", line 243 in __exit__
File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1016 in _rollback_atomics
File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 1073 in _fixture_teardown
File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 919 in _post_teardown
File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 217 in __call__
File "/usr/lib/python3.5/unittest/suite.py", line 122 in run
File "/usr/lib/python3.5/unittest/suite.py", line 84 in __call__
File "/usr/lib/python3.5/unittest/runner.py", line 176 in run
File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 494 in run_suite
File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 533 in run_tests
File "./runtests.py", line 275 in django_tests
File "./runtests.py", line 458 in <module>

Maybe SQLite 3.12 makes it more likely to trigger to the problem or something
like that. I can reproduce the crash with Python 2 and Python 3.

Cheers,

[Florian Apolloner]

This is the list of SQL statements we are executing: https://dpaste.de/Mqu1/raw first column is thread id, second is cursor id and third is the statement. I am a little bit confused about why the cursor ids seem to be "jumpy" like that -- but in general this should not cause issues I think

[Sam Cooke]

We ran into this issue a couple of months ago - I'm not sure how helpful this is but just in case extra information is useful I'll share our workarounds. The faulthandler output was roughly the same - the error happened during a rollback. This also happened on multiple python and sqlite versions.

It started when we removed a model and the one foreign key to that model. The quickest workaround we found was to simply add the model and foreign key back when running tests - which implies to me that the structure of the database was related to whether or not it segfaulted (the test where the segfault occurred was unrelated to the model that was removed).

The second option was we could remove the creation of a model during a test (i.e. something that meant a rollback was required).

The third option was to use an on-disk sqlite database rather than in memory.

I hope some of that is useful,

Sam

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/813967fd-3a1f-4b2a-93cc-c14f320f920f%40googlegroups.com.

[Florian Apolloner]

I was able to reduce the admin_view testcases to https://github.com/apollo13/django/blob/f086934f02efaca272e7e9b29921a31fd47821aa/tests/admin_views/tests.py (see my sqlite_segfault branch)

On Thursday, April 7, 2016 at 3:06:14 PM UTC+2, Florian Apolloner wrote:

[Florian Apolloner]

I've created an application which shows the behavior: https://github.com/apollo13/sqlite_bug -- I cannot yet reproduce it without running it through the testing process -- ./manage.py test admin_views triggers the bug in this application!

[Florian Apolloner]

mkdir ~/sqlite
cd ~/sqlite
git clone g...@github.com:django/django.git
git clone g...@github.com:apollo13/sqlite_bug.git
export PYTHONPATH=`pwd`/django
cd sqlite_bug
gdb python
(in gdb shell) r ./manage.py test admin_views

Please tell me if this does not cause a segfault for you.

Cheers,
Florian

On Thu, Apr 7, 2016, at 06:07 PM, Richard Hipp wrote:
> On 4/7/16, Florian Apolloner <f.apo...@gmail.com> wrote:
> > I've created an application which shows the behavior:
> > https://github.com/apollo13/sqlite_bug -- I cannot yet reproduce it without
> >
> > running it through the testing process -- ./manage.py test admin_views
> > triggers the bug in this application!
>

> I have not touched Python in over a decade. Please help me track down
> the problem by giving me step-by-step instructions on how to get this
> test running, inside of gdb, on Ubuntu 14.04.3 LTS.
> --
> D. Richard Hipp
> d...@sqlite.org

[Aymeric Augustin]

For the record, the discussion continues here:
https://www.sqlite.org/src/info/7f7f8026eda38

--
Aymeric.

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1460046444.1161220.572000497.6B596361%40webmail.messagingengine.com.