Possible bug transaction rollback or SQLite backend

[Jonas H]

Hi there,

there seems to be something wrong with either the SQLite backend, the SQLite library itself or Django's transaction handling. Whenever I try to run the test suite of a specific project of mine (https://github.com/jonashaag/fahrtkostenrechner), using `fahrtkostenrechner/manage.py test events`, it shows the following behavior:

* On Linux, it doesn't crash and shows actual test failures.

* On OSX, with SQLite's in-memory database as test database, a segfault (!) occurs. (I'm not familiar with the lower-level Python SQLite bindings, but I guess a segfault should *never* happen in the first place?!)

* On OSX, with a on-disk test database, a "django.db.utils.ProgrammingError: Cannot operate on a closed database." (caused by `_cursor()`: https://github.com/django/django/blob/4d0f8831a7498ab3b1ebcf4cafa2ee234503e4f8/django/db/backends/base/base.py#L206) occurs.

It looks to me that the cause of this is some unfinished transaction rollback, but I'm not sure, so I'm asking for guidance here. In the file linked above in L193, is it legitimate for `needs_rollback` to  be True? In my debugging I found `connection != None` and `needs_rollback=True` whenever `_cursor()` fails in L206, so I figured maybe there might be an outstanding rollback for some reason?

I use Python 3.5, SQLite 3.9.2 and I can reproduce the crash on Django >=1.8.4,<1.9.

Jonas

[Tim Graham]

Hi Jonas, I wonder if https://code.djangoproject.com/ticket/24080 might be related. If you could distill the project you linked into a minimal example to reproduce that would be ideal. By the way, "is it a bug?" questions normally go to django-users. Tim

[Jonas H]

Tim,

thanks for the pointer. I'm not sure if my issue is related. I've found a minimal example for another (same?) issue: https://code.djangoproject.com/ticket/25860

[Carl Meyer]

Hi Jonas,

On 12/03/2015 07:02 AM, Jonas H wrote:
> Hi there,
>
> there seems to be something wrong with either the SQLite backend, the
> SQLite library itself or Django's transaction handling. Whenever I try
> to run the test suite of a specific project of mine
> (https://github.com/jonashaag/fahrtkostenrechner), using
> `fahrtkostenrechner/manage.py test events`, it shows the following behavior:
>
> * On Linux, it doesn't crash and shows actual test failures.
> * On OSX, with SQLite's in-memory database as test database, a segfault
> (!) occurs. (I'm not familiar with the lower-level Python SQLite
> bindings, but I guess a segfault should *never* happen in the first place?!)

Certainly a segfault should not happen :-) The fact that one is even
possible does point to a bug in Python's sqlite module (AFAIK any case
where one can trigger a segfault from pure Python code is considered a
bug in CPython). It may be that we can avoid triggering this particular
segfault bug, if we can figure out how we are triggering it.

I'd be extremely surprised to find _two_ segfault bugs in the SQLite
bindings (that both occur only on OS X) so it seems almost certain to me
that https://code.djangoproject.com/ticket/24080 is the same underlying
issue one way or another.

> * On OSX, with a on-disk test database, a
> "django.db.utils.ProgrammingError: Cannot operate on a closed database."
> (caused by `_cursor()`:
> https://github.com/django/django/blob/4d0f8831a7498ab3b1ebcf4cafa2ee234503e4f8/django/db/backends/base/base.py#L206)
> occurs.

That's a useful clue! If it consistently tracks with the segfault, where
the only difference is on-disk vs memory db, it seems likely that it's
the same situation that's causing a segfault in the memory-db case.

> It looks to me that the cause of this is some unfinished transaction
> rollback, but I'm not sure, so I'm asking for guidance here. In the file
> linked above in L193, is it legitimate for `needs_rollback` to be True?
> In my debugging I found `connection != None` and `needs_rollback=True`
> whenever `_cursor()` fails in L206, so I figured maybe there might be an
> outstanding rollback for some reason?

`connection != None` and `needs_rollback = True` is a legitimate state
for the connection to be in when someone asks for a cursor, AFAIK. It
would be most useful to trace back and find out a) what caused
`needs_rollback` to be set to `True`, and b) when/how/why was the
database closed?

> I use Python 3.5, SQLite 3.9.2 and I can reproduce the crash on Django
>>=1.8.4,<1.9.

This might be useful information to add to
https://code.djangoproject.com/ticket/24080, if it's not already present
there.

Does the version range you give imply that the issue goes away in Django
1.9, or just that you've never tried it there?

Carl

[Jonas H.]

> On 03 Dec 2015, at 17:04, Carl Meyer <ca...@oddbird.net> wrote:
>
>> I use Python 3.5, SQLite 3.9.2 and I can reproduce the crash on Django
>>> =1.8.4,<1.9.
>
> This might be useful information to add to
> https://code.djangoproject.com/ticket/24080, if it's not already present
> there.
>
> Does the version range you give imply that the issue goes away in Django
> 1.9, or just that you've never tried it there?

So, having ported my application to Django 1.9, I can say that the error is gone in Django 1.9. It happens in Django 1.8.X, X>=4.

I can’t test on Django 1.8.X, X<4, because the application needs #25160 fixed. I can’t test at all on Django < 1.8.

Another observation is that the error only appears *without* migrations, i.e. if the “migrations” directory doesn’t exist. It works for 1.8.X, X>=4 if proper migrations exists.

Also, the error only happens when `raise SkipTest` isn’t used correctly, as shown earlier in this thread.