[Django] #27683: Change default transaction isolation level to READ COMMITTED on MySQL
Django
-------------------------------------+-------------------------------------
Reporter: Shai | Owner: nobody
Berger |
Type: New | Status: new
feature |
Component: Database | Version: master
layer (models, ORM) |
Severity: Release | Keywords:
blocker |
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 1 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
There have been plenty of bug reports and discussions about the problems
caused for users by MySQL's implementation of the REPEATABLE READ
transaction isolation level, and the fact that it is the default. Django
is mostly written for READ COMMITTED; of all supported backends, MySQL is
the only one to use a different level by default. Things will probably run
more in line with users expectations under READ COMMITTED, and reusable
apps' behavior will be more similar across database backends.
Some history: This has been raised already in the [#13906 1.2 era], and
possibly even before that; the question was raised in different forms
[#14026 again] and [#21670 again] until one more [#26347 instance] caused
it to be brought up on the mailing list. To be sure, it wasn't the first
time the issue was brought to the list either, but
[https://groups.google.com/forum/#!msg/django-
developers/6pWbpV1_6Us/a3CNmojACwAJ this discussion] seemed to conclude
with a rough consensus that the default transaction isolation level for
MySQL should change. As far as I'm aware, the issue has not been discussed
since then.
There are two kinds of backwards-compatibility problems I already see with
this change:
1) The obvious one -- Apps that are written to be correct under MySQL's
REPEATABLE READ isolation level may become subtly incorrect by default.
This is a very real problem, and hard to tackle because writing tests to
capture the problems is very hard -- one needs to make and use at least
two separate connections to the same database, and the Django ORM (and
testing framework) does not make that easy. The counter-argument to that
is that we likely have plenty of apps -- reusable or otherwise -- that are
currently subtly wrong because MySQL's REPEATABLE READ semantics is
surprising and unintuitive.
2) The less obvious one -- the way to set transaction isolation levels is
with SQL executed when the connection is opened, and we need to make sure
we don't interrupt with the user's own `init_command` if there is one.
Some very basic intro to transaction isolation levels in general and
MySQL's REPEATABLE READ in particular is in my DC.EU.2016 lightning talk
about this:
https://opbeat.com/community/posts/lightning-talks-day-1/ (starting around
4:00).
--
Ticket URL: <https://code.djangoproject.com/ticket/27683>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Adam Chainz):
An alternative to changing the default would be to have a system check
highly recommending READ-COMMITTED, similar to what we do for MySQL's
`sql_mode` (see
https://github.com/django/django/blob/master/django/db/backends/mysql/validation.py#L6
). This would also help educate users a bit more about the change.
Possibly this could be part of a path to changing the default... "Warning:
In Django 2.1 the default tx_isolation will change..."
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:1>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
Replying to [comment:1 Adam Chainz]:
> An alternative to changing the default would be to have a system check
highly recommending READ-COMMITTED [...] "Warning: In Django 2.1 the
default tx_isolation will change..."
That may indeed be the way to go. Two issues arise:
1) We need to make sure which deprecations can happen when (WRT our
deprecation policies around LTSs)
2) It may be seen as annoying: "if you know that it's correct to use that
other level, why don't you just go and do it? why bug me?"
The big advantage is that it completely solves my 2nd backward-
compatibility issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:2>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Old description:
New description:
There have been plenty of bug reports and discussions about the problems
caused for users by MySQL's implementation of the REPEATABLE READ
transaction isolation level, and the fact that it is the default. Django
is mostly written for READ COMMITTED; of all supported backends, MySQL is
the only one to use a different level by default. Things will probably run
more in line with users expectations under READ COMMITTED, and reusable
apps' behavior will be more similar across database backends.
Some history: This has been raised already in the [ticket:13906 1.2 era],
and possibly even before that; the question was raised in different forms
[ticket:14026 again] and [ticket:21670 again] until one more [ticket:26347
--
Comment (by Karen Tracey):
We really should do this! (Unfortunately I have no energy/time to devote.
But I did fix up the ticket links in the description.)
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:3>
Django
-------------------------------------+-------------------------------------
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aymeric Augustin):
Like I said in earlier discussions, I'm in favor of doing this change.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_docs: 1 => 0
* type: New feature => Cleanup/optimization
* severity: Release blocker => Normal
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:5>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
So, with respect to the option to use checks and deprecation warnings:
1) We can deprecate whatever we like whenever we like, the LTS limitation
is only about removing deprecation shims in the same major version (so,
shim introduced in 1.11 **can** be removed in 2.1).
2) Annoyance stands, so I think the correct path is at least to minimize
it by explicitly supporting isolation-level in `OPTIONS`. That's also the
easy route (both for users and implementation) to deprecation.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:6>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
[https://github.com/django/django/pull/7826 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:7>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Shai Berger
* status: new => assigned
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:8>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:9>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
Comment:
Review comments addressed.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:10>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
As I thought about this again, I wanted to ask if a deprecation cycle
brings any value compared to this alternate strategy. It doesn't seem to
me that projects need two release cycles to choose their isolation level
-- in the easiest case they just specify what's being used now. If we
changed the default isolation level now, we could have a "compatibility"
system check raise a warning if `'isolation_level'` isn't specified in the
database settings and end up in the same place (where all MySQL projects
have to specify an isolation level to silence a warning). We could remove
that check in Django 2.0 and avoid another 8 months of new MySQL projects
having a check or deprecation warning out of the box.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:11>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
At a first approximation, I suspect it is better to be cautious about this
sort of change. However, this may be a good question to ask the community.
The deprecation period may be required because although the warnings and
settings are at the project level, there may be code changes required at
the app level, and it may be better to keep the new projects aware of the
issue while 3rd parties adapt.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:12>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
Ok, I'll write to the mailing list shortly. With the current patch, would
the system check ever trigger in Django 2.1 when the default transaction
level changes?
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:13>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
Replying to [comment:13 Tim Graham]:
> With the current patch, would the system check ever trigger in Django
2.1 when the default transaction level changes?
No, it never would -- there will be no case where the user hasn't
specified a preference and the isolation level would still be REPEATABLE
READ.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:14>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
Was your thinking that the system check is needed because the deprecation
warning is silent by default or did you have another idea in mind?
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:15>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Shai Berger):
The system check was suggested by Adam in comment:1, and I thought the
analogy with strict-mode holds. I felt odd about there being both the
check and the deprecation warning (and commented about it in the original
PR) but yes, I felt that the fact they were both mostly silent (the check
is only vocal in `migrate` by default) sort-of justifies keeping both.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:16>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Tim Graham):
Thanks. Here's the [https://groups.google.com/d/topic/django-
developers/JiXalqmohUc/discussion django-developers thread].
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:17>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):
In [changeset:"f01ad1cb6a27669fd44b2079089e14f9dc4d3ca3" f01ad1c]:
{{{
#!CommitTicketReference repository=""
revision="f01ad1cb6a27669fd44b2079089e14f9dc4d3ca3"
Refs #27683 -- Allowed setting isolation level in DATABASES ['OPTIONS'] on
MySQL.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:18>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:19>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/7978 PR] to change the default
isolation to read committed.
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:20>
Django
-------------------------------------+-------------------------------------
Type: | Berger
Cleanup/optimization | Status: assigned
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):
In [changeset:"c4e18bb1cebc78e1e42c765608248ce3ead9deb7" c4e18bb]:
{{{
#!CommitTicketReference repository=""
revision="c4e18bb1cebc78e1e42c765608248ce3ead9deb7"
Refs #27683 -- Split up MySQL isolation level tests.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:21>
Django
-------------------------------------+-------------------------------------
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Shai Berger => (none)
* status: assigned => new
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:22>
Django
-------------------------------------+-------------------------------------
Type: | <noreply@…>
Cleanup/optimization | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => GitHub <noreply@…>
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"924af638e4d4fb8eb46a19ac0cafcb2e83480cf3" 924af638]:
{{{
#!CommitTicketReference repository=""
revision="924af638e4d4fb8eb46a19ac0cafcb2e83480cf3"
Fixed #27683 -- Made MySQL default to the read committed isolation level.
Thanks Shai Berger for test help and Adam Johnson for review.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:23>
Django
-------------------------------------+-------------------------------------
Type: | <noreply@…>
Cleanup/optimization | Status: closed
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by GitHub <noreply@…>):
In [changeset:"ab83d4d8fe76c2c6bc3b0a96aa2baed9c9f07f19" ab83d4d8]:
{{{
#!CommitTicketReference repository=""
revision="ab83d4d8fe76c2c6bc3b0a96aa2baed9c9f07f19"
Added multi_db=True to test cases that access the 'other' db connection.
Fixed a failure in the context processors tests when running in
reverse on MySQL due to an extra query after refs #27683.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27683#comment:24>