[Django] #34852: Django Unit Tests break when using replicated MySQL Cluster v8.0.28
Django
-------------------------------------+-------------------------------------
Reporter: Aaron | Owner: nobody
Blair |
Type: Bug | Status: new
Component: Testing | Version: 4.1
framework |
Severity: Normal | Keywords: Test MySQL Cluster
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 1
UI/UX: 0 |
-------------------------------------+-------------------------------------
When running unit tests, the internal testing of a db for using
transactions breaks when using MySQL Cluster server v8.0.28 because it
requires a primary key for tables, and this line in
https://github.com/django/django/blob/main/django/db/backends/base/features.py
breaks it :
{{{
@cached_property
def supports_transactions(self):
"""Confirm support for transactions."""
with self.connection.cursor() as cursor:
cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
}}}
We have been patch-fixing that file as follows:
<- cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT PRIMARY
KEY)")
---
-> cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
--
Ticket URL: <https://code.djangoproject.com/ticket/34852>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Aaron Blair | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Description changed by Aaron Blair:
Old description:
> When running unit tests, the internal testing of a db for using
> transactions breaks when using MySQL Cluster server v8.0.28 because it
> requires a primary key for tables, and this line in
> https://github.com/django/django/blob/main/django/db/backends/base/features.py
> breaks it :
>
> {{{
> @cached_property
> def supports_transactions(self):
> """Confirm support for transactions."""
> with self.connection.cursor() as cursor:
> cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
> }}}
>
> We have been patch-fixing that file as follows:
>
> <- cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT PRIMARY
> KEY)")
> ---
> -> cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
New description:
When running unit tests, the internal testing of a db for using
transactions breaks when using MySQL Cluster server v8.0.28 because that
db version requires a primary key for tables, and this line in
https://github.com/django/django/blob/main/django/db/backends/base/features.py
breaks it :
{{{
@cached_property
def supports_transactions(self):
"""Confirm support for transactions."""
with self.connection.cursor() as cursor:
cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
}}}
We have been patch-fixing that file as follows:
<- cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT PRIMARY
KEY)")
---
-> cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT)")
--
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:1>
Django
------------------------------------+------------------------------------
Type: Bug | Status: new
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:2>
Django
------------------------------------+------------------------------------
Reporter: Aaron Blair | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+------------------------------------
Comment (by Can Huynh):
Can I try this issue?
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:3>
Django
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
Changes (by Aaron Blair):
* owner: nobody => Can Huynh
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:4>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Can Huynh):
Can I get an instruction on how to reproduce this issue?
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:5>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Aaron Blair):
Replying to [comment:5 Can Huynh]:
> Can I get an instruction on how to reproduce this issue?
Install MySQL Cluster server v8.0.28 onto multiple (say 3) servers and
start replication.
Install Django (any version that uses cursor.execute("CREATE TABLE
ROLLBACK_TEST (X INT)") in django.db.backends.base.features.py this exists
on 4.1 but it was there for several versions previous to 4.1 as well)
Create any model that uses autocreate id as pk (ie do not specify an id or
pk field)
Write any unit test bases on django.test.TestCase for the model which
accesses the db
Run the unit test
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:6>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
* cc: Pieter Cardillo Kwok (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:7>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Can Huynh):
Hi, I have attempted to install MySQL Cluster v8.0.28 but it doesn't seem
to be working. I may have done it incorrectly. I tried doing this locally
and on AWS but was not successful. Are you able to provide me
instructions/resources on how to install this so I can continue with this
ticket?
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:8>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
* Attachment "test_case.png" added.
--
Ticket URL: <https://code.djangoproject.com/ticket/34852>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Changes (by Can Huynh):
* Attachment "test_output.png" added.
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Can Huynh):
Replying to [comment:6 Aaron Blair]:
Can you guide me on how you set up your unit test and what your test
output looked like, if possible? I have attached my test suits and test
output, please let me know if there is anything I need to modify. Thank
you in advance and sorry for being inactive for a long time.
[[Image( test_case.png)]] [[Image(test_output.png)]]
> Replying to [comment:5 Can Huynh]:
> > Can I get an instruction on how to reproduce this issue?
>
> Install MySQL Cluster server v8.0.28 onto multiple (say 3) servers and
start replication.
> Install Django (any version that uses cursor.execute("CREATE TABLE
ROLLBACK_TEST (X INT)") in django.db.backends.base.features.py this exists
on 4.1 but it was there for several versions previous to 4.1 as well)
> Create any model that uses autocreate id as pk (ie do not specify an id
or pk field)
> Write any unit test bases on django.test.TestCase for the model which
accesses the db
> Run the unit test
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:9>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Aaron Blair):
Replying to [comment:9 Can Huynh]:
A very simple example:
my_app.models:
{{{
from django.db import models
class Company(models.Model):
name = models.CharField(max_length=30, unique=True)
}}}
my_app/tests/test_models_myapp.py:
{{{
from django.test import TestCase
from my_app.models import Company
class CompanyModelTest(TestCase):
def test_saving_and_retrieving_items(self):
company = Company()
company.name = "ACompany"
company.save()
companies = Company.objects.all()
self.assertEqual(companies.count(), 1)
first_company = companies[0]
self.assertEqual(first_company.name, "ACompany")
}}}
> Replying to [comment:6 Aaron Blair]:
>
> Can you guide me on how you set up your unit test and what your test
output looked like, if possible? I have attached my test suits and test
output, please let me know if there is anything I need to modify. Thank
you in advance and sorry for being inactive for a long time.
>
> [[Image( test_case.png)]] [[Image(test_output.png)]]
>
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:10>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Aaron Blair):
Replying to [comment:9 Can Huynh]:
Can you add your unittest_app.models.py?
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:11>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
* Attachment "unittest_app_models.png" added.
--
Ticket URL: <https://code.djangoproject.com/ticket/34852>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Can Huynh):
Here is my model file. I have tried applying your recommendation to my
test case, and I still cannot make the test fail. I have tried with the
most recent version, the 4.1 version, and the fix that I changed locally
ie: cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT PRIMARY KEY)")
I think that I didn't install and setup MySQL Cluster correctly but I am
not sure how to verify it
[[Image(unittest_app_models.png)]]
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:12>
Django
------------------------------------+-------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: assigned
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution:
Keywords: Test MySQL Cluster | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
------------------------------------+-------------------------------------
Comment (by Aaron Blair):
That model looks fine. As long as you have multiple mysql cluster servers
of version 8.0.28 and they have replication set up with each other, it
will break tests without the the fix. Remove the fix, and check that
replication is set up with the mysql cluster servers. (Log into one of
them add use the command - select * from
performance_schema.replication_group_members; - be sure all servers are
listed as ONLINE state and PRIMARY role).
Replying to [comment:12 Can Huynh]:
> Here is my model file. I have tried applying your recommendation to my
test case, and I still cannot make the test fail. I have tried with the
most recent version, the 4.1 version, and the fix that I changed locally
ie: cursor.execute("CREATE TABLE ROLLBACK_TEST (X INT PRIMARY KEY)")
>
> I think that I didn't install and setup MySQL Cluster correctly but I am
not sure how to verify it
>
> [[Image(unittest_app_models.png)]]
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:13>
Django
Reporter: Aaron Blair | Owner: Can Huynh
Component: Testing framework | Version: 4.1
Keywords: Test MySQL Cluster | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
------------------------------------+--------------------------------------
Changes (by Mariusz Felisiak):
* status: assigned => closed
* resolution: => invalid
* easy: 1 => 0
* stage: Accepted => Unreviewed
Comment:
The `django.db.backends.mysql` backend has its own implementation of
`DatabaseFeatures.supports_transactions` and doesn't use the default one.
I don't think there is anything to fix in Django itself.
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:14>
Django
------------------------------------+--------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: closed
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution: invalid
Keywords: Test MySQL Cluster | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+--------------------------------------
Thank you for the update!
Replying to [comment:14 Mariusz Felisiak]:
`DatabaseFeatures.supports_transactions` and doesn't use the default one.
I don't think there is anything to fix in Django itself.
--
Django
------------------------------------+--------------------------------------
Reporter: Aaron Blair | Owner: Can Huynh
Type: Bug | Status: closed
Component: Testing framework | Version: 4.1
Severity: Normal | Resolution: invalid
Keywords: Test MySQL Cluster | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------------+--------------------------------------
Comment (by Can Huynh):
@Mariusz Felisiak
--
Ticket URL: <https://code.djangoproject.com/ticket/34852#comment:16>