[Django] #25471: Random test failures with parallel selenium tests
Django
-----------------------------------+--------------------
Reporter: MoritzS | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------
When running selenium tests in parallel I often get test failures that
seem to be some kind of race condition. However with `--parallel 1` these
errors don't occur and sometimes even when running in parallel all tests
pass.
I usually run the tests with `xvfb-runner`, however I also tested it
without xvfb and the errors still occur.
{{{
$ chromium --version
Chromium 45.0.2454.101
$ chromedriver --version
ChromeDriver 2.16
$ firefox --version
Mozilla Firefox 41.0
}}}
Here are some of the results of running the test suite multiple times with
`./runtests --selenium` (running on 8 cores):
{{{
======================================================================
ERROR: test_ForeignKey
(admin_widgets.tests.AdminRawIdWidgetSeleniumFirefoxTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element: {"method":"link text","selector":"Bogey Blues"}
Stacktrace:
at FirefoxDriver.prototype.findElementInternal_
(file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdr...@googlecode.com/components
/driver-component.js:10647)
at FirefoxDriver.prototype.findElement
(file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdr...@googlecode.com/components
/driver-component.js:10656)
at DelayedCommand.prototype.executeInternal_/h
(file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdr...@googlecode.com/components
/command-processor.js:12617)
at DelayedCommand.prototype.executeInternal_
(file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdr...@googlecode.com/components
/command-processor.js:12622)
at DelayedCommand.prototype.execute/<
(file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdr...@googlecode.com/components
/command-processor.js:12564)
======================================================================
ERROR: test_many_to_many
(admin_widgets.tests.AdminRawIdWidgetSeleniumFirefoxTests)
----------------------------------------------------------------------
selenium.common.exceptions.TimeoutException: Message:
======================================================================
ERROR: test_prepopulated_fields
(admin_views.tests.SeleniumAdminViewsChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: no such
element
(Session info: chrome=45.0.2454.101)
(Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)
======================================================================
FAIL: test_multiple_locales
(i18n.test_compilation.MultipleLocaleCompilationTests)
----------------------------------------------------------------------
AssertionError: False is not true
----------------------------------------------------------------------
Ran 10308 tests in 61.830s
}}}
{{{
======================================================================
ERROR: test_calendar_nonday_class
(admin_widgets.tests.DateTimePickerSeleniumChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: no such
element
(Session info: chrome=45.0.2454.101)
(Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)
======================================================================
FAIL: test_add_inlines (admin_inlines.tests.SeleniumChromeTests)
----------------------------------------------------------------------
AssertionError: 0 != 1
----------------------------------------------------------------------
Ran 10308 tests in 62.124s
}}}
{{{
======================================================================
ERROR: test_filter
(admin_widgets.tests.HorizontalVerticalFilterSeleniumChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.StaleElementReferenceException: Message: stale
element reference: element is not attached to the page document
(Session info: chrome=45.0.2454.101)
(Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)
----------------------------------------------------------------------
Ran 10308 tests in 65.913s
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/25471>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: MoritzS | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by aaugustin):
* needs_better_patch: => 0
* stage: Unreviewed => Accepted
* needs_tests: => 0
* needs_docs: => 0
Comment:
I don't know if Selenium is supposed to handle parallel execution of
multiple processes.
The easiest solution is probably to serialize execution of these tests
with SerializeMixin.
--
Ticket URL: <https://code.djangoproject.com/ticket/25471#comment:1>
Django
Reporter: MoritzS | Owner: nobody
Type: Bug | Status: closed
Component: Testing framework | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by MoritzS):
* status: new => closed
* resolution: => fixed
Comment:
I can't reproduce the problem any more. I assume it was fixed by #26048.
--
Ticket URL: <https://code.djangoproject.com/ticket/25471#comment:2>