[Django] #27132: Test PyLibMCCache on Jenkins
Django
--------------------------------------+--------------------
Reporter: edmorley | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: master
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+--------------------
Currently jenkins only tests against MemcachedCache (ie python-memcached).
In order to make changes like those in #20892 easier to test, it would be
useful if PyLibMCCache were tested on jenkins too.
This will likely involve:
* Adding a PyLibMCCache config to the jenkins configs (not sure whether
these are in a public repo somewhere?)
* Refactoring tests/cache/tests.py, such that:
- `MemcachedCacheTests` is split into a base class (that isn't a
subclass of `TestCase`) plus `MemcachedCacheTests` and `PyLibMCCacheTests`
- the vast majority of tests are kept in the base class, with just the
binding-specific tests in the subclasses
- the subclasses have appropriate `@skipUnless()` and
`@override_settings()`
For example:
{{{
MemcachedCache_params = {}
PyLibMCCache_params = {}
for _cache_params in settings.CACHES.values():
backend = _cache_params['BACKEND']
if backend == 'django.core.cache.backends.memcached.MemcachedCache':
MemcachedCache = _cache_params
elif backend == 'django.core.cache.backends.memcached.PyLibMCCache':
PyLibMCCache_params = _cache_params
# ...
class BaseMemcachedTests(BaseCacheTests):
def test_foo(self):
# ...
@unittest.skipUnless(MemcachedCache_params, "MemcachedCache backend not
configured")
@override_settings(CACHES=caches_setting_for_tests(
base=MemcachedCache_params,
exclude=memcached_excluded_caches,
))
class MemcachedCacheTests(BaseMemcachedTests, TestCase):
def test_python_memcached__foo(self):
# ...
@unittest.skipUnless(PyLibMCCache_params, "PyLibMCCache backend not
configured")
@override_settings(CACHES=caches_setting_for_tests(
base=PyLibMCCache_params,
exclude=memcached_excluded_caches,
))
class PyLibMCCacheTests(BaseMemcachedTests, TestCase):
def test_pylibmc_foo(self):
# ...
}}}
However, there are both class level uses of `@override_settings()` and
also per-test uses, and the per-test uses are for tests that will be in
the base class `BaseMemcachedTests`.
What's the preferred way to inherit settings from the class-level
`@override_settings()` usage of the subclasses (eg `PyLibMCCacheTests`),
and to then modify them further on the tests run on the base class?
--
Ticket URL: <https://code.djangoproject.com/ticket/27132>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: edmorley | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I don't have much guidance about the settings overrides. Whatever works.
I'll take care of updating the Jenkins configuration. I'd guess it'll end
up looking something like this:
{{{
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
},
'memcached': {...},
'pylibmc': {...}
}
}}}
Ideally, we might have some feature flags on the classes so as not to
hardcode class paths like
"django.core.cache.backends.memcached.MemcachedCache". That may make the
test suite reusable for third-party subclasses.
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:1>
Django
Reporter: edmorley | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:2>
Django
Reporter: edmorley | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by edmorley):
> I don't have much guidance about the settings overrides. Whatever works.
Ah I've just seen that `override_settings` can be used as a context
manager too and not just as a decorator. I didn't realise that from
reading
https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.override_settings
(I didn't scroll up to see the similar entry for `modify_settings()`.
That's fine then, for the per-test overrides we can use it to retrieve the
backend specific settings from the base class at runtime (which wouldn't
have worked with the decorator).
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:3>
Django
Reporter: edmorley | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by edmorley):
Should I also add pylibmc to the py2.txt/py3.txt test requirements files?
(I'm not sure whether Jenkins uses those?)
Installation of pylibmc requires libmemcached, so we'll need that on the
Jenkins nodes too.
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:4>
Django
Reporter: edmorley | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Yes, I installed `libmemcached-dev` on Jenkins.
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:5>
Django
Reporter: edmorley | Owner: edmorley
Type: Cleanup/optimization | Status: assigned
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by edmorley):
* cc: emorley@… (added)
* status: new => assigned
* has_patch: 0 => 1
* owner: nobody => edmorley
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:6>
Django
Reporter: edmorley | Owner: edmorley
Type: Cleanup/optimization | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"5d978c46216df53884fbca590a9abe660a739774" 5d978c46]:
{{{
#!CommitTicketReference repository=""
revision="5d978c46216df53884fbca590a9abe660a739774"
Refs #27132 -- Added pylibmc to test requirements.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:7>
Django
Reporter: edmorley | Owner: edmorley
Component: Core (Cache system) | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"047c1d48a613cc2a16f078a9094cc799f06e6b0c" 047c1d4]:
{{{
#!CommitTicketReference repository=""
revision="047c1d48a613cc2a16f078a9094cc799f06e6b0c"
Fixed #27132 -- Allowed testing MemcachedCache and PyLibMCCache during the
same test run.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:8>
Django
--------------------------------------+------------------------------------
Reporter: edmorley | Owner: edmorley
Type: Cleanup/optimization | Status: closed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:9>
Django
--------------------------------------+------------------------------------
Reporter: edmorley | Owner: edmorley
Type: Cleanup/optimization | Status: closed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Ed Morley <emorley@…>):
In [changeset:"306545d80508a0d6d70c329667640e7b183c30f0" 306545d8]:
{{{
#!CommitTicketReference repository=""
revision="306545d80508a0d6d70c329667640e7b183c30f0"
[1.10.x] Fixed #27132 -- Allowed testing MemcachedCache and PyLibMCCache
during the same test run.
Backport of 047c1d48a613cc2a16f078a9094cc799f06e6b0c from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:11>
Django
--------------------------------------+------------------------------------
Reporter: edmorley | Owner: edmorley
Type: Cleanup/optimization | Status: closed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Ed Morley <emorley@…>):
In [changeset:"547c7e67569b2a0c323b8b64c1ce835206c23e3a" 547c7e6]:
{{{
#!CommitTicketReference repository=""
revision="547c7e67569b2a0c323b8b64c1ce835206c23e3a"
[1.10.x] Refs #27132 -- Added pylibmc to test requirements.
Backport of 5d978c46216df53884fbca590a9abe660a739774 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27132#comment:10>