[Django] #27301: Better handling of errors that are not pickleable when testing in parallel
Django
----------------------------------+----------------------------------------
Reporter: Adam Wróbel | Owner: Tim Graham
Type: Uncategorized | Status: assigned
Component: Testing | Version: 1.10
framework |
Severity: Normal | Keywords: testrunner parallel pickle
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+----------------------------------------
When running test in parallel the tests results are communicated between
processes in serialized form. Unfortunately not all test errors are
serializable using `pickle` module.
Django currently detects errors that fail `pickle.dumps`, reports them to
`STDERR` and raises exception that interrupts test runner.
This ticket has two parts:
The first is a minor enhancement. In addition to detecting `pickle.dumps`
errors also detect `pickle.loads` errors. I've implemented it in a
[https://github.com/django/django/pull/7325 Github PR #7325].
The second is a proposal to instead of raising pickling exception that
interrupts tests and displays useless
`multiprocessing.pool.RemoteTraceback` wrap the error type and message in
some kind of `UnpickleableError` container and pass it to parent process
with the usual `addError`/`addFailure`/`addSubTest`/`addExpectedFailure`
methods. This will let the test suite continue, display final error
statistics and perform teardown. This is especially important for
`addExpectedFailure` which currently stops the entire suite if the
expected error is unpickleable (my actual use case).
--
Ticket URL: <https://code.djangoproject.com/ticket/27301>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage:
pickle | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* status: assigned => new
* needs_better_patch: => 0
* needs_tests: => 0
* owner: Tim Graham => (none)
* needs_docs: => 0
* type: Uncategorized => Cleanup/optimization
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:1>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:2>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Chris Jerdonek):
* cc: chris.jerdonek@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:3>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
In the PR, you say--
> One example of error like that is botocore.exceptions.ClientError which
pickles as "ClientError(message)", but its initializer is actually
ClientError(response, operation_name).
In addition to creating this ticket, have you also tried reporting this to
the botocore project?
> This ticket has two parts:
This sounds like it should be taken care of by two separate PR's (and
indeed you created one PR to solve the first part). Does this mean you
should have two separate tickets? This way the two issues can be discussed
in isolation without confusing the two.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:4>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Wróbel):
Replying to [comment:4 Chris Jerdonek]:
> In the PR, you say--
>
> > One example of error like that is botocore.exceptions.ClientError
which pickles as "ClientError(message)", but its initializer is actually
ClientError(response, operation_name).
>
> In addition to creating this ticket, have you also tried reporting this
to the botocore project?
I've started writing a PR for botocore, but consider it a lower priority:
we can try to fix thousands of unpickleable errors, but with a robust test
runner it might not matter at all.
> > This ticket has two parts:
>
> This sounds like it should be taken care of by two separate PR's (and
indeed you created one PR to solve the first part). Does this mean you
should have two separate tickets? This way the two issues can be discussed
in isolation without confusing the two.
Indeed I could try to write a second PR if the second part of this ticket
was accepted.
As for two tickets, I was really hoping to have that minor adjustment from
first PR merged without any ticket at all. That one-direction-only check
seems like a trivial omission. I think @aaugustin would agree.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:5>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
> I've started writing a PR for botocore, but consider it a lower
priority:
Can you post a link to the issue?
> we can try to fix thousands of unpickleable errors, but with a robust
test runner it might not matter at all.
Thousands seems like an exaggeration to me. A dozen seems like a more
reasonable upper bound.
In any case, if a particular exception isn't picklable, it seems like that
should be reported as a bug each time we notice it -- regardless of the
state of Django's test runner, and regardless of the number of instances
we notice. Also, as long as Django relies on the picklability of
exceptions for the nicest form of error messages, it seems we should want
those exceptions to be picklable.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:6>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Wróbel):
Replying to [comment:6 Chris Jerdonek]:
> > I've started writing a PR for botocore, but consider it a lower
priority:
>
> Can you post a link to the issue?
Well, I said I started writing PR, not that I posted it. But hey, here's
me hoping to please you:
https://github.com/boto/botocore/pull/1044
In either case botocore issue shouldn't in any way delay our progress
here. They might fix it or not, but we should be able to handle problems
like theirs.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:7>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Adam Wróbel):
I am beginning to think that making `botocore` errors picklable is a lost
cause. They support python 2.6 and 2.7 which is even worse at pickling (my
current botocore PR only passes on 3.X). They also inherit exceptions from
`requests` package , similarly rewriting the `__init__` signature and
affecting picklability (I've accidentally omitted that one error class in
my PR, but now I don't know how to tackle it).
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:8>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Accepted
pickle |
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
Thanks. Re: botocore, it would seem reasonable to me for them to accept a
PR that fixes it only for 3.x and/or future versions of botocore if fixing
it for 2.x is too difficult and/or would break backwards compatibility.
Also, on a related note, if it turns out that picklability of exceptions
is too much to expect in general, then I think we should consider not
having Django's test runner error out if the exception isn't picklable.
But that can be handled in the "part 2" ticket when that is created.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:9>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
pickle | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* has_patch: 0 => 1
* stage: Accepted => Ready for checkin
Comment:
[https://github.com/django/django/pull/7325 PR] looks good to me. Wouldn't
mind a +1 from someone else.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:10>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
I can take a look, but it would be later today.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:11>
Django
Reporter: Adam Wróbel | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution:
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Chris Jerdonek):
I added three comments to the PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:12>
Django
Reporter: Adam Wróbel | Owner: Tim
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* owner: (none) => Tim Graham <timograham@…>
* resolution: => fixed
Comment:
In [changeset:"52188a5ca6bafea0a66f17baacb315d61c7b99cd" 52188a5]:
{{{
#!CommitTicketReference repository=""
revision="52188a5ca6bafea0a66f17baacb315d61c7b99cd"
Fixed #27301 -- Prevented exceptions that fail unpickling from crashing
the parallel test runner.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:13>
Django
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution: fixed
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Comment (by nessita <124304+nessita@…>):
In [changeset:"548209e620b3ca34396a360453f07c8dbb8aa6c7" 548209e]:
{{{#!CommitTicketReference repository=""
revision="548209e620b3ca34396a360453f07c8dbb8aa6c7"
Made RemoteTestResultTest.test_pickle_errors_detection() compatible with
tblib 3.2+.
tblib 3.2+ makes exception subclasses with __init__() and the default
__reduce__() picklable. This broke the test for
RemoteTestResult._confirm_picklable(), which expects a specific
exception to fail unpickling.
https://github.com/ionelmc/python-
tblib/blob/master/CHANGELOG.rst#320-2025-10-21
This fix defines ExceptionThatFailsUnpickling.__reduce__() in a way
that pickle.dumps(obj) succeeds, but pickle.loads(pickle.dumps(obj))
raises TypeError.
Refs #27301. This preserves the intent of the regression test from
52188a5ca6bafea0a66f17baacb315d61c7b99cd without skipping it.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:14>
Django
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution: fixed
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Comment (by Natalia <124304+nessita@…>):
In [changeset:"500c5d325f6188f3dfeaab58e7b83d77bd4441f6" 500c5d3]:
{{{#!CommitTicketReference repository=""
revision="500c5d325f6188f3dfeaab58e7b83d77bd4441f6"
[6.0.x] Made RemoteTestResultTest.test_pickle_errors_detection()
tblib 3.2+ makes exception subclasses with __init__() and the default
__reduce__() picklable. This broke the test for
RemoteTestResult._confirm_picklable(), which expects a specific
exception to fail unpickling.
https://github.com/ionelmc/python-
tblib/blob/master/CHANGELOG.rst#320-2025-10-21
This fix defines ExceptionThatFailsUnpickling.__reduce__() in a way
that pickle.dumps(obj) succeeds, but pickle.loads(pickle.dumps(obj))
raises TypeError.
Refs #27301. This preserves the intent of the regression test from
52188a5ca6bafea0a66f17baacb315d61c7b99cd without skipping it.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/27301#comment:15>
Django
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution: fixed
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Comment (by Natalia <124304+nessita@…>):
{{{#!CommitTicketReference repository=""
revision="9b37bd5fe73b0d614ad8e503071d6f7bf2bdf6b2"
[5.2.x] Made RemoteTestResultTest.test_pickle_errors_detection()
tblib 3.2+ makes exception subclasses with __init__() and the default
__reduce__() picklable. This broke the test for
RemoteTestResult._confirm_picklable(), which expects a specific
exception to fail unpickling.
https://github.com/ionelmc/python-
tblib/blob/master/CHANGELOG.rst#320-2025-10-21
This fix defines ExceptionThatFailsUnpickling.__reduce__() in a way
that pickle.dumps(obj) succeeds, but pickle.loads(pickle.dumps(obj))
raises TypeError.
Refs #27301. This preserves the intent of the regression test from
52188a5ca6bafea0a66f17baacb315d61c7b99cd without skipping it.
Backport of 548209e620b3ca34396a360453f07c8dbb8aa6c7 from main.
}}}
--
Django
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution: fixed
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Comment (by Natalia <124304+nessita@…>):
{{{#!CommitTicketReference repository=""
revision="c361494cbb134540f2f35a6364ea9259f0084a58"
[5.1.x] Made RemoteTestResultTest.test_pickle_errors_detection()
tblib 3.2+ makes exception subclasses with __init__() and the default
__reduce__() picklable. This broke the test for
RemoteTestResult._confirm_picklable(), which expects a specific
exception to fail unpickling.
https://github.com/ionelmc/python-
tblib/blob/master/CHANGELOG.rst#320-2025-10-21
This fix defines ExceptionThatFailsUnpickling.__reduce__() in a way
that pickle.dumps(obj) succeeds, but pickle.loads(pickle.dumps(obj))
raises TypeError.
Refs #27301. This preserves the intent of the regression test from
52188a5ca6bafea0a66f17baacb315d61c7b99cd without skipping it.
Backport of 548209e620b3ca34396a360453f07c8dbb8aa6c7 from main.
}}}
--
Django
Cleanup/optimization | Status: closed
Component: Testing framework | Version: 1.10
Severity: Normal | Resolution: fixed
Keywords: testrunner parallel | Triage Stage: Ready for
pickle | checkin
Has patch: 1 | Needs documentation: 0
Comment (by Natalia <124304+nessita@…>):
{{{#!CommitTicketReference repository=""
revision="cd3b21bcaccf03a1dc49512bdb8efd796fba8100"
[4.2.x] Made RemoteTestResultTest.test_pickle_errors_detection()
tblib 3.2+ makes exception subclasses with __init__() and the default
__reduce__() picklable. This broke the test for
RemoteTestResult._confirm_picklable(), which expects a specific
exception to fail unpickling.
https://github.com/ionelmc/python-
tblib/blob/master/CHANGELOG.rst#320-2025-10-21
This fix defines ExceptionThatFailsUnpickling.__reduce__() in a way
that pickle.dumps(obj) succeeds, but pickle.loads(pickle.dumps(obj))
raises TypeError.
Refs #27301. This preserves the intent of the regression test from
52188a5ca6bafea0a66f17baacb315d61c7b99cd without skipping it.
Backport of 548209e620b3ca34396a360453f07c8dbb8aa6c7 from main.
}}}
--