[Django] #29023: running tests in parallel doesn't show exception chain (even with tblib)

[Django]

#29023: running tests in parallel doesn't show exception chain (even with tblib)
-------------------------------------+-------------------------------------
Reporter: Chris | Owner: nobody
Jerdonek |
Type: | Status: new
Cleanup/optimization |
Component: Testing | Version: master
framework | Keywords:
Severity: Normal | parallel,exception,chain,traceback,pickling,tblib
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Running tests in parallel doesn't show more than the first link of an
exception chain. For example, with the following test (with `tblib`
installed):

{{{#!python
def test(self):
time.sleep(2)
try:
raise ValueError('foo')
except Exception:
raise KeyError('bar')
}}}

this is what the failure looks like when running tests in parallel:

{{{
Traceback (most recent call last):
File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line
59, in testPartExecutor
yield
File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line
605, in run
testMethod()
File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
raise KeyError('bar')
KeyError: 'bar'
}}}

In contrast, this is what it looks like when running them not in parallel:

{{{
Traceback (most recent call last):
File "/Users/.../myproject/myproject/tests/test_a.py", line 11, in test2
raise ValueError('foo')
ValueError: foo

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
raise KeyError('bar')
KeyError: 'bar'
}}}

Side question: why must pickling be used at all? Why can't we simply
render the traceback to a string using Python's `traceback` module, and
then send the string back? It seems like that would result in simpler,
more understandable code and sidestep many of the difficulties.

(This was reported using Python 3.6.4. and `master` as of Jan. 15, 2018:
02365d3f38a64a5c2f3e932f23925a381d5bb151 .)

--
Ticket URL: <https://code.djangoproject.com/ticket/29023>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#29023: running tests in parallel doesn't show exception chain (even with tblib)
-------------------------------------+-------------------------------------

Reporter: Chris Jerdonek | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage:
parallel,exception,chain,traceback,pickling,tblib| Unreviewed
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Chris Jerdonek):

* cc: Chris Jerdonek (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:1>

[Django]

#29023: running tests in parallel doesn't show exception chain (even with tblib)
-------------------------------------+-------------------------------------

Reporter: Chris Jerdonek | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage:
parallel,exception,chain,traceback,pickling,tblib| Unreviewed

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Chris Jerdonek):

One more thing to add: in the case where the full chain isn't being shown,
there's also no way for the user to know that anything is missing (without
also running in non-parallel mode and comparing the test outputs).

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:2>

[Django]

#29023: running tests in parallel doesn't show exception chain (even with tblib)
-------------------------------------+-------------------------------------

Reporter: Chris Jerdonek | Owner: nobody
Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage:
parallel,exception,chain,traceback,pickling,tblib| Unreviewed

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

> Side question: why must pickling be used at all? Why can't we simply
render the traceback to a string using Python's traceback module, and then
send the string back? It seems like that would result in simpler, more
understandable code and sidestep many of the difficulties.

From what I know It's just the default behavior of the `multiprocessing`
module for inter process exception bubbling. I feel like it's a bit
harder than that as you still have to construct a `traceback` object from
the string on the parent process side to raise the appropriate exception.

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:3>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib

-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: master
Severity: Normal | Resolution:

Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|
Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Tim Graham):

* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:4>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Chris Jerdonek):

For future reference, here is one possible work-around that I think would
be better than the status quo. If an exception raised during a test either
(1) fails the "pickleable test," or (2) has a chain of length bigger than
one (e.g. non-`None` `__cause__` attribute), then the traceback could be
formatted to a string in the child process and a pickleable exception of
the form `UnpickleableTestException(traceback)` could be sent back to the
parent process. The end result won't look as clean as the pickled case,
but at least the information of the full exception chain should display.
Also, in the case of unpickleable exceptions, it should prevent the test
run from aborting completely. So it would address two issues.

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:5>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Chris Jerdonek):

FYI, I implemented a proof-of-concept for the idea I described in my
previous comment and posted it here:
https://github.com/cjerdonek/django/tree/ticket_29023

In particular, it handles both chained exceptions and non-picklable
exceptions, and without aborting the test run in the latter case.

I included example "before" and "after" outputs for both cases of chained
exceptions and non-picklable exceptions in the commit message of this
commit:
https://github.com/cjerdonek/django/commit/0040040ca4fec08a9980a24f08bcad900e2b0f22

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:6>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib
-------------------------------------+-------------------------------------

Reporter: Chris Jerdonek | Owner: Ahmad
Type: | Abdallah
Cleanup/optimization | Status: assigned

Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Ahmad Abdallah):

* owner: nobody => Ahmad Abdallah
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:7>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib
-------------------------------------+-------------------------------------

Reporter: Chris Jerdonek | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: dev

Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sage Abdullah):

* cc: Sage Abdullah (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:7>

[Django]

#29023: running tests in parallel doesn't show exception chain, even with tblib
-------------------------------------+-------------------------------------
Reporter: Chris Jerdonek | Owner: nobody

Type: | Status: new
Cleanup/optimization |

Component: Testing framework | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
parallel,exception,chain,traceback,pickling,tblib|

Has patch: 0 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Ülgen Sarıkavak):

* cc: Ülgen Sarıkavak (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/29023#comment:8>