Tests failing randomly

421 views

Skip to first unread message

em

unread,

Jul 8, 2022, 5:04:45 AM7/8/22

to Dedalus Users

I've installed Dedalus 3 on two servers and on each one I cannot get the tests( python3 -m dedalus test ) to run and produce the same result each time. Each run produces a different series of failures, or sometimes they all pass. When there are errors the test process does not finish - it hangs after a few thousand tests(different each time).

Dedalus was installed using Conda.

The machines are Dell servers with Xeon processors running Ubuntu 21.10. They both have over 100GB RAM.

Tests were performed when the servers were idle.

Please let me know if there is anything I should be looking at. This has me perplexed and out of ideas!

Thanks.

em

unread,

Jul 27, 2022, 4:16:24 AM7/27/22

to Dedalus Users

I am still unable to get Dedalus to pass tests reliably. I also tried it on my Linux (Ubuntu 22.04) laptop and got similar results same though with a higher success rate.

We have a researcher who wishes to use Dedalus but I can't recommend it until we can get the tests to pass. Can anybody offer any suggestions? Do the tests always work 100% every time you run them?

Any assistance would be greatly appreciated!

Keaton Burns

unread,

Jul 27, 2022, 7:57:53 AM7/27/22

to dedalu...@googlegroups.com

Hello,

Yes the tests should pass deterministically. They run in about 2 minutes on my mac laptop, ubuntu desktop, and all the clusters I have access to. Are you actually seeing test failures marked “f", or just the “x” and “s” marks on the pytest output for xfail/skipped tests? If you see failures, can you attach a log that shows some of them? I can’t recommend anything until we see what the actual errors are.

Best,

-Keaton

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/dd7a0a20-936a-4dc5-8cc0-1f3434dc780fn%40googlegroups.com.

em

unread,

Jul 27, 2022, 9:11:05 AM7/27/22

to Dedalus Users

The failures are marked with an 'E'.

When there are failures the test process always hangs so no log of errors can be obtained. I always stop it with ctrl-z and then kill it. Below you can see two subsequent runs, each showing a different pattern of errors. Each was stopped after the process hung(no progress for over 30 seconds).

myself@edmonton:~$ . /usr/local/dedalus/init
(dedalus3) myself@edmonton:~$ python3 -m dedalus test
============================================================================================ test session starts ============================================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests
plugins: cov-3.0.0, parallel-0.1.1, benchmark-3.4.1
collected 7034 items / 1618 deselected / 5416 selected
pytest-parallel: 64 workers (processes), 1 test per worker (thread)
......................................................................................................................................................................................................................................................E..........................E........................EE......................E......E....EEEEEEEE....................EE.........................................E....E.......................................................................................E.................E........................................E........E.....................................................E...........................E.....................................................................E......................................................................................................................................E................................................................................................................................................E...........................................E.......................................E.........................E...............E...............................E............E.............................................................E..E................E..........E..............E...............E..........E.....E...E.E..E..................................................................E..........EE...................E........E.......................E..................E..................E.....E...............E...............................E.........E..........................E......E.............E.E..................................E....E........xE............................................E....E....EE..E..Exx..xxExxx..x..x..E....EE..E....xx....E....EE..EE......EE..E..........xxE..........EE......E....ExxXX..XXEE..EExx......Ex....xx..xx....xx..EEEE..E......E........EE......EEEE..EEEE..EEE..EEEEEEE..E......EE......EEEEEE...E...E....EE..E..E....EE..E....E..E....E..E....E....E..EE..........E.............E.......E.......E...xx.x.........xxx..x..xx..E........Exxxx.x.xExxExxxx..xxxxExxxxxxxx..EE..E..E........xx..EE..........E..xxE..........E......xxE....E..xx........E......E....E.......E.........xx..E..........E........E........E........E....E..E....E....E....Exx..E..E......E..........xxE..E........E........E....E....E..........E..E..........E..E.......................E.E..............EE..........EE....EE....E........E..........EE........EE................E..EXX......E..........E........XX................E..E..................EE........................E.....E.................E....E..............E....................E........E............................................................................................E.....................................................................E......................E.......E............................E..E........E..E..................E....................EE....E..........E..E........E....E........E....E..E......E..E........E......E......E................E....E..E..........E....E........E....E........E....EE........E....E......E..E....E..E........EE........EE..E......E......E....EE..........E......EE....EE......E..E....E........E....E....E..........E....E....E....E..E....E....E........E......E....E......E..E..................................E..E..............E....E........EE................EE............EE..............EE........E..E............E....E......E......E........E........E..............E..................E................E..........E..E....EEE....E....E..E....EE....E......E..........E....E..EE......E......EE..E....E..EE....E....E....E..........EE....EE......E..E......EE......E..EEE......EE.....E.......E..E......EE....E....E....E..E..E..E....EE....E..E....EE..EE......EE......E....EEE..E..E..EE....EE........EE...............E.E....E..E..EE......E....E......E..E......EE....EE......E..........E......EE....E..E..........E..E......E..EE..EE........E..........xxxxxxxxxxxxx..xx........E............Exx..xx..xxxxxEEExx..xxE..E....EE....EE................EE............EE......EE..............E..E......EE........E......E..E......E......E..E..E..E..EE..EEEE....EE......E..E....E..EEE..EE......E........E......E..ssssssssssssssssss.s.ssssss..E..E....EE......EE..EE..EE......EE....EE....E....E......E........E......E..E..............E....E................E......E....E..E......E......E..............E..E....E..........E..........EE..................EE..............EE......................EE................E......E..................E................................E...................................................E..............................................................E.............E..........................................EE..........E....E......E...E.......................E..........E................E..E........E..........E....................EE....................EE..................EE........E......E.............E.......................E....E..E......E....E......E..E......E........E..........E....E............EE..EE........E.E.....E..........EE....E..........E......E....EE..E..........EE....E..E....E..EE......E....EE..E..E..E..EE....E..EE..E....E..................EEE......E..EE....E..E..E..E....E....E....E....E....E......E....EEEE....E..........E....E..E............E....E....E..E....EE..EE.......E.EEE..EEEE......EEE....EE..........E..EEEE......E..E........EEEE....EEEE....EE....EE..E......E....EEE..EEEEE...E.......E..E..EE....EE..E..EE......EEE....E......E....EE..EE....EE....E..E..EE....EE......E.E...EEE..E..EE.........E...EE........E....E..E..E..E..EEE..E..E......EE...........E.E..E........E......E..E....EEEEEE....EEEEEEE..E....EEE..E..E....E............EE..........................EE........................EE............E........................EE..................................E.............E......................................E............................................................................E..............................................................E..........................E...................................................................E....................................E............E.E.E..................E.........E.....................................................................................................................E...E....................E.......................E................EE...EE.........E..E......................EE..............EE........E..E..........EE..EE..............E....E..........E..........E....................E.................E...................E..........E........EE....E..E........E..E....................E............E....................E..................................EE............................................E.........E.......................................................E............................................................................................E......................................................................E................................................................................E......E......................................E.........E...................E.....E...E................................E....E..................E......E......E......E........E........E......E......E..E............E..................E..............E....E............EE....EE........E................EE......E..EE..E....E......E..E....E......E........E..E..........E...............................E.........................E......E......................E........E..................E......E..........................E............E..................................E........................................E....................E..E............EE........E....E........E........E..........E........E....................E..................E..........................E....E...............................E.......E......................................E......................................E.E...............................E......E..................E........E..................E....................................E......................E..E..E..E............E....E......................E................E.............................E.............................E........................E.................E.........................E..................................EE..........................................E.E................................EE............................................EE...........................................E..........................E....................E........................E..E..........................E....................................E..............E........E......E..................................E............E..E......................E..........E..............EE....E....E..............E..E......E....E..E..E........E..................EE......EE....................EE....EE...............E.............E......E..E..........EE........EE..................E..E........E..E............E..................E........................EE......................E.E.............E....................EE..................E........E..........E........E............................E........................E..........E................E......................E....E..........................E..E..............E......E..EE..EE........E..........E................E..................E..........E.............................E.E....................EE......EE....E...............E.......EE..........E....E..................................................E........E....E..........E....E....E....EEEEEEEEEEEEEE..E..E......E..E..........E....E..........E..E............E..E..........EE..E..E....EE..........E........E......E............................................................................................................^Z
[1]+ Stopped python3 -m dedalus test
(dedalus3) myself@edmonton:~$ kill %1

[1]+ Stopped python3 -m dedalus test
(dedalus3) myself@edmonton:~$
[1]+ Terminated python3 -m dedalus test
(dedalus3) myself@edmonton:~$ python3 -m dedalus test^C
(dedalus3) myself@edmonton:~$ ps -ef|grep -i dedalus
myself 2830225 2825991 0 07:48 pts/2 00:00:00 grep -i dedalus
(dedalus3) myself@edmonton:~$ python3 -m dedalus test
======================================== test session starts ========================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests
plugins: cov-3.0.0, parallel-0.1.1, benchmark-3.4.1
collected 7034 items / 1618 deselected / 5416 selected
pytest-parallel: 64 workers (processes), 1 test per worker (thread)
....................................................................................................................................................................................................................................................................................................................................E....E..EEEEE..E.......E...........E..E....................................E...............E...................................E............................................................................E......E............................................E.............E...............................E............E..................................E..................................................E.............................................................E.E............................................................................................................................................................................................................................................E..........................................................E.........................................E...............E................E...........................E............................................E..E............E..........E................E.........E...........E.......E.....E...E.E.E........E....................E............................................E..E...........E...........E........E....................................E......E........................E........E..............E................E..........................E......E..................E......E..........E................E..........................E..E..E......E........................................EEEE....xxxExxxxxx..xxxE....EEE..x....x..x..E....E....E....EE......EE......E..EEE..................E..xx.E.EXXE....E....Exx....x..x..xxx..xx..xEE......E..E..EE....EE....EE..EEEE..EEEEEEEE..EEE..EEE....EE..E..EEE........EEEEE........EEE..EE......EE....E........E..............E....xx..E..........E..E................E....E..........x..xxx......xx..xx..x..x..........EExxxxxxxxxxxxE..xxExx..xxxxE..xxE..........E....E....E........Exx....xx..Exx........E..E..........E........E..........E......E..............E......E..............E..E......E....E......E......E....EEEE..xxEE............E..xx..............E..EE..................E..E....E......E..........E....E......................EE..........E..E........EE......E....E..E....E......E......XXE.................E.E..........XX....E..............E..E....................E..E.......................E.E..............................E....E................E..........E........E..................E..............E................................E..................................................E........................................E..............E...............................E.....................................E..............E....E..............E....E..E..E............E....E......E........E......E......................EE............EE....EE......EE..................E........E..........EE..........E....E..E......E........E..........E......E....E..E..EE..EE..E..E....E........E......E..........E....E....E....EE....E........E........EE......E....EE......E..E..E....E..E......E......E......E......E........E..........E..................E......EE............................EE..........E....E........EE........EE......EE............EE..............E..........E........E....E..........E............E...E.......E..................E......................E........EE......EE......EE........E..E......E....E.........E.....E........E..E....E..E..EE....E....E..E......E........E....EE..........EE..E..E....E..E..E..E....EE....EE......EE........EE........E........E..EE..EE........EEEE....EE......EE........E..E..EE..EE........EE..E....E........E..E........E....E......E............EE....E..EE....E..........E....EE....EE..EE..............EE................EE........E..E....EE..EE....E..E........xxxxxxxxxx..xxxxxx..xx....xE....E....xx..E......xx....ExxE..xxE..E....E..E....E..............E..........E..........E....EE....E....E......E....E....E....EE......E......EEE......E..E........E..EE..E..EE..........E..E..E..E..EEEEE...E...E..E......E....E....ssssssssssssssssEsssssssssssss..................E..EEEEE....E..E..E....E....E....E....E........E....E............E....E........E..E..........E..E..............E..........EE..................E............E..........E......E..............EE................EE..................E........E..........E........E........E...............E...E..................................E.......E.......................................................E...........................................................E...........................E......E..........E....E..........E..E......................E..............E......................E..........................E..........E................E..E....E..E.................E.E....EE............EE..................EE................E.....E.....E............E..E......E........E....E........E.........E.......E..EE........EE......EE............EE....EE....E..E........E............E..E....E..E..E........EE..EE....EE..E..E..EE..EE........EE..E..E........EE....EE..............EE......E..E..E..E....EE........E......E..E..E........E........EEE....E....EE..E..........E............E..E..EE..E..E....EE..EE..EEEE..EE..........EE....E....EE..E......E....E..E....E....E..E....EEEE..EE..EE........EEEE......E..E..E..E....E..E....E..E....E....E..EEEE....EE......EE..E..E..E..E..E.E...E..EE....E..E..E..EE..EEE............E......EE....EE..E..E..E..EEE................E........EE..E....E..E......E..E........E..........EEE......EE..EEEE......EE....EEEEEE..........E....E......................E..............................E........EE........E..E......................E............E...........................E...............E................................E.......................................................E.............E............................................E...............E.................................................E......................................................................E..............E....EEE..E.................................................................E.............................E...........................................................................................E.....E..................................E.EEE........E...E........E......E................EE....E....E......E......E....E..E..E..............E..E......E......E................E..........................EE..................EE......E....E..............E..E........EE..............................EE...........................E.E....................................E............................................E......................................E..........................E......................................E...................................................E................................E.........................................................................................E....................E..............................................E....................E..E....E........E............E............E..........E..........E........EE..............EE..........................E......E....E......E..................E......E......E......E............E..E..E............E............EE......EE......E.E.......EE..E..E..............E..E................E................E........E..........................E..E..................................E........E.............E.......E....................................E..................................E......E..........................................................E............EE..........E..E......E..E................E......E......E............E..................E............................E................EE.......................................E.E...........................................................E...E........................................EE.............................E.E..........................E....E....................................E..................E..EE............................E....E............................E............E..............E.............E.................E..................................................................................E................E....................................E............E......................E................................E......................E............................................E..E....................................E..E..............................E........................E................E.............E.......................EE..........E................E..................................EE........E......................................E......E..E............EE........E..E....EE....EE......................E..........E..............E............E..E....E....E..............EE......EE......EE......E..E..........E..........E..............E....E................E...E...............................E........................E......E................................E..E......................E......E......E........E............................E......................E................E..........E........E.........E...........................................E..................EE......E..EEE..EE............E.........................E...........E........E......................E....................E............E....E..E......E..E.........E...E................E....E.....................E.............E..............E..........E..E..........E....E......E..E......EEEEE..EE....EE..EEEE..E..........EE........EE....EE........E.....E.........E..E........E........EE................E.E...................................................................................................................^Z
[1]+ Stopped python3 -m dedalus test
(dedalus3) myself@edmonton:~$ kill %1

[1]+ Stopped python3 -m dedalus test

Jeffrey S. Oishi

unread,

Jul 27, 2022, 9:19:20 AM7/27/22

to dedalus-users

Hi,

How odd! Those are not test failures, they are errors. Could you try hitting ctrl-C as soon as the first E pops up? That might trigger the error to print immediately.

Like Keaton, I am unable to reproduce this behavior.

Thanks,

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/5b20a179-55b9-4b27-9e76-fcf5ab75ef62n%40googlegroups.com.

Keaton Burns

unread,

Jul 27, 2022, 9:29:39 AM7/27/22

to dedalu...@googlegroups.com

It may also help to run a subset of tests directly using pytest rather than through the dedalus command line interface. For instance, you can cd to the test folder (something like …miniforge3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests) and then just do pytest test_lbvp.py for instance, to serially run a small number of tests, and hopefully get a more useful output at the end.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CAAOZTN8_GvvPMg-4HeDOPY-rRvZ2Oh0%3DshD2zCgz5Yg6NP6SHg%40mail.gmail.com.

em

unread,

Jul 27, 2022, 10:16:12 AM7/27/22

to Dedalus Users

When subsets of tests are run they pass without error(warnings though). Here is a partial run of the full test suite interrupted by CTRL-C:

(dedalus3) myself@edmonton:~$ python3 -m dedalus test
======================================== test session starts ========================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests
plugins: cov-3.0.0, parallel-0.1.1, benchmark-3.4.1
collected 7034 items / 1618 deselected / 5416 selected
pytest-parallel: 64 workers (processes), 1 test per worker (thread)

...................................................................................................................................................................................................................................................E..................E.........................E.......E.........E....................E..E..EEEE.E.....^CE2022-07-27 08:46:15,712 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,714 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.0e+01/s
2022-07-27 08:46:15,715 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
............E........EE.......................................................................................................................................................
============================================== ERRORS ===============================================
__________________ ERROR at setup of test_trace_explicit[g-float64-1-16-build_FC] ___________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5db2e0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
. lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
. return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
._ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_trace_explicit[g-float64-1-16-build_FC]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_________________________ ERROR at setup of test_clenshaw_vector[2-1-1-0-8] _________________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55f010>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
.
. @classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_clenshaw_vector[2-1-1-0-8]>

. def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

./usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_______________ ERROR at setup of test_waves_1d_first_order[ChebyshevT-32-complex128] _______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5dadd0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
. lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
./usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
. return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_waves_1d_first_order[ChebyshevT-32-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_________________ ERROR at setup of test_transpose_implicit[float64-1-16-build_FFF] _________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5db1c0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
. lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
. return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_transpose_implicit[float64-1-16-build_FFF]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_____________ ERROR at setup of test_ball_bessel_eigenfunction[1-10-3-31-3-complex128] ______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5db130>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
Process Process-37:
.Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
.. return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_bessel_eigenfunction[1-10-3-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,345 subsystems 0/1 INFO :: Building subproblem matrices 1/10 (~10%) Elapsed: 0s, Remaining: 1s, Rate: 1.6e+01/s
_____________ ERROR at setup of test_ball_bessel_eigenfunction[1.5-0-2-31-3-complex128] _____________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb562ef0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
. return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_bessel_eigenfunction[1.5-0-2-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_____________ ERROR at setup of test_ball_bessel_eigenfunction[1.5-5-0-31-3-complex128] _____________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55ea70>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
. """Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
. Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
. lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
./usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
. item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_bessel_eigenfunction[1.5-5-0-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_______________ ERROR at setup of test_ball_diffusion[no-slip-1.5-1-31-3-complex128] ________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5dadd0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
. return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[no-slip-1.5-1-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
_______________ ERROR at setup of test_ball_diffusion[no-slip-1.5-3-31-3-complex128] ________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55ff40>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
. def from_call(
. cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
. return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[no-slip-1.5-3-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,475 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,476 subsystems 0/1 INFO :: Building subproblem matrices 3/10 (~30%) Elapsed: 0s, Remaining: 0s, Rate: 2.0e+01/s
2022-07-27 08:46:15,477 subsystems 0/1 INFO :: Building subproblem matrices 4/10 (~40%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,478 subsystems 0/1 INFO :: Building subproblem matrices 28/64 (~44%) Elapsed: 1s, Remaining: 1s, Rate: 5.0e+01/s
2022-07-27 08:46:15,478 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,482 subsystems 0/1 INFO :: Building subproblem matrices 4/10 (~40%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,482 subsystems 0/1 INFO :: Building subproblem matrices 3/10 (~30%) Elapsed: 0s, Remaining: 0s, Rate: 2.0e+01/s
2022-07-27 08:46:15,483 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,483 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.4e+01/s
2022-07-27 08:46:15,484 subsystems 0/1 INFO :: Building subproblem matrices 1/64 (~2%) Elapsed: 0s, Remaining: 18s, Rate: 3.4e+00/s
2022-07-27 08:46:15,485 subsystems 0/1 INFO :: Building subproblem matrices 26/256 (~10%) Elapsed: 1s, Remaining: 5s, Rate: 4.7e+01/s
2022-07-27 08:46:15,487 subsystems 0/1 INFO :: Building subproblem matrices 3/10 (~30%) Elapsed: 0s, Remaining: 0s, Rate: 2.0e+01/s
2022-07-27 08:46:15,487 subsystems 0/1 INFO :: Building subproblem matrices 4/10 (~40%) Elapsed: 0s, Remaining: 0s, Rate: 2.0e+01/s
2022-07-27 08:46:15,494 subsystems 0/1 INFO :: Building subproblem matrices 3/10 (~30%) Elapsed: 0s, Remaining: 0s, Rate: 2.0e+01/s
2022-07-27 08:46:15,495 subsystems 0/1 INFO :: Building subproblem matrices 28/64 (~44%) Elapsed: 1s, Remaining: 1s, Rate: 4.8e+01/s
2022-07-27 08:46:15,502 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
______________ ERROR at setup of test_ball_diffusion[stress-free-1-2-31-3-complex128] _______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb5db130>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
._ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[stress-free-1-2-31-3-complex128]>
..
def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,538 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.6e+01/s
2022-07-27 08:46:15,539 subsystems 0/1 INFO :: Building subproblem matrices 4/10 (~40%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
______________ ERROR at setup of test_ball_diffusion[stress-free-1-3-31-3-complex128] _______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55f6d0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
..... def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[stress-free-1-3-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly
...
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,549 subsystems 0/1 INFO :: Building subproblem matrices 26/256 (~10%) Elapsed: 1s, Remaining: 5s, Rate: 4.2e+01/s
2022-07-27 08:46:15,552 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,553 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,559 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,560 subsystems 0/1 INFO :: Building subproblem matrices 6/10 (~60%) Elapsed: 0s, Remaining: 0s, Rate: 2.2e+01/s
2022-07-27 08:46:15,562 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,563 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,563 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,566 subsystems 0/1 INFO :: Building subproblem matrices 6/10 (~60%) Elapsed: 0s, Remaining: 0s, Rate: 2.2e+01/s
2022-07-27 08:46:15,567 subsystems 0/1 INFO :: Building subproblem matrices 5/10 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,574 subsystems 0/1 INFO :: Building subproblem matrices 28/64 (~44%) Elapsed: 1s, Remaining: 1s, Rate: 4.2e+01/s
2022-07-27 08:46:15,575 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
_____________ ERROR at setup of test_ball_diffusion[stress-free-1.5-1-31-3-complex128] ______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb562950>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[stress-free-1.5-1-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,576 subsystems 0/1 INFO :: Building subproblem matrices 5/10 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,579 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,579 subsystems 0/1 INFO :: Building subproblem matrices 6/10 (~60%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,581 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,585 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,588 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,589 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,591 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 1s, Rate: 5.1e+00/s
2022-07-27 08:46:15,593 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,597 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.6e+01/s
2022-07-27 08:46:15,597 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,601 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,601 subsystems 0/1 INFO :: Building subproblem matrices 1/256 (~0%) Elapsed: 0s, Remaining: 34s, Rate: 7.5e+00/s
2022-07-27 08:46:15,605 subsystems 0/1 INFO :: Building subproblem matrices 28/64 (~44%) Elapsed: 1s, Remaining: 1s, Rate: 4.0e+01/s
2022-07-27 08:46:15,605 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,606 subsystems 0/1 INFO :: Building subproblem matrices 7/10 (~70%) Elapsed: 0s, Remaining: 0s, Rate: 2.2e+01/s
2022-07-27 08:46:15,607 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 1s, Rate: 5.3e+00/s
2022-07-27 08:46:15,608 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,611 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,611 subsystems 0/1 INFO :: Building subproblem matrices 7/10 (~70%) Elapsed: 0s, Remaining: 0s, Rate: 2.2e+01/s
2022-07-27 08:46:15,613 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
2022-07-27 08:46:15,613 subsystems 0/1 INFO :: Building subproblem matrices 6/10 (~60%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,613 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,615 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,616 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,619 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,619 subsystems 0/1 INFO :: Building subproblem matrices 1/4 (~25%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
_____________ ERROR at setup of test_ball_diffusion[stress-free-1.5-2-31-3-complex128] ______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55feb0>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

. @classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[stress-free-1.5-2-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
.2022-07-27 08:46:15,620 subsystems 0/1 INFO :: Building subproblem matrices 6/10 (~60%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,621 subsystems 0/1 INFO :: Building subproblem matrices 7/10 (~70%) Elapsed: 0s, Remaining: 0s, Rate: 2.1e+01/s
2022-07-27 08:46:15,628 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,633 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,634 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
2022-07-27 08:46:15,638 subsystems 0/1 INFO :: Building subproblem matrices 4/4 (~100%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
2022-07-27 08:46:15,639 subsystems 0/1 INFO :: Building subproblem matrices 35/64 (~55%) Elapsed: 1s, Remaining: 1s, Rate: 4.9e+01/s
2022-07-27 08:46:15,644 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.8e+01/s
______________ ERROR at setup of test_ball_diffusion[potential-1.5-1-31-3-complex128] _______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55e320>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

. @classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[potential-1.5-1-31-3-complex128]>

. def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,716 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 1.9e+01/s
______________ ERROR at setup of test_ball_diffusion[conducting-1.5-2-31-3-complex128] ______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb55ef80>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

. @classmethod
def from_call(
. cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[conducting-1.5-2-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,787 subsystems 0/1 INFO :: Building subproblem matrices 2/4 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 1.7e+01/s
_____________ ERROR at setup of test_ball_diffusion[stress-free-1.5-3-31-3-complex128] ______________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f0efb563e20>, when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.

:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
. precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:154: in pytest_runtest_setup
item.session._setupstate.setup(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytest_parallel.ThreadLocalSetupState object at 0x7f0f069eeaa0>
item = <Function test_ball_diffusion[stress-free-1.5-3-31-3-complex128]>

def setup(self, item: Item) -> None:
"""Setup objects along the collector chain to the item."""
needed_collectors = item.listchain()

# If a collector fails its setup, fail its entire subtree of items.
# The setup is not retried for each item - the same exception is used.
for col, (finalizers, exc) in self.stack.items():
> assert col in needed_collectors, "previous item was not torn down properly"
E AssertionError: previous item was not torn down properly

/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py:482: AssertionError
--------------------------------------- Captured stdout setup ---------------------------------------
2022-07-27 08:46:15,699 subsystems 0/1 INFO :: Building subproblem matrices 3/4 (~75%) Elapsed: 0s, Remaining: 0s, Rate: 9.9e+00/s
========================================= warnings summary ==========================================
../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tools/cache.py:59: 158 warnings
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tools/cache.py:59: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
argnames, _, _, defaults = inspect.getargspec(function)

../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/cacheprovider.py:433
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/.pytest_cache/v/cache/nodeids
config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/cacheprovider.py:387
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/cacheprovider.py:387: PytestCacheWarning: cache could not write path /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/.pytest_cache/v/cache/lastfailed
config.cache.set("cache/lastfailed", self.lastfailed)

../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/stepwise.py:52
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/stepwise.py:52: PytestCacheWarning: cache could not write path /usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/.pytest_cache/v/cache/stepwise
session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================== short test summary info ======================================
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_cartesian_operators.py::test_trace_explicit[g-float64-1-16-build_FC]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_clenshaw.py::test_clenshaw_vector[2-1-1-0-8]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_waves_1d_first_order[ChebyshevT-32-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_cartesian_operators.py::test_transpose_implicit[float64-1-16-build_FFF]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_bessel_eigenfunction[1-10-3-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_bessel_eigenfunction[1.5-0-2-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_bessel_eigenfunction[1.5-5-0-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[no-slip-1.5-1-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[no-slip-1.5-3-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[stress-free-1-2-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[stress-free-1-3-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[stress-free-1.5-1-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[stress-free-1.5-2-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[potential-1.5-1-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[conducting-1.5-2-31-3-complex128]
ERROR ../../usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_evp.py::test_ball_diffusion[stress-free-1.5-3-31-3-complex128]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/popen_fork.py:27: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
..=================== 209 passed, 1618 deselected, 161 warnings, 16 errors in 2.88s ===================
.Process Process-13:
.Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
...EProcess Process-44:
Process Process-65:
.Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
. File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
..Process Process-16:
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
.......Process Process-23:
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
.ETraceback (most recent call last):
.. File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
. File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Process Process-11:
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
....Process Process-8:
.Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
.Exception in thread Thread-1 (process_responses):
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 359, in process_responses
event_name, kwargs = queue.get()
File "<string>", line 2, in get
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 833, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 260, in serve_client
self.id_to_local_proxy_obj[ident]
KeyError: '7f0f069f7280'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 262, in serve_client
raise ke
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 256, in serve_client
obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '7f0f069f7280'
---------------------------------------------------------------------------
Exception in thread Thread-2:
Process Process-31:
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 93, in run
run_test(self.session, item, None)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 54, in run_test
Exception in thread Thread-2:
item.ihook.pytest_runtest_protocol(item=item, nextitem=nextitem)
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
Process Process-54:
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 93, in run
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
KeyboardInterrupt
run_test(self.session, item, None)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 54, in run_test
return outcome.get_result()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
item.ihook.pytest_runtest_protocol(item=item, nextitem=nextitem)
raise ex[1].with_traceback(ex[2])
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
res = hook_impl.function(*args)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 111, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 124, in runtestprotocol
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
rep = call_and_report(item, "setup", log)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 223, in call_and_report
hook.pytest_runtest_logreport(report=report)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 348, in pytest_runtest_logreport
self.send_response('testreport', report=data)
return outcome.get_result()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 339, in send_response
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
self.responses_queue.put((event_name, arguments))
File "<string>", line 2, in put
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
raise ex[1].with_traceback(ex[2])
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in process_with_threads
[t.join() for t in threads]
res = hook_impl.function(*args)
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 70, in <listcomp>
[t.join() for t in threads]
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 111, in pytest_runtest_protocol
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1089, in join
self._wait_for_tstate_lock()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
runtestprotocol(item, nextitem=nextitem)
KeyboardInterrupt
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 130, in runtestprotocol
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 98, in run
self.errors.put((self.name, pickle.dumps(sys.exc_info())))
File "<string>", line 2, in put
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 101, in run
self.queue.task_done()
File "<string>", line 2, in task_done
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 93, in run
run_test(self.session, item, None)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 54, in run_test
item.ihook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 111, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 124, in runtestprotocol
rep = call_and_report(item, "setup", log)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/_pytest/runner.py", line 223, in call_and_report
hook.pytest_runtest_logreport(report=report)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 348, in pytest_runtest_logreport
self.send_response('testreport', report=data)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 339, in send_response
self.responses_queue.put((event_name, arguments))
File "<string>", line 2, in put
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 98, in run
self.errors.put((self.name, pickle.dumps(sys.exc_info())))
File "<string>", line 2, in put
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/pytest_parallel/__init__.py", line 101, in run
self.queue.task_done()
File "<string>", line 2, in task_done
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
^CException ignored in atexit callback: <function _exit_function at 0x7f0f06f7a4d0>
Traceback (most recent call last):
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
p.join()
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/popen_fork.py", line 43, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt:
(dedalus3) myself@edmonton:~$

Jeffrey S. Oishi

unread,

Jul 27, 2022, 11:30:22 AM7/27/22

to dedalus-users

Which subsets did you try? You could try running through a bunch of
them to try and isolate the ones that error.

On Wed, Jul 27, 2022 at 10:16 AM 'em' via Dedalus Users

> To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/f1807bb0-15a2-4331-a7df-9e7157a29077n%40googlegroups.com.

em

unread,

Jul 28, 2022, 1:03:31 AM7/28/22

to Dedalus Users

I tried them all one-by-one. The following caused errors:

test_cartesian_ncc.py - 60 failed, 80 passed, 4920 warnings in 26.04s

test_output.py - 72 failed, 24 xfailed, 1689 warnings in 5.09s

test_spherical_ncc.py - 1200 failed, 80 passed, 33017 warnings in 1802.43s (0:30:02)

Keaton Burns

unread,

Jul 28, 2022, 9:26:37 AM7/28/22

to dedalu...@googlegroups.com

Those NCC files have no failures and are skipped when you run the command-line tests, so my guess is that the problem is coming from test_output.py. Could you try temporarily moving that file to a different folder, and seeing if you can then complete the command-line tests?

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/844c5368-d3c8-4b88-860d-5a41c5891293n%40googlegroups.com.

em

unread,

Jul 28, 2022, 1:55:56 PM7/28/22

to Dedalus Users

Removing test_output.py did not change anything. Removing all except test_output.py and running tests produces no errors.

Trying each test file individually:

test_cartesian_operators.py - four errors, hangs

test_clenshaw.py - no errors. first run ok. subsequent runs hang after last test.

test_fourier_operators.py - hangs after last test

test_transforms.py - lots of errors. hangs

test_nlbvp.py - OK

test_grid_operators.py - first run ok. subsequent ones have single errror and hang

test_ivp.py - hangs, sometimes has single error

test_jacobi_operators.py - errors, hangs

I did not continue beyond that.

If I run a test like this:

pytest test_fourier_operators.py

There are no problems.

If I run it in parallel:

pytest --workers=64 test_fourier_operators.py

There are errors and/or hangs. I tried various number of workers. It looks like the more workers I use the greater chance of problems.

The server has 2 CPUs(Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz), 32 cores each.

em

unread,

Aug 1, 2022, 3:20:03 PM8/1/22

to Dedalus Users

Can I assume that the problems are with pytest and not Dedalus? Below are more details on whats going on when a single test is run that hangs.

All tests pass reliably if run one at a time. I tested this like this:

Edit envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/__init__.py and add

"--workers=1" to the args in base_cmd(). Then run "python3 -m dedalus test".

Result:

5337 passed, 16 skipped, 1618 deselected, 57 xfailed, 6 xpassed, 158 warnings in 733.44s (0:12:13)

When I check the hung processes I see one parent and one or more child processes:

root@rossby:~# ps -ef|grep deda
mathit 3035134 2902381 5 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not ncc -vv test_fourier_operators.py
mathit 3035138 3035134 1 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not ncc -vv test_fourier_operators.py
mathit 3035223 3035134 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not ncc -vv test_fourier_operators.py
mathit 3035246 3035134 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not ncc -vv test_fourier_operators.py
root 3035521 2902400 0 13:28 pts/1 00:00:00 grep --color=auto deda

root@rossby:~# for pid in 3035134 3035138 3035223 3035246 ; do ps -fp $pid; cat /proc/$pid/stack;echo;done
UID PID PPID C STIME TTY TIME CMD
mathit 3035134 2902381 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not
[<0>] do_wait+0x173/0x2f0
[<0>] kernel_wait4+0xaf/0x150
[<0>] __do_sys_wait4+0x85/0x90
[<0>] __x64_sys_wait4+0x1c/0x20
[<0>] do_syscall_64+0x61/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

UID PID PPID C STIME TTY TIME CMD
mathit 3035138 3035134 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not
[<0>] futex_wait_queue_me+0xba/0x120
[<0>] futex_wait+0x105/0x250
[<0>] do_futex+0x12c/0x1d0
[<0>] __x64_sys_futex+0x78/0x1e0
[<0>] do_syscall_64+0x61/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

UID PID PPID C STIME TTY TIME CMD
mathit 3035223 3035134 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not
[<0>] futex_wait_queue_me+0xba/0x120
[<0>] futex_wait+0x105/0x250
[<0>] do_futex+0x12c/0x1d0
[<0>] __x64_sys_futex+0x78/0x1e0
[<0>] do_syscall_64+0x61/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

UID PID PPID C STIME TTY TIME CMD
mathit 3035246 3035134 0 13:28 pts/2 00:00:00 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not
[<0>] futex_wait_queue_me+0xba/0x120
[<0>] futex_wait+0x105/0x250
[<0>] do_futex+0x12c/0x1d0
[<0>] __x64_sys_futex+0x78/0x1e0
[<0>] do_syscall_64+0x61/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae

And backtraces from gdb:

(dedalus3) mathit@rossby:/usr/local/dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests$ gdb --args /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k "not ncc" -vv test_fourier_operators.py
GNU gdb (Ubuntu 11.1-0ubuntu2) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/dedalus/miniconda3/envs/dedalus3/bin/python3...
(gdb) run
Starting program: /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/bin/python3 /usr/local/dedalus/miniconda3/envs/dedalus3/bin/pytest --benchmark-disable --workers=auto -k not\ ncc -vv test_fourier_operators.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after vfork from child process 3040805]
[Detaching after vfork from child process 3040806]
[Detaching after fork from child process 3040807]
[Detaching after vfork from child process 3040809]

============================================================================================ test session starts ============================================================================================

platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0 -- /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/bin/python3
cachedir: .pytest_cache

benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)

rootdir: /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests
plugins: cov-3.0.0, parallel-0.1.1, benchmark-3.4.1
collecting ... [Detaching after fork from child process 3040810]
[New Thread 0x7ffff6056640 (LWP 3040814)]
[New Thread 0x7ffff5677640 (LWP 3040815)]
collected 60 items
pytest-parallel: 40 workers (processes), 1 test per worker (thread)
[New Thread 0x7fffe6cd1640 (LWP 3040827)]
[Detaching after fork from child process 3040829]
[Detaching after fork from child process 3040831]
[Detaching after fork from child process 3040836]
[Detaching after fork from child process 3040841]

test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py [Detaching after fork from child process 3040844]
[Detaching after fork from child process 3040853]

test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py [Detaching after fork from child process 3040862]
PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py [Detaching after fork from child process 3040870]
PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED[Detaching after fork from child process 3040873]
PASSED

test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED[Detaching after fork from child process 3040882]
PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040886]

test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040892]
PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040900]
PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py [Detaching after fork
from child process 3040905]
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040912]
PASSEDPASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSED

test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py
test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[g-float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSEDPASSEDPASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[g-complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040964]
[Detaching after fork from child process 3040969]
[Detaching after fork from child process 3040978]

test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3040987]
[Detaching after fork from child process 3040996]
[Detaching after fork from child process 3041005]

test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3041014]
[Detaching after fork from child process 3041023]
[Detaching after fork from child process 3041032]
[Detaching after fork from child process 3041040]
[Detaching after fork from child process 3041050]
[Detaching after fork from child process 3041058]

[Detaching after fork from child process 3041068]
test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3041077]
[Detaching after fork from child process 3041086]
[Detaching after fork from child process 3041093]
[Detaching after fork from child process 3041103]

test_fourier_operators.py::test_fourier_convert_constant[c-float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3041113]
[Detaching after fork from child process 3041122]
[Detaching after fork from child process 3041129]
[Detaching after fork from child process 3041140]

test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3041149]
[Detaching after fork from child process 3041158]
[Detaching after fork from child process 3041167]

test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED[Detaching after fork from child process 3041176]

test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_convert_constant[c-complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_differentiate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_interpolate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds0-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-10] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[complex128-bounds1-12] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED
test_fourier_operators.py::test_fourier_integrate[float64-bounds1-8] <- ../../../../../../../../../../dedalus/miniconda3/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/test_fourier_operators.py PASSED^C
Thread 1 "python3" received signal SIGINT, Interrupt.
0x00007ffff7d6172f in __GI___wait4 (pid=pid@entry=3040912, stat_loc=stat_loc@entry=0x7fffffff9ed4, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
(gdb) bt
#0 0x00007ffff7d6172f in __GI___wait4 (pid=pid@entry=3040912, stat_loc=stat_loc@entry=0x7fffffff9ed4, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1 0x00007ffff7d616ab in __GI___waitpid (pid=pid@entry=3040912, stat_loc=stat_loc@entry=0x7fffffff9ed4, options=options@entry=0) at waitpid.c:38
#2 0x0000555555792d0e in os_waitpid_impl (module=<optimized out>, options=0, pid=3040912) at /usr/local/src/conda/python-3.10.4/Modules/posixmodule.c:8418
#3 os_waitpid (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at /usr/local/src/conda/python-3.10.4/Modules/clinic/posixmodule.c.h:4021
#4 0x00005555556985bf in cfunction_vectorcall_FASTCALL (func=0x7ffff7c19940, args=0x7fffe6e9cd10, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/local/src/conda/python-3.10.4/Objects/methodobject.c:430
#5 0x000055555568c29b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e9cd10, callable=0x7ffff7c19940, tstate=0x55555591f560)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#6 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e9cd10, callable=0x7ffff7c19940) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#7 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffffff9ff0, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#8 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4181
#9 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7fffe6e9cb80, tstate=0x55555591f560) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#10 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff66bc200, tstate=0x55555591f560)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#11 _PyFunction_Vectorcall (func=0x7ffff66bc1f0, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#12 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe00026e0, callable=0x7ffff66bc1f0, tstate=0x55555591f560)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#13 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe00026e0, callable=0x7ffff66bc1f0) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#14 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffffffa1b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#15 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#16 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7fffe0002560, tstate=0x55555591f560) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#17 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff66bc290, tstate=0x55555591f560)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#18 _PyFunction_Vectorcall (func=0x7ffff66bc280, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#19 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe00018b0, callable=0x7ffff66bc280, tstate=0x55555591f560)
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#20 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe00018b0, callable=0x7ffff66bc280) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123

...SNIP...

(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7ffff7c73740 (LWP 3040801) "python3" 0x00007ffff7d6172f in __GI___wait4 (pid=pid@entry=3040912, stat_loc=stat_loc@entry=0x7fffffff9ed4, options=options@entry=0, usage=usage@entry=0x0)
at ../sysdeps/unix/sysv/linux/wait4.c:30
2 Thread 0x7ffff6056640 (LWP 3040814) "python3" 0x00007ffff7d8fd3f in __GI___poll (fds=0x7ffff0000b60, nfds=1, timeout=3599818) at ../sysdeps/unix/sysv/linux/poll.c:29
3 Thread 0x7ffff5677640 (LWP 3040815) "python3" 0x00007ffff7d9cd8e in epoll_wait (epfd=22, events=0x555555dcc700, maxevents=32, timeout=2100000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
4 Thread 0x7fffe6cd1640 (LWP 3040827) "python3" __GI___libc_read (nbytes=4, buf=0x7fffe6ea2ff0, fd=30) at ../sysdeps/unix/sysv/linux/read.c:26
(gdb)

(gdb) t 2
[Switching to thread 2 (Thread 0x7ffff6056640 (LWP 3040814))]
#0 0x00007ffff7d8fd3f in __GI___poll (fds=0x7ffff0000b60, nfds=1, timeout=3599818) at ../sysdeps/unix/sysv/linux/poll.c:29
29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) bt
#0 0x00007ffff7d8fd3f in __GI___poll (fds=0x7ffff0000b60, nfds=1, timeout=3599818) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007ffff610a8c1 in poll_dispatch () from /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/mpi4py/../../.././libopen-pal.so.40
#2 0x00007ffff610311f in opal_libevent2022_event_base_loop () from /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/mpi4py/../../.././libopen-pal.so.40
#3 0x00007ffff60c3caf in progress_engine () from /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/mpi4py/../../.././libopen-pal.so.40
#4 0x00007ffff7d0c947 in start_thread (arg=<optimized out>) at pthread_create.c:435
#5 0x00007ffff7d9ca44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

(gdb) t 3
[Switching to thread 3 (Thread 0x7ffff5677640 (LWP 3040815))]
#0 0x00007ffff7d9cd8e in epoll_wait (epfd=22, events=0x555555dcc700, maxevents=32, timeout=2100000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30 ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) bt
#0 0x00007ffff7d9cd8e in epoll_wait (epfd=22, events=0x555555dcc700, maxevents=32, timeout=2100000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007ffff6100681 in epoll_dispatch () from /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/mpi4py/../../.././libopen-pal.so.40
#2 0x00007ffff610311f in opal_libevent2022_event_base_loop () from /usr/local/Dedalus/dedalus-3b/miniconda3/envs/dedalus3/lib/python3.10/site-packages/mpi4py/../../.././libopen-pal.so.40
#3 0x00007ffff57e005f in progress_engine () from /usr/local/dedalus/miniconda3/envs/dedalus3/lib/openmpi/mca_pmix_pmix3x.so
#4 0x00007ffff7d0c947 in start_thread (arg=<optimized out>) at pthread_create.c:435
#5 0x00007ffff7d9ca44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

(gdb) t 4
[Switching to thread 4 (Thread 0x7fffe6cd1640 (LWP 3040827))]
#0 __GI___libc_read (nbytes=4, buf=0x7fffe6ea2ff0, fd=30) at ../sysdeps/unix/sysv/linux/read.c:26
26 ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) bt
#0 __GI___libc_read (nbytes=4, buf=0x7fffe6ea2ff0, fd=30) at ../sysdeps/unix/sysv/linux/read.c:26
#1 __GI___libc_read (fd=fd@entry=30, buf=buf@entry=0x7fffe6ea2ff0, nbytes=nbytes@entry=4) at ../sysdeps/unix/sysv/linux/read.c:24
#2 0x000055555574cfd6 in read (__nbytes=<optimized out>, __buf=<optimized out>, __fd=<optimized out>, __fd=<optimized out>, __buf=<optimized out>, __nbytes=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python-split_1648142860430/_build_env/x86_64-conda-linux-gnu/sysroot/usr/include/bits/unistd.h:45
#3 _Py_read (fd=30, buf=0x7fffe6ea2ff0, count=4) at /usr/local/src/conda/python-3.10.4/Python/fileutils.c:1746
#4 0x00005555557993d6 in os_read_impl (module=<optimized out>, length=4, fd=30) at /usr/local/src/conda/python-3.10.4/Modules/posixmodule.c:9464
#5 os_read (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at /usr/local/src/conda/python-3.10.4/Modules/clinic/posixmodule.c.h:4815
#6 0x00005555556985bf in cfunction_vectorcall_FASTCALL (func=0x7ffff7c19da0, args=0x7ffff6630e98, nargsf=<optimized out>, kwnames=<optimized out>)
at /usr/local/src/conda/python-3.10.4/Objects/methodobject.c:430
#7 0x0000555555687bcc in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6630e98, callable=0x7ffff7c19da0, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#8 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6630e98, callable=0x7ffff7c19da0) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#9 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6ccfe70, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#10 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4213
#11 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7ffff6630cf0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#12 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff6663a40, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#13 _PyFunction_Vectorcall (func=0x7ffff6663a30, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#14 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6619f88, callable=0x7ffff6663a30, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#15 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6619f88, callable=0x7ffff6663a30) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#16 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd0030, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#17 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#18 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7ffff6619e00, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#19 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff6663b60, tstate=0x555556468540)
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#20 _PyFunction_Vectorcall (func=0x7ffff6663b50, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#21 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6651298, callable=0x7ffff6663b50, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#22 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff6651298, callable=0x7ffff6663b50) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#23 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd01f0, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#24 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#25 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7ffff6651120, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#26 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff66636e0, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#27 _PyFunction_Vectorcall (func=0x7ffff66636d0, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#28 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55555650df60, callable=0x7ffff66636d0, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#29 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55555650df60, callable=0x7ffff66636d0) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#30 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd03b0, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#32 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x55555650dda0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#33 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff66a74a0, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#34 _PyFunction_Vectorcall (func=0x7ffff66a7490, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#35 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe7f5fd40, callable=0x7ffff66a7490, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#36 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe7f5fd40, callable=0x7ffff66a7490) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#37 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd0570, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
--Type <RET> for more, q to quit, c to continue without paging--
#39 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7fffe7f5fbc0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#40 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7fffe6e49a30, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#41 _PyFunction_Vectorcall (func=0x7fffe6e49a20, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#42 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe0000d08, callable=0x7fffe6e49a20, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#43 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe0000d08, callable=0x7fffe6e49a20) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#44 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd0730, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#45 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#46 0x00005555556a5932 in _PyEval_EvalFrame (throwflag=0, f=0x7fffe0000b70, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#47 _PyEval_Vector (kwnames=0x0, argcount=<optimized out>, args=0x7fffe6cd0820, locals=0x0, con=0x7ffff679ef90, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#48 _PyFunction_Vectorcall (kwnames=0x0, nargsf=<optimized out>, stack=0x7fffe6cd0820, func=0x7ffff679ef80) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#49 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6cd0820, callable=0x7ffff679ef80, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#50 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/classobject.c:83
#51 0x000055555568a3c5 in do_call_core (kwdict=0x7ffff6847100, callargs=0x7fffed65e080, func=0x7ffff6a8a5c0, trace_info=0x7fffe6cd0940, tstate=<optimized out>)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5919
#52 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4277
#53 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7fffe7f5fa00, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#54 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff78d3a40, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#55 _PyFunction_Vectorcall (func=0x7ffff78d3a30, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#56 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e0cf40, callable=0x7ffff78d3a30, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#57 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e0cf40, callable=0x7ffff78d3a30) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
--Type <RET> for more, q to quit, c to continue without paging--
#58 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd0b00, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#59 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#60 0x00005555556983cf in _PyEval_EvalFrame (throwflag=0, f=0x7fffe6e0cdd0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#61 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7ffff78d3d10, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#62 _PyFunction_Vectorcall (func=0x7ffff78d3d00, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#63 0x0000555555687f2f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e85290, callable=0x7ffff78d3d00, tstate=0x555556468540)
at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#64 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fffe6e85290, callable=0x7ffff78d3d00) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:123
#65 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7fffe6cd0cc0, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5867
#66 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:4198
#67 0x00005555556a5a57 in _PyEval_EvalFrame (throwflag=0, f=0x7fffe6e85120, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/internal/pycore_ceval.h:46
#68 _PyEval_Vector (kwnames=0x0, argcount=1, args=0x7fffe6cd0da8, locals=0x0, con=0x7ffff78d3ad0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Python/ceval.c:5065
#69 _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x7fffe6cd0da8, func=0x7ffff78d3ac0) at /usr/local/src/conda/python-3.10.4/Objects/call.c:342
#70 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fffe6cd0da8, callable=0x7ffff78d3ac0, tstate=0x555556468540) at /usr/local/src/conda/python-3.10.4/Include/cpython/abstract.h:114
#71 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.10.4/Objects/classobject.c:61
#72 0x0000555555792835 in thread_run (boot_raw=0x7fffe6ea01b0) at /usr/local/src/conda/python-3.10.4/Modules/_threadmodule.c:1090
#73 0x00005555557927e4 in pythread_wrapper (arg=<optimized out>) at /usr/local/src/conda/python-3.10.4/Python/thread_pthread.h:248
#74 0x00007ffff7d0c947 in start_thread (arg=<optimized out>) at pthread_create.c:435
#75 0x00007ffff7d9ca44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb)

em

unread,

Aug 3, 2022, 8:02:55 AM8/3/22

to Dedalus Users

The problem appears to only happen in systems with multiple physical CPUs.

I spun up two VMs: Debian 11.4 and Ubuntu 22.04 Server. Dedalus tests out perfectly on them. They are configured with 1 cpu with 8 cores.

My laptop also runs Dedalus tests perfectly: Ubuntu 22.04, one cpu, 16 cores.

All of our servers that have problems running the Dedalus tests have two CPUs.

I tried keeping the test process on a single CPU by selecting only those cores that were on physical CPU #0:
taskset --cpu-list `echo {0..38..2}|tr ' ' ,` python3 -m dedalus test

The first run went without error(as happens some times). All subsequent runs had errors and hung.

One thing I noticed is that the machines that have the testing problems always leave orphaned pytest threads when I stop the tests with CTRL-C. The machines where Dedalus tests run perfectly do
not do this; all threads exit properly.

Keaton Burns

unread,

Aug 3, 2022, 4:42:19 PM8/3/22

to dedalu...@googlegroups.com

Hmm yes it sounds like it’s something complicated with pytest. Dedalus doesn’t do any multithreading / process management outside of MPI. Here I think pytest is using Python’s multiprocessing to do parallel tests, and my guess would be that’s where things are going wrong.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/def39b5f-d381-4ea0-a61f-80225f26bb66n%40googlegroups.com.

em

unread,

Aug 4, 2022, 11:37:23 AM8/4/22

to Dedalus Users

I'll be running tests like this:

pytest --workers=1 -k 'not ncc' /PATH/TO/CONDA/INSTALL/envs/dedalus3/lib/python3.10/site-packages/dedalus/tests/

Thanks.

黃瑪瑪

unread,

Aug 27, 2025, 11:58:54 AM (13 days ago) Aug 27

to Dedalus Users

Dear all,

Thanks for the discussion. I also found my installing has "6586 passed, 16 skipped, 143 xfailed, 18 xpassed in 403.80s".

It seems that the installation step, "conda env config vars set OMP_NUM_THREADS=1", was not effective. When I ran "python3 -m dedalus tests" on my dedalus3 conda virtual environment, the output error message relates some keywords about multiprocessing:

"/home/yahuei/anaconda3/envs/dedalus3/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=7666) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
...

pytest-parallel: 8 workers (processes), 1 test per worker (thread)

/home/yahuei/anaconda3/envs/dedalus3/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=7666) is multi-threaded, use of fork() may lead to deadlocks in the child.

self.pid = os.fork() ...

I installed dedalus3 via conda, and my OS is Ubuntu 18.04.6 LTS.

Please let me know about things I can try if you spot something from my output error!

Thanks in advance.

em 在 2022年8月4日星期四上午11:37:23 [UTC-4] 的信中寫道：

Reply all

Reply to author

Forward

0 new messages