Unable to run example code in parallel
67 views
Skip to first unread message
Ali Asad
Jan 26, 2021, 4:20:40 AM1/26/21
to Dedalus Users
Hi all,
I wanted to test parallel code. When I run the following example
I get the following errors,
rbc$ mpiexec --hostfile hostfile -n 4 python rbc.py
2021-01-22 23:44:14,134 pencil 0/4 INFO :: Building pencil matrix 1/1024 (~0%) Elapsed: 0s, Remaining: 38s, Rate: 2.7e+01/s
2021-01-22 23:44:17,885 pencil 0/4 INFO :: Building pencil matrix 103/1024 (~10%) Elapsed: 4s, Remaining: 34s, Rate: 2.7e+01/s
2021-01-22 23:44:21,484 pencil 0/4 INFO :: Building pencil matrix 206/1024 (~20%) Elapsed: 7s, Remaining: 29s, Rate: 2.8e+01/s
2021-01-22 23:44:24,113 pencil 0/4 INFO :: Building pencil matrix 281/1024 (~27%) Elapsed: 10s, Remaining: 26s, Rate: 2.8e+01/s
2021-01-22 23:44:25,095 pencil 0/4 INFO :: Building pencil matrix 309/1024 (~30%) Elapsed: 11s, Remaining: 25s, Rate: 2.8e+01/s
2021-01-22 23:44:28,707 pencil 0/4 INFO :: Building pencil matrix 412/1024 (~40%) Elapsed: 15s, Remaining: 22s, Rate: 2.8e+01/s
2021-01-22 23:44:32,317 pencil 0/4 INFO :: Building pencil matrix 515/1024 (~50%) Elapsed: 18s, Remaining: 18s, Rate: 2.8e+01/s
2021-01-22 23:44:34,129 pencil 0/4 INFO :: Building pencil matrix 567/1024 (~55%) Elapsed: 20s, Remaining: 16s, Rate: 2.8e+01/s
2021-01-22 23:44:35,912 pencil 0/4 INFO :: Building pencil matrix 618/1024 (~60%) Elapsed: 22s, Remaining: 14s, Rate: 2.8e+01/s
2021-01-22 23:44:39,502 pencil 0/4 INFO :: Building pencil matrix 721/1024 (~70%) Elapsed: 25s, Remaining: 11s, Rate: 2.8e+01/s
2021-01-22 23:44:43,096 pencil 0/4 INFO :: Building pencil matrix 824/1024 (~80%) Elapsed: 29s, Remaining: 7s, Rate: 2.8e+01/s
2021-01-22 23:44:44,115 pencil 0/4 INFO :: Building pencil matrix 853/1024 (~83%) Elapsed: 30s, Remaining: 6s, Rate: 2.8e+01/s
2021-01-22 23:44:46,690 pencil 0/4 INFO :: Building pencil matrix 927/1024 (~91%) Elapsed: 33s, Remaining: 3s, Rate: 2.8e+01/s
[asad-workstation:05714] *** Process received signal ***
[asad-workstation:05714] Signal: Segmentation fault (11)
[asad-workstation:05714] Signal code: Address not mapped (1)
[asad-workstation:05714] Failing at address: 0x8df90808
[asad-workstation:05714] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f92aa1a3390]
[asad-workstation:05714] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7f92a75cf6b7]
[asad-workstation:05714] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7f929f413a5e]
[asad-workstation:05714] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7f929f414a46]
[asad-workstation:05714] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7f929f181f90]
[asad-workstation:05714] [ 5] python(PyCFunction_Call+0x56)[0x55e98bf21f76]
[asad-workstation:05714] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7f929f161a0d]
[asad-workstation:05714] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7f929f190dd8]
[asad-workstation:05714] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7f929f19970a]
[asad-workstation:05714] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55e98bedf7d4]
[asad-workstation:05714] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55e98bf62e35]
[asad-workstation:05714] [11] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [12] python(PyVectorcall_Call+0x71)[0x55e98bedf041]
[asad-workstation:05714] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55e98bf6499b]
[asad-workstation:05714] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55e98bf2ca92]
[asad-workstation:05714] [15] python(_PyObject_FastCallDict+0x20c)[0x55e98bf2e48c]
[asad-workstation:05714] [16] python(_PyObject_Call_Prepend+0x63)[0x55e98bf2e733]
[asad-workstation:05714] [17] python(+0x18c83a)[0x55e98bf2e83a]
[asad-workstation:05714] [18] python(_PyObject_MakeTpCall+0x22f)[0x55e98bedf85f]
[asad-workstation:05714] [19] python(+0x18bfa1)[0x55e98bf2dfa1]
[asad-workstation:05714] [20] python(+0x10077f)[0x55e98bea277f]
[asad-workstation:05714] [21] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [22] python(+0x18be79)[0x55e98bf2de79]
[asad-workstation:05714] [23] python(PyVectorcall_Call+0x71)[0x55e98bedf041]
[asad-workstation:05714] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55e98bf6499b]
[asad-workstation:05714] [25] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [26] python(+0x10075e)[0x55e98bea275e]
[asad-workstation:05714] [27] python(+0x18bc0b)[0x55e98bf2dc0b]
[asad-workstation:05714] [28] python(+0x10077f)[0x55e98bea277f]
[asad-workstation:05714] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55e98bf2ca92]
[asad-workstation:05714] *** End of error message ***
2021-01-22 23:44:14,134 pencil 0/4 INFO :: Building pencil matrix 1/1024 (~0%) Elapsed: 0s, Remaining: 38s, Rate: 2.7e+01/s
2021-01-22 23:44:17,885 pencil 0/4 INFO :: Building pencil matrix 103/1024 (~10%) Elapsed: 4s, Remaining: 34s, Rate: 2.7e+01/s
2021-01-22 23:44:21,484 pencil 0/4 INFO :: Building pencil matrix 206/1024 (~20%) Elapsed: 7s, Remaining: 29s, Rate: 2.8e+01/s
2021-01-22 23:44:24,113 pencil 0/4 INFO :: Building pencil matrix 281/1024 (~27%) Elapsed: 10s, Remaining: 26s, Rate: 2.8e+01/s
2021-01-22 23:44:25,095 pencil 0/4 INFO :: Building pencil matrix 309/1024 (~30%) Elapsed: 11s, Remaining: 25s, Rate: 2.8e+01/s
2021-01-22 23:44:28,707 pencil 0/4 INFO :: Building pencil matrix 412/1024 (~40%) Elapsed: 15s, Remaining: 22s, Rate: 2.8e+01/s
2021-01-22 23:44:32,317 pencil 0/4 INFO :: Building pencil matrix 515/1024 (~50%) Elapsed: 18s, Remaining: 18s, Rate: 2.8e+01/s
2021-01-22 23:44:34,129 pencil 0/4 INFO :: Building pencil matrix 567/1024 (~55%) Elapsed: 20s, Remaining: 16s, Rate: 2.8e+01/s
2021-01-22 23:44:35,912 pencil 0/4 INFO :: Building pencil matrix 618/1024 (~60%) Elapsed: 22s, Remaining: 14s, Rate: 2.8e+01/s
2021-01-22 23:44:39,502 pencil 0/4 INFO :: Building pencil matrix 721/1024 (~70%) Elapsed: 25s, Remaining: 11s, Rate: 2.8e+01/s
2021-01-22 23:44:43,096 pencil 0/4 INFO :: Building pencil matrix 824/1024 (~80%) Elapsed: 29s, Remaining: 7s, Rate: 2.8e+01/s
2021-01-22 23:44:44,115 pencil 0/4 INFO :: Building pencil matrix 853/1024 (~83%) Elapsed: 30s, Remaining: 6s, Rate: 2.8e+01/s
2021-01-22 23:44:46,690 pencil 0/4 INFO :: Building pencil matrix 927/1024 (~91%) Elapsed: 33s, Remaining: 3s, Rate: 2.8e+01/s
[asad-workstation:05714] *** Process received signal ***
[asad-workstation:05714] Signal: Segmentation fault (11)
[asad-workstation:05714] Signal code: Address not mapped (1)
[asad-workstation:05714] Failing at address: 0x8df90808
[asad-workstation:05714] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f92aa1a3390]
[asad-workstation:05714] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7f92a75cf6b7]
[asad-workstation:05714] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7f929f413a5e]
[asad-workstation:05714] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7f929f414a46]
[asad-workstation:05714] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7f929f181f90]
[asad-workstation:05714] [ 5] python(PyCFunction_Call+0x56)[0x55e98bf21f76]
[asad-workstation:05714] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7f929f161a0d]
[asad-workstation:05714] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7f929f190dd8]
[asad-workstation:05714] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7f929f19970a]
[asad-workstation:05714] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55e98bedf7d4]
[asad-workstation:05714] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55e98bf62e35]
[asad-workstation:05714] [11] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [12] python(PyVectorcall_Call+0x71)[0x55e98bedf041]
[asad-workstation:05714] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55e98bf6499b]
[asad-workstation:05714] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55e98bf2ca92]
[asad-workstation:05714] [15] python(_PyObject_FastCallDict+0x20c)[0x55e98bf2e48c]
[asad-workstation:05714] [16] python(_PyObject_Call_Prepend+0x63)[0x55e98bf2e733]
[asad-workstation:05714] [17] python(+0x18c83a)[0x55e98bf2e83a]
[asad-workstation:05714] [18] python(_PyObject_MakeTpCall+0x22f)[0x55e98bedf85f]
[asad-workstation:05714] [19] python(+0x18bfa1)[0x55e98bf2dfa1]
[asad-workstation:05714] [20] python(+0x10077f)[0x55e98bea277f]
[asad-workstation:05714] [21] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [22] python(+0x18be79)[0x55e98bf2de79]
[asad-workstation:05714] [23] python(PyVectorcall_Call+0x71)[0x55e98bedf041]
[asad-workstation:05714] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55e98bf6499b]
[asad-workstation:05714] [25] python(_PyFunction_Vectorcall+0x10b)[0x55e98bf2d86b]
[asad-workstation:05714] [26] python(+0x10075e)[0x55e98bea275e]
[asad-workstation:05714] [27] python(+0x18bc0b)[0x55e98bf2dc0b]
[asad-workstation:05714] [28] python(+0x10077f)[0x55e98bea277f]
[asad-workstation:05714] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55e98bf2ca92]
[asad-workstation:05714] *** End of error message ***
2021-01-22 23:44:50,076 pencil 0/4 INFO :: Building pencil matrix 1024/1024 (~100%) Elapsed: 36s, Remaining: 0s, Rate: 2.8e+01/s
2021-01-22 23:44:50,082 __main__ 0/4 INFO :: Solver built
2021-01-22 23:44:50,082 __main__ 0/4 INFO :: Solver built
[asad-workstation:05712] *** Process received signal ***
[asad-workstation:05712] Signal: Segmentation fault (11)
[asad-workstation:05712] Signal code: Address not mapped (1)
[asad-workstation:05712] Failing at address: 0x8f882118
[asad-workstation:05712] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2a222a7390]
[asad-workstation:05712] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7f2a1f6d36b7]
[asad-workstation:05712] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7f2a17509a5e]
[asad-workstation:05712] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7f2a1750aa46]
[asad-workstation:05712] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7f2a17277f90]
[asad-workstation:05712] [ 5] python(PyCFunction_Call+0x56)[0x55cb8d910f76]
[asad-workstation:05712] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7f2a17257a0d]
[asad-workstation:05712] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7f2a17286dd8]
[asad-workstation:05712] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7f2a1728f70a]
[asad-workstation:05712] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55cb8d8ce7d4]
[asad-workstation:05712] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55cb8d951e35]
[asad-workstation:05712] [11] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [12] python(PyVectorcall_Call+0x71)[0x55cb8d8ce041]
[asad-workstation:05712] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55cb8d95399b]
[asad-workstation:05712] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55cb8d91ba92]
[asad-workstation:05712] [15] python(_PyObject_FastCallDict+0x20c)[0x55cb8d91d48c]
[asad-workstation:05712] [16] python(_PyObject_Call_Prepend+0x63)[0x55cb8d91d733]
[asad-workstation:05712] [17] python(+0x18c83a)[0x55cb8d91d83a]
[asad-workstation:05712] [18] python(_PyObject_MakeTpCall+0x22f)[0x55cb8d8ce85f]
[asad-workstation:05712] [19] python(+0x18bfa1)[0x55cb8d91cfa1]
[asad-workstation:05712] [20] python(+0x10077f)[0x55cb8d89177f]
[asad-workstation:05712] [21] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [22] python(+0x18be79)[0x55cb8d91ce79]
[asad-workstation:05712] [23] python(PyVectorcall_Call+0x71)[0x55cb8d8ce041]
[asad-workstation:05712] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55cb8d95399b]
[asad-workstation:05712] [25] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [26] python(+0x10075e)[0x55cb8d89175e]
[asad-workstation:05712] [27] python(+0x18bc0b)[0x55cb8d91cc0b]
[asad-workstation:05712] [28] python(+0x10077f)[0x55cb8d89177f]
[asad-workstation:05712] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55cb8d91ba92]
[asad-workstation:05712] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[asad-workstation:05715] *** Process received signal ***
[asad-workstation:05715] Signal: Segmentation fault (11)
[asad-workstation:05715] Signal code: Address not mapped (1)
[asad-workstation:05715] Failing at address: 0x82eb3668
[asad-workstation:05715] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7feca5ae1390]
[asad-workstation:05715] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7feca2f0d6b7]
[asad-workstation:05715] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7fec9ad50a5e]
[asad-workstation:05715] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7fec9ad51a46]
[asad-workstation:05715] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7fec9aabef90]
[asad-workstation:05715] [ 5] python(PyCFunction_Call+0x56)[0x55c581b80f76]
[asad-workstation:05715] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7fec9aa9ea0d]
[asad-workstation:05715] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7fec9aacddd8]
[asad-workstation:05715] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7fec9aad670a]
[asad-workstation:05715] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55c581b3e7d4]
[asad-workstation:05715] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55c581bc1e35]
[asad-workstation:05715] [11] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [12] python(PyVectorcall_Call+0x71)[0x55c581b3e041]
[asad-workstation:05715] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55c581bc399b]
[asad-workstation:05715] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55c581b8ba92]
[asad-workstation:05715] [15] python(_PyObject_FastCallDict+0x20c)[0x55c581b8d48c]
[asad-workstation:05715] [16] python(_PyObject_Call_Prepend+0x63)[0x55c581b8d733]
[asad-workstation:05715] [17] python(+0x18c83a)[0x55c581b8d83a]
[asad-workstation:05715] [18] python(_PyObject_MakeTpCall+0x22f)[0x55c581b3e85f]
[asad-workstation:05715] [19] python(+0x18bfa1)[0x55c581b8cfa1]
[asad-workstation:05715] [20] python(+0x10077f)[0x55c581b0177f]
[asad-workstation:05715] [21] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [22] python(+0x18be79)[0x55c581b8ce79]
[asad-workstation:05715] [23] python(PyVectorcall_Call+0x71)[0x55c581b3e041]
[asad-workstation:05715] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55c581bc399b]
[asad-workstation:05715] [25] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [26] python(+0x10075e)[0x55c581b0175e]
[asad-workstation:05715] [27] python(+0x18bc0b)[0x55c581b8cc0b]
[asad-workstation:05715] [28] python(+0x10077f)[0x55c581b0177f]
[asad-workstation:05715] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55c581b8ba92]
[asad-workstation:05715] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 2 with PID 0 on node asad-workstation exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[asad-workstation:05712] Signal: Segmentation fault (11)
[asad-workstation:05712] Signal code: Address not mapped (1)
[asad-workstation:05712] Failing at address: 0x8f882118
[asad-workstation:05712] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2a222a7390]
[asad-workstation:05712] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7f2a1f6d36b7]
[asad-workstation:05712] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7f2a17509a5e]
[asad-workstation:05712] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7f2a1750aa46]
[asad-workstation:05712] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7f2a17277f90]
[asad-workstation:05712] [ 5] python(PyCFunction_Call+0x56)[0x55cb8d910f76]
[asad-workstation:05712] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7f2a17257a0d]
[asad-workstation:05712] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7f2a17286dd8]
[asad-workstation:05712] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7f2a1728f70a]
[asad-workstation:05712] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55cb8d8ce7d4]
[asad-workstation:05712] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55cb8d951e35]
[asad-workstation:05712] [11] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [12] python(PyVectorcall_Call+0x71)[0x55cb8d8ce041]
[asad-workstation:05712] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55cb8d95399b]
[asad-workstation:05712] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55cb8d91ba92]
[asad-workstation:05712] [15] python(_PyObject_FastCallDict+0x20c)[0x55cb8d91d48c]
[asad-workstation:05712] [16] python(_PyObject_Call_Prepend+0x63)[0x55cb8d91d733]
[asad-workstation:05712] [17] python(+0x18c83a)[0x55cb8d91d83a]
[asad-workstation:05712] [18] python(_PyObject_MakeTpCall+0x22f)[0x55cb8d8ce85f]
[asad-workstation:05712] [19] python(+0x18bfa1)[0x55cb8d91cfa1]
[asad-workstation:05712] [20] python(+0x10077f)[0x55cb8d89177f]
[asad-workstation:05712] [21] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [22] python(+0x18be79)[0x55cb8d91ce79]
[asad-workstation:05712] [23] python(PyVectorcall_Call+0x71)[0x55cb8d8ce041]
[asad-workstation:05712] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55cb8d95399b]
[asad-workstation:05712] [25] python(_PyFunction_Vectorcall+0x10b)[0x55cb8d91c86b]
[asad-workstation:05712] [26] python(+0x10075e)[0x55cb8d89175e]
[asad-workstation:05712] [27] python(+0x18bc0b)[0x55cb8d91cc0b]
[asad-workstation:05712] [28] python(+0x10077f)[0x55cb8d89177f]
[asad-workstation:05712] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55cb8d91ba92]
[asad-workstation:05712] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[asad-workstation:05715] *** Process received signal ***
[asad-workstation:05715] Signal: Segmentation fault (11)
[asad-workstation:05715] Signal code: Address not mapped (1)
[asad-workstation:05715] Failing at address: 0x82eb3668
[asad-workstation:05715] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7feca5ae1390]
[asad-workstation:05715] [ 1] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/mpi4py/../../../libmpi.so.40(PMPI_Comm_rank+0x37)[0x7feca2f0d6b7]
[asad-workstation:05715] [ 2] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_guru+0x8e)[0x7fec9ad50a5e]
[asad-workstation:05715] [ 3] /home/asad/anaconda3/envs/dedalus/lib/libfftw3_mpi.so.3(fftw_mpi_local_size_many_transposed+0x136)[0x7fec9ad51a46]
[asad-workstation:05715] [ 4] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x2bf90)[0x7fec9aabef90]
[asad-workstation:05715] [ 5] python(PyCFunction_Call+0x56)[0x55c581b80f76]
[asad-workstation:05715] [ 6] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0xba0d)[0x7fec9aa9ea0d]
[asad-workstation:05715] [ 7] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x3add8)[0x7fec9aacddd8]
[asad-workstation:05715] [ 8] /home/asad/anaconda3/envs/dedalus/lib/python3.8/site-packages/dedalus/core/transposes.cpython-38-x86_64-linux-gnu.so(+0x4370a)[0x7fec9aad670a]
[asad-workstation:05715] [ 9] python(_PyObject_MakeTpCall+0x1a4)[0x55c581b3e7d4]
[asad-workstation:05715] [10] python(_PyEval_EvalFrameDefault+0x475)[0x55c581bc1e35]
[asad-workstation:05715] [11] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [12] python(PyVectorcall_Call+0x71)[0x55c581b3e041]
[asad-workstation:05715] [13] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55c581bc399b]
[asad-workstation:05715] [14] python(_PyEval_EvalCodeWithName+0x2d2)[0x55c581b8ba92]
[asad-workstation:05715] [15] python(_PyObject_FastCallDict+0x20c)[0x55c581b8d48c]
[asad-workstation:05715] [16] python(_PyObject_Call_Prepend+0x63)[0x55c581b8d733]
[asad-workstation:05715] [17] python(+0x18c83a)[0x55c581b8d83a]
[asad-workstation:05715] [18] python(_PyObject_MakeTpCall+0x22f)[0x55c581b3e85f]
[asad-workstation:05715] [19] python(+0x18bfa1)[0x55c581b8cfa1]
[asad-workstation:05715] [20] python(+0x10077f)[0x55c581b0177f]
[asad-workstation:05715] [21] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [22] python(+0x18be79)[0x55c581b8ce79]
[asad-workstation:05715] [23] python(PyVectorcall_Call+0x71)[0x55c581b3e041]
[asad-workstation:05715] [24] python(_PyEval_EvalFrameDefault+0x1fdb)[0x55c581bc399b]
[asad-workstation:05715] [25] python(_PyFunction_Vectorcall+0x10b)[0x55c581b8c86b]
[asad-workstation:05715] [26] python(+0x10075e)[0x55c581b0175e]
[asad-workstation:05715] [27] python(+0x18bc0b)[0x55c581b8cc0b]
[asad-workstation:05715] [28] python(+0x10077f)[0x55c581b0177f]
[asad-workstation:05715] [29] python(_PyEval_EvalCodeWithName+0x2d2)[0x55c581b8ba92]
[asad-workstation:05715] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 2 with PID 0 on node asad-workstation exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
It would be very helpful if you help me resolve this issue. Thank You.
Regards,
Ali
Reply all
Reply to author
Forward
0 new messages