Need help in debugging the python core

[Amresh Sajjanshetty]

Dear All,
 
I’m using asyncio and paramiko to multiplex different channels into a single SSH connection. Things were working fine till recently but suddenly started seeing that python getting crashed whenever I tried to write to the channel. I have very limited knowledge on how python interpreter works, so I’m finding difficulty in understanding the stack trace. Can you please help in understanding the below backtarce.
 
bash-4.2$ gdb /usr/software/bin/python3.4.3 core.60015
Traceback (most recent call last):
  File "<string>", line 70, in <module>
  File "<string>", line 67, in GdbSetPythonDirectory
  File "/usr/software/share/gdb/python/gdb/__init__.py", line 19, in <module>
    import _gdb
ImportError: No module named _gdb
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/software/bin/python3.4.3...done.
 
warning: core file may not match specified executable file.
[New LWP 60015]
[New LWP 60018]
[New LWP 60019]
[New LWP 60020]
[New LWP 60021]
[New LWP 60022]
[New LWP 60023]
[New LWP 60024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/software/lib/libthread_db.so.1".
Core was generated by `/usr/software/bin/python3.4.3 /x/eng/bbrtp/users/amresh/sshproxy_3896926_160824'.
Program terminated with signal 11, Segmentation fault.
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
1159    Objects/obmalloc.c: No such file or directory.
(gdb) bt
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
#1  0x00007ff2e511474a in PyUnicode_New (maxchar=<optimized out>, size=3) at Objects/unicodeobject.c:1093
#2  PyUnicode_New (size=3, maxchar=<optimized out>) at Objects/unicodeobject.c:1033
#3  0x00007ff2e5139da2 in _PyUnicodeWriter_PrepareInternal (writer=writer@entry=0x7fff3d5c8640, length=<optimized out>, maxchar=<optimized out>, maxchar@entry=127) at Objects/unicodeobject.c:13327
#4  0x00007ff2e513f38b in PyUnicode_DecodeUTF8Stateful (s=s@entry=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=size@entry=3,
    errors=errors@entry=0x7ff2dee5dd70 "strict", consumed=consumed@entry=0x0) at Objects/unicodeobject.c:4757
#5  0x00007ff2e5140690 in PyUnicode_Decode (s=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=3, encoding=0x7ff2dee5df28 "utf-8", errors=0x7ff2dee5dd70 "strict")
    at Objects/unicodeobject.c:3012
#6  0x00007ff2de49bfdf in unpack_callback_raw (o=<synthetic pointer>, l=3, p=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", u=0x7fff3d5c8840, b=<optimized out>)
    at msgpack/unpack.h:229
#7  unpack_execute<true> (ctx=ctx@entry=0x7fff3d5c8840, data=0x7ff2e3572ec0 "\205\245_auth\300\245_call\246expect\243_i", <incomplete sequence \332>, len=<optimized out>, off=off@entry=0x7fff3d5c8820)
    at msgpack/unpack_template.h:312
#8  0x00007ff2de49fe3d in __pyx_pf_7msgpack_9_unpacker_2unpackb (__pyx_v_packed=__pyx_v_packed@entry=0x7ff2e3572ea0, __pyx_v_object_hook=__pyx_v_object_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>,
    __pyx_v_list_hook=__pyx_v_list_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>, __pyx_v_use_list=1, __pyx_v_encoding=0x7ff2dee5df08, __pyx_v_unicode_errors=0x7ff2dee5dd50,
    __pyx_v_object_pairs_hook=0x7ff2e54934b0 <_Py_NoneStruct>, __pyx_v_ext_hook=0x13db2d8, __pyx_v_max_str_len=__pyx_v_max_str_len@entry=2147483647,
    __pyx_v_max_bin_len=__pyx_v_max_bin_len@entry=2147483647, __pyx_v_max_array_len=2147483647, __pyx_v_max_map_len=2147483647, __pyx_v_max_ext_len=__pyx_v_max_ext_len@entry=2147483647,
    __pyx_self=<optimized out>) at msgpack/_unpacker.pyx:139
#9  0x00007ff2de4a1395 in __pyx_pw_7msgpack_9_unpacker_3unpackb (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at msgpack/_unpacker.pyx:102
#10 0x00007ff2e5174ed3 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fff3d5d2b80, func=0x7ff2df20ddc8) at Python/ceval.c:4463
#11 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d2b80) at Python/ceval.c:4264
#12 PyEval_EvalFrameEx (f=f@entry=0x7ff2def02208, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#13 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x7ff2deefec30, kwcount=0, defs=0x0,
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#14 0x00007ff2e51734da in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3d5d2e10, func=0x7ff2dee9b7b8) at Python/ceval.c:4344
#15 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d2e10) at Python/ceval.c:4262
#16 PyEval_EvalFrameEx (f=f@entry=0x7ff2deefea98, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#17 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x14566c8, kwcount=0,
    defs=0x7ff2deeaedb8, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#18 0x00007ff2e51734da in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3d5d30a0, func=0x7ff2dee2dd90) at Python/ceval.c:4344
#19 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d30a0) at Python/ceval.c:4262
#20 PyEval_EvalFrameEx (f=f@entry=0x1456478, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#21 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7ff2d87364c0, argcount=1, kws=kws@entry=0x7ff2dee1de40,
    kwcount=kwcount@entry=3, defs=defs@entry=0x7ff2e0820fd8, defcount=defcount@entry=3, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#22 0x00007ff2e50d3320 in function_call (func=0x7ff2df1e9a60, arg=0x7ff2d87364a8, kw=0x7ff2d8738248) at Objects/funcobject.c:632
#23 0x00007ff2e50a76ca in PyObject_Call (func=func@entry=0x7ff2df1e9a60, arg=arg@entry=0x7ff2d87364a8, kw=kw@entry=0x7ff2d8738248) at Objects/abstract.c:2040
#24 0x00007ff2e50be55d in method_call (func=0x7ff2df1e9a60, arg=0x7ff2d87364a8, kw=0x7ff2d8738248) at Objects/classobject.c:347
#25 0x00007ff2e50a76ca in PyObject_Call (func=0x7ff2dee30e88, arg=arg@entry=0x7ff2e433d048, kw=kw@entry=0x7ff2d8738248) at Objects/abstract.c:2040
#26 0x00007ff2e51d9301 in partial_call (pto=0x7ff2deee1db8, args=<optimized out>, kw=0x0) at ./Modules/_functoolsmodule.c:127
#27 0x00007ff2e50a76ca in PyObject_Call (func=func@entry=0x7ff2deee1db8, arg=arg@entry=0x7ff2e433d048, kw=kw@entry=0x0) at Objects/abstract.c:2040
#28 0x00007ff2e51700a0 in ext_do_call (nk=-466366392, na=0, flags=<optimized out>, pp_stack=0x7fff3d5d3540, func=0x7ff2deee1db8) at Python/ceval.c:4561
#29 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2878
#30 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3710, func=0x7ff2e1540730) at Python/ceval.c:4334
#31 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3710) at Python/ceval.c:4262
#32 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#33 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d38f0, func=0x7ff2e12f2f28) at Python/ceval.c:4334
#34 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d38f0) at Python/ceval.c:4262
#35 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#36 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3ad0, func=0x7ff2e12f0c80) at Python/ceval.c:4334
#37 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3ad0) at Python/ceval.c:4262
#38 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#39 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3cb0, func=0x7ff2df1e9ae8) at Python/ceval.c:4334
#40 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3cb0) at Python/ceval.c:4262
#41 PyEval_EvalFrameEx (f=f@entry=0xf796b8, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#42 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=_co@entry=0x7ff2e400c660, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488, args=args@entry=0x0, argcount=argcount@entry=0,
    kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:3588
#43 0x00007ff2e517601b in PyEval_EvalCode (co=co@entry=0x7ff2e400c660, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488) at Python/ceval.c:775
#44 0x00007ff2e519c09e in run_mod (arena=0xfc2990, flags=0x7fff3d5d3f50, locals=0x7ff2e42df488, globals=0x7ff2e42df488, filename=0x7ff2e41d64b0, mod=0x103b4f0) at Python/pythonrun.c:2180
#45 PyRun_FileExFlags (fp=fp@entry=0xf77b60, filename_str=filename_str@entry=0x7ff2e41d81d0 "/x/eng/bbrtp/users/amresh/sshproxy_3896926_1608240818/test/nate/lib/NATE/Service/SSHProxy.py",
    start=start@entry=257, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:2133
#46 0x00007ff2e519ced5 in PyRun_SimpleFileExFlags (fp=fp@entry=0xf77b60, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:1606
---Type <return> to continue, or q <return> to quit---
#47 0x00007ff2e519df09 in PyRun_AnyFileExFlags (fp=fp@entry=0xf77b60, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:1292
#48 0x00007ff2e51b6af5 in run_file (p_cf=0x7fff3d5d3f50, filename=0xf0e7f0 L"/x/eng/bbrtp/users/amresh/sshproxy_3896926_1608240818/test/nate/lib/NATE/Service/SSHProxy.py", fp=0xf77b60) at Modules/main.c:319
#49 Py_Main (argc=argc@entry=6, argv=argv@entry=0xee6010) at Modules/main.c:751
#50 0x0000000000400aa6 in main (argc=6, argv=<optimized out>) at ./Modules/python.c:69
(gdb)
 
Thanks and Regards,
Amresh