right list for SIGABRT python binary question ?

[Karsten Hilbert]

Hi all,

is this the right list to ask for help in debugging a
SIGABORT (?) happening on shutdown of a Python 2.7 script ?

If not, which one is ?

Karsten
--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

[Steve D'Aprano]

On Tue, 17 Oct 2017 07:32 am, Karsten Hilbert wrote:

> Hi all,
>
> is this the right list to ask for help in debugging a
> SIGABORT (?) happening on shutdown of a Python 2.7 script ?
>
> If not, which one is ?
>
> Karsten

You should try here first.

Please ensure you read and follow this first:

http://sscce.org/


and post a *minimum* example of your code. (Sorry for the redundant
instructions if you already know this, but you would be surprised how many
people don't.)


--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Karsten Hilbert]

On Tue, Oct 17, 2017 at 12:04:09PM +1100, Steve D'Aprano wrote:

> > is this the right list to ask for help in debugging a
> > SIGABORT (?) happening on shutdown of a Python 2.7 script ?
> >
> > If not, which one is ?
>

> You should try here first.
>
> Please ensure you read and follow this first:
>
> http://sscce.org/
>
> and post a *minimum* example of your code. (Sorry for the redundant
> instructions if you already know this, but you would be surprised how many
> people don't.)

Thanks. I'll work towards a postable example.

Running a debug build of py27 gave me a first lead: this
Debian system (Testing, upgraded all the way from various
releases ago) carries an incompatible mxDateTime which I'll
take care of.

*** You don't have the (right) mxDateTime binaries installed !
Traceback (most recent call last):
File "./bootstrap_gm_db_system.py", line 87, in <module>
from Gnumed.pycommon import gmCfg2, gmPsql, gmPG2, gmTools, gmI18N
File "/home/ncq/Projekte/gm-git/gnumed/gnumed/Gnumed/pycommon/gmPG2.py", line 34, in <module>
from Gnumed.pycommon import gmDateTime
File "/home/ncq/Projekte/gm-git/gnumed/gnumed/Gnumed/pycommon/gmDateTime.py", line 52, in <module>
import mx.DateTime as mxDT
File "/usr/lib/python2.7/dist-packages/mx/DateTime/__init__.py", line 8, in <module>
from DateTime import *
File "/usr/lib/python2.7/dist-packages/mx/DateTime/DateTime.py", line 9, in <module>
from mxDateTime import *
File "/usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/__init__.py", line 13, in <module>
raise ImportError, why
ImportError: /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so: undefined symbol: Py_InitModule4

[Karsten Hilbert]

OK, here's the first bit which might be part of the puzzle of
why my Python script SIGABRT's.

When run under "normal" py2.7 it runs all the way through but
upon shutdown (*after* sys.exit(0)) faulthandler shows a
problem (and returns 134 which made me think of SIGABRT):

*** Error in `python': free(): invalid pointer: 0x00770b14 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6738a)[0xb7ccc38a]
/lib/i386-linux-gnu/libc.so.6(+0x6dfc7)[0xb7cd2fc7]
/lib/i386-linux-gnu/libc.so.6(+0x6e806)[0xb7cd3806]
python(PyObject_Free+0xfb)[0x44affb]
python(+0x110b51)[0x534b51]
python(+0x110b51)[0x534b51]
python(+0x110b51)[0x534b51]
python(+0xf3ea7)[0x517ea7]
python(PyDict_SetItem+0x201)[0x4d4f81]
python(_PyModule_Clear+0x150)[0x539630]
python(PyImport_Cleanup+0x61d)[0x53927d]
python(Py_Finalize+0x9d)[0x53635d]
python(Py_Exit+0x14)[0x55cb24]
python(+0x136102)[0x55a102]
python(PyErr_PrintEx+0x35)[0x559975]
python(+0x3dd41)[0x461d41]
python(Py_Main+0x753)[0x4d2c83]
python(main+0x1b)[0x4d251b]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf6)[0xb7c7d286]
python(+0xae3f3)[0x4d23f3]
======= Memory map: ========
00424000-00763000 r-xp 00000000 08:01 6285092 /usr/bin/python2.7
00763000-00764000 r--p 0033e000 08:01 6285092 /usr/bin/python2.7
00764000-007c4000 rw-p 0033f000 08:01 6285092 /usr/bin/python2.7
007c4000-007d9000 rw-p 00000000 00:00 0
026f1000-02921000 rw-p 00000000 00:00 0 [heap]
b6800000-b6821000 rw-p 00000000 00:00 0
b6821000-b6900000 ---p 00000000 00:00 0
b6995000-b69b0000 r-xp 00000000 08:01 8151085 /lib/i386-linux-gnu/libgcc_s.so.1
b69b0000-b69b1000 r--p 0001a000 08:01 8151085 /lib/i386-linux-gnu/libgcc_s.so.1
b69b1000-b69b2000 rw-p 0001b000 08:01 8151085 /lib/i386-linux-gnu/libgcc_s.so.1
b69f5000-b6b35000 rw-p 00000000 00:00 0
b6b35000-b6b40000 r-xp 00000000 08:01 8151337 /lib/i386-linux-gnu/libnss_files-2.24.so
b6b40000-b6b41000 r--p 0000a000 08:01 8151337 /lib/i386-linux-gnu/libnss_files-2.24.so
b6b41000-b6b42000 rw-p 0000b000 08:01 8151337 /lib/i386-linux-gnu/libnss_files-2.24.so
b6b42000-b6b48000 rw-p 00000000 00:00 0
b6b48000-b6b53000 r-xp 00000000 08:01 8151339 /lib/i386-linux-gnu/libnss_nis-2.24.so
b6b53000-b6b54000 r--p 0000a000 08:01 8151339 /lib/i386-linux-gnu/libnss_nis-2.24.so
b6b54000-b6b55000 rw-p 0000b000 08:01 8151339 /lib/i386-linux-gnu/libnss_nis-2.24.so
b6b64000-b6b6b000 r--p 00000000 08:01 6286752 /usr/share/locale/de/LC_MESSAGES/libpq5-10.mo
b6b6b000-b6b72000 r--s 00000000 08:01 6903281 /usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
b6b72000-b6b98000 r--p 00000000 08:01 6288754 /usr/share/locale/de/LC_MESSAGES/libc.mo
b6b98000-b6c98000 rw-p 00000000 00:00 0
b6c9e000-b6cb4000 r-xp 00000000 08:01 8151334 /lib/i386-linux-gnu/libnsl-2.24.so
b6cb4000-b6cb5000 r--p 00016000 08:01 8151334 /lib/i386-linux-gnu/libnsl-2.24.so
b6cb5000-b6cb6000 rw-p 00017000 08:01 8151334 /lib/i386-linux-gnu/libnsl-2.24.so
b6cb6000-b6cb8000 rw-p 00000000 00:00 0
b6cb8000-b6cc0000 r-xp 00000000 08:01 8151335 /lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc0000-b6cc1000 r--p 00007000 08:01 8151335 /lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc1000-b6cc2000 rw-p 00008000 08:01 8151335 /lib/i386-linux-gnu/libnss_compat-2.24.so
b6cc2000-b6d02000 rw-p 00000000 00:00 0
b6d02000-b6d09000 r-xp 00000000 08:01 6899491 /usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d09000-b6d0a000 r--p 00006000 08:01 6899491 /usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d0a000-b6d0b000 rw-p 00007000 08:01 6899491 /usr/lib/i386-linux-gnu/libffi.so.6.0.4
b6d0b000-b6d96000 r-xp 00000000 08:01 6899509 /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d96000-b6d97000 r--p 0008a000 08:01 6899509 /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d97000-b6d98000 rw-p 0008b000 08:01 6899509 /usr/lib/i386-linux-gnu/libgmp.so.10.3.2
b6d98000-b6dcc000 r-xp 00000000 08:01 6899139 /usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dcc000-b6dcd000 r--p 00033000 08:01 6899139 /usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dcd000-b6dce000 rw-p 00034000 08:01 6899139 /usr/lib/i386-linux-gnu/libhogweed.so.4.3
b6dce000-b6e08000 r-xp 00000000 08:01 6897991 /usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e08000-b6e09000 ---p 0003a000 08:01 6897991 /usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e09000-b6e0a000 r--p 0003a000 08:01 6897991 /usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e0a000-b6e0b000 rw-p 0003b000 08:01 6897991 /usr/lib/i386-linux-gnu/libnettle.so.6.3
b6e0b000-b6e1e000 r-xp 00000000 08:01 6898338 /usr/lib/i386-linux-gnu/libtasn1.so.6.5.4
b6e1e000-b6e1f000 r--p 00012000 08:01 6898338 /usr/lib/i386-linux-gnu/libtasn1.so.6.5.4
b6e1f000-b6e20000 rw-p 00013000 08:01 6898338 /usr/lib/i386-linux-gnu/libtasn1.so.6.5.4
b6e20000-b6f8c000 r-xp 00000000 08:01 6899693 /usr/lib/i386-linux-gnu/libunistring.so.2.0.0
b6f8c000-b6f8d000 ---p 0016c000 08:01 6899693 /usr/lib/i386-linux-gnu/libunistring.so.2.0.0
b6f8d000-b6f8f000 r--p 0016c000 08:01 6899693 /usr/lib/i386-linux-gnu/libunistring.so.2.0.0
b6f8f000-b6f90000 rw-p 0016e000 08:01 6899693 /usr/lib/i386-linux-gnu/libunistring.so.2.0.0
b6f90000-b6fac000 r-xp 00000000 08:01 6899147 /usr/lib/i386-linux-gnu/libidn2.so.0.3.1
b6fac000-b6fad000 r--p 0001b000 08:01 6899147 /usr/lib/i386-linux-gnu/libidn2.so.0.3.1
b6fad000-b6fae000 rw-p 0001c000 08:01 6899147 /usr/lib/i386-linux-gnu/libidn2.so.0.3.1
b6fae000-b70fe000 r-xp 00000000 08:01 6899036 /usr/lib/i386-linux-gnu/libp11-kit.so.0.3.0
b70fe000-b7104000 r--p 0014f000 08:01 6899036 /usr/lib/i386-linux-gnu/libp11-kit.so.0.3.0
b7104000-b7109000 rw-p 00155000 08:01 6899036 /usr/lib/i386-linux-gnu/libp11-kit.so.0.3.0
b7109000-b710a000 rw-p 00000000 00:00 0
b710a000-b7299000 r-xp 00000000 08:01 6898093 /usr/lib/i386-linux-gnu/libgnutls.so.30.14.7
b7299000-b72a1000 r--p 0018e000 08:01 6898093 /usr/lib/i386-linux-gnu/libgnutls.so.30.14.7
b72a1000-b72a2000 rw-p 00196000 08:01 6898093 /usr/lib/i386-linux-gnu/libgnutls.so.30.14.7
b72a2000-b72a3000 rw-p 00000000 00:00 0
b72a3000-b72bf000 r-xp 00000000 08:01 6897853 /usr/lib/i386-linux-gnu/libsasl2.so.2.0.25
b72bf000-b72c0000 r--p 0001b000 08:01 6897853 /usr/lib/i386-linux-gnu/libsasl2.so.2.0.25
b72c0000-b72c1000 rw-p 0001c000 08:01 6897853 /usr/lib/i386-linux-gnu/libsasl2.so.2.0.25
b72c1000-b72d5000 r-xp 00000000 08:01 8151343 /lib/i386-linux-gnu/libresolv-2.24.so
b72d5000-b72d6000 r--p 00013000 08:01 8151343 /lib/i386-linux-gnu/libresolv-2.24.so
b72d6000-b72d7000 rw-p 00014000 08:01 8151343 /lib/i386-linux-gnu/libresolv-2.24.so
b72d7000-b72d9000 rw-p 00000000 00:00 0
b72d9000-b730b000 r-xp 00000000 08:01 6898091 /usr/lib/i386-linux-gnu/libk5crypto.so.3.1
b730b000-b730c000 ---p 00032000 08:01 6898091 /usr/lib/i386-linux-gnu/libk5crypto.so.3.1
b730c000-b730d000 r--p 00032000 08:01 6898091 /usr/lib/i386-linux-gnu/libk5crypto.so.3.1
b730d000-b730e000 rw-p 00033000 08:01 6898091 /usr/lib/i386-linux-gnu/libk5crypto.so.3.1
b730e000-b730f000 rw-p 00000000 00:00 0
b730f000-b73e2000 r-xp 00000000 08:01 6898964 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
b73e2000-b73e3000 ---p 000d3000 08:01 6898964 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
b73e3000-b73e9000 r--p 000d3000 08:01 6898964 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
b73e9000-b73eb000 rw-p 000d9000 08:01 6898964 /usr/lib/i386-linux-gnu/libkrb5.so.3.3
b73eb000-b7441000 r-xp 00000000 08:01 6899719 /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2.10.8
b7441000-b7442000 ---p 00056000 08:01 6899719 /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2.10.8
b7442000-b7443000 r--p 00056000 08:01 6899719 /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2.10.8
b7443000-b7444000 rw-p 00057000 08:01 6899719 /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2.10.8
b7444000-b7445000 rw-p 00000000 00:00 0
b7445000-b7495000 r-xp 00000000 08:01 6898215 /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2.2
b7495000-b7496000 r--p 0004f000 08:01 6898215 /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2.2
b7496000-b7497000 rw-p 00050000 08:01 6898215 /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2.2
b7497000-b74e1000 r-xp 00000000 08:01 6899474 /usr/lib/i386-linux-gnu/libpq.so.5.10
b74e1000-b74e3000 r--p 00049000 08:01 6899474 /usr/lib/i386-linux-gnu/libpq.so.5.10
b74e3000-b74e4000 rw-p 0004b000 08:01 6899474 /usr/lib/i386-linux-gnu/libpq.so.5.10
b74e8000-b74ed000 r-xp 00000000 08:01 6365428 /usr/lib/python2.7/dist-packages/faulthandler.i386-linux-gnu.so
b74ed000-b74ee000 r--p 00004000 08:01 6365428 /usr/lib/python2.7/dist-packages/faulthandler.i386-linux-gnu.so
b74ee000-b74ef000 rw-p 00005000 08:01 6365428 /usr/lib/python2.7/dist-packages/faulthandler.i386-linux-gnu.so
b74ef000-b74f6000 r-xp 00000000 08:01 6367548 /usr/lib/python2.7/lib-dynload/_csv.i386-linux-gnu.so
b74f6000-b74f7000 r--p 00006000 08:01 6367548 /usr/lib/python2.7/lib-dynload/_csv.i386-linux-gnu.so
b74f7000-b74f9000 rw-p 00007000 08:01 6367548 /usr/lib/python2.7/lib-dynload/_csv.i386-linux-gnu.so
b74f9000-b750c000 r-xp 00000000 08:01 6367559 /usr/lib/python2.7/lib-dynload/_json.i386-linux-gnu.so
b750c000-b750d000 r--p 00012000 08:01 6367559 /usr/lib/python2.7/lib-dynload/_json.i386-linux-gnu.so
b750d000-b750e000 rw-p 00013000 08:01 6367559 /usr/lib/python2.7/lib-dynload/_json.i386-linux-gnu.so
b750e000-b7523000 r-xp 00000000 08:01 6367569 /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
b7523000-b7524000 r--p 00014000 08:01 6367569 /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
b7524000-b7527000 rw-p 00015000 08:01 6367569 /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
b7527000-b7559000 r-xp 00000000 08:01 6479968 /usr/lib/python2.7/dist-packages/psycopg2/_psycopg.i386-linux-gnu.so
b7559000-b755a000 r--p 00031000 08:01 6479968 /usr/lib/python2.7/dist-packages/psycopg2/_psycopg.i386-linux-gnu.so
b755a000-b755e000 rw-p 00032000 08:01 6479968 /usr/lib/python2.7/dist-packages/psycopg2/_psycopg.i386-linux-gnu.so
b755e000-b756e000 r-xp 00000000 08:01 8151113 /lib/i386-linux-gnu/libbz2.so.1.0.4
b756e000-b756f000 r--p 0000f000 08:01 8151113 /lib/i386-linux-gnu/libbz2.so.1.0.4
b756f000-b7570000 rw-p 00010000 08:01 8151113 /lib/i386-linux-gnu/libbz2.so.1.0.4
b7576000-b7584000 r-xp 00000000 08:01 6897979 /usr/lib/i386-linux-gnu/liblber-2.4.so.2.10.8
b7584000-b7585000 r--p 0000d000 08:01 6897979 /usr/lib/i386-linux-gnu/liblber-2.4.so.2.10.8
b7585000-b7586000 rw-p 0000e000 08:01 6897979 /usr/lib/i386-linux-gnu/liblber-2.4.so.2.10.8
b7586000-b7589000 r-xp 00000000 08:01 8151178 /lib/i386-linux-gnu/libkeyutils.so.1.5
b7589000-b758a000 r--p 00002000 08:01 8151178 /lib/i386-linux-gnu/libkeyutils.so.1.5
b758a000-b758b000 rw-p 00003000 08:01 8151178 /lib/i386-linux-gnu/libkeyutils.so.1.5
b758b000-b7596000 r-xp 00000000 08:01 6899192 /usr/lib/i386-linux-gnu/libkrb5support.so.0.1
b7596000-b7597000 r--p 0000a000 08:01 6899192 /usr/lib/i386-linux-gnu/libkrb5support.so.0.1
b7597000-b7598000 rw-p 0000b000 08:01 6899192 /usr/lib/i386-linux-gnu/libkrb5support.so.0.1
b7598000-b759f000 r-xp 00000000 08:01 8151344 /lib/i386-linux-gnu/librt-2.24.so
b759f000-b75a0000 r--p 00006000 08:01 8151344 /lib/i386-linux-gnu/librt-2.24.so
b75a0000-b75a1000 rw-p 00007000 08:01 8151344 /lib/i386-linux-gnu/librt-2.24.so
b75a1000-b75b1000 r-xp 00000000 08:01 6570027 /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
b75b1000-b75b2000 r--p 0000f000 08:01 6570027 /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
b75b2000-b75b3000 rw-p 00010000 08:01 6570027 /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so
b75b3000-b77fe000 r-xp 00000000 08:01 6897994 /usr/lib/i386-linux-gnu/libcrypto.so.1.1
b77fe000-b77ff000 ---p 0024b000 08:01 6897994 /usr/lib/i386-linux-gnu/libcrypto.so.1.1
b77ff000-b7810000 r--p 0024b000 08:01 6897994 /usr/lib/i386-linux-gnu/libcrypto.so.1.1
b7810000-b7817000 rw-p 0025c000 08:01 6897994 /usr/lib/i386-linux-gnu/libcrypto.so.1.1
b7817000-b781a000 rw-p 00000000 00:00 0
b781a000-b7881000 r-xp 00000000 08:01 6899137 /usr/lib/i386-linux-gnu/libssl.so.1.1
b7881000-b7884000 r--p 00066000 08:01 6899137 /usr/lib/i386-linux-gnu/libssl.so.1.1
b7884000-b7888000 rw-p 00069000 08:01 6899137 /usr/lib/i386-linux-gnu/libssl.so.1.1
b7888000-b7988000 rw-p 00000000 00:00 0
b798a000-b798d000 r-xp 00000000 08:01 8151240 /lib/i386-linux-gnu/libcom_err.so.2.1
b798d000-b798e000 r--p 00002000 08:01 8151240 /lib/i386-linux-gnu/libcom_err.so.2.1
b798e000-b798f000 rw-p 00003000 08:01 8151240 /lib/i386-linux-gnu/libcom_err.so.2.1
b798f000-b7998000 r-xp 00000000 08:01 6367575 /usr/lib/python2.7/lib-dynload/bz2.i386-linux-gnu.so
b7998000-b7999000 r--p 00008000 08:01 6367575 /usr/lib/python2.7/lib-dynload/bz2.i386-linux-gnu.so
b7999000-b799b000 rw-p 00009000 08:01 6367575 /usr/lib/python2.7/lib-dynload/bz2.i386-linux-gnu.so
b799b000-b79fc000 rw-p 00000000 00:00 0
b79fc000-b7a00000 r-xp 00000000 08:01 6367555 /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
b7a00000-b7a01000 r--p 00003000 08:01 6367555 /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
b7a01000-b7a02000 rw-p 00004000 08:01 6367555 /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
b7a02000-b7a05000 r-xp 00000000 08:01 6367622 /usr/lib/python2.7/lib-dynload/termios.i386-linux-gnu.so
b7a05000-b7a06000 r--p 00002000 08:01 6367622 /usr/lib/python2.7/lib-dynload/termios.i386-linux-gnu.so
b7a06000-b7a08000 rw-p 00003000 08:01 6367622 /usr/lib/python2.7/lib-dynload/termios.i386-linux-gnu.so
b7a08000-b7a48000 rw-p 00000000 00:00 0
b7a48000-b7be3000 r--p 00000000 08:01 6299936 /usr/lib/locale/locale-archive
b7be3000-b7c65000 rw-p 00000000 00:00 0
b7c65000-b7e16000 r-xp 00000000 08:01 8151265 /lib/i386-linux-gnu/libc-2.24.so
b7e16000-b7e18000 r--p 001b0000 08:01 8151265 /lib/i386-linux-gnu/libc-2.24.so
b7e18000-b7e19000 rw-p 001b2000 08:01 8151265 /lib/i386-linux-gnu/libc-2.24.so
b7e19000-b7e1c000 rw-p 00000000 00:00 0
b7e1c000-b7e6f000 r-xp 00000000 08:01 8151332 /lib/i386-linux-gnu/libm-2.24.so
b7e6f000-b7e70000 r--p 00052000 08:01 8151332 /lib/i386-linux-gnu/libm-2.24.so
b7e70000-b7e71000 rw-p 00053000 08:01 8151332 /lib/i386-linux-gnu/libm-2.24.so
b7e71000-b7e8a000 r-xp 00000000 08:01 8151148 /lib/i386-linux-gnu/libz.so.1.2.8
b7e8a000-b7e8b000 r--p 00018000 08:01 8151148 /lib/i386-linux-gnu/libz.so.1.2.8
b7e8b000-b7e8c000 rw-p 00019000 08:01 8151148 /lib/i386-linux-gnu/libz.so.1.2.8
b7e8c000-b7e8e000 r-xp 00000000 08:01 8151355 /lib/i386-linux-gnu/libutil-2.24.so
b7e8e000-b7e8f000 r--p 00001000 08:01 8151355 /lib/i386-linux-gnu/libutil-2.24.so
b7e8f000-b7e90000 rw-p 00002000 08:01 8151355 /lib/i386-linux-gnu/libutil-2.24.so
b7e90000-b7e93000 r-xp 00000000 08:01 8151331 /lib/i386-linux-gnu/libdl-2.24.so
b7e93000-b7e94000 r--p 00002000 08:01 8151331 /lib/i386-linux-gnu/libdl-2.24.so
b7e94000-b7e95000 rw-p 00003000 08:01 8151331 /lib/i386-linux-gnu/libdl-2.24.so
b7e95000-b7eae000 r-xp 00000000 08:01 8151342 /lib/i386-linux-gnu/libpthread-2.24.so
b7eae000-b7eaf000 r--p 00018000 08:01 8151342 /lib/i386-linux-gnu/libpthread-2.24.so
b7eaf000-b7eb0000 rw-p 00019000 08:01 8151342 /lib/i386-linux-gnu/libpthread-2.24.so
b7eb0000-b7eb2000 rw-p 00000000 00:00 0
b7eb4000-b7ef8000 rw-p 00000000 00:00 0
b7ef8000-b7efb000 r--p 00000000 00:00 0 [vvar]
b7efb000-b7efd000 r-xp 00000000 00:00 0 [vdso]
b7efd000-b7f20000 r-xp 00000000 08:01 8151050 /lib/i386-linux-gnu/ld-2.24.so
b7f20000-b7f21000 r--p 00022000 08:01 8151050 /lib/i386-linux-gnu/ld-2.24.so
b7f21000-b7f22000 rw-p 00023000 08:01 8151050 /lib/i386-linux-gnu/ld-2.24.so
bf9f5000-bfa16000 rw-p 00000000 00:00 0 [stack]
Fatal Python error: Aborted

Current thread 0xb7c63d00 (most recent call first):

When run under a debug build it sayeth right away (simulated
as a minimal working example):

root@hermes:~/bin# python2.7-dbg
Python 2.7.14 (default, Sep 17 2017, 18:50:44)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mx.DateTime

*** You don't have the (right) mxDateTime binaries installed !
Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/usr/lib/python2.7/dist-packages/mx/DateTime/__init__.py", line 8, in <module>
from DateTime import *
File "/usr/lib/python2.7/dist-packages/mx/DateTime/DateTime.py", line 9, in <module>
from mxDateTime import *
File "/usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/__init__.py", line 13, in <module>
raise ImportError, why
ImportError: /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so: undefined symbol: Py_InitModule4

[47287 refs]
>>> exit()
[21664 refs]
root@hermes:~/bin# apt-cache policy python-egenix-mxdatetime
python-egenix-mxdatetime:
Installiert: 3.2.9-1
Installationskandidat: 3.2.9-1
Versionstabelle:
*** 3.2.9-1 990
500 http://httpredir.debian.org/debian stretch/main i386 Packages
990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status
root@hermes:~/bin#

Does this ring a bell or does anyone know what I should
further look into ?

For good measure I have filed a bug with Debian asking for
recompilation of python-egenix-mxdatetime.

Next, I'll remove mxDateTime from the script and see what
prevails.

Thanks,

[Thomas Jollans]

I don't really what exactly is going on here, but in general extension
modules compiled for non-debug builds won't work with debug builds. The
Debian package is almost certainly fine.

> For good measure I have filed a bug with Debian asking for
> recompilation of python-egenix-mxdatetime.

Even if the maintainers do that, it won't help. Check that the module
works on its own with the regular Python build and then close the bug if
the maintainers don't beat you to it.

> When run under "normal" py2.7 it runs all the way through but
> upon shutdown (*after* sys.exit(0)) faulthandler shows a
> problem (and returns 134 which made me think of SIGABRT):

We still don't know what "it" is.

Strip down your script as much as possible. It looks like you're using a
lot of extension modules, and any one of them could (in theory) be
causing problems.

--
Thomas Jollans

[dieter]

Karsten Hilbert <Karsten...@gmx.net> writes:
> ...

> When run under "normal" py2.7 it runs all the way through but
> upon shutdown (*after* sys.exit(0)) faulthandler shows a
> problem (and returns 134 which made me think of SIGABRT):
>
> *** Error in `python': free(): invalid pointer: 0x00770b14 ***

This indicates some form of memory corruption, usually caused by
some C extension. Unfortunately, memory corruption is rarely noticed
locally; therefore, the localization is typically complex.

Maybe, you are lucky and the "mxdatetime" is to blame.

[Karsten Hilbert]

On Wed, Oct 18, 2017 at 02:07:46PM +0200, Thomas Jollans wrote:

> > When run under a debug build it sayeth right away (simulated
> > as a minimal working example):
> >
> > root@hermes:~/bin# python2.7-dbg
> > Python 2.7.14 (default, Sep 17 2017, 18:50:44)
> > [GCC 7.2.0] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import mx.DateTime
> > *** You don't have the (right) mxDateTime binaries installed !
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > File "/usr/lib/python2.7/dist-packages/mx/DateTime/__init__.py", line 8, in <module>
> > from DateTime import *
> > File "/usr/lib/python2.7/dist-packages/mx/DateTime/DateTime.py", line 9, in <module>
> > from mxDateTime import *
> > File "/usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/__init__.py", line 13, in <module>
> > raise ImportError, why
> > ImportError: /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so: undefined symbol: Py_InitModule4
>
> I don't really what exactly is going on here, but in general extension
> modules compiled for non-debug builds won't work with debug builds.

OK, that helps. I found -dbg builds of mx.DateTime in Debian.

> > For good measure I have filed a bug with Debian asking for
> > recompilation of python-egenix-mxdatetime.
>
> Even if the maintainers do that, it won't help. Check that the module
> works on its own with the regular Python build and then close the bug if
> the maintainers don't beat you to it.

Done.

> > When run under "normal" py2.7 it runs all the way through but
> > upon shutdown (*after* sys.exit(0)) faulthandler shows a
> > problem (and returns 134 which made me think of SIGABRT):
>

> We still don't know what "it" is.
>
> Strip down your script as much as possible. It looks like you're using a
> lot of extension modules, and any one of them could (in theory) be
> causing problems.

I know. However, stripping down isn't quite so easy (it never
is, which is a rather lame excuse for not doing so yet :-)

The script itself is but a meager 1843 lines (but using lots
of Python modules, both stdlib and my own). What it does is
take a large number of SQL files (roughly a thousand) and
replaying them against PostgreSQL thereby bootstrapping a
database layout from scratch up to the current version (21).
The abort-on-exit only happens if "enough" SQL scripts have
been run (sounds like a resource leak) and the bootstrapper
script exits (having encountered a database problem or not
makes no difference). Running various parts of the whole
procedure does not make a difference as to whether the fault
occurs when exiting, if only "enough" SQL scripts are run.

I am currently running the bootstrapper with mxdatetime as a
dbg build to see what gives. The only other C extension I am
aware of that is in use is psycopg2.

However, none of these two (nor any other modules) have been
added to the bootstrapper (or updated) recently (in fact,
this setup has been in use for, what, a decade?) so it is
quite unclear why they should be at fault now.

The one thing that did change is PostgreSQL going from 9.6 to
10, effecting a libpq5 recompiled against the newer PG
client. I wonder whether psycopg2 needs a recompile against
that libpq5 (despite there not having been changes advertised
by PG).

I guess I'll have to downgrade libpq5 to 9.6 and see what
happens.

I'll report what I find.

(oh, and it's all available on github)

Thanks,

[Karsten Hilbert]

On Thu, Oct 19, 2017 at 07:27:45PM +0200, Karsten Hilbert wrote:

> I am currently running the bootstrapper with mxdatetime as a
> dbg build to see what gives. The only other C extension I am
> aware of that is in use is psycopg2.

So here's the final console output of that:

==> bootstrapping "v20_fixups-pre_v21" ...
==> dropping pre-existing target database [gnumed_v21] ...
==> cloning [gnumed_v20] (72 MB) as target database [gnumed_v21] ...
==> reindexing target database (can take a while) ...
==> transferring users ...
==> bootstrapping "v20-v21-static" ...
==> bootstrapping "v20-v21-dynamic" ...
==> bootstrapping "v21-fixups" ...
==> setting up auditing ...
==> setting up encounter/episode FKs and IDXs ...
==> setting up encounter/episode FK sanity check triggers ...
==> setting up generic notifications ...
==> upgrading reference data sets ...
==> verifying target database schema ...
==> checking migrated data for plausibility ...
Done bootstrapping GNUmed database: We very likely succeeded.
log: /home/ncq/Projekte/gm-git/gnumed/gnumed/server/bootstrap/bootstrap-latest.log
Debug memory block at address p=0x717b7c: API ''
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
at p-3: 0x03 *** OUCH
at p-2: 0x4e *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0x717b7c are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API '', verified using API 'o'
./bootstrap-latest.sh: Zeile 80: 28023 Abgebrochen ./bootstrap_gm_db_system.py --log-file=${LOG} --conf-file=${CONF} --${QUIET}
Bootstrapping "gnumed_v21" did not finish successfully (134). Aborting.

Note there's not faulthandler output here because I don't yet
know how to get a debug build of *that* (Debian does not seem
to have one and neither does pypi to my knowledge).

That'd be needed because of:

root@hermes:~/bin# python2.7-dbg
Python 2.7.14 (default, Sep 17 2017, 18:50:44)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import faulthandler

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

ImportError: /usr/lib/python2.7/dist-packages/faulthandler.i386-linux-gnu.so: undefined symbol: Py_InitModule4
[45316 refs]
>>>

Can anyone give more guidance on what the above python debug
output might vaguely point to ?

Thanks,

[dieter]

Karsten Hilbert <Karsten...@gmx.net> writes:
> ...

> So here's the final console output of that:

> ...

> Debug memory block at address p=0x717b7c: API ''
> 0 bytes originally requested
> The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
> at p-3: 0x03 *** OUCH
> at p-2: 0x4e *** OUCH
> at p-1: 0x00 *** OUCH
> Because memory is corrupted at the start, the count of bytes requested
> may be bogus, and checking the trailing pad bytes may segfault.
> The 4 pad bytes at tail=0x717b7c are not all FORBIDDENBYTE (0xfb):
> at tail+0: 0x00 *** OUCH
> at tail+1: 0x00 *** OUCH
> at tail+2: 0x00 *** OUCH
> at tail+3: 0x00 *** OUCH
> The block was made by call #0 to debug malloc/realloc.
> Fatal Python error: bad ID: Allocated using API '', verified using API 'o'

> ...

> Can anyone give more guidance on what the above python debug
> output might vaguely point to ?

It points to a memory corruption.

I would approach the problem by means of debugging: put a write
breakpoint at the corrupted address "0x717b7c" and check what part
of the system accesses it (this assumes you are using a CPU
supporting write breakpoints).
It may be very tedious as the address might be accessed very often
legally before it gets corrupted.

Another approach may be to use a tool designed for memory debugging,
e.g. "valgrind".

[M.-A. Lemburg]

On 17.10.2017 09:30, Karsten Hilbert wrote:
> On Tue, Oct 17, 2017 at 12:04:09PM +1100, Steve D'Aprano wrote:
>

>>> is this the right list to ask for help in debugging a
>>> SIGABORT (?) happening on shutdown of a Python 2.7 script ?
>>>
>>> If not, which one is ?
>>

>> You should try here first.
>>
>> Please ensure you read and follow this first:
>>
>> http://sscce.org/
>>
>> and post a *minimum* example of your code. (Sorry for the redundant
>> instructions if you already know this, but you would be surprised how many
>> people don't.)
>

> Thanks. I'll work towards a postable example.
>
> Running a debug build of py27 gave me a first lead: this
> Debian system (Testing, upgraded all the way from various
> releases ago) carries an incompatible mxDateTime which I'll
> take care of.
>

> *** You don't have the (right) mxDateTime binaries installed !

> Traceback (most recent call last):

> File "./bootstrap_gm_db_system.py", line 87, in <module>
> from Gnumed.pycommon import gmCfg2, gmPsql, gmPG2, gmTools, gmI18N
> File "/home/ncq/Projekte/gm-git/gnumed/gnumed/Gnumed/pycommon/gmPG2.py", line 34, in <module>
> from Gnumed.pycommon import gmDateTime
> File "/home/ncq/Projekte/gm-git/gnumed/gnumed/Gnumed/pycommon/gmDateTime.py", line 52, in <module>
> import mx.DateTime as mxDT

> File "/usr/lib/python2.7/dist-packages/mx/DateTime/__init__.py", line 8, in <module>
> from DateTime import *
> File "/usr/lib/python2.7/dist-packages/mx/DateTime/DateTime.py", line 9, in <module>
> from mxDateTime import *
> File "/usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/__init__.py", line 13, in <module>
> raise ImportError, why
> ImportError: /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so: undefined symbol: Py_InitModule4

This error suggests that you have 32- and 64-bit versions of
Python and mxDateTime mixed in your installation.

Py_InitModule4 is only available in the 32-bit build of
Python. With the 64-bit build, it's called Py_InitModule4_64.

Since you're getting the same error from faulthandler,
this is where I'd start to investigate.

"nm" will list all exported and required symbols. As first step,
you should probably check the python binary for its symbols and
see whether it exports Py_InitModule* symbols.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 21 2017)
>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> Python Database Interfaces ... http://products.egenix.com/
>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
http://www.malemburg.com/

[Karsten Hilbert]

Thanks for your input !

The python2.7-dbg build is 32 bits:

root@hermes:~# nm /usr/bin/python2.7-dbg | grep Py_InitM
00155b9f T Py_InitModule4TraceRefs


python2.7-dbg:
Installiert: 2.7.14-2
Installationskandidat: 2.7.14-2
Versionstabelle:
*** 2.7.14-2 500

500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

2.7.13-2 990

500 http://httpredir.debian.org/debian stretch/main i386 Packages
990 http://httpredir.debian.org/debian buster/main i386 Packages

The python2.7 build (no -dbg) does not have symbols.

mxDateTime really should be 32 bits, too:

python-egenix-mxdatetime:
Installiert: 3.2.9-1
Installationskandidat: 3.2.9-1
Versionstabelle:
*** 3.2.9-1 990
500 http://httpredir.debian.org/debian stretch/main i386 Packages
990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

Let me check the .so file:

root@hermes:~# nm /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime_d.so | grep Py_InitM
U Py_InitModule4TraceRefs

It seems it is - hm ...

Thanks,

[Karsten Hilbert]

On Sun, Oct 22, 2017 at 10:15:51PM +0200, Karsten Hilbert wrote:

> The python2.7-dbg build is 32 bits:

...

> The python2.7 build (no -dbg) does not have symbols.

More to the point:

/usr/bin/python2.7: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=91226399fb434d138f166a5c8eca04385ea2e41f, stripped

/usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=f09c7cc78b41421d3cee1f418b4d45772a3db701, stripped

Regards,

[Karsten Hilbert]

Hi Dieter,

thanks for your guidance. I fear this approach is out of my class.

[Karsten Hilbert]

On Sun, Oct 22, 2017 at 11:10:33PM +0200, Karsten Hilbert wrote:

>> It points to a memory corruption.
>

> thanks for your guidance. I fear this approach is out of my class.

For what it's worth I have run a successful overnight memory
stress test (memtest86+) so it shouldn't be a bad bit in
hardware.

[Karsten Hilbert]

On Sat, Oct 21, 2017 at 09:31:54AM +0200, dieter wrote:

> It points to a memory corruption.

While running valgrind and friends is beyond my capabilitis I
have run the problem through gdb to at least obtain a stack
trace to see what gives:

...
==> verifying target database schema ...
==> checking migrated data for plausibility ...
Done bootstrapping GNUmed database: We very likely succeeded.
log: /home/ncq/Projekte/gm-git/gnumed/gnumed/server/bootstrap/bootstrap-latest.log

Debug memory block at address p=0x6aab7c: API ' '

0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):

at p-3: 0x33 *** OUCH
at p-2: 0x47 *** OUCH

at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.

The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):

at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API ' ', verified using API 'o'

Program received signal SIGABRT, Aborted.
0xb7fd9ce9 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7fd9ce9 in __kernel_vsyscall ()
#1 0xb7d6edd0 in __libc_signal_restore_set (set=0xbfffed40) at ../sysdeps/unix/sysv/linux/nptl-signals.h:79
#2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xb7d70297 in __GI_abort () at abort.c:89
#4 0x0055fb74 in Py_FatalError (msg=0xbfffefec "bad ID: Allocated using API '\037', verified using API 'o'") at ../Python/pythonrun.c:1700
#5 0x00499adb in _PyObject_DebugCheckAddressApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1640
#6 0x004997a5 in _PyObject_DebugFreeApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1527
#7 0x0049964f in _PyObject_DebugFree (p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1471
#8 0x00471043 in int_dealloc (v=0x6aab7c <_Py_ZeroStruct>) at ../Objects/intobject.c:139
#9 0x00497bee in _Py_Dealloc (op=False) at ../Objects/object.c:2262
#10 0x00489bad in dict_dealloc (mp=0xb7888f34) at ../Objects/dictobject.c:1085
Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode bytes in position 1-2: unexpected end of data:
#11 0x00497bee in _Py_Dealloc (op=) at ../Objects/object.c:2262
Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode bytes in position 1-2: unexpected end of data:
#12 0x004b9ab7 in subtype_dealloc (self=) at ../Objects/typeobject.c:1035
Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode bytes in position 1-2: unexpected end of data:
#13 0x00497bee in _Py_Dealloc (op=) at ../Objects/object.c:2262
#14 0x0044dc7a in instancemethod_dealloc (im=0xb78984b4) at ../Objects/classobject.c:2388
#15 0x00497bee in _Py_Dealloc (op=<instancemethod at remote 0xb78984b4>) at ../Objects/object.c:2262
#16 0x004885d7 in insertdict_by_entry (mp=0xb78880d4, key='_shutdown', hash=598970216, ep=0x88d6d4, value=None) at ../Objects/dictobject.c:519
#17 0x00488857 in insertdict (mp=0xb78880d4, key='_shutdown', hash=598970216, value=None) at ../Objects/dictobject.c:556
#18 0x0048910f in dict_set_item_by_hash_or_entry (
op={'current_thread': <function at remote 0xb78a28f4>, '_BoundedSemaphore': None, 'currentThread': <function at remote 0xb78a28f4>, '_Timer': None, '_format_exc': None, 'Semaphore': <function at remote 0xb789b6c4>, '_deque': None, 'activeCount': <function at remote 0xb78a2a34>, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': <type at remote 0xb784c034>, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': <function at remote 0xb789ba34>, '_shutdown': None, '__all__': ['activeCount', 'active_count', 'Condition', 'currentThread', 'current_thread', 'enumerate', 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Timer', 'setprofile', 'settrace', 'local', 'stack_size'], '_Event': <type at remote 0xb789e034>, 'active_count': <function at remote 0xb78a2a34>, '__package__': None, '_Condition': <type at remote 0xb784c9e4>, '_RLock': <type at remote 0xb784c7f4>, '_test': <function at remote 0xb78a2b74>, 'local': <type at remote 0x6fd420>, '__doc__': "Thread modul...(truncated), key='_shutdown', hash=598970216, ep=0x0, value=None) at ../Objects/dictobject.c:795
#19 0x00489285 in PyDict_SetItem (
op={'current_thread': <function at remote 0xb78a28f4>, '_BoundedSemaphore': None, 'currentThread': <function at remote 0xb78a28f4>, '_Timer': None, '_format_exc': None, 'Semaphore': <function at remote 0xb789b6c4>, '_deque': None, 'activeCount': <function at remote 0xb78a2a34>, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': <type at remote 0xb784c034>, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': <function at remote 0xb789ba34>, '_shutdown': None, '__all__': ['activeCount', 'active_count', 'Condition', 'currentThread', 'current_thread', 'enumerate', 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Timer', 'setprofile', 'settrace', 'local', 'stack_size'], '_Event': <type at remote 0xb789e034>, 'active_count': <function at remote 0xb78a2a34>, '__package__': None, '_Condition': <type at remote 0xb784c9e4>, '_RLock': <type at remote 0xb784c7f4>, '_test': <function at remote 0xb78a2b74>, 'local': <type at remote 0x6fd420>, '__doc__': "Thread modul...(truncated), key='_shutdown', value=None) at ../Objects/dictobject.c:848
#20 0x00492758 in _PyModule_Clear (m=<module at remote 0xb7892364>) at ../Objects/moduleobject.c:125
#21 0x0054a33b in PyImport_Cleanup () at ../Python/import.c:530
#22 0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
#23 0x0055fe9c in Py_Exit (sts=0) at ../Python/pythonrun.c:1783
#24 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
#25 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1161
#26 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
#27 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:952
#28 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:752
#29 0x00577cb0 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:645
#30 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20
(gdb) quit
A debugging session is active.

Inferior 1 [process 32024] will be killed.

Quit anyway? (y or n) y
Dropping obsoleted staging database gnumed_v2 ...
(you may need to provide the password for root)
Dropping obsoleted staging database gnumed_v3 ...
...

Vague hints from the above might suggest that I go look for
whether I do have a second thread and where improperly formed
unicode data might come from.

[Karsten Hilbert]

On Mon, Oct 23, 2017 at 03:26:18PM +0200, Thomas Jollans wrote:

> >> It points to a memory corruption.
> >
> > While running valgrind and friends is beyond my capabilitis I
> > have run the problem through gdb to at least obtain a stack
> > trace to see what gives:
>

> I wouldn't read too much into that. You know that somehow some memory
> got corrupted. You know at which point Python trips over the error - but
> that doesn't really tell you anything about how the memory initially got
> corrupted.

Thanks, I know. However, I am grasping for straws here since
I am unable to "properly" debug the python binary as such.

[M.-A. Lemburg]

> The python2.7-dbg build is 32 bits:
>

> root@hermes:~# nm /usr/bin/python2.7-dbg | grep Py_InitM
> 00155b9f T Py_InitModule4TraceRefs
>
>
> python2.7-dbg:
> Installiert: 2.7.14-2
> Installationskandidat: 2.7.14-2
> Versionstabelle:
> *** 2.7.14-2 500
> 500 http://httpredir.debian.org/debian unstable/main i386 Packages
> 100 /var/lib/dpkg/status
> 2.7.13-2 990
> 500 http://httpredir.debian.org/debian stretch/main i386 Packages
> 990 http://httpredir.debian.org/debian buster/main i386 Packages
>

> The python2.7 build (no -dbg) does not have symbols.
>

> mxDateTime really should be 32 bits, too:
>
> python-egenix-mxdatetime:
> Installiert: 3.2.9-1
> Installationskandidat: 3.2.9-1
> Versionstabelle:
> *** 3.2.9-1 990
> 500 http://httpredir.debian.org/debian stretch/main i386 Packages
> 990 http://httpredir.debian.org/debian buster/main i386 Packages
> 500 http://httpredir.debian.org/debian unstable/main i386 Packages
> 100 /var/lib/dpkg/status
>
> Let me check the .so file:
>
> root@hermes:~# nm /usr/lib/python2.7/dist-packages/mx/DateTime/mxDateTime/mxDateTime_d.so | grep Py_InitM
> U Py_InitModule4TraceRefs
>
> It seems it is - hm ...

Could you check whether you have similar import errors with
other modules that have C extensions ? E.g. lxml.

What you're seeing appears to be a compilation problem
with Python 2.7.14 on Debian. The executable doesn't appear
to export its symbols to the .so files, or only some of them.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts

>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> Python Database Interfaces ... http://products.egenix.com/
>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

[Karsten Hilbert]

On Tue, Oct 24, 2017 at 08:47:58PM +0200, M.-A. Lemburg wrote:

> >> This error suggests that you have 32- and 64-bit versions of
> >> Python and mxDateTime mixed in your installation.
> >>
> >> Py_InitModule4 is only available in the 32-bit build of
> >> Python. With the 64-bit build, it's called Py_InitModule4_64.

...
> Could you check whether you have similar import errors with
> other modules that have C extensions ? E.g. lxml.
>
> What you're seeing appears to be a compilation problem
> with Python 2.7.14 on Debian. The executable doesn't appear
> to export its symbols to the .so files, or only some of them.

Let's see:

python-lxml:
Installiert: 4.0.0-1
Installationskandidat: 4.0.0-1
Versionstabelle:
*** 4.0.0-1 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

3.7.1-1 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

python-lxml-dbg:
Installiert: (keine)
Installationskandidat: 4.0.0-1
Versionstabelle:
4.0.0-1 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages

3.7.1-1 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

ncq@hermes:~$ python2.7-dbg

Python 2.7.14 (default, Sep 17 2017, 18:50:44)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import lxml
[45350 refs]
>>>

ncq@hermes:~$ python

Python 2.7.14 (default, Sep 17 2017, 18:50:44)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import lxml
>>>

Also, psycogp2 imports just fine.

Now that I have

python-egenix-mx-base-dbg:

Installiert: 3.2.9-1
Installationskandidat: 3.2.9-1
Versionstabelle:
*** 3.2.9-1 990
500 http://httpredir.debian.org/debian stretch/main i386 Packages
990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

mx.DateTime imports fine as well (but the SIGABRT persists).

[M.-A. Lemburg]

n 25.10.2017 11:51, Karsten Hilbert wrote:
> On Tue, Oct 24, 2017 at 08:47:58PM +0200, M.-A. Lemburg wrote:
>
>>>> This error suggests that you have 32- and 64-bit versions of
>>>> Python and mxDateTime mixed in your installation.
>>>>
>>>> Py_InitModule4 is only available in the 32-bit build of
>>>> Python. With the 64-bit build, it's called Py_InitModule4_64.
> ...
>> Could you check whether you have similar import errors with
>> other modules that have C extensions ? E.g. lxml.
>>
>> What you're seeing appears to be a compilation problem
>> with Python 2.7.14 on Debian. The executable doesn't appear
>> to export its symbols to the .so files, or only some of them.
>
> Let's see:

> ... using -dbg packages for everything, the imports work ...

Ah, so you were mixing debug packages with non-debug ones. This
explains the import errors.

> mx.DateTime imports fine as well (but the SIGABRT persists).

Do you just see the SIGABRT when running the debug versions or
also with the production versions (memory management works differently
in Python debug mode) ?

BTW: It would help if you'd post the stack trace with symbols.
The one you posted in one of your earlier emails only includes
addresses.

PS: Please CC me on replies as I don't regularly read c.l.p anymore.

Thanks,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 30 2017)

>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/
>>> Python Database Interfaces ... http://products.egenix.com/
>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

http://www.malemburg.com/

[Karsten Hilbert]

On Mon, Oct 30, 2017 at 02:02:25PM +0100, M.-A. Lemburg wrote:

> PS: Please CC me on replies as I don't regularly read c.l.p anymore.

Sure.

> >> Could you check whether you have similar import errors with
> >> other modules that have C extensions ? E.g. lxml.
> >>
> >> What you're seeing appears to be a compilation problem
> >> with Python 2.7.14 on Debian. The executable doesn't appear
> >> to export its symbols to the .so files, or only some of them.
> >
> > Let's see:
> > ... using -dbg packages for everything, the imports work ...
>
> Ah, so you were mixing debug packages with non-debug ones. This
> explains the import errors.

I am not sure I do.

I installed normal and -dbg packages for modules with
c extensions, say psycopg2:

python-psycopg2:
Installiert: 2.7.3-1
Installationskandidat: 2.7.3-1
Versionstabelle:
*** 2.7.3-1 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

2.6.2-1 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

python-psycopg2-dbg:
Installiert: 2.7.3-1
Installationskandidat: 2.7.3-1
Versionstabelle:
*** 2.7.3-1 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

2.6.2-1 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

Now, when running either python2.7 or python2.7-dbg, which
are installed alongside

python2.7-dbg:
Installiert: 2.7.14-2
Installationskandidat: 2.7.14-2
Versionstabelle:

*** 2.7.14-2 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

2.7.13-2 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

python2.7:

Installiert: 2.7.14-2
Installationskandidat: 2.7.14-2
Versionstabelle:

*** 2.7.14-2 990

990 http://httpredir.debian.org/debian buster/main i386 Packages
500 http://httpredir.debian.org/debian unstable/main i386 Packages
100 /var/lib/dpkg/status

2.7.13-2 500

500 http://httpredir.debian.org/debian stretch/main i386 Packages

the Debian version does (should ?) take care to import either
the -dbg or the normal version of modules. It seems it so
does because when I got both module versions installed I
don't get any import errors. So I don't think I am mixing, at
least not to my knowledge.

(But I still get the SIGABORT on shutdown regardless.)

> Do you just see the SIGABRT when running the debug versions or
> also with the production versions (memory management works differently
> in Python debug mode) ?

Either version fails.

Why haven't I tried running this under python3 ? Because the
whole shebang is part of a wxPython application (GNUmed) and
there are no Debian packages for wxPython/wxPhoenix on py3 as
far as I know.

> BTW: It would help if you'd post the stack trace with symbols.
> The one you posted in one of your earlier emails only includes
> addresses.

I'd gladly comply if I knew how. This is what apt says about python2.7-dbg:

Package: python2.7-dbg
Version: 2.7.14-2
Description-en: Debug Build of the Python Interpreter (version 2.7)
The package holds two things:
.
- A Python interpreter configured with --pydebug. Dynamically loaded modules
are searched as <foo>_d.so first. Third party extensions need a separate
build to be used by this interpreter.
- Debug information for standard python interpreter and extensions.

So from that I conferred it does contain symbols.

I am getting a fresh run under gdb with python2.7-dbg. Maybe
you can point out where you'd have expected to see symbols
and help me figure out which -dbg package is missing on this
Debian system:

[...]

==> verifying target database schema ...
==> checking migrated data for plausibility ...
Done bootstrapping GNUmed database: We very likely succeeded.
log: /home/ncq/Projekte/gm-git/gnumed/gnumed/server/bootstrap/bootstrap-latest.log
Debug memory block at address p=0x6aab7c: API ' '
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
at p-3: 0x33 *** OUCH
at p-2: 0x47 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API ' ', verified using API 'o'

Program received signal SIGABRT, Aborted.
0xb7fd9ce9 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7fd9ce9 in __kernel_vsyscall ()

#1 0xb7d70dd0 in __libc_signal_restore_set (set=0xbfffed40) at ../sysdeps/unix/sysv/linux/nptl-signals.h:79

#2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48

#3 0xb7d72297 in __GI_abort () at abort.c:89

#4 0x0055fb74 in Py_FatalError (msg=0xbfffefec "bad ID: Allocated using API '\037', verified using API 'o'") at ../Python/pythonrun.c:1700
#5 0x00499adb in _PyObject_DebugCheckAddressApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1640
#6 0x004997a5 in _PyObject_DebugFreeApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1527
#7 0x0049964f in _PyObject_DebugFree (p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1471
#8 0x00471043 in int_dealloc (v=0x6aab7c <_Py_ZeroStruct>) at ../Objects/intobject.c:139
#9 0x00497bee in _Py_Dealloc (op=False) at ../Objects/object.c:2262

#10 0x00489bad in dict_dealloc (mp=0xb79f10d4) at ../Objects/dictobject.c:1085
Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0xb7 in position 3: invalid start byte:

#11 0x00497bee in _Py_Dealloc (op=) at ../Objects/object.c:2262

Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0xb7 in position 3: invalid start byte:

#12 0x004b9ab7 in subtype_dealloc (self=) at ../Objects/typeobject.c:1035

Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0xb7 in position 3: invalid start byte:

#13 0x00497bee in _Py_Dealloc (op=) at ../Objects/object.c:2262

#14 0x0044dc7a in instancemethod_dealloc (im=0xb79e9034) at ../Objects/classobject.c:2388
#15 0x00497bee in _Py_Dealloc (op=<instancemethod at remote 0xb79e9034>) at ../Objects/object.c:2262
#16 0x004885d7 in insertdict_by_entry (mp=0xb7891214, key='_shutdown', hash=598970216, ep=0x88e1ec, value=None) at ../Objects/dictobject.c:519
#17 0x00488857 in insertdict (mp=0xb7891214, key='_shutdown', hash=598970216, value=None) at ../Objects/dictobject.c:556

#18 0x0048910f in dict_set_item_by_hash_or_entry (

op={'current_thread': <function at remote 0xb79f23a4>, '_BoundedSemaphore': None, 'currentThread': <function at remote 0xb79f23a4>, '_Timer': None, '_format_exc': None, 'Semaphore': <function at remote 0xb79ec174>, '_deque': None, 'activeCount': <function at remote 0xb79f24e4>, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': <type at remote 0xb7850414>, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': <function at remote 0xb79ec4e4>, '_shutdown': None, '__all__': ['activeCount', 'active_count', 'Condition', 'currentThread', 'current_thread', 'enumerate', 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Timer', 'setprofile', 'settrace', 'local', 'stack_size'], '_Event': <type at remote 0xb79ee224>, 'active_count': <function at remote 0xb79f24e4>, '__package__': None, '_Condition': <type at remote 0xb7850bd4>, '_RLock': <type at remote 0xb78509e4>, '_test': <function at remote 0xb79f2624>, 'local': <type at remote 0x6fd420>, '__doc__': "Thread modul...(truncated), key='_shutdown', hash=598970216, ep=0x0, value=None) at ../Objects/dictobject.c:795

#19 0x00489285 in PyDict_SetItem (

op={'current_thread': <function at remote 0xb79f23a4>, '_BoundedSemaphore': None, 'currentThread': <function at remote 0xb79f23a4>, '_Timer': None, '_format_exc': None, 'Semaphore': <function at remote 0xb79ec174>, '_deque': None, 'activeCount': <function at remote 0xb79f24e4>, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': <type at remote 0xb7850414>, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': <function at remote 0xb79ec4e4>, '_shutdown': None, '__all__': ['activeCount', 'active_count', 'Condition', 'currentThread', 'current_thread', 'enumerate', 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Timer', 'setprofile', 'settrace', 'local', 'stack_size'], '_Event': <type at remote 0xb79ee224>, 'active_count': <function at remote 0xb79f24e4>, '__package__': None, '_Condition': <type at remote 0xb7850bd4>, '_RLock': <type at remote 0xb78509e4>, '_test': <function at remote 0xb79f2624>, 'local': <type at remote 0x6fd420>, '__doc__': "Thread modul...(truncated), key='_shutdown', value=None) at ../Objects/dictobject.c:848
#20 0x00492758 in _PyModule_Clear (m=<module at remote 0xb78a1154>) at ../Objects/moduleobject.c:125

#21 0x0054a33b in PyImport_Cleanup () at ../Python/import.c:530
#22 0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
#23 0x0055fe9c in Py_Exit (sts=0) at ../Python/pythonrun.c:1783
#24 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
#25 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1161
#26 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
#27 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:952
#28 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:752
#29 0x00577cb0 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:645
#30 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20
(gdb) q

A debugging session is active.

Inferior 1 [process 12740] will be killed.

Quit anyway? (y or n) y

Does that help ?

Thanks,

[dieter]

Karsten Hilbert <Karsten...@gmx.net> writes:

> On Sat, Oct 21, 2017 at 09:31:54AM +0200, dieter wrote:
>> It points to a memory corruption.
> While running valgrind and friends is beyond my capabilitis I
> have run the problem through gdb to at least obtain a stack
> trace to see what gives:

The i386/x64 architecture supports memory access breakpoints
and GDB, too, has support for this. You know the address which
gets corrupted. Thus, the following apporach could have a chance
to succeed:

Put a "memory write" breakpoint on the address which gets corrupted.
this should stop the program each time this address is written;
Check then the backtrace. As the address forms part of the
address block prologue, it should only be accessed from
Python's "malloc" (and "free") implementation. Any other access
indicates bad behaviour.

Should your program get stopped too often (because the memory
block is reused too often), you must find a good point in your
program to activate the memory access breakpoint in a delayed way.
You could use the Python debugger for this: execute significant
parts of your program in the Python debugger; switching to
GDB, check whether the address is already corrupted - if
so, restart and refine the checks above in the portion of your program
which caused the corruption. If the part in your program
is sufficiently small, you can activate the memory access breakpoint.
This approach may also be feasible, should you use a CPU
without memory access breakpoints.

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 09:21:46AM +0100, dieter wrote:

> >> It points to a memory corruption.
>

> The i386/x64 architecture supports memory access breakpoints
> and GDB, too, has support for this. You know the address which
> gets corrupted. Thus, the following apporach could have a chance
> to succeed:
>
> Put a "memory write" breakpoint on the address which gets corrupted.
> this should stop the program each time this address is written;
> Check then the backtrace. As the address forms part of the
> address block prologue, it should only be accessed from
> Python's "malloc" (and "free") implementation. Any other access
> indicates bad behaviour.

I understand. Thank you for the explanation. This may seem
like a dumb question: the actual address that gets corrupted
varies from run to run (it may be the same "place" in the
code but that place gets put at a different address each
run). Since I don't know the address of a given run, how do I
set a break point on that address ? It seems to me I first
need to map the "place" to its address before the SIGABRT
happens. How do I find out out which "place" needs to be
mapped from the output I already have ?

(the "place" is the memory block you refer to)

Other than that I understand what you are getting at and am
willing to try.

Thanks,
Karsten

> Should your program get stopped too often (because the memory
> block is reused too often), you must find a good point in your
> program to activate the memory access breakpoint in a delayed way.
> You could use the Python debugger for this: execute significant
> parts of your program in the Python debugger; switching to
> GDB, check whether the address is already corrupted - if
> so, restart and refine the checks above in the portion of your program
> which caused the corruption. If the part in your program
> is sufficiently small, you can activate the memory access breakpoint.
> This approach may also be feasible, should you use a CPU
> without memory access breakpoints.

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 10:27:54AM +0100, Karsten Hilbert wrote:

> > >> It points to a memory corruption.
> >
> > The i386/x64 architecture supports memory access breakpoints
> > and GDB, too, has support for this. You know the address which
> > gets corrupted. Thus, the following apporach could have a chance
> > to succeed:
> >
> > Put a "memory write" breakpoint on the address which gets corrupted.
> > this should stop the program each time this address is written;
> > Check then the backtrace. As the address forms part of the
> > address block prologue, it should only be accessed from
> > Python's "malloc" (and "free") implementation. Any other access
> > indicates bad behaviour.
>
> I understand. Thank you for the explanation. This may seem
> like a dumb question: the actual address that gets corrupted
> varies from run to run (it may be the same "place" in the
> code but that place gets put at a different address each
> run). Since I don't know the address of a given run, how do I
> set a break point on that address ? It seems to me I first
> need to map the "place" to its address before the SIGABRT
> happens. How do I find out out which "place" needs to be
> mapped from the output I already have ?

Or rather: I need to find out which "place" a given address
refers to, check whether the changing addresses always belong
to the same "place" between runs and _then_ map a "place" to
its address and breakpoint that address on yet another run.

It might seem

gdb> info symbol <the address>

should give me the "place".

Then

gdb> info address <the symbol>

on another run. Or even just "watch <the symbol". I'll try.

Karsten

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 11:14:08AM +0100, Karsten Hilbert wrote:

> Or rather: I need to find out which "place" a given address
> refers to, check whether the changing addresses always belong
> to the same "place" between runs and _then_ map a "place" to
> its address and breakpoint that address on yet another run.
>
> It might seem
>
> gdb> info symbol <the address>
>
> should give me the "place".

Given this:

Debug memory block at address p=0x6aab7c: API ''
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
at p-3: 0x33 *** OUCH
at p-2: 0x47 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API '', verified using API 'o'

Program received signal SIGABRT, Aborted.
0xb7fd9ce9 in __kernel_vsyscall ()

(gdb) info symbol 0x6aab7c
_Py_ZeroStruct in section .data of /usr/bin/python2.7-dbg
(gdb)

my assumption would be that something clobbers 0x6aab7c,
which seems to be in (?) _Py_ZeroStruct in this run. I'll
re-run a few times to make sure the corruption "reliably"
hits _Py_ZeroStruct.

If so, I'll set a memory write breakpoint on _Py_ZeroStruct.

Am I on the right track ?

Thanks,
Karsten

BTW, the backtrace for this run was ...

(gdb) bt
#0 0xb7fd9ce9 in __kernel_vsyscall ()

#1 0xb7d70dd0 in __libc_signal_restore_set (set=0xbfffee90) at ../sysdeps/unix/sysv/linux/nptl-signals.h:79

#2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xb7d72297 in __GI_abort () at abort.c:89

#4 0x0055fb74 in Py_FatalError (msg=0xbffff13c "bad ID: Allocated using API '\037', verified using API 'o'") at ../Python/pythonrun.c:1700

#5 0x00499adb in _PyObject_DebugCheckAddressApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1640
#6 0x004997a5 in _PyObject_DebugFreeApi (api=111 'o', p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1527
#7 0x0049964f in _PyObject_DebugFree (p=0x6aab7c <_Py_ZeroStruct>) at ../Objects/obmalloc.c:1471
#8 0x00471043 in int_dealloc (v=0x6aab7c <_Py_ZeroStruct>) at ../Objects/intobject.c:139

... so I could've known without "info symbol" :-)

#9 0x00497bee in _Py_Dealloc (op=False) at ../Objects/object.c:2262

#10 0x004885d7 in insertdict_by_entry (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, ep=0x7c5c08, value=None) at ../Objects/dictobject.c:519
#11 0x00488857 in insertdict (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, value=None) at ../Objects/dictobject.c:556
#12 0x0048910f in dict_set_item_by_hash_or_entry (
op={
'setrecursionlimit': None,
'dont_write_bytecode': None,
'getfilesystemencoding': <built-in function getfilesystemencoding>,
'long_info': <sys.long_info at remote 0xb7f936e8>,
'path_importer_cache': None,
'stdout': <file at remote 0xb7fcd098>,
'getprofile': <built-in function getprofile>,
'__stdin__': <file at remote 0xb7fcd028>,
'version_info': <sys.version_info at remote 0xb7fcfd80>,
'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', hash=591857026, ep=0x0, value=None
) at ../Objects/dictobject.c:795
#13 0x00489285 in PyDict_SetItem (
op={'setrecursionlimit': None, 'dont_write_bytecode': None, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'long_info': <sys.long_info at remote
0xb7f936e8>, 'path_importer_cache': None, 'stdout': <file at remote 0xb7fcd098>, 'getprofile': <built-in function getprofile>, '__stdin__': <file at remote 0xb7fcd028>, 'version_info': <sys.version_info at remote 0xb7fcfd80>, 'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', value=None) at ../Objects/dictobject.c:848
#14 0x0049281f in _PyModule_Clear (m=<module at remote 0xb7f935d4>) at ../Objects/moduleobject.c:139
#15 0x0054a3ec in PyImport_Cleanup () at ../Python/import.c:540
#16 0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
#17 0x0055fe9c in Py_Exit (sts=1) at ../Python/pythonrun.c:1783
#18 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
#19 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1161
#20 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
#21 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:952
#22 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:752
#23 0x00577cb0 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:645
#24 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20

--

[Karsten Hilbert]

> my assumption would be that something clobbers 0x6aab7c,
> which seems to be in (?) _Py_ZeroStruct in this run. I'll
> re-run a few times to make sure the corruption "reliably"
> hits _Py_ZeroStruct.
>
> If so, I'll set a memory write breakpoint on _Py_ZeroStruct.

Interestingly, on subsequent runs, it seems to hit the same
address, 0x6aab7c, belonging to the same symbol, _Py_ZeroStruct.

This is what happens:

(gdb) watch *0x6aab7c
Hardware watchpoint 1: *0x6aab7c
(gdb) run
Starting program: /usr/bin/python2.7-dbg ./bootstrap_gm_db_system.py --log-file=./bootstrap-latest.log --conf-file=bootstrap-latest.conf --
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Hardware watchpoint 1: *0x6aab7c

Old value = 0
New value = -1208182272
_Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
70 ../Objects/object.c: Datei oder Verzeichnis nicht gefunden.
(gdb)

which means I'll probably have to apply the delayed
breakpoint setting strategy, or else it is just some initial
relocation at startup. Let's see what "cont" brings. The next
hit after the Python script has run until just before it
usually aborts:

Hardware watchpoint 1: *0x6aab7c

Old value = -1208182272
New value = 0
_Py_ForgetReference (op=False) at ../Objects/object.c:2255
2255 in ../Objects/object.c
(gdb)

The backtrace at this point says:

(gdb) bt
#0 _Py_ForgetReference (op=False) at ../Objects/object.c:2255
#1 0x00497be0 in _Py_Dealloc (op=False) at ../Objects/object.c:2261
#2 0x004885d7 in insertdict_by_entry (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, ep=0x7c5c08, value=None) at ../Objects/dictobject.c:519
#3 0x00488857 in insertdict (mp=0xb7fc5674, key='dont_write_bytecode', hash=591857026, value=None) at ../Objects/dictobject.c:556
#4 0x0048910f in dict_set_item_by_hash_or_entry (

op={'setrecursionlimit': None, 'dont_write_bytecode': None, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'long_info': <sys.long_info at remote
0xb7f936e8>, 'path_importer_cache': None, 'stdout': <file at remote 0xb7fcd098>, 'getprofile': <built-in function getprofile>, '__stdin__': <file at remote 0xb7fcd028>, 'version_info': <sys.version_info at remote 0xb7fcfd80>, 'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', hash=591857026, ep=0x0, value=None) at ../Objects/dictobject.c:795

#5 0x00489285 in PyDict_SetItem (

op={'setrecursionlimit': None, 'dont_write_bytecode': None, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'long_info': <sys.long_info at remote
0xb7f936e8>, 'path_importer_cache': None, 'stdout': <file at remote 0xb7fcd098>, 'getprofile': <built-in function getprofile>, '__stdin__': <file at remote 0xb7fcd028>, 'version_info': <sys.version_info at remote 0xb7fcfd80>, 'exc_clear': <built-in function exc_clear>, 'gettotalrefcount': <built-in function gettotalrefcount>, 'getrefcount': <built-in function getrefcount>, 'byteorder': 'little', '_clear_type_cache': None, 'excepthook': <built-in function excepthook>, 'subversion': ('CPython', '', ''), '_multiarch': None, 'exc_type': None, 'ps1': None, '__excepthook__': <built-in function excepthook>, 'executable': '/usr/bin/python2.7-dbg', 'float_info': None, 'copyright': 'Copyright (c) 2001-2017 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for Nation...(truncated), key='dont_write_bytecode', value=None) at ../Objects/dictobject.c:848

#6 0x0049281f in _PyModule_Clear (m=<module at remote 0xb7f935d4>) at ../Objects/moduleobject.c:139
#7 0x0054a3ec in PyImport_Cleanup () at ../Python/import.c:540
#8 0x0055c53c in Py_Finalize () at ../Python/pythonrun.c:458
#9 0x0055fe9c in Py_Exit (sts=1) at ../Python/pythonrun.c:1783
#10 0x0055e0fc in handle_system_exit () at ../Python/pythonrun.c:1151
#11 0x0055e152 in PyErr_PrintEx (set_sys_last_vars=1) at ../Python/pythonrun.c:1161
#12 0x0055dd5b in PyErr_Print () at ../Python/pythonrun.c:1064
#13 0x0055d61f in PyRun_SimpleFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:952
#14 0x0055cc4e in PyRun_AnyFileExFlags (fp=0x7ee010, filename=0xbffff7e6 "./bootstrap_gm_db_system.py", closeit=1, flags=0xbffff4f4) at ../Python/pythonrun.c:752
#15 0x00577cb0 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:645
#16 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20

And continuing hits the SIGABRT right away:

(gdb) cont
Continuing.

Debug memory block at address p=0x6aab7c: API ''
0 bytes originally requested
The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
at p-3: 0x33 *** OUCH
at p-2: 0x47 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 4 pad bytes at tail=0x6aab7c are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x00 *** OUCH
at tail+1: 0x00 *** OUCH
at tail+2: 0x00 *** OUCH
at tail+3: 0x00 *** OUCH
The block was made by call #0 to debug malloc/realloc.
Fatal Python error: bad ID: Allocated using API '', verified using API 'o'

Program received signal SIGABRT, Aborted.
0xb7fd9ce9 in __kernel_vsyscall ()
(gdb)

Does that help ?

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 12:40:56PM +0100, Karsten Hilbert wrote:

> Interestingly, on subsequent runs, it seems to hit the same
> address, 0x6aab7c, belonging to the same symbol, _Py_ZeroStruct.
>
> This is what happens:
>
> (gdb) watch *0x6aab7c
> Hardware watchpoint 1: *0x6aab7c
> (gdb) run
> Starting program: /usr/bin/python2.7-dbg ./bootstrap_gm_db_system.py --log-file=./bootstrap-latest.log --conf-file=bootstrap-latest.conf --
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
>
> Hardware watchpoint 1: *0x6aab7c
>
> Old value = 0
> New value = -1208182272
> _Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
> 70 ../Objects/object.c: Datei oder Verzeichnis nicht gefunden.
> (gdb)

I forgot the backtrace from this point:

(gdb) bt
#0 _Py_AddToAllObjects (op=False, force=0) at ../Objects/object.c:70
#1 0x0051e052 in _PyBuiltin_Init () at ../Python/bltinmodule.c:2708
#2 0x0055bebf in Py_InitializeEx (install_sigs=1) at ../Python/pythonrun.c:233
#3 0x0055c4dd in Py_Initialize () at ../Python/pythonrun.c:381
#4 0x00577805 in Py_Main (argc=5, argv=0xbffff684) at ../Modules/main.c:551
#5 0x004259c8 in main (argc=5, argv=0xbffff684) at ../Modules/python.c:20

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 11:58:53AM -0400, Dennis Lee Bieber wrote:

> >I understand. Thank you for the explanation. This may seem
> >like a dumb question: the actual address that gets corrupted
> >varies from run to run (it may be the same "place" in the
>

> Since the process virtual memory space should be the same on each run
> -- even heap allocations should be deterministic, UNLESS the program is
> doing things with external non-deterministic data (the time of day, random
> numbers not initialized with a repeatable seed, dynamic web pages, multiple
> threads that are non-deterministic due to I/O delays, etc.),

Mine doesn't to my knowledge hence I can expect "fixed"
addresses -- which is confirmed by what I found during
consecutive runs. Thanks for the affirmative explanation.

> the only other source for varying addresses would be
> OS-level shared libraries that are getting mapped at
> different locations due to changes in the order of other
> programs running.

IOW I should take care I haven't run any relevant amount of
Python code before debugging this problem, it seems ?

> The physical memory may vary due to paging fragmentation,
> but should still be the same virtual addresses.

I see.

[Grant Edwards]

On 2017-11-01, Dennis Lee Bieber <wlf...@ix.netcom.com> wrote:
> On Wed, 1 Nov 2017 10:27:54 +0100, Karsten Hilbert
><Karsten...@gmx.net> declaimed the following:

>
>
>>
>>I understand. Thank you for the explanation. This may seem
>>like a dumb question: the actual address that gets corrupted
>>varies from run to run (it may be the same "place" in the
>
> Since the process virtual memory space should be the same on each run

Not with modern OS kernels:

https://en.wikipedia.org/wiki/Address_space_layout_randomization

--
Grant Edwards grant.b.edwards Yow! ... My pants just went
at on a wild rampage through a
gmail.com Long Island Bowling Alley!!

[dieter]

Karsten Hilbert <Karsten...@gmx.net> writes:
>> >> It points to a memory corruption.
>>
>> The i386/x64 architecture supports memory access breakpoints
>> and GDB, too, has support for this. You know the address which
>> gets corrupted. Thus, the following apporach could have a chance
>> to succeed:
>>
>> Put a "memory write" breakpoint on the address which gets corrupted.
>> this should stop the program each time this address is written;
>> Check then the backtrace. As the address forms part of the
>> address block prologue, it should only be accessed from
>> Python's "malloc" (and "free") implementation. Any other access
>> indicates bad behaviour.
>

> I understand. Thank you for the explanation. This may seem
> like a dumb question: the actual address that gets corrupted
> varies from run to run (it may be the same "place" in the

> code but that place gets put at a different address each
> run).

That's sad.

It is a long time ago (more than 10 years)
that I had to analyse such a kind of memory corruption. Fortunately,
in my case, the address was stable accross runs.
Likely, ASLR was not yet used by that time on a standard Linux platform.

Maybe, you find a way to disable ASLR.

If ASLR is the cause of the randomness, it might also be possible to
compute the new address. More on this later.


In another message, you reported how you tried to obtain an invariant
for the affected address by using "info symbol". I have not much
hope that this will succeed:
It is most likely, that the corrupted memory block is part of the
heap (in may also be a stack block, wrongly freed; this would be
a local error - and easily detectable from the traceback).
If you use "info symbol" on a heap address, you get not very
reliable information - especially, if ASLR is in effect (which
randomizes the various process segments and the heap blocks
independently from one another).


Back to an approach how to maybe compute the corrupted address
for a new run. The approach assumes a heap address and
uses that "mallog" (and friends) request large (which means hopefully few)
memory blocks from the OS which are then split into smaller blocks internally.
You can then catalog the large memory block requests and determine
the index of the block and the corrupted offset. In a following run,
you determine the new base address of this block and apply the
same offset to find the corrupted address.

Of cause, this assumes that your application is totally deterministic
(apart from maybe ASLR).

[Karsten Hilbert]

On Wed, Nov 01, 2017 at 04:28:02PM +0000, Grant Edwards wrote:

> >>I understand. Thank you for the explanation. This may seem
> >>like a dumb question: the actual address that gets corrupted
> >>varies from run to run (it may be the same "place" in the
> >

> > Since the process virtual memory space should be the same on each run
>
> Not with modern OS kernels:
>
> https://en.wikipedia.org/wiki/Address_space_layout_randomization

I feared as much. However, I discovered that during
subsequent runs the address seems stable due to shared
libraries being preloaded: On the first run the affected code
is loaded at some randomized address and the corruption is
hit at a certain address giving me the value to watch on
during subsequent runs, as long as the affected code is not
evicted from being preloaded the address is stable.

I have posted backtraces taken from the address being
watched. Does that help any at all ?

Thanks,

[dieter]

Karsten Hilbert <Karsten...@gmx.net> writes:
> ...

> I have posted backtraces taken from the address being
> watched. Does that help any at all ?

Only in the case that the error is "local", i.e. detected
(quite) immediately.

You might be in this case as you have observed that the address
is stable after library preload. Thus, it might not be a heap
address but one associated with one of the libraries. Such
a memory block should never be "freed". The backtrace would allow
you to determine the library affected. Obtain its source. Recompile
with symbols and try to find out where this memory block comes from.

[Karsten Hilbert]

Dieter, thanks for taking the time to explain the general
procedure. However, recompiling a library and trying to find
out where given block of memory comes from is way beyond
skills. I fear I have reached the end of what I can do.