Linking errors when compiled with both python and python3
Nico Raffo
python3interp, errors occur when importing many python modules. To
reproduce, compile as described then try something like:
:py import termios
or
:py3 import termios
Which gives the error:
ImportError: /usr/lib/python3.1/lib-dynload/termios.so: undefined
symbol: PyExc_TypeError
This doesn't happen with all modules. Mostly just those with odd .so
files.
The modules will import fine outside of Vim, or if python 2 or python
3 is compiled into Vim individually. I attempted adding some more
compiler flags as suggested at the bottom of http://docs.python.org/extending/embedding.html
without much luck.
Nico
James Vega
> If Vim is compiled with both --enable-pythoninterp and --enable-
> python3interp, errors occur when importing many python modules. To
> reproduce, compile as described then try something like:
>
> :py import termios
> or
> :py3 import termios
>
> Which gives the error:
>
> ImportError: /usr/lib/python3.1/lib-dynload/termios.so: undefined
> symbol: PyExc_TypeError
>
> This doesn't happen with all modules. Mostly just those with odd .so
> files.
I had been meaning to check whether something like that would happen with the
Python interfaces. I saw something similar when testing the Perl interface.
Does the attached patch fix it?
--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@jamessan.com>
Nico Raffo
> On Thu, Jul 22, 2010 at 12:36:27AM -0700, Nico Raffo wrote:
> > If Vim is compiled with both --enable-pythoninterp and --enable-
> > python3interp, errors occur when importing many python modules. To
> > reproduce, compile as described then try something like:
>
> > :py import termios
>
> Python interfaces. I saw something similar when testing the Perl interface.
> Does the attached patch fix it?
>
both versions of python installed. However if you import it from both
during one Vim session then Vim crashes and freezes my terminal.
To reproduce, compile the latest with --enable-pythoninterp and --
enable-python3interp and with James' patch. Launch Vim and run
:py import termios
:py3 import termios
Vim: Caught deadly signal ABRT
At this point Vim is gone and my gnome terminal has to be killed.
Nico
Bram Moolenaar
James Vega wrote:
> On Thu, Jul 22, 2010 at 12:36:27AM -0700, Nico Raffo wrote:
> > If Vim is compiled with both --enable-pythoninterp and --enable-
> > python3interp, errors occur when importing many python modules. To
> > reproduce, compile as described then try something like:
> >
> > :py import termios
> > or
> > :py3 import termios
> >
> > Which gives the error:
> >
> > ImportError: /usr/lib/python3.1/lib-dynload/termios.so: undefined
> > symbol: PyExc_TypeError
> >
> > This doesn't happen with all modules. Mostly just those with odd .so
> > files.
>
> I had been meaning to check whether something like that would happen
> with the Python interfaces. I saw something similar when testing the
> Perl interface. Does the attached patch fix it?
Thanks, I'll include it. It fixes the problem on my system.
--
Compilation process failed successfully.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ download, build and distribute -- http://www.A-A-P.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
Bram Moolenaar
Nico Raffo wrote:
I also see the crash. I can recover the terminal (by starting Vim
again).
The output is:
*** glibc detected *** ./vim: free(): invalid pointer: 0xb76990e0 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x1053591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x1054de8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x1057ecd]
/usr/lib/libpython2.6.so(PyObject_Free+0x54)[0x4168ea4]
/usr/lib/libpython2.6.so(+0x9d213)[0x4190213]
/usr/lib/libpython3.1.so(PyUnicode_InternInPlace+0xd6)[0x11ca576]
/usr/lib/libpython3.1.so(PyUnicode_InternFromString+0x33)[0x11d17e3]
/usr/lib/libpython3.1.so(+0x785d8)[0x11ba5d8]
/usr/lib/libpython3.1.so(PyType_Ready+0x87)[0x11bf7c7]
/usr/lib/libpython3.1.so(_Py_ReadyTypes+0xa4)[0x11ac174]
/usr/lib/libpython3.1.so(Py_InitializeEx+0x86)[0x1218ff6]
/usr/lib/libpython3.1.so(Py_Initialize+0x1e)[0x12195ae]
./vim[0x8211611]
./vim[0x821167c]
./vim(ex_py3+0x41)[0x821179c]
./vim[0x80c89d9]
./vim(do_cmdline+0x944)[0x80c62b2]
./vim[0x814db42]
./vim(normal_cmd+0xf59)[0x8147278]
./vim(main_loop+0x584)[0x8109079]
./vim(main+0xd7c)[0x8108ad0]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xffebd6]
./vim[0x8073f71]
In gdb the stack trace is:
#0 0x0012d422 in __kernel_vsyscall ()
#1 0x00a14651 in *__GI_raise (sig=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00a17a82 in *__GI_abort () at abort.c:92
#3 0x00a4b49d in __libc_message (do_abort=2,
fmt=0xb1ff98 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4 0x00a55591 in malloc_printerr (action=<value optimized out>,
str=0x6 <Address 0x6 out of bounds>, ptr=0xb7dbe0e0) at malloc.c:6264
#5 0x00a56de8 in _int_free (av=<value optimized out>, p=<value optimized out>)
at malloc.c:4792
#6 0x00a59ecd in *__GI___libc_free (mem=0xb7dbe0e0) at malloc.c:3738
#7 0x00f65ea4 in PyObject_Free () from /usr/lib/libpython2.6.so
#8 0x00f8d213 in ?? () from /usr/lib/libpython2.6.so
#9 0x0136f576 in PyUnicode_InternInPlace () from /usr/lib/libpython3.1.so
#10 0x013767e3 in PyUnicode_InternFromString () from /usr/lib/libpython3.1.so
#11 0x0135f5d8 in ?? () from /usr/lib/libpython3.1.so
#12 0x013647c7 in PyType_Ready () from /usr/lib/libpython3.1.so
#13 0x01351174 in _Py_ReadyTypes () from /usr/lib/libpython3.1.so
#14 0x013bdff6 in Py_InitializeEx () from /usr/lib/libpython3.1.so
#15 0x013be5ae in Py_Initialize () from /usr/lib/libpython3.1.so
#16 0x08211611 in Python3_Init () at if_python3.c:527
#17 0x0821167c in DoPy3Command (eap=0xbfffee8c, cmd=0x83cf76c "import termios")
at if_python3.c:588
#18 0x0821179c in ex_py3 (eap=0xbfffee8c) at if_python3.c:643
#19 0x080c89d9 in do_one_cmd (cmdlinep=0xbffff040, sourcing=0,
cstack=0xbffff048, fgetline=0x80dbb9d <getexline>, cookie=0x0)
at ex_docmd.c:2656
#20 0x080c62b2 in do_cmdline (cmdline=0x0, getline=0x80dbb9d <getexline>,
cookie=0x0, flags=0) at ex_docmd.c:1122
#21 0x0814db42 in nv_colon (cap=0xbffff3ac) at normal.c:5295
#22 0x08147278 in normal_cmd (oap=0xbffff46c, toplevel=1) at normal.c:1188
#23 0x08109079 in main_loop (cmdwin=0, noexmode=0) at main.c:1256
#24 0x08108ad0 in main (argc=1, argv=0xbffff6b4) at main.c:964
That it jumps from libpython3.1.so to a function in libpython2.6.so
looks bad.
--
BROTHER MAYNARD: Armaments Chapter Two Verses Nine to Twenty One.
ANOTHER MONK: And St. Attila raised his hand grenade up on high saying "O
Lord bless this thy hand grenade that with it thou mayest
blow thine enemies to tiny bits, in thy mercy. "and the Lord
did grin and people did feast upon the lambs and sloths and
carp and anchovies and orang-utans and breakfast cereals and
fruit bats and...
BROTHER MAYNARD: Skip a bit brother ...
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
James Vega
>
> James Vega wrote:
>
>> On Thu, Jul 22, 2010 at 12:36:27AM -0700, Nico Raffo wrote:
>> > If Vim is compiled with both --enable-pythoninterp and --enable-
>> > python3interp, errors occur when importing many python modules. To
>> > reproduce, compile as described then try something like:
>> >
>> > :py import termios
>> > or
>> > :py3 import termios
>> >
>> > Which gives the error:
>> >
>> > ImportError: /usr/lib/python3.1/lib-dynload/termios.so: undefined
>> > symbol: PyExc_TypeError
>> >
>> > This doesn't happen with all modules. Mostly just those with odd .so
>> > files.
>>
>> I had been meaning to check whether something like that would happen
>> with the Python interfaces. I saw something similar when testing the
>> Perl interface. Does the attached patch fix it?
>
> Thanks, I'll include it. It fixes the problem on my system.
The attached patch corrects how DYNAMIC_PYTHON_DLL is specified so that
it correctly includes the SONAME (libpython2.6.so.1 instead of
libpython2.6.so). The former is always available while the latter is
typically only available when the relevant -dev package is installed.
I'm looking into the crash that was reported as well, but I'm not sure
if that can be avoided as long as both interpreters are getting loaded.
At a quick glance, it seems like if_python may not be handling locking
properly. I'll dig into this some more later tonight.
James Vega
Because we have to use RTLD_GLOBAL (since the C extensions aren't linked
against libpython), it's not safe to use both the Python 2 & 3
interfaces in the same session of Vim. It looks like the options are
either:
1) Only allow one interface to be enabled at a time.
2) Keep track of whether one of the Python interfaces has been
initialized and error out if the other one tries to be initialized.
I'd personally prefer the latter since that still allows one to build a
single Vim binary that supports both Python 2 & 3, but only one of the
interfaces can be used per Vim session.
Andy Kittner
>Because we have to use RTLD_GLOBAL (since the C extensions aren't linked
>against libpython), it's not safe to use both the Python 2 & 3
This seems to be system dependent then. On my system the original
example worked fine before the patch and I can also use python2 and 3
in the same vim session (just checked with the current hg head and
manually removed the RTLD_GLOBAL)
Running ldd on the files in the lib-dynload directory reveals that out
of my installed python versions (2.4.6, 2.5.4, 2.6.5, 2.7, 3.1.2)
revealed that all c-extensions except those for 2.4.6 are linked against
the corresponding libpython.so
>interfaces in the same session of Vim. It looks like the options are
>either:
>
>1) Only allow one interface to be enabled at a time.
>2) Keep track of whether one of the Python interfaces has been
> initialized and error out if the other one tries to be initialized.
>
>I'd personally prefer the latter since that still allows one to build a
>single Vim binary that supports both Python 2 & 3, but only one of the
>interfaces can be used per Vim session.
Given my above observations the "with cherry on top" solution might be
2) + a configure time check to see whether RTLD_GLOBAL and thus this
restriction is necessary at all. However I'm not sure if the added
complexity would be worth it.
Regards,
Andy
--
Be nice to people on the way up, because you'll meet them on your way down.
-- Wilson Mizner
James Vega
> On Fri, Jul 23, 2010 at 02:30:19PM -0400, James Vega wrote:
>
> >Because we have to use RTLD_GLOBAL (since the C extensions aren't linked
> >against libpython), it's not safe to use both the Python 2 & 3
> This seems to be system dependent then. On my system the original
> example worked fine before the patch and I can also use python2 and
> 3
> in the same vim session (just checked with the current hg head and
> manually removed the RTLD_GLOBAL)
>
> Running ldd on the files in the lib-dynload directory reveals that
> out of my installed python versions (2.4.6, 2.5.4, 2.6.5, 2.7,
> 3.1.2) revealed that all c-extensions except those for 2.4.6 are
> linked against the corresponding libpython.so
That's interesting. As far as I can tell, Python's build system doesn't
do this normally. It would appear your distribution (which is?) has
changed that.
Kazuo Teramoto
>
> That's interesting. As far as I can tell, Python's build system doesn't
> do this normally. It would appear your distribution (which is?) has
> changed that.
In my system the files on lib-dynload are linked to libpython.so too.
I'm using ArchLinux.
$ ldd /usr/lib/python2.6/lib-dynload/termios.so|grep libpython
libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 (0x00007fdd89f0f000)
ArchLinux use a very simple build system, so if you know bash you can
read the build script at
http://repos.archlinux.org/wsvn/packages/python/trunk/PKGBUILD
--
«Dans la vie, rien n'est à craindre, tout est à comprendre»
Marie Sklodowska Curie.
Andy Kittner
I'm on gentoo linux, but it doesn't look related to any of the gentoo
specific patches to python. I manually compiled python fresh from the
original source tarball without any patches using the following commands:
./configure --prefix=/tmp/pytest2 --enable-shared && make && make install
Which results in this output from ldd:
ldd /tmp/pytest2/lib/python2.6/lib-dynload/termios.so
linux-vdso.so.1 => (0x00007fffa8eac000)
libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 (0x00007f6bc5553000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f6bc5336000)
libc.so.6 => /lib/libc.so.6 (0x00007f6bc4fd6000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f6bc4dd2000)
libutil.so.1 => /lib/libutil.so.1 (0x00007f6bc4bcf000)
libm.so.6 => /lib/libm.so.6 (0x00007f6bc494c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6bc5b72000)
So as you can see it is linked against the libpython.so.
(I case you are wondering, pytest1 was a plain configure without any arguments
but --prefix, but that didn't even build a shared python lib that could be dlopen()ed)
Regards,
Andy
--
Steal this tagline. I did.
Bram Moolenaar
James Vega wrote:
> >>> On Thu, Jul 22, 2010 at 12:36:27AM -0700, Nico Raffo wrote:
> >>> > If Vim is compiled with both --enable-pythoninterp and --enable-
> >>> > python3interp, errors occur when importing many python modules. To
> >>> > reproduce, compile as described then try something like:
> >>> >
> >>> > :py import termios
> >>> > or
> >>> > :py3 import termios
> >>> >
> >>> > Which gives the error:
> >>> >
> >>> > ImportError: /usr/lib/python3.1/lib-dynload/termios.so: undefined
> >>> > symbol: PyExc_TypeError
> >>> >
> >>> > This doesn't happen with all modules. Mostly just those with odd .so
> >>> > files.
> >>>
> >>> I had been meaning to check whether something like that would happen
> >>> with the Python interfaces. =A0I saw something similar when testing the
> >>> Perl interface. =A0Does the attached patch fix it?
> >>
> >> Thanks, I'll include it. =A0It fixes the problem on my system.
> >
> > The attached patch corrects how DYNAMIC_PYTHON_DLL is specified so that
> > it correctly includes the SONAME (libpython2.6.so.1 instead of
> > libpython2.6.so). =A0The former is always available while the latter is
> > typically only available when the relevant -dev package is installed.
> >
> > I'm looking into the crash that was reported as well, but I'm not sure
> > if that can be avoided as long as both interpreters are getting loaded.
> > At a quick glance, it seems like if_python may not be handling locking
> > properly. =A0I'll dig into this some more later tonight.
>
> Because we have to use RTLD_GLOBAL (since the C extensions aren't linked
> against libpython), it's not safe to use both the Python 2 & 3
> interfaces in the same session of Vim. It looks like the options are
> either:
>
> 1) Only allow one interface to be enabled at a time.
> 2) Keep track of whether one of the Python interfaces has been
> initialized and error out if the other one tries to be initialized.
>
> I'd personally prefer the latter since that still allows one to build a
> single Vim binary that supports both Python 2 & 3, but only one of the
> interfaces can be used per Vim session.
The problem is that then the first :python or :py3 command blocks the
others. But crashing is worse, thus we should do this at least.
Would it be possible to unload the other Python version? Thus when
using ":python" after ":py3" it would unload Python 3 and load Python 2.
Would that work?
I did a quick try by calling python_end() when Python 3 is loaded, and
calling python3_end() when Python is loaded, but switching back and
forth causes a SEGV:
#0 0x00fe0a20 in ?? ()
#1 0x0820e059 in Python_RestoreThread () at if_python.c:472
#2 0x0820e1e2 in DoPythonCommand (eap=0xbfffee8c,
cmd=0x83d5403 "print \"hello\"") at if_python.c:634
#3 0x0820e269 in ex_python (eap=0xbfffee8c) at if_python.c:673
Apparently the _end() functions don't cleaup properly.
--
SIGFUN -- signature too funny (core dumped)