Re: Vim 7.3: Python3 support
Bram Moolenaar
Roland -
You have been quiet for a while. I still have the workaround that in
one Vim session one can only use either :python or :py3 to avoid
problems mixing the shared library. Is there a better solution? I hope
you can come up with something really soon, since we are getting very
close to the 7.3 release and I don't want to risc making changes that
could crash Vim.
--
hundred-and-one symptoms of being an internet addict:
26. You check your mail. It says "no new messages." So you check it again.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ download, build and distribute -- http://www.A-A-P.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
James Vega
> Roland -
>
> You have been quiet for a while. I still have the workaround that in
> one Vim session one can only use either :python or :py3 to avoid
> problems mixing the shared library. Is there a better solution?
As far as I know, the only way to do this would be to run distinct
processes for each interpreter in order to keep the libraries from
running in the same namespace. So, any ":py*" or ":py3*" command would
start a separate process, dlopen the library, load the symbols, run
whatever command was requested, and then exit.
--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@jamessan.com>
Roland Puntaier
sorry about my late response, first I was on holiday, then I was quite busy.
I compared the vim7.3d sources to my original ones on Saturday.
Most of the functions are basically the same,
but I spotted a relevant difference:
I had:
#define load_dll(n) dlopen((n),RTLD_LAZY)
Vim 7.3 has:
#define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
I recalled, that back then I had had the same crash, probably because global static variables got mixed up.
I made the test in Vim 7.3: After removing RTLD_GLOBAL there was no crash any more.
Regards,
Roland
Br...@Moolenaar.net schrieb am 06.08.2010 21:16:39:
> Von: Bram Moolenaar <Br...@Moolenaar.net>
Bram Moolenaar
Roland Puntaier wrote:
> Hello Bram,
>
> sorry about my late response, first I was on holiday, then I was quite
> busy.
>
> I compared the vim7.3d sources to my original ones on Saturday.
> Most of the functions are basically the same,
> but I spotted a relevant difference:
>
> I had:
> #define load_dll(n) dlopen((n),RTLD_LAZY)
>
> Vim 7.3 has:
> #define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
>
> I recalled, that back then I had had the same crash, probably because
> global static variables got mixed up.
>
> I made the test in Vim 7.3: After removing RTLD_GLOBAL there was no crash
> any more.
If I'm not mistaken this flag was added to make "import termios" work.
Or something like that.
Perhaps there is another solution to make that import work without
RTLD_GLOBAL ?
--
I am also told that there is a logical proof out there somewhere
that demonstrates that there is no task which duct tape cannot handle.
-- Paul Brannan
James Vega
>
> Roland Puntaier wrote:
>
> > Hello Bram,
> >
> > sorry about my late response, first I was on holiday, then I was quite
> > busy.
> >
> > I compared the vim7.3d sources to my original ones on Saturday.
> > Most of the functions are basically the same,
> > but I spotted a relevant difference:
> >
> > I had:
> > #define load_dll(n) dlopen((n),RTLD_LAZY)
> >
> > Vim 7.3 has:
> > #define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
> >
> > I recalled, that back then I had had the same crash, probably because
> > global static variables got mixed up.
> >
> > I made the test in Vim 7.3: After removing RTLD_GLOBAL there was no crash
> > any more.
>
> If I'm not mistaken this flag was added to make "import termios" work.
> Or something like that.
Right. It's needed to be able to load any of the C extensions on
systems where they aren't linked to libpython. I know this is the case
on Debian-based systems.
Andy Kittner
Bram Moolenaar <Br...@Moolenaar.net> wrote:
>
> Roland Puntaier wrote:
... snip...
> >
> > I made the test in Vim 7.3: After removing RTLD_GLOBAL there was no crash
> > any more.
>
> If I'm not mistaken this flag was added to make "import termios" work.
> Or something like that.
The issue was that at least on some systems pythons C-Extensions seemingly
aren't linked against the python shared library, therefore they can't be
imported unless the host application provides the required symbols.
See this thread: "Linking errors when compiled with both python and python3"
http://groups.google.com/group/vim_dev/browse_frm/thread/10be77eb81ad1c2d/fed7c6d9e3932ef5?tvc=1
Regards,
Andy
Roland Puntaier
Andy Kittner <and...@gmx.de> wrote:
> > > I made the test in Vim 7.3: After removing RTLD_GLOBAL there wasno crash
> >
> > If I'm not mistaken this flag was added to make "import termios" work.
> > Or something like that.
>
> The issue was that at least on some systems pythons C-Extensions seemingly
> aren't linked against the python shared library, therefore they can't be
> imported unless the host application provides the required symbols.
>
> See this thread: "Linking errors when compiled with both python and python3"
>
Hi Bram,
I missed that thread.
I had the same problem with PyQt 4.7, first, but at a later try it worked, probably because that linked against libpython 3.1.2.
Andy reported termios to work as well on his system.
Maybe the problematic termios was older or not linked against libpython, because it was not there at the time termios was configured.
Maybe we can optimistically assume that for most python libraries / systems it works.
I would opt for the following:
If only one, Python 2.x xor Python 3.x, is configured (and if DYNAMIC_PYTHON), then use RTLD_GLOBAL:
#define load_dll(n) dlopen((n),RTLD_LAZY)
James Vega
<Roland....@br-automation.com> wrote:
> On Mon, 09 Aug 2010 13:49:43
> Andy Kittner <and...@gmx.de> wrote:
>> > > I made the test in Vim 7.3: After removing RTLD_GLOBAL there wasno
>> > > crash
>> > > any more.
>> >
>> > If I'm not mistaken this flag was added to make "import termios" work.
>> > Or something like that.
>>
>> The issue was that at least on some systems pythons C-Extensions seemingly
>> aren't linked against the python shared library, therefore they can't be
>> imported unless the host application provides the required symbols.
>>
>> See this thread: "Linking errors when compiled with both python and
>> python3"
>>
> http://groups.google.com/group/vim_dev/browse_frm/thread/10be77eb81ad1c2d/fed7c6d9e3932ef5?tvc=1
>
> Hi Bram,
>
> I missed that thread.
>
> I had the same problem with PyQt 4.7, first, but at a later try it worked,
> probably because that linked against libpython 3.1.2.
> Andy reported termios to work as well on his system.
> Maybe the problematic termios was older or not linked against libpython,
> because it was not there at the time termios was configured.
termios is a standard module in the Python distribution, so it seems to
be up to the build process that's used. I've checked my Fedora system
and the lib-dynload modules there aren't linked against libpython
either. So, it looks like at least Debian and Red Hat based distros
don't link the lib-dynload modules against libpython.
> Maybe we can optimistically assume that for most python libraries / systems
> it works.
>
> I would opt for the following:
>
> If only one, Python 2.x xor Python 3.x, is configured (and if
> DYNAMIC_PYTHON), then use RTLD_GLOBAL:
> #define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
>
> If both are configured use without RTLD_GLOBAL:
> #define load_dll(n) dlopen((n),RTLD_LAZY)
I would instead suggest a configure-time check to determine whether it's
possible to load one of the lib-dynload modules without RTLD_GLOBAL set.
Simply deciding not to use RTLD_GLOBAL because both Python versions have
been chosen means that neither will work on systems where the
lib-dynload modules aren't linked against libpython since some of the
lib-dynload modules are always used by Python.
Bram Moolenaar
James Vega wrote:
What do you suggest we do when RTLD_GLOBAL is needed? Disallow building
with two versions of Python? Or restrict use at runtime, as it is now?
We could add an option to switch using RTLD_GLOBAL on/off, but it's
really ugly. I would rather tell users to build two versions of Vim,
one with Python 2 and one with Python 3.
--
'Well, here's something to occupy you and keep your mind off things.'
'It won't work, I have an exceptionally large mind.'
-- Douglas Adams, "The Hitchhiker's Guide to the Galaxy"
Bram Moolenaar
Roland Puntaier wrote:
OK, that's a reasonable compromise, should make most things work for
most people.
Can you please document this in the help? We need to update the section
about the Python 3 support anyway! What differences are there?
--
Tips for aliens in New York: Land anywhere. Central Park, anywhere.
No one will care or indeed even notice.
Roland Puntaier
> James Vega wrote:
> > On Mon, Aug 9, 2010 at 9:49 AM, Roland Puntaier
> > <Roland....@br-automation.com> wrote:
> > > On Mon, 09 Aug 2010 13:49:43
> > > Andy Kittner <and...@gmx.de> wrote:
> > >> > > I made the test in Vim 7.3: After removing RTLD_GLOBAL there wasno
> > >> > > crash
> > >> > > any more.
> > >> >
> > >> > If I'm not mistaken this flag was added to make "import termios" work.
> > >> > Or something like that.
> > >>
> > >> The issue was that at least on some systems pythons C-
> Extensions seemingly
> > >> aren't linked against the python shared library, therefore they can't be
> > >> imported unless the host application provides the required symbols.
> > >>
> > >> See this thread: "Linking errors when compiled with both python and
> > >> python3"
> > >>
> > > http://groups.google.com/group/vim_dev/browse_frm/thread/10be77eb81ad1c2d/fed7c6d9e3932ef5?tvc=1
> > >
> > > I had the same problem with PyQt 4.7, first, but at a later try it worked,
> > > probably because that linked against libpython 3.1.2.
> > > Andy reported termios to work as well on his system.
> >
> libraries / systems
> > > it works.
> > >
> > > I would opt for the following:
> > >
> > > If only one, Python 2.x xor Python 3.x, is configured (and if
> > > DYNAMIC_PYTHON), then use RTLD_GLOBAL:
> > > #define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
> > >
> > > If both are configured use without RTLD_GLOBAL:
> > > #define load_dll(n) dlopen((n),RTLD_LAZY)
> >
> > I would instead suggest a configure-time check to determine whether it's
> > possible to load one of the lib-dynload modules without RTLD_GLOBAL set.
> > Simply deciding not to use RTLD_GLOBAL because both Python versions have
> > been chosen means that neither will work on systems where the
> > lib-dynload modules aren't linked against libpython since some of the
> > lib-dynload modules are always used by Python.
>
> What do you suggest we do when RTLD_GLOBAL is needed? Disallow building
> with two versions of Python? Or restrict use at runtime, as it is now?
> We could add an option to switch using RTLD_GLOBAL on/off, but it's
> really ugly. I would rather tell users to build two versions of Vim,
> one with Python 2 and one with Python 3.
So without RTLD_GLOBAL we have found that certain python C extension modules that were not linked to libpython cannot be loaded.
This is less a problem than a crash.
It can also be tackled by recompiling that extension module or the system package. For termios, I suppose recompiling python with --enable-shared should work.
My assumption is that certain distros distribute python without --enable-shared and the libpython gets into the system by other packages.
On the other hand a crash is not acceptable and thus RTLD_GLOBAL when both python version are enable is not acceptable.
To find out during configuration is expensive and the outcome is only that enabling both can lead to one, possibly both version with reduced functionality.
Such a message could be issued at configuration time, but without the expense to actually verify, whether there will be reduced functionality.
To disable one python version either at configuration time or runtime is against the intention of the user to have both enabled.
Maybe the user can live with the reduced python functionality. If not, there are options:
1) reconfigure vim for one python version only
2) recompile and properly link the problematic library, or python as a whole in case of its standard libraries.
Regards,
Roland
Roland Puntaier
> > I would opt for the following:
> >
> > If only one, Python 2.x xor Python 3.x, is configured (and if
> > DYNAMIC_PYTHON), then use RTLD_GLOBAL:
> > #define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
> >
> > If both are configured use without RTLD_GLOBAL:
> > #define load_dll(n) dlopen((n),RTLD_LAZY)
>
> OK, that's a reasonable compromise, should make most things work for
> most people.
>
> Can you please document this in the help? We need to update the section
> about the Python 3 support anyway! What differences are there?
and the possible restrictions in python functionality, if both versions are enabled, as discussed in this thread.
I had a look at the current documentation. You already mention Python 3.
Since there are no other differences this should suffice.
With the above compromise I suggest the following as last chapter in if_pyth.txt.
==============================================================================
7. Python 3 *python3*
*:py3* *:python3*
The |:py3| and |:python3| commands work similar to |:python|.
*:py3file*
The |:py3file| command works similar to |:pyfile|.
Vim can be built in four ways (:version output):
1. No Python support (-python, -python3)
2. Python 2 support only (+python or +python/dyn, -python3)
3. Python 3 support only (-python, +python3 or +python3/dyn)
4. Python 2 and 3 support (+python/dyn, +python3/dyn)
You can see that when Python 2 and Python 3 are both supported they must be
loaded dynamically.
On Linux/Unix systems this can only be done without importing global symbols.
In this case python's "import" might fail, if the library expects the symbols
to be provided by vim. To work around this
1. either the problematic library, or python in case of
standard libraries, must be recompiled to link to the
according libpython.so file (--enable-shared in case of python).
2. Or you recompile vim for only one python version.
In this case all symbols can be imported into vim.
==============================================================================
Regards,
Roland
Bram Moolenaar
Roland Puntaier wrote:
> > What do you suggest we do when RTLD_GLOBAL is needed? Disallow building
> > with two versions of Python? Or restrict use at runtime, as it is now?
> > We could add an option to switch using RTLD_GLOBAL on/off, but it's
> > really ugly. I would rather tell users to build two versions of Vim,
> > one with Python 2 and one with Python 3.
>
> So without RTLD_GLOBAL we have found that certain python C extension
> modules that were not linked to libpython cannot be loaded.
> This is less a problem than a crash.
Certainly.
> It can also be tackled by recompiling that extension module or the
> system package. For termios, I suppose recompiling python with
> --enable-shared should work.
> My assumption is that certain distros distribute python without
> --enable-shared and the libpython gets into the system by other packages.
>
> On the other hand a crash is not acceptable and thus RTLD_GLOBAL when both
> python version are enable is not acceptable.
> To find out during configuration is expensive and the outcome is only that
> enabling both can lead to one, possibly both version with reduced
> functionality.
> Such a message could be issued at configuration time, but without the
> expense to actually verify, whether there will be reduced functionality.
> To disable one python version either at configuration time or runtime is
> against the intention of the user to have both enabled.
> Maybe the user can live with the reduced python functionality. If not,
> there are options:
> 1) reconfigure vim for one python version only
> 2) recompile and properly link the problematic library, or python
> as a whole in case of its standard libraries.
This is how it works now. When building with one of the interfaces
RTLD_GLOBAL is used an everything should work. When building with both
RTLD_GLOBAL is not used and Vim won't crash but some imports may fail.
The need for indluding both interfaces mainly comes from Linux
distributors, who would like to have one version of Vim package that
works for all users. For these it might actually be better to use
RTLD_GLOBAL and disallow using both ":python" and ":py3" in one session.
In fact, only one of the two libraries may be installed.
A configure check would help to decide whether everything works without
RTLD_GLOBAL global. If it does, then the current solution is best. If
it doesn't it might be better to switch to the other solution: do use
RTLD_GLOBAL but disallow using both python commands.
--
On the other hand, you have different fingers.
-- Steven Wright
Bram Moolenaar
Roland Puntaier wrote:
> There are no differences apart from :py3 <-> :py[thon], and the discussed
> consequences,
> and the possible restrictions in python functionality, if both versions
> are enabled, as discussed in this thread.
>
> I had a look at the current documentation. You already mention Python 3.
> Since there are no other differences this should suffice.
>
> With the above compromise I suggest the following as last chapter in
> if_pyth.txt.
Thanks, I'll include this.
--
Change is inevitable, except from a vending machine.
James Vega
> A configure check would help to decide whether everything works without
> RTLD_GLOBAL global. If it does, then the current solution is best. If
> it doesn't it might be better to switch to the other solution: do use
> RTLD_GLOBAL but disallow using both python commands.
Attached is a configure check that determines if RTLD_GLOBAL is needed,
when building with dynamic python. If so, it enables the "only one
Python interface can be used per session" code.
Roland Puntaier
> On Tue, Aug 10, 2010 at 11:15:41PM +0200, Bram Moolenaar wrote:
> > A configure check would help to decide whether everything works without
> > RTLD_GLOBAL global. If it does, then the current solution is best. If
> > it doesn't it might be better to switch to the other solution: do use
> > RTLD_GLOBAL but disallow using both python commands.
>
> Attached is a configure check that determines if RTLD_GLOBAL is needed,
> when building with dynamic python. If so, it enables the "only one
> Python interface can be used per session" code.
I agree with you, that it makes sense, to have such a test at configuration time.
It makes a good guess for the system.
Since we are at the configuration step, one could turn off Python2 or Python3 altogether,
but I agree that it's not so bad to keep them enabled, so,
when starting vim, the first Python version used will be switched on,
i.e. the user can have both in a binary distribution, just not both at the same time.
The user is unlikely to need both python versions at the same time, I would say.
My other arguments are for those who do their own configuration and would like to have both enabled
even if one will not work for some extension libraries.
But probably one can also expect from them that they manually define PY_NO_RTLD_GLOBAL.
The C code checks for Python 3 only.
Whether Python 2 needs RTLD_GLOBAL is independent of whether Python3 needs RTLD_GLOBAL.
The crash happens if any one or both use RTLD_GLOBAL.
I changed it. Here is the new diff.
The define should be PY_NO_RTLD_GLOBAL, so if not defined, as is the case for a configuration of only one dynamically linked Python version, it defaults to using RTLD_GLOBAL.
But I've seen that unlike for e.g. PERL or LUA --enable-pythoninterp=dynamic does not work. I have changed that, too.
Roland
Bram Moolenaar
Roland Puntaier wrote:
Thanks. I'll have a closer look tomorrow.
Can you please also write some text for the help about this?
I'll change the E999 to a valid number, we should have a tag in the help
file that explains the message.
--
hundred-and-one symptoms of being an internet addict:
46. Your wife makes a new rule: "The computer cannot come to bed."
Dominique Pellé
> I'll change the E999 to a valid number, we should have a tag in the help
> file that explains the message.
Should each translator update the E999 in src/po/*.po files to the new
value or can it be done automatically?
Here are the po file which already contain E999:
$ cd vim/src/po/ && grep -l E999 *.po
eo.po
fi.po
fr.po
it.po
pl.cp1250.po
pl.po
pl.UTF-8.po
sk.cp1250.po
sk.po
uk.cp1251.po
uk.po
Regards
-- Dominique
Bram Moolenaar
Dominique Pelle wrote:
So short before a release I'll just change the number in the .po files.
It should not have been E999 anyway, it was a placeholder that I forgot
to update.
--
The real
trick is
this: to
keep the
lines as
short as
possible
and keep
the size
the same
yet free
from the
need for
hyphena-
Dammit!! (Matthew Winn)
Roland Puntaier
Br...@Moolenaar.net wrote :
> Can you please also write some text for the help about this?
> I'll change the E999 to a valid number, we should have a tag in the help
> file that explains the message.
In my yesterday's patch I have changed the E999 line in if_python.c and if_python3.c.
Please restore it to the original line.
Here is the diff for if_pyth.txt with a E999 tag.
Bram Moolenaar
Roland Puntaier wrote:
Thanks, I'll include it.
--
hundred-and-one symptoms of being an internet addict:
57. You begin to wonder how on earth your service provider is allowed to call
200 hours per month "unlimited."