Dynamically loading Python library
björn
A couple of people have requested that MacVim be linked against Python
2.5 on Leopard but this presents problems since the binary I release
must run on Tiger as well (which only comes with Python 2.3). It was
then suggested that Python could be loaded dynamically (like on
Windows) but I was reluctant to look into this myself and was hoping
somebody who actually used Python 2.5 would do this and send me the
patch. Well, that didn't happen so I took a quick peek myself.
Now, I have managed to get Python to load dynamically and have tested
on Leopard/Intel and Tiger/PPC and on the former it loads 2.5, on the
latter 2.3. Yay! Or not. It turns out that there was an API change
between 2.3 and 2.5 and this is checked via a define
PYTHON_API_VERSION which is defined inside a Python header
(modsupport.h). Since I compile on Leopard the API version is 1013,
but 2.3 has API version 1012 so when I call ":python ..." on Tiger I
get the following warning:
sys:1: RuntimeWarning: Python C API version mismatch for module vim:
This Python has API version 1012, module vim has version 1013.
It feels like I've put in an effort here so now I am hoping somebody
will help me out with this problem because I am not sure what to do.
I don't see a solution: I have to build using some Python headers --
but which? I can't exactly "dynamically load" which header to build
against, if you catch my drift. How is this solved on Windows?
Please, even if nobody knows the answer to this question I urge you to
look into it and help me out, else I will be forced to continue
distributing MacVim with Python 2.3.
The attached archive contains a patch (against if_python.c) and the
"config.mk" file I used. To build, I ran the following configure
line:
./configure --enable-gui=macvim --enable-pythoninterp --with-mac-arch=both
and then I edited "src/auto/config.mk" as follows:
1. add -DDYNAMIC_PYTHON to "DEFS"
2. clear PYTHON_LIBS, PYTHON_CONFDIR, PYTHON_GETPATH_CFLAGS so that
Python isn't linked statically (these lines are filled in by the
configure script)
The "config.mk" in the archive has exactly these changes so you should
be able to copy it into "src/auto" after running configure and then
call "make".
Thanks in advance,
Björn
Jason Foreman
Hi Björn,
On Nov 1, 2008, at 11:57 AM, björn wrote:
> How is this solved on Windows?
I believe you must have installed the version of Python that your Vim
distribution was built with. For standard builds I think this is
2.3. Cream builds use 2.5 or maybe 2.4. I can verify this on Monday
at work.
> Please, even if nobody knows the answer to this question I urge you to
> look into it and help me out, else I will be forced to continue
> distributing MacVim with Python 2.3.
I don't have an answer, but I'm willing to help if I can. I'm not
100% sure there will be a solution short of having separate Leopard
and Tiger builds, but it's worth trying.
One thing that might work is to use "Py_InitModule3" instead of
"Py_InitModule4" in if_python.c:2361. The former does not take an API
version parameter, so that check won't be performed. Even the online
Python docs [1] say that use of "Py_InitModule3" is preferred in most
cases. I haven't got Tiger setup at the moment to try this out. I'll
work on getting Tiger going this weekend so I can try your patch.
Another thing to try would be downloading the Python 2.3 sources and
using the headers from there when building, instead of the Python 2.5
headers included with Leopard. This would ensure the
PYTHON_API_VERSION matches. But I'm not sure if this would fix the
problem or just invert it (no warn on Tiger, warn on Leopard).
Jason
[1] http://www.python.org/doc/2.5.2/api/allocating-objects.html
George V. Reilly
> On Nov 1, 2008, at 11:57 AM, björn wrote:
>> How is this solved on Windows?
>
> I believe you must have installed the version of Python that your Vim
> distribution was built with. For standard builds I think this is 2.3.
> Cream builds use 2.5 or maybe 2.4. I can verify this on Monday at work.
This is certainly correct for the Visual C++ build, Make_mvc.mak, and
seems to be true of the MinGW build too.
Try asking on http://mail.python.org/mailman/listinfo/capi-sig. That's
where the experts on embedding Python hang out.
If Python is dynamically loaded, then people running Python 2.3 on
Tiger don't pay any cost until they use :python. I'm inclined to think
that most people who want to use :python have Python 2.5 or 2.6
installed.
--
/George V. Reilly geo...@reilly.org
http://www.georgevreilly.com/blog http://blogs.cozi.com/tech
cfr...@imapmail.org
> 2008/11/1 Jason Foreman <ja...@threeve.org>:
>> On Nov 1, 2008, at 11:57 AM, bj?rn wrote:
>>> How is this solved on Windows?
>>
>> I believe you must have installed the version of Python that your Vim
>> distribution was built with. For standard builds I think this is 2.3.
>> Cream builds use 2.5 or maybe 2.4. I can verify this on Monday at work.
>
> This is certainly correct for the Visual C++ build, Make_mvc.mak, and
> seems to be true of the MinGW build too.
>
> Try asking on http://mail.python.org/mailman/listinfo/capi-sig. That's
> where the experts on embedding Python hang out.
>
> If Python is dynamically loaded, then people running Python 2.3 on
> Tiger don't pay any cost until they use :python. I'm inclined to think
> that most people who want to use :python have Python 2.5 or 2.6
> installed.
In case it matters, Python 2.5 on Tiger (from python.org) does not
include the dynamically linked shared library. This is so whether one
uses the installer provided or whether one compiles the framework build
from source and is true even if one explicitly requests that the
library be built and installed. I don't have Leopard to compare, but I
believe it does include the library in Python just as the Apple
provided python 2.3 provides it on Tiger.
As I say, I don't know if this matters or not.
- cfr
>
Ben Schmidt
> on Leopard/Intel and Tiger/PPC and on the former it loads 2.5, on the
> latter 2.3. Yay! Or not. It turns out that there was an API change
> between 2.3 and 2.5 and this is checked via a define
> PYTHON_API_VERSION which is defined inside a Python header
> (modsupport.h).
Hehe. I could have told you that and saved you the trouble. It's come up
on one of the Vim mailing lists before. It's not solved on Windows, but
IIRC, gives even more drastic problems than it seems to on the Mac. :-\
Unless you know exactly what the API changes are, you're pretty much
stuffed. Any change in a structure Vim uses or the signature or return
type of a function Vim calls, or a constant/macro definition Vim uses,
and things will go wrong if the wrong version is dynamically loaded. In
a sense, the warning is the least of your worries.
You could ask the Python guys to try to find out exactly what the API
changes are and whether they apply to Vim or not, but this could be
difficult to ascertain--these things tend to be a bit of a haystack. On
the other hand, it could be the change is very simple and irrelevant,
and you can just somehow disable the warning--if this is a likely
scenario, the Python guys almost certainly will have provided a way to
disable it.
If it can't be ascertained whether change is irrelevant, or it can be
ascertained that it *is* relevant, it would be technically possible to
make a Vim that worked with both API versions, but you'd have to
basically duplicate if_python.c, change all the function names so there
were two versions of each, compile one against the 2.3 API and another
against the 2.5 API and then have a runtime check to decide which
version of the functions to call. It would be quite a refactoring job,
and I would say not worth the effort.
One definite solution, though not ideal, but which is easy to set up for
building, as well as easy for users to understand: have separate MacVim
builds for Tiger and Leopard. Stacks of apps do it, and Mac users are
fairly used to downloading the version appropriate to their OS version.
Ben.
Ben Schmidt
> Tiger don't pay any cost until they use :python. I'm inclined to think
> that most people who want to use :python have Python 2.5 or 2.6
> installed.
I'm not so sure about that. I do use :python, but only indirectly, by
using plugins that use it. As such, if it's good enough for the plugin,
it's good enough for me, and I won't bother installing a later version
than the one the OS provides unless something really requires it. I'm
certainly not going to install one just for the love of Python or of
being on the cutting edge. And to be honest, I wouldn't be sure that if
I did install a later one, that would be the one MacVim would find first
in the path. I suspect there are other users like me. In fact, I suspect
more people use :python via plugins than use it directly in their own
scripts or from the commandline.
None of this actually affects me, as I compile MacVim from source
myself, linking statically against whatever Python it finds. But I'm
just pointing out that I expect it would affect others.
Ben.
George V. Reilly
> If Python is dynamically loaded, then people running Python 2.3 onI'm not so sure about that. I do use :python, but only indirectly, by
> Tiger don't pay any cost until they use :python. I'm inclined to think
> that most people who want to use :python have Python 2.5 or 2.6
> installed.
using plugins that use it.
When I ported the Vim Python interface code to Win64, I looked around for Vim plugins to exercise it, and I had a hard time finding any. That was about a year ago. Maybe people have started using it more.
Anyway, separate builds for Tiger and Leopard is probably the best way to go.
Ben Schmidt
> 2008/11/2 Ben Schmidt <mail_ben...@yahoo.com.au
> <mailto:mail_ben...@yahoo.com.au>>
>
> > If Python is dynamically loaded, then people running Python 2.3 on
> > Tiger don't pay any cost until they use :python. I'm inclined to
> think
> > that most people who want to use :python have Python 2.5 or 2.6
> > installed.
>
> I'm not so sure about that. I do use :python, but only indirectly, by
> using plugins that use it.
>
> When I ported the Vim Python interface code to Win64, I looked around
> for Vim plugins to exercise it, and I had a hard time finding any. That
> was about a year ago. Maybe people have started using it more.
I expect there are more plugins that use it on non-Windows systems.
I think I have a couple installed that use it, but certainly the main
one is
http://www.vim.org/scripts/script.php?script_id=1929
which I use from time to time to debug PHP. I first started using it
about a year ago too. It's the sort of thing that can make a big
difference to your coding experience, requires Python to be working, but
doesn't need the latest Python version. And certainly the user doesn't
want to be fiddling with any of the Python code or other Python-related
problems (like me, may not even know any Python). They already have
problems in their own code...
Grins,
Ben.
björn
>
> You could ask the Python guys to try to find out exactly what the API
> changes are and whether they apply to Vim or not, but this could be
> difficult to ascertain--these things tend to be a bit of a haystack. On
> the other hand, it could be the change is very simple and irrelevant,
> and you can just somehow disable the warning--if this is a likely
> scenario, the Python guys almost certainly will have provided a way to
> disable it.
I don't think it is a feasible solution...e.g. what happens the next
time the API changes? No, I think loading Python dynamically is out
of the question. :(
> If it can't be ascertained whether change is irrelevant, or it can be
> ascertained that it *is* relevant, it would be technically possible to
> make a Vim that worked with both API versions, but you'd have to
> basically duplicate if_python.c, change all the function names so there
> were two versions of each, compile one against the 2.3 API and another
> against the 2.5 API and then have a runtime check to decide which
> version of the functions to call. It would be quite a refactoring job,
> and I would say not worth the effort.
Ugh, yeah, that's no good either.
> One definite solution, though not ideal, but which is easy to set up for
> building, as well as easy for users to understand: have separate MacVim
> builds for Tiger and Leopard. Stacks of apps do it, and Mac users are
> fairly used to downloading the version appropriate to their OS version.
It seems this is the only way to go then. But...I wonder if it really
is worth my time making two distributions just so a few users get to
use Python 2.5 instead of 2.3. I think I need a better reason than
that for something as drastic as distributing two different versions
of the binary. Maybe I underestimate the number of users who would
benefit from 2.5, but for now I'll just keep building with 2.3. :-/
Thanks to everybody for the feedback,
Björn
Jason Foreman
> 2008/11/3 Ben Schmidt <mail_ben...@yahoo.com.au>:
>> have separate MacVim builds for Tiger and Leopard.
>
> is worth my time making two distributions just so a few users get to
> use Python 2.5 instead of 2.3. I think I need a better reason than
> that for something as drastic as distributing two different versions
> of the binary. Maybe I underestimate the number of users who would
> benefit from 2.5, but for now I'll just keep building with 2.3. :-/
FWIW, I've been toying with the idea of making my own Leopard-only
builds available recently. I always build my MacVim from source using
Python 2.5 and 10.5 SDK. I do also use a couple of extra patches in
my build (relative number, Lua, working on vimgdb). If there is
interest, I could put my builds up somewhere for others to use. My
main concern would be increasing the support load due to unofficial
builds floating around.
Jason
Matt Tolton
> is worth my time making two distributions just so a few users get to
> use Python 2.5 instead of 2.3. I think I need a better reason than
> that for something as drastic as distributing two different versions
> of the binary. Maybe I underestimate the number of users who would
> benefit from 2.5, but for now I'll just keep building with 2.3. :-/
What about feature freezing the current 10.4 version and continuing
development for 10.5 only? Then you could enable garbage collection
which would be much more convenient for development -- this issue bugs
me especially because if someone wants to make a plugin, I'm pretty
sure they can't use garbage collection since MacVim doesn't.
Nico Weber
>> really
>> is worth my time making two distributions just so a few users get to
>> use Python 2.5 instead of 2.3. I think I need a better reason than
>> that for something as drastic as distributing two different versions
>> of the binary. Maybe I underestimate the number of users who would
>> benefit from 2.5, but for now I'll just keep building with 2.3. :-/
That sounds reasonable.
> What about feature freezing the current 10.4 version and continuing
> development for 10.5 only? Then you could enable garbage collection
> which would be much more convenient for development -- this issue bugs
> me especially because if someone wants to make a plugin, I'm pretty
> sure they can't use garbage collection since MacVim doesn't.
I don't use Tiger, but I don't think dropping Tiger already is a good
idea (see e.g. http://update.omnigroup.com/).
Nico
Ben Schmidt
>>> It seems this is the only way to go then. But...I wonder if it
>>> really
>>> is worth my time making two distributions just so a few users get to
>>> use Python 2.5 instead of 2.3. I think I need a better reason than
>>> that for something as drastic as distributing two different versions
>>> of the binary. Maybe I underestimate the number of users who would
>>> benefit from 2.5, but for now I'll just keep building with 2.3. :-/
>
> That sounds reasonable.
+1.
Real Python-heads can always compile themselves.
>> What about feature freezing the current 10.4 version and continuing
>> development for 10.5 only? Then you could enable garbage collection
>> which would be much more convenient for development -- this issue bugs
>> me especially because if someone wants to make a plugin, I'm pretty
>> sure they can't use garbage collection since MacVim doesn't.
>
> I don't use Tiger, but I don't think dropping Tiger already is a good
> idea (see e.g. http://update.omnigroup.com/).
I agree, but I do use Tiger. 10.4.9. :-)
Ben.
Panos
went ahead a compiled from source. Easy as pie, plus now I have python
2.5 support :)
Thanks for making it that easy, although I still believe that people
that actually use python are way past 2.3 by now.
Still it's easier to have them compile it themselves, than to ask them
to move up a version of their OS, so I see why we're having this
issue.
It should be made easier to pick python version depending on the build
platform, hope we can figure this out.
björn
>
> FWIW, I've been toying with the idea of making my own Leopard-only
> builds available recently. I always build my MacVim from source using
> Python 2.5 and 10.5 SDK. I do also use a couple of extra patches in
> my build (relative number, Lua, working on vimgdb). If there is
> interest, I could put my builds up somewhere for others to use. My
> main concern would be increasing the support load due to unofficial
> builds floating around.
Jason, feel free to make your own builds available -- I have no
objections to this.
(Sorry for taking so long to reply to your post!)
Björn