PATCH: dynamically load Python on Solaris

[Danek Duvall]

I've been wanting to code this up for a while, and almost had it a while
back, but couldn't quite get it working, and put it aside until today.

It's not ready for integration, since it's not at all tied into the
autoconf setup, and requires that you modify src/auto/config.mk to add
-DDYNAMIC_PYTHON to PYTHON_CFLAGS and wrap -lpython2.4 in PYTHON_LIBS with
-zlazyload / -znolazyload, but otherwise it seems to work: python isn't
loaded until the first python command is executed, and (as far as I've been
able to tell) executes python code just fine after that, has("python") is
true, etc.

Comments on the patch itself are appreciated. Suggestions on how it might
be made to work on Linux are also appreciated, though I don't really have
any means of testing it at the moment. And help with getting it tied in
with autoconf would be great; I'm not sure how to move forward on that. If
there are any Python script torture tests I should run, let me know; I'm
not sure what the best python scripts are to play with.

I haven't looked at doing the other interpreters yet, though I'd like to do
that, as well as see if it's possible to dynamically load the GUI.

Thanks for any thoughts,
Danek

[Doug Kearns]

On 2/15/09, Danek Duvall <duv...@comfychair.org> wrote:

<snip>

> If
> there are any Python script torture tests I should run, let me know; I'm
> not sure what the best python scripts are to play with.

You could play with the distributed Python omni completion:
autoload/pythoncomplete.vim

<snip>

Regards,
Doug

[Bram Moolenaar]

Danek Duvall wrote:

I haven't looked at the patch yet, but I would like to encourage others
to help make this work for as many systems as possible. It's so much
nicer when Vim can be build with Python and the binary distributed to
systems where Python is not available. Less versions, more flexibility.

I assume that for building the Python headers and tools are still
required, thus it only helps for when distributing a Vim binary.

--
From "know your smileys":
:-)-O Smiling doctor with stethoscope

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ download, build and distribute -- http://www.A-A-P.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Danek Duvall]

On Sun, Feb 15, 2009 at 02:57:35PM +0100, Bram Moolenaar wrote:

> I haven't looked at the patch yet, but I would like to encourage others
> to help make this work for as many systems as possible. It's so much
> nicer when Vim can be build with Python and the binary distributed to
> systems where Python is not available. Less versions, more flexibility.
>
> I assume that for building the Python headers and tools are still
> required, thus it only helps for when distributing a Vim binary.

Correct.

One question I have is how flexible it should be. I did this the way I did
because I could easily hang off the work that had been done for Windows,
but if I were to make a dynamic module -- consisting of if_python.o and
py_config.o -- then I could make one for each version of Python available
on the system at build time, and then try loading them in turn until one
was successful. So you could build on a system that had 2.4, 2.5, and 2.6,
and work on a system that had any of the three.

Of course, fully fleshing that idea out means giving some control to the
user of which version they want to use, or a preference ordering of the
versions.

And further down that road would be to experiment with whether multiple
versions of Python can be loaded into memory at the same time. If that's
the case, then you could have different scripts running with different
versions in the same vim session. I don't know how useful that would be in
practice, though.

Danek

[Danek Duvall]

Ah, thanks. Seems to work.

Danek

[Danek Duvall]

Okay, I've hacked configure.in to make it possible to do this from
configure. The patch is attached. It applies only on top of my previous
patch, as it passes the python library name in as a define. This is gross,
but it gets the job done for now.

Thanks,
Danek

[Danek Duvall]

So here's a re-work that should allow for dynamic loading on platforms
other than Solaris. The autoconf bits are tuned for Solaris, but change
-Kpic to -fpic, -G to -shared, and SunOS to Linux, and I think it might
work, or at least come close.

The difference here is that if_python.c itself is converted into a shared
object which can be loaded by vim. That way, if_python.so can link simply
against python and not have to worry about the mess of dlsym calls, lazy
loading, etc, that you needed in the main program in my first attempt.
Then vim either loads or fails to load if_python.so.

New patch is attached.

Thanks,
Danek

[björn]

Hi Danek,

2009/2/15 Danek Duvall:

>
> And further down that road would be to experiment with whether multiple
> versions of Python can be loaded into memory at the same time. If that's
> the case, then you could have different scripts running with different
> versions in the same vim session. I don't know how useful that would be in
> practice, though.

Hi Danek,

This would be very useful for the Mac OS X port of Vim (MacVim) since
10.4 comes with Python 2.3 and 10.5 ships with Python 2.5. At the
moment I link MacVim against 2.3 so that it works on both 10.4 and
10.5 but several users have requested 2.5. I actually implemented
dynamic loading of Python for OS X but since there are Python API
changes between 2.3 and 2.5 you get a warning when trying to run on
2.3 if the binary was built using 2.5 (even though the library itself
is successfullly loaded dynamically).

Anyway, if you get around to implementing this I would be very eager
to test it and include it in MacVim. By the way, how exactly do you
propose to do this? Looking at your latest patch I assume you'd build
one if_python module for each version and then decide which one to
load when python is first used?

Björn

[Danek Duvall]

björn wrote:

> 2009/2/15 Danek Duvall:
> >
> > And further down that road would be to experiment with whether multiple
> > versions of Python can be loaded into memory at the same time. If that's
> > the case, then you could have different scripts running with different
> > versions in the same vim session. I don't know how useful that would be in
> > practice, though.
>

> This would be very useful for the Mac OS X port of Vim (MacVim) since
> 10.4 comes with Python 2.3 and 10.5 ships with Python 2.5. At the
> moment I link MacVim against 2.3 so that it works on both 10.4 and
> 10.5 but several users have requested 2.5.

So that speaks for having multiple versions available, but not necessarily
for having them all co-resident.

> I actually implemented dynamic loading of Python for OS X but since there
> are Python API changes between 2.3 and 2.5 you get a warning when trying
> to run on 2.3 if the binary was built using 2.5 (even though the library
> itself is successfullly loaded dynamically).

Yup, I saw that as well when playing around.

> By the way, how exactly do you propose to do this? Looking at your
> latest patch I assume you'd build one if_python module for each version
> and then decide which one to load when python is first used?

That's the idea. I'm just not sure how that decision should be made. I
guess I'm looking for some guidance from Bram, though not being terribly
familiar with the vim development model, I don't know if I should be
looking elsewhere for that guidance.

At any rate, if I were to hack something together right now, I'd probably
introduce a new variable that would let you set the order:

set pythonversions=2.3,2.4,2.5

with tags of "oldest" and "newest" available as well. The first module
matching an element in the series that loaded properly would be the one
used, defaulting to either oldest or newest if the variable isn't set (or
if none of them match?). The modules would need to be named consistently
-- if_python<ver>.so -- and in a collatable order, though I suppose I could
always try to query the module for the definitive version if the collation
proves too difficult.

But there might be some pitfalls behind that. Suggestions and guidance are
welcomed.

Thanks,
Danek

[björn]

2009/2/17 Danek Duvall:

>
> björn wrote:
>
>> 2009/2/15 Danek Duvall:
>> >
>> > And further down that road would be to experiment with whether multiple
>> > versions of Python can be loaded into memory at the same time. If that's
>> > the case, then you could have different scripts running with different
>> > versions in the same vim session. I don't know how useful that would be in
>> > practice, though.
>>
>> This would be very useful for the Mac OS X port of Vim (MacVim) since
>> 10.4 comes with Python 2.3 and 10.5 ships with Python 2.5. At the
>> moment I link MacVim against 2.3 so that it works on both 10.4 and
>> 10.5 but several users have requested 2.5.
>
> So that speaks for having multiple versions available, but not necessarily
> for having them all co-resident.

Yes that is all I am after -- I can't see any particular use for
having two or more co-resident. (?)

>> By the way, how exactly do you propose to do this? Looking at your
>> latest patch I assume you'd build one if_python module for each version
>> and then decide which one to load when python is first used?
>
> That's the idea. I'm just not sure how that decision should be made. I
> guess I'm looking for some guidance from Bram, though not being terribly
> familiar with the vim development model, I don't know if I should be
> looking elsewhere for that guidance.
>
> At any rate, if I were to hack something together right now, I'd probably
> introduce a new variable that would let you set the order:
>
> set pythonversions=2.3,2.4,2.5
>
> with tags of "oldest" and "newest" available as well. The first module
> matching an element in the series that loaded properly would be the one
> used, defaulting to either oldest or newest if the variable isn't set (or
> if none of them match?). The modules would need to be named consistently
> -- if_python<ver>.so -- and in a collatable order, though I suppose I could
> always try to query the module for the definitive version if the collation
> proves too difficult.
>
> But there might be some pitfalls behind that. Suggestions and guidance are
> welcomed.

I think this is overkill. Why not just see which modules are
installed and try loading from newest to oldest? Well, maybe it isn't
that simple since loading will probably work as long as the module is
present even if the matching python version is not installed. But it
provides pretty much the same functionality as the option you
suggested (the user simply has to make sure that only the right module
is installed) and it means less code. I'm pretty new to Vim
development myself, but it seems the less new options you introduce
the greater the chance that your patch will be accepted.

Anyway, as far as OS X is concerned the best way would be if the most
recent version of python installed was automatically used (judging
from the requests I've had from users) without the need of user
involvement. Would this be possible? (Assuming the user has the
matching if_python modules installed...in the case of OS X these would
be distributed with the Vim binary inside the so-called application
bundle.)

Björn

[Danek Duvall]

On Tue, Feb 17, 2009 at 07:32:35PM +0100, björn wrote:

> Yes that is all I am after -- I can't see any particular use for
> having two or more co-resident. (?)

The use case would be if you had a script that could only run with 2.3 and
another that could only run with 2.6, and you wanted to use both in the
same vim process. I agree that it's all that worth worrying about, though.

> I think this is overkill. Why not just see which modules are
> installed and try loading from newest to oldest?

Works for me. I can always change it later to be fancier if people find
the simple method to not be enough.

> Well, maybe it isn't that simple since loading will probably work as long
> as the module is present even if the matching python version is not
> installed.

That'll probably depend on the features of the dynamic linker. On Solaris,
at least, the module will fail to load if a dependency it has isn't there.

I may not get to it before this weekend, but I'll post another patch as
soon as I have it. In the meantime, if you have a chance to try my latest
patch out on OS X, I'd be curious to know if it works there.

Thanks,
Danek

[björn]

2009/2/17 Danek Duvall:

>
> I may not get to it before this weekend, but I'll post another patch as
> soon as I have it. In the meantime, if you have a chance to try my latest
> patch out on OS X, I'd be curious to know if it works there.

The following bit of configure.in looked a bit strange, so I changed
it. No idea what that special "if" for MACOSX does though.

diff --git a/src/configure.in b/src/configure.in
index b623c6f..bd32b12 100644
--- a/src/configure.in
+++ b/src/configure.in
@@ -742,16 +742,12 @@ eof
fi
PYTHON_SRC="if_python.c"
dnl For Mac OSX 10.2 config.o is included in the Python library.
- if test "x$MACOSX" = "xyes"; then
- PYTHON_OBJ="objects/if_python.o"
- else
- if test "${enable_pythoninterp}" = "dynamic"; then
- PYTHON_OBJ="objects/if_python_stub.o"
- PYTHON_SO_OBJ="objects/if_python.o objects/py_config.o"
- else
- PYTHON_OBJ="objects/if_python_stub.o objects/if_python.o objects/py_
- fi
- fi
+ if test "${enable_pythoninterp}" = "dynamic"; then
+ PYTHON_OBJ="objects/if_python_stub.o"
+ PYTHON_SO_OBJ="objects/if_python.o objects/py_config.o"
+ else
+ PYTHON_OBJ="objects/if_python_stub.o objects/if_python.o objects/py_c
+ fi
if test "${vi_cv_var_python_version}" = "1.4"; then
PYTHON_OBJ="$PYTHON_OBJ objects/py_getpath.o"
fi

With this change I did regenerated the configure script and tried
building but something's up but I haven't got time to investigate
right now. Probably just some flag to the compiler/linker that is
missing...but I get the following errors:

gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
-Wno-unknown-pragmas -pipe -I. -Iproto -DMACOS_X_UNIX -no-cpp-precomp
-I/Developer/Headers/FlatCarbon -g -O -D_FORTIFY_SOURCE=1
-DDYNAMIC_PYTHON
-I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5
-Kpic -o objects/if_python.o if_python.c
gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
-Wno-unknown-pragmas -pipe -I. -Iproto -DMACOS_X_UNIX -no-cpp-precomp
-I/Developer/Headers/FlatCarbon -g -O -D_FORTIFY_SOURCE=1
-DDYNAMIC_PYTHON -o objects/py_config.o
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/config.c
\
-I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5
-I/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config
-DHAVE_CONFIG_H -DNO_MAIN
gcc -o if_python.so objects/if_python.o objects/py_config.o
Undefined symbols:
"_initimp", referenced from:
__PyImport_Inittab in py_config.o
"_ml_get_buf", referenced from:
_RBSlice in if_python.o
_GetBufferLine in if_python.o
"_u_savesub", referenced from:
_SetBufferLine in if_python.o
"_PyInt_Type", referenced from:
_PyInt_Type$non_lazy_ptr in if_python.o
"_ml_append", referenced from:
_RBAssSlice in if_python.o
_RBAppend in if_python.o
_RBAppend in if_python.o
"_win_setwidth", referenced from:
_WindowSetattr in if_python.o
"_init_symtable", referenced from:
__PyImport_Inittab in py_config.o
"_curwin", referenced from:
_curwin$non_lazy_ptr in if_python.o

-snip-

I'll look into it later on, but the compilation at least worked
fine...I just have to get the dynamic library to work (any ideas?).

Björn

[Danek Duvall]

On Tue, Feb 17, 2009 at 10:42:00PM +0100, björn wrote:

> The following bit of configure.in looked a bit strange, so I changed
> it. No idea what that special "if" for MACOSX does though.

Well, there's that comment:

dnl For Mac OSX 10.2 config.o is included in the Python library.

Can't speak to its relevance, though.

> gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall
> -Wno-unknown-pragmas -pipe -I. -Iproto -DMACOS_X_UNIX -no-cpp-precomp
> -I/Developer/Headers/FlatCarbon -g -O -D_FORTIFY_SOURCE=1
> -DDYNAMIC_PYTHON
> -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5
> -Kpic -o objects/if_python.o if_python.c

-Kpic is specific to the Sun compiler. I'm not sure what gcc did with
that, but it should probably be -fpic.

> gcc -o if_python.so objects/if_python.o objects/py_config.o
> Undefined symbols:
> "_initimp", referenced from:
> __PyImport_Inittab in py_config.o
> "_ml_get_buf", referenced from:
> _RBSlice in if_python.o
> _GetBufferLine in if_python.o
> "_u_savesub", referenced from:
> _SetBufferLine in if_python.o
> "_PyInt_Type", referenced from:
> _PyInt_Type$non_lazy_ptr in if_python.o
> "_ml_append", referenced from:
> _RBAssSlice in if_python.o
> _RBAppend in if_python.o
> _RBAppend in if_python.o
> "_win_setwidth", referenced from:
> _WindowSetattr in if_python.o
> "_init_symtable", referenced from:
> __PyImport_Inittab in py_config.o
> "_curwin", referenced from:
> _curwin$non_lazy_ptr in if_python.o

This is probably happenine because gcc isn't being told to create a shared
library. My patch used -G to pass to Studio cc, but gcc takes -shared.

Not sure if those two changes will be sufficient, but they're likely
necessary.

Danek

[björn]

2009/2/17 Danek Duvall:

Well, this is turning out to be quite complicated. After quite a bit
of searching I found that passing

-dynamic -undefined suppress -flat_namespace

when linking if_python.so gets rid of all those error messages. So
the relevant lines inside Makefile should say something like:

$(PYTHON_SO): $(PYTHON_SO_OBJ)
$(CClink) -dynamic -undefined suppress -flat_namespace
$(PYTHON_SO_LIBS) -o $@ $(PYTHON_SO_OBJ)

However, trying ":py print 'hi'" inside Vim results in a crash (SEGV).
I know that's not at all helpful but I just wanted to say that I am
trying to get this patch to work on OS X. To anybody else using OS X:
feel free to help me out here. At any rate, I'll give this another go
when I get a chance again (next weekend perhaps).

Björn

[Danek Duvall]

On Sun, Feb 22, 2009 at 07:04:08PM +0100, bj�rn wrote:

> Well, this is turning out to be quite complicated. After quite a bit
> of searching I found that passing
>
> -dynamic -undefined suppress -flat_namespace
>
> when linking if_python.so gets rid of all those error messages. So
> the relevant lines inside Makefile should say something like:

Hm. Try perhaps -dynamiclib instead of -dynamic? You could also see how
the Python dynamic modules are compiled, since they're likely loaded by the
Python interpreter the same way we're loading this module in vim.

Danek

[björn]

2009/2/22 Danek Duvall:

After lots of fiddling it turns out it may be as simple as swapping
-dynamic for -bundle (so your suggestion was good).

At any rate, I've got the simple ":py print 'hi'" test working now but
I made a few other changes along the way so I'm not sure that was the
only necessary change. Hold on...yes, if I swap back to -dynamic I
get "Bus Error" again so -bundle is what is needed. The other two
flags are necessary to avoid masses of linking errors.

I'm looking forward to a patch with dynamic loading of the most recent
Python version! I know that a lot of MacVim users will appreciate
this feature.

Björn