[vim/vim] Build from source linking to Anaconda python fails (#2549)

[Ben Lindsay]

I'm trying to build from source so I can link to a different python interpreter. I'm on MacOS Sierra 10.12.1. I can build just fine with all the options I want except for python integration. So if I install with this script it works:

start_dir=$(pwd)
downloads_dir=$HOME/Downloads
python_dir=$HOME/miniconda
vim_install_dir=$HOME/.local/vim

mkdir -p $downloads_dir
cd $downloads_dir
git clone https://github.com/vim/vim.git
cd vim

export LD_LIBRARY_PATH=$python_dir/lib:$LD_LIBRARY_PATH \
  export PATH=$python_dir/bin:$PATH
  ./configure \
  --prefix=$vim_install_dir \
  --with-features=huge \
  --enable-multibyte \
  --enable-cscope \
  --enable-gui=no \
  # --includedir=$python_dir/include/python3.6m \
  # --enable-python3interp \
  # --with-python3-config-dir=$python_dir/lib/python3.6/config-3.6m-darwin \

make && make install

cd $start_dir

but if I delete all the built and downloaded files, uncomment the python-related lines, and run it again, it fails. Here's a file with the output of the command above with the 3 lines uncommented: install.log

A couple other things to verify that the directories I'm using are correct:

$ find ~/miniconda -name config.c
/Users/benlindsay/miniconda/envs/django/lib/python3.6/config-3.6m-darwin/config.c
/Users/benlindsay/miniconda/lib/python3.6/config-3.6m-darwin/config.c
/Users/benlindsay/miniconda/pkgs/python-3.5.4-he720263_23/lib/python3.5/config-3.5m/config.c
/Users/benlindsay/miniconda/pkgs/python-3.6.3-h47c878a_7/lib/python3.6/config-3.6m-darwin/config.c
/Users/benlindsay/miniconda/pkgs/python-3.6.3-h6804ab2_0/lib/python3.6/config-3.6m-darwin/config.c

$ find ~/miniconda/include -type d -name '*python*'                                                                    
/Users/benlindsay/miniconda/include/python3.6m

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub

[Kazunobu Kuriyama]

but if I delete all the built and downloaded files, uncomment the python-related lines, and run it again, it fails.

From this remark, I cannot be sure of what you did or how it failed specifically, but guessing from the shell script you showed, I'm afraid neither the removal nor the uncommenting might be the cause of the issue.

It seems to me that, at least, in order to run an executable which was built with a script like the one you gave, you also need to specify PATH and LD_LIBRARY_PATH at runtime in accordance with those given at build time. So, say, you need something like

$ LD_LIBRARY_PATH=$python_dir/lib:$LD_LIBRARY_PATH; PATH=$python_dir/bin:$PATH; vim

instead of simple $ vim. (Probably you did that already, so just to make sure...)

[Ben Lindsay]

Thanks for the fast response! Sorry, I wasn't very clear about the specific problem I'm seeing. The error I'm seeing when I try to build with python support is a compile-time error. The last couple lines look like this:

error: Unknown attribute kind (52) (Producer: 'LLVM4.0.1' Reader: 'LLVM APPLE_1_800.0.42.1_0')
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [vim] Error 1
make: *** [first] Error 2

The rest of the make output is in the file I attached in my previous comment.

Thanks for the note that the LD_LIBRARY_PATH and PATH need to be right at runtime as well as compile time! I didn't realize that was the case, but I'm not to that problem just yet until I can actually compile with python support :)

To clarify, when I said I delete all the built and downloaded files, I meant that after successfully building without python support just to make sure that works, I deleted the vim git repo and the binaries I had just built so that I was starting fresh for an attempt to build with python support. That's when I get a compile-time error and no binaries are built.

Any other advice you have would be greatly appreciated

[Kazunobu Kuriyama]

Thanks for the clarification.

Then, as suggested in the error message, how about adding LDFLAGS='-v' or CFLAGS='-Wl,-v' to the command-line when you run configure in order to see what the linker is complaining about? For instance,

export LDFLAGS='-v' \
export LD_LIBRARY_PATH=$python_dir/lib:$LD_LIBRARY_PATH \
export PATH=$python_dir/bin:$PATH
  ./configure \
<snip>

Though I'm not 100% sure if that works well for you, but if it does, then the resulting verbose output could guide you in fixing the python-related configuration parameters.

[Ben Lindsay]

Hmm, I just tried both the LDFLAGS and CFLAGS approaches and it doesn't seem to be giving me any more information...

[Kazunobu Kuriyama]

Seems we need another approach to the issue...

In the install.log file, the following line seems to be unusual to me:

checking Python3's dll name... libpython3.6m.a

I'm wondering why the name looks like an archive or static library rather than a name for a shared one (It should be libpython3.6m.dylib or something like that in our case).

So, would you check, after spotting the directory in which the .a file is (Probably, in /Users/benlindsay/miniconda/lib/python3.6/config-3.6m-darwin/), what file(1) says about the file? (Or, do $ otool -l /your/path/to/libpython3.6m.a | grep -A1 filetype and check what number is given to the filetype.) Also, please check if there's a .dylib file corresponding to the .a file.

If you have the .a file there, if file(1) says that it is actually a dynamically linked shared library, and if there's no corresponding .dylib file, then doing the following before invoking make(1) might make a difference:

$ cd /Users/benlindsay/miniconda/lib/python3.6/config-3.6m-darwin/
$ ln -s libpython3.6m.a libpython3.6m.dylib

Although creating a symbolic link that way may look strange, that's what is seen in Python on Mac when it is installed as an OS X framework; both libpython*.a and libpython*.dylib are symbolic links to a file named Python which file(1) says isMach-O 64-bit dynamically linked shared library x86_64.

Anyway, all you need to do is to give correct arguments of the -L and -l options which are suitable for specifying your target Python. Naturally, it's not easy to get configure to cover all possible non-standard configurations and installations without knowing specific use cases in advance. So, when you find it the case, please consider specifying those options manually, not relying on configure's automatic detection; LDFLAGS given from command-line is meant for that.

[Ben Lindsay]

I'm not sure what you mean by file(1), but if I just use file I get this:

benlindsay@macbook:miniconda/ $ file ./lib/python3.6/config-3.6m-darwin/libpython3.6m.a
./lib/python3.6/config-3.6m-darwin/libpython3.6m.a: current ar archive random library

Using the otool command (without piping to grep) gives this:

benlindsay@macbook:miniconda/ $ otool -l ./lib/python3.6/config-3.6m-darwin/libpython3.6m.a
Archive : ./lib/python3.6/config-3.6m-darwin/libpython3.6m.a

With grep it outputs nothing.

Looking for all possible libpython3.6m.a or libpython3.6m.dylib files shows this:

benlindsay@macbook:miniconda/ $ find . \( -name libpython3.6m.a -o -name libpython3.6m.dylib \)
./envs/django/lib/libpython3.6m.a
./envs/django/lib/libpython3.6m.dylib
./envs/django/lib/python3.6/config-3.6m-darwin/libpython3.6m.a
./lib/libpython3.6m.a
./lib/libpython3.6m.dylib
./lib/python3.6/config-3.6m-darwin/libpython3.6m.a
./pkgs/python-3.6.3-h47c878a_7/lib/libpython3.6m.a
./pkgs/python-3.6.3-h47c878a_7/lib/libpython3.6m.dylib
./pkgs/python-3.6.3-h47c878a_7/lib/python3.6/config-3.6m-darwin/libpython3.6m.a
./pkgs/python-3.6.3-h6804ab2_0/lib/libpython3.6m.a
./pkgs/python-3.6.3-h6804ab2_0/lib/libpython3.6m.dylib
./pkgs/python-3.6.3-h6804ab2_0/lib/python3.6/config-3.6m-darwin/libpython3.6m.a

So I guess another candidate is ~/miniconda/lib/libpython3.6m.dylib. Running the otool + grep command on that gives:

benlindsay@macbook:miniconda/ $ otool -l ./lib/libpython3.6m.dylib | grep -A1 filetype
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           6    17       2288 0x00100085

Honestly I have no idea what this means though. Do you think that .dylib file is what vim is supposed to see instead?

[Ben Lindsay]

For good measure I tried dynamic linking you suggested, which resulted in failing in a different way (...progress?). The message came at the same place, but it says

dyld: Library not loaded: @rpath/libpython3.6m.dylib
  Referenced from: /Users/benlindsay/.local/vim/bin/vim
  Reason: image not found
make[2]: *** [vimtags] Abort trap: 6
make[1]: [installrtbase] Error 2 (ignored)
ls: tags: No such file or directory

Here's the full output log if you're interested: post_symlink_vim_install.log

Also, not sure if this is expected, but the libpython3.6m.a file is also symlinked to a file of that name 2 directories above it. i.e.

benlindsay@macbook:~/ $ cd miniconda/lib
benlindsay@macbook:lib/ $ ls -l **/libpython3.6m.{a,dylib}
-rwxr-xr-x  2 benlindsay  staff  5382520 Oct  6 13:09 libpython3.6m.a
-rwxr-xr-x  1 benlindsay  staff  3032740 Dec  7 22:09 libpython3.6m.dylib
lrwxr-xr-x  1 benlindsay  staff       21 Dec  7 22:09 python3.6/config-3.6m-darwin/libpython3.6m.a -> ../../libpython3.6m.a
lrwxr-xr-x  1 benlindsay  staff       15 Jan 14 11:48 python3.6/config-3.6m-darwin/libpython3.6m.dylib -> libpython3.6m.a

I also tried symlinking the libpython3.6m.dylib file directly to the one 2 directories up as well, but that resulted in failing the same way as without any symlinks.

[Kazunobu Kuriyama]

I'm not sure what you mean by file(1)

That's a widely used notational convention which means that you can view the on-line manual of file of the topic section 1 with $ man 1 file.

Do you think that .dylib file is what vim is supposed to see instead?

On Mac, usually, yes.

For good measure I tried dynamic linking you suggested,

Sorry but I didn't suggest anything that way. I suggested creating a symlink if the three conditions I enumerated in my previous comment are met; otherwise, all bets are off.

dyld: Library not loaded: .6m.dylib

(Assuming libpython3.6m.dylib is a symlink of libpython3.6m.a that you made with your own decision) Contrary to what its suffix suggests, if libpython3.6m.a was a dynamically linked shared library as is seen with the pre-installed Python on Mac, that error might not happen. That's why I carefully enumerated the conditions "If you have the .a file there, if file(1) says that it is actually a dynamically linked shared library, and if there's no corresponding .dylib file, then..." to avoid something like that.

With your Python 3.6 installation, what you got with file(1) or otool(1) suggests libpython3.6m.a is literally a static library; however, that's not an issue per se. I think the true problem is being suggested by the list you showed:

<snip>
./lib/libpython3.6m.a
./lib/libpython3.6m.dylib
./lib/python3.6/config-3.6m-darwin/libpython3.6m.a
<snip>

You noticed that ./lib/libpython3.6m.a is a symlink to ./lib/python3.6/config-3.6m-darwin/libpython3.6m.a. Then, it is quite natural to guess that ./lib/libpython3.6m.dylib is also a symlink to something. But there's no libpython3.6m.dylib in ./lib/python3.6/config-3.6m-darwin/ in the list. Is that a typo or did anything go wrong with the installation? I'm wondering what happened to real libpython3.6m.dylib.

[Kazunobu Kuriyama]

@benlindsay

Since the given information was too vague for me to handle the issue properly, I installed Python 3.6.3|Anaconda by myself on my macOS 10.11.6 through the Miniconda installer to look into the issue further.

Having checked how that particular version of Python was installed, I was able to build Vim linked against its libpython3.6m.dylib successfully. The procedure was as follows:

1. Install Python 3.6.3|Anaconda through the installer mentioned above. (As to the settings, I accepted the defaults since there was no reason for me to do otherwise)

2. Create a symbolic link as follows:

$ cd ~/miniconda3/lib/python3.6/config-3.6m-darwin
$ ln -s ../../libpython3.6m.dylib .

(This procedure was needed to avoid conflicts or linker errors between some system libraries and those installed with the Python, e.g., ncurses in particular).

3. Go to vim/src, and run ./configure --enable-python3interp there.

4. After configure is done, apply the following patch to auto/config.mk:

diff --git a/auto/config.mk.orig b/auto/config.mk
index 927e3690b..4ff3104b3 100644
--- a/auto/config.mk.orig
+++ b/auto/config.mk
@@ -71,7 +71,7 @@ PYTHON_GETPATH_CFLAGS =
 PYTHON3_SRC    = if_python3.c
 PYTHON3_OBJ    = objects/if_python3.o
 PYTHON3_CFLAGS = -I/Users/kazu/miniconda3/include/python3.6m -DPYTHON3_HOME='L"/Users/kazu/miniconda3"' -fPIE
-PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
+PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -rpath /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
 PYTHON3_CONFDIR        = /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin

 TCL            =

(The -rpath option was needed to add since the libpython3.6m.dylib was configure and built to use it.)

5. make test to build and test Vim.

6 ./vim, then check if the if_python3 works or not. For example, do

:py3 import sys
:py3 print(sys.version_info)

to see if Vim is linked against the target Python.

[Ben Lindsay]

Closed #2549.

[Ben Lindsay]

Wow, thank you so much for putting that much time into my specific edge case! Following this process worked great. I just added the other flags in the ./configure step to get it to install to the correct local directory and include the set of features I want, and things are working like a charm.

Thanks again!

[Socheng]

Wow, thank you so much for putting that much time into my specific edge case! Following this process worked great. I just added the other flags in the ./configure step to get it to install to the correct local directory and include the set of features I want, and things are working like a charm.

Thanks again!

@benlindsay

Since the given information was too vague for me to handle the issue properly, I installed Python 3.6.3|Anaconda by myself on my macOS 10.11.6 through the Miniconda installer to look into the issue further.

Having checked how that particular version of Python was installed, I was able to build Vim linked against its libpython3.6m.dylib successfully. The procedure was as follows:

1. Install Python 3.6.3|Anaconda through the installer mentioned above. (As to the settings, I accepted the defaults since there was no reason for me to do otherwise)
2. Create a symbolic link as follows:

$ cd ~/miniconda3/lib/python3.6/config-3.6m-darwin
$ ln -s ../../libpython3.6m.dylib .

(This procedure was needed to avoid conflicts or linker errors between some system libraries and those installed with the Python, e.g., ncurses in particular).

1. Go to vim/src, and run ./configure --enable-python3interp there.
2. After configure is done, apply the following patch to auto/config.mk:

diff --git a/auto/config.mk.orig b/auto/config.mk
index 927e3690b..4ff3104b3 100644
--- a/auto/config.mk.orig
+++ b/auto/config.mk
@@ -71,7 +71,7 @@ PYTHON_GETPATH_CFLAGS =
 PYTHON3_SRC    = if_python3.c
 PYTHON3_OBJ    = objects/if_python3.o
 PYTHON3_CFLAGS = -I/Users/kazu/miniconda3/include/python3.6m -DPYTHON3_HOME='L"/Users/kazu/miniconda3"' -fPIE
-PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
+PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -rpath /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
 PYTHON3_CONFDIR        = /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin

 TCL            =

(The -rpath option was needed to add since the libpython3.6m.dylib was configured and built to use it. And, of course, replace the account name kazu with yours.)

1. make test to build and test Vim.

1. ./vim, then check if the if_python3 works or not. For example, do

:py3 import sys
:py3 print(sys.version_info)

to see if Vim is linked against the target Python.

UPDATE: Fixed a minor typo and the list format.

Hello mate!

I solved my problem using your solution perfectly.

The thing is, according to your suggestion, it can solve the case only when the library is static, not dynamically loaded; i.e. I cannot have python and python3 both enabled.

Again, you helped me a lot. Just dunno if you can answer my question.

Regards,

Kenichi

[richard-moss]

Hey Nuko - just wanted to add my thanks as well ...solved it compiling for me as well

[Dreamingo]

PYTHON3_SRC    = if_python3.c
 PYTHON3_OBJ    = objects/if_python3.o
 PYTHON3_CFLAGS = -I/Users/kazu/miniconda3/include/python3.6m -DPYTHON3_HOME='L"/Users/kazu/miniconda3"' -fPIE
-PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
+PYTHON3_LIBS   = -L/Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -rpath /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
 PYTHON3_CONFDIR        = /Users/kazu/miniconda3/lib/python3.6/config-3.6m-darwin

Hi, could you provide a version for Linux users? it seems -rpath -framework is not suitable for gcc?? Thanks.

—
You are receiving this because you are subscribed to this thread.

Reply to this email directly, view it on GitHub, or unsubscribe.