refer to installed ortools source code to build my own package

[Sergey Gerasimov]

Hi folks,

I develop my own python package which includes an extension module (cython) that needs ortools C++ header files and compiled ortools library during build. Can you please advise how to handle this in setup.py of my package?

If ortools has some installation mode that provides C++ header files I can specify the corresponding path in Extension definition in my setup.py. But unfortunately, I see that "pip install ortools" installs only *.py files and compiled libraries. 

Sorry if this question is too general.

Thanks a lot for any advice!

Sergey.

[Mizux Seiha]

If I were you I would:
1. use CMake as buildsystem for your underlying C++ module since it is more or less integrated in the python setuptools ecosystem

2. thus in your CMakeLists.txt you can try to include or-tools as a dependency using FetchContent()..

[Sergey Gerasimov]

Thanks a lot, Mizux!

It is really cool way. Will dig into how to  combine setuptools + (cython + cmake). The last sum is most interesting :-)

What do you think: is it better to distribute my package in binary way (bdist)? Looks like git clone of ortools and further build will take a lot of time..

вторник, 18 января 2022 г. в 11:21:43 UTC+3, mizu...@gmail.com:

[Sergey Gerasimov]

Sorry.. One more question. How to correctly link ortools statically?

When I use FetchContent() and do:

add_library(mylib MODULE ${mylib})

target_link_libraries(mylib ortools::ortools)

I see that ortools is linked as dynamic .so.

вторник, 18 января 2022 г. в 11:21:43 UTC+3, mizu...@gmail.com:

If I were you I would:

[Mizux Seiha]

we use 
```cmake
add_library(${PROJECT_NAME} "")
```
src: https://github.com/google/or-tools/blob/2cb85b4eead4c38e1c54b48044f92087cf165bce/cmake/cpp.cmake#L6

So you should be able to control it using BUILD_SHARED_LIBS as usual
ref: https://cmake.org/cmake/help/latest/variable/BUILD_SHARED_LIBS.html

note: you should do it in a subdirectory to have this variable scoped
e.g. https://github.com/google/or-tools/blob/stable/cmake/dependencies/CMakeLists.txt#L36-L57

include from:
https://github.com/google/or-tools/blob/stable/CMakeLists.txt#L216

[Sergey Gerasimov]

Thanks a lot, Mizux!

I tried many ways to build my Python extension (cython), but still have problem that finally I have _opt.so (that corresponds to my extension) and I also have separate libortools.so, libortools.so.9 etc. provided by presence of FetcchContent() of ortools. During importing of my extension "import opt" I get "ImportError: libortools.so.9: cannot open shared object file: No such file or directory.

What I want to do is to build my extension _opt.so statically linked with libortools (2nd way is to provide both .so files _opt.so and libortools.so to final users of my extension but I didn't find how to do this except of setting of LD_LIBRARY_PATH which looks not promising )

My final CMakeLists.txt looks as follow. If you have some quick hint in mind I will appreciate if you can share it.

In the code below I use macros from https://github.com/scikit-build/scikit-build like add_cython_target and python_extension_module. Scikit-build helps me to combine setup.py +cython+cmake.

Playing with BUILD_SHARED_LIBS (you can see it in comments did not help).

include(FetchContent)
FetchContent_Declare(
or-tools
GIT_REPOSITORY https://github.com/google/or-tools.git
GIT_TAG master
)
FetchContent_MakeAvailable(or-tools)

set(BUILD_DEPS ON)
set(BUILD_SAMPLES OFF)
set(BUILD_EXAMPLES OFF)
set(BUILD_Protobuf ON)
#set(BUILD_SHARED_LIBS OFF)

# provide ortools C/C++ header files to my extension
include_directories(./ ${or-tools_SOURCE_DIR} ${absl_SOURCE_DIR})

add_cython_target(_opt CXX)
add_library(_opt MODULE ${_opt})
target_link_libraries(_opt ortools::ortools)
python_extension_module(_opt)

install(TARGETS _opt LIBRARY DESTINATION opt)

There is also something like 3rd possible way.

I can specify ortools pypi package as a dependency, so during installation of my package the e library _opt.so will be build and installed (so I will use only c/c++ header files from ortools, do not know ho achieve this in cmake with FetchContent() of ortools since I may need not only clone ortools source code but maybe do some extra-steps like generation of protobuf headers...) and I can somehow guess to _opt.so that libortools.so is located in the installation directory of ortools pypi package.  

I think this way is also good, it needs to preserve equality of version of ortools used in FetchContent() and version of ortools pypi package.

But: I do not know how to guess to my _opt.so that libortools.so is located in  .libs subfolder (as I remember) of the ortools installation folder.

Another cons - I think ortools pypi package has no SCIP on board (?) and I need SCIP.

пятница, 21 января 2022 г. в 11:36:07 UTC+3, mizu...@gmail.com:

[Mizux Seiha]

few points:

0. I deeply apology, I should have point you to https://github.com/or-tools/cmake_or-tools/tree/main/FetchContent
warning: this repo need to be updated since it is a little bit rusty but  could have been helpful

1. FetchContent_MakeAvailable(or-tools) should be call AFTER the all the set()

2. when using `target_link_libraries(_opt PRIVATE ortools::ortools)` you should have access to include directories otherwise there is a bug in or-tools
2bis: also you should use target_include_directories
see: https://github.com/google/or-tools/blob/2cb85b4eead4c38e1c54b48044f92087cf165bce/cmake/cpp.cmake#L88-L92

3.I don't see a easy way to generate protobuf header without compiling ortools, need to investigate

EDIT: maybe by using `ortools_proto` target ? (aka use an `add_dependencies(_ ortools_proto)`
https://github.com/google/or-tools/blob/2cb85b4eead4c38e1c54b48044f92087cf165bce/cmake/cpp.cmake#L170-L211

ref: https://cmake.org/cmake/help/latest/command/add_dependencies.html

4. since python interpreter load all module in the same process
maybe using  `import ortools` before loading your module should load all symbols needed by your module.
note: you can see the `import ortools` as a way to LD_PRELOAD libortools in the python process ;)

[Sergey Gerasimov]

Thanks a bunch Mizux for all advice!

All of them helped me a lot.

I become a big fan of cmake :-)

Now the only actual problem is "ImportError: libortools.so.9: cannot open shared object file: No such file or directory" during import of my module ("import opt").

I see that .so file of my module now stores rpath (thanks to your examples and to "set_target_properties(_opt PROPERTIES

INSTALL_RPATH "\$ORIGIN/../lib"") and I see the link to libortools.so:

ldd ~/miniconda3/envs/pyopt/lib/python3.9/site-packages/sdo-0.1.0-py3.9-linux-x86_64.egg/opt/_opt.cpython-39-x86_64-linux-gnu.so | grep ortools
        libortools.so.9 => /home/sergun/miniconda3/envs/pyopt/lib/python3.9/site-packages/sdo-0.1.0-py3.9-linux-x86_64.egg/opt/../lib/libortools.so.9 (0x00007f00f5b5d000)

But "import opt" works only if I execute python with LD_LIBRARY_PATH explicitely passed:

LD_LIBRARY_PATH=/home/sergun/miniconda3/envs/pyopt/lib/python3.9/site-packages/sdo-0.1.0-py3.9-linux-x86_64.egg/opt/../lib python

Maybe you also have some good solution for this problem?

суббота, 22 января 2022 г. в 12:30:31 UTC+3, mizu...@gmail.com: