Re: [cython-users] Cython and linking to multiple DLLs

[Chris Barker]

On Wed, Oct 3, 2012 at 7:21 AM, Brad Buran <bbu...@gmail.com> wrote:
> I'm trying to write a Cython wrapper around a library for reading from a
> (proprietary) binary file format. The vendor has released versions of the
> library (called "MCStream" on Linux and "MC_StreamAnsiLib" on Windows)
> compiled for Linux and Windows. I've been able to successfully get the
> Cython wrapper working under Linux; however, the version of the vendor's
> library for Windows is linked against the boost C++ library. Hence, I need
> to compile the boost library on Windows for linking.

Wow! this is pretty painful for a binary-only proprietary library (I'm
assuming you do not get the source code so that you could re-compile
the lib yourself?)


> I was able to successfully do this and compile/link a test C++ file with the
> MCStream library and the boost libraries, but only when using Microsoft
> Visual Studio 10. Based on what I understand from various newsgroups, the
> boost library must be compiled/linked using the same version of MS Visual
> Studio that MCStream was compiled under.

that does seem likely but probaly not your issue anyway...

> However, I understand that Cython uses Visual Studio 9 (at least that's the
> default executable used when I run "python setup.py build_ext --inplace").

To be clear, it is not Cython that uses an particular version of a
compiler, it is distutils, which is designed to compile Python
extensions with the same compiler that python itself was built with.
The Python binaries distributed by python.org were built with
particular versions (usually a bit old) 2.7 was built with VS2008, for
instance (is that the same as 9? -- the dula naming conversion drives
me crazy!)


> Consequently, I get a lot of "unresolved external symbol" errors that are
> associated with the failure to properly link the boost libraries with the
> MCStream executable.

HMM -- if I have this right: MCStream uses Boost (or some of it,
anyway). But that should be independent of your Cython code -- it
should be calling the MCStream API directly (or are there data types,
etc, provided by Boost that you need to use to call MCStream?)

Anyway -- while distutils, by default, uses the compiler PYthon was
built with, as I understand nit, it isn't absolutely required that
that be the case -- the issues are less with the compiler than with
the the stdlib that is used by the compiler -- and only SOME of the
stdlib functions require compatibility (I think file handles is a
common issue, for instance) but if the two versions of the compiler
have a compatible ABI (which I"d guess they do?) then you *might* be
able to compile your extension with the newer compiler.

I'm not sure how to force distutils to use a different one, but I'd
start by compiling an extension by hand to test whether it will work.

You may also be abel to make a little layer: put the boost stuff in a
C++ layer that calls the dll, and have your Cyton code call this layer
(free of Boost types).

A final option is to re-compile Python itself with VS 9 -- then it'll
all the the same.

> below, actual error is at the bottom). As you can see, the error relates to
> it being unable to find the references defined in the MCStream library;

I don't know if minGW is ABI-compatible with MS VS for C++ (it does
work for C, but there is no standard ABI for C++)

HTH,
-Chris



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Brad Buran]

Wow! this is pretty painful for a binary-only proprietary library (I'm
assuming you do not get the source code so that you could re-compile
the lib yourself?)

They don't give us access to the source code unfortunately. 
 

To be clear, it is not Cython that uses an particular version of a
compiler, it is distutils, which is designed to compile Python
extensions with the same compiler that python itself was built with.
The Python binaries distributed by python.org were built with
particular versions (usually a bit old) 2.7 was built with VS2008, for
instance (is that the same as 9? -- the dula naming conversion drives
me crazy!)

Sorry, I should have been clear.  Yes, version 9 = VS2008.
 

HMM -- if I have this right: MCStream uses Boost (or some of it,
anyway). But that should be independent of your Cython code -- it
should be calling the MCStream API directly (or are there data types,
etc, provided by Boost that you need to use to call MCStream?)

Yes, MCStream uses boost.  As far as I can tell Cython code doesn't need to interact with Boost in any way or use the data types available from Boost.  However, in my setup.py file, I need to ensure that the boost libraries are on the library path otherwise I get errors about not being able to link against some of the Boost libraries.

Anyway -- while distutils, by default, uses the compiler PYthon was
built with, as I understand nit, it isn't absolutely required that
that be the case -- the issues are less with the compiler than with
the the stdlib that is used by the compiler -- and only SOME of the
stdlib functions require compatibility (I think file handles is a
common issue, for instance) but if the two versions of the compiler
have a compatible ABI (which I"d guess they do?) then you *might* be
able to compile your extension with the newer compiler.

Thanks for the suggestion!  This worked.  In the past I've always made sure I used VS2008 since that's what seems to work the best when installing modules, but I removed VS2008 and installed VS2010.  I needed to run the following extra steps to get python setup.py build_ext to work:

SET VS90COMNTOOLS=%VS100COMNTOOLS%
Add extra_compile_args=['/MANIFEST'] to the setup.py script

If anyone else is having similar problems, there's a good write-up on various problems encountered when compiling with VS2010 at http://magic-smoke.blogspot.com/2012/07/building-pyliblo-on-windows-using.html that may be helpful.

I'm not sure how to force distutils to use a different one, but I'd
start by compiling an extension by hand to test whether it will work.

Me either, but the trick I used was to uninstall VS2008 and run
SET VS90COMNTOOLS=%VS100COMNTOOLS%

Thanks for the help,
Brad

[Chris Barker]

On Thu, Oct 4, 2012 at 7:39 AM, Brad Buran <bbu...@gmail.com> wrote:

> Yes, MCStream uses boost. As far as I can tell Cython code doesn't need to
> interact with Boost in any way or use the data types available from Boost.
> However, in my setup.py file, I need to ensure that the boost libraries are
> on the library path otherwise I get errors about not being able to link
> against some of the Boost libraries.

hmm -- that's odd -- I"d think that linking would have happened when
MCStream was built (and/or at runtime) -- so I'm surprised you need
that. But I don't pretent to understand Windows linking.

> Thanks for the suggestion! This worked. In the past I've always made sure
> I used VS2008 since that's what seems to work the best when installing
> modules,

yup -- it is a safer bet.

> but I removed VS2008 and installed VS2010. I needed to run the
> following extra steps to get python setup.py build_ext to work:
>
> SET VS90COMNTOOLS=%VS100COMNTOOLS%
> Add extra_compile_args=['/MANIFEST'] to the setup.py script

Thanks for the tip.

I still think you may be able build your extension with VS2008, and
just link to MCStream, but if you've got something working, you've got
something working.

I wonder why they don't statically link the boost stuff in to theior
DLL? most of boost (that I've seen) is headers/templates-only anyway.

[Brad Buran]

>> Yes, MCStream uses boost. As far as I can tell Cython code doesn't need to
>> interact with Boost in any way or use the data types available from Boost.
>> However, in my setup.py file, I need to ensure that the boost libraries are
>> on the library path otherwise I get errors about not being able to link
>> against some of the Boost libraries.
>
> hmm -- that's odd -- I"d think that linking would have happened when
> MCStream was built (and/or at runtime) -- so I'm surprised you need
> that. But I don't pretent to understand Windows linking.

That's what I would have though too. They have a static and
dynamic-link version of their libraries but for some reason I still
needed the boost libraries. I don't understand that part.

> I still think you may be able build your extension with VS2008, and
> just link to MCStream, but if you've got something working, you've got
> something working.

My thoughts exactly :)

[Sturla Molden]

On 05.10.2012 14:52, Brad Buran wrote:

> That's what I would have though too. They have a static and
> dynamic-link version of their libraries but for some reason I still
> needed the boost libraries. I don't understand that part.

This is a curse of C++ templates.

With templates, the compiler must see the full implementation as well as
the definition. We cannot compile a libary separately and just include
the headers files (.h). Instead all templated code (i.e. the .h and the
.cpp files) are included together in a .hpp file. That is why C++
templated code is very slow to compile and always results in bloatware.
To get template code to compile and link correctly, you must have both
the .h and the .cpp files (i.e. the whole library source) in the
includes. Since Boost use templates, you get this issue. I.e. you must
include the sources of Boost as well as the headers. That is what the
.hpp includes do. You also have the same issue with the STL. It is
actually compiled from source every time you use it. It is not a library
you can just link.

If you want a clean separation of compilation and linkage, you should
use C or C++ without templates. That will prevent the kind of problems
that C++ templates generates (slow compilation, bloated binaries).

Thus, if you have 100 .cpp files that use a std::vector<double>, you get
100 copies of the std::vector<double> source code compiled into the
final binary. And for every other template specification you use, new
copies of std::Vector are compiled into the binary. What you finally end
up with is an overgrown dinosaur with multiple copies of the same code.

By the way: Cython's "fused types" is also a bloatware generator, but
not as bad as C++ templates. You must limit the amount of bloat that is
generated, and you don't get multiple copies of the same binary code
linked into the .pyd file.

Java and .NET generics are also designed not to produce the kind of
bloat that C++ templates do.

All in all, C++ templates have served as an example of horror of how not
to implement generics in every language that has followed.


Sturla

[Brad Buran]

Apologies for the delayed reply, but I just had a thought:

With templates, the compiler must see the full implementation as well as
the definition. We cannot compile a libary separately and just include
the headers files (.h). Instead all templated code (i.e. the .h and the
.cpp files) are included together in a .hpp file. That is why C++
templated code is very slow to compile and always results in bloatware.
To get template code to compile and link correctly, you must have both
the .h and the .cpp files (i.e. the whole library source) in the
includes. Since Boost use templates, you get this issue. I.e. you must
include the sources of Boost as well as the headers. That is what the
.hpp includes do. You also have the same issue with the STL. It is
actually compiled from source every time you use it. It is not a library
you can just link.

If I ask for the *.hpp includes from the vendor, would this be sufficient for me to link against the static version of the MCStream library without having to link against the Boost libraries as well?

Brad

[Sturla Molden]

On 14.11.2012 15:48, Brad Buran wrote:

> If I ask for the *.hpp includes from the vendor, would this be
> sufficient for me to link against the static version of the MCStream
> library without having to link against the Boost libraries as well?

You must have Boost installed, and even use the same C++ compiler,
standard library version, and Boost version as the vendor of the binary
"MCStream" library. Otherwise, you must build everything from source
with your own C++ compiler.

S.M.

[Brad Buran]

That's too bad.  Would be nice if it was easier.  Thanks for the advice!