I'm trying to write a Cython wrapper around a library for reading from a (proprietary) binary file format. The vendor has released versions of the library (called "MCStream" on Linux and "MC_StreamAnsiLib" on Windows) compiled for Linux and Windows. I've been able to successfully get the Cython wrapper working under Linux; however, the version of the vendor's library for Windows is linked against the boost C++ library. Hence, I need to compile the boost library on Windows for linking.
I was able to successfully do this and compile/link a test C++ file with the MCStream library and the boost libraries, but only when using Microsoft Visual Studio 10. Based on what I understand from various newsgroups, the boost library must be compiled/linked using the same version of MS Visual Studio that MCStream was compiled under.
However, I understand that Cython uses Visual Studio 9 (at least that's the default executable used when I run "python setup.py build_ext --inplace"). Consequently, I get a lot of "unresolved external symbol" errors that are associated with the failure to properly link the boost libraries with the MCStream executable. My understanding is that Cython needs to use Visual Studio 9 because that's the version the Python is compiled with.
Is there any way to resolve this conflict? I tried compiling using mingw32 instead to see if this resolved the issue. I passed the flag --compiler=mingw32 to "python setup.py build_ext --inplace"; however, this produced a different set of errors which I do not understand (full output is below, actual error is at the bottom). As you can see, the error relates to it being unable to find the references defined in the MCStream library; however, both the -lMC_StreamAnsiLib and -L<path to library> flags are correct. I'm a bit stumped as to what to try next to get this to compile. Any pointers would be greatly appreciated.
Thanks, Brad
C:\Users\bburan\projects\MCStream\python>python setup.py build_ext --inplace --c ompiler=mingw32 running build_ext skipping 'test.cpp' Cython extension (up-to-date) building 'test' extension C:\MinGW32-xy\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Users\bburan\projects\ MCStream\include -IC:\Python27\lib\site-packages\numpy\core\include -IC:\Python2 7\include -IC:\Python27\PC -c test.cpp -o build\temp.win32-2.7\Release\test.o In file included from test.cpp:278:0: C:\Users\bburan\projects\MCStream\include/typedefs.h:80:0: warning: "RGB" redefi ned c:\mingw32-xy\bin\../lib/gcc/mingw32/4.5.2/../../../../include/wingdi.h:268 8:0: note: this is the location of the previous definition C:\Python27\lib\site-packages\numpy\core\include/numpy/__multiarray_api.h:1 532:1 : warning: 'int _import_array()' defined but not used C:\Python27\lib\site-packages\numpy\core\include/numpy/__ufunc_api.h:226:1: warn ing: 'int _import_umath()' defined but not used C:\Python27\include/datetime.h:188:25: warning: 'PyDateTimeAPI' defined but not used test.cpp:1506:12: warning: 'int __pyx_pw_5numpy_7ndarray_1__getbuffer__(PyObject *, Py_buffer*, int)' defined but not used test.cpp:2322:13: warning: 'void __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyO bject*, Py_buffer*)' defined but not used writing build\temp.win32-2.7\Release\test.def C:\MinGW32-xy\bin\g++.exe -mno-cygwin -shared -s build\temp.win32-2.7\Release\te st.o build\temp.win32-2.7\Release\test.def -LC:\Users\bburan\projects\MCStream\l ib\win32 -LC:\Python27\libs -LC:\Python27\PCbuild -lMC_StreamAnsiLib -lpython27 -lmsvcr90 -o C:\Users\bburan\projects\MCStream\python\test.pyd build\temp.win32-2.7\Release\test.o:test.cpp:(.text+0x395): undefined reference to `CMCSAStreamFile::CMCSAStreamFile()' build\temp.win32-2.7\Release\test.o:test.cpp:(.text+0xb39): undefined reference to `CMCSAStreamFile::OpenFile(char const*)' collect2: ld returned 1 exit status error: command 'g++' failed with exit status 1
On Wed, Oct 3, 2012 at 7:21 AM, Brad Buran <bbu...@gmail.com> wrote:
> I'm trying to write a Cython wrapper around a library for reading from a
> (proprietary) binary file format. The vendor has released versions of the
> library (called "MCStream" on Linux and "MC_StreamAnsiLib" on Windows)
> compiled for Linux and Windows. I've been able to successfully get the
> Cython wrapper working under Linux; however, the version of the vendor's
> library for Windows is linked against the boost C++ library. Hence, I need
> to compile the boost library on Windows for linking.
Wow! this is pretty painful for a binary-only proprietary library (I'm
assuming you do not get the source code so that you could re-compile
the lib yourself?)
> I was able to successfully do this and compile/link a test C++ file with the
> MCStream library and the boost libraries, but only when using Microsoft
> Visual Studio 10. Based on what I understand from various newsgroups, the
> boost library must be compiled/linked using the same version of MS Visual
> Studio that MCStream was compiled under.
that does seem likely but probaly not your issue anyway...
> However, I understand that Cython uses Visual Studio 9 (at least that's the
> default executable used when I run "python setup.py build_ext --inplace").
To be clear, it is not Cython that uses an particular version of a
compiler, it is distutils, which is designed to compile Python
extensions with the same compiler that python itself was built with.
The Python binaries distributed by python.org were built with
particular versions (usually a bit old) 2.7 was built with VS2008, for
instance (is that the same as 9? -- the dula naming conversion drives
me crazy!)
> Consequently, I get a lot of "unresolved external symbol" errors that are
> associated with the failure to properly link the boost libraries with the
> MCStream executable.
HMM -- if I have this right: MCStream uses Boost (or some of it,
anyway). But that should be independent of your Cython code -- it
should be calling the MCStream API directly (or are there data types,
etc, provided by Boost that you need to use to call MCStream?)
Anyway -- while distutils, by default, uses the compiler PYthon was
built with, as I understand nit, it isn't absolutely required that
that be the case -- the issues are less with the compiler than with
the the stdlib that is used by the compiler -- and only SOME of the
stdlib functions require compatibility (I think file handles is a
common issue, for instance) but if the two versions of the compiler
have a compatible ABI (which I"d guess they do?) then you *might* be
able to compile your extension with the newer compiler.
I'm not sure how to force distutils to use a different one, but I'd
start by compiling an extension by hand to test whether it will work.
You may also be abel to make a little layer: put the boost stuff in a
C++ layer that calls the dll, and have your Cyton code call this layer
(free of Boost types).
A final option is to re-compile Python itself with VS 9 -- then it'll
all the the same.
> below, actual error is at the bottom). As you can see, the error relates to
> it being unable to find the references defined in the MCStream library;
I don't know if minGW is ABI-compatible with MS VS for C++ (it does
work for C, but there is no standard ABI for C++)
HTH,
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
> Wow! this is pretty painful for a binary-only proprietary library (I'm > assuming you do not get the source code so that you could re-compile > the lib yourself?)
They don't give us access to the source code unfortunately.
> To be clear, it is not Cython that uses an particular version of a > compiler, it is distutils, which is designed to compile Python > extensions with the same compiler that python itself was built with. > The Python binaries distributed by python.org were built with > particular versions (usually a bit old) 2.7 was built with VS2008, for > instance (is that the same as 9? -- the dula naming conversion drives > me crazy!)
Sorry, I should have been clear. Yes, version 9 = VS2008.
> HMM -- if I have this right: MCStream uses Boost (or some of it, > anyway). But that should be independent of your Cython code -- it > should be calling the MCStream API directly (or are there data types, > etc, provided by Boost that you need to use to call MCStream?)
Yes, MCStream uses boost. As far as I can tell Cython code doesn't need to interact with Boost in any way or use the data types available from Boost. However, in my setup.py file, I need to ensure that the boost libraries are on the library path otherwise I get errors about not being able to link against some of the Boost libraries.
Anyway -- while distutils, by default, uses the compiler PYthon was
> built with, as I understand nit, it isn't absolutely required that > that be the case -- the issues are less with the compiler than with > the the stdlib that is used by the compiler -- and only SOME of the > stdlib functions require compatibility (I think file handles is a > common issue, for instance) but if the two versions of the compiler > have a compatible ABI (which I"d guess they do?) then you *might* be > able to compile your extension with the newer compiler.
Thanks for the suggestion! This worked. In the past I've always made sure I used VS2008 since that's what seems to work the best when installing modules, but I removed VS2008 and installed VS2010. I needed to run the following extra steps to get python setup.py build_ext to work:
SET VS90COMNTOOLS=%VS100COMNTOOLS% Add extra_compile_args=['/MANIFEST'] to the setup.py script
On Thu, Oct 4, 2012 at 7:39 AM, Brad Buran <bbu...@gmail.com> wrote:
> Yes, MCStream uses boost. As far as I can tell Cython code doesn't need to
> interact with Boost in any way or use the data types available from Boost.
> However, in my setup.py file, I need to ensure that the boost libraries are
> on the library path otherwise I get errors about not being able to link
> against some of the Boost libraries.
hmm -- that's odd -- I"d think that linking would have happened when
MCStream was built (and/or at runtime) -- so I'm surprised you need
that. But I don't pretent to understand Windows linking.
> Thanks for the suggestion! This worked. In the past I've always made sure
> I used VS2008 since that's what seems to work the best when installing
> modules,
yup -- it is a safer bet.
> but I removed VS2008 and installed VS2010. I needed to run the
> following extra steps to get python setup.py build_ext to work:
> SET VS90COMNTOOLS=%VS100COMNTOOLS%
> Add extra_compile_args=['/MANIFEST'] to the setup.py script
Thanks for the tip.
I still think you may be able build your extension with VS2008, and
just link to MCStream, but if you've got something working, you've got
something working.
I wonder why they don't statically link the boost stuff in to theior
DLL? most of boost (that I've seen) is headers/templates-only anyway.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
>> Yes, MCStream uses boost. As far as I can tell Cython code doesn't need to
>> interact with Boost in any way or use the data types available from Boost.
>> However, in my setup.py file, I need to ensure that the boost libraries are
>> on the library path otherwise I get errors about not being able to link
>> against some of the Boost libraries.
> hmm -- that's odd -- I"d think that linking would have happened when
> MCStream was built (and/or at runtime) -- so I'm surprised you need
> that. But I don't pretent to understand Windows linking.
That's what I would have though too. They have a static and
dynamic-link version of their libraries but for some reason I still
needed the boost libraries. I don't understand that part.
> I still think you may be able build your extension with VS2008, and
> just link to MCStream, but if you've got something working, you've got
> something working.
> That's what I would have though too. They have a static and
> dynamic-link version of their libraries but for some reason I still
> needed the boost libraries. I don't understand that part.
This is a curse of C++ templates.
With templates, the compiler must see the full implementation as well as the definition. We cannot compile a libary separately and just include the headers files (.h). Instead all templated code (i.e. the .h and the .cpp files) are included together in a .hpp file. That is why C++ templated code is very slow to compile and always results in bloatware. To get template code to compile and link correctly, you must have both the .h and the .cpp files (i.e. the whole library source) in the includes. Since Boost use templates, you get this issue. I.e. you must include the sources of Boost as well as the headers. That is what the .hpp includes do. You also have the same issue with the STL. It is actually compiled from source every time you use it. It is not a library you can just link.
If you want a clean separation of compilation and linkage, you should use C or C++ without templates. That will prevent the kind of problems that C++ templates generates (slow compilation, bloated binaries).
Thus, if you have 100 .cpp files that use a std::vector<double>, you get 100 copies of the std::vector<double> source code compiled into the final binary. And for every other template specification you use, new copies of std::Vector are compiled into the binary. What you finally end up with is an overgrown dinosaur with multiple copies of the same code.
By the way: Cython's "fused types" is also a bloatware generator, but not as bad as C++ templates. You must limit the amount of bloat that is generated, and you don't get multiple copies of the same binary code linked into the .pyd file.
Java and .NET generics are also designed not to produce the kind of bloat that C++ templates do.
All in all, C++ templates have served as an example of horror of how not to implement generics in every language that has followed.
Apologies for the delayed reply, but I just had a thought:
With templates, the compiler must see the full implementation as well as
> the definition. We cannot compile a libary separately and just include > the headers files (.h). Instead all templated code (i.e. the .h and the > .cpp files) are included together in a .hpp file. That is why C++ > templated code is very slow to compile and always results in bloatware. > To get template code to compile and link correctly, you must have both > the .h and the .cpp files (i.e. the whole library source) in the > includes. Since Boost use templates, you get this issue. I.e. you must > include the sources of Boost as well as the headers. That is what the > .hpp includes do. You also have the same issue with the STL. It is > actually compiled from source every time you use it. It is not a library > you can just link.
If I ask for the *.hpp includes from the vendor, would this be sufficient for me to link against the static version of the MCStream library without having to link against the Boost libraries as well?
> If I ask for the *.hpp includes from the vendor, would this be
> sufficient for me to link against the static version of the MCStream
> library without having to link against the Boost libraries as well?
You must have Boost installed, and even use the same C++ compiler, standard library version, and Boost version as the vendor of the binary "MCStream" library. Otherwise, you must build everything from source with your own C++ compiler.
> If I ask for the *.hpp includes from the vendor, would this be
>> sufficient for me to link against the static version of the MCStream
>> library without having to link against the Boost libraries as well?
> You must have Boost installed, and even use the same C++ compiler,
> standard library version, and Boost version as the vendor of the binary
> "MCStream" library. Otherwise, you must build everything from source with
> your own C++ compiler.
