Cannot build PyImath library on Windows

[Lana Melnichuk]

I am working on building PyAlembic on Windows. I am having problems building and running the imath module from PyIlmBase. PyIex builds fine and so I can import iex module and use it. No such luck with PyImath. With Boost 1.4x, I can build PyImath but then it will crash when I try to import the imath module. With Boost 1.5x, PyImath won't compile at all.

I've been looking around the internet to see if someone else has ran into the same problem and found this. I've seen this post on other forums but there is no followup on any of them.

Has anyone successfully build a working win64 binary of PyImath?

[Piotr Stanczyk]

PyImath won't build with boost 1.50, we're hoping to get round to it
pretty soon, but if anyone else wants to jump in ....


Piotr

> --
>
>

[Lana Melnichuk]

I am hoping to use it with Boost 1.42 but imath module crashes on import (win64 PyImath.dll renamed to imath.pyd and used as ">>>import imath"). Any idea what may be causing the crash?

[Barnaby Robson]

Crashes like this on Linux are usually due to library mismatches.  One thing to make sure of is that the libraries used to build the dll are the same libraries that are being loaded at run time.  Have you run depends.exe on the library ?

barnaby.

On 09/21/2012 10:15 AM, Lana Melnichuk wrote:

I am hoping to use it with Boost 1.42 but imath module crashes on import (win64 PyImath.dll renamed to imath.pyd and used as ">>>import imath"). Any idea what may be causing the crash?

--
 
 

[Lucas Miller]

I had terrible luck getting the bindings to build with boost python 1.42 on Linux as well.

It worked MUCH better with boost python 1.46.

Lucas

On Fri, Sep 21, 2012 at 10:15 AM, Lana Melnichuk <melnichuk...@gmail.com> wrote:

I am hoping to use it with Boost 1.42 but imath module crashes on import (win64 PyImath.dll renamed to imath.pyd and used as ">>>import imath"). Any idea what may be causing the crash?

--
 
 

[Ben Houston]

Just a quick note that we will be distributing binary builds of the python bindings for linux and windows in our distribution soon. May be a convenience for many. This should happen next week. 

Sent from my iPhone

--
 
 

[Barnaby Robson]

Ben, out of interest which boost version are you building pyimath with on windows ?

Thanks

barnaby.

--
 
 

[Ben Houston]

Barnaby: I haven't yet built them. I've covered our code base to use
Alembic 1.1 this week and the Python bindings are on my schedule for
next week. Initially I'll probably
try Boost 1.44 initially and switch if necessary. Given that we
statically link in boost, it should be relatively straightforward to
mix and match boost versions across the various Alembic tools -- just
whatever works. I'll let you know once I do figure it out on our end.
-ben

> --
>
>



--
Best regards,
Ben Houston
CTO, Exocortex Technologies, Inc.
http://www.exocortex.com

[Lana Melnichuk]

VS9 debug stack trace:

>    imath.pyd!PyIex::TypeTranslator<Iex::BaseExc>::firstClassDesc()  Line 281 + 0x5 bytes    C++
     imath.pyd!PyIex::registerExc<Imath::NullVecExc,Iex::MathExc>(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & name="NullVecExc", const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & module="imath")  Line 240 + 0xd bytes    C++
     imath.pyd!init_module_imath()  Line 458 + 0x41 bytes    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::detail::function::void_function_ref_invoker0<void (__cdecl*)(void),void>::invoke(boost::detail::function::function_buffer & function_obj_ptr={...})  Line 189    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::function0<void>::operator()()  Line 1014    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::exception_handler::operator()(const boost::function0<void> & f={...})  Line 75    C++
     iex.pyd!00000001800ccd2b()    
     [Frames below may be incorrect and/or missing, no symbols loaded for iex.pyd]   
     iex.pyd!00000001800cb95a()    
     iex.pyd!00000001800c7ea7()    
     iex.pyd!00000001800c08e1()    
     boost_python-vc90-mt-gd-1_42.dll!boost::function2<bool,boost::python::detail::exception_handler const & __ptr64,boost::function0<void> const & __ptr64>::operator()(const boost::python::detail::exception_handler & a0={...}, const boost::function0<void> & a1={...})  Line 1014    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::exception_handler::handle(const boost::function0<void> & f={...})  Line 42    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::handle_exception_impl(boost::function0<void> * f=0x000000000021f5e0)  Line 24 + 0x11 bytes    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::handle_exception<void (__cdecl*)(void)>(void (void)* f=0x00000000026b30bc)  Line 30    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::init_module(const char * name=0x0000000003e6b40c, void (void)* init_function=0x00000000026b30bc)  Line 53 + 0xa bytes    C++
     imath.pyd!initimath()  Line 164 + 0x2e bytes    C++

[Lana Melnichuk]

Sorry, forgot to rebuild PyIex in debug. Here's what I get now (with python26_d.lib as well):


>    boost_python-vc90-mt-gd-1_42.dll!boost::python::api::setattr(const boost::python::api::object & target={...}, const char * key=0x0000000003e6c414, const boost::python::api::object & value={...})  Line 64 + 0x28 bytes    C++
     imath.pyd!boost::python::api::attribute_policies::set(const boost::python::api::object & target={...}, const char * key=0x0000000003e6c414, const boost::python::api::object & value={...})  Line 88    C++
     imath.pyd!boost::python::api::proxy<boost::python::api::attribute_policies>::operator=<boost::python::handle<_object> >(const boost::python::handle<_object> & rhs={...})  Line 33 + 0x37 bytes    C++
     imath.pyd!init_module_imath()  Line 169 + 0x6b bytes    C++

     boost_python-vc90-mt-gd-1_42.dll!boost::detail::function::void_function_ref_invoker0<void (__cdecl*)(void),void>::invoke(boost::detail::function::function_buffer & function_obj_ptr={...})  Line 189    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::function0<void>::operator()()  Line 1014    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::exception_handler::operator()(const boost::function0<void> & f={...})  Line 75    C++

     iex.pyd!boost::python::detail::translate_exception<Iex::BaseExc,void (__cdecl*)(Iex::BaseExc const & __ptr64)>::operator()(const boost::python::detail::exception_handler & handler={...}, const boost::function0<void> & f={...}, void (const Iex::BaseExc &)* const translate=0x000000018000cd60)  Line 48 + 0x10 bytes    C++
     iex.pyd!boost::_bi::list3<boost::arg<1>,boost::arg<2>,boost::_bi::value<void (__cdecl*)(Iex::BaseExc const & __ptr64)> >::operator()<bool,boost::python::detail::translate_exception<Iex::BaseExc,void (__cdecl*)(Iex::BaseExc const & __ptr64)>,boost::_bi::list2<boost::python::detail::exception_handler const & __ptr64,boost::function0<void> const & __ptr64> >(boost::_bi::type<bool> __formal={...}, boost::python::detail::translate_exception<Iex::BaseExc,void (__cdecl*)(Iex::BaseExc const &)> & f={...}, boost::_bi::list2<boost::python::detail::exception_handler const &,boost::function0<void> const &> & a={...}, boost::_bi::type<bool> __formal={...})  Line 383    C++
     iex.pyd!boost::_bi::bind_t<bool,boost::python::detail::translate_exception<Iex::BaseExc,void (__cdecl*)(Iex::BaseExc const & __ptr64)>,boost::_bi::list3<boost::arg<1>,boost::arg<2>,boost::_bi::value<void (__cdecl*)(Iex::BaseExc const & __ptr64)> > >::operator()<boost::python::detail::exception_handler,boost::function0<void> >(const boost::python::detail::exception_handler & a1={...}, const boost::function0<void> & a2={...})  Line 103    C++
     iex.pyd!boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool,boost::python::detail::translate_exception<Iex::BaseExc,void (__cdecl*)(Iex::BaseExc const & __ptr64)>,boost::_bi::list3<boost::arg<1>,boost::arg<2>,boost::_bi::value<void (__cdecl*)(Iex::BaseExc const & __ptr64)> > >,bool,boost::python::detail::exception_handler const & __ptr64,boost::function0<void> const & __ptr64>::invoke(boost::detail::function::function_buffer & function_obj_ptr={...}, const boost::python::detail::exception_handler & a0={...}, const boost::function0<void> & a1={...})  Line 133    C++

     boost_python-vc90-mt-gd-1_42.dll!boost::function2<bool,boost::python::detail::exception_handler const & __ptr64,boost::function0<void> const & __ptr64>::operator()(const boost::python::detail::exception_handler & a0={...}, const boost::function0<void> & a1={...})  Line 1014    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::exception_handler::handle(const boost::function0<void> & f={...})  Line 42    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::handle_exception_impl(boost::function0<void> * f=0x000000000021f5e0)  Line 24 + 0x11 bytes    C++
     boost_python-vc90-mt-gd-1_42.dll!boost::python::handle_exception<void (__cdecl*)(void)>(void (void)* f=0x00000000026b30bc)  Line 30    C++

     boost_python-vc90-mt-gd-1_42.dll!boost::python::detail::init_module(const char * name=0x0000000003e6c40c, void (void)* init_function=0x00000000026b30bc)  Line 53 + 0xa bytes    C++

[Barnaby Robson]

Hi Lana,

Could you let us know which actual versions of boost and python you are using ?  Also that stack trace looks fairly innocent .. I'm talking with Piotr and we're not sure what the actual error is, can you describe what you believe it tells us ?

barnaby.

--
 
 

[Barnaby Robson]

Ok .. don't worry about the version questions .. it says in the stack trace boost 1.42 and python 2.6 you mentioned. Still some more details on what you think is going on would be good. Thanks !

barnaby.

--
 
 

[Lana Melnichuk]

Hi Barnaby! Thanks for taking a look at my problem.

I am using boost 1.42 (boost_python-vc90-mt-1_42.dll), ilmbase 1.0.3, pyilmbase 1.0.0, python 2.6 and Visual Studio 9. I am able to build static/shared and release/debug versions of all necessary libraries. I can build and use PyIex.dll (iex.pyd). I can build but not use PyImath.dll (imath.pyd) due to a runtime crash.

I believe the problem is with PyImath/imathmodule.cpp. The stack errors suggest that there is something going on in BOOST_PYTHON_MODULE(imath), lines 166-169:

166    handle<> iex(PyImport_ImportModule("iex"));
167    if (PyErr_Occurred()) boost::python::throw_error_already_set();
168
169    scope().attr("iex") = iex;


I took the three lines above and inserted them into a simple program based on boost.python extending.cpp:

#include <boost/python.hpp>

// original code
char const* greet()
{
    return "hello, world";
}

BOOST_PYTHON_MODULE(hello_ext)
{
    // original code
    using namespace boost::python;

    // test code
    handle<> iex(PyImport_ImportModule("iex"));
    if (PyErr_Occurred()) boost::python::throw_error_already_set();
    scope().attr("iex") = iex;

    // original code
    def("greet", greet);
}

Importing the hello_ext module now give an error "No module named iex" which is what I am expecting. Adding iex.pyd to the folder solves that problem. Okay, so this tells me that those three lines are not a problem on their own. There are references to Iex in PyImath/imathmodule.cpp, lines 458 - 462:

458    PyIex::registerExc<Imath::NullVecExc,Iex::MathExc>("NullVecExc","imath");
459    PyIex::registerExc<Imath::NullQuatExc,Iex::MathExc>("NullQuatExc","imath");
460    PyIex::registerExc<Imath::SingMatrixExc,Iex::MathExc>("SingMatrixExc","imath");
461    PyIex::registerExc<Imath::ZeroScaleExc,Iex::MathExc>("ZeroScaleExc","imath");
462    PyIex::registerExc<Imath::IntVecNormalizeExc,Iex::MathExc>("IntVecNormalizeExc","imath");

Adding those to my short example (in addition to lines 166-169) results in a runtime crash on import. Perhaps lines 458-462 are the cause of the crash?

[Lana Melnichuk]

Basically, I believe the problem is with getting access and/or data contained in iex.pyd from inside imath.pyd at runtime.

[Lana Melnichuk]

Hi Barnaby!
Unfortunately, it crashes silently. This is my work flow:

C:\myprojectdirectory>python
Python 2.6.1 (r26:67517, Dec  4 2008, 16:59:09) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import imath

Then it crashes, and a Visual Studio Just-In-Time Debugger window pops up asking me to launch a new instance of Visual Studio 2008. When I do, the error I get is "Unhandled exception ... in python.exe... Access violation reading location...". When I break, it takes me to object_protocol.cpp line 64:

60   BOOST_PYTHON_DECL void setattr(object const& target, char const* key, object const& value)
61   {
62       if (PyObject_SetAttrString(
63               target.ptr(), const_cast<char*>(key), value.ptr()) == -1
64           )
65       {
66           throw_error_already_set();
67       }
68   }

In the mean while, nothing else is printed to the console. If there's something different he'd have me do, please let me know. I could also send you my iex.pyd and imath.pyd modules (and any relevant dependencies) if you think that could help diagnose the problem.

[Shih-Chin]

Lana Melnichuk於 2012年9月25日星期二UTC+8上午6時16分14秒寫道：

Basically, I believe the problem is with getting access and/or data contained in iex.pyd from inside imath.pyd at runtime.

 Hi Lana, 

I've run into the similar issues before. At first, the libs I built are PyIex.lib, iex.pyd, PyImath.lib and imath.pyd.

But I got the run-time crash when it executed the following code:

PyIex::registerExc<IMATH_NAMESPACE::NullVecExc,IEX_NAMESPACE::MathExc>("NullVecExc","imath");
PyIex::registerExc<IMATH_NAMESPACE::NullQuatExc,IEX_NAMESPACE::MathExc>("NullQuatExc","imath");
PyIex::registerExc<IMATH_NAMESPACE::SingMatrixExc,IEX_NAMESPACE::MathExc>("SingMatrixExc","imath");
PyIex::registerExc<IMATH_NAMESPACE::ZeroScaleExc,IEX_NAMESPACE::MathExc>("ZeroScaleExc","imath");
PyIex::registerExc<IMATH_NAMESPACE::IntVecNormalizeExc,IEX_NAMESPACE::MathExc>("IntVecNormalizeExc","imath");

When I traced the error in debug mode, I found that the value of _baseExcTranslator from PyIex.cpp 

wasn't initialized properly (it's still a null pointer), and this the reason of crash. 

Actually, the _baseExcTranslator would have duplicate instances in iex.pyd and imath.pyd.

The workaround I found is to compile PyIex as a shared library:

http://stackoverflow.com/questions/3186926/shared-global-variable-in-c-static-library

Then, I deploy PyIex.dll, iex.pyd and imath.pyd to C:/PythonXX/DLLs, and both imath and iex could be successfully imported in my case.

The attached files are the CMake scripts I used, Hope it might help. :)

Unfortunately, this is how far I can get so far. When I compile PyAlembic with those libs, it's ok to compile and link. But I get TypeError when I import alembic.pyd:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No Python class registered for C++ class class PyImath::FixedArray<bool>

Currently, I don't have any clue about how to solve this problem...

My PC configuration is:

Boost: 1.48

OpenEXR/Ilmbase: the latest commit from github (https://github.com/openexr/openexr)

HDF5: 1.8.9

Zlib: 1.2.7

All dependent libs are built as static libs and use multi-threading c-runtime-library (i.e. /MD)

Shih-Chin

[Mark]

I am having the same problem with the null pointer value from _baseExcTranslator in PyIex.cpp... and the ensuing segfault

[Mark]

I had to patch imathmodule.cpp to fix the import. 

diff  /work/proddev/tpc/python/temp/pyilmbase-1.0.0-v1.7/PyImath/imathmodule.cpp /work/proddev/tpc/python/temp/pyilmbase-1.0.0.patched-v1.7/PyImath/imathmodule.cpp

458,462c458,462
<     PyIex::registerExc<Imath::NullVecExc,Iex::MathExc>("NullVecExc","imath");
<     PyIex::registerExc<Imath::NullQuatExc,Iex::MathExc>("NullQuatExc","imath");
<     PyIex::registerExc<Imath::SingMatrixExc,Iex::MathExc>("SingMatrixExc","imath");
<     PyIex::registerExc<Imath::ZeroScaleExc,Iex::MathExc>("ZeroScaleExc","imath");
<     PyIex::registerExc<Imath::IntVecNormalizeExc,Iex::MathExc>("IntVecNormalizeExc","imath");
---
>     PyIex::registerExc<Imath::NullVecExc,Iex::BaseExc>("NullVecExc","iex");
>     PyIex::registerExc<Imath::NullQuatExc,Iex::BaseExc>("NullQuatExc","iex");
>     PyIex::registerExc<Imath::SingMatrixExc,Iex::BaseExc>("SingMatrixExc","iex");
>     PyIex::registerExc<Imath::ZeroScaleExc,Iex::BaseExc>("ZeroScaleExc","iex");
>     PyIex::registerExc<Imath::IntVecNormalizeExc,Iex::BaseExc>("IntVecNormalizeExc","iex");

Not sure what exactly's broken/missing, but this allowed me to import all the modules without segfault (Could use some input from pyImath devs).