mingwpy installed via anaconda produces a Win64 DLL that crashes in the initialization function

[vital...@gmail.com]

I am experiencing this problem when building https://github.com/numenta/nupic.core (commit 4e9d0904dce71883dddd93bd27b15a5b19513487) via mingwpy installed via `pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy` on Windows Server 2012 R2 64-bit. It happens on both AppVeyor as well as on my own windows instance from AWS/EC2.

I recently switched the mingwpy toolchain that we use to build https://github.com/numenta/nupic.core for Win64 (the failure is demonstrated at nupic.core commit 4e9d0904dce71883dddd93bd27b15a5b19513487). Previously, we installed mingwpy via libpython-cp27-none-win_amd64.7z and mingwpy_amd64_vc90.7z from https://bitbucket.org/carlkl/mingw-w64-for-python/downloads. This used to work and still does.

However, I was made to understand that the preferred mingwpy installation is via `pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy`.  On the Win64 box, this presently downloads and installs https://pypi.anaconda.org/carlkl/simple/mingwpy/0.1.0b3/mingwpy-0.1.0b3-cp27-none-win_amd64.whl (82.4MB). The DLL in question is an extension DLL for nupic.bindings.math. Why I try to import nupic.bindings.math from python 2.7 shell (Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32), it fails with the exception:

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed

Windows Application Log shows access violation exception 0xc0000005 at offset 0x00000000001e2917:

Log Name:      Application

Source:        Application Error

Date:          4/12/2016 6:45:18 PM

Event ID:      1000

Task Category: (100)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      WIN-PJ2KGO4VBDS

Description:

Faulting application name: python.exe, version: 0.0.0.0, time stamp: 0x5488acd7

Faulting module name: _math.pyd, version: 0.0.0.0, time stamp: 0x570c5794

Exception code: 0xc0000005

Fault offset: 0x00000000001e2917

Faulting process id: 0xf74

Faulting application start time: 0x01d194eb75144b17

Faulting application path: C:\Python27\python.exe

Faulting module path: c:\users\administrator\nta\nupic.core\bindings\py\nupic\bindings\_math.pyd

Report Id: b32c20d0-00de-11e6-8123-020a3cfbcbf7

Faulting package full name: 

Faulting package-relative application ID:

The odd fault offset 0x00000000001e2917 looks particularly strange. Once again, the 64-bit toolchain installed from https://bitbucket.org/carlkl/mingw-w64-for-python/downloads doesn't exhibit this problem.

To reproduce the failure:

Set up toolchain and source code

1. Get a Windows Server 2012 R2 64-bit machine. On AWS, I used Windows_Server-2012-R2_RTM-English-64Bit-Base-2016.03.09 (ami-1719f677)
2. Install git
   
3. Make unix-compatible patch.exe available to the build. I hacked it together from git\usr\bin as follows (couldn't add git\usr\bin to PATH because sh.exe there upsets the toolchain)
   
1. mkdir "C:\Program Files\PatchFromGit"
2. copy "C:\Program Files\Git\usr\bin\patch.exe" "C:\Program Files\PatchFromGit"
3. copy "C:\Program Files\Git\usr\bin\msys*.dll" "C:\Program Files\PatchFromGit"
4. Add C:\Program Files\PatchFromGit to the PATH environment variable
4. Install CMAKE: https://cmake.org/ (Latest Release 3.5.1 msi)
5. Install Python 2.7.9 via Windows x86-64 MSI installer from https://www.python.org/downloads/release/python-279/
   
6. Add python and pip executable directories to PATH env var (C:\Python27\;C:\Python27\Scripts on my machine)
   
7. Upgrade pip and setuptools: python -m pip install -U pip setuptools
8. Install wheel: pip install wheel
9. Install mingwpy toolchain per instructions in https://anaconda.org/carlkl/mingwpy
   
1. pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
10. Clone https://github.com/numenta/nupic.core at commit 4e9d0904dce71883dddd93bd27b15a5b19513487
11. Set NUPIC_CORE env var to the absolute path of the cloned nupic.core directory (e.g., C:\Users\Administrator\nta\nupic.core)
    
12. Under nupic.core, create directories build/scripts
13. Setup MinGW-w64 GCC as a valid distutils compiler
    
1. copy nupic.core\external\windows64-gcc\bin\distutils.cfg C:\Python27\Lib\distutils\distutils.cfg
   
14. Install NumPy and C++ headers per https://anaconda.org/mingwpy/numpy
    
1. pip install -i https://pypi.anaconda.org/mingwpy/simple numpy

Build:

1. cd %NUPIC_CORE%\build\scripts
   
2. cmake -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=%NUPIC_CORE%\build\release -DPY_EXTENSIONS_DIR=%NUPIC_CORE%\bindings\py\nupic\bindings %NUPIC_CORE%
   
3. cmake.exe --build "%NUPIC_CORE%\build\scripts" --target install --config Release
   

Install:

1. cd %NUPIC_CORE%
2. python setup.py install

Reproduce the Access Violation:

1. python -c "import nupic.bindings.math"
   
1. This prints the simple exception message: ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed
2. Windows Application Log shows the access violation exception 0xc0000005 with more details.
   

[vitaly.kru...@gmail.com]

Here is the actual trackeback from `python -c "import nupic.bindings.math"`

C:\Users\Administrator\nta\nupic.core>python -c "import nupic.bindings.math"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "C:\Python27\lib\site-packages\nupic.bindings-0.4.2.dev0-py2.7-win-amd64.egg\nupic\bindings\math.py", line 28, in

 <module>

    _math = swig_import_helper()

  File "C:\Python27\lib\site-packages\nupic.bindings-0.4.2.dev0-py2.7-win-amd64.egg\nupic\bindings\math.py", line 24, in

 swig_import_helper

    _mod = imp.load_module('_math', fp, pathname, description)

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

And the corresponding Windows Application Log item:

Log Name:      Application

Source:        Application Error

Date:          4/13/2016 3:49:14 PM

Event ID:      1000

Task Category: (100)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      WIN-PJ2KGO4VBDS

Description:

Faulting application name: python.exe, version: 0.0.0.0, time stamp: 0x5488acd7

Faulting module name: _math.pyd, version: 0.0.0.0, time stamp: 0x570da6ef

Exception code: 0xc0000005

Fault offset: 0x00000000001e2917

Faulting process id: 0xa30

Faulting application start time: 0x01d1959c07b0ea69

Faulting application path: C:\Python27\python.exe

Faulting module path: C:\Python27\lib\site-packages\nupic.bindings-0.4.2.dev0-py2.7-win-amd64.egg\nupic\bindings\_math.pyd

Report Id: 4577b084-018f-11e6-8123-020a3cfbcbf7

Faulting package full name: 

Faulting package-relative application ID: 

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Application Error" />

    <EventID Qualifiers="0">1000</EventID>

    <Level>2</Level>

    <Task>100</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2016-04-13T15:49:14.000000000Z" />

    <EventRecordID>7070</EventRecordID>

    <Channel>Application</Channel>

    <Computer>WIN-PJ2KGO4VBDS</Computer>

    <Security />

  </System>

  <EventData>

    <Data>python.exe</Data>

    <Data>0.0.0.0</Data>

    <Data>5488acd7</Data>

    <Data>_math.pyd</Data>

    <Data>0.0.0.0</Data>

    <Data>570da6ef</Data>

    <Data>c0000005</Data>

    <Data>00000000001e2917</Data>

    <Data>a30</Data>

    <Data>01d1959c07b0ea69</Data>

    <Data>C:\Python27\python.exe</Data>

    <Data>C:\Python27\lib\site-packages\nupic.bindings-0.4.2.dev0-py2.7-win-amd64.egg\nupic\bindings\_math.pyd</Data>

    <Data>4577b084-018f-11e6-8123-020a3cfbcbf7</Data>

    <Data>

    </Data>

    <Data>

    </Data>

  </EventData>

</Event>

[carlkl]

To my best guess this error is due to the fact, that nupic may need long double with so called extended precision. long doubles in mingwpy have been mapped to double precision for MSVC and Windows API compatibility. This is not the case for the older so-called mingw-static archives on bitbucket.

To verify this statement you can remove the occurences of: -mlong-double-64 in the sections: cc1_options and cc1plus in the specs file of the mingwpy compiler toolchain and test again with mingwpy. The next version of mingwpy should have the possibility to configure this behaviour properly during compile and link time.

Carl

[vitaly.kru...@gmail.com]

Hi Carl, many thanks for your insight, you were spot-on! I can confirm that after removing the occurrences of -mlong-double-64 in the sections cc1_options and cc1plus of the specs file, the DLL-loading problem described in this topic disappeared.

What are the next steps? I would prefer to switch nupic.core to the new toolchain rather than continuing to use the old one from bitbucket.

Best regards,

Vitaly

[carlkl]

A new version of mingwpy based on gcc-5.3.0 is in preparation and should not to far away.

Do you have a special need to use long doubles with extended precision for nupic?

Carl

[vitaly.kru...@gmail.com]

I will have my ear to the ground for the new version of mingwpy.

Regarding 

Do you have a special need to use long doubles with extended precision for nupic?

 I will check with the numenta algorithms team about it and get back to you here.

Best,

Vitaly

[Nathaniel Smith]

Ideally, you should modify nupic.core so that it works correctly when sizeof(long double) == sizeof(double) (i.e. the largest available float is float64; there is no extended precision available). This is how the standard windows abi works (as defined by msvc among others), and mingwpy is moving to match it for compatibility.

(The problem is that even if you only use long double inside your own code, for this to work right mingwpy has to set some global extended precision settings in the fpu, and in programs that mix mingwpy and msvc code then their disagreement about what to do with this global state is problematic.)

-n

--
You received this message because you are subscribed to the Google Groups "mingwpy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mingwpy+u...@googlegroups.com.
To post to this group, send email to min...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mingwpy/fd14ab93-192b-4ef6-bb3b-c064d9146e7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vitaly.kru...@gmail.com]

Hi Nathaniel, thank you for the recommendation. Would you mind providing or pointing me to an example of how to accomplish this?

Best,

Vitaly

[Nathaniel Smith]

It depends entirely on how your code is using long double. Somewhere in your codebase the string "long double" appears -- that's the code that you need to fix :-)

--
You received this message because you are subscribed to the Google Groups "mingwpy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mingwpy+u...@googlegroups.com.
To post to this group, send email to min...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mingwpy/0e21890e-f669-4e18-b1ee-ab5b4ca09cf6%40googlegroups.com.

[vitaly.kru...@gmail.com]

Nathaniel, I thought that you were hinting at something deeper than that. I suspect that the conflict in nupic.core might derive from the use of `long double` in swig interface definition file src/nupic/bindings/numpy.i, but could be wrong.

[carlkl]

The swig files defines long doubles according to the definition found in _numpyconfig.h. For the used numpy binary it is:

..
#define NPY_SIZEOF_DOUBLE 8
#define NPY_SIZEOF_COMPLEX_DOUBLE 16
#define NPY_SIZEOF_LONGDOUBLE 8
#define NPY_SIZEOF_COMPLEX_LONGDOUBLE 16
..
#define NPY_HAVE_COMPLEX_LONG_DOUBLE 1
..

As numpy defines long double == double I assume the problem is  how boost (part of nupic.core) is compiled. According to http://info.prelert.com/blog/the-pitfalls-of-long-double it is possible to compile boost without support for long double at all: use BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS.

I have no understanding of nupic src code, but with a quick llok it seems, that nupic dosn't use long doubles at all.

Carl

[Olivier Grisel]

If BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS if confirmed to fix the
issue reported by Vitaly, we should add it to the documentation of
mingwpy, for instance to the FAQ.

--
Olivier

[vital...@gmail.com]

Regarding 

Do you have a special need to use long doubles with extended precision for nupic?

I followed up with the algorithms team, and they confirmed that there is no need in nupic.core for long doubles of any kind. `long double` was removed from nupic.core, and I will be trying a build with BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS as suggested by Carl. Thx.

[vital...@gmail.com]

Clarification: "long double" was removed from nupic.core proper, but not from boost embedded in nupic.core, so hopefully the flag BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS will have the desired effect.

[vital...@gmail.com]

Gentlemen, the suggestion to use BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS worked, and I now have a successful nupic.core build using the anaconda-based mingwpy 0.1.0b3: https://github.com/NumentaCorp/grok-projects/pull/250.

Many thanks!

Vitaly

[Carl Kleffner]

Great to hear,

Carl

--

You received this message because you are subscribed to the Google Groups "mingwpy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mingwpy+u...@googlegroups.com.
To post to this group, send email to min...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mingwpy/e13c01a1-30ac-41ec-9328-b900a3ae7ca7%40googlegroups.com.