Problems building Numpy with OpenBLAS for Windows x64(aka: x86_64 and amd64)

[Mark Mikofski]

A. Is there any other way to build Numpy with OpenBLAS on windows x64 other than using Carl Kleffner's static gcc toolchain?

B. Has anyone other than Carl Kleffner successfully built numpy on Windows x64 using openblas, sdk7 (vc90) and mingw-w64 gcc_seh-4.9.1 (from mingw-build)?

C. Has anyone successfuly built numpy on Windows x64 using openblas, sdk7(vc90) and clang(llvm)?

Any help would be appreciated.

Thanks!

--Mark

PS: Sorry if this thread is redundant, but I have searched and found only the following support for building Numpy with OpenBLAS on windows x64:

Carl Kleffner has created a static toolchain for mingw-w64 that links with mscvr90.dll and static version of gcc.lib and gfortran.lib.

Here is what I have tried:

1. build numpy using OpenBLAS binaries from sf.net

    (a) copy import library "OpenBLAS\lib\libopenblas.dll.a" -> "OpenBLAS\bin\openblas.lib"

    (b) add "OpenBLAS\bin" to PATH environmental variable

    (c) add "mingw64\bin" to PATH

    (d) edit "numpy\site.cfg" to read:

         [openblas]

         library_dirs=c:\path\to\OpenBLAS\bin

    (e) run python setup.py config --compiler=msvc --fcompiler=gnu95 build --compiler=msvc --fcompiler=gnu95 install

    (f) run python and import numpy returns following traceback

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\__init__.py", line 170, in <module>

    from . import add_newdocs

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>

    from numpy.lib import add_newdoc

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\lib\__init__.py", line 18, in <module>

    from .polynomial import *

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\lib\polynomial.py", line 19, in <module>

    from numpy.linalg import eigvals, lstsq, inv

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\linalg\__init__.py", line 51, in <module>

    from .linalg import *

  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\linalg\linalg.py", line 29, in <module>

    from numpy.linalg import lapack_lite, _umath_linalg

ImportError: DLL load failed: The specified procedure could not be found.

2. repeat using clean build of OpenBLAS

    (a) clone openblas master v0.2.12

    (b) make

    (c) make install

    (d) repeat steps from [1] above

[Mark Mikofski]

I forgot to add that I ran dependency walker on the lapack_lite.pyd and umath_linalg.pyd and they return the following errors:

Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module.

Error: Modules with different CPU types were found.

Warning: At least one delay-load dependency module was not found.

Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.

libopenblas.dll doesn't have matching exports for some imports, and msvcr90.dll from an x86 module (intel opencl sdk)  is picked up , but I don't understand these results, still learning

[Mark Mikofski]

Okay, maybe this is due to ABI imcompatibilties, but the issue might be that for some totally odd and crazy reason, according to dependency walker, lapack_lite.pyd is looking in libopenblas.dll for functons that are normally found in python27.dll, functions such as PyArg_ParseTuple, PyDict_SetItemString.This function is supposed to call the libopenblas functions dgelsd, dgegrf, dorgqr, zgelsd, zgegrf, zungqr and zerbla. Maybe this is an error with dependency walker. There is at least some scrambling of implicit imports in every compiled module.

On Monday, October 20, 2014 12:32:46 PM UTC-7, Mark Mikofski wrote:

[Zhang Xianyi]

Hi Mark,

I will give a try about numpy and OpenBLAS on Windows x64 this week.

Xianyi

--
You received this message because you are subscribed to the Google Groups "OpenBLAS-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openblas-user...@googlegroups.com.
To post to this group, send email to openbla...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Mikofski]

Thanks Xianyi. I was using gcc-4.9.1 (seh error handling, with winpthreads) from mingw-builds, windows sdk7(msvc90), Python-2.7.8 and Numpy-1.9.0 - not sure if that was the problem.

There are several threads on the Numpy-Discussion mailist in reference to this topic

* 64-bit windows numpy / scipy wheels for testing (http://mail.scipy.org/pipermail/numpy-discussion/2014-April/070028.html)

* Default builds of OpenBLAS development branch are now fork safe (http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069827.html)

* The BLAS problem (was: Re: Wiki page for building numerical stuff on Windows) (http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069868.html)

* Wiki page for building numerical stuff on Windows (http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069860.html)

[ckl]

Hi Mark,

feel free to use https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/2014-08-28_OpenBLAS-0.2.12dev_PR440.7z or compile OpenBLAS-0.12 yourself. You may use https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingw64static-2014-07.7z or https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingw32static-2014-07.7z to ensure to get the gcc runtime code statically linked and thus avoid the need to deploy libgcc_s_seh-1.dll and libgfortran-3.dll.
https://bitbucket.org/carlkl/mingw-w64-for-python/downloads has also some patches for numpy as well as numpy/scipy wheels based on OpenBLAS.

Regards,

Carl

[Mark Mikofski]

Thanks Carl, and thanks for your contribution. I looked through your static toolchain and patches, and I think it is a huge contribution. I have some concerns though. I am worried that a static toolchain is hard to maintain, and that it is an impediment to lay users who want a general purpose binary build system that can deal with all extensions, not just numpy. I like that it uses static libraries, but I would prefer to see OpenBLAS community generate and distribute static libraries for linking, that could then be used by any compatible compiler (eg: msvc90 for py27 on win7 x64) to build numpy. Can your OpenBLAS static libraries be used with MSVC? If so then maybe your static toolchain could be incorporated into the OpenBLAS project and used to generate x86_64 windows static libraries for distribution with their current binaries (EG: contain both dynamic and static libraries). Also is your numpy.patch required to build numpy with the OpenBLAS static library? What is the status of the patch - will it be pulled into numpy? Thanks again!

[ckl]

The static toolchain on my bitbucket site is specially adapted to work around problems with the usage of mingw-w64 based python extensions together with the official MSVC build Python distributions. This toolchain can be used as well for building the libopenblas.dll with the gcc-runtimes linked into the libopenblas.dll. Other static mingw-w64 based distributions can be used as well, i.e. TDM's distribution http://tdm-gcc.tdragon.net or gcc from equation.com.

IMPORTANT HINT: Don't ever mix import files from different mingw-w64 compiler flavours. If you need an import file for the DLL please build it yourself with your own compiler:

1. gendef libopenblas.dll
2. dlltool -D libopenblas.dll -d libopenblas.def -l libopenblas.dll.a

or use the MSVC tools if you want to use Visual C; see https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio.

If you want to use precompiled static import libraries (libopenblas.a) it is best to use the very same compiler version (gcc major version; appropriate choosen thread model). TDM's mingw-w64 version AFAIK is known to be ABI incompatible with other mingw-w64 toolchains.

Cheers

Carl

Am Donnerstag, 23. Oktober 2014 21:02:33 UTC+2 schrieb Mark Mikofski:

Thanks Carl, and thanks for your contribution. I looked through your static toolchain and patches, and I think it is a huge contribution. I have some concerns though. I am worried that a static toolchain is hard to maintain, and that it is an impediment to lay users who want a general purpose binary build system that can deal with all extensions, not just numpy. I like that it uses static libraries, but I would prefer to see OpenBLAS community generate and distribute static libraries for linking, that could then be used by any compatible compiler (eg: msvc90 for py27 on win7 x64) to build numpy. Can your OpenBLAS static libraries be used with MSVC? If so then maybe your static toolchain could be incorporated into the OpenBLAS project and used to generate x86_64 windows static libraries for distribution with their current binaries (EG: contain both dynamic and static libraries). Also is your numpy.patch required to build numpy with the OpenBLAS static library? What is the status of the patch - will it be pulled into numpy? Thanks again!

The numpy patch will be changed to a numpy PR very time soon.
 

[Zhang Xianyi]

Hi Carl,

Thank you for the solution.

Could you write a wiki about Numpy and OpenBLAS on https://github.com/xianyi/OpenBLAS/wiki ? 

Xianyi

[Mark Mikofski]

Carl this is great! Unfortunately, probably because I didn't apply the numpy.patch, I get several unresolved references when it gets to init_dotblas. here is a partial stack trace.

 __imp_OpenProcessToken

__imp_LookupPrivilegeValueA

__imp_AdjustTokenPrivileges

 ___chkstk_ms

I used the openblas section of the site.cfg.example, uncommenting

[openblas]

libraries = openblas

library_dirs = c:\path\to\2014-08-28_OpenBLAS-0.2.12dev_PR440\amd64\lib

and I copied libopenblas.a to openblas.lib

This is kind of the holy grail. If anyone can make this static library "openblas.lib" compile with numpy, then we won't need to depend on Intel MKL or C.Gohlke anymore, and numpy can be distributed as a wheel.

What can I do to help?

On Thursday, October 23, 2014 6:07:05 AM UTC-7, ckl wrote:

[ckl]

Do you try to use VisualC together with OpenBLAS? In this case you should read How-to-use-OpenBLAS-in-Microsoft-Visual-Studio.
You should create an import library for libopenblas.dll with the lib.exe. Do not try to link against the static import library libopenblas.a. I'm not sure, that this would ever work.

BTW: the first three missing symbols are part of advapi.dll AFAIK, the last one is mingw-w64 specific: it is part of libgcc.a. The MS counterpart is _chkstk, see: adventures-with-chkstk.htm

[Mark Mikofski]

@ckl, Success! Thank you!

* Yes I am using MSVC90 (via SDK7) which is the runtime for Python-2.7. I applied the vcvarsall.bat.patch (https://gist.github.com/mikofski/11024332) so I can use msvc90 from any console.

* I tried using your amd64\bin\libopenblas.dll and it works! Note, using the static library library libopenblas.a did not work.

So now my questions are:

------------------------

* Can this binary dll be built by openblas team and available on their website? maybe it is already? is it the one that is available on the site now? maybe I made mistakes before building it? Or it didn't build without CKL's numpy patch?

* Can the wheel that uses openblas.dll be uploaded top PyPI and sourceforge for x64 windows users to use? Let's make this the status quo instead of the crazy hoops we have to jump through now.

* How can we benchmark the performance of NumPy-OpenBLAS vs NumPy-MKL?

* can openblas.dll be put into Python DLL's folder or dynload folder for Numpy to build on automatically? this would make the process even easier and make it available for other python packages to use it too

Exact steps to build NumPy with CKL's dynamic OpenBLAS library:

---------------------------------------------------------------

1. Create definitions file and a new dll export lib.

* Unfortunately using the lib export file created by openblas build (aka amd64\lib\libopenblas.dll.a which is  about 5kb) does not work

* you will need wither mingw-w64/msys2 gendef.exe or mingw/msys pexports.exe

* you will also need microsoft sdk 7, the following commands used the sdk 7 environment shell

C:\> setenv /x64 /release
C:\> pexports.exe libopenblas.dll > libopenblas.def
C:\> lib.exe /machine:X64 /def:libopenblas.def

* the new libopenblas.lib file is about 1kb

2. copy both dll, lib files to numpy/core

3. copy header files to numpy/core/openblas (is this necessary?)

4. create site.cfg

[openblas]
libraries = libopenblas
library_dirs = c:\path\to\numpy-1.9.1\numpy\core
include_dirs = c:\path\to\numpy-1.9.1\numpy\core\openblas
The dll and exports library must be in core, or I get the traceback that the DLL could not be imported.

* The dll and exports library must be in core, or I get the traceback that the DLL could not be imported

* Can relative paths be used here? eg just "numpy\core" instead of absolute path

* from here on out I ran these in a windows cmd console in a virtualenv called "venv" with only pip, setuptools, wheel and nose

(venv) C:\> python setup.py config # check config is okay
(venv) C:\> python setup.py config build install

5. copy libopenblas.dll to lib\site-packages\numpy\core

6. start python and import numpy, yay! it works! show the config and run the tests

(venv) C:\Users\mmikofski\Downloads>python
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__config__.show()
lapack_opt_info:
    libraries = ['libopenblas', 'libopenblas']
    library_dirs = ['c:\\path\\to\\numpy-1.9.1\\numpy\\core']
    language = f77
blas_opt_info:
    libraries = ['libopenblas', 'libopenblas']
    library_dirs = ['c:\\path\\to\\numpy-1.9.1\\numpy\\core']
    language = f77
openblas_info:
    libraries = ['libopenblas', 'libopenblas']
    library_dirs = ['c:\\path\\to\\numpy-1.9.1\\numpy\\core']
    language = f77
openblas_lapack_info:
    libraries = ['libopenblas', 'libopenblas']
    library_dirs = ['c:\\path\\to\\numpy-1.9.1\\numpy\\core']
    language = f77
blas_mkl_info:
  NOT AVAILABLE
>>> numpy.test()
Running unit tests for numpy
NumPy version 1.9.1
NumPy is installed in C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy
Python version 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)]
nose version 1.3.4
----------------------------------------------------------------------

Ran 5178 tests in 33.850s


OK (KNOWNFAIL=10, SKIP=19)
<nose.result.TextTestResult run=5178 errors=0 failures=0>

7. create a binary distribution windows installer and convert to wheel (attached)

* I had to add this to setup.py to add the dll to the binary distribution windows installer

--- setup.py.orig       Wed Nov 12 14:51:41 2014
+++ setup.py    Wed Nov 12 14:41:20 2014
@@ -213,6 +213,7 @@
         platforms = ["Windows", "Linux", "Solaris", "Mac OS-X", "Unix"],
         test_suite='nose.collector',
         cmdclass={"sdist": sdist_checked},
+        package_data={'numpy.core': ['*.dll']}
     )


     # Run build

* Now make the binary distribution windows installer and convert it to a wheel.

C:\> python setup.py config build bdist_wininst
C:\> wheel convert dist\numpy-1.9.1.win-amd64-py2.7.exe

As I mentioned above, I also tried using libopenblas.dll.a as the exports library as it says in the wiki, but I got this traceback.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\__init__.py", line 170, in <module>
    from . import add_newdocs
  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\add_newdocs.py",

line 13, in <module>
    from numpy.lib import

add_newdoc
  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\lib\__init__.py", line 18, in <module>
    from .polynomial import *
  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\lib\polynomial.py", line 19, in <module>

    from numpy.linalg import eigvals, lstsq,

inv
  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\linalg\__init__.py", line 51, in <module>
    from .linalg import *
  File "C:\path\to\numpy-1.9.1\venv\lib\site-packages\numpy\linalg\linalg.py", line 29, in <module>

Thanks again!

[Olivier Grisel]

Thanks Mark for the step by step guide.

I have started a PR to build and run tests for numpy under windows
using http://appveyor.com/ :

https://github.com/numpy/numpy/pull/5328

Right now the build fails:

https://ci.appveyor.com/project/ogrisel/numpy

because I did not properly generate the import libs (see the
appveyor.yml file for the details of the current broken setup).

Would you be interested in helping setting up this build environment?
If numpy can get continuous integration with stable tests, I am sure
it would help convince the numpy developers package numpy with
OpenBLAS by default in binary wheel packages on PyPI for the future
releases.

If you prefer to push to my branch to work on the same PR I can also
give you push rights to my numpy repo, just give me your github id.

--
Olivier

[Zhang Xianyi]

It looks like OpenBLAS win binary package needs Carl's static gcc toolchain and building on windows machine.