With regard to include mingw-w64 in the 64bit distributions

[Yuxiang Wang]

Hi all,

I have revisited the ticket #24 today, on http://sourceforge.net/p/winpython/tickets/24/ and was starting to think about the possibility to include the mingw-w64 on the 64 bit version, both Python 2 and 3.

I have talked to one of the Cython developers about how mingw-w64 is not recommended for cython in windows, however he told me that the page saying "mingw-w64 is unstable yet" is totally outdated. Mingw64 is stable now (though few compatibility issues with MSVC still exist), and works better with Cython (e.g. include OpenMP support while VC2010 express does not, and also can use pyximport with appropriate settings etc.).

I also did successfully set up mingw-w64 with Python 3. It is correct that Python 3 lacks libmsvcr100.a and libpython33.a, and that can be solved by installing Christopher Gholke's libpython binary installer. And everything else is the same with mingw, as Pierre said in https://code.google.com/p/winpython/source/browse/make.py#28

Mine is TDM-GCC and it worked great so far.

I do realize that Pierre is really busy and is less related to Python programming as he moves up to a more managerial position (congrats again!), and I just wanted to throw this idea out here in case someone else will continue on the project and have a chance to include it.

Lastly - thanks again Pierre for the great work!

-Shawn

[Yuxiang Wang]

Hi all...

Sorry that I said "Mine is TDM-GCC and it worked great so far." This changed...

It cannot work with memoryview, as of now (2014-06-15).

For more details, please see:   https://groups.google.com/forum/?fromgroups#!topic/cython-users/IHqn8QokZo8

-Shawn

[stonebig]

Hi Yuxiang Wang,

I did build a winpython 3.4 - 64 with wingw64, but I have issues (cython compiles but doesn't link)
Could you post the directory tree of you TDM-GCC installation, and how you installed it, so I may get a clue of what I did wrong ?

[carlkl]

Hi,

you may try my experimental mingw-w64 builds (32bit/64bit) on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads . These builds are made for building python extensions and easy depolyment, see https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/readme.txt . To ensure exclusive linking to msvcr90.dll or msvcr100.dll you have to manually choose a suitable specs-file according to the readme.txt in the archive (look for "hints for Python usage"). Default is msvcr90 linkage.
This mingw distribution includes manifest linkage and statically linking of the gcc-mingw runtimes. That means, NO dependancy of any of the mingw-w64 specific runtime-dlls like libgcc_s_seh.dll, libgfortran.dll, libstdc++6.dll. In contrast to TDM's distribution this distributions ensures fully ABI compatibility to the standard mingw-w64 distributions.

gcc-4.8.2 64bit : https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingw64static-2014-07.7z
gcc-4.8.2 32bit : https://bitbucket.org/carlkl/mingw-w64-for-python/downloads/mingw32static-2014-07.7z

Any feedback is welcome, as this distribution should be considered as experimenatl right now.

Cheers,

Carl

[stonebig]

Hi Carl,

Thank you for the link.

I will try again experimenting with "usb" 64 bit compilers when cython 0.21 will be out.

Regards,

[stonebig]

Hi Carl,

I finally tried it with mingw64static-2014-07.7z, and it blew up on me :
- pc blocked during long seconds,
- ipython notebook message suggesting a python 3 issue ? (see after)
- and in the dos window (that I unfortunately closed) maybe a message of missalignment

==> advices welcomed, as I really whish cython to work on Python 3.4.2 64 bit !

D:\result_tests\WinPython-64bit-3.4.1.2_build07\python-3.4.1.amd64\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    191     # but it's overkill for just that one bit of state.
    192     def magic_deco(arg):
--> 193         call = lambda f, *a, **k: f(*a, **k)
    194 
    195         if callable(arg):

D:\result_tests\WinPython-64bit-3.4.1.2_build07\python-3.4.1.amd64\lib\site-packages\IPython\extensions\cythonmagic.py in cython(self, line, cell)
    269             self._code_cache[key] = module_name
    270 
--> 271         module = imp.load_dynamic(module_name, module_path)
    272         self._import_all(module)
    273 

D:\result_tests\WinPython-64bit-3.4.1.2_build07\python-3.4.1.amd64\Scripts\stringsource in init _cython_magic_8ff05497b86c77c64d80657a18455241 (D:\result_tests\WinPython-64bit-3.4.1.2_build07\settings\.ipython\cython\_cython_magic_8ff05497b86c77c64d80657a18455241.c:13258)()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 1: invalid start byte

[Carl Kleffner]

Thank you for trying this out.

My mingw-w64 distribution is workable but has some still undocumented rough edges.

Please try the following:

(1) make the compiler emit MSVC100 dependant binaries. See the readme inside the archive:

default crt ist msvcr90

- make msvcr100 the default crt: (Python >= 3.3)

  copy lib\gcc\x86_64-w64-mingw32\4.8.2\specs100 to lib\gcc\x86_64-w64-mingw32\4.8.2\specs

You can test compiled binaries with the ntldd command: > ntldd test.exe

Or use www.dependencywalker.com/

(2) ensure to use correct import file for python34.dll

- locate the folder containing python34.dll

> remove or backup an existing libpython34*.a file

> gendef python34.dll      (creates python34.def)
> dlltool -D python34.dll -d python34.def -l libpython34.a

> move libpython34.a to the libs folder

> and remove any msvcrXX.a import files in the libs folder

Best Regards,

Carl

--
You received this message because you are subscribed to a topic in the Google Groups "WinPython" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/winpython/HUJD20S9_yg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to winpython+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[stonebig]

Hi carlkl,

I did spend the week-end on it :
  https://groups.google.com/forum/#!topic/cython-users/GnBAjzbL4rY

Globally, it works when I do not use numpy :
- HelloWorld,
- %%cython
   print (2*3)
- but I have a compilation issue (my memory is filled in 1 second) when using numpy.

==> What does it validate as "ok" among your suggestions ?

For the moment I suspect it's :
- the numpy\distutils that doesn't like my "un-registered" winpython,
- please pinpoint me the suggestion you just made that could be also the problem


I don't know exactly how to nail down  the issue, as my pc blocs when it fails.
==> Any suggestion to help for this "bug" tracking is welcome.

[stonebig]

checking cautiously :
from numpy.distutils.misc_util import msvc_runtime_library, get_build_architecture
get_build_architecture()

'AMD64'

==> that looks ok

import os
import sys
import subprocess

p = subprocess.Popen(['gcc', '-dumpversion'], shell=True,  stdout=subprocess.PIPE)
out_string = p.stdout.read()
p.stdout.close()

>>> out_string
b'4.8.2\r\n'

==> that is curious, as I see this directory in your distribution :
lib\gcc\mingw32\4.8.1

I have also a "PC installation" C:\MinGW\lib\gcc\mingw32\4.8.1 

==> How this 4.8.2 can arrive ?

[stonebig]

I mistake in my control , I'm looking ok at this :


- I have rigthfully a 4.8.2 : WinPython-64bit-3.4.1.2_build14\tools\Mingw32\lib\gcc\x86_64-w64-mingw32\4.8.2
- specs = specs100

so I don't guess where is the problem

On Monday, September 29, 2014 5:19:13 PM UTC+2, carlkl wrote:

[Carl Kleffner]

I made the experience, that the compilation is very picky about the import library. So you have to create a fresh one specific to the compiler - see (2)

Cheers,

Carl

[stonebig]

hi,

I'm just in the process of trying that, as it's my last clue for the day.

[stonebig]

generated libpython34.a was 947k, but existing is 945k.

I get a fail result on basic test that is working If I revert to existing libpython34.a

%%cython
print(5*7, 'e')

==> Does it tell us something ?

---------------------------------------------------------------------------
LinkError                                 Traceback (most recent call last)
<ipython-input-2-fa5a48e13e19> in <module>()
----> 1 get_ipython().run_cell_magic('cython', '', "print(5*7, 'e')")

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\site-packages\IPython\core\interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2160             magic_arg_s = self.var_expand(line, stack_depth)
   2161             with self.builtin_trap:
-> 2162                 result = fn(magic_arg_s, cell)
   2163             return result
   2164 

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\site-packages\IPython\extensions\cythonmagic.py in cython(self, line, cell)

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)

    191     # but it's overkill for just that one bit of state.

    192     def magic_deco(arg):

--> 193         call = lambda f, *a, **k: f(*a, **k)
    194 
    195         if callable(arg):

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\site-packages\IPython\extensions\cythonmagic.py in cython(self, line, cell)
    266             build_extension.build_temp = os.path.dirname(pyx_file)
    267             build_extension.build_lib  = lib_dir
--> 268             build_extension.run()

    269             self._code_cache[key] = module_name
    270 

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\command\build_ext.py in run(self)
    346 
    347         # Now actually compile and link everything.
--> 348         self.build_extensions()
    349 
    350     def check_extensions_list(self, extensions):

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\command\build_ext.py in build_extensions(self)
    455         for ext in self.extensions:
    456             try:
--> 457                 self.build_extension(ext)
    458             except (CCompilerError, DistutilsError, CompileError) as e:
    459                 if not ext.optional:

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\command\build_ext.py in build_extension(self, ext)
    542             debug=self.debug,
    543             build_temp=self.build_temp,
--> 544             target_lang=language)
    545 
    546     def swig_sources(self, sources, extension):

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\ccompiler.py in link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    715                   libraries, library_dirs, runtime_library_dirs,
    716                   export_symbols, debug,
--> 717                   extra_preargs, extra_postargs, build_temp, target_lang)
    718 
    719 

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\cygwinccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    246                            None, # export_symbols, we do this in our def-file
    247                            debug, extra_preargs, extra_postargs, build_temp,
--> 248                            target_lang)
    249 
    250     # -- Miscellaneous methods -----------------------------------------

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\lib\distutils\unixccompiler.py in link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    194                 self.spawn(linker + ld_args)
    195             except DistutilsExecError as msg:
--> 196                 raise LinkError(msg)
    197         else:
    198             log.debug("skipping %s (up-to-date)", output_filename)

LinkError: command 'D:\\result_tests\\WinPython-64bit-3.4.1.2_build14\\python-3.4.1.amd64\\..\\tools\\mingw32\\bin\\gcc.exe' failed with exit status 1

[Carl Kleffner]

It just tells, that you still have mingw32 in your path.

LinkError: command 'D:\\result_tests\\WinPython-64bit-3.4.1.2_build14\\python-3.4.1.amd64\\..\\tools\\mingw32\\bin\\gcc.exe' failed with exit status 1

Cheers,

Carl

[stonebig]

I put your compiler solution under this same directory name as what I use for 32bit python, to avoid redoing some script.
I should later user a Mingw "neutral" name.

[Carl Kleffner]

Hi,

I see. Is the helloworld testcase on the cython ML something I could try out? In this case I would take a look (tomorrow).

Carl

[stonebig]

well it's just the tutorial of cython
http://docs.cython.org/src/tutorial/cython_tutorial.html

As  I just realize this is the compilation process that goes wild
so I use Ipython Notebook :

%load_ext cythonmagic

then

%%cython
print (5*7, 'e')

(which works)

then , hey, thinking about it, this not the execution that blows up my pc, but the compilation process itself.

%%cython
import numpy as np
cimport cython
from libc.math cimport sqrt

@cython.boundscheck(False)
@cython.wraparound(False)
def pairwise_cython(double[:, ::1] X):
    cdef int M = X.shape[0]
    cdef int N = X.shape[1]
    cdef double tmp, d
    cdef double[:, ::1] D = np.empty((M, M), dtype=np.float64)
    for i in range(M):
        for j in range(M):
            d = 0.0
            for k in range(N):
                tmp = X[i, k] - X[j, k]
                d += tmp * tmp
            D[i, j] = sqrt(d)
    return np.asarray(D)

==> I may try to compile that without using the Ipython notebook and maybe I will get a better hint of the problem

[Carl Kleffner]

I will try it tomorrow as I have no installation under my fingers right now.

Carl

[stonebig]

hi,

This morning I try a numpy cython example, compiled by hand (not by %%cython ipython notebook), and it doesn't fail.

... so fail condition seems a little more limited : "%%cython with numpy"



D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64>python setup.
py build_ext --inplace
Compiling convolve1.pyx because it changed.
Cythonizing convolve1.pyx
running build_ext
building 'convolve1' extension
D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\..\tools\ming
w32\bin\gcc.exe -DMS_WIN64 -mdll -O -Wall -ID:\result_tests\WinPython-64bit-3.4.
1.2_build14\python-3.4.1.amd64\include -ID:\result_tests\WinPython-64bit-3.4.1.2
_build14\python-3.4.1.amd64\include -c convolve1.c -o build\temp.win-amd64-3.4\R
elease\convolve1.o
writing build\temp.win-amd64-3.4\Release\convolve1.def
D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64\..\tools\ming
w32\bin\gcc.exe -shared -s build\temp.win-amd64-3.4\Release\convolve1.o build\te
mp.win-amd64-3.4\Release\convolve1.def -LD:\result_tests\WinPython-64bit-3.4.1.2
_build14\python-3.4.1.amd64\libs -LD:\result_tests\WinPython-64bit-3.4.1.2_build
14\python-3.4.1.amd64\PCbuild\amd64 -lpython34 -lmsvcr100 -o D:\result_tests\Win
Python-64bit-3.4.1.2_build14\python-3.4.1.amd64\convolve1.pyd

D:\result_tests\WinPython-64bit-3.4.1.2_build14\python-3.4.1.amd64>python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import convolve1
>>> convolve1.naive_convolve(np.array([[1, 1, 1]], dtype=np.int), np.array([[1],
[2],[1]], dtype=np.int))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'np' is not defined
>>> import numpy as np
>>> import convolve1
>>> convolve1.naive_convolve(np.array([[1, 1, 1]], dtype=np.int), np.array([[1],
[2],[1]], dtype=np.int))
array([[1, 1, 1],
       [2, 2, 2],
       [1, 1, 1]])

[stonebig]

even more curious , this last example doesn't fail with %%cython (python3.4 64 bit)
(with the speed-up of 25% given by not helping cython)

==>  I may try from there at which point of "cythonic" additions it will fail again

%%cython
from __future__ import division
import numpy as np
def naive_convolve1(f, g):
    # f is an image and is indexed by (v, w)
    # g is a filter kernel and is indexed by (s, t),
    #   it needs odd dimensions
    # h is the output image and is indexed by (x, y),
    #   it is not cropped
    if g.shape[0] % 2 != 1 or g.shape[1] % 2 != 1:
        raise ValueError("Only odd dimensions on filter supported")
    # smid and tmid are number of pixels between the center pixel
    # and the edge, ie for a 5x5 filter they will be 2.
    #
    # The output size is calculated by adding smid, tmid to each
    # side of the dimensions of the input image.
    vmax = f.shape[0]
    wmax = f.shape[1]
    smax = g.shape[0]
    tmax = g.shape[1]
    smid = smax // 2
    tmid = tmax // 2
    xmax = vmax + 2*smid
    ymax = wmax + 2*tmid
    # Allocate result image.
    h = np.zeros([xmax, ymax], dtype=f.dtype)
    # Do convolution
    for x in range(xmax):
        for y in range(ymax):
            # Calculate pixel value for h at (x,y). Sum one component
            # for each pixel (s, t) of the filter g.
            s_from = max(smid - x, -smid)
            s_to = min((xmax - x) - smid, smid + 1)
            t_from = max(tmid - y, -tmid)
            t_to = min((ymax - y) - tmid, tmid + 1)
            value = 0
            for s in range(s_from, s_to):
                for t in range(t_from, t_to):
                    v = x - smid + s
                    w = y - tmid + t
                    value += g[smid - s, tmid - t] * f[v, w]
            h[x, y] = value
    return h

[Carl Kleffner]

Hi,

glad to hear that. I have been sucessfull compiling some packages with WinPython (3.4 64bit) and my mingw-w64. Usually I use miniconda or a plain python installation

I didn't tried %%cython though

Carl

[stonebig]

Is this ugly theory  possible ?
- an initial compilation was done in 32 bit,
- when I did switch to 64 bit, the source code was not touched, so cython decided there was no needs to recompile,
- I will blow up again and again, until I change the source code a little.

[stonebig]

Hum,

I did change the code, and now it doesn't blow up.
==> I suspect the problem is something along these lines.

...

[stonebig]

Tomorrow !

[stonebig]

Done !  (only a build for python3.4 64 bit at the moment)

https://sourceforge.net/projects/stonebig.u/files/Winpython_3.4betas/

To activate Mingw for winpython64, follow the same procedure as for 32bit, which includes the live patch on distutils to add -DMS_WIN64 compilation option

http://nbviewer.ipython.org/github/winpython/winpython_afterdoc/blob/master/examples/using_cython_under_winpython32bit.ipynb

[Yuxiang Wang]

Way to go stonebig!! Thank you so much for the excellent work. And I assume it is MinGW-w64 instead of mingw?

[stonebig]

yes,

But playing the procedure via ipython instead of command line , i See a small issue.

==> click directly on script\make_cython_use_mingw.bat
or
patcht this line

Find_And_replace.vbs "%WINPYDIR%\Lib\distutils\cygwinccompiler.py" "-O -W" "-O -DMS_WIN64 -W"
to
%~dp0Find_And_replace.vbs "%WINPYDIR%\Lib\distutils\cygwinccompiler.py" "-O -W" "-O -DMS_WIN64 -W"

==> otherwise you distutils won't be patch properly (you will see an error "Find_And_replace.vbs not a command

==> I'm going to remove that build and repush another one to avoid the problem

[Yuxiang Wang]

Hi stonebig,

Thanks again for the hard work.

I remembered that while I was using mingw-w64 a while ago, the memoryview would not work (https://groups.google.com/forum/?fromgroups#!topic/cython-users/IHqn8QokZo8). 

I will check to see whether this has been fixed after you got the new build ready. Thanks again!

-Shawn

[stonebig]

You did try the TDM-GCC mingw64 version.

I picked the mingw64 'static' version used by numpy, which I hope works better.

[Yuxiang Wang]

Hi stonebig,

That makes total sense... Gotcha. Thanks! :)

Looking forward to the new build,

-Shawn

[stonebig]

hi,

So build is there :
http://sourceforge.net/projects/stonebig.u/files/Winpython_3.4betas/WinPython-64bit-3.4.1.2_build17.exe/download

Procedure/Demo is there :
http://nbviewer.ipython.org/github/stonebig/winpython_afterdoc/blob/WinPython-64bit-3.4.1.2_build17/examples/using_cython_under_winpython64bit.ipynb

You'll notice two 'details'  :
- problems when trying to compile twice the some cython function on Ipython
  (seems a mingw64 or Ipython issue, I believe I saw several references to this issue on internet)
  ==> should I suggest a regular cleanup of WinPython-64bit-3.4.1.2_build17\settings\.ipython\cython ?

- a warning when compiling numba (seems not important)
  ==> I don't know how to fix this detail.

I'm not a cython expert, so maybe there are some easy fix to do.
Let me know what you think of it.

Regards,

[Wang Yuxiang]

PERFECT.

Thank you stonebig!! You just made using cython+gcc under Windows much easier :)

Two thoughts:

1) I am really new to writing software, but I think there should be a way to run "make_cython_use_mingw.bat" automatically after installation I guess...? Or are you making the manual installation intentionally?

2) I haven't tried to build winpython myself yet, but it seems that if we modify winpython/winpython/data/packages.ini we could add descriptions to the new packages. I am happy to try to get a PR on that file if you think appropriate?

-Shawn

[Wang Yuxiang]

Oh, and a side question - when you are preparing the binaries, you need the packages downloaded from Chris Gholke right? Did you use a web-crawler to do that? Do you know whether there is ready code for that purpose?

Thanks,

Shawn

[stonebig]

Hi Shawn,

* Winpython using by default its internal compiler is a tempting option :
  . it may allow to install "source packages" that need compilations,
  . numba would work out of the box, ...
  . but :
      . mingw64 must be made working for other python flavors,
      . winpython is to patch further.
      ==> I'm not sure if this will be ready (or wise) for next release.

* Package Descriptions are at https://github.com/winpython/winpython/blob/master/winpython/data/packages.ini
==> Feel free to do a pull request to add some new Descriptions.

* Christoph Gohlke made it difficult to use a web-crawler to download his binaries : server capacity, intel compiler licence issue, technical  matter, ... whatever.
==> I don't think you'll find a web-crawler that fits.

[Yuxiang Wang]

Hi stonebig,

Thank you for your response! And I agree with your thoughts on whether to set mingw as a default. That is probably the same reason why Pierre did not make mingw default in the 32-bit versions, too.

A side question: just curious, why was pyside taken out of the build...?

-Shawn

[Carl Kleffner]

Hi,

great to see the compiler toolchain in the WinPython beta now.

I have some remarks though:

the MSVCRT linkage is controlled by a so called "specs" file in the folder: tools\mingw32\lib\gcc\x86_64-w64-mingw32\4.8.2

removing the "specs" file yields the standard linkage to MSVCRT.DLL . I provided two specs files as templates: "specs100" and "specs90". The specs file in the mingw64static archive is identical to specs90. To get linkage to MSVCR90.DLL or MSVCR100.DLL simply copy the template to "specs". Unfortunately the specs file from the archive wasn't changed to msvcr100 linkage.

I cannot emphasize enough how important it is to link correct msvcr runtimes to the binary extension files. That means:

- use msvcr90 linkage: Python2.6, 2.7, 3.1, 3.2

- use msvcr100 linkage: Python3.3, 3.4

Sometimes incorrect settings seems to work on one systems but fails on anoher systems or fails under unknown circumstances. This problem was one of the reasons to create a dedicated mingw-w64 toolchain for python extension building.

Another important detail is to ONLY USE correct import libraries inside the folder python-3.4.1.amd64\libs . That means generate the python import library by the mingw toolchain you are going to use and copy the msvcr100 (or msvcr90) import file from the toolchain inside to libs folder to prevent creation of a possibly non-working import file that will hide the import file of the toolchain when compiling packages with extended distutils (numpy does that i.e.).

You can use the following lines as a batch file:

-----
REM start "WinPython Command Prompt.exe"
REM You are now at <installation path>\python-3.4.1.amd64
REM make msvcr100 linkage the default: python-3.3 and up
REM
copy  /Y ..\tools\mingw32\lib\gcc\x86_64-w64-mingw32\4.8.2\specs100 ..\tools\mingw32\lib\gcc\x86_64-w64-mingw32\4.8.2\specs
REM hint: Python versions 2.6 ... 3.2 needs msvcr90 linkage
REM
REM generate and copy import files
REM first backup distributed files
move libs\libpython34.a libs\libpython34.a_bak
move libs\libmsvcr100.a libs\libmsvcr100.a_bak
REM
REM copy msvcr100 import file to prevent generation later on
copy /Y ..\tools\mingw32\x86_64-w64-mingw32\lib\libmsvcr100.a  libs\libmsvcr100.a
REM
REM generate python.34 import file
gendef python34.dll
dlltool -D python34.dll -d python34.def -l libpython34.dll.a
move libpython34.dll.a libs
del python34.def
REM
REM you are done
-----

With that I could run using_cython_under_winpython64bit.ipynb.

This steps are mandantory to get a functional system for non trivial extensions.

By now the compiler toolchain itself is quite stable, but not batlte tested by a wider community and should be considered as beta version.

-carlkl

--

[stonebig]

Hi Shawn,

I took Pyside out because :
- I wanted to make a big room to be able to include :
   . an integrated compilation solution,
   . more graphic packages,
   . more sql packages.
- Pyside was looking  redundant clone with PyQt (at the exception of the licence), and Pierre wrote PyQt was preferable.

Discussion is open on :
- how to improve current single' flavor' : what should be in and what should be out, 
- what should be several 'flavors',
- how to get users picking/doing/contributing their own 'flavor',
- ....

Regards,

[stonebig]

Hi carlkl,

Ooups, you're right :
- I did tweak that in the previous installed build on my pc,
- but I forgot to put it back on the build 'model'.

==> I will try fix that in next build.

Thank you for the remark and proposed batch.

[stonebig]

these  lines

***

dlltool -D python34.dll -d python34.def -l libpython34.dll.a
move libpython34.dll.a libs

***
isn't it rather ?

***
dlltool -D python34.dll -d python34.def -l libpython34.a
move libpython34.a libs
***

It seems my %%cython "DLL in memory" problem is gone indeed.

So, if a Winpython installation wants to shift back from mingw64 to visual studio :
we must replace libmscvcr100.a and libpython34.a with the former ones ?

[Yuxiang Wang]

Hi stonebig,

Again thank you for your response.

1) IMO, I would be reluctant to remove a package because there could be users using WinPython with PySide before (actually I am one of them). But in the same time, I can edit my code for PyQt4, or just pip install pyside. So I guess... I don't know, but if this is just for the sake of saving space, I wouldn't prefer having it removed. Just my two cents:)

2) I like the flavor idea! I think the "main" flavor that Pierre had would be useful for most of the users, and they can install their specific packages themselves. I would really love a second flavor of Cython etc. (the mess with c compiler), because that is non-trivial to make by oneself. Are you thinking in the same line, or are you referring to something else when saying "flavors"?

Final note... As a user I have very limited knowledge in software engineering, so please excuse me for the naiveness :)

-Shawn

[stonebig]

functional flavors :
- "education",
- "performance",
- "Qt lovers",
- "big data / cloud", ...

origin flavors :
- bigstone taste
- Shawn taste,
- Jim taste,
- Sebix taste, ...

circumstance flavors :
- winpython 101,
- french course,
- winpython usb-vc2010, ....

[Yuxiang Wang]

Stonebig,

Sounds like an interesting idea! I'd propose a "basic" version though, which is similar to Pierre's original selection for backward compatibility. 

We should probably open a new thread and talk about what to put in each taste!

-Shawn

--

You received this message because you are subscribed to a topic in the Google Groups "WinPython" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/winpython/HUJD20S9_yg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to winpython+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Yuxiang "Shawn" Wang
Gerling Research Lab
University of Virginia
yw...@virginia.edu
+1 (434) 284-0836
https://sites.google.com/a/virginia.edu/yw5aj/

[carlkl]

Using libpython34.dll.a instead of libpython34.a is a convention. Usually the ###.dll.a denotes the import library for a DLL rather than a static import library. In this case you can use libpython34.a as well, as no static import library is available.

Switching back to VS means, that libpython34.a  (or libpython34.dll.a) and libmsvcr100.a are simply ignored. The former import files you provided are not needed other than for testing purposes or comparison. Are this files from C. Gohlke's repository?

-carlkl

[stonebig]

Hi carlkl,

Yes : until now I get the cython and libpython packages from Christoph.

I think I should pre-include more of the "mingw" preparation in the downloaded stuff.
==> Is there a problem if I pre-launched your batch "before" doing the distribution ?
(If it's independant from the target PC, only dependant of the python an mingw version, it should be ok, shoudn't it ?)

should I :
  * just not use Christoph  libpython  package  ? it seems just to contain libmsvcr100.a and libpython34.a, so to be over-written by your own batch
  * or shall I keep his versions, as other binaries are coming from him today ?
  (one is obviously to remove, but I'm not sure of the best choice)

Regards.

[Carl Kleffner]

Hi,

don't use Christoph's libpython package at all. The best is to prepare libpython34.a, libmsvcr100.a and the specs file according to the batch file I've provided and then deploy a new version of WinPython.

New versions of the mingw64static archive will be provided with a postinstall script to adapt the libs folder as well as the specs file.

Another possibility is do add OpenBLAS or ATLAS libraries and headers into mingw to support BLAS/LAPACK.

-carlkl

--

[stonebig]

Hi Carlkl,

I will try that :
- don't include Christoph libpython,
- apply you batch when preparing winpython build, to have the relevant libpython34.a, libmsvcr100.a in place out-of-the-box

[stonebig]

Hi Shawn,


Did your OpenMP support work  with this build ?

what do you think of https://bitbucket.org/carlkl/mingw-w64-for-python "2014-08-28_OpenBLAS-0.2.12dev_PR440.7z" ?

What do you think of Numpy 1.9 ?

[Yuxiang Wang]

Hi stonebig,

I haven't tried it yet, but it sounds easy to test it out. I will get back to you before this week ends.

-Shawn

[Yuxiang Wang]

Hi stonebig,

Briefly speaking, 1) OpenMP support worked 2) Sorry but I do not think I have the credibility to comment on Carl's OpenBLAS build due to my limited knowledge on this topic...

Code I used to test openmp:

# test_compiler.pyx

import numpy as np

cimport numpy as np

from cython.parallel cimport prange

cimport openmp

openmp.omp_set_dynamic(0)

openmp.omp_set_num_threads(4)  

def foo():

    cdef int i, j, n

    x = np.zeros((200, 2000), float)

    n = x.shape[0]

    for i in prange(n, nogil=True):

        with gil:

            for j in range(100):

                x[i,:] = np.cos(x[i,:])

    return x

And then the setup_test_compiler.py

from distutils.core import setup

from distutils.extension import Extension

from Cython.Distutils import build_ext

import numpy as np

ext_module = Extension(

    'test_compiler',

    ['test_compiler.pyx'],

    include_dirs = [np.get_include()],

    extra_compile_args = ['-fopenmp'],

    extra_link_args = ['-fopenmp'],

)

setup(

    name = 'test_compiler',

    cmdclass = {'build_ext': build_ext},

    ext_modules = [ext_module],

)

And in ipython, when number of threads is 4, time is 234 ms; when there is only 1 thread, time is 381 ms.

Shawn

On Wednesday, October 8, 2014 3:59:58 PM UTC-4, stonebig wrote:

[stonebig]

Hourra, we can close this thread with a success !

[Yuxiang Wang]

Yay!! :)