Python 3.x and cython (update)

[Antonio]

Hi to all,

Given the undergoing migration from Legacy Python (2.7) to Python ( 3.4 or 3.5), I'm wondering what is the plan to handle cython extensions (optimization modules, not interfacing with C libs) under Windows using Anaconda.

Currently, gcc in Anaconda (installed with `libpython` build dependency) cannot compile cython extension in python 3.4. I reported the issue a few months ago with no answer:

https://groups.google.com/a/continuum.io/forum/#!searchin/conda/python3$20cython/conda/y4VAkVg2jfo/N6QhE5lTrHsJ

There are a few issues on github that left me confused on what is the plan for the future. In addition this post highlights new changes from python 3.5 onward:

http://stevedower.id.au/blog/building-for-python-3-5/

Here are my questions:

1. What is the suggested way to build cython extensions under windows with Anaconda? 

2. Should we dig GCC in favour of MS Visual studio?

3. When building a package with MS Visual studio compiler, is the user guaranteed to have all the runtime dependencies?

The nice thing about GCC is that it's open source, it's the same on all platforms and any runtime can be legally included. 

Regarding the MS Compiler, so far I installed Windows SDK v7.1 and compiled extensions successfully. But it requires opening a special shell included in the SDK which sets all the environmental variables. I need to use that special shell (which has yellow fonts) to build the conda package (fails otherwise). But I'm not sure if I'm introducing any runtime dependency this way, so the package will not work on some machines.

Any suggestion on which is the way to go is appreciated.

Thanks,
Antonio

[Carlos Córdoba]

Hi,

My answers are below

El 29/09/15 a las 16:55, Antonio escribió:

Hi to all,

Given the undergoing migration from Legacy Python (2.7) to Python ( 3.4 or 3.5), I'm wondering what is the plan to handle cython extensions (optimization modules, not interfacing with C libs) under Windows using Anaconda.

Currently, gcc in Anaconda (installed with `libpython` build dependency) cannot compile cython extension in python 3.4. I reported the issue a few months ago with no answer:

https://groups.google.com/a/continuum.io/forum/#!searchin/conda/python3$20cython/conda/y4VAkVg2jfo/N6QhE5lTrHsJ

There are a few issues on github that left me confused on what is the plan for the future. In addition this post highlights new changes from python 3.5 onward:

http://stevedower.id.au/blog/building-for-python-3-5/

Here are my questions:

1. What is the suggested way to build cython extensions under windows with Anaconda?

Based on my experience, the best way to compile Python extensions is to use a conda recipe. Conda-build calls the right version of vcvarsall.bat according to your Python version. And after a recent pull request, you can even override that detection and use the Visual Studio compiler you want :-)

2. Should we dig GCC in favour of MS Visual studio?

Yes. Visual Studio is much better than MinGW, at least for Win 64. It took me like 50 hours to notice that :-)

3. When building a package with MS Visual studio compiler, is the user guaranteed to have all the runtime dependencies?

Right now Anaconda provides the runtime dlls for VS 2008, 2010 and 2015. Only if you use a different compiler, you need to pack its respective dlls.

The nice thing about GCC is that it's open source, it's the same on all platforms and any runtime can be legally included.

There is a new version called MinGWpy (or something like that) that's supposed to be optimized to work with Python. But the Python extensions I've built are bindings for big and complex C++ libraries, and in that case there's no other option but to use VS to compile both library and bindings.

However, it'd be nice if Anaconda moved to use MinGWpy for Cython extensions :-)

Regarding the MS Compiler, so far I installed Windows SDK v7.1 and compiled extensions successfully. But it requires opening a special shell included in the SDK which sets all the environmental variables. I need to use that special shell (which has yellow fonts) to build the conda package (fails otherwise). But I'm not sure if I'm introducing any runtime dependency this way, so the package will not work on some machines.

This is not necessary. Using that special shell is just equivalent to calling vcvarsall.bat, which conda-build already does, as I said. Maybe you have a problem with your installation :-)


Cheers,
Carlos

Any suggestion on which is the way to go is appreciated.

Thanks,
Antonio

--
You received this message because you are subscribed to the Google Groups "conda - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to conda+un...@continuum.io.
To post to this group, send email to co...@continuum.io.
Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Matt Craig]

Hi,

A couple comments only in places that differ from what Carlos said.

> 2. Should we dig GCC in favour of MS Visual studio?

Philosophically, maybe, but python.org builds python on Windows with MSVC so if you want to guarantee compatibility the change needs to happen elsewhere. I have (not infrequently recently) had packages that are mostly python with some C or cython fail tests when not compiled with MSVC on Windows. 

> 3. When building a package with MS Visual studio compiler, is the user guaranteed to have all the runtime dependencies?

In my experience, yes. This is one of the nice things about anaconda :)

> Regarding the MS Compiler, so far I installed Windows SDK v7.1 and compiled extensions successfully. But it requires opening a special shell included in the SDK which sets all the environmental variables. I need to use that special shell (which has yellow fonts) to build the conda package (fails otherwise). But I'm not sure if I'm introducing any runtime dependency this way, so the package will not work on some machines.

*shuddder* My experience with MSVC 2010 has been similar to yours, even when using conda-build. There are several examples on github of windows cmd files that will ensure everything is set up correctly; the one I've used most recently part of obvious-ci at https://github.com/pelson/Obvious-CI/blob/master/scripts/obvci_appveyor_python_build_env.cmd (though it derives from earlier examples).

For python 2.7 MS has made installation really easy, for python 3 it was still painful the last time I did it from scratch (about a year ago). Best guide to setup I've seen recently is at the cython github wiki: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows

Matt Craig

[Antonino Ingargiola]

Thanks Carlos and Matt,

now I know that using MS compiler is the way to go.

With the excuse of switching to anaconda with python 3 in the root env, I'll make a fresh installation and see if conda build will detect the MSVC compiler. 

The fact that anaconda includes all the necessary runtime is reassuring!

Python 3.5 requires MSVC 14.0 or greater. My understanding is that this compiler is freely available only through MS Visual Studio, no SDK anymore. Since from python 3.5 on, things should become easier I'm tempted to skip python 3.4 support.

Is there a time frame for a possible switch to 3.5 as default python in anaconda?

Thanks,

Antonio

[Chris Barker]

A few more notes:

1. What is the suggested way to build cython extensions under windows with Anaconda?

It's been said, but I'll repeat: use the (correct) MS compiler.

 And after a recent pull request, you can even override that detection and use the Visual Studio compiler you want :-)

but no, you don't want to do that -- really.

 

3. When building a package with MS Visual studio compiler, is the user guaranteed to have all the runtime dependencies?

Right now Anaconda provides the runtime dlls for VS 2008, 2010 and 2015.

The python.org builds come with the correct run-times too, so that should not be an issue.

 Only if you use a different compiler, you need to pack its respective dlls.

But DO NOT DO THAT. YOu really, really want to use the version of teh compiler that python itself was built with:

py2.7: VS2008 (The MS compiler for python 2.7)

py3.3 and 3.4 VS2010 (the SDK for Windows7)

py3.5: VS2015 (VS2015 community Edition)

(in the future, in theory, MS compilers will be less tied to the run time, so that we should be abel to build for py3.5 with future version of VS -- but that's an issue for the future, only) 

The nice thing about GCC is that it's open source, it's the same on all platforms

it's the same on all platforms, but not all platforms are the some ;-) You've just shifted your problems down one level....

 

This is not necessary. Using that special shell is just equivalent to calling vcvarsall.bat, which conda-build already does, as I said. Maybe you have a problem with your installation :-)

All too easy....and the right vcvarsall.bat needs to be called.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Aaron Meurer]

On Wed, Sep 30, 2015 at 10:50 AM, Chris Barker <chris....@noaa.gov> wrote:
> A few more notes:
>
>> 1. What is the suggested way to build cython extensions under windows with
>> Anaconda?
>
> It's been said, but I'll repeat: use the (correct) MS compiler.
>
>> And after a recent pull request, you can even override that detection and
>> use the Visual Studio compiler you want :-)
>
>
> but no, you don't want to do that -- really.
>
> 3. When building a package with MS Visual studio compiler, is the user
> guaranteed to have all the runtime dependencies?
>>
>>
>> Right now Anaconda provides the runtime dlls for VS 2008, 2010 and 2015.
>
>
> The python.org builds come with the correct run-times too, so that should
> not be an issue.
>
>> Only if you use a different compiler, you need to pack its respective
>> dlls.
>
>
> But DO NOT DO THAT. YOu really, really want to use the version of teh
> compiler that python itself was built with:
>
> py2.7: VS2008 (The MS compiler for python 2.7)
> py3.3 and 3.4 VS2010 (the SDK for Windows7)
> py3.5: VS2015 (VS2015 community Edition)

I agree, use the right compiler. If you want to use a different
compiler you will have to recompile the entire stack, including Python
itself. Existing C extensions built against the "correct" compiler may
not work.

This feature was added to conda-build because some people wish to do
exactly that, but it's not a simple undertaking, and you should only
do it if you have a good reason and you know what you are getting
into.

Aaron Meurer

>
> (in the future, in theory, MS compilers will be less tied to the run time,
> so that we should be abel to build for py3.5 with future version of VS --
> but that's an issue for the future, only)
>
>> The nice thing about GCC is that it's open source, it's the same on all
>> platforms
>
> it's the same on all platforms, but not all platforms are the some ;-)
> You've just shifted your problems down one level....
>
>>
>> This is not necessary. Using that special shell is just equivalent to
>> calling vcvarsall.bat, which conda-build already does, as I said. Maybe you
>> have a problem with your installation :-)
>
>
> All too easy....and the right vcvarsall.bat needs to be called.
>
> -Chris
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris....@noaa.gov
>

[Antonino Ingargiola]

Chris, Aaron, thanks for your answers.

On a different but related issue, are all Anaconda packages compiled with MS compiler on windows x64?

I ask because there the latest version of pytables (3.2) in Anaconda is broken under windows x64 (on python 3):

https://github.com/PyTables/PyTables/issues/492

Seems like Christoph Gohlke binary works though (and he's using MS compilers AFAICT). 

If you look at the log in the previous bug report, seems like Anaconda is using GCC for compiling pytables. Is it possible that the wrong compiler was selected to build pytables 3.2 conda package?

Antonio

[Carlos Córdoba]

Hi,

El 30/09/15 a las 12:36, Antonino Ingargiola escribió:

Chris, Aaron, thanks for your answers.

On a different but related issue, are all Anaconda packages compiled with MS compiler on windows x64?

Yes, they are compiled with the right Visual Studio version according to the Python version in use.

Cheers,
Carlos

[Antonino Ingargiola]

Hi Carlos,

On Thu, Oct 1, 2015 at 8:51 AM, Carlos Córdoba <ccord...@gmail.com> wrote:

El 30/09/15 a las 12:36, Antonino Ingargiola escribió:

On a different but related issue, are all Anaconda packages compiled with MS compiler on windows x64?

Yes, they are compiled with the right Visual Studio version according to the Python version in use.

Ok, great! 

I ask because there the latest version of pytables (3.2) in Anaconda is broken under windows x64 (on python 3):

https://github.com/PyTables/PyTables/issues/492

Seems like Christoph Gohlke binary works though (and he's using MS compilers AFAICT). 

If you look at the log in the previous bug report, seems like Anaconda is using GCC for compiling pytables. Is it possible that the wrong compiler was selected to build pytables 3.2 conda package?

Can you please take a look at the aforementioned pytables crash log? It has the gcc string all over the place. It may be that it was compiled with the wrong compiler?

[Aaron Meurer]

There is also more information about this at this discussion
https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/iLAxGBWlC3g
(in particular the links
https://github.com/cython/cython/wiki/CythonExtensionsOnWindows and
http://siomsystems.com/mixing-visual-studio-versions/ are very
relevant here).

Aaron Meurer