differences between conda build and distutils

[Darren Dale]

All,

Please excuse my weakness with windows build environments. I'm seeing different results between a standard (successful) distutils build of an extenssion module and a (work in progress) conda build that simply calls "%PYTHON% setup.py install". These are both performed with the same anaconda installation, the only python environment on my computer.

For example, here is one line from the distutils build:

C:\Users\darren\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Users\darren\Anaconda\lib\site-packages\numpy\core\include\numpy -Ihexrd/transforms -IC:\Users\darren\Anaconda\include -IC:\Users\darren\Anaconda\PC -c hexrd/transforms\transforms_CAPI.c -o build\temp.win-amd64-2.7\Release\hexrd\transforms\transforms_capi.o

Here is the corresponding line from the conda build:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\darren\Anaconda\envs\_build\lib\site-packages\numpy\core\include\numpy -Ihexrd/transforms -IC:\Users\darren\Anaconda\envs\_build\include -IC:\Users\darren\Anaconda\envs\_build\PC /Tchexrd/transforms\transforms_CFUNC.c /Fobuild\temp.win-amd64-2.7\Release\hexrd/transforms\transforms_CFUNC.obj

transforms_CFUNC.c

hexrd/transforms\transforms_CFUNC.c(100) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(101) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(105) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(106) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(193) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(310) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(312) : error C2065: 'NAN' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(320) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(597) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(598) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(599) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(602) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(603) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(604) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(650) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(651) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(652) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(663) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(675) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(676) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(677) : error C2065: 'M_PI' : undeclared identifier

hexrd/transforms\transforms_CFUNC.c(774) : error C2057: expected constant expression

hexrd/transforms\transforms_CFUNC.c(774) : error C2466: cannot allocate an array of constant size 0

hexrd/transforms\transforms_CFUNC.c(774) : error C2133: 'q2s' : unknown size

hexrd/transforms\transforms_CFUNC.c(795) : error C2065: 'NAN' : undeclared identifier

error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\amd64\\cl.exe' failed with exit status 2

Why are two seemingly identical calls to setup.py invoking different compilers? Is there a way for me to instruct conda to use the same compiler and settings as distutils?

Thanks,

Darren

[Aaron Meurer]

conda build sets some environment variables to find the msvc compiler
from vsvarsall.bat (see
https://github.com/conda/conda-build/blob/a9864163a5303de29b3fc44aaac4727ffb63c5d9/conda_build/windows.py#L59
and https://github.com/conda/conda-build/blob/a9864163a5303de29b3fc44aaac4727ffb63c5d9/conda_build/windows.py#L110).
If you know how to improve this code, we would love a pull request. I
suspect it only works specific to the way we have had Visual Studio
set up.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at http://groups.google.com/a/continuum.io/group/conda/.

[Darren Dale]

Unfortunately, this is not any area of expertise for me and I don't have time to invest. I decided to tweak the c code to meet the constraints of VC++ 2008 instead.

[Matt Craig]

Hi Darren,

Could you please provide the command you used to "build with distutils"? I'm guessing it was just "python setup.py install", but would be helpful. 

My interest in this is wanting to get an automatic windows build of some python packages going because most of the undergrads I work with are windows based. 

Thanks,

Matt

PS I think I noticed from your g+ profile that you are at cornell...graduated from there ~25 years ago!

—
Sent from Mailbox

[Darren Dale]

Hi Matt,

Yes, it was just the standard "python setup.py build".

[Chris Barker]

There are multiple issues here, but the key one is:

Anaconda is set to use mingw by default -- thus python setup.py install tries to use that.

conda build re-sets this to use the MS compiler be default.

As both are created by the same organization, and are designed very much to work together, this is a odd and confusing choice.

But there you have it.

However, in theory, conda is designed to work with any python distribution, not just Anaconda -- so it seems what it should do is use the default compiler configured by python (distutils) by default, while letting the user (either the build script writer or the end-user) specify what compiler they want to use, like distutils does.

Is this in the works?

-CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Aaron Meurer]

So Ilan just explained to me that the difference here is the libpython
package. Anaconda comes with it, but by default, it is not installed
into the build environment when you do a conda build. If libpython is
installed, distutils will use mingw. Otherwise, it will use Visual
Studio. If you want to build with mingw with conda build, add
libpython as a build dependency. If you want to not build with it
outside of conda build, either set the compiler manually, or conda
remove libpython.

We should definitely document this better.

Aaron Meurer

[Ilan Schnell]

The way this works is that the libpython conda package (which includes the so-called Python import library, i.e. libpython.a) also includes the file Lib\distutils\distutils.cfg with the following content:

[build]
compiler=mingw32

- Ilan

[Chris Barker]

On Mon, Aug 11, 2014 at 7:19 PM, Ilan Schnell <il...@continuum.io> wrote:

The way this works is that the libpython conda package (which includes the so-called Python import library, i.e. libpython.a) also includes the file Lib\distutils\distutils.cfg with the following content:

[build]
compiler=mingw32

that explain it yes. Thanks. Still confused as to why the default behavior for Anaconda out of the box and conda build is different, but that's a policy/ Anaconda issue.

Also, the mingw build didn't work for me out of the box on Anaconda anyway on two machines so far -- but I'll test more to nail it down and report on the Anaconda list.

-Thanks,

   -Chris