[fwrap-users] Cython+Fortran builds

[Dag Sverre Seljebotn]

Alas, the time has come that I must distribute my software to others.
And I'm realizing that my beloved scons scripts may no longer be up to
the task, which don't support different Fortran compilers and
installation...

Kurt, you've looked a lot of this for fwrap, but I failed to find a
clear conclusion on the mailing lists... do you have a conclusion now?
What tool would you pick to build a Cython+Fortran project? Is e.g.
numscons currently up for the job?

--
Dag Sverre

--
You received this message because you are subscribed to the Google Groups "Fwrap Users" group.
To post to this group, send email to fwrap...@googlegroups.com.
To unsubscribe from this group, send email to fwrap-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fwrap-users?hl=en.

[Kurt Smith]

On Wed, Apr 21, 2010 at 8:13 AM, Dag Sverre Seljebotn
<da...@student.matnat.uio.no> wrote:
> Alas, the time has come that I must distribute my software to others. And
> I'm realizing that my beloved scons scripts may no longer be up to the task,
> which don't support different Fortran compilers and installation...
>
> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
> conclusion on the mailing lists... do you have a conclusion now? What tool
> would you pick to build a Cython+Fortran project? Is e.g. numscons currently
> up for the job?

In an ideal world -- i.e. a world where there's unlimited time for
FOSS projects -- I'd help David Cournapeau and get something working
with toydist and waf.

The latest thread on the topic in numpy-discussion [0], as you
probably remember, concluded that the best thing for fwrap to do is
delegate the F90 source build to the user.

To keep fwrap from having yet another third-party dependency, I'm
using a hybrid cython distutils and numpy distutils script that builds
the extension module. Since the user supplies the compiled .o and
.mod files, this solution doesn't have to worry about all the things
that waf and numscons do. The testsuite will require a simple
builder, but I'm going to use the above distutils solutions.

I likely have less expertise with scons/numscons than you to make an
informed assessment of those build systems.

I can say, long term, that I'll be contributing to a solution --
probably taking up David C.'s lead.

Kurt

[0] http://old.nabble.com/numpy-build-system-questions-for-use-in-another-project-%28fwrap%29-td28159999.html

[Dag Sverre Seljebotn]

Kurt Smith wrote:
> On Wed, Apr 21, 2010 at 8:13 AM, Dag Sverre Seljebotn
> <da...@student.matnat.uio.no> wrote:
>> Alas, the time has come that I must distribute my software to others. And
>> I'm realizing that my beloved scons scripts may no longer be up to the task,
>> which don't support different Fortran compilers and installation...
>>
>> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
>> conclusion on the mailing lists... do you have a conclusion now? What tool
>> would you pick to build a Cython+Fortran project? Is e.g. numscons currently
>> up for the job?
>
> In an ideal world -- i.e. a world where there's unlimited time for
> FOSS projects -- I'd help David Cournapeau and get something working
> with toydist and waf.
>
> The latest thread on the topic in numpy-discussion [0], as you
> probably remember, concluded that the best thing for fwrap to do is
> delegate the F90 source build to the user.

Thanks for the quick answer. Does this include the wrapper .f90 file, or
is that built as part of the Python extension from .f90 source using
numpy distutils?

>
> To keep fwrap from having yet another third-party dependency, I'm
> using a hybrid cython distutils and numpy distutils script that builds
> the extension module. Since the user supplies the compiled .o and
> .mod files, this solution doesn't have to worry about all the things
> that waf and numscons do. The testsuite will require a simple
> builder, but I'm going to use the above distutils solutions.
>
> I likely have less expertise with scons/numscons than you to make an
> informed assessment of those build systems.
>
> I can say, long term, that I'll be contributing to a solution --
> probably taking up David C.'s lead.

I hope to do so too, but first there's at least my MSc. Scientific
Cython desperately needs a new "default" build system dealing with
Fortran, BLAS, etc.

[Kyle Mandli]

I have been trying to find a solution for this exact problem for quite
awhile but have not found one besides using a combination of make and
distutils. If someone has a better solution, it would be much
appreciated!

Kyle

> [0]http://old.nabble.com/numpy-build-system-questions-for-use-in-another...

>
>
>
> > --
> > Dag Sverre
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Fwrap Users" group.
> > To post to this group, send email to fwrap...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > fwrap-users...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/fwrap-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "Fwrap Users" group.
> To post to this group, send email to fwrap...@googlegroups.com.
> To unsubscribe from this group, send email to fwrap-users...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/fwrap-users?hl=en.

[Kurt Smith]

On Wed, Apr 21, 2010 at 10:58 AM, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
> Kurt Smith wrote:
>>
>> On Wed, Apr 21, 2010 at 8:13 AM, Dag Sverre Seljebotn
>> <da...@student.matnat.uio.no> wrote:
>>>
>>> Alas, the time has come that I must distribute my software to others. And
>>> I'm realizing that my beloved scons scripts may no longer be up to the
>>> task,
>>> which don't support different Fortran compilers and installation...
>>>
>>> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
>>> conclusion on the mailing lists... do you have a conclusion now? What
>>> tool
>>> would you pick to build a Cython+Fortran project? Is e.g. numscons
>>> currently
>>> up for the job?
>>
>> In an ideal world -- i.e. a world where there's unlimited time for
>> FOSS projects -- I'd help David Cournapeau and get something working
>> with toydist and waf.
>>
>> The latest thread on the topic in numpy-discussion [0], as you
>> probably remember, concluded that the best thing for fwrap to do is
>> delegate the F90 source build to the user.
>
> Thanks for the quick answer. Does this include the wrapper .f90 file, or is
> that built as part of the Python extension from .f90 source using numpy
> distutils?

The latter. David C. has consistently pushed using numpy distutils
for fortran builds at the present time. Fwrap needs it for a very
simple and well-defined build dependency structure, so much of the
ugliness in distutils is mitigated. I've not looked at numscons for
fortran builds -- that's likely an option to pursue. There's been
some activity on the waf mailinglist for getting fortran support
working -- once that's up and running I'd like to move towards waf.

>
>>
>> To keep fwrap from having yet another third-party dependency, I'm
>> using a hybrid cython distutils and numpy distutils script that builds
>> the extension module.  Since the user supplies the compiled .o and
>> .mod files, this solution doesn't have to worry about all the things
>> that waf and numscons do.  The testsuite will require a simple
>> builder, but I'm going to use the above distutils solutions.
>>
>> I likely have less expertise with scons/numscons than you to make an
>> informed assessment of those build systems.
>>
>> I can say, long term, that I'll be contributing to a solution --
>> probably taking up David C.'s lead.
>
> I hope to do so too, but first there's at least my MSc. Scientific Cython
> desperately needs a new "default" build system dealing with Fortran, BLAS,
> etc.

Yes, very much so.

[Ondrej Certik]

On Wed, Apr 21, 2010 at 6:13 AM, Dag Sverre Seljebotn
<da...@student.matnat.uio.no> wrote:
> Alas, the time has come that I must distribute my software to others. And
> I'm realizing that my beloved scons scripts may no longer be up to the task,
> which don't support different Fortran compilers and installation...
>
> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
> conclusion on the mailing lists... do you have a conclusion now? What tool
> would you pick to build a Cython+Fortran project? Is e.g. numscons currently
> up for the job?

I use cmake and I use cython, C and fortran in my project. It
generates makefiles that work nicely in parallel (that's very
important for me). And it's very robust, I can only recommend.

Ondrej

[Dag Sverre Seljebotn]

Ondrej Certik wrote:
> On Wed, Apr 21, 2010 at 6:13 AM, Dag Sverre Seljebotn
> <da...@student.matnat.uio.no> wrote:
>> Alas, the time has come that I must distribute my software to others. And
>> I'm realizing that my beloved scons scripts may no longer be up to the task,
>> which don't support different Fortran compilers and installation...
>>
>> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
>> conclusion on the mailing lists... do you have a conclusion now? What tool
>> would you pick to build a Cython+Fortran project? Is e.g. numscons currently
>> up for the job?
>
> I use cmake and I use cython, C and fortran in my project. It
> generates makefiles that work nicely in parallel (that's very
> important for me). And it's very robust, I can only recommend.

Some questions:

- How do you/do you need to ensure that Python extension modules are
correctly linked with Fortran code? Does it automatically switch to
using fortran as the linker when linking Fortran into the extension modules?
- Do you support multiple Fortran compilers or just gfortran?
- How about multiple BLAS implementations?
- How are packages distributed? (.spkg I guess, so that you don't have
to deal with it?)

The cmake support for compiling Cython modules you showed me earlier is
very dangerous and non-crossplatform, but I guess it is easy enough to
fix with some calls to distutils.sysconfig.

--
Dag Sverre

[Ondrej Certik]

On Wed, Apr 21, 2010 at 10:00 AM, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>>
>> On Wed, Apr 21, 2010 at 6:13 AM, Dag Sverre Seljebotn
>> <da...@student.matnat.uio.no> wrote:
>>>
>>> Alas, the time has come that I must distribute my software to others. And
>>> I'm realizing that my beloved scons scripts may no longer be up to the
>>> task,
>>> which don't support different Fortran compilers and installation...
>>>
>>> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
>>> conclusion on the mailing lists... do you have a conclusion now? What
>>> tool
>>> would you pick to build a Cython+Fortran project? Is e.g. numscons
>>> currently
>>> up for the job?
>>
>> I use cmake and I use cython, C and fortran in my project. It
>> generates makefiles that work nicely in parallel (that's very
>> important for me). And it's very robust, I can only recommend.
>
> Some questions:
>
>  - How do you/do you need to ensure that Python extension modules are
> correctly linked with Fortran code? Does it automatically switch to using
> fortran as the linker when linking Fortran into the extension modules?

It's using gcc, as that works for me. I didn't know I need to use
gfotran to link it. Why? If that is the requirement, I think it is
possible to make cmake use gfortran.

>  - Do you support multiple Fortran compilers or just gfortran?

Cmake supports all kinds of fortran compilers. It's true that I only
use gfortran, but I think any compiler would work.

>  - How about multiple BLAS implementations?

cmake discovers blas itself (of course you can do it yourself too), so
I think it works with all blas implementations.

>  - How are packages distributed? (.spkg I guess, so that you don't have to
> deal with it?)

Source, spkg or binaries. Maybe I don't understand the question.

>
> The cmake support for compiling Cython modules you showed me earlier is very
> dangerous and non-crossplatform, but I guess it is easy enough to fix with

Well, it does work on Mac, Linux and Windows (cygwin), as I have
tested it, so I think it's actually pretty cross platform.

> some calls to distutils.sysconfig.

Yep.

Ondrej

[Dag Sverre Seljebotn]

Different Fortran compilers require different RTLs linked in, and the
Fortran compiler deals with it, that's all. In my case, I just hardcode
to use ifort as the linker...

numscons has support to use gcc but link in the right RTL depending on
the Fortran compiler; perhaps cmake does the same.

Anyway, thanks, I'll have a look, though it sounds very similar to what
I already have with scons -- except perhaps that it can generate
makefiles, so no dependency beyond make on the users.

--
Dag Sverre

[Ondrej Certik]

On Wed, Apr 21, 2010 at 10:24 AM, Dag Sverre Seljebotn

Actually, users need cmake, as it generates makefiles for the exact
platform/compilers.

Ondrej

[Kurt Smith]

On Wed, Apr 21, 2010 at 12:00 PM, Dag Sverre Seljebotn

<da...@student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>>
>> On Wed, Apr 21, 2010 at 6:13 AM, Dag Sverre Seljebotn
>> <da...@student.matnat.uio.no> wrote:
>>>
>>> Alas, the time has come that I must distribute my software to others. And
>>> I'm realizing that my beloved scons scripts may no longer be up to the
>>> task,
>>> which don't support different Fortran compilers and installation...
>>>
>>> Kurt, you've looked a lot of this for fwrap, but I failed to find a clear
>>> conclusion on the mailing lists... do you have a conclusion now? What
>>> tool
>>> would you pick to build a Cython+Fortran project? Is e.g. numscons
>>> currently
>>> up for the job?
>>
>> I use cmake and I use cython, C and fortran in my project. It
>> generates makefiles that work nicely in parallel (that's very
>> important for me). And it's very robust, I can only recommend.
>
> Some questions:
>
>  - How do you/do you need to ensure that Python extension modules are
> correctly linked with Fortran code? Does it automatically switch to using
> fortran as the linker when linking Fortran into the extension modules?

Question on this point: is using fortran as the linker a requirement
so that the fortran RTLs are properly handled? Because if I
understand the intent of the ISO C BINDING stuff correctly, it should
be possible (and recommended?) to use gcc/icc/etc as the linker so
long as you specify the RTLs with appropriate link flags. If there
are other problems or pitfalls with using a C compiler as the linker
in hybrid apps I'm all ears. I've been using gcc as the linker for
fwrap tests and haven't run into any problems, but there might be some
dark corners.

[Dag Sverre Seljebotn]

No, sorry for causing confusion. I just don't know how to link in the
RTL of all the different Fortrans out there, I need a build system which
can do it for me :-)

The easy way out in my own pure SCons build system was to link using the
Fortran compiler.

[Dag Sverre Seljebotn]

I just wasted some time on numscons; it didn't look too promising.

- First I had to get it to recognize my ifort (change a regex, remove a
misplaced Windows-only import)
- I had to change suffixes from f90 to f95
- Then it wouldn't build f95 files as shared objects when used as
sources for a Python extension by default

At that point I stopped hacking and went back to my dumber,
no-autodetection system. The promise that lured me to numscons was the
"setup.py install" bit. But I think I'll wait for toydist.

Dag Sverre