Fortran extensions and setup.py

[george...@noaa.gov]

Hi,

I am creating Cython extension to a FORTRAN library. I do follow advice from www.fortran90.org, that is write an interface using iso_c_binding. My issue is how to compile the code from distutils setup.py. I came up with the following solution:

In setup.py:

import build_ftn_ext    # works only with gcc

def get_extensions():
    if use_cython:
        ext_modules = [
            Extension('nceputils._funcphys',
                      ['nceputils/_funcphys.pyx', 'nceputils/physif.f90'],
                      libraries=['phys', 'gfortran'],
...

where physif.f90 is my interface to library libphys.so and build_ftn_ext is a monkey patch:

import distutils.command.build_ext

# Allows gcc to compile FORTRAN code
_tmp = distutils.command.build_ext.build_ext

class build_ftn_ext(_tmp):
    description = "As build_ext(), allows FORTRAN objects to be made by gcc"

    def build_extensions(self):
        self.compiler.src_extensions.extend(['.f90'])
        for ext in self.extensions:
            ext.extra_compile_args.extend(['-J', self.build_temp])
        _tmp.build_extensions(self)

distutils.command.build_ext.build_ext = build_ftn_ext

This works fine for me, I do not anticipate having to switch to a different compiler. But, is there a better solution?

George

[Björn Dahlgren]

On Wednesday, 18 February 2015 20:06:26 UTC+1, george...@noaa.gov wrote:

This works fine for me, I do not anticipate having to switch to a different compiler. But, is there a better solution?

In my experience, linking C/C++/Fortran is a mess if you want to do it in an exportable manner for different compilers,
because linking for e.g. a C++/Fortran extension is done with the fortran compiler for Intel and with c++ compiler for GNU.
I have made a (very) humble attempt to write a small package to do help with this: github.com/bjodah/pycompilation
but I would not advocate to use it for production critical things, but it might be a useful starting point? There are quite
a few options around to do this (CMake, WAF, numpydistutils, bento) and I too would be curious to hear about peoples
experience with these!

[Sturla Molden]

The only portable way to link Fortran 90 to C is via Fortran 2003 ISO C
bindings. But you are limited to compilers that actually implement this
standard (or provide it as an extension to Fortran 90). You also need to
write a layer of Fortran 2003 proxy code between C and Fortran 90.
numpy.distutils also does not recognize .f03 or .f08 files.

When using numpy.distutils I prefer to compile the Fortran 90 code to a
static library (use add_library) and then compile the Cython generated C or
C++ code with add_extension. You must manually run Cython on the .pyx
files. This is portable in the sence that it works on all compilers
numpy.distutils supports, but you still need to take care of the
communication between C and Fortran 90. Take a look at how Fortran
libraries are compiled in SciPy:

https://github.com/scipy/scipy/blob/master/scipy/fftpack/setup.py


Sturla

[Sturla Molden]

numpy.distutils works well for Fortran 90, but you need to compile the
Fortran code as a static library. It has some limitations though. It does
not allow different optimisation levels, unless you compile the Fortran
into multiple libraries.
And if you use Fortran 90 you must also know the ABI of the compiler.

In SciPy we only interface C with Fortran 77 (or a subset of Fortran 90
which would be valid Fortran 77) and then proceed as follows:

A default Fortran integer is a C int

Never call Fortran functions from C, write a subroutine wrapper

We wrap the g77 ABI in Apple Accelerate Framework

Assume gfortran ABI

Use this for name mangling:
https://github.com/insertinterestingnamehere/scipy/blob/cyblas/scipy/linalg/fortran_defs.h

The compiler will get the name mangling defines form the lapack_opt
https://github.com/insertinterestingnamehere/scipy/blob/cyblas/scipy/linalg/setup.py

A bit PITA...

Sturla

[Ian Bell]

I've had remarkably good luck using the FortranCInterface in CMake to generate a mangling interface header - the CMake module runs a small test to determine the mangling scheme in use, so no manual intervention is required.  So far it has worked with both gfortran and intel compiler on both windows and Linux.  Also, using CMake means incremental builds, which is key when you have a little bit of C/C++ wrapping a lot of FORTRAN.  CMake can be used to generate the entire package, or just to generate a static library including the c/c++ wrapper which you then link into the cython module.

 

For interfacing with python, I compile a static library of Fortran sources, and a shim layer in C++ that calls directly into the Fortran functions.  And then the shim layer can be called conveniently from cython.  It's quite a lot of layers, but as Sturla says, interfacing anything with FORTRAN that isn't FORTRAN is a PITA.

 

I could probably put together a small example demonstrating how all the bits and bobs go together if there is interest.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hai Nguyen]

Hi,

I am interested in your example. I've been working on writing a Python
wrapper for a C++ program for data analysis (biomolecular simulation)
(with Cython). And I am thinking about wrapping some Fortran code for
my package too. It's great to learn from your (and others) cases.
https://github.com/pytraj/pytraj

Hai

[george...@noaa.gov]

Thanks. I did peruse numpy.distutils, but was left with the impression that it only handles Fortran 77. Will have to look again.
            
My thinking is that if I provide a Fortran library that can be of interest outside of my Python package, then I can bundle a C interface and build it with standard tools (make, configure, cmake, ...). But when the library comes from another provider, I need to create an interface. I can do it in C, but then, as you said, I have to deal with name mangling and other subtleties of compiler ABI. I thougt ISO C bindings module would take care of that. So my interest is to compile the interface file with minimal hassle. gcc can do it, if allowed by distutils.
A crazy idea: a script that would invoke C or Fortran compiler, depending on the file extension (if -c is one of the options). Possibly massaging other compiler options - the difficult part. That could work with the current, monkey-patched distutils.This was floated around years ago in a different content: https://mail.python.org/pipermail/distutils-sig/2002-August/002946.html

I see there are no easy solutions. If I have to write a Fortran extension, I'd probably opt for numpy.distutils.

George

[Sturla Molden]

On 20/02/15 19:30, george...@noaa.gov wrote:

> Thanks. I did peruse numpy.distutils, but was left with the impression
> that it only handles Fortran 77. Will have to look again.

numpy.distutils can handle Fortran 77 and Fortran 90/95.

It cannot handle Fortran 2003 or Fortran 2008.

> My thinking is that if I provide a Fortran library that can be of
> interest outside of my Python package, then I can bundle a C interface
> and build it with standard tools (make, configure, cmake, ...). But when
> the library comes from another provider, I need to create an interface.
> I can do it in C, but then, as you said, I have to deal with name
> mangling and other subtleties of compiler ABI. I thougt ISO C bindings
> module would take care of that.

ISO C bindings is Fortran 2003, which numpy.distutils cannot handle. :-(

I did open an issue for this a while ago, but nobody has worked on it yet.

https://github.com/numpy/numpy/issues/5486


Sturla