fwrap prospects: Version controlled wrappers
Dag Sverre Seljebotn
On 11/10/2010 08:03 PM, Ondrej Certik wrote:
> On Wed, Nov 10, 2010 at 12:40 AM, Dag Sverre Seljebotn
> <da...@student.matnat.uio.no> wrote:
>
>> On 11/10/2010 09:08 AM, Ondrej Certik wrote:
>>
>>>
>>> I am interested too. I ended up using plain Cython without fwrap at
>>> all, for two reasons:
>>>
>>> * no support for f90 modules
>>> * better control of the build process, and less files to worry about
>>>
>>>
>>>
>> The build part is much better now in trunk (Kurt dropped distutils and
>> switched to waf, which tends to help). I'll make it easier to fit into
>> custom build systems this autumn. F90 modules remain the main challenge for
>> real usability.3) Build and distribution workshop
>>
> Right. I forgot to add another important thing --- if you look at the
> actual wrappers here:
>
> https://github.com/hpfem/phaml/blob/master/src/phaml/simple_wrapper.pyx
>
> I like that I design my own wrappers, just like if I was wrapping some
> C code. While with f2py and also fwrap, I don't have the option --- as
> fwrap decides itself how to wrap the functionality. If I want to
> create some more Python like interface (like my cdef class above), I
> end up with two layers.
>
> So to me, fwrap/f2py are on the same level as some automatic tool,
> that parses C code and generates Cython wrappers. This took me some
> time to realize. Which is not necessarily a bad thing, but I never use
> any tools to generate Cython when wrapping C either (well, except when
> I was wrapping opengl), I prefer to write it by hand. And the same
> with Fortran. (In fortran, there is one more little step involved,
> that I need to use the iso_c_module.)
>
If you can use iso_c_module from Fortran without too much hassle, I say
go for it. The goal of fwrap as I understand it is to be usable in
situations where using iso_c_module properly would be too longwinded.
As for workflow, exactly this is something I'll fix for the SciPy
refactoring project. I intend to add an additional mode where the .pyx
files are not temporary build artifacts, but managed by version control.
When something changes Fortran-side and one reruns fwrap, the changes
made to the wrapper can then be merged with git.
This is the interface I envision now (no code yet):
$ fwrap init --versioned # creates fwrap.cfg
$ fwrap add module1 myfort.f90 --exclude=foo
$ fwrap status
Python modules:
module1: myfort.90
All subroutines/functions, except "foo"
fwrap generated files are under version control (git)
$ fwrap update # Creates wrappers, commits, and records commit ID to
fwrap.cfg (and commits again)
Now, one can modify the generated module1.pyx as normal. When myfort.f90
changes (or fwrap upgraded with new features), one re-runs
$ fwrap update
Now, the fwrap command starts at the point where the previous "fwrap
update" was run, and creates a temporary git branch (say, "_fwrap"),
creating the new wrappers there in order to generate a changeset of
"what changed due to changed source/updated fwrap". Then the new commit
ID is saved. Then, the user must merge with _fwrap (and delete it).
Potentially, when fwrap is upgraded to use, e.g., a different naming
scheme, it could also set up a branch with the manually changed wrappers
run through some regexp replacements.
There's a few precautions to make (always order functions in the same
way etc.), but beyond that it seems to me that things should work OK...
Notice that I essentially reinvented the concept of a git branch there,
with storing a changeset ID for each step, but I think keeping track of
the branch manually this way has many advantages to using a dedicated
git branch.
And yes, I'll hard-code (optional) support for git at the moment.
Thoughts welcome.
Dag Sverre
Ondrej Certik
Can you give me some example? Maybe lapack? Things where you don't
want to write all the wrappers by hand?
The iso_c_module will eventually be supported by all compilers, I
think it already is, but I know that for example the sage fortran on
Mac doesn't support it (because it's gfortran 4.2, and one needs
gfortran 4.3).
Why can't the .pyx just be committed, and when you do "make fwrap" or
something, it would simply overwrite them, and then you do "git diff"
and decide if you want to keep the changes, or not.
I use a similar approach for Cython -- we were having constant
problems in our project, that the Ubuntu version of Cython was too old
and didn't compile our .pyx correctly, so I simply committed the
generated .c files into git and by default I don't run Cython. On
"make cython", the generated files are regenerated and I decide
manually if I want to commit it or not.
Ondrej
Dag Sverre Seljebotn
a) For assumed-shape arrays, dimension(:, :), it gets tedious to pass
all the shape arguments by hand.
b) Derived types/"structs" etc.
Not saying fwrap handles everything gracefully now, just describing the
goal.
Also, I'd say the average Fortran programmer don't know their way around
in C as well as we do. Consider somebody who only wants Fortran who
wants to dive into Python to do some plotting. That's a target
demographic as well.
Put in your shoes, I may well have opted for iso_c_binding myself. fwrap
is not for everyone.
> The iso_c_module will eventually be supported by all compilers, I
> think it already is, but I know that for example the sage fortran on
> Mac doesn't support it (because it's gfortran 4.2, and one needs
> gfortran 4.3).
>
fwrap just auto-generates all the iso_c_binding code, that's what I meant.
Hmm. I'll need to reconsider this. I was thinking that when the changes
between fwrap generated wrappers are recorded seperately, it would cause
more graceful merges. But perhaps not, and perhaps the complexity is
better skipped. I'm probably demonstrating my ignorance of the inner
workings of git here.
Dag Sverre
Dag Sverre Seljebotn
Sorry, I wasn't paying enough attention. I do this in order to invoke
the 3-way merging algorithm at all. Example:
fwrap outputs:
#
# Wrapper generated by fwrap by Kurt W. Smith et al.
#
We modify it to
#
# Wrapper around our fantastic library
# Copyright 2010 Dag Sverre Seljebotn, Ondrej Certik
#
# (generated by fwrap; Kurt W. Smith et al.)
#
When updating the wrapper using your method, the diff will contain a
reversal of this change. With my proposed method, this shouldn't clutter
up the diff at all, as long as the comment generated by fwrap is not
changed.
Dag Sverre
Ondrej Certik
Ok, I get it. I think the generated files should never be modified by
hand. I think the complexity will just rise up.
Ondrej
Dag Sverre Seljebotn
Well, I can see that the idea is a bit risky. For my particular project
(moving SciPy from f2py to Cython, essentially), I am rather confident
that this workflow should work OK, and represents the easiest way to
have flexible wrappers that are easy to maintain. But the manual changes
are rather limited (it will be all the manual C code that is present in
the .pyf files...example from scipy.special.specfun.clqmn: For some odd
reason, Fortran function takes real and imaginary part as seperate
arguments; Python-side we want a single complex argument.)
When more extensive changes to the wrapper is needed I am less sure.
This is mostly proposed as the alternative to all the various wrapper
customization options in the .pyf files.
As you say, manual wrapping may work better for you. If you have a
better idea for fwrap workflow that lands us somewhere in-between fully
automatic and fully manual, I'd love to hear it. The problem can be
stated: One has lots of functions to wrap. Most functions can be dealt
with automatically, but one wants to change the interface of a few
(perhaps some additional error handling, etc.) That is,
"computer-assisted wrapping".
Other possible solutions (that I can think of):
- Input configuration domain specific language. This is what f2py
does: You run it once to generate a .pyf, then modify the .pyf, with
options to insert custom C code in-line which is then dropped at the
appropriate place in the wrapper. Somehow I'd rather modify the wrapper
directly than learn how to hack things in using a pyf-like language.
- Always have another level of manually written wrappers on top of the
automatically generated ones.
Dag Sverre