Issue 39 in f2py: STOP command kills entire python instance

[f2...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 39 by samg...@googlemail.com: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

What steps will reproduce the problem?
1. Run a Fortran program wrapped using f2py in Python; any STOP command in
Fortran will kill the entire Python instance


What version of the product are you using? On what operating system?
Version 2

Please provide any additional information below.

I am trying to wrap a large, existing Fortran program
(http://www.itp.uzh.ch/~teyssier/ramses/RAMSES.html) with multiple files
and modules in f2py and call it from a Python package. The Fortran program
makes liberal use of STOP commands in both upper and lower case and is
under active development, so removing the STOP commands manually will be a
pain. I would like to keep the Python instance active after the Fortran
program ends, rather than having the Fortran program kill the Python script
upon exit. Putting the f2py call in a separate thread doesn't appear to
help; a STOP command in a spawned thread will kill the Python instance that
spawned it. If you can suggest work-arounds that don't involve calling the
program from the command line, let me know!

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[f2...@googlecode.com]

Comment #1 on issue 39 by pearu.peterson: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

Note that in Fortran STOP is a statement of the language that executes
system level function. As a result, it is impossible for f2py catch the
results of STOP execution.

I would write a script that walks through the source code of the Fortran
program and replace all STOP statements with something more appropriate,
e.g. a special function call that records the STOP statement argument and
makes a longjmp call to the wrapper function, all necessary modifications
to the generated wrapper function, eg defining longjmp points, etc. are
supported by f2py signature file specifications.

In principle, fparser could be used for that but it could be overkill as a
simple python script should be sufficient for replacing STOP statements
with something other.

[f2...@googlecode.com]

Comment #2 on issue 39 by samg...@googlemail.com: STOP command kills entire
python instance
http://code.google.com/p/f2py/issues/detail?id=39

Ah, I see, that makes sense. Thinking about it, it shouldn't be too hard to
add a simple script before compilation that replaces all the STOP commands.
Thanks for your advice - I'll let you know if I have any other problems
with this. My only other idea was to spawn a separate Python instance with
the Fortran code in it so that if it goes down the main thread stays alive,
but I can't think of a clean way to do this short of calling Python with
os.system() or similar.

[f2...@googlecode.com]

Comment #3 on issue 39 by samg...@googlemail.com: STOP command kills entire
python instance
http://code.google.com/p/f2py/issues/detail?id=39

I wrote some code to skim the Fortran and replace the STOP calls with
something that references longjmp (and isetjmp), and this works, but
apparently gfortran can't find longjmp/isetjmp ("undefined reference to
`longjmp_' / collect2: error: ld returned 1 exit status"). It's possible
it's been removed: http://gcc.gnu.org/ml/fortran/2010-11/msg00009.html -
I'll keep poking, but it seems quite difficult to find meaningful
documentation on longjmp in gfortran in any case.

This stackoverflow seems to propose a working C++ wrapper that calls
longjmp in C++, meaning you don't need to modify the Fortran at all:
http://stackoverflow.com/questions/19596375/intercepting-fortran-stop-from-c
(no idea how stable or portable this fix is, of course). I'd rather not go
to the trouble of wrapping Fortran in C++ and then wrapping that in Python,
of course, but if it works...

[f2...@googlecode.com]

Comment #4 on issue 39 by pearu.peterson: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

1) I don't think that you need to deal with longjmp in Fortran. You can
replace Fortran STOP statement with a call to C function that will deal
with the longjmp stuff.

2) Finding the exit function as described in
intercepting-fortran-stop-from-c is equivalent to replacing STOP statement.
You would not need to change Fortran code at the expense of loosing
portability (is portability relevant in your case?). Otherwise, longjmp
stuff has to be implemented anyway.

[f2...@googlecode.com]

Comment #5 on issue 39 by samg...@googlemail.com: STOP command kills entire
python instance
http://code.google.com/p/f2py/issues/detail?id=39

OK, it works, thanks! It's a bit convoluted but seems to work. For
reference/posterity, my solution was the following (I've changed around
some of the names and simplified my code a bit, but it should work as
written):
1) Replace every "STOP" call with "call long_jump" using a Python script (I
can post this up if you like; I jumped through a few hoops to get rid of
stuff like comments/strings including the characters "stop" or functions
called "stop_stuff", etc.
2) Write these files:
---
clean_stop.c:
#include <stdio.h>
#include <setjmp.h>

static jmp_buf buf;

void run_fortran_();

void long_jump_()
{
longjmp(buf,1);
}

int set_jump_()
{
if(!setjmp(buf))
{
run_fortran_();
}
}
---
myfortran.f90:
program myfortran
! This is optional; allows you to run it as a command-line program too
call set_jump
end program myfortran

subroutine run
! This is what you call in Python
call set_jump
end subroutine run

subroutine run_fortran
! FORTAN CODE GOES HERE

end subroutine run_fortran
---
3) Compile both as object files in gfortran and gcc
4) Run "f2py -m myfortran -h myfortran.pyf myfortran.f90" (you can delete
the interface call to run_fortran if you like, might make things cleaner
for the users)
5) Run "f2py -lgfortran" -c myfortran.pyf *.o
6) In Python, call "import myfortran; myfortran.run()"
(Note: the output of this program is as files, I assume returning variables
should be a simple thing to hack into the solution above)

I suspect it might have been simpler to embed the C code in Python as the
entry point, but never mind.

[f2...@googlecode.com]

Comment #6 on issue 39 by pearu.peterson: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

Thanks for your feedback!

After a thought, I found perhaps an easier solution. It is described in

https://code.google.com/p/f2py/wiki/FAQ2ed

[f2...@googlecode.com]

Comment #7 on issue 39 by samg...@googlemail.com: STOP command kills entire
python instance
http://code.google.com/p/f2py/issues/detail?id=39

Ah, cunning. I might try your solution #1 if I have to revisit the problem
for whatever reason; it seems a bit less invasive than the solution I gave
above as that way I don't need to mess about with C or modify the Makefile.
Thanks again for your help!

[f2...@googlecode.com]

Updates:
Status: Done

Comment #8 on issue 39 by pearu.peterson: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

(No comment was entered for this change.)

[f2...@googlecode.com]

Comment #9 on issue 39 by pearu.peterson: STOP command kills entire python
instance
http://code.google.com/p/f2py/issues/detail?id=39

Note that I have changed the FAQ a little: actually there is no reason to
remove STOP statements, they can be left in provided that they precede a
call to Python function that raises an exception. In this way the Fortran
sources will be still usable by Fortran programs that need to provide the
f2pystop function that does nothing.

[f2...@googlecode.com]

Comment #10 on issue 39 by samg...@googlemail.com: STOP command kills

entire python instance
http://code.google.com/p/f2py/issues/detail?id=39

Sorry for dragging this back up again, but one thing I noticed was the
module Multiprocessing:
http://docs.python.org/2/library/multiprocessing.html - I haven't had a
chance to confirm, but I believe this spawns multiple Python instances, so
in theory if the stop command is called it should only affect its
containing process (I found this as I didn't manage to get the !f2py
threadsafe command to work).