Getting a file pointer from a python file object

[Chris Barker - NOAA Federal]

HI folks,

There is some discussion on mpl-dev about whether/how to use Cython as
a core technology in Matplotlib.

As part of this, someone posted a Cython wrapper of the libpng lib, to
compare with a hand-coded C wrapper. However, to my eyes at least,
Cython was used at a lower level than need be (making calls to the
CPython API that Cython may take care of for you), so I'm trying to
simplify it a bit.

One issue:

How to get a pointer to pass to a C function expecting a stdio.FILE*.

He's got some python-version dependent code that seems to work, but I
suspect that Cython may well know how to do this built in. So, is
there a slightly easier way to write:

cdef stdio.FILE* fp

if PyFile_CheckExact(file_obj):
fp = PyFile_AsFile(file_obj)
the_c_function(..., fp, ...)


granted, that's not that hard, but particularly as it's different for
Py3, it would be nice to simply do:

cdef PyFile file_obj
the_c_function(..., file_obj, ...)

and have Cython do the checking, etc for me.

I couldn't find anything in the Wiki or googling, but this can't be a rare need.

-thanks,
-Chris



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Dag Sverre Seljebotn]

On 12/03/2012 10:21 PM, Chris Barker - NOAA Federal wrote:
> HI folks,
>
> There is some discussion on mpl-dev about whether/how to use Cython as
> a core technology in Matplotlib.
>
> As part of this, someone posted a Cython wrapper of the libpng lib, to
> compare with a hand-coded C wrapper. However, to my eyes at least,
> Cython was used at a lower level than need be (making calls to the
> CPython API that Cython may take care of for you), so I'm trying to
> simplify it a bit.
>
> One issue:
>
> How to get a pointer to pass to a C function expecting a stdio.FILE*.
>
> He's got some python-version dependent code that seems to work, but I
> suspect that Cython may well know how to do this built in. So, is
> there a slightly easier way to write:
>
> cdef stdio.FILE* fp
>
> if PyFile_CheckExact(file_obj):
> fp = PyFile_AsFile(file_obj)
> the_c_function(..., fp, ...)
>
>
> granted, that's not that hard, but particularly as it's different for
> Py3, it would be nice to simply do:
>
> cdef PyFile file_obj
> the_c_function(..., file_obj, ...)
>
> and have Cython do the checking, etc for me.

I'm pretty sure this is not implemented, you'll need to go the route you
say above.

I think this can cause problems on Windows if your Python is compiled
with Visual C and the Cython module with mingw (i.e. the FILE comes from
two different C runtime libraries).

Dag Sverre

[Lars Buitinck]

2012/12/3 Chris Barker - NOAA Federal <chris....@noaa.gov>:

> How to get a pointer to pass to a C function expecting a stdio.FILE*.

[snip]

> He's got some python-version dependent code that seems to work, but I
> suspect that Cython may well know how to do this built in. So, is
> there a slightly easier way to write:
>
> cdef stdio.FILE* fp
>
> if PyFile_CheckExact(file_obj):
> fp = PyFile_AsFile(file_obj)
> the_c_function(..., fp, ...)
>
> granted, that's not that hard, but particularly as it's different for
> Py3, it would be nice to simply do:

Speaking of Py3, wasn't the I/O completely redesigned to decouple it
from C stdio? PyFile_AsFile seems to be gone in the 3.2 C API. That
means your conversion from Python file to FILE* would have to get the
fileno from the file and fdopen it. Personally, I find that a bit much
to expect from an implicit conversion.

--
Lars Buitinck
Scientific programmer, ILPS
University of Amsterdam

[Bradley Froehle]

For Python 3, you'll probably want code similar to (parts of) `get_file` from `Python/import.c`.  Here's the version from Python 3.2:

static FILE *

get_file(char *pathname, PyObject *fob, char *mode)

{

    FILE *fp;

    if (mode[0] == 'U')

        mode = "r" PY_STDIOTEXTMODE;

    if (fob == NULL) {

        fp = fopen(pathname, mode);

    }

    else {

        int fd = PyObject_AsFileDescriptor(fob);

        if (fd == -1)

            return NULL;

        if (!_PyVerify_fd(fd))

            goto error;

        /* the FILE struct gets a new fd, so that it can be closed

         * independently of the file descriptor given

         */

        fd = dup(fd);

        if (fd == -1)

            goto error;

        fp = fdopen(fd, mode);

    }

    if (fp)

        return fp;

error:

    PyErr_SetFromErrno(PyExc_IOError);

    return NULL;

}

Are you sure that you even want to work with FILE* streams anyway?  Maybe it'd be easier to write the interface at a higher level instead?

-Brad

[Pauli Virtanen]

04.12.2012 01:06, Bradley Froehle kirjoitti:
> For Python 3, you'll probably want code similar to (parts of) `get_file`
> from `Python/import.c`. Here's the version from Python 3.2:
>
> static FILE *
> get_file(char *pathname, PyObject *fob, char *mode)
> {

[clip]

> int fd = PyObject_AsFileDescriptor(fob);
> if (fd == -1)
> return NULL;
> if (!_PyVerify_fd(fd))
> goto error;
> /* the FILE struct gets a new fd, so that it can be closed
> * independently of the file descriptor given
> */
> fd = dup(fd);
> if (fd == -1)
> goto error;
> fp = fdopen(fd, mode);

[clip]

You probably need also to call the Python-side flush() before the
AsFileDescriptor, and Python-side seek() after fclose(), to deal with
buffering. We do essentially this in Numpy for Python 3.

I recall, however, seeing some complaints that even this wasn't enough
in some cases. However, I'm unable to find where this was, and I'm not
able to reproduce any errors myself... (If someone knows cases that
fail, please file bug reports!)

So I'd say that doing anything with FILE* and Python files is not good,
especially on Python 3. Unfortunately, Python has no C-API for this, so
I don't know how parsers etc. that read one character at a time are
supposed to work effectively (or maybe the Python call overhead is small
enough not to matter).

--
Pauli Virtanen

[Stefan Behnel]

Pauli Virtanen, 04.12.2012 00:36:

> I'd say that doing anything with FILE* and Python files is not good,
> especially on Python 3.

+1, my usual way to deal with this is to set

#define PyFile_AsFile(obj) (NULL)

in Python 3 and then just let it take the usual fall back path for
file-like objects.

Stefan