Support for #include in cdef
1,962 views
Skip to first unread message
Sarvi Shanmugham
Aug 23, 2012, 8:29:47 PM8/23/12
to pytho...@googlegroups.com
Is there a technical reason for not supporting #include in cdef()?
Or
Is it just a matter of it needing more work.
Is there a plan to support it in the future?
It would be convenient to simply do
ffi=FFI()
ffi.cdef("""
#incude "mylibrary.h"
""")
mylibrary=ffi.dlopen("mylibrary.so")
instead of having to cut and paste content from the include files.
Thx,
Sarvi
Sarvi Shanmugham
Aug 24, 2012, 8:22:31 PM8/24/12
to pytho...@googlegroups.com
I read a bit more, took mylibrary.h and put it through the gcc preprocessor as follows to expand the #includes
gcc -E mylibrary.h > mylibrary.h.out
mylibrary is simply:
#include <stdio.h>
I thought this might be able to go into a cdef() as is.
Didn't Work. Got a traceback
>>> ff.cdef(x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/cffi-0.3-py2.7-linux-x86_64.egg/cffi/api.py", line 79, in cdef
self._parser.parse(csource, override=override)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/cffi-0.3-py2.7-linux-x86_64.egg/cffi/cparser.py", line 71, in parse
self._internal_parse(csource)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/cffi-0.3-py2.7-linux-x86_64.egg/cffi/cparser.py", line 76, in _internal_parse
ast, macros = self._parse(csource)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/cffi-0.3-py2.7-linux-x86_64.egg/cffi/cparser.py", line 64, in _parse
ast = _get_parser().parse(csource)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/pycparser-2.07-py2.7.egg/pycparser/c_parser.py", line 127, in parse
return self.cparser.parse(text, lexer=self.clex, debug=debuglevel)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/ply-3.4-py2.7.egg/ply/yacc.py", line 265, in parse
return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/ply-3.4-py2.7.egg/ply/yacc.py", line 971, in parseopt_notrack
p.callable(pslice)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/pycparser-2.07-py2.7.egg/pycparser/c_parser.py", line 462, in p_decl_body
self._parse_error('Multiple type specifiers with a type tag', coord)
File "/ws/sarvi-sjc/mytools/lib/python2.7/site-packages/pycparser-2.07-py2.7.egg/pycparser/plyparser.py", line 54, in _parse_error
raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: ?: Multiple type specifiers with a type tag
Sarvi
Armin Rigo
Aug 25, 2012, 4:03:13 AM8/25/12
to pytho...@googlegroups.com
Hi Sarvi,
On Sat, Aug 25, 2012 at 2:22 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> I read a bit more, took mylibrary.h and put it through the gcc preprocessor
beginning, we considered doing this, which would be nice because you'd
only need to write the "#include <foo.h>".
However, there are a number of problems making the approach fragile.
For example, foo.h might use unsupported C extensions (for example
GCC's). This is true even if foo.h doesn't explicitly *use* them but
just inherits them from #include'ing a standard header. For example,
"#include <stdint.h>" might expand to some type declarations, say
"long long long", which were added recently to GCC. So even though
foo.h doesn't actually use "long long long" and compiles fine on old
GCCs, doing the macro-expansion will try to declare a "long long long"
type and crash, depending on whether the installed stdint.h is from a
recent GCC or not. By contrast, the current approach should work more
robustly: on any system where you'd successfully compile a piece of C
code using the header, it should work in CFFI too.
Another issue is what to do with complex macro declarations. For
example there is no way we can just #include <ncurses.h> and be done,
because it has macros like this:
#define getyx(win,y,x) <some code which writes to both x and y>
which need a bit of custom code to be supported.
This is why we took the current approach. Note that some people have
some early success using large libraries with CFFI by writing large
headers into a local .h file and just doing
ffi.cdef(open('local.h').read()). The point is still that this local
header is not exactly the same as the installed one. You need to go
once through the original .h file manually and copy and fix things,
e.g. replacing too-precise declarations with "...", removing parts,
etc.
A bientôt,
Armin.
On Sat, Aug 25, 2012 at 2:22 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> I read a bit more, took mylibrary.h and put it through the gcc preprocessor
> Didn't Work. Got a traceback
This is basically why it is not the supported approach. At the
beginning, we considered doing this, which would be nice because you'd
only need to write the "#include <foo.h>".
However, there are a number of problems making the approach fragile.
For example, foo.h might use unsupported C extensions (for example
GCC's). This is true even if foo.h doesn't explicitly *use* them but
just inherits them from #include'ing a standard header. For example,
"#include <stdint.h>" might expand to some type declarations, say
"long long long", which were added recently to GCC. So even though
foo.h doesn't actually use "long long long" and compiles fine on old
GCCs, doing the macro-expansion will try to declare a "long long long"
type and crash, depending on whether the installed stdint.h is from a
recent GCC or not. By contrast, the current approach should work more
robustly: on any system where you'd successfully compile a piece of C
code using the header, it should work in CFFI too.
Another issue is what to do with complex macro declarations. For
example there is no way we can just #include <ncurses.h> and be done,
because it has macros like this:
#define getyx(win,y,x) <some code which writes to both x and y>
which need a bit of custom code to be supported.
This is why we took the current approach. Note that some people have
some early success using large libraries with CFFI by writing large
headers into a local .h file and just doing
ffi.cdef(open('local.h').read()). The point is still that this local
header is not exactly the same as the installed one. You need to go
once through the original .h file manually and copy and fix things,
e.g. replacing too-precise declarations with "...", removing parts,
etc.
A bientôt,
Armin.
Sarvi Shanmugham
Aug 29, 2012, 6:16:22 PM8/29/12
to pytho...@googlegroups.com, ar...@tunes.org
Makes sense.
May be we should allow #include and ignore them or better provide a callback function for #include directives for expanding as needed.
That would allow us to still do the following,
ffi.cdef(open("mylib.h").read())
and if it requires any specific definitions from the #includes within "mylib.h" we still have the option of selectively defining them or including the whole header file.
Let me think of other ways of doing this.
Thanks,
Sarvi
Andreas Kloeckner
Aug 29, 2012, 9:04:31 PM8/29/12
to Armin Rigo, pytho...@googlegroups.com
Hi Armin,
Armin Rigo <ar...@tunes.org> writes:
> On Sat, Aug 25, 2012 at 2:22 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
>> I read a bit more, took mylibrary.h and put it through the gcc preprocessor
>> Didn't Work. Got a traceback
>
> This is basically why it is not the supported approach. At the
> beginning, we considered doing this, which would be nice because you'd
> only need to write the "#include <foo.h>".
>
> However, there are a number of problems making the approach fragile.
> For example, foo.h might use unsupported C extensions (for example
> GCC's).
just as an FYI--I've written an add-on to pycparser that lets it digest
many current GNU extensions.
https://github.com/inducer/pycparserext
Note that I'm not suggesting the 'digest preprocessed source'
route. That's unsustainable for precisely the reason you mention...
Andreas
Armin Rigo <ar...@tunes.org> writes:
> On Sat, Aug 25, 2012 at 2:22 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
>> I read a bit more, took mylibrary.h and put it through the gcc preprocessor
>> Didn't Work. Got a traceback
>
> This is basically why it is not the supported approach. At the
> beginning, we considered doing this, which would be nice because you'd
> only need to write the "#include <foo.h>".
>
> However, there are a number of problems making the approach fragile.
> For example, foo.h might use unsupported C extensions (for example
> GCC's).
many current GNU extensions.
https://github.com/inducer/pycparserext
Note that I'm not suggesting the 'digest preprocessed source'
route. That's unsustainable for precisely the reason you mention...
Andreas
Armin Rigo
Aug 30, 2012, 3:31:27 AM8/30/12
to pytho...@googlegroups.com
Hi Sarvi,
On Thu, Aug 30, 2012 at 12:16 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> May be we should allow #include and ignore them or better provide a callback
> function for #include directives for expanding as needed.
>
> That would allow us to still do the following,
> ffi.cdef(open("mylib.h").read())
You can already do it more explicitly than with callbacks. Just
preprocess the source yourself before passing it to ffi.cdef(), e.g.:
lines = open("mylib.h").readlines()
for line in lines:
if line.startswith('#include '):
xxx
ffi.cdef(''.join(lines))
You can also extract a part of the file only, or anything; it's up to
you. In general I'm still thinking that sending whole unmodified
header files to cdef() is slightly fragile, but I see the point for
convenience.
A bientôt,
Armin.
On Thu, Aug 30, 2012 at 12:16 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> May be we should allow #include and ignore them or better provide a callback
> function for #include directives for expanding as needed.
>
> That would allow us to still do the following,
> ffi.cdef(open("mylib.h").read())
preprocess the source yourself before passing it to ffi.cdef(), e.g.:
lines = open("mylib.h").readlines()
for line in lines:
if line.startswith('#include '):
xxx
ffi.cdef(''.join(lines))
You can also extract a part of the file only, or anything; it's up to
you. In general I'm still thinking that sending whole unmodified
header files to cdef() is slightly fragile, but I see the point for
convenience.
A bientôt,
Armin.
Reply all
Reply to author
Forward
0 new messages