Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Progress migrating cffi and pycparser to libclang
89 views
Skip to first unread message
Etienne Robillard
Jan 4, 2018, 4:50:53 AM1/4/18
to
Hi,
I will be creating a repository for this:
https://bitbucket.org/tkadm30/cffi-libclang
The goal is to generate a AST object from a C header by preprocessing
with clang -E then compile the python bindings with CFFI...
ffi.cdef(open('uwsgi.h').read()) # <-- XXX need to modify internal
parsing code to use clang.cindex!
ffi.dlopen('/usr/local/lib/libuwsgi.so')
Right now the parsing of the C source file is done by pycparser, but i
would like to use the clang.cindex module.
What do you think?
Etienne
--
Etienne Robillard
tka...@yandex.com
https://www.isotopesoftware.ca/
I will be creating a repository for this:
https://bitbucket.org/tkadm30/cffi-libclang
The goal is to generate a AST object from a C header by preprocessing
with clang -E then compile the python bindings with CFFI...
ffi.cdef(open('uwsgi.h').read()) # <-- XXX need to modify internal
parsing code to use clang.cindex!
ffi.dlopen('/usr/local/lib/libuwsgi.so')
Right now the parsing of the C source file is done by pycparser, but i
would like to use the clang.cindex module.
What do you think?
Etienne
--
Etienne Robillard
tka...@yandex.com
https://www.isotopesoftware.ca/
Paul Moore
Jan 4, 2018, 6:42:05 AM1/4/18
to
On 4 January 2018 at 09:50, Etienne Robillard <tka...@yandex.com> wrote:
> Hi,
>
> I will be creating a repository for this:
> https://bitbucket.org/tkadm30/cffi-libclang
>
> The goal is to generate a AST object from a C header by preprocessing with
> clang -E then compile the python bindings with CFFI...
>
> ffi.cdef(open('uwsgi.h').read()) # <-- XXX need to modify internal parsing
> code to use clang.cindex!
>
> ffi.dlopen('/usr/local/lib/libuwsgi.so')
>
>
> Right now the parsing of the C source file is done by pycparser, but i would
> like to use the clang.cindex module.
>
> What do you think?
Presumably that will introduce a dependency on some clang module? You
> Hi,
>
> I will be creating a repository for this:
> https://bitbucket.org/tkadm30/cffi-libclang
>
> The goal is to generate a AST object from a C header by preprocessing with
> clang -E then compile the python bindings with CFFI...
>
> ffi.cdef(open('uwsgi.h').read()) # <-- XXX need to modify internal parsing
> code to use clang.cindex!
>
> ffi.dlopen('/usr/local/lib/libuwsgi.so')
>
>
> Right now the parsing of the C source file is done by pycparser, but i would
> like to use the clang.cindex module.
>
> What do you think?
mention clang.cindex - but the only clang package I can find on PyPI
says "OBSOLETE. LLVM-CLANG NOW PUBLISHES PYTHON PACKAGE.
JUST ADD THE OFFICIAL llvm-3.7 repo in your apt." but doesn't provide
binaries or explain how to install clang on, say, Windows (to pick an
example relevant to me :-)).
As a fork/extension for cffi, I have no particular opinion (I'm
unlikely to ever use it). But the advantage of pycparser is that it's
cross-platform and pure Python, so I doubt this will be acceptable for
inclusion into CFFI itself.
Paul
Etienne Robillard
Jan 4, 2018, 4:05:08 PM1/4/18
to
Hi Paul,
Le 2018-01-04 à 06:41, Paul Moore a écrit :
> Presumably that will introduce a dependency on some clang module? You
> mention clang.cindex - but the only clang package I can find on PyPI
> says "OBSOLETE. LLVM-CLANG NOW PUBLISHES PYTHON PACKAGE.
> JUST ADD THE OFFICIAL llvm-3.7 repo in your apt." but doesn't provide
> binaries or explain how to install clang on, say, Windows (to pick an
> example relevant to me :-)).
I'm targeting the Linux/x86 machine class.
On debian systems, it's pretty easy to install clang and the python
bindings with apt-get. :-)
C directives like #define and #include in the ffi.cdef() function.
The easiest solution is to migrate the internal parsing code to
libclang, a state-of-the art C/C++ compiler based on LLVM.
Best regards,
Le 2018-01-04 à 06:41, Paul Moore a écrit :
> Presumably that will introduce a dependency on some clang module? You
> mention clang.cindex - but the only clang package I can find on PyPI
> says "OBSOLETE. LLVM-CLANG NOW PUBLISHES PYTHON PACKAGE.
> JUST ADD THE OFFICIAL llvm-3.7 repo in your apt." but doesn't provide
> binaries or explain how to install clang on, say, Windows (to pick an
> example relevant to me :-)).
On debian systems, it's pretty easy to install clang and the python
bindings with apt-get. :-)
>
> As a fork/extension for cffi, I have no particular opinion (I'm
> unlikely to ever use it). But the advantage of pycparser is that it's
> cross-platform and pure Python, so I doubt this will be acceptable for
> inclusion into CFFI itself.
CFFI/pycparser definitely need to be patched to support parsing standard
> As a fork/extension for cffi, I have no particular opinion (I'm
> unlikely to ever use it). But the advantage of pycparser is that it's
> cross-platform and pure Python, so I doubt this will be acceptable for
> inclusion into CFFI itself.
C directives like #define and #include in the ffi.cdef() function.
The easiest solution is to migrate the internal parsing code to
libclang, a state-of-the art C/C++ compiler based on LLVM.
Best regards,
Paul Moore
Jan 4, 2018, 4:25:58 PM1/4/18
to
On 4 January 2018 at 21:02, Etienne Robillard <tka...@yandex.com> wrote:
>> As a fork/extension for cffi, I have no particular opinion (I'm
>> unlikely to ever use it). But the advantage of pycparser is that it's
>> cross-platform and pure Python, so I doubt this will be acceptable for
>> inclusion into CFFI itself.
>
> CFFI/pycparser definitely need to be patched to support parsing standard C
> directives like #define and #include in the ffi.cdef() function.
>
> The easiest solution is to migrate the internal parsing code to libclang, a
> state-of-the art C/C++ compiler based on LLVM.
I would strongly object to adding a dependency to cffi that couldn't
>> As a fork/extension for cffi, I have no particular opinion (I'm
>> unlikely to ever use it). But the advantage of pycparser is that it's
>> cross-platform and pure Python, so I doubt this will be acceptable for
>> inclusion into CFFI itself.
>
> CFFI/pycparser definitely need to be patched to support parsing standard C
> directives like #define and #include in the ffi.cdef() function.
>
> The easiest solution is to migrate the internal parsing code to libclang, a
> state-of-the art C/C++ compiler based on LLVM.
be automatically installed by pip as part of standard dependency
resolution (i.e., a PyPI hosted Python project with wheels available
for all common platforms - Linux, Mac OS and Windows). But ultimately
if you're proposing this as a change to cffi, you should be getting
the opinions of the cffi devs, not just asking on this list. (I notice
you have posted to the cffi mailing list, but haven't had any response
yet).
Paul
Etienne Robillard
Jan 5, 2018, 4:17:58 AM1/5/18
to
Forwarding this thread to the CFFI developers...
Re Paul: Thanks for your feedback.
My intended audience are developers who can use hg to fetch/build source
code without pip.
Best regards,
Etienne
-------- Message transféré --------
Sujet : Re: Progress migrating cffi and pycparser to libclang
Date : Thu, 4 Jan 2018 21:25:27 +0000
De : Paul Moore <p.f....@gmail.com>
Pour : Etienne Robillard <tka...@yandex.com>
Copie à : Python <pytho...@python.org>
Re Paul: Thanks for your feedback.
My intended audience are developers who can use hg to fetch/build source
code without pip.
Best regards,
Etienne
-------- Message transféré --------
Sujet : Re: Progress migrating cffi and pycparser to libclang
Date : Thu, 4 Jan 2018 21:25:27 +0000
De : Paul Moore <p.f....@gmail.com>
Pour : Etienne Robillard <tka...@yandex.com>
Copie à : Python <pytho...@python.org>
Armin Rigo
Jan 5, 2018, 10:35:49 AM1/5/18
to
Hi Etienne,
On 5 January 2018 at 10:15, Etienne Robillard <tka...@yandex.com> wrote:
> Forwarding this thread to the CFFI developers...
>
If you're asking whether we could add libclang as a dependency to CFFI, the
answer is no, sorry.
I feel that I've already explained exactly this to you several times in
private e-mails, so I'm writing down this fact here on the public
python-cffi mailing list. Please stop asking the same question.
For reference, here's my answer again. CFFI is meant to be used in a
certain way. I know from experience that a few people keep thinking about
it in a different way---sometimes for good reasons: it is tedious to wrap a
very large library even if it is easy to copy-paste-tweak individual
function definitions. So to automate the process these people usually want
to parse real .h files, using libclang or better pycparser integration or
something else; and that's cool. As I said there are already third-party
projects that try to do something similar. Usually it is never completely
general, but more focused on a particular style of .h files coming from a
particular large project. Maybe you'll run into similar issues and make
something that works in your case but not in others. Or maybe you'll
manage to make it general enough. In all cases this won't be included
inside CFFI, but could nevertheless be a successful separate project.
Please think about it as a project *above* CFFI, maybe something you'll
import and use inside a CFFI build script in order to provide content to
ffi.cdef().
A bientôt,
Armin.
On 5 January 2018 at 10:15, Etienne Robillard <tka...@yandex.com> wrote:
> Forwarding this thread to the CFFI developers...
>
answer is no, sorry.
I feel that I've already explained exactly this to you several times in
private e-mails, so I'm writing down this fact here on the public
python-cffi mailing list. Please stop asking the same question.
For reference, here's my answer again. CFFI is meant to be used in a
certain way. I know from experience that a few people keep thinking about
it in a different way---sometimes for good reasons: it is tedious to wrap a
very large library even if it is easy to copy-paste-tweak individual
function definitions. So to automate the process these people usually want
to parse real .h files, using libclang or better pycparser integration or
something else; and that's cool. As I said there are already third-party
projects that try to do something similar. Usually it is never completely
general, but more focused on a particular style of .h files coming from a
particular large project. Maybe you'll run into similar issues and make
something that works in your case but not in others. Or maybe you'll
manage to make it general enough. In all cases this won't be included
inside CFFI, but could nevertheless be a successful separate project.
Please think about it as a project *above* CFFI, maybe something you'll
import and use inside a CFFI build script in order to provide content to
ffi.cdef().
A bientôt,
Armin.
Eli Bendersky
Jan 6, 2018, 12:53:19 PM1/6/18
to
On Fri, Jan 5, 2018 at 1:15 AM, Etienne Robillard <tka...@yandex.com>
of pycparser. What are you trying to accomplish?
One of the issues with parsing "original" headers (with #defines and
#includes and all that jazz) is that you have to match the compilation
environment very precisely. When a compiler compiles a large project it
could set dozens of -Ds, as well as -Is pointing to directories with
additional headers. This is usually specified in Makefiles or something
similar. Clang and libclang deal with this by having the concept of a
compilation database which remembers for each file what the exact flags to
compile it are (https://clang.llvm.org/docs/JSONCompilationDatabase.html)
To make this *really* work for CFFI you'd have to take all of this into
account -- do you really want to deal with this on the CFFI level?
Eli
>
> -------- Message transféré --------
> Sujet : Re: Progress migrating cffi and pycparser to libclang
> Date : Thu, 4 Jan 2018 21:25:27 +0000
> De : Paul Moore <p.f....@gmail.com> <p.f....@gmail.com>
> Pour : Etienne Robillard <tka...@yandex.com> <tka...@yandex.com>
> Copie à : Python <pytho...@python.org> <pytho...@python.org>
> -- python-cffi: To unsubscribe from this group, send email to
> python-cffi...@googlegroups.com. For more options, visit this
> group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "python-cffi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to python-cffi...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
wrote:
> Forwarding this thread to the CFFI developers...
>
> Forwarding this thread to the CFFI developers...
>
> Re Paul: Thanks for your feedback.
>
> My intended audience are developers who can use hg to fetch/build source
> code without pip.
>
> Best regards,
>
> Etienne
>
I'd like to understand the requirements here better, from the point of view
>
> My intended audience are developers who can use hg to fetch/build source
> code without pip.
>
> Best regards,
>
> Etienne
>
of pycparser. What are you trying to accomplish?
One of the issues with parsing "original" headers (with #defines and
#includes and all that jazz) is that you have to match the compilation
environment very precisely. When a compiler compiles a large project it
could set dozens of -Ds, as well as -Is pointing to directories with
additional headers. This is usually specified in Makefiles or something
similar. Clang and libclang deal with this by having the concept of a
compilation database which remembers for each file what the exact flags to
compile it are (https://clang.llvm.org/docs/JSONCompilationDatabase.html)
To make this *really* work for CFFI you'd have to take all of this into
account -- do you really want to deal with this on the CFFI level?
Eli
>
> -------- Message transféré --------
> Sujet : Re: Progress migrating cffi and pycparser to libclang
> Date : Thu, 4 Jan 2018 21:25:27 +0000
> Pour : Etienne Robillard <tka...@yandex.com> <tka...@yandex.com>
> Copie à : Python <pytho...@python.org> <pytho...@python.org>
>
> On 4 January 2018 at 21:02, Etienne Robillard <tka...@yandex.com> <tka...@yandex.com> wrote:
> >> As a fork/extension for cffi, I have no particular opinion (I'm
> >> unlikely to ever use it). But the advantage of pycparser is that it's
> >> cross-platform and pure Python, so I doubt this will be acceptable for
> >> inclusion into CFFI itself.
> >
> > CFFI/pycparser definitely need to be patched to support parsing standard C
> > directives like #define and #include in the ffi.cdef() function.
> >
> > The easiest solution is to migrate the internal parsing code to libclang, a
> > state-of-the art C/C++ compiler based on LLVM.
>
> I would strongly object to adding a dependency to cffi that couldn't
> be automatically installed by pip as part of standard dependency
> resolution (i.e., a PyPI hosted Python project with wheels available
> for all common platforms - Linux, Mac OS and Windows). But ultimately
> if you're proposing this as a change to cffi, you should be getting
> the opinions of the cffi devs, not just asking on this list. (I notice
> you have posted to the cffi mailing list, but haven't had any response
> yet).
>
> Paul
>
>
> --
> On 4 January 2018 at 21:02, Etienne Robillard <tka...@yandex.com> <tka...@yandex.com> wrote:
> >> As a fork/extension for cffi, I have no particular opinion (I'm
> >> unlikely to ever use it). But the advantage of pycparser is that it's
> >> cross-platform and pure Python, so I doubt this will be acceptable for
> >> inclusion into CFFI itself.
> >
> > CFFI/pycparser definitely need to be patched to support parsing standard C
> > directives like #define and #include in the ffi.cdef() function.
> >
> > The easiest solution is to migrate the internal parsing code to libclang, a
> > state-of-the art C/C++ compiler based on LLVM.
>
> I would strongly object to adding a dependency to cffi that couldn't
> be automatically installed by pip as part of standard dependency
> resolution (i.e., a PyPI hosted Python project with wheels available
> for all common platforms - Linux, Mac OS and Windows). But ultimately
> if you're proposing this as a change to cffi, you should be getting
> the opinions of the cffi devs, not just asking on this list. (I notice
> you have posted to the cffi mailing list, but haven't had any response
> yet).
>
> Paul
>
>
> -- python-cffi: To unsubscribe from this group, send email to
> python-cffi...@googlegroups.com. For more options, visit this
> group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "python-cffi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to python-cffi...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
0 new messages