Extracting AST from Cython files.

[Fabio Zadrozny]

Hi All,

I'd like to be able to convert Cython to an AST to extract information for outline, code-completion, etc.

Right now I have a pretty ad-hoc manual parser just to extract some info (which is in a very poor shape right now) as it doesn't seem like Cython has an actual grammar (i.e.: there's https://raw.githubusercontent.com/cython/cython/084a25f55d0b4cf8b4c3cd496ec57bb3e2f57f71/Cython/Parser/Grammar but from comments it doesn't seem to be used).

So, I was wondering : does Cython have some supported API where I can provide it a file and it can give me back syntax errors and an AST?

Thanks,

Fabio

[Robert Bradshaw]

No, unfortunately, it doesn't. (You could try of course to invoke the
same utilities that cythonize invokes, but those are considered
internal details more than a public API and may change). The grammar
file was a first, small, attempt in this direction (though, as you
mention, still unused).

[Fabio Zadrozny]

Thanks for the reply...

 I'm not really fond of the idea of using internal utilities or private APIs, so, I've created a feature request: https://github.com/cython/cython/issues/2592 ;)

Thanks,

Fabio

[walt3k]

Fabio,
I saw your comment here 
https://github.com/cython/cython/issues/1628

and was wondering if I could get your advice with something I'd like to do.

I have used the ast module in python to write custom syntax checks on user programs.  They are pretty simple; for example, for a program to work in our infrastructure the user must put certain decorators in a certain order otherwise things go badly.  Since we load the python interpreter and subsequent user program in C++ code, we run this checker prior to invoking the user program which saves a lot of debug and headaches.

I'd like to be able to do the same thing with .pyx cython files.  I am wondering if you think the example you shared
```
from Cython.Compiler.TreeFragment import parse_from_strings
mod = parse_from_strings(name, source)
```
would allow me to do the same simple checks like I can with the ast module where I would somehow process mod.  

If so, do you know if there any documentation like the ast module has on how to visit nodes and such ?  If not, I am wondering if you'd be willing to share a bit of your wisdom and or/examples on how to do the equivalent basic operations the python ast module provides.  I'm not sure how to get started and could really use any info you could share.

Thx a bunch, Walt.

[da-woods]

I'm not Fabio, but:

On 24/04/2022 23:07, walt3k wrote:
> If so, do you know if there any documentation like the ast module
> has on how to visit nodes and such ?  If not, I am wondering if you'd
> be willing to share a bit of your wisdom and or/examples on how to do
> the equivalent basic operations the python ast module provides.  I'm
> not sure how to get started and could really use any info you could share.
>

There is really much documentation (because they're largely internal).
The classes to use are in Cython/Compiler/Visitor.py (there's a few
different options to use depending on what state you want to keep track
of, and whether you want to modify the tree). Look in
Cython/Compiler/ParseTreeTransforms.py to see examples of these classes
in use.

[walt3k]

Thanks a bunch!