More Python parsing libraries

[Brian Theado]

I was looking up mutation testing libraries for python and came across several references to Python parsing which I thought in light of recent work, Edward would be interested in. Mutation testing deserves its own thread. Here I'm just sharing the parsing references.

The mutation testing library and related article is at  https://pypi.org/project/mutmut/ and

https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78. The article has this bit (emphasis mine):

"I decided that I absolutely wanted a feature both Cosmic Ray and mutpy lacked: being able to apply a mutation on a source file and not screw up the entire file. Cosmic Ray and mutpy use Pythons built in AST library but it has the unfortunate property of not representing formatting in the AST, making it impossible to just dump the AST back out and get the original file. So if I can’t use Pythons own AST, what then? Enter baron, an independently developed Python->AST library specifically created to be able to round trip without changing your source files. Baron doesn’t support all of Python 3 syntax yet unfortunately, but it looks like people are working on it.

[EDIT: Since this article I’ve replaced Baron with Parso and now I fully support Python 3!]"

https://github.com/PyCQA/baron

"Baron is a Full Syntax Tree (FST) library for Python. By opposition to an AST which drops some syntax information in the process of its creation (like empty lines, comments, formatting), a FST keeps everything and guarantees the operation fst_to_code(code_to_fst(source_code)) == source_code."

https://parso.readthedocs.io/en/latest/

"Parso is a Python parser that supports error recovery and round-trip parsing for different Python versions (in multiple Python versions). Parso is also able to list multiple syntax errors in your python file."

https://github.com/davidhalter/jedi - not related to parsing, but it uses parso. Dropping the link leo being an editor might be able to make use of it:

"Jedi is a static analysis tool for Python that can be used in IDEs/editors. Jedi has a focus on autocompletion and goto functionality. Jedi is fast and is very well tested. It understands Python and stubs on a deep level."

[Edward K. Ream]

On Thu, Jan 16, 2020 at 8:13 AM Brian Theado <brian....@gmail.com> wrote:

I ... came across several references to Python parsing which I thought in light of recent work, Edward would be interested in.

Thanks for these links. I've bookmarked them.

baron and parso would have made my work much simpler. Alas, they further "Balkanize" python's world as far as parse trees go.

The advantages of my present work:

1. It works with Python's standard ast and tokenize modules.

2. It allows devs to work either with tokens or with parse trees: both/and, not either/or.

Edward