Discuss: a proposed answer to python issue #33337

[Edward K. Ream]

Here is the first draft of a reply to Python issue #33337. This long-dormant issue discusses possible improvements to python tools.

I would appreciate it if you would read the thread in detail, and let me know if the following reply seems on topic and useful. Thanks.

===== Start Post

> So if users want to write tools that manipulate Python code, the standard
library doesn't provide them with great options.

> I suggest the following plan:

Hello all,

I would like to suggest another possibility.

For about three months I have been obsessed with the general problem of manipulating Python source code *easily*. I have long experience with such tools.  Along the way I have studied the
[asttokens](https://pypi.org/project/asttokens/),
[fstringify](https://pypi.org/project/fstringify/), and
[black](https://pypi.org/project/black/) projects.
Imo, each in its own way illustrates that something significantly better would be welcome. Each project uses complex, idiosyncratic, opaque generators to generate results. What if something much simpler were possible?

In late October, 2019 I completed a token-based version of fstringify. I could have stopped there, but then a new thought intruded. What if we could define a **token-order traversal** of parse trees? This new idea sparked an avalanche of new ideas. I couldn't let them go.

After three months of literally day and night development this single idea has turned into a major new tool, contained at present in [leoAst.py](https://github.com/leo-editor/leo-editor/blob/fstrings/leo/core/leoAst.py). The code is simple, general, flexible, and fast, in stark contrast to the tools mentioned above.

Leo's [issue #1440](https://github.com/leo-editor/leo-editor/issues/1440) (Unify the ast and token worlds)
contains extensive documentation for this project. The first comment contains an overview of the project, the second comment contains a Theory of Operation, and the third contains a History of the Project, with links to the many Engineering Notebook (ENB) post that I wrote along the way.

Imo, the TokenOrderGenerator (TOG) and TokenOrderTraverser (TOT) might be candidates for inclusion in the standard library, but first you Python devs will want to decide whether this project is actually something that is even vaguely of interest to you :-)

I will be happy to answer any questions you may have. Please let me know your thoughts.

Edward K. Ream
An old man, crazy about computer programming

===== End Post

Edward

[Terry Brown]

I wonder if a couple of demos would help, "see how much better this is" rather than "ask me about how much better this is".  Maybe a comparison like: https://gist.github.com/kennethreitz/973705

which compares `urllib2` to `requests` code to do the same thing.

And perhaps some context like "I want to know if this seems like a candidate for inclusion before I put together all the integrated docs. / tests / implementation pull request" to make it obvious you not expecting someone else to do all that.

A slightly different route might be to do the above demo / comparison stuff in a fully fledged package on GitHub / pypi / readthedocs so that you can get some feedback from people trying it out then propose getting into core libs by referencing the package.  Maybe you've already started in that direction, not sure.

Cheers -Terry

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/c70a9009-31d5-4cc4-b8d4-6e3af34d6d9f%40googlegroups.com.

[Edward K. Ream]

On Tue, Jan 14, 2020 at 3:00 PM Terry Brown <terry...@gmail.com> wrote:

I wonder if a couple of demos would help,

Thanks for this suggestion.  Imo, the demo is the TOG class compared with similar code in asttokens, fstringify and black. The differences are striking.

Edward

[Brian Theado]

This puts the burden on all the readers to go look at the two versions of fstringify (and other) code and be able to figure out on their own exactly what striking differences you are referring to. If you do the work to present the differences in an easy to read way, then all readers will benefit. Are there small pieces you can put side-by-side which show the improvements? But that's extra work you may not want to do and maybe it is hard to do. I'm just sharing my thoughts.

[Matt Wilkie]

Here is the first draft of a reply to Python issue #33337. This long-dormant issue discusses possible improvements to python tools.

I would appreciate it if you would read the thread in detail, and let me know if the following reply seems on topic and useful. Thanks.

I still need to read through with attention, but my first blush feedback is: you've worked incredibly hard to this point, and it must be really exciting/enticing to be near a point of release and of shouting "hear ye, hear ye" loud enough to attract attention, but don't make noise just yet!

At the very least I think leoAst needs to be run on it's own (e.g. not rely on `import leo ...`), and then have a concrete quick start example or two like Terry and Btheado mentioned.

-matt

[Edward K. Ream]

Thanks for this suggestion.  Imo, the demo is the TOG class compared with similar code in asttokens, fstringify and black. The differences are striking.

This puts the burden on all the readers to go look at the two versions of fstringify (and other) code and be able to figure out on their own exactly what striking differences you are referring to.

Yeah, I agree with you and Terry that there is a hole in the docs, and in my proposed letter.

If you do the work to present the differences in an easy to read way, then all readers will benefit. Are there small pieces you can put side-by-side which show the improvements?

I don't think snippets will plug the hole. What's needed is some discussion of what this tool is actually good for. This is something that I probably have never discussed, because it's so clear to me. I'll attempt a proper answer in an upcoming post.

Edward

[Edward K. Ream]

On Tue, Jan 14, 2020 at 8:00 PM Matt Wilkie <map...@gmail.com> wrote:

> you've worked incredibly hard to this point, and it must be really exciting/enticing to be near a point of release and of shouting "hear ye, hear ye" loud enough to attract attention, but don't make noise just yet!

I agree that more work is needed.

At the very least I think leoAst needs to be run on it's own (e.g. not rely on `import leo ...`), and then have a concrete quick start example or two like Terry and Btheado mentioned.

A quick-start example is a good idea. The first that comes to mind is the code that would fstringify a file. leoBeautify.py contains three fstringify commands that could be reworked. However, those commands do "too much" in some sense.

The challenge is to create a motivating example, which can be solved with only a few lines of code. That's a big ask, because I imagine anything useful might run to a few pages of code

I think the Python devs (and everyone else) need to be told in more detail why the tool is useful. That involves a discussion of what "unify the token and ast worlds" means, and why it would be useful. That's what I'll discuss in an upcoming post.

Edward

[Edward K. Ream]

On Tuesday, January 14, 2020 at 3:00:44 PM UTC-5, Terry Brown wrote:

I wonder if a couple of demos would help

I'd like to thank you, Brian and Matt for your comments. Those comments have helped me not to make a fool of myself straightaway :-)

Something shocking has just happened. I'll discuss what's in a separate thread.

Edward

[Brad]

Edward,

This is very interesting work.

As you probably know, the Python core developers are a bit curmudgeonly when it comes to suggestions that haven't come from themselves.

Be aware that the responses could focus exclusively on all of the potential problems, with the inevitable: "Why don't you put this on PyPI first and see how it is received?"

-Brad

[Edward K. Ream]

On Wed, Jan 15, 2020 at 2:45 PM Brad <brad.r...@gmail.com> wrote:

This is very interesting work.

Thank you.

As you probably know, the Python core developers are a bit curmudgeonly when it comes to suggestions that haven't come from themselves.

Hehe. I have some experience with that :-)

Be aware that the responses could focus exclusively on all of the potential problems, with the inevitable: "Why don't you put this on PyPI first and see how it is received?"

An excellent idea, even if it does come from the python devs ;-).  Matt is working on the packaging.

Thanks for your feedback.

Edward