Using importhooks for "preparsing"

[Nils Bruin]

With the adoption of Python 3 in Sage, we have access to a lot of more recent development in python. One of which is the new "import" machinery, which makes hooking in particular processors a lot easier. For some examples:

https://github.com/aroberge/ideas

http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking-python.html

https://macropy3.readthedocs.io

I'm not suggesting that we should adopt one of these to implement our preprocessor, but I think some of the ideas in there could help with a cleaner implementation. Obviously, sage's REPL doesn't put its code through import hooks, but appropriate modularisation of the process would probably make it relatively straightforward to implement an IPython REPL that uses similar steps (it still needs to tokenize, parse, transform ASTs, and compile)

I don't have concrete ideas for, or experience with, rewriting the preparser. I'm just flagging the opportunity in case there is someone with the right skill set who would find this a fun project. It would require some experimentation to get right, but if it works correctly (and more efficiently?) I think there will be appetite for adopting it.

[Matthias Koeppe]

On Sunday, July 26, 2020 at 4:10:25 PM UTC-7, Nils Bruin wrote:

With the adoption of Python 3 in Sage, we have access to a lot of more recent development in python. One of which is the new "import" machinery, which makes hooking in particular processors a lot easier. [...]

+1. In particular, I think this would be a key improvement that would make it easier for users to distribute packages consisting of Sage code.

[Nils Bruin]

Hm, do we actually want to go there? It would be possible, but I think our compatibility with the rest of the python ecosystem now is better by mandating that library code is in plain python. The only obnoxious one for that is replacing "^" with "**". The generator mangling is a bit too magical to be used outside of a REPL anyway.

[Matthias Koeppe]

I am sure it will be useful for sharing preliminary code, and for easing the transition from "one long .sage file" to a proper Python package (distribution). 

Users often need to do two steps at a time: Thinking about modularization of their code into several Python modules; and translating from the preparsed Sage language to plain Python. By supporting importable modules written in the Sage language, these two steps would become independent of each other.

(Also, the preparsed language is quite convenient for code that has lots of integer/rational constants.)

Regarding the compatibility with the Python ecosystem: This is certainly an important point, and we should address it by documenting "best practices" for distribution of user packages, which would include recommending to avoid .sage modules.