Does Leo have something like ImportMagic?

[Yaakov Belch]

Hi I am starting to use Leo for programming --- and wonder whether it comes with some feature (like importmagic) to add python import statements as needed to the code?

In other words -- do you remember the file paths of all your features and write out the import statements for each new file by hand or do you have some tool to help you with that?

[Edward K. Ream]

On Mon, Feb 17, 2020 at 7:20 AM Yaakov Belch <yaakov...@gmail.com> wrote:

Hi I am starting to use Leo for programming --- and wonder whether it comes with some feature (like importmagic) to add python import statements as needed to the code?

In other words -- do you remember the file paths of all your features and write out the import statements for each new file by hand or do you have some tool to help you with that?

I'm not sure I understand your question. However, I always enable pyflakes on write, and pyflakes will warn about any undefined symbol.

If you are a python programmer, then you definitely should set:

@bool run-pyflakes-on-write = True

pyflakes catches roughly 95% of my programming blunders.

Edward

[Yaakov Belch]

Thank you very much for your swift response and for suggesting pyflakes.

My question aimed at a different angle.  Let me explain it again, with a concrete example:

Assume my file `pkg/message_decoder.py` defines a function `decode_message`.

I am now writing another file `pkg/api.py` which contains (among others) the following line:

    message = decode_message(requests.get(api_url, data=request_data).json())

For this to work, the Declarations of the `api.py` file need to contain at least the following two lines:

    import requests

    from pkg.message_decoder import decode_message

I am looking for a solution to create these two import statements automatically --- without me remembering the sources that define each symbol.

I imagine myself editing just a clone of the node (a child of @clean pkg/api.py) that defines the above line `message = ...`.

Pyflakes will mark the two symbols `decode_message` and `requests` as undefined --- and I want to press some button and have the right import lines added to the right Declarations node.

The problems I want to solve are:

* I don't want to remember the sources of each feature.

* I don't want to trace back the right Declaration node to add it to.

So my question is: Is there a good solution for this in leo --- and if not, what do you do when you add code that requires additional imports?

Yaakov

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS2oN5NsFZOVLGhkt6sCQf3gGkYv2NDOr5vSo%2B0PFD40ww%40mail.gmail.com.

[Edward K. Ream]

On Mon, Feb 17, 2020 at 9:41 AM Yaakov Belch <yaakov...@gmail.com> wrote:

The problems I want to solve are:

* I don't want to remember the sources of each feature.

* I don't want to trace back the right Declaration node to add it to.

So my question is: Is there a good solution for this in leo.

No, there isn't.

what do you do when you add code that requires additional imports?

I add the imports. I don't understand why this is a big deal for you.

I typically prefer not to use 'from x import y', preferring (usually) to reference x.y. That makes clear where symbols come from.

Edward

[Yaakov Belch]

Hi Edward,

As there is no ready Python-Auto-Import solution yet in Leo -- I will try to develop one myself.  I am new to Leo, I would appreciate help in choosing the right design and APIs.

But first:  Why are automatic imports important for me?  For me, editing import statements whenever a new dependency is added to a module is a "deep interruption" of my work flow.  I really just want to use a function --- but I have to (1) remember or research where exactly that function was defined.  I have to (2) jump to a different location to edit the import statement and I have to (3) jump back to the place where I was working.

In the preferred Leo style, step (2) becomes more difficult:  I usually will edit cloned nodes and finding the corresponding Declaration nodes is more involved than scrolling up to the top of the file.

The bottom line of the interruption --- I lost the momentum of the thought of just using that function.

I have used an automatic import feature with PyCharm for a while and it made a real impact on my work.  But I don't want to copy the user experience from PyCharm.  I want something simpler and more aligned with my work flow.

Here is my initial design sketch:

(1) All information about importable names must be unambiguously in the titles of the project outline:

* Local features (functions and classes defined in this project) are already exposed as nodes below @clear.

* Each external dependency (e.g.  `from docutils.nodes import field`) must be declared as one node title for the whole project.

* There will be a scan of the whole outline to create an `import_dict` mapping symbol names to their import sources.

(2) External files with python-auto-import semantics would be explicitly defined (e.g. with @auto-import similar to @others or with @python-src instead of @clear).  When saving such a file, we want to:

* Expand the body of the file (without any automatic `import` statements).

* Extract undefined symbols from this collected source (per file).

* Translate these symbols into import statements (using `import_dict` from above) and prepend them to the code.

* When loading the external file from disk -- recognize the initial header of import statements and ignore them.

(3) There are some special cases (e.g. related to circular imports) that need special attention.  I believe the programmer should explicitly resolve these (rare) cases rather than making the automatic import more complex and more fragile by adding automatic special cases.

Can you advise about the best way to hook into the creation of external files and into the reading to accomplish these modifications?

Yaakov

[Edward K. Ream]

On Tue, Feb 18, 2020 at 9:02 AM Yaakov Belch <yaakov...@gmail.com> wrote:

I have used an automatic import feature with PyCharm for a while and it made a real impact on my work. 

Good to know.

 

But I don't want to copy the user experience from PyCharm.  I want something simpler and more aligned with my work flow.

Simple and elegant is a great starting point :-)

Edward

[Matt Wilkie]

 

But first:  Why are automatic imports important for me?  For me, editing import statements whenever a new dependency is added to a module is a "deep interruption" of my work flow.  I really just want to use a function --- but I have to (1) remember or research where exactly that function was defined.  I have to (2) jump to a different location to edit the import statement and I have to (3) jump back to the place where I was working.

For #1, I find it important to remember what the module has the function. I wouldn't want to get os.path confused with foorbar.null.path. When I don't like the extra typing I use the idiom "from foobar import null as fbn" and then use it as "fbn.path".

For #2 and thus #3: just skip them! When I'm in flow and need a new module I just "import foo" right where I am, inside the function even. Later, after the thing is working well enough, I move the import where it belongs. There are probably subtle (or not so subtle!) scenarios where this pattern causes problems but so far it hasn't happened to me.

I'm not arguing against an auto-import feature, just answering the "what do you do?" part of the question.

I saw a python utilty somewhere on pypi.org that handled move-imports-automatically as part of a code linting process but didn't feel the need to follow up on it. I see an active feature request for Black to move and order imports: https://github.com/psf/black/issues/333. A quick read of the comments seems to indicate it might be forthcoming soon. If it does, then I expect Leo to be able to use it straightforwardly.

-matt

[Yaakov Belch]

Hi Matt,

Thanks for sharing your workflow.  Here are some comments:

For #1 (remember where the function comes from -- yb), I find it important to remember what the module has the function. I wouldn't want to get os.path confused with foorbar.null.path. When I don't like the extra typing I use the idiom "from foobar import null as fbn" and then use it as "fbn.path".

 

My plan is that the programmer explicitly declares where functions are imported from --- but he does this only for external dependencies and only once per project.  Imports for all python files will be built from these definitions.  The rules are:

* If you define a class, function or global variable `path` in your project --- then any undefined use of a name `path` will be provided with an automatic matching import statement.

* The first time I want to use an external dependency (e.g. `Pathlib.path`) I have to define the corresponding import statement.

* However, you can reuse this definition for all python files in this project --- and copy your favorite definitions in bulk to other projects.

* If there is any conflict --- multiple definitions for the same name --- then you get an error message.  You will have to decide on a unique name to feature map of names that you want to use in this project.

As a result, there will be no ambiguity.

For #2 and thus #3 (move the import statement to the top -- yb): just skip them! When I'm in flow and need a new module I just "import foo" right where I am, inside the function even. Later, after the thing is working well enough, I move the import where it belongs. There are probably subtle (or not so subtle!) scenarios where this pattern causes problems but so far it hasn't happened to me.

 

Here is a problem with automatically moving import out of the body of functions or methods to the top of the file:  When you have cyclic data dependencies between classes, you can resolve them by putting some import statements inside functions and not at the top of the file.  These imports will be executed at run time (when the function is actually called) and hence break the import cycle.

I came across a few special cases where swapping import statements with other statements can cause problems:

* If a statement affects the import mechanism (such as setting the search path for modules) then it has to come before the imports.

* gevent.monkey_patch() needs to run before other imports --- otherwise it won't work as intended.

I'm not arguing against an auto-import feature, just answering the "what do you do?" part of the question.

 That's understood --- I appreciate the discussion.

I saw a python utilty somewhere on pypi.org that handled move-imports-automatically as part of a code linting process but didn't feel the need to follow up on it. I see an active feature request for Black to move and order imports: https://github.com/psf/black/issues/333. A quick read of the comments seems to indicate it might be forthcoming soon. If it does, then I expect Leo to be able to use it straightforwardly.

The 'optimize imports' feature of PyCharm (taken as a motivating example of this issue 333) does not move imports past function definitions.  It only optimizes contiguous runs of import statements --- for the sake of correctness.

[Yaakov Belch]

Hi Edward, 

I wrote my first Leo plugin: python_auto_imports.  It works for me --- but there is still a lot of work to be done before it can be shared.

My question is: What is the best place to discuss the different aspects of the work and to keep the work itself?

On the top level, we have two choices:

(A) Do the work in private until we have some project that can be integrated and used by others.

(B) "Work in the open" --- discuss the design choices with anyone interested, share the work-in-progress code in github and release it to the official Leo source when it's ready.

For option (B), there may be different appropriate locations for the different parts of the work:

(1) Motivation and normal UX design/documentation.

(2) UX design/documentation for error reporting and user choices --- integration with pylint and other error detection extensions.

(3) Technical integration with the Leo sources.

Here is a high-level overview of these three parts of work: 

(1) Motivation and normal UX design/documentation:

In order for other people to use the plugin, I need to clearly motivate it and explain how to use it.  

For most parts, the `import` statements in python are trivial -- but for some case (e.g. import cycles and import side-effects), things can become complicated.  These problems need to be discovered and discussed.

For documentation, I believe a simple LeoVue page could collect all the relevant information.  For discussion we may either continue in this e-mail thread or create a github issue in the leo-editor repository for this extension.

(2) UX design/documentation for user choices and error reporting --- fix-errors and pyflakes integration:

Instead of copying the UI/UX of the automatic imports from PyCharm (that opens modal input windows whenever the user needs to make a choice), I want to create a UX that lives inside the Leo outline. This same UX will integrate with other features that can detect errors and warnings about the project currently edited -- e.g. pyflakes and capturing stack traces from test exceptions.  

Here is the idea: You define all choices as configurations (e.g. with @auto-import nodes) in the outline of your project. It's OK not to create any such configuration at the outset.  This will simply trigger errors --- to be fixed with `fix-errors`.

The fix-errors command works similar to the cfa command: it populates a 'fix-errors' node at the bottom of the outline with explanations of any detected errors/warnings, suggestions of corrections and clones of the offending source nodes.  These child nodes will be organized into an easy-to navigate tree so that you can quickly get to the action that you want to perform to resolve the problem.

For example, you may see just two direct children of the `fix-errors` node: One exception trace (showing just the message of the exception) and one node titled '4 auto-import problems'.  

By expanding the first child (exceptions) -- you would see the code lines of the exception stack trace that are appear as clones of the actual sources of these lines.  Just with the cursor keys, you can flip through these code nodes until you find the problem and fix it on the spot.

Alternatively, you could expand the node '4 auto-import problems' -- which has four children.  One for each discovered problem.  By expanding one of these problems, you will see the relevant problem information --- and children with suggested solutions (e.g. imports from external libraries).  If we have more information about a suggestion, we can add as sub-children to each option.  Hence, just with your cursor keys you can investigate all problems, suggested solutions --- and solve the problem on the spot.

This fix-errors extension is a project by itself --- and should be documented in a separate project page and discussed in a separate e-mail thread or in a separate github issue.

(3) Technical integration with the Leo sources:

Currently, my plugin is a hack.  I just started to play with Leo a few days ago and needed some solution quickly to continue with my main day job (with Leo).  Very likely, I will need to change the current Leo source to add hooks that my extensions can connect to.

For coding we have two choices: Either work on feature branches inside the leo-editor repository or to clone this repository and send pull requests from this external copy.

What are your preferences for these choices?

Yaakov

[Edward K. Ream]

On Fri, Feb 21, 2020 at 7:20 AM Yaakov Belch <yaakov...@gmail.com> wrote:

I wrote my first Leo plugin: python_auto_imports.  It works for me --- but there is still a lot of work to be done before it can be shared.

Many for this work. My apologies for the delay in responding.

My question is: What is the best place to discuss the different aspects of the work and to keep the work itself?

Yes. This is the place to discuss everything related to Leo.

On the top level, we have two choices:

(A) Do the work in private until we have some project that can be integrated and used by others.

(B) "Work in the open" --- discuss the design choices with anyone interested, share the work-in-progress code in github and release it to the official Leo source when it's ready.

Do whichever you like. I encourage public discussion whenever possible.

For option (B), there may be different appropriate locations for the different parts of the work:

(1) Motivation and normal UX design/documentation.

(2) UX design/documentation for error reporting and user choices --- integration with pylint and other error detection extensions.

(3) Technical integration with the Leo sources.

Here is a high-level overview of these three parts of work: [Big snip]

I suggest you publish your plugin in "devel". This will safe provided the plugin doesn't monkey-patch Leo's core. Even then, the plugin wouldn't be dangerous unless people enabled it.

I've just sent you an invitation to be part of Leo's core team. That should give you write access to devel.

Edward

[Yaakov Belch]

Thank you very much.  

I will move my python-auto-import extension into the devel branch.  As already said, this may take some time --- and I will discuss how to resolve the hacks here on this e-mail list.

Yaakov

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS1tdh%2BtCXoeS39jJKi9JdfRqm3qptQdX2GDF%3DzNkV0ipg%40mail.gmail.com.

[Edward K. Ream]

On Thu, Feb 27, 2020 at 2:01 PM Yaakov Belch <yaakov...@gmail.com> wrote:

Thank you very much.  

You're welcome.

I will move my python-auto-import extension into the devel branch.  As already said, this may take some time --- and I will discuss how to resolve the hacks here on this e-mail list.

I'll look forward to seeing what you do.

Edward