Leo and black

[Edward K. Ream]

The black python formatter is worthy of serious consideration.  Its best feature is line wrapping, moderated by a maximum line length setting (code setting in the API and/or command-line arg).

The "black" branch contains experimental code to reformat @file nodes using black.  At present, the code only reports the diffs that would be produced if the code were actually changed.

After playing with black for awhile (with various values of the line length setting),  my reactions were:

1. This is pretty much how I break lines myself.

2. Why didn't anybody think about doing this before?

3. Leo's beautify commands should do something similar.

Problems with black

Leo's beautify commands are token oriented.  In contrast, black's code is based on parse trees.  Parse trees would seem like the industrial-strength way. Alas, black looks considerably slower than Leo's beautify commands.  Speed matters a lot if one is intending to "blacken" code on the fly.  Otoh, I may be misunderstanding the underlying speed of black.

One of my speed experiments was to blacken an entire file at once.  This sidesteps problems with using Leonine syntax (@doc parts, section references, Leo directives) on a node-by-node basis.  Alas, this experiment failed because black thinks Leo's sentinel comments should have a space between the '#" and the '@'. Yes, I could monkey-patch black's comment check to work around this, but it would be very ugly, and would probably difficult to do accurately in all situations.

Improving Leo's beautify commands

An alternative to using black itself would be to improve Leo's existing beautify commands so they follow black's line-breaking strategy.  I believe this would be relatively straightforward.  This would be a major departure for Leo's beautify commands.  At present, these commands never insert or delete newlines. Still I plan to experiment with allowing the beautify commands to insert or delete newlines.

Summary

Black's line breaking algorithm is good enough to justify black's basic premise:

    [When using black], you agree to cede control over minutiae of hand-formatting.

    In return, Black gives you speed, determinism, and freedom from [pep8-based] nagging about formatting.

    You will save time and mental energy for more important matters.

Because of the problems with black discussed above, I'll look into adding black's line-breaking algorithm to Leo's beautify commands.  The ultimate goal would be to adjust all lines as we type.

Edward

[Terry Brown]

I've been using black for quite a while. I agree it's probably not
fast enough to continuously reformat a file as you type, but that would
probably be annoying. I just have a "run black on file" shortcut

noremap <Leader>l mtgg!Gblack -S -q -l 79 -<CR>`t

that binds \l to run black, including setting a mark (t) and jumping
back to it when done.

Python mode in vim runs 2-3 checkers on save, one of which reports
PEP-8 violations, so I run it when I see that, or when I've typed
something in a lazy mess and want it cleaned up.

Cheers -Terry

On Sun, 28 Jul 2019 14:03:36 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> The black python formatter <https://github.com/psf/black>is worthy of

> serious consideration. Its best feature is line wrapping

> <https://github.com/psf/black#how-black-wraps-lines>, moderated by a

> maximum line length setting (code setting in the API and/or
> command-line arg).
>
> The "black" branch

> <https://github.com/leo-editor/leo-editor/issues/1058> contains

> experimental code to reformat @file nodes using black. At present,
> the code only reports the diffs that would be produced if the code
> were actually changed.
>
> After playing with black for awhile (with various values of the line
> length setting), my reactions were:
>
> 1. This is pretty much how I break lines myself.
> 2. Why didn't anybody think about doing this before?
> 3. Leo's beautify commands should do something similar.
>
>

> *Problems with black*

>
> Leo's beautify commands are token oriented. In contrast, black's
> code is based on parse trees. Parse trees would seem like the

> industrial-strength way. Alas, black looks considerably times slower

> than Leo's beautify commands. Speed matters a lot if one is
> intending to "blacken" code on the fly. Otoh, I may be
> misunderstanding the underlying speed of black.
>
> One of my speed experiments was to blacken an entire file at once.
> This sidesteps problems with using Leonine syntax (@doc parts,
> section references, Leo directives) on a node-by-node basis. Alas,

> this experiment failed because *black thinks Leo's sentinel comments
> should have a space between the '#" and the '@'*. Yes, I could

> monkey-patch black's comment check to work around this, but it would
> be very ugly, and would probably difficult to do accurately in all
> situations.
>

> *Improving Leo's beautify commands*

>
> An alternative to using black itself would be to improve Leo's
> existing beautify commands so they follow black's line-breaking
> strategy. I believe this would be relatively straightforward. This
> would be a major departure for Leo's beautify commands. At present,
> these commands never insert or delete newlines. Still I plan to
> experiment with allowing the beautify commands to insert or delete
> newlines.
>

> *Summary*

>
> Black's line breaking algorithm is good enough to justify black's
> basic premise:
>
> [When using black], you agree to cede control over minutiae of
> hand-formatting.

> In return, *Black* gives you speed, determinism, and freedom from

[Edward K. Ream]

On Sun, Jul 28, 2019 at 5:15 PM Terry Brown <terry...@gmail.com> wrote:

I've been using black for quite a while.  I agree it's probably not
fast enough to continuously reformat a file as you type, but that would
probably be annoying. 

Thanks for these comments.  The prototype code in the "black" branch could be adapted to the work flow you suggest.  That code understands Leo-specific constructs, and works on nodes, not files, and there would be blacken-tree and blacken-node commands, just as with the beautify commands.

However, Leo's existing beautify commands appear to be three times faster than the blacken commands. Furthermore, Leo's beautify commands depend only on Python standard library for tokenizing.  So my plan is to see if I can use black's line-breaking ideas in Leo's beautify commands.  That should take just a few hours.

Edward

[Edward K. Ream]

On Sunday, July 28, 2019 at 4:03:37 PM UTC-5, Edward K. Ream wrote:

[improving] Leo's existing beautify commands so they follow black's line-breaking strategy [should] be relatively straightforward.  This would be a major departure for Leo's beautify commands.

A quick prototype of Leo's token-based beautify code reveals no gotchas:

- The code will need a new token type, say 'optional-line-end', in addition to the existing 'line-end' token.  'optional-line-end' represents a newline within one or more open (unmatched) parens or square/curly brackets.  'line-end' tokens represent newlines outside such unmatched parens or brackets.

- The new code might even avoid some nasty logic involving backslash newlines. Black generally deletes backslash newlines.

- The prototype code scans back through the output_tokens list looking for the previous 'line-end' or 'file-start' token. Instead, the revised code will likely remember the position of the previous 'line-end' or 'file-start' token.

- Calculating the length of one or more lines is easy:

    len(''.join([z.to_string() for z in self.code_list[i:]]))

where i is the start of the lines.

Edward

[Edward K. Ream]

On Sunday, July 28, 2019 at 4:03:37 PM UTC-5, Edward K. Ream wrote:

> black looks considerably slower than Leo's beautify commands.

On second thought, it looks like Leo can use black with very little extra work.

Black looks fast enough

The blacken-node command (in the "black" branch) takes about 0.01 sec for typically sized nodes. At present, this code doesn't actually change p.b.  It only reports diffs.  So it looks like Leo could optionally run blacken-node whenever selecting and/or unselecting a node for which @language python is in effect.

Leonine constructs are not gotchas

black.make_chunks adapts black to Leo. It calls black only for sequences of lines not containing Leonine constructs.  At present, it doesn't recognize @language so as to call black only for sequences of python lines.

Black's (mis-) handling of Leo's sentinel lines doesn't matter because sentinel lines never appear in p.b.

We might add an @no-black directive, or maybe even retire the @no-beautify directive.

Summary

Leo could (optionally) run the blacken-node command when selecting nodes containing python source code.

Edward

[vitalije]

On Monday, July 29, 2019 at 2:44:10 PM UTC+2, Edward K. Ream wrote:

So it looks like Leo could optionally run blacken-node whenever selecting and/or unselecting a node for which @language python is in effect.

That would be rather invasive and most of the time unnecessary spending CPU time. Selecting nodes is already too complicated and not very fast (to say the least). Please don't add any more burden to it. Instead let the user decide when to execute this command. If one wishes to perform it on every node selection, then it would be trivial to add it as a plugin so that the Leo's core dealing with selecting nodes, remains free of this feature.

The benefit of having this command executed is a rather rare thing. One must:

1. write Python code
2. write a lengthy line of code that could be splitted by this command

How many times one can have these conditions met? Against how many node selections where these conditions are not fulfilled? I believe the ratio is negligibly small.

Vitalije

[Terry Brown]

Now that I think about it I started using Black before I stopped using Leo.  I disagree with Vitalije's assessment of how often you'd execute it, I found I quite often wanted a node tidied up.  But I would agree with Vitalije on taking a minimalist light weight approach to using Black in Leo.  I just set up a button to run it on the current node - thinking at the time how cool it is that Leo makes doing that so easy.

Totally unrelated PSA: rabbitMQ - high performance cross language / cross machine (cloud) message queue system - I'm probably just late to the party, but it seems like the missing link in easy decoupling / connecting of software.  Not saying it has some immediate relevance to Leo, just that it's good to know about such things in general.

Cheers -Terry

On Mon, Jul 29, 2019 at 8:07 AM vitalije <vita...@gmail.com> wrote:

On Monday, July 29, 2019 at 2:44:10 PM UTC+2, Edward K. Ream wrote:

So it looks like Leo could optionally run blacken-node whenever selecting and/or unselecting a node for which @language python is in effect.

That would be rather invasive and most of the time unnecessary spending CPU time. Selecting nodes is already too complicated and not very fast (to say the least). Please don't add any more burden to it. Instead let the user decide when to execute this command. If one wishes to perform it on every node selection, then it would be trivial to add it as a plugin so that the Leo's core dealing with selecting nodes, remains free of this feature.

The benefit of having this command executed is a rather rare thing. One must:

1. write Python code
2. write a lengthy line of code that could be split by this command

How many times one can have these conditions met? Against how many node selections where these conditions are not fulfilled? I believe the ratio is negligibly small.

Vitalije

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/ec0a3148-9636-4977-b134-aa75ca421913%40googlegroups.com.

[vitalije]

On Monday, July 29, 2019 at 4:04:57 PM UTC+2, Terry Brown wrote:

I disagree with Vitalije's assessment of how often you'd execute it,

I didn't want to say that this command should be used rarely. I was just pointing out that unless you have changed the code in a node, there is no point in executing this command. While reading code  you presumably select nodes more often than you write code that needs beautifying. And perhaps while you write code, you wish to see it beautified right away and not just after you had selected some other node and then come back to the original node to see the beautified code.

I am mostly concerned with the present complexity of the tree selection/deselection code. I feel like it has already too much staff there, and perhaps something needs to be removed, reduced, simplified. Adding more code to it worries me.

Vitalije

[Edward K. Ream]

On Mon, Jul 29, 2019 at 8:06 AM vitalije <vita...@gmail.com> wrote:

That would be rather invasive and most of the time unnecessary spending CPU time. Selecting nodes is already too complicated and not very fast (to say the least). Please don't add any more burden to it.

Blackening nodes automatically would be a user option, off by default.

Edward

[Edward K. Ream]

On Mon, Jul 29, 2019 at 9:33 AM vitalije <vita...@gmail.com> wrote:

> I didn't want to say that this command should be used rarely. I was just pointing out that unless you have changed the code in a node, there is no point in executing this command.

The node selection logic already compares the old and new versions of p.b and p.h.  It must do this in order to set the dirty "bit" and the corresponding icon.  So yes, Leo should blacken a node only if it's body text changes.

Edward

[Edward K. Ream]

On Mon, Jul 29, 2019 at 9:04 AM Terry Brown <terry...@gmail.com> wrote:

rabbitMQ - high performance cross language / cross machine (cloud) message queue system..it seems like the missing link in easy decoupling / connecting of software. 

Thanks for this.  I've bookmarked https://www.rabbitmq.com/

Edward

[Matt Wilkie]

After playing with black for a bit, I quite like it. The only thing I've seen it change that I haven't wholesale agreed with is dedenting "next line comments".

func(arg1, arg2 ... keyword=thing)
    # indented comment about above line because inline would
    # be too long, and before-line is sub-optimal for some
    # other reason

-matt

[Edward K. Ream]

On Fri, Aug 2, 2019 at 4:38 PM Matt Wilkie <map...@gmail.com> wrote:

After playing with black for a bit, I quite like it. The only thing I've seen it change that I haven't wholesale agreed with is dedenting "next line comments".

This doesn't bother me.  In this black issue, amb states:

"Comments are not and will not be hand-edited by Black. Black is not able to understand if the comment is safe to rewrap. Use your editor to re-flow the comments which are safe and Black will respect your edit."

This is probably good enough for me, though I haven't actually blackened much.  We'll see.

I am dithering whether to add black-like line breaking to Leo's own (faster) beautify command.  I think I have to do it, just as a possibly-useful experiment.  If the experiment works, we can insert any desired options, regardless of what the black devs think about such things ;-)

Important: Leo will support @int blacken-max-line-length, default (I think) 88.  This is one of the few black options, and it is the crucial one.

Edward