Proposal for 'easy indentation mode' (e.g. for Python)

[Mikhail V]

Hello all.

I am new to this group. I am using NPP, which as I know now uses
Scintilla for editing. I am mostly Python programmer.
I want to share a feature description, that I think would be very
helpful for Python programmer (probably not only Python).
If there is already such feature then let me know.

In short, I think it would be really great if python code
would be parsed (loaded), and edited in such a manner
that would disallow a user to edit the indentation whitespace.
This means, that when a user moves the cursor at the line beginning,
it just stops on the indent level an not let them select or edit
the actual whitespace which belongs to the indentation,
i.e. spaces or tabs.

This is a difficult task, and may not 100% work for all style of coding,
e.g. if one adds whitespace additionally to align something
in arbitrary manner, and the question is of course, how it would deal
with copy-pasting.
But I think, in general it is not a rocket-science, and even with limitations,
such feature would really make life easier for indentation-based code.
Further, this could help with plain text formatting or other codes.

Export/import would follow specific rules, based on
two global parameters:
- indent_character
- amount_of_chars_per_level

The actual indentation level would then be controlled
in the editor by sending commands, e.g. :
- increase/decrease indent by 1 level

So I want to know what the authors and developers think?
If there is something like this already I'd also want to
participate in tests or prototyping.


Regards,
Mikhail

[Lex Trotman]

IMHO there are too many differing indentation/alignment styles to try
to have it integrated into a low level editing widget, especially for
languages where the indentation does not determine the program
structure, see https://en.wikipedia.org/wiki/Indentation_style.

Existing projects will not want Scintilla modifying their chosen scheme.

There are many code formatters of varying levels of sophistication and
accuracy that projects can use to support indentation schemes, from
external programs, eg uncrustify or astyle or clangformat, that the
editor/IDE can run code through on user request, to libraries that can
be integrated into the application to provide accurate language
specific guidance to style and more in "real time", eg clanglib for
C/C++.

Making such facilities available is a decision for the editor/IDE
project to fit in with their use-cases, not something that should be
enforced by Scintilla.

> --
> You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scintilla-inter...@googlegroups.com.
> To post to this group, send email to scintilla...@googlegroups.com.
> Visit this group at https://groups.google.com/group/scintilla-interest.
> For more options, visit https://groups.google.com/d/optout.

[Mikhail V]

Thanks for your thoughts, Lex.
Well, I am not very familiar with C/C++, nor am I familiar with
algorithms in Scintilla so it is hard for me to comment on this.

Generally I was willing to hear the points about the (future) potential benefits of such method.
And of course not as replacement for current state of things, but
rather as an additional experimental method, that users might want to try.
This might be the main complexity point - it can become hard to maintain and test
when having two methods instead of one 'universal' method. i.e. having all
un-interpreted characters on the screen and for editing, for now.

As for future benefits I do see some important things here.
Firstly, the indent level being parsed as , say a parameter of each line object,
may provide more robust and effective input, e.g. indent-aware multiline copy/pasting.
Empty indent areas  in the line beginning can be further utilized for mouse GUI interaction,
e.g. for line selection together with indent level - dragging the mouse on
the indent column could select lines with desired amount of indent.

Another point that I was thinking of - tabulations. Such feature as manual tabstops
would require some 'fixation' of the indent coordinates. And implementing correct
behavior can be PITA if all those whitespace are still wobbling around.
So say, first coordinate of a tabstop (inside the text), defined as ,say, 100 px, should
take the indent in account, namely it should start the column at 100 px
starting from the coordinate of the first visible character on the line.
Otherwise tabstobs will be messed up by changing the indent.


Regards,
Mikhail

> To unsubscribe from this group and stop receiving emails from it, send an email to scintilla-interest+unsub...@googlegroups.com.

[Neil Hodgson]

Mikhail V:

> In short, I think it would be really great if python code
> would be parsed (loaded), and edited in such a manner
> that would disallow a user to edit the indentation whitespace.

This style of editing is often referred to as “syntax-directed editing” and it seeks to ensure code correctness by limiting editing freedom. Editors like this have been researched for over 30 years but have not yet been successful.

https://en.wikipedia.org/wiki/Structure_editor
http://www.cs.wichita.edu/~rodney/scheutz/textsde5.pdf
http://bat8.inria.fr/~lang/papers/trondheim86/usefulness-syntax-directed-editors-19860616-18.pdf

> This means, that when a user moves the cursor at the line beginning,
> it just stops on the indent level an not let them select or edit
> the actual whitespace which belongs to the indentation,
> i.e. spaces or tabs.

Python source code contains regions where indentation is structurally-significant and other regions such as multi-line strings where the indentation does not affect the code structure.

Restricting selection as a mechanism for imposing editing modes has proved complex as projects trying to use the current ‘changeable’ style attribute have found.

> So I want to know what the authors and developers think?
> If there is something like this already I'd also want to
> participate in tests or prototyping.

I think this is a difficult and would require many changes in different areas. It should be thoroughly explored in a branch before any merging into mainline Scintilla.

Neil

[Mikhail V]

On Thursday, November 30, 2017 at 10:56:53 PM UTC+1, Neil Hodgson wrote:

Mikhail V:

> In short, I think it would be really great if python code
> would be parsed (loaded), and edited in such a manner
> that would disallow a user to edit the indentation whitespace.

   This style of editing is often referred to as “syntax-directed editing” and it seeks to ensure code correctness by limiting editing freedom. Editors like this have been researched for over 30 years but have not yet been successful.

Formally speaking, yes. Though methods I am currently thinking of are not reading structure or

try to understand the meaning of code lines, so I'd still call it just formatting aid.

And formatting IMO can make big difference. E.g. manually setting precise vertical indentation

can really improve readability.

> This means, that when a user moves  the cursor at the line beginning,
> it just stops on the indent level an not let them select or edit
> the actual whitespace which belongs to the indentation,
> i.e. spaces or tabs.

   Python source code contains regions where indentation is structurally-significant and other regions such as multi-line strings where the indentation does not affect the code structure.

   Restricting selection as a mechanism for imposing editing modes has proved complex as projects trying to use the current ‘changeable’ style attribute have found.

Yes one of the main obstacles to a purist solution are multi-line contents, e,g, this is valid syntax:

def many (
        somelongvariable1,
        somelongvariable2,
        somelongvariable3,
        somelongvariable4 ) :

Even more, I can mix preceding tabs and spaces in this example, and still it'll compile correct.

But many users wonder how would one align things nicely here?

There is no definite answer, but there are workarounds.

Having a separate control character for user-defined tabs versus old U+0009 tabs,

could IMO provide a purist solution to the problematic, still Python does not have it.

> So I want to know what the authors and developers think?
> If there is something like this already I'd also want to
> participate in tests or prototyping.

   I think this is a difficult and would require many changes in different areas. It should be thoroughly explored in a branch before any merging into mainline Scintilla.

Thanks for the feedback. As for prototyping, the things I personally could make:

small graphical applications which implement code rendering and particular interactive features

and their specification. This would be however Python apps, using Pygame (SDL wrapper).

Other options I don't have, but at least I have understanding how one writes python app so

it can be easily ported to C.

So I was working time to time on some related projects,

just as hobby. Mostly typography stuff, such as toolchains for bitmap font creation and

typesetting algorithms.

Modifying Scintilla code self unfortunately is not realistic for me, just in case.

Regards,

Mikhail

[Colomban Wendling]

Hi,

Le 30/11/2017 à 17:35, Mikhail V a écrit :
> On Thursday, November 30, 2017 at 10:56:53 PM UTC+1, Neil Hodgson wrote:
>

> […] E.g. manually setting precise vertical indentation can really
> improve readability.
I'm not sure, but maybe you'd be interested about what some refer as
"elastic tab stops". I'm not sure if it made it to Scintilla, but I
think some people we working on it.

>    Python source code contains regions where indentation is
> structurally-significant and other regions such as multi-line
> strings where the indentation does not affect the code structure.
>
>    Restricting selection as a mechanism for imposing editing modes
> has proved complex as projects trying to use the current
> ‘changeable’ style attribute have found.
>
>
> Yes one of the main obstacles to a purist solution are multi-line
> contents, e,g, this is valid syntax:
>
> def many (
>         somelongvariable1,
>         somelongvariable2,
>         somelongvariable3,
>         somelongvariable4 ) :
>
> Even more, I can mix preceding tabs and spaces in this example, and
> still it'll compile correct.

It's not indentation in the Python sense of it here, that's why you can
use anything. But understanding this means you actually understand
Python, which IMO is partly out of the scope of Scintilla, and thus the
liberty should be left to the application.

> But many users wonder how would one align things nicely here?
> There is no definite answer, but there are workarounds.
> Having a separate control character for user-defined tabs versus old
> U+0009 tabs,
> could IMO provide a purist solution to the problematic, still Python
> does not have it.

Python actually recommends 4 spaces per indentation levels, not one tab.
Other people recommend indenting with tabs and aligning with spaces
(generally, not specifically for Python), which kind of make sense to me
-- although it's not really what I use myself most of the time, but
that's another question.

Anyway, PEP8 suggests a few options for your above example:
https://www.python.org/dev/peps/pep-0008/#indentation
My favorite is the "visual indent" with the opening delimiter, but
indeed you might want to leave this up to the user… but if you're
already hiding the indentation editing from your user, you could as well
do all the "code style" (wide sense of "indentation") automagically.
Which suggests to me the only feature you want is a non-editable region,
and leave the whole control of its content to the application. Possibly
having a "non-editable indentation" option could make sense to ease
implementation of this, but that seem to be about it to me right now --
and as Neil mentioned, it might even be a problem for some of the
indentation that's actually not really indentation)

> Thanks for the feedback. As for prototyping, the things I personally

> could make: […]

You probably could try using Scintilla's support for readonly portions
of text as Neil mentioned some apps are trying to make use of:
http://scintilla.org/ScintillaDoc.html#SCI_STYLESETCHANGEABLE

I'm not sure of the challenges here, but I guess it'd be possible to
implement the "simple" extra API to increase/decrease/set indentation
level/size and leave it unmodifiable by editing in your app's side.

Regards,
Colomban

[Mikhail V]

On Friday, December 1, 2017 at 4:22:22 AM UTC+1, Colomban Wendling wrote:

Hi,

I'm not sure, but maybe you'd be interested about what some refer as
"elastic tab stops".  I'm not sure if it made it to Scintilla, but I
think some people we working on it.

I can't remember the exact specification of elastic tabstops, I might
re-read the docs about it.
In general, implementing real-time tracking of text change and
auto-setting of tabstops seems not so good idea to me.
Not only it complicates the core algoritms, but there is
not so many universal rules as to know the ranges of lines,
where the algorithm should apply the settings.

IMO such formatting works best when _human_ defines line ranges
where to apply formatting, and controls the result visually.
So the first thing I personally will try to implement, are manual tabulations.
Important is, the lines, once formatted, must remain so, even
if I move them around the document or set other indentation level.

If the formatting information cannot be stored in source files,
then it should be possible to store separately, e.g. for 'main.py'
have a 'main.py.formats' file, with entries indexed by source line indexes.
VIM  for example stores some such information in a single viminfo file.
Anyway storing meta-data, like formatting, bookmarks,etc.
is unavoidable step for development of human-oriented editing.

So, thanks again for insights. I will be working further on prototypes
and once ready, I'll share them here. Real examples always show
better than words.



Mikhail

[George Hummet]

>IMO such formatting works best when _human_ defines line ranges

> where to apply formatting.
---
Cool.

>But I think, in general it is not a rocket-science, and even with limitations,
>such feature would really make life easier for indentation-based code.
>Further, this could help with plain text formatting or other codes.

At least if you dont force yourself to build a Mars reachin Spaceship :)
Beginnin easy and just grow with the task sounds quite reasonable to me.

Have Fun!