Restarting the vim-emulation project

[Edward K. Ream]

Encouraged by the success of the simplified persistence project, I am going to restart the vim-emulation project.

All aspects of the design will be based on simplicity and accuracy. I have spent several hours recently playing with vim and noticing subtleties about vim's screen and handling of the dot. I shall attempt the best possible emulation of vim, constrained by the following simplifying principles:

1. For now, and probably forever, the code will assume the typical vim key bindings.  Not sure about vim key mapping, but that can wait.

2. The vim key bindings will apply only to plain keys (when @bool vim_mode = True).  Leo's existing key bindings will still apply for non-plain (Alt, Ctrl, Meta) keys, even when vim_mode is True.  However, it's possible that a few control keys will be handled differently in vim_mode in a hard-wired way...

This "great divide" of keystrokes is the simplest thing that could possibly work.  Furthermore, it allows the vim emulation to work in the much-more-complex Leonine environment *without* having to change any plain vim bindings!  The trick is to use the Leo bindings for the Alt-Arrow keys and <return> to shift focus to and from the outline. <Return> never makes any sense when in the outline (in vim insert mode), so it may as well shift focus back to the body pane and enter normal mode.  Slick.

This "great divide" does have a few complications involving non-plain keys, especially Ctrl-G and Ctrl-R, but we'll muddle through somehow.  The point is that vim is focused on plain keys, so getting them *exactly* right is the main problem.

3. I'll attempt as accurate an emulation as possible, including "Visual" and "Selection" modes, and especially attempting to recreate the cursor and selection areas exactly.  I don't necessarily plan a *complete* emulation of every possible combination of plain keys.  I'll start off with the most useful, and add others if and when anyone requests them.

4. Vim emulation will be based on a set of "official" commands whose names start with "vim_":  vim_a, vim_dot, etc.  These commands will update vim's dot as appropriate, and emulate the corresponding vim commands as closely as possible.

5. Vim's ':' key in normal mode will probably be a synonym for Alt-X.  This seems reasonable because ":" commands do not affect the dot.

That's all for now.  Your comments and corrections are welcome.

I expect to be able to use vim mode comfortably myself in just a few days.

Edward

[Zoltan Benedek]

Excellent!

For me the main obstacle in adopting Leo was Vim. Vim is so powerful and efficient, you can work by keyboard only.

Thanks Edward, for your endeavor and work.

Zoltan

[Edward K. Ream]

On Wednesday, July 23, 2014 7:54:36 AM UTC-5, Edward K. Ream wrote:

> Encouraged by the success of the simplified persistence project, I am going to restart the vim-emulation project.

> All aspects of the design will be based on simplicity and accuracy.

I have now started the vim project.  I immediately became immersed in the code because I started to focus on the integration of vim emulation with the rest of Leo.  How vim mode handles plain keys is almost irrelevant at this stage! What matters now are:

1. Getting the mode indicator correct.  At present, it lags behind by one key.

2. Having ":" in vim normal mode enter the minibuffer.  At present, there are all sorts of problems.

3. Displaying the cursor and selection area differently in normal, insert and selection modes.  At present, the cursor never changes.  I'll emulate vim's box cursor using a one-character selection range.  Somehow Leo's core will have to be aware of this "extra" selection.

4. Having <Return> in headlines switch to the body pane in normal mode.  Doesn't happen yet.

5. Handling all Alt and Control keys the same, regardless of vim mode.  Not sure what the present status is. Later, some exceptions will be made for heavily used vim control keys.

The above 5 points will require changes to Leo's core. Happily, all such changes will start with::

    if c.vim_mode

or

    if c.vim_mode and << predicate depending on vc.method >>

As a result, working on this project should be completely safe, because c.vim_mode is True only if the user has set::

    @bool vim_mode = True

Everything should work *exactly* as before as long as @bool vim_mode is False.

So this is good.  Getting these points exactly right right at the start is a good strategy for the following reasons:

1. They are crucial to a good user experience.
2. They will make working on the vim commands much less confusing.
3. They will demonstrate that the vim code can be integrated seamlessly into Leo's core, especially into the very complex code in leoKeys.py (k.masterKeyHandler, k.masterCommandHandler and their allies).
4. They will strongly affect the design of helper methods within leoKeys.py and corresponding methods within leoVim.py.

That's all for now.  I expect to be spending at least a week on this part of the vim project.  It should be straightforward to add bells and whistles to the actual vim commands when this preliminary work is done,

Edward

[Edward K. Ream]

On Thursday, July 31, 2014 6:21:25 AM UTC-5, Edward K. Ream wrote:

My appologies for the double post.  The first got posted by mistake and it would be too confusing if allowed to remain.

> So this is good.  Getting these points exactly right right at the start is a good strategy for the following reasons:

> 1. They are crucial to a good user experience.
> 2. They will make working on the vim commands much less confusing.
> 3. They will demonstrate that the vim code can be integrated seamlessly into Leo's core, especially into the very complex code in > leoKeys.py (k.masterKeyHandler, k.masterCommandHandler and their allies).
> 4. They will strongly affect the design of helper methods within leoKeys.py and corresponding methods within leoVim.py.

This strategy has been a spectacular success.  As of rev d9cca94...

- Vim mode code is now simply and correctly connected to code in leoKeys.py.
- Vim modes shows the correct state at all times.
- Crucially, ':' in normal mode and Alt-X do exactly the same thing.
- More generally, Leo handles *all* non-plain keys exactly as in non-vim mode(!)
  However, Ctrl-R is hard-bound to redo.  Exceptions may also have to be made for arrow keys.
- Undo and redo "just work", using Leo's existing undo handler, *regardless* of how vim mode changes text(!!)
- The parsing code now handles constructs such as:  dd, 2dd, d2j, 3d2j and many others.

Much work remains:

- Parsing is likely to be revised further.  This is a very tricky subject.
- d2j and 3d2j don't work exactly as in vim.  Soon it will, as described below.
- The i,o,O,a and A commands do not support a leading repeat count.
- No support for dot yet.  It should be straightforward.
- The h and l motions can move past newlines, unlike in vim.  This will be corrected.
- Vim mode must show (probably in the status line) the command keys as the user types them.
  For example, it should show 2d2 while the user types the 3d2j command.

I believe I have grokked an important principle underlying the real vim, one that I will emulate in Leo's vim mode code.  The idea is that parsers for movements like 2j or 0 or $ or gg, etc, **actually move the cursor**.  This simplifies command handling.

For example, in a command like 3d2j, the code for d will remember the leading repeat count and the insertion point, say I1, before parsing the last part of the command, in this case 2j.  The parser for 2j moves the cursor down two lines and calls a "callback" for the d command.  The callback (for the d command) compares the saved insertion point I1 with the present insertion point, I2 and then does the following:

- If I2 is on the same line as I1, leave I2 unchanged, othewise:
    A. If I1 < I2, we move I2 to the end of the line (if it isn't already there).
    B. If I1 > I2, we move I2 to the beginning of the line (if it isn't already there).

Imo, this will make dj work *exactly* as in vim: it deletes *two* lines, not one.  Similarly for all other movements, like d2h or d0 or d$ or dgg etc.

Exactly the same general strategy will support repeats counts for the i,a,A,o and O commands.  The command remembers the starting insert point, I1, and the callback for the i,a,A,o and O commands will compute the text actually inserted by comparing the new insertion point I2 with I2.  The inserted text is everything between I1 and I2 (with perhaps a few tweaks...).

So yesterday was a very good day.  I expect to have a usable (and truly compatible!) vim mode in just a few days.

Edward

P.S. I stated above that the h and l commands will soon be "anchored" to present line.  We want this for strict compatibility with vim.  This compatibility is important.  For example, dj deletes by lines, but one can imagine situations in which d5h can be combined with the dot to delete columns.  In that case, it's important not to move past line boundaries!

Edward

[Edward K. Ream]

On Fri, Aug 1, 2014 at 6:38 AM, Edward K. Ream <edre...@gmail.com> wrote:

> This strategy has been a spectacular success. As of rev d9cca94...

[big snip]

The ultimate reason for this success is that throughout the
experimentation process I have been guided by how vim actually works.
I have vim open at all times, and when I want to discover what happens
in vim I just take a look.

This see-what-vim-does "meta-strategy" seems obvious only in
retrospect. It's much better than studying either vim's code or
documentation. That study hasn't been a total waste: it gave me an
overall impression of vim's code and commands.

As a result of this meta-strategy, I am now much more comfortable with
the real vim, especially u and Ctrl-R :-) I'm not sure whether I'll
ever be truly comfortable with vim mode. We shall see. In this
regard, the difference in cursors between normal, input and visual
modes are surprisingly important visual cues. I'll be adding those
cues to Leo's vim mode asap.

In short, I am tremendously excited by the present state of Leo's vim
project. It's much farther along than I dared hoped.

Edward

P.S. A crucial part of the recent success was abandoning the idea of
allowing "flexible" vim-bindings. Even with hard-coded bindings, the
simplest thing that could possibly work (as far as code goes) is far
from simple! Adding extra complexity would have made the
vim-emulation code too complex to understand or to get right.

P.P.S. A hinted above, I have studied vim's actual C code in detail.
The uniformity (as far as it goes) of the real vim's commands is made
possibly only with some hairy scanning hacks.

Leo's vim-emulation scanning code is also fairly complicated.
However, Leo's code should remain simpler than the real vim's code.
Imo, the real vim doesn't use enough helper functions. Perhaps it was
a matter of style; perhaps the original developers were too concerned
with scanning speed. Certainly that speed is completely irrelevant
today.

EKR

[Terry Brown]

On Fri, 1 Aug 2014 08:49:12 -0500
"Edward K. Ream" <edre...@gmail.com> wrote:

> In this
> regard, the difference in cursors between normal, input and visual
> modes are surprisingly important visual cues. I'll be adding those
> cues to Leo's vim mode asap.

I would recommend doing it this way (this is current @data
qt-gui-plugin-style-sheet):

QLineEdit#lineEdit {
color: @minibuffer_foreground_color;
background-color: @minibuffer_background_color;
}

QLineEdit#lineEdit[style_class ~= 'minibuffer_warning'] {
background-color: @minibuffer_warning_color;
}

QLineEdit#lineEdit[style_class ~= 'minibuffer_error'] {
color: @minibuffer_error_color;
background-color: @minibuffer_warning_color;
}

the point here is not the @substitutions, although you might want to
use those too, but the [style_class ~= 'minibuffer_warning']
selectors. `style_class` is arbitrary, I chose it to match the CSS
class attribute without using the reserved word class itself. So the
styling is visible and editable where it belongs, in the stylesheet,
and in the code you just do:

some_widget.setProperty('style_class', 'vim_insert')
or
some_widget.setProperty('style_class', 'vim_transmogrify')

etc.

I can't remember if you have to

some_widget.setStylesheet(some_widget.stylesheet())

every time to get the change to be visible, possibly. If that's the
case, it's better to setStylesheet on the target widget than just
setting it on c.frame.top or whatever, as that seems to be a
performance hit.

Ah, google gives this, which is better:

style()->unpolish(this);
style()->polish(this);

it avoids having a stylesheet set on a lower level widget, translated
that would be

some_widget.style().unpolish(some_widget)
some_widget.style().polish(some_widget)

which avoids attaching a copy of the stylesheet to a widget way down
the hierarchy.

I'll have to try that for the places Leo currently applies the
stylesheet on individual widgets for performance reasons.

Cheers -Terry

[Edward K. Ream]

On Fri, Aug 1, 2014 at 8:49 AM, Edward K. Ream <edre...@gmail.com> wrote:

> Leo's vim-emulation scanning code is also fairly complicated. However, Leo's code should remain simpler than the real vim's code.

Just a two or three more comments and then I'll be on to coding. The
comments in this thread are pre-writing for a "Theory of Vim
Emulation" section in Leo's documentation.

1. A big reason that Leo's code will be simpler than vim's own code
is the Python language. The goal is always simplicity, and the
resulting generality the simplicity makes possible. Well, it's almost
infinitely easier to see patterns in Python rather than C. That single
perceptual advantage leads to better code.

2. I am proud of the work I did yesterday. I spent about 6 hours
trying all sorts of parsing schemes. Ironically, the more work I did,
the simpler the code became, and the less work I appeared to do.
Looking at git diffs gives you *no* idea of the effort involved!

3. I am particularly proud of the seamless integration of Leo's code
into the massively complex code in leoKeys.py. At present there are
only four "hooks" into the code in leoKeys.py:

The most important hook is at the top level of k.masterKeyHandler.
Its placement is brilliantly simple. Leo will handle all "modes" as
it always does. This includes handling the minibuffer.

After handling modes, k.masterKeyHandler calls vc.do_key if vim_mode
is in effect. vc.do_key returns True if it has completely handled the
key. In that case, k.masterKeyHandler just returns. Otherwise,
k.masterKeyHandler continues as usual. This has the effect of
handling *all* bindings exactly as before. This is clearly the
simplest thing that could possibly work.

The other three hooks are in k.keyboardQuit, k.resetLabel and
k.showStateAndMode. When vim mode is in effect, these hooks just call
vc methods to reset or display the vim state. Totally simple. I
don't expect any more hooks will be needed; they would be very simple
in any case.

In short, the vim code is on a solid footing. I can reorganize code
within leoVim.py with absolutely no effect on the rest of Leo.

Edward

[Edward K. Ream]

On Friday, August 1, 2014 9:16:23 AM UTC-5, Terry wrote:

On Fri, 1 Aug 2014 08:49:12 -0500
"Edward K. Ream" <edre...@gmail.com> wrote:

 

>> the difference in cursors between normal, input and visual modes are surprisingly important visual cues.  I'll be adding those cues to Leo's vim mode asap.

> I would recommend doing it [with Leo's qt stylesheets].

Thanks for this!  This makes sense, and I'm glad there is a way to add vim-related "bindings".

> Ah, google gives this, which is better:

     style()->unpolish(this);
     style()->polish(this);

Cool.  I wondered what those were for ;-)

Thanks for these tips, Terry.  They come at a good time.  I'll ask for your help should I need it.

Edward

[Edward K. Ream]

On Friday, August 1, 2014 9:25:45 AM UTC-5, Edward K. Ream wrote:

> Just two or three more comments and then I'll be on to coding...

Two or three or four more:

1. It's easy to find the hooks for (interface to) vim mode.  Just look for c.vim_mode (or even just c.vim) in leoKeys.py.  This is how I'll find the hooks two months from now ;-)

2. Each hook has the following general form::

    if c.vim_mode and c.vimCommands:
        c.vimCommands.do_something()
        return

This pattern is why we can be *absolutely* sure that vim emulation has no effect on Leo if @bool vim_mode = False.

The all-important call to vc.do_key in k.masterKeyHandler is::

    if c.vim_mode and c.vimCommands:
        ok = c.vimCommands.do_key(event)
        if trace: g.trace('vc.do_key returns',ok)
        if ok: return

That is, do_key returns True if it claims to have completely handled the key.  Otherwise, k.masterKeyHandler handles the key *exactly* as before.  I emphasize these simple-seeming lines because they (and their location!) are the heart of the clean interface.

3. There is a small, simple, important hack in vc.do_key.  In one place we need the following lines::

    val = f() # This may set vc.next_func again.
    # A hack.  A simple return implies we have handled the key fully.
    if val is None: val = True

The effect that handlers for particular *single* keys usually just return.  They only need to return "False" if they have discovered that they can't really handle the key.  Alternatively, I suppose they could set a "fail" flag.

In any case, communication between the various mode handlers and their delegates is tricky and crucial.  It's possible that it will be completely rewritten.  Even so, the code in k.masterKeyHandler is likely to remain unchanged.

4. Eventually, I plan to create leoVimUnitTests.leo.  It will have @bool vim_mode = True, and will contain all vim-mode-specific unit tests.  I expect to build in support for such commands in leoVim.py...

That's all that's in the bean for now...

Edward