ENB: collapsing complexity in vim mode
27 views
Skip to first unread message
Edward K. Ream
Aug 3, 2014, 8:00:58 AM8/3/14
to leo-e...@googlegroups.com
This is an engineering notebook post. It consists of notes to myself and other core developers, plus a brag in the post script. Feel free to ignore.
In the last four days, the vim-emulation code has consistently gotten simpler. This has kept my energy levels extremely high. It's been hard to stay asleep ;-)
When I awoke this morning, I saw how to make the vim-mode code dead simple. I'll revise the code today. The result will be much simpler than the real vim's main line code.
==== The problem
The present code works, and some programmers might be satisfied with it. However, it's tricky to know (and remember!) just how scanning works and just how the accumulating command (vc.command_list) and the previous dot (vc.dot_list) are updated.
In fact, it's not obvious that they *are* updated properly in all situations. The code feels fragile.
==== The solution
The solution is to apply the principle, "explicit is better than implicit" as follows:
1. The vc.handler var will contain the method used to handle the next incoming key. Initially, the handler will be the vc.do_normal_mode method.
vc.handler ivar collapses vc.do_key to something like this::
def do_key(vc,event):
'''Handle the next key in vim mode.'''
vc.init_scanner_vars(event)
vc.return_value = None
vc.handler()
return vc.return_value
This is, quite obviously, the simplest thing that could possibly work. As an internal check, I'll replace the return statement with::
if vc.return_value not in (True,False):
<< issue internal error message >>
return True
else:
return vc.return_value
2. Every handler must end with a (explicit!) call to one of the following **acceptance methods**:
vc.accept(handler,...)
# Add the key to the accumulating command.
vc.ignore(...)
# Ignore the key, issue a warning (by default) and retain all present state.
vc.done(handler,...)
# End the command and (by default) update the dot.
Every acceptance function sets vc.return_value to True or False, which is why the checking code at the end of vc.do_key will work.
By default, the acceptance functions set vc.return_value to True, indicating that vim mode has completely handled the key and that k.masterKeyHandler can just return.
Insert mode will call vc.accept(handler,return_val=False). As a result, k.masterKeyHandler will handle the key *exactly* as it does in non-vim mode.
That's it. The (explicit!) acceptance methods will collapse the complexity and increase the regularity of *all* the key-handling code in leoVim.py. It's a big step forward.
Edward
P.S. There are two kinds of handlers, **mode-handlers** and **command-handlers**.
vc.do_normal_mode will be a mode handler. It will look something like::
def do_normal_mode(vc):
'''Handle an outer normal mode command.'''
func = vc.normal_mode_dispatch_dict.get(vc.stroke)
if func:
func() # vc.do_key checks that func calls an acceptance method!
elif vc.is_plain_key(vc.stroke):
# ignore plain keys with a complaint.
vc.ignore()
else:
# Pass non-plain keys to k.masterKeyHandler
vc.accept(handler=vc.do_normal_mode,return_value=False)
The vim_d_... family of methods are examples of command handlers. In rough outline, they will become...
def vim_d(vc):
vc.accept(handler=vc.vim_d2)
# Accept the 'd' and let vc.vim_d2 handle the next character.
def vim_d2(vc):
if vc.stroke == 'd':
<< delete the line >>
vc.done()
else:
vc.handler=None
vc.begin_motion(after_motion_handler=vc.vim_d3)
# vc.vim_d3 will be called when the motion is completed.
assert vc.handler
# vc.begin_motion (or a helper!) must set vc.handler *now*.
def vim_d3(vc):
<< Complete the d command after the cursor has moved >>
vc.done()
# Properly updates the dot.
This is not completely trivial, but it makes explicit what must happen.
P.P.S. <brag>
I said in the post https://groups.google.com/d/msg/leo-editor/5T71y-ePMSI/mOEXkVT4QpIJ
"A big reason that Leo's code will be simpler than vim's own code is the Python language. The goal is always simplicity, and the
resulting generality the simplicity makes possible. Well, it's almost infinitely easier to see patterns in Python rather than C. That single perceptual advantage leads to better code."
Two comments:
1. I think it's fair to say that Leo also plays an important part in simplifying code :-) The ability to keep both the big picture and all the details constantly in view is crucial.
2. I am a mediocre programmer in most respects. But I have one great programming talent: the ability to keep revising code until it *obviously* is the simplest code possible. Later today, after the latest collapse in complexity, you will be able to see this talent to its full extent. Just compare Leo's vim-parsing code with the real vim's main line. You will be amazed. I certainly am :-)
</brag>
EKR
In the last four days, the vim-emulation code has consistently gotten simpler. This has kept my energy levels extremely high. It's been hard to stay asleep ;-)
When I awoke this morning, I saw how to make the vim-mode code dead simple. I'll revise the code today. The result will be much simpler than the real vim's main line code.
==== The problem
The present code works, and some programmers might be satisfied with it. However, it's tricky to know (and remember!) just how scanning works and just how the accumulating command (vc.command_list) and the previous dot (vc.dot_list) are updated.
In fact, it's not obvious that they *are* updated properly in all situations. The code feels fragile.
==== The solution
The solution is to apply the principle, "explicit is better than implicit" as follows:
1. The vc.handler var will contain the method used to handle the next incoming key. Initially, the handler will be the vc.do_normal_mode method.
vc.handler ivar collapses vc.do_key to something like this::
def do_key(vc,event):
'''Handle the next key in vim mode.'''
vc.init_scanner_vars(event)
vc.return_value = None
vc.handler()
return vc.return_value
This is, quite obviously, the simplest thing that could possibly work. As an internal check, I'll replace the return statement with::
if vc.return_value not in (True,False):
<< issue internal error message >>
return True
else:
return vc.return_value
2. Every handler must end with a (explicit!) call to one of the following **acceptance methods**:
vc.accept(handler,...)
# Add the key to the accumulating command.
vc.ignore(...)
# Ignore the key, issue a warning (by default) and retain all present state.
vc.done(handler,...)
# End the command and (by default) update the dot.
Every acceptance function sets vc.return_value to True or False, which is why the checking code at the end of vc.do_key will work.
By default, the acceptance functions set vc.return_value to True, indicating that vim mode has completely handled the key and that k.masterKeyHandler can just return.
Insert mode will call vc.accept(handler,return_val=False). As a result, k.masterKeyHandler will handle the key *exactly* as it does in non-vim mode.
That's it. The (explicit!) acceptance methods will collapse the complexity and increase the regularity of *all* the key-handling code in leoVim.py. It's a big step forward.
Edward
P.S. There are two kinds of handlers, **mode-handlers** and **command-handlers**.
vc.do_normal_mode will be a mode handler. It will look something like::
def do_normal_mode(vc):
'''Handle an outer normal mode command.'''
func = vc.normal_mode_dispatch_dict.get(vc.stroke)
if func:
func() # vc.do_key checks that func calls an acceptance method!
elif vc.is_plain_key(vc.stroke):
# ignore plain keys with a complaint.
vc.ignore()
else:
# Pass non-plain keys to k.masterKeyHandler
vc.accept(handler=vc.do_normal_mode,return_value=False)
The vim_d_... family of methods are examples of command handlers. In rough outline, they will become...
def vim_d(vc):
vc.accept(handler=vc.vim_d2)
# Accept the 'd' and let vc.vim_d2 handle the next character.
def vim_d2(vc):
if vc.stroke == 'd':
<< delete the line >>
vc.done()
else:
vc.handler=None
vc.begin_motion(after_motion_handler=vc.vim_d3)
# vc.vim_d3 will be called when the motion is completed.
assert vc.handler
# vc.begin_motion (or a helper!) must set vc.handler *now*.
def vim_d3(vc):
<< Complete the d command after the cursor has moved >>
vc.done()
# Properly updates the dot.
This is not completely trivial, but it makes explicit what must happen.
P.P.S. <brag>
I said in the post https://groups.google.com/d/msg/leo-editor/5T71y-ePMSI/mOEXkVT4QpIJ
"A big reason that Leo's code will be simpler than vim's own code is the Python language. The goal is always simplicity, and the
resulting generality the simplicity makes possible. Well, it's almost infinitely easier to see patterns in Python rather than C. That single perceptual advantage leads to better code."
Two comments:
1. I think it's fair to say that Leo also plays an important part in simplifying code :-) The ability to keep both the big picture and all the details constantly in view is crucial.
2. I am a mediocre programmer in most respects. But I have one great programming talent: the ability to keep revising code until it *obviously* is the simplest code possible. Later today, after the latest collapse in complexity, you will be able to see this talent to its full extent. Just compare Leo's vim-parsing code with the real vim's main line. You will be amazed. I certainly am :-)
</brag>
EKR
Edward K. Ream
Aug 3, 2014, 2:29:19 PM8/3/14
to leo-e...@googlegroups.com
On Sunday, August 3, 2014 7:00:58 AM UTC-5, Edward K. Ream wrote:
> ...keep revising code until it *obviously* is the simplest code possible.
Here are vim-mode's present (not pushed) state handlers. They are neither tested, nor integrated with the key handlers, but pylint is happy :-)
def do_inner_motion(vc):
'''Handle strokes in motions.'''
return vc.do_state(vc.motion_dispatch_dict,'motion')
def do_insert_mode(vc):
'''Handle insert mode: delegate all strokes to k.masterKeyHandler.'''
vc.delegate()
def do_normal_mode(vc):
'''Handle strokes in normal mode.'''
return vc.do_state(vc.normal_mode_dispatch_dict,'normal')
def do_visual_mode(vc):
'''Handle strokes in visual mode.'''
return vc.do_state(vc.vis_dispatch_dict,'visual')
Three of them use the following helper::
def do_state(vc,dispatch_dict,mode_name):
'''General dispatcher code.'''
trace = False and not g.unitTesting
func = dispatch_dict.get(vc.stroke)
if func:
if trace: g.trace(mode_name,vc.stroke,func.__name__)
func()
elif vc.is_plain_key(vc.stroke):
vc.ignore()
else:
# Pass non-plain keys to k.masterKeyHandler
vc.delegate()
vc.return_value = False
To make this scheme work, all key handlers must end in an acceptance method: vc.accept, vc.delegate or vc.ignore.
As a result, key handlers know nothing about state handlers and vice versa!
Nothing could be simpler or more flexible. This is the way it is written in The Book.
EKR
> ...keep revising code until it *obviously* is the simplest code possible.
Here are vim-mode's present (not pushed) state handlers. They are neither tested, nor integrated with the key handlers, but pylint is happy :-)
def do_inner_motion(vc):
'''Handle strokes in motions.'''
return vc.do_state(vc.motion_dispatch_dict,'motion')
def do_insert_mode(vc):
'''Handle insert mode: delegate all strokes to k.masterKeyHandler.'''
vc.delegate()
def do_normal_mode(vc):
'''Handle strokes in normal mode.'''
return vc.do_state(vc.normal_mode_dispatch_dict,'normal')
def do_visual_mode(vc):
'''Handle strokes in visual mode.'''
return vc.do_state(vc.vis_dispatch_dict,'visual')
Three of them use the following helper::
def do_state(vc,dispatch_dict,mode_name):
'''General dispatcher code.'''
trace = False and not g.unitTesting
func = dispatch_dict.get(vc.stroke)
if func:
if trace: g.trace(mode_name,vc.stroke,func.__name__)
func()
elif vc.is_plain_key(vc.stroke):
vc.ignore()
else:
# Pass non-plain keys to k.masterKeyHandler
vc.return_value = False
To make this scheme work, all key handlers must end in an acceptance method: vc.accept, vc.delegate or vc.ignore.
As a result, key handlers know nothing about state handlers and vice versa!
Nothing could be simpler or more flexible. This is the way it is written in The Book.
EKR
Edward K. Ream
Aug 4, 2014, 7:05:35 AM8/4/14
to leo-e...@googlegroups.com
On Sunday, August 3, 2014 1:29:19 PM UTC-5, Edward K. Ream wrote:
>> ...keep revising code until it *obviously* is the simplest code possible.
>> ...keep revising code until it *obviously* is the simplest code possible.
>To make this scheme work, all key handlers must end in an acceptance method: vc.accept, vc.delegate or vc.ignore.
> As a result, [the new] key handlers [will] know nothing about state handlers and vice versa!
As discussed in another thread, rev ebacdc1... puts this scheme into practice.
Here, I'll discuss these changes in detail, including complications that I discovered yesterday and further simplifications that I'll make today.
1. In general, ending each command handler with an acceptance method has been a great success. Acceptance methods clearly indicate what should happen when a command handler ends.
Acceptance handlers hide the existence of the vc.next_func and vc.return_value ivars from all command handlers. This makes each command handler much cleaner in appearance. Furthermore, it is now possible to tweak acceptance methods without changing command handlers in any way.
In other words, even though the acceptance methods follow the principle, "explicit is better than implicit", they *also* hide implementation details in a most useful way. It's the best of both "implicit" and "explicit".
2. Besides the **direct** acceptance methods, vc.accept, vc.delegate or vc.ignore, vc.done & vc.quit, command handlers can also end with **indirect** acceptance methods: vc.begin_insert_mode, vc.begin_motion, vc.end_insert_mode, and vc.vim_digits. Indirect acceptance methods (eventually!) call direct acceptance methods.
**Important**: command handlers can use different acceptance methods (direct or indirect) in different branches of their code, provided that all branches end in a call to exactly one acceptance method. In practice, checking this requirement is easy. Furthermore, the checking code in vc.do_state will warn if this requirement has not been met.
3. I did get one surprise: vc.do_inner_motion can't just call vc.do_state(motion_dispatch_d,'motion') because vc.do_inner_motion must call the after-motion callback, vc.motion_func. No big deal, except...
4. At present, vc.done calls vc.reinit_ivars, and that caused problems for vc.do_inner_motion because vc.reinit_ivars wiped out the ivars needed when calling the motion callback, including vc.motion_func itself!
The temporary hack was simply to save/restore the needed ivars in vc.do_inner_motion. Happily, this morning I saw a cleaner, more explicit way to init ivars if and when needed. This will simplify the init code, and more importantly, clarify exactly what is happening and why.
To recap: the problem is that vc.reinit_ivars clears too many ivars. Furthermore, the dot ivars are handled as special cases. The solution is to define the following methods:
- vc.init_dot_vars()
- vc.init_motion_vars()
- vc.init_state_vars()
- vc.init_persistent_vars()
Doh! We have now clearly grouped *all* ivars into *disjoint* classes. We can now define the following:
def init_all_ivars(vc):
'''Init all vim-mode ivars. Called only from the ctor.'''
vc.init_dot_vars()
vc.init_motion_vars()
vc.init_state_vars()
vc.init_persistent_vars()
def init_ivars_after_quit(vc):
'''Init vim-mode ivars after the keyboard-quit command.'''
vc.init_motion_vars()
vc.init_state_vars()
def init_ivars_after_done(vc):
'''Init vim-mode ivars when a command completes.'''
if not vc.in_motion:
vc.init_motion_ivars()
vc.init_state_ivars()
**Important**: it would be very bad design to call vc.init_ivars_after_quit inside vc.init_ivars_after_done just because (for now!) the effect would be the same. Glen Meyers calls reusing common code "code-level" binding. It can cause all kinds of problems.
Instead, we want what Meyers calls "functional" binding. The (proper) code shown above makes vc.init_after_quit and vc.init_after_done independent of each other, regardless of what happens in future.
So this is good. We have now created a higher-level grouping of ivars that makes clear what is intended. This scheme is simple and flexible: new (disjoint!) groups of ivars can be created at any time. The new scheme is yet another example of using abstraction as a design and coding tool.
===== Conclusions
Leo's vim code is now spectacularly different, both visually and functionally, from the real vim's code. Wherever possible, Leo uses methods to hide the blah, blah, blah of implementation details. Imo, the result is *far* better design and code than vim's.
Imo, if vim were recreated today from scratch, it would be reasonable to use Leo's vim-mode code as a starting point.
Edward
P.S. To emphasize what I said before in another post: Python encourages simplifications that are, in practice, denied to C programmers. Leo's dispatch dicts for vim mode are an example. Creating such dicts is difficult in plain C, and non-trivial in C++. So C programmers don't create those dicts and miss all the simplifying possibilities afforded by them. There are many other ways that Python aids simplification while C inhibits it.
EKR
As discussed in another thread, rev ebacdc1... puts this scheme into practice.
Here, I'll discuss these changes in detail, including complications that I discovered yesterday and further simplifications that I'll make today.
1. In general, ending each command handler with an acceptance method has been a great success. Acceptance methods clearly indicate what should happen when a command handler ends.
Acceptance handlers hide the existence of the vc.next_func and vc.return_value ivars from all command handlers. This makes each command handler much cleaner in appearance. Furthermore, it is now possible to tweak acceptance methods without changing command handlers in any way.
In other words, even though the acceptance methods follow the principle, "explicit is better than implicit", they *also* hide implementation details in a most useful way. It's the best of both "implicit" and "explicit".
2. Besides the **direct** acceptance methods, vc.accept, vc.delegate or vc.ignore, vc.done & vc.quit, command handlers can also end with **indirect** acceptance methods: vc.begin_insert_mode, vc.begin_motion, vc.end_insert_mode, and vc.vim_digits. Indirect acceptance methods (eventually!) call direct acceptance methods.
**Important**: command handlers can use different acceptance methods (direct or indirect) in different branches of their code, provided that all branches end in a call to exactly one acceptance method. In practice, checking this requirement is easy. Furthermore, the checking code in vc.do_state will warn if this requirement has not been met.
3. I did get one surprise: vc.do_inner_motion can't just call vc.do_state(motion_dispatch_d,'motion') because vc.do_inner_motion must call the after-motion callback, vc.motion_func. No big deal, except...
4. At present, vc.done calls vc.reinit_ivars, and that caused problems for vc.do_inner_motion because vc.reinit_ivars wiped out the ivars needed when calling the motion callback, including vc.motion_func itself!
The temporary hack was simply to save/restore the needed ivars in vc.do_inner_motion. Happily, this morning I saw a cleaner, more explicit way to init ivars if and when needed. This will simplify the init code, and more importantly, clarify exactly what is happening and why.
To recap: the problem is that vc.reinit_ivars clears too many ivars. Furthermore, the dot ivars are handled as special cases. The solution is to define the following methods:
- vc.init_dot_vars()
- vc.init_motion_vars()
- vc.init_state_vars()
- vc.init_persistent_vars()
Doh! We have now clearly grouped *all* ivars into *disjoint* classes. We can now define the following:
def init_all_ivars(vc):
'''Init all vim-mode ivars. Called only from the ctor.'''
vc.init_dot_vars()
vc.init_motion_vars()
vc.init_state_vars()
vc.init_persistent_vars()
def init_ivars_after_quit(vc):
'''Init vim-mode ivars after the keyboard-quit command.'''
vc.init_motion_vars()
vc.init_state_vars()
def init_ivars_after_done(vc):
'''Init vim-mode ivars when a command completes.'''
if not vc.in_motion:
vc.init_motion_ivars()
vc.init_state_ivars()
**Important**: it would be very bad design to call vc.init_ivars_after_quit inside vc.init_ivars_after_done just because (for now!) the effect would be the same. Glen Meyers calls reusing common code "code-level" binding. It can cause all kinds of problems.
Instead, we want what Meyers calls "functional" binding. The (proper) code shown above makes vc.init_after_quit and vc.init_after_done independent of each other, regardless of what happens in future.
So this is good. We have now created a higher-level grouping of ivars that makes clear what is intended. This scheme is simple and flexible: new (disjoint!) groups of ivars can be created at any time. The new scheme is yet another example of using abstraction as a design and coding tool.
===== Conclusions
Leo's vim code is now spectacularly different, both visually and functionally, from the real vim's code. Wherever possible, Leo uses methods to hide the blah, blah, blah of implementation details. Imo, the result is *far* better design and code than vim's.
Imo, if vim were recreated today from scratch, it would be reasonable to use Leo's vim-mode code as a starting point.
Edward
P.S. To emphasize what I said before in another post: Python encourages simplifications that are, in practice, denied to C programmers. Leo's dispatch dicts for vim mode are an example. Creating such dicts is difficult in plain C, and non-trivial in C++. So C programmers don't create those dicts and miss all the simplifying possibilities afforded by them. There are many other ways that Python aids simplification while C inhibits it.
EKR
Reply all
Reply to author
Forward
0 new messages