Dear Bram

1632 views
Skip to first unread message

graywh

unread,
Feb 4, 2011, 11:05:34 AM2/4/11
to vim_dev
Please fix vim's input queue mechanism with key info structures a.la
libtermkey so that LeoNerd can fix terminal input and Gvim and
everyone will be happy. Thanks.

Erik Falor

unread,
Feb 4, 2011, 11:17:36 AM2/4/11
to vim...@googlegroups.com

I second this request.

--
Erik Falor
Registered Linux User #445632 http://counter.li.org

Bram Moolenaar

unread,
Feb 4, 2011, 1:14:29 PM2/4/11
to graywh, vim_dev

Graywh wrote:

Send me a patch and I'll look into it.

If you want a more useful response, use your real name and include more
information about what you actually want, what system, etc.

--
hundred-and-one symptoms of being an internet addict:
195. Your cat has its own home page.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Paul LeoNerd Evans

unread,
Feb 17, 2011, 11:48:16 AM2/17/11
to vim...@googlegroups.com, graywh
On Fri, Feb 04, 2011 at 07:14:29PM +0100, Bram Moolenaar wrote:
> Graywh wrote:
>
> > Please fix vim's input queue mechanism with key info structures a.la
> > libtermkey so that LeoNerd can fix terminal input and Gvim and
> > everyone will be happy. Thanks.
>
> Send me a patch and I'll look into it.
>
> If you want a more useful response, use your real name and include more
> information about what you actually want, what system, etc.

Hi Bram,

I would like this. For all systems. Everywhere.

This is a large undertaking. This isn't a "send me a patch" request.

I am talking about ripping out the byte-queue input system and replacing
it with a structured keypress queue. The lot. Everything. Remove the
"stuck in the 1970s all the world is a DEC-style glass virtual terminal"
input queue, and replace it with one that has structures; remembers
modifiers, etc.. With a structure we can properly represent "Ctrl-I" as
being the letter I with a control key, totally separate from the Tab.

Having done this, we can feed in the input events from a terminal in
EXACTLY the same way it currently works. We can take events out to match
them up to map/etc.. much neater. We can feed events in from GTK or
similar GUIs, without having to go near legacy 1970s terminal
technology, meaning that GTK (et.al) users can map Tab, Ctrl-I and
Ctrl-Shift-I, all independently of each other.

Having done this, I can then quite easily send you a patch to use
libtermkey if that is wanted, because then it will be able to recognise
Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
properly-configured xterm. But that is PURELY an extension mechanism. My
primary request is to fix the underlying queue mechanism.

This is an issue that has been dragging on for years now - 7 or 8 years
by my count. 7 or 8 years I have been sitting in #vim on Freenode; in
all that time almost every week we'll get someone or other asking how
they can map some key or other, that we have to keep explaining "No,
sorry, you can't. Not even in GTK.".

I am happy to sit and discuss it with you for potentially hours if
necessary. What I cannot do is "send a patch" - it's not a small simple
few-lines fix like that.

I think it is about time we ended this unfortunate situation, and
properly fixed the real underlying mechanism.

Yours, with my real name,

--
Paul "LeoNerd" Evans

leo...@leonerd.org.uk
ICQ# 4135350 | Registered Linux# 179460
http://www.leonerd.org.uk/

signature.asc

Ingo Karkat

unread,
Feb 17, 2011, 12:03:31 PM2/17/11
to vim...@googlegroups.com
On 17-Feb-2011 17:48, Paul LeoNerd Evans wrote:
> On Fri, Feb 04, 2011 at 07:14:29PM +0100, Bram Moolenaar wrote:
>>
>> Send me a patch and I'll look into it.
>>
>
> This is a large undertaking. This isn't a "send me a patch" request.

I would love to have this, too, if only because after so many happy years with
Vim (a heartfelt thanks to Bram and all developers!), I'm running out of keys
for my custom mappings :-)

Though I'm too ignorant of the internals (and can therefore only offer to do
some testing), I agree that this probably cannot be done in a single patch. But,
with the Mercurial repository, it should be easy to create a feature branch that
allows devs to collaborate and test this, and only have this merged in after
it's done (hopefully in time for Vim 7.4 / 8.0?).

-- regards, ingo

Bram Moolenaar

unread,
Feb 17, 2011, 2:14:05 PM2/17/11
to Paul LeoNerd Evans, vim...@googlegroups.com, graywh

Paul Evans wrote:

> I would like this. For all systems. Everywhere.
>
> This is a large undertaking. This isn't a "send me a patch" request.
>
> I am talking about ripping out the byte-queue input system and replacing
> it with a structured keypress queue. The lot. Everything. Remove the
> "stuck in the 1970s all the world is a DEC-style glass virtual terminal"
> input queue, and replace it with one that has structures; remembers
> modifiers, etc.. With a structure we can properly represent "Ctrl-I" as
> being the letter I with a control key, totally separate from the Tab.
>
> Having done this, we can feed in the input events from a terminal in
> EXACTLY the same way it currently works. We can take events out to match
> them up to map/etc.. much neater. We can feed events in from GTK or
> similar GUIs, without having to go near legacy 1970s terminal
> technology, meaning that GTK (et.al) users can map Tab, Ctrl-I and
> Ctrl-Shift-I, all independently of each other.

You can just keep the existing queue and make this work. It actually
already works for most keys, using a modifier sequence. Using
structures only makes it bigger.

Main problem is that this won't be backwards compatible, currently
CTRL-I is the same as Tab. I don't know how to make this work without
breaking backwards compatibilty.

> Having done this, I can then quite easily send you a patch to use
> libtermkey if that is wanted, because then it will be able to recognise
> Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
> properly-configured xterm. But that is PURELY an extension mechanism. My
> primary request is to fix the underlying queue mechanism.
>
> This is an issue that has been dragging on for years now - 7 or 8 years
> by my count. 7 or 8 years I have been sitting in #vim on Freenode; in
> all that time almost every week we'll get someone or other asking how
> they can map some key or other, that we have to keep explaining "No,
> sorry, you can't. Not even in GTK.".

Actually, in the GUI many key combinations do work. Not all though.

> I am happy to sit and discuss it with you for potentially hours if
> necessary. What I cannot do is "send a patch" - it's not a small simple
> few-lines fix like that.
>
> I think it is about time we ended this unfortunate situation, and
> properly fixed the real underlying mechanism.
>
> Yours, with my real name,
>
> --
> Paul "LeoNerd" Evans

--
A cow comes flying over the battlements, lowing aggressively. The cow
lands on GALAHAD'S PAGE, squashing him completely.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

Davertron

unread,
Feb 17, 2011, 12:04:19 PM2/17/11
to vim_dev
Please, please, PLEASE make this happen.

On Feb 17, 11:48 am, Paul LeoNerd Evans <leon...@leonerd.org.uk>
wrote:
> leon...@leonerd.org.uk
> ICQ# 4135350       |  Registered Linux# 179460http://www.leonerd.org.uk/
>
>  signature.asc
> < 1KViewDownload

Paul LeoNerd Evans

unread,
Feb 17, 2011, 5:07:55 PM2/17/11
to vim...@googlegroups.com, graywh
On Thu, Feb 17, 2011 at 08:14:05PM +0100, Bram Moolenaar wrote:
> You can just keep the existing queue and make this work. It actually
> already works for most keys, using a modifier sequence. Using
> structures only makes it bigger.

OK, well which ever way works.. I'm not fussy about the internals of
implementation. So long as it is capable of knowing that Enter is not
Ctrl-M is not Ctrl-Shift-M, I am happy.

> Main problem is that this won't be backwards compatible, currently
> CTRL-I is the same as Tab. I don't know how to make this work without
> breaking backwards compatibilty.

:set oldterminalmode

:map <Ctrl-I> <Tab>
:map <Ctrl-M> <CR>

For that matter, make it default to this 1970s-like legacy behaviour and
if people want the ability to map newly-recognised keys,

:set newterminalmode

> Actually, in the GUI many key combinations do work. Not all though.

Well, then lets make them all work. Please?

Seriously - I am not the only person here. There is a constant stream of
users on #vim, every week you can guarantee someone will want to map
Ctrl-I or Shift-Space or Shift-Enter or some other key combination that
seems perfectly logical to any outside observer, to want to map. Every
week we have to tell them "no". I'm getting tired of it, when there
isn't any technical reason why it can't be done, only "it hasn't been
done this way so far".

So can we please have a plan? If you care deeply about retaining
1970s-like semantics, I'd be happy for a compromise solution. Some
boolean setting that defaults to a value whereby these aliases do occur,
where Ctrl-I and Ctrl-Shift-I -are- synonymous with Tab. But implemented
in a way where I or whichever knowledgable user can say "Yes vim, I know
that Ctrl-I used to mean Tab in the 1970s but I would like an extra
Ctrl key now and 'I' looks a good choice - let me have it". And then I
could toggle setting, and map it. Yes, I'm aware that not -every-
terminal can represent that. Yes I'm aware if I ever go use a real DEC
VT220 glass teletype I've now lost the ability to type it. But that
should be -my- problem - right now vim pointlessly restricts a whole
selection of possible key mappings from GTK or Mac OS X or Win32 or
proper xterms or all sorts of other places, simply for the
backward-looking legacy of retaining compatibility with what the
behaviour used to be in a time before many vim users were even born.

I think it's about time we opened this up a bit, and allowed users to
map the keypresses they want to.

signature.asc

Paul LeoNerd Evans

unread,
Feb 17, 2011, 6:08:12 PM2/17/11
to vim...@googlegroups.com, graywh
On Thu, Feb 17, 2011 at 08:14:05PM +0100, Bram Moolenaar wrote:
> You can just keep the existing queue and make this work. It actually
> already works for most keys, using a modifier sequence. Using
> structures only makes it bigger.

Oh, and I'm not sure I follow your logic here.

Yes, it would make it bigger. By maybe bytes. Consider

struct keypress {
uint16_t flags; /* Unicode or special */
uint16_t modifiers; /* A bitmask */
uint32_t codepoint;
};

that's 8 bytes. As compared a single byte keypress that's 7 bytes
bigger. Less so, compared to e.g. a 3 or 4 byte modifier prefixed
sequence. Not a massive growth here. Lets be really pessimistic and say
we're using 7 bytes extra. Lets now say we're really memory-constrained,
and can only afford a single extra kilobyte of memory. That's still 146
keypresses of queue we can afford. Is vim's queue that big? Can we
really afford no more than a single kilobyte?

Ofcourse, this was a simplistic structure. If we care more about saving
memory at a slight CPU overhead we could

struct keypress {
unsigned int is_special : 1;
unsigned int modifiers : 10;
unsigned int codepoint : 21;
};

that's 4 bytes, to store the same information, noting that Unicode is
only a 21bit code space, a Unicode/special flag, and storage space for
10 modifiers.

4 bytes.

Those variable-length-encoded prefixed keypresses are getting on for
that long anyway. Plus now we're in a fixed-length encoding, so perhaps
we've saved CPU time and associated code space to store the program used
to encode and decode them.

I'm not sure an argument of memory consumption can be made against using
this encoding scheme.

Plus, it's only the queue of incoming keypresses - that queue isn't
going to stay very big for very long.

signature.asc

Ben Schmidt

unread,
Feb 18, 2011, 12:12:47 AM2/18/11
to vim...@googlegroups.com
> Plus, it's only the queue of incoming keypresses - that queue isn't
> going to stay very big for very long.

It's not just the input queue that's in question here, it's everywhere
in Vim where keypresses are represented. For instance, the right hand
sides of mappings are not primarily characters, but lists of keypresses.
They need the same amount of expressive power to work properly.

When macros are recorded, registers, which usually are primarily lists
of characters, are used to store keypresses. Likewise, for feedkeys() to
work, its input, a string, needs to be able to represent keypresses.

And I'm sure there are plenty more subtleties.

The bottom line, though, is that changing to a struct-based approach
could make the job absolutely huge, requiring reworking and/or
redesigning how maps, registers, etc. all work. And it might not even be
possible since, e.g. registers need to be able to do both characters and
keys. It would be much simpler to extend the current approach which uses
an 'escape mechanism' for representing special keys. Just hit qq and
type some arrow keys, q to end recording, and then inspect register q
and you'll see what's going on here. Vim uses a byte <80> followed by
two more bytes to represent special keys.

IMHO, what needs to be done is for this to be tidied up and made
consistent and documented so that these codes can be more readily
generated, interpreted, and even viewed and understood by users (e.g.
when I do :registers, it would be nice to see <S-Up> rather than
<80><fd>^D, when appropriate). Included in this is a clear specification
of what control characters mean, e.g. does ^I (<09>) mean tab or
control-I? And which of those keys can or does use the escape mechanism?
Then the behaviour of mappings needs to be defined--if there is a
mapping for ^I (<09>) and I push tab, will it be triggered? If in a
terminal which can't distinguish control-I and tab, and a ^I is
received, should the mapping for Tab or control-I be triggered? If
there's a mapping for ^I as well as Tab, which has precedence? All these
kinds of questions need clear answers, and sensible specifications and
design need to address them, avoid ambiguity, and take care to require
as little as possible work for users, plugin authors, etc. to update
their code and mappings.

Then the input code needs to be reworked in all the GUIs and in the
terminal handling to generate the appropriate internal codes,
consistently across all the different GUIs, etc., in line with that
specification/design.

Ben.

Adrien "Axioplase" Piérard

unread,
Feb 18, 2011, 1:04:05 AM2/18/11
to vim...@googlegroups.com
> It's not just the input queue that's in question here, it's everywhere
> in Vim where keypresses are represented.

> And I'm sure there are plenty more subtleties.

What I see here, if done, is a potential new major version of vim.
Such a new version may trade off backward compatibility for up-to-date
technology and/or design constraints.

Since the changes may be huge, I concur with the suggestion of
creating a branch, and having discussions *and* documentation on its
evolution.
You raised some very interesting questions, and I hope that other
people who know about the internals give their point of view too.

As far as memory usage is concerned, I am now very egoistic and
believe that I have enough RAM to run vim. If I don't, then I'll use
vi, or an old version of vim.
I used to collect and use old hardware, but I'm now getting my job
done and bringing home the bacon thanks to vim on machines with more
memory that I could think of not so long ago…

Anyway,

Cheers,

P!

--
Français, English, 日本語, 한국어

Stephen Lee

unread,
Feb 17, 2011, 5:15:08 PM2/17/11
to vim_dev
Hello I just join discussion because I love vim and want to give a
small suggestion:

Is it possible to put a parameter in .vimrc like:

set keymap=byte-queue, structured, etc..
(default is byte-queue)
so that both sets of mapping could be kept...

sorry if I see the problem too surface.
>  /// Bram Moolenaar -- B...@Moolenaar.net --http://www.Moolenaar.net  \\\
> ///        sponsor Vim, vote for features --http://www.Vim.org/sponsor/\\\
> \\\  an exciting new programming language --http://www.Zimbu.org       ///

Stephen Lee

unread,
Feb 17, 2011, 6:11:32 PM2/17/11
to vim_dev
Would it be better to have a new folder called "Keyboard", putting all
the key mapping definition files like dvorak.vim, 101-keyboard.vim,
etc. and let people set in .vimrc?

It may be more flexible than just new or old term mode...

Philippe Vaucher

unread,
Feb 18, 2011, 8:35:38 AM2/18/11
to vim_dev
FWIW, you have my vote for making this happen. I think it's not that
hard to make everyone happy on this subject by the mean of .vimrc
settings, like we have between vi and vim for :set compatible.

Wether we use structures or the byte 80 for special key is
implementation detail, I mean wether macros, mappings or registers
simply store a list of bytes or a list of structures shouldn't change
anything: you just need an explicit mapping between a displayed
keypress and a byte or a struct (<S-Up> can map internally to \x80\xFD
\xAA or { 1, MOD_SHIFT, KEY_UP } and it shouldn't matter).

Structures just look more sane as they're more generic/expressive and
accurate about the problem, they also are easier to expand on.

Philippe

Paul LeoNerd Evans

unread,
Feb 18, 2011, 10:52:00 AM2/18/11
to vim...@googlegroups.com
On Fri, Feb 18, 2011 at 05:35:38AM -0800, Philippe Vaucher wrote:
> FWIW, you have my vote for making this happen. I think it's not that
> hard to make everyone happy on this subject by the mean of .vimrc
> settings, like we have between vi and vim for :set compatible.

:set compatible does suggest the possibility for more similar options,
ways to optionally extend the featureset beyond its existing capability,
without upsetting people who rely on the current behaviour, limits and
all.

> Wether we use structures or the byte 80 for special key is
> implementation detail, I mean wether macros, mappings or registers
> simply store a list of bytes or a list of structures shouldn't change
> anything: you just need an explicit mapping between a displayed
> keypress and a byte or a struct (<S-Up> can map internally to \x80\xFD
> \xAA or { 1, MOD_SHIFT, KEY_UP } and it shouldn't matter).
>
> Structures just look more sane as they're more generic/expressive and
> accurate about the problem, they also are easier to expand on.

Another thought occurs to me: UTF-8 sequences vs. Alt keys.

I don't know if it's an accident of implementation, but I have never
been able to get them all working. Lets consider this example

é is U+00e9 in Unicode, encodes to two UTF-8 bytes C3 A9

C is U+0043 in Unicode, encodes to one UTF-8 byte 43

If vim simply uses an 8th-bit-high to indicate Alt+ prefix on simple
ASCII, then what does

C3

mean? Is it the first of two UTF-8 bytes, or is it Alt+C ?

Alternatively, perhaps we'll represent it as a two-byte sequence with an
<Escape> prefix. Now we've conflated someone typing <Escape> C from
someone typing Alt+C. Again, in the 1970s that may be what it was, but
these days it's far from clear from GUIs, that this behaviour is what's
wanted. Again, a simple 'set' option should default this:

:set noaltisescape (defaults to :set altisescape)


The reason I suggested a simple neat orthogonal structure representation
is that it can disambiguate all of the following cases in a clear
logical way; a way that can be easily fed from GUI events, and fed into
map tables or other internals.

Tab { 1, 0, KEY_TAB }
Ctrl-I { 0, MOD_CTRL, 'i' }
Ctrl-Shift-I { 0, MOD_CTRL, 'I' }

Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
Alt+C { 0, MOD_ALT, 'C' }
é { 0, 0, 0xe9 }

signature.asc

Paul LeoNerd Evans

unread,
Feb 18, 2011, 11:15:10 AM2/18/11
to vim...@googlegroups.com
On Fri, Feb 18, 2011 at 04:12:47PM +1100, Ben Schmidt wrote:
> >Plus, it's only the queue of incoming keypresses - that queue isn't
> >going to stay very big for very long.
>
> It's not just the input queue that's in question here, it's everywhere
> in Vim where keypresses are represented. For instance, the right hand
> sides of mappings are not primarily characters, but lists of keypresses.
> They need the same amount of expressive power to work properly.

Yes. And the scheme I created is no less expressive than the byte
queues. In fact it is more expressive, able to disambiguate neatly
situations that cannot normally be represented - see also my other mail
on the thread a few minutes ago.

http://groups.google.com/group/vim_dev/msg/7bb4cc9b127cddb3

> When macros are recorded, registers, which usually are primarily lists
> of characters, are used to store keypresses. Likewise, for feedkeys() to
> work, its input, a string, needs to be able to represent keypresses.
>
> And I'm sure there are plenty more subtleties.

Yes - again, my point of these structures is that they exactly _do_
represent these keypresses.

> The bottom line, though, is that changing to a struct-based approach
> could make the job absolutely huge, requiring reworking and/or
> redesigning how maps, registers, etc. all work. And it might not even be
> possible since, e.g. registers need to be able to do both characters and
> keys.

But every character -can- be represented as a key - namely, the key that
generates it. Let me again repeat my structure:

struct keypress {
unsigned int is_special : 1;
unsigned int modifiers : 10;
unsigned int codepoint : 21;
};

Any character has a Unicode codepoint (cp). That's represented by

{ 0, 0, cp }

Any keypress has a symbolic key number, taken from some arbitrary
enumeration - I care not where.

{ 1, mod, key }

Now, we can also represent those modified Unicode keys such as Ctrl-I
and Shift-Space which previously were impossible:

{ 0, MOD_CTRL, 'i' } /* Ctrl-I */
{ 0, MOD_SHIFT, ' ' } /* Shift-Space */

> It would be much simpler to extend the current approach which uses
> an 'escape mechanism' for representing special keys. Just hit qq and
> type some arrow keys, q to end recording, and then inspect register q
> and you'll see what's going on here. Vim uses a byte <80> followed by
> two more bytes to represent special keys.
>
> IMHO, what needs to be done is for this to be tidied up and made
> consistent and documented so that these codes can be more readily
> generated, interpreted, and even viewed and understood by users (e.g.
> when I do :registers, it would be nice to see <S-Up> rather than
> <80><fd>^D, when appropriate). Included in this is a clear specification
> of what control characters mean, e.g. does ^I (<09>) mean tab or
> control-I? And which of those keys can or does use the escape mechanism?

OK - well, if you feel confident that the existing prefix-escape
mechanism can completely an unambiguously represent all these possible
keystrokes, then sure, that way might result in less code change
overall.

But do make quite sure it can represent them. Specifically consider the
two tricky special-cases I suggested - repeated here again:

Tab { 1, 0, KEY_TAB }
Ctrl-I { 0, MOD_CTRL, 'i' }
Ctrl-Shift-I { 0, MOD_CTRL, 'I' }

Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
Alt+C { 0, MOD_ALT, 'C' }
é { 0, 0, 0xe9 }

> Then the behaviour of mappings needs to be defined--if there is a


> mapping for ^I (<09>) and I push tab, will it be triggered? If in a
> terminal which can't distinguish control-I and tab, and a ^I is
> received, should the mapping for Tab or control-I be triggered? If
> there's a mapping for ^I as well as Tab, which has precedence?

I propose a pair of boolean settings, with the following defaults:

:set nomodifiedunicode
:set altisescape

Under these settings,

:map <Ctrl-I> ... vs :map <Tab> ... shall have the same effect, each
overwriting the effects of the other; last one wins.

Pressing the Tab key or Ctrl-I shall both invoke the last mapping
registered.

:map <Escape>C ... vs :map <Alt+C> ... shall behave similarly, each
overwrites the other. Typing either key sequence will invoke the
last map to be registered.

If a user decides "I want to use those extra Ctrl- keys", they can set
in their .vimrc

:set modifiedunicode

At this point,

:map <Ctrl-I> and :map <Tab> shall fill two -different-
slots of the mapping list, and typing either key will activate the
indicated mapping.

If a user further decides "I want the Alt modifier to not mean Escape
prefixing", then

:set noaltisescape

At which point,

:map <Escape>C and :map <Alt+C> shall also be different.

The only remaining ambiguity to be answered is, what happens in the
following case:

:set nomodifiedunicode
:map <Tab> ONE
:map <Ctrl-I> TWO
:set modifiedunicode

now press Tab or Ctrl-I

I don't have an easy answer for this case; no particular behaviour jumps
out at me as "obviously correct".

> All these
> kinds of questions need clear answers, and sensible specifications and
> design need to address them, avoid ambiguity, and take care to require
> as little as possible work for users, plugin authors, etc. to update
> their code and mappings.
>
> Then the input code needs to be reworked in all the GUIs and in the
> terminal handling to generate the appropriate internal codes,
> consistently across all the different GUIs, etc., in line with that
> specification/design.

I propose that the structure approach makes this simpler. It is now
clearly obvious how e.g. GTK should fill in these structures, as GDK
keypress events already have unicode / modifier bits or symbolic keys.
It's also obvious how to render such a structure as a string, possibly
wrapping <> around it, possibly prefixing C- or A- or whatever as
appropriate.

(( This sort of behaviour is already implemented by libtermkey, even
going so far as to support

char buffer[256];
termkey_snprint_key(tk, buffer, sizeof buffer, &key, TERMKEY_FORMAT_VIM);

for exactly this purpose. :) ))

signature.asc

Ben Schmidt

unread,
Feb 19, 2011, 10:03:58 AM2/19/11
to vim...@googlegroups.com, Paul LeoNerd Evans
>>> Plus, it's only the queue of incoming keypresses - that queue isn't
>>> going to stay very big for very long.
>>
>> It's not just the input queue that's in question here, it's everywhere
>> in Vim where keypresses are represented. For instance, the right hand
>> sides of mappings are not primarily characters, but lists of keypresses.
>> They need the same amount of expressive power to work properly.
>
> Yes. And the scheme I created is no less expressive than the byte
> queues. In fact it is more expressive, able to disambiguate neatly
> situations that cannot normally be represented - see also my other mail
> on the thread a few minutes ago.

Yeah, there was never anything lacking in your scheme in terms of
expressive power. It's just that it's not only the input queue that
would need to use it. It's a lot more.

>> When macros are recorded, registers, which usually are primarily lists
>> of characters, are used to store keypresses. Likewise, for feedkeys() to
>> work, its input, a string, needs to be able to represent keypresses.
>>
>> And I'm sure there are plenty more subtleties.
>
> Yes - again, my point of these structures is that they exactly _do_
> represent these keypresses.

Yes. But what happens when you then edit that macro by putting the
register into a buffer, changing it, and yanking it again? This is not
uncommonly done. How should the registers be stored in .viminfo? How do
you write the input to the feedkeys function as a string in vimscript?
Etc.. These are the kinds of issues I was trying to raise.

>> The bottom line, though, is that changing to a struct-based approach
>> could make the job absolutely huge, requiring reworking and/or
>> redesigning how maps, registers, etc. all work. And it might not even be
>> possible since, e.g. registers need to be able to do both characters and
>> keys.
>
> But every character -can- be represented as a key - namely, the key that
> generates it.

When there is one. Or a codepoint in some encoding. Yeah, your struct
does allow for that. The struct isn't inexpressive, it's just that it's
big, and if it needs to be used in a LOT of places (which seem to just
be increasing...input queue, mappings, registers, strings, buffers?!),
suddenly not just one thing is getting a bit bigger--a lot of things are
getting bigger--potentially a lot bigger! If every time you yank text
into a register it gets 12 times larger, that may cause some problems!

> Let me again repeat my structure:
>
> struct keypress {
> unsigned int is_special : 1;
> unsigned int modifiers : 10;
> unsigned int codepoint : 21;
> };
>
> Any character has a Unicode codepoint (cp). That's represented by

Well, a codepoint in whatever encoding you are using.

> { 0, 0, cp }
>
> Any keypress has a symbolic key number, taken from some arbitrary
> enumeration - I care not where.
>
> { 1, mod, key }
>
> Now, we can also represent those modified Unicode keys such as Ctrl-I
> and Shift-Space which previously were impossible:
>
> { 0, MOD_CTRL, 'i' } /* Ctrl-I */
> { 0, MOD_SHIFT, ' ' } /* Shift-Space */

Yeah. The benefit here is that different keyboard layouts can be
represented, e.g. keyboards which have single keys for accented
characters can represent those keys with modifiers, easily.

> OK - well, if you feel confident that the existing prefix-escape
> mechanism can completely an unambiguously represent all these possible
> keystrokes, then sure, that way might result in less code change
> overall.

I think with some careful design it could, which may well require some
reworking of how it's currently done. There is the issue, of course,
that <80> is a valid character in some encodings, too, and I don't know
if this is accounted for. It needs to be. So maybe the escape mechanism
needs to be a bit more formal (if it can be--but maybe there's no way
around this, or maybe that's a longer-term consideration). Definitely
the issues have to be bashed out. I still think this would be the
easiest and best way forward, though. But it's far from my call!

Regarding that whole meta issue that you raised earlier--I think, yes,
using an 8-bit-high representation for meta is completely out of the
question. Part of the input code should be transforming that to a proper
Vim-internal escape sequence for terminals that use it. Same for other
things. The input code should transform input into Vim's internal
representation which should be carefully designed not to be ambiguous,
etc., but which, nevertheless, is at its most fundamental, a byte stream
so it can be used in the input queue, mappings, registers, strings and
even buffers.

> But do make quite sure it can represent them. Specifically consider the
> two tricky special-cases I suggested - repeated here again:
>
> Tab { 1, 0, KEY_TAB }
> Ctrl-I { 0, MOD_CTRL, 'i' }
> Ctrl-Shift-I { 0, MOD_CTRL, 'I' }
>
> Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
> Alt+C { 0, MOD_ALT, 'C' }
> é { 0, 0, 0xe9 }

Yes. This all has to be carefully accounted for.

Maybe the way to do it is to have printable characters represented just
as printable characters, 'default' control characters optionally just as
themselves (e.g. <09> for tab), to use when a terminal that can't
distinguish between this and Ctrl-I is used, and (whenever possible),
use <80>+flag_byte+character for control/special keys. One of the
flag_byte flags could be a 'special' flag, which means the 'character'
in the third byte would be some kind of mnemonic for the key (e.g. u for
the up arrow). When the key is not 'special' the character it produces
would be the third byte (and perhaps later for multibyte characters,
e.g. if ctrl is used with an accented character on keyboards that have a
specific key for this). How all this works depends a bit on how the
keyboard interfacing works--maybe it should just use keycodes if
characters are not readily obtainable. And maybe you need two flag bytes
if there are more than 7 modifier keys that need to be distinguished.

>> Then the behaviour of mappings needs to be defined--if there is a
>> mapping for ^I (<09>) and I push tab, will it be triggered? If in a
>> terminal which can't distinguish control-I and tab, and a ^I is
>> received, should the mapping for Tab or control-I be triggered? If
>> there's a mapping for ^I as well as Tab, which has precedence?
>
> I propose a pair of boolean settings, with the following defaults:
>
> :set nomodifiedunicode
> :set altisescape
>
> Under these settings,
>
> :map<Ctrl-I> ... vs :map<Tab> ... shall have the same effect, each
> overwriting the effects of the other; last one wins.
>
> Pressing the Tab key or Ctrl-I shall both invoke the last mapping
> registered.
>
> :map<Escape>C ... vs :map<Alt+C> ... shall behave similarly, each
> overwrites the other. Typing either key sequence will invoke the
> last map to be registered.
>
> If a user decides "I want to use those extra Ctrl- keys", they can set
> in their .vimrc
>
> :set modifiedunicode
>
> At this point,
>
> :map<Ctrl-I> and :map<Tab> shall fill two -different-
> slots of the mapping list, and typing either key will activate the
> indicated mapping.

What should Vim do in a terminal that can't distinuish between those
keys when this option is in effect? This is the same question as below,
really.

Maybe we need an annotation to the mapping, like <default> or something
that says "this mapping is acceptable to use when we're not sure if this
exact combo was pressed, but we got a code that 'could be it' in some
terminal."

> If a user further decides "I want the Alt modifier to not mean Escape
> prefixing", then
>
> :set noaltisescape
>
> At which point,
>
> :map<Escape>C and :map<Alt+C> shall also be different.

I think there might already be an option for this. Or something in
termcap? I seem to remember reading some stuff about it in the Vim
manual. I guess, though, the 'non-alt prefixing' of Vim probably means
to use 8-bit-high for meta, which we already said, is pretty useless.

I'm not sure two options are needed. Probably a single option suffices
to deal with both situations.

> The only remaining ambiguity to be answered is, what happens in the
> following case:
>
> :set nomodifiedunicode
> :map<Tab> ONE
> :map<Ctrl-I> TWO
> :set modifiedunicode
>
> now press Tab or Ctrl-I
>
> I don't have an easy answer for this case; no particular behaviour jumps
> out at me as "obviously correct".

I agree. If we go with the <default> annotation to mappings (preferably
with a better keyword), I guess that is the one that wins and takes
precedence. It may not even need to delete the other ones, so both slots
could remain active, but it would just run the 'default' one.

>> All these
>> kinds of questions need clear answers, and sensible specifications and
>> design need to address them, avoid ambiguity, and take care to require
>> as little as possible work for users, plugin authors, etc. to update
>> their code and mappings.
>>
>> Then the input code needs to be reworked in all the GUIs and in the
>> terminal handling to generate the appropriate internal codes,
>> consistently across all the different GUIs, etc., in line with that
>> specification/design.
>
> I propose that the structure approach makes this simpler. It is now
> clearly obvious how e.g. GTK should fill in these structures, as GDK
> keypress events already have unicode / modifier bits or symbolic keys.
> It's also obvious how to render such a structure as a string, possibly
> wrapping<> around it, possibly prefixing C- or A- or whatever as
> appropriate.

That's good for mappings, and I guess the \< escape in vimscript
strings, but it doesn't help for registers (which can end up in
buffers), etc..

Maybe it's a sensible 'middle point', though, for the input queue. The
GUI specific code could go into a structure like this, which generic
code can then translate into internal byte-stream representation as
appropriate, and then that byte-stream representation is the standard
throughout the rest of Vim. Basically the structure would only be used
as a mechanism to deal with the code that deals with hardware
keypresses, making it consistent between GUIs, platforms, etc..

> (( This sort of behaviour is already implemented by libtermkey, even
> going so far as to support
>
> char buffer[256];

> termkey_snprint_key(tk, buffer, sizeof buffer,&key, TERMKEY_FORMAT_VIM);


>
> for exactly this purpose. :) ))

Ben.

Philippe Vaucher

unread,
Feb 20, 2011, 6:17:47 AM2/20/11
to vim_dev

> Yes. But what happens when you then edit that macro by putting the
> register into a buffer, changing it, and yanking it again? This is not
> uncommonly done. How should the registers be stored in .viminfo? How do
> you write the input to the feedkeys function as a string in vimscript?
> Etc.. These are the kinds of issues I was trying to raise.

Well, I don't see what the problem is? Just do it like today, maybe
with some expansions e.g "<C-A><S-I><Tab>somechars<U-234>". I mean
this is simple key-as-text representation and then text-representation-
to-struct creation no?

For example, the user records a macro into register q that is then
displayed as "ihello<Tab>worlld<Esc><Left><Left>x". He then edits the
macro in a buffer, yanks it into register q, and when @q is executed
all it does is interpret the string?

We'll probably need some escape mechanism to differentiate between the
text "<Tab>" and the tab key, but this looks fairly easy to find no?
Wether it being a special byte before the <> block or some \ or
whatever.

Maybe I'm oversimplifying it but it'd help if you were more specific
about the "issues" the new design have. To me it looks like we just
need to agree on a new set of conventions by describing how the new
system would work for all the current (and future) macros/registers/
whatever scenarios.

I suggest maybe we make some kind of "new design" draft which would
answer all the issues raised (memory size/1970 support/macros
concerns) ?

Philippe

Milan Vancura

unread,
Feb 21, 2011, 4:53:03 AM2/21/11
to vim...@googlegroups.com
> Yes. But what happens when you then edit that macro by putting the
> register into a buffer, changing it, and yanking it again? This is not
> uncommonly done. How should the registers be stored in .viminfo? How do
> you write the input to the feedkeys function as a string in vimscript?
> Etc.. These are the kinds of issues I was trying to raise.

Hi.

Wouldn't be it same as now, only used more often? There is already a
possibility to write "<F4>" or "<S-Space>". The only problem is that, at least
the second, we can't press strongly enough to push it to vim :-)

But, on the other hand, you are right there still will be (and must be)
ambiguities. We can't do anything about that, in general - it's a user who must
decide how does he want to understand his keyboard.

For example: I, as Czech, have some Czech accented letters accessible via
modifier+key on my keyboard. To make the example more specific, think about
Mod5+s as "s with hook" (U0161) and redefined my keymap so Capslock key acts as
Mod5. It's up to me, and only me, if I define some vim mapping as

:map <Mod5+S> ...rhs...
- or -
:map <U0161> ...rhs...

Both do the same on my current keyboard but start to behave differently if I
change my keyboard setting in X window system, of course. If I switched to
another kind of Czech keyboard (called "typewriter one"), <U0161> appears at a
key of "number 3" and Mod5 would be on right Alt or not defined at all.

As I wrote above, we can't do anything about that, as far as I know.

Milan

--
Milan Vancura, Prague, Czech Republic, Europe

Stephen Lee

unread,
Feb 22, 2011, 1:35:40 PM2/22/11
to vim_dev
Therefore I propose letting users choose their preferred keyboard
layouts instead of forcing any specific one to them.

In your case it seems you are talking about QWERTY and QWERTZ (from
wiki), so the following would be keyboard layouts in the new directory
called "Keyboard"

(General keyboard layouts)
QWERTY.vim
QWERTZ.vim
AZERTY.vim
QZERTY.vim
Dvorak.vim
Colemak.vim
JCUKEN.vim
Neo.vim
Turkish.vim
.
.
OR (Keyboard layout by Country Name)





and there would be 4 files:

QWERTYsequence.vim
QWERTZsequence.vim

QWERTYstructured.vim
QWERTZstructured.vim




OR (specific for your country)

CzechQWERTYsequence.vim
CzechQWERTZsequence.vim

CzechQWERTYstructured.vim
CzechQWERTZstructured.vim




OR simply (having advantage of adding new keyboard layouts in future
but disadvantage of difficult to find which is which when changing
them in command)

CzechLayoutA1.vim
CzechLayoutA2.vim

CzechLayoutB1.vim
CzechLayoutB2.vim




In .vimrc, there would be:

set keyboardlayout=....




Maybe a step forward to change the layout on-the-fly by the following
command (when changing keyboard setting in X window system):

:set keyboardlayout=...



I am not a programmer, but the concept is to make vim a converter and
convert keys on-the-fly:

Input ---> vim(search and map in the layout file) ---> output




The concept is inspired by the following plugins:

VimIM : Vim Input Method
ywvim : Another input method(IM) for VIM, supports all modes

and the Keyboard layout system in Windows because when inputting
Chinese we rely heavily on mapping different keys so as to generate
one Chinese character.




All we need would be desiging 2 sets of clear layout files for every
different kind of keyboard layout

Hope this help.

Stephen Lee

unread,
Feb 22, 2011, 2:10:00 PM2/22/11
to vim_dev
Sorry everyone,

I just browsed through vim and found a directory called "keymap"
can we reuse and change or add different layout files in there?

Adrien "Pied" Piérard

unread,
Feb 22, 2011, 9:55:06 PM2/22/11
to vim...@googlegroups.com
Hi.


2011/2/23 Stephen Lee <stephenl...@gmail.com>:


> Therefore I propose letting users choose their preferred keyboard
> layouts instead of forcing any specific one to them.
> In your case it seems you are talking about QWERTY and QWERTZ (from
> wiki), so the following would be keyboard layouts in the new directory
> called "Keyboard"
>
> (General keyboard layouts)
> QWERTY.vim

> ...

Hum, as a user of a home-made variant of dvorak-bépo (because my
keyboard, a kinesis, is *not* practical to use with standard bépo), I
hope it's going to be easy to set up and modify such a setting,
because you are not going to give an exhaustive list of all layouts…

Ben Fritz

unread,
Feb 22, 2011, 11:23:44 PM2/22/11
to vim_dev


On Feb 22, 8:55 pm, Adrien "Pied" Piérard <axiopl...@gmail.com> wrote:
> Hi.
>
> 2011/2/23 Stephen Lee <stephenletter...@gmail.com>:
>
> > Therefore I propose letting users choose their preferred keyboard
> > layouts instead of forcing any specific one to them.
> > In your case it seems you are talking about QWERTY and QWERTZ (from
> > wiki), so the following would be keyboard layouts in the new directory
> > called "Keyboard"
>
> > (General keyboard layouts)
> > QWERTY.vim
> > ...
>
> Hum, as a user of a home-made variant of dvorak-bépo (because my
> keyboard, a kinesis, is *not* practical to use with standard bépo), I
> hope it's going to be easy to set up and modify such a setting,
> because you are not going to give an exhaustive list of all layouts…
>

I'm really not sure how the discussion got to be about keyboard
layouts. The proposal at hand is to replace Vim's input mechanism so
that ANY key, modified or otherwise, coming from ANY keyboard, will
work with Vim as a user would expect. Nobody is talking about forcing
specific keyboard layouts.

Benjamin R. Haskell

unread,
Feb 22, 2011, 11:43:39 PM2/22/11
to vim...@googlegroups.com
On Thu, 17 Feb 2011, Paul LeoNerd Evans wrote:

>> Graywh wrote:
>>
>>> Please fix vim's input queue mechanism with key info structures a.la
>>> libtermkey so that LeoNerd can fix terminal input and Gvim and
>>> everyone will be happy. Thanks.
>>

>> [...]


>
> I am talking about ripping out the byte-queue input system and replacing

> it with a structured keypress queue. [...]


>
> Having done this, I can then quite easily send you a patch to use

> libtermkey [...]


>
> Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a

> properly-configured xterm. [...]

So, from other responses, it seems (to me) that the problem isn't
necessarily in the input queue, per se. "Special" keys are already
represented in some internal format (perhaps not-well-documented, since
it's internal, but still represented). (example of undocumented nature
is the K_IGNORE special key trick that stopped working when its keycode
was either changed or eliminated [1]).

Taking that as a given, I'm going to ignore that aspect of this issue.
My question is about libtermkey. I was curious enough to download it,
unpack it, `make` it, and run ./demo. And, surprisingly, given that it
seems to be the favored example, pressing <Tab>, then <Ctrl>+<i>, then
<Ctrl>+<Shift>+<i> gave me the output sequence:

<Tab>
<Tab>
<Tab>

So, I'm wondering: what does libtermkey bring to the table?

I tested under both rxvt-unicode (my preferred terminal emulator) and
uxterm (which, AFAIK, is a "properly-configured" xterm). If
"properly-configured" actually means "specially-configured", how would
using libtermkey be anything less than a huge PITA when working on
multiple systems that aren't so configured? Or is this a
chicken-and-egg problem? (Nothing is configured to send anything other
than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)

--
Best,
Ben

[1] http://groups.google.com/group/vim_use/browse_thread/thread/8c535e5cf2b35f63/319b3113716e52d5

Ben Schmidt

unread,
Feb 23, 2011, 5:57:11 AM2/23/11
to vim...@googlegroups.com, Stephen Lee
First, Stephen, please don't top-post. Like it or not (and personally I
don't!) that is the convention on this list. I must admit, though, that
particularly after somebody else has already bottom-posted, top-posting
makes things a big mess. In future, please write your replies under
short, relevant extracts of the mail you are replying to.

> Therefore I propose letting users choose their preferred keyboard
> layouts instead of forcing any specific one to them.

Sorry, but this is not what we are discussing. Vim already has this
functionality. As you discovered, it can work through what it calls
'keymaps' (:help keymap). It also works with a number of different input
methods on different platforms, particularly when using Gvim.

> Maybe a step forward to change the layout on-the-fly by the following
> command (when changing keyboard setting in X window system):
>
> :set keyboardlayout=...

And there is even better than this already: Vim's 'iminsert' option
(:help iminsert). Of course, you can write mappings (as I have done) to
change the 'keymap' option between values, too.

> and there would be 4 files:
>
> QWERTYsequence.vim
> QWERTZsequence.vim
>
> QWERTYstructured.vim
> QWERTZstructured.vim

You are miles ahead of us, Stephen! It is indeed possible that this kind
of thing could evolve. In fact, the functionality for this part of it is
already there. What we need, though, is a way to get the structured
kepress information into vim in the first place, so that keymaps (and
more importantly, mappings) can refer to them.

Mappings are more important, because what we are discussing isn't really
about text input, but about shortcut keys. At the moment, Vim sometimes
can't tell the difference between Tab and Control-I (and a lot of other
things). And because Vim sometimes has no way of telling the difference,
at the moment, it doesn't even bother to notice the difference when it
*can* tell. We want to change that.

After we have changed that, then your idea of having multiple keyboard
layouts, some for when these keys can be differentiated, and some for
when they cannot, may become useful. I hope we won't need that idea,
though, but will be able to write one keymap file that works well in
both situations.

But that is for later.

First we need to get all the keys recognised as properly as possible.

Ben.

Ben Schmidt

unread,
Feb 23, 2011, 6:48:39 AM2/23/11
to vim...@googlegroups.com, Philippe Vaucher
On 20/02/11 10:17 PM, Philippe Vaucher wrote:
>> Yes. But what happens when you then edit that macro by putting the
>> register into a buffer, changing it, and yanking it again? This is not
>> uncommonly done. How should the registers be stored in .viminfo? How do
>> you write the input to the feedkeys function as a string in vimscript?
>> Etc.. These are the kinds of issues I was trying to raise.
>
> Well, I don't see what the problem is? Just do it like today, maybe
> with some expansions e.g "<C-A><S-I><Tab>somechars<U-234>". I mean
> this is simple key-as-text representation and then text-representation-
> to-struct creation no?
>
> For example, the user records a macro into register q that is then
> displayed as "ihello<Tab>worlld<Esc><Left><Left>x". He then edits the
> macro in a buffer, yanks it into register q, and when @q is executed
> all it does is interpret the string?

OK, so you seem to be proposing changing the way the q and @ commands
work, so that instead of exposing internal key representation, they
record and play back <> notation. They would also have to represent < as
<lt> of course. The feedkeys() function would also need to be similarly
changed. It would be hard to know what to do about \<xxx> escapes in
strings (:help expr-string)--whether to make it a no-op or do the old
thing, or something else.

So this would be a backwards-incompatible change. It would not just
affect places where special keys are currently represented in internal
notation, but everywhere macros are used. Now there are currently
probably quite a lot of places where internal keys are represented, and
changing the internal representation as I suggest, may affect them; but
if the current representation is sparse enough, perhaps it can be done
in a backwards compatible way. There may be quite little impact, though,
as wherever possible, plugin authors use <> notation anyway; it is only
when they cannot do this that they resort to internal
representation--and even then, they often use \<xxx> escapes when they
can. There is no way your suggestion could be backwards compatible, I
don't think, and it could potentially affect a lot more macros (e.g. all
macros that insert xml and html, so use < and > a lot). It could break a
lot of plugins as well as home-grown code.

Now, that doesn't rule it out, and I think it is probably a way to solve
the problem. But I think it would need more careful thought than the
alternative option of changing the internal representation while keeping
it byte-stream based. I also think it would be much more work, because
it requires not just macro recording, but every problem involving keys
to be separately solved from scratch, and every one would probably be
backwards incompatible. And I think Vim and Bram have a history of
maintaining backwards compatibility (we still have :set compatible!) so
it may be that a more compatible change is preferred.

> Maybe I'm oversimplifying it but it'd help if you were more specific
> about the "issues" the new design have.

This would be easier if we actually had a solid 'new design'. We don't.
We just have a bunch of rough ideas. One of those ideas had a drafted
structure and suggestion that it be used only in the input queue, where
the memory inefficiency would not be a concern. I have pointed out that
there are many other problems to solve: right hand sides of mappings,
feedkeys(), registers for macro recordings (which indirectly affect
buffers, as registers must be able to be put and yanked), representation
in strings, and more (I just thought of the :normal command, and I'm
sure there is still more...). I pointed out that if you use the
structure to solve all these problems, memory efficiency is most
certainly a concern.

So I think I have been quite specific. I think I've been specific enough
for the purpose of this discussion.

> To me it looks like we just need to agree on a new set of conventions
> by describing how the new system would work for all the current (and
> future) macros/registers/ whatever scenarios.

Yep, this seems to sum up your approach nicely, which is basically
design and solve every key-related problem from scratch. The most
specific thing you have done is suggested a different way to using the
structure to solve at least the macro recording problem.

My response, to both that specific proposal and that general approach is
that I don't think it is a good way, because it is massively
backwards-incompatible and a lot of work.

> I suggest maybe we make some kind of "new design" draft which would
> answer all the issues raised (memory size/1970 support/macros
> concerns) ?

I suggested earlier a rough idea in response to the structure proposal
of extending/changing Vim's existing mechanism. That was also
springboarding off Bram's comment that he thought this could be done
using the existing mechanism. The existing mechanism is there for all to
see, so is already a much more well-developed design than any of the
alternatives. My rough idea of how it could be extended is in a previous
post, too. I think it would work, and be more efficient, less work, and
more compatible than the other directions we have explored.

But it's not zero work, and I'm not at the point where I can volunteer
to do work on it. Even putting together a careful detailed proposal is
probably not a contribution I can commit. I'm more than willing to
discuss and brainstorm ideas, though, and comment on any serious
proposal someone wants to make.

In the end, it's up to Bram as maintainer, and whoever's going to put
the (large amount of) time into implementing this, what goes ahead,
though.

But as a community member, I'd prefer it if it affected my code and
plugins as little as possible, and I consider it essential that Vim
doesn't become significantly less efficient because of this. And if it
were me considering an implementation, I'd certainly be looking for the
option that was the least work! YMMV.

Ben.

Paul LeoNerd Evans

unread,
Feb 23, 2011, 11:13:25 AM2/23/11
to vim...@googlegroups.com
On Tue, Feb 22, 2011 at 11:43:39PM -0500, Benjamin R. Haskell wrote:
> Taking that as a given, I'm going to ignore that aspect of this
> issue. My question is about libtermkey. I was curious enough to
> download it, unpack it, `make` it, and run ./demo. And,
> surprisingly, given that it seems to be the favored example,
> pressing <Tab>, then <Ctrl>+<i>, then <Ctrl>+<Shift>+<i> gave me the
> output sequence:
>
> <Tab>
> <Tab>
> <Tab>
>
> So, I'm wondering: what does libtermkey bring to the table?

It parses CSIs, the way that xterm et.al. send modified keys.

Start an old vim. Go into insert mode. Press Ctrl-Left.

xterm sends CSI 1;5D

Without a CSI parser, using only a 1970s-style terminfo-driven prefix
hunting algorithm, there is no terminfo key sequence matching this. So
vim gives up. It then sees Escape [ 1 ; 5 D. The upshot is that vim
leaves insert mode (Escape), does something unimportant with [ 1 ; then
deletes 5 lines of text (5D).

With a CSI parser such as libtermkey, this is recognised as a real CSI D
call, with the parameters 1 and 5. This is reported to the application
as Left (CSI D), with the Ctrl modifier (2nd param being 5).

The primary thing that libtermkey brings, is not blowing up and deleting
5 lines of your text just because xterm happened to send a valid CSI
sequence that isn't in the terminfo.

> I tested under both rxvt-unicode (my preferred terminal emulator)

rxvt uses its own, totally-incompatible encoding scheme for modified
keypresses. This scheme is different to any other terminal that has such
abilities (all the others follow xterm's CSI scheme). It is also
incompatible with ECMA-48 (whereas xterm extends it), inextensible
beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
characters seemingly at random, to represent the Shift, Ctrl and
Ctrl+Shift states).

> and uxterm (which, AFAIK, is a "properly-configured" xterm). If
> "properly-configured" actually means "specially-configured",

Specifically-configured means

modifyOtherKeys: 2

Except that on its own now breaks most other applications, because xterm
is too keen to use the CSI encoding scheme on "normal" keypresses, such
as Ctrl-D. xterm doesn't yet send the correct and useful sequences here,
but it is not difficult to fix its code to do so - I have been in
occasional contact with Thomas Dickey on this issue; I need to poke him
again sometime soon to check he's properly looking into it.

What is needed is a hybrid scheme, whereby the simplest key encoding is
used in a nicer way; where Ctrl-D would send the single 0x04 byte (so
kernel's pty driver can recognise "here's an EOF"), but Ctrl-Shift-D
sends CSI 68;5 u to encode it correctly.

> how
> would using libtermkey be anything less than a huge PITA when
> working on multiple systems that aren't so configured? Or is this a
> chicken-and-egg problem? (Nothing is configured to send anything
> other than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)

Rightnow, xterm will send the same byte for Tab, Ctrl-I or Ctrl-Shift-I.
Setting modifyOtherKeys to 2 will send real keypresses that it can tell,
but now everything else cannot.

Ultimately, yes. It is chicken and egg, in order for -everything- to
work in a happy shiney way. But even without terminal changes,
applications can benefit from using libtermkey now.

So to answer your question on "how would using libtermkey be anything
less than a huge PITA" - it isn't. It already recognises, via its
terminfo-driven parser, every keypress that terminfo/curses already
recognises. It cannot be worse. It is strictly better than, because of
the full CSI parser I mentioned above. Even if it cannot pass on that
information to its containing program, it can at least parse it. If
libtermkey was used by some application that has no concept of modifier
keys, then modifiers would just be ignored; i.e. that pressing Ctrl-Left
would simply be read as a Left keypress, not resulting in a beep and
deleting 5 lines of text (vim), or beeping and appending ;5D to your
shell prompt (readline, irssi, etc...).

Furthermore it is much more future-proof. Just as in 1970 nobody
considered modifier keys, right now nobody has, say, pressure-sensitive
keyboards, so nobody bothers reporting pressure information. Maybe in
another 20 years time we'll all have keyboards that can recognise a
gentle vs. hard press of a key, or multi-touch with two fingers, or
whatever. It would be quite natural to use the 3rd CSI argument, say, to
encode this.

Up CSI A
Ctrl-Up CSI 1;5 A
Forceful-Ctrl-Up CSI 1;5;3 A

A terminfo-driven string prefix parser will of course blow up here. The
code in libtermkey, right now today, will not blow up. It won't
-understand- that 3rd parameter, it cannot -report- that to its
containing application, but it will correctly parse it to ignore it -
reporting to its application the press of Ctrl-A, by ignoring the
Forceful part.

Using libtermkey today leaves terminal developers of the future the
freedom to extend the CSI parameters in whatever way they see fit,
knowing that if old applications can't make use of this new information,
they are not at least confused and inconvenienced by it.


Hope that helps,

signature.asc

John Little

unread,
Feb 23, 2011, 5:58:03 PM2/23/11
to vim_dev
On Feb 24, 5:13 am, Paul LeoNerd Evans <leon...@leonerd.org.uk> wrote:
...
>  Forceful-Ctrl-Up  CSI 1;5;3 A

Now there's an idea :)))

Regards, John

Benjamin R. Haskell

unread,
Feb 24, 2011, 5:25:17 AM2/24/11
to vim...@googlegroups.com
On Wed, 23 Feb 2011, Paul LeoNerd Evans wrote:

> On Tue, Feb 22, 2011 at 11:43:39PM -0500, Benjamin R. Haskell wrote:
>>

>> [...] I'm wondering: what does libtermkey bring to the table?
>
> It parses CSIs, [...]


>
> Without a CSI parser, using only a 1970s-style terminfo-driven prefix
> hunting algorithm, there is no terminfo key sequence matching this. So

> vim gives up. [...]


>
>> I tested under both rxvt-unicode (my preferred terminal emulator)
>
> rxvt uses its own, totally-incompatible encoding scheme for modified
> keypresses. This scheme is different to any other terminal that has
> such abilities (all the others follow xterm's CSI scheme). It is also
> incompatible with ECMA-48 (whereas xterm extends it), inextensible
> beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
> characters seemingly at random, to represent the Shift, Ctrl and
> Ctrl+Shift states).

The whole situation is pretty arbitrary. E.g. ./demo's output under a
modifyOtherKeys:2 UXTerm, compressed to one row, for <Shift>+ the top
row of my typical US 105-key or variant:

<S-~> ! <S-@> # $ % <S-^> & * ( ) <S-_> + <Backspace>

Why the Shift modifier on ~, @, ^, and _, but not !, #, $, etc?
(Guessing hysterical raisins.)

Didn't realize ECMA-48 existed, though. Thanks.


>> and uxterm (which, AFAIK, is a "properly-configured" xterm). If
>> "properly-configured" actually means "specially-configured",
>
> Specifically-configured means
>
> modifyOtherKeys: 2
>
> Except that on its own now breaks most other applications, because
> xterm is too keen to use the CSI encoding scheme on "normal"
> keypresses, such as Ctrl-D. xterm doesn't yet send the correct and
> useful sequences here, but it is not difficult to fix its code to do
> so - I have been in occasional contact with Thomas Dickey on this
> issue; I need to poke him again sometime soon to check he's properly
> looking into it.

Cool. Works as advertised (i.e. currently not in anything other than
the `demo` program, but still interesting to see the ability is built
in).


> What is needed is a hybrid scheme, whereby the simplest key encoding
> is used in a nicer way; where Ctrl-D would send the single 0x04 byte
> (so kernel's pty driver can recognise "here's an EOF"), but
> Ctrl-Shift-D sends CSI 68;5 u to encode it correctly.

How do <Tab> and <Ctrl-i> fit into this scheme? What's the "simplest
key encoding"?


>> how would using libtermkey be anything less than a huge PITA when
>> working on multiple systems that aren't so configured? Or is this a
>> chicken-and-egg problem? (Nothing is configured to send anything
>> other than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)
>
> Rightnow, xterm will send the same byte for Tab, Ctrl-I or
> Ctrl-Shift-I. Setting modifyOtherKeys to 2 will send real keypresses
> that it can tell, but now everything else cannot.
>
> Ultimately, yes. It is chicken and egg, in order for -everything- to
> work in a happy shiney way. But even without terminal changes,
> applications can benefit from using libtermkey now.
>
> So to answer your question on "how would using libtermkey be anything
> less than a huge PITA" - it isn't. It already recognises, via its
> terminfo-driven parser, every keypress that terminfo/curses already
> recognises. It cannot be worse. It is strictly better than, because of
> the full CSI parser I mentioned above.

Thanks. This paragraph combined with the first section about parsing
unknown CSI sequences is plenty convincing. (trimmed rest)

--
Best,
Ben

Ben Schmidt

unread,
Feb 23, 2011, 6:29:02 PM2/23/11
to vim...@googlegroups.com, Paul LeoNerd Evans
>> So, I'm wondering: what does libtermkey bring to the table?
>
> It parses CSIs, the way that xterm et.al. send modified keys.

Of course, Vim can already do this to an extent.

:help keycodes
:help xterm-8bit
:help +termresponse

It's only keys like Control-I which are still indistingishable from keys
like Tab (and Enter, and Backspace, and I think shifted Control keys).
And perhaps that can't be fixed in the terminal now, but it can
certainly be fixed in the GUI.

And in doing so, the notation and representation for it can be tidied up
so there aren't a bunch of internal ambiguities and at least all the
remaining ambiguities are external ones that we are stuck with anyway
(and with luck will go away one day).

> rxvt uses its own, totally-incompatible encoding scheme for modified
> keypresses. This scheme is different to any other terminal that has such
> abilities (all the others follow xterm's CSI scheme). It is also
> incompatible with ECMA-48 (whereas xterm extends it), inextensible
> beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
> characters seemingly at random, to represent the Shift, Ctrl and
> Ctrl+Shift states).

And AFAIK, Vim doesn't deal with this. Does libtermkey or does it only
do CSI stuff?

Ben.

Ben Schmidt

unread,
Feb 24, 2011, 7:30:44 AM2/24/11
to vim...@googlegroups.com, Benjamin R. Haskell
> The whole situation is pretty arbitrary. E.g. ./demo's output under a
> modifyOtherKeys:2 UXTerm, compressed to one row, for <Shift>+ the top row of my
> typical US 105-key or variant:
>
> <S-~> ! <S-@> # $ % <S-^> & * ( ) <S-_> + <Backspace>
>
> Why the Shift modifier on ~, @, ^, and _, but not !, #, $, etc? (Guessing
> hysterical raisins.)

This can't explain ~, but...

My guess is that these are keys which are low enough in ASCII that they
can be control keys. There should be 32 of them. A-Z makes 26, plus the
three above which are @ ^ and _, then there's [ ] and \, and that's it.
I guess you can test the theory and see if { } and | appear with shift
modifiers--though I guess they may not, since they actually generate
different characters. Still, would be interesting to know.

Maybe ~ by some other convention does something special with control.
IIRC, it is right down the other end of ASCII, at 126 or something.
Maybe it's equivalent to another one of the keys, if ctrl-a is the same
as ctrl-A, maybe ctrl-~ is equivalent to whatever is at
codepoint...126-32=94, which is definitely one of those symbols above...
It's strange, in my terminal, pressing ctrl-~ seems to do nothing,
whereas control with other 'non-control' characters makes a beep.

At any rate, since it's only these few keys which can be control
characters in the first place, it makes sense that these have the
special treatment, and the others don't.

Ben.

Paul LeoNerd Evans

unread,
Feb 24, 2011, 12:57:02 PM2/24/11
to vim...@googlegroups.com
On Thu, Feb 24, 2011 at 10:29:02AM +1100, Ben Schmidt wrote:
> >rxvt uses its own, totally-incompatible encoding scheme for modified
> >keypresses. This scheme is different to any other terminal that has such
> >abilities (all the others follow xterm's CSI scheme). It is also
> >incompatible with ECMA-48 (whereas xterm extends it), inextensible
> >beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
> >characters seemingly at random, to represent the Shift, Ctrl and
> >Ctrl+Shift states).
>
> And AFAIK, Vim doesn't deal with this. Does libtermkey or does it only
> do CSI stuff?

It doesn't at the moment, but if someone wants to do the research into
what sequences it sends, and write some code in the library (similar to
driver-csi.c and driver-ti.c), I could add that in sure.

signature.asc

Stephen Lee

unread,
Feb 24, 2011, 3:27:25 PM2/24/11
to vim_dev
Sorry Ben

This is my first time joining a discussion, I had a hard time just to
open an account and send my suggestion. I use the default reply
function and I don't mean to upset anyone. Hopefull now this reply is
well-edited (I removed the quote).

About vim, I misunderstood that it was a keyboard problem. As Bram
said it is internally set as <CTRL-I> = <Tab> and people requesting
having structured input mechanism, all I can think of is to break
apart all internal linkages and revert to the their original settings:

I found that <CTRL-M> and <Enter> both have the same keymap code 0x0d,
and both <Tab> and <CTRL-I> gives 0x09, maybe that is why vim gives
the same output.

How about checking the Scan Codes directly?

<Enter> = 1c 9c
<L-CTRL> = 1d 9d
<M> = 32 b2

<Tab> = 0f 8f
<L-CTRL> = 1d 9d
<I> = 17 97


A shortcut table could be made to map between <Enter> and <L-CTRL>
+<M>, and load when vim starts.


So the new process would be:

detecting Scan Codes --> map between Scan Codes to alias like
"<Enter>" or "<Tab>"

Different keyboards from different countries maybe need different Scan-
Codes-to-alias mapping layouts, and therefore many layout files.

I know it is more complicated than getting the keycode conveniently
from the system, but this is all I know to my understanding.


About Paul's type:

> Tab { 1, 0, KEY_TAB }
> Ctrl-I { 0, MOD_CTRL, 'i' }
> Ctrl-Shift-I { 0, MOD_CTRL, 'I' }

I see is a switch type setting, it should work and require less
changes but I really don't know how to differentiate if it is
systemwise that <Tab> and <CTRL-I> gives the same 0x09.


Also about your question:

> When there is one. Or a codepoint in some encoding. Yeah, your struct
> does allow for that. The struct isn't inexpressive, it's just that it's
> big, and if it needs to be used in a LOT of places (which seem to just
> be increasing...input queue, mappings, registers, strings, buffers?!),
> suddenly not just one thing is getting a bit bigger--a lot of things are
> getting bigger--potentially a lot bigger! If every time you yank text
> into a register it gets 12 times larger, that may cause some problems!

Is it possilbe to swap those registers and histories onto hard disk so
as to control the memory size? It has advantages of large capacity,
infinite undo, and can be called to check the complete list.

Sorry again if I gave useless suggestions.

Stephen

Philippe Vaucher

unread,
Feb 25, 2011, 5:48:50 AM2/25/11
to vim_dev
> This would be easier if we actually had a solid 'new design'. We don't.
> We just have a bunch of rough ideas. One of those ideas had a drafted
> structure and suggestion that it be used only in the input queue, where
> the memory inefficiency would not be a concern. I have pointed out that
> there are many other problems to solve: right hand sides of mappings,
> feedkeys(), registers for macro recordings (which indirectly affect
> buffers, as registers must be able to be put and yanked), representation
> in strings, and more (I just thought of the :normal command, and I'm
> sure there is still more...). I pointed out that if you use the
> structure to solve all these problems, memory efficiency is most
> certainly a concern.

> My response, to both that specific proposal and that general approach is
> that I don't think it is a good way, because it is massively
> backwards-incompatible and a lot of work.

> But as a community member, I'd prefer it if it affected my code and
> plugins as little as possible, and I consider it essential that Vim
> doesn't become significantly less efficient because of this. And if it
> were me considering an implementation, I'd certainly be looking for the
> option that was the least work! YMMV.

So basically backward compatibility and memory efficiency are what
hold vim back in 1970. You made a lot of good points and reasons for
it to be so, but I'm always sad when good ideas are refused just
because of old scenarios. I mean I wouldn't care if vim went as far as
doubling its memory usage and broke backward compatibility if what I
gain in exchange is modernity, but I guess not everyone is like me. I
just think old hardware should use old software and that new hardware
should use new software, I understand backward compatibility is what
makes a software like vim able to run on my toaster, but I wouldn't
mind using an older version of vim on that toaster.

That said, I think there is a compromise. Vim has features, maybe the
new input mechanism could simply be a feature, something like
+enhanced_term_input (like we have for +python or +eval), and plugins
could simply check with stuffs like:

if has('enhanced_input') == 0
echom 'Vim wasn't compiled with +enhanced_input, the plugin's
default mappings (<C-i>, etc) were remaped'
call s:UseOldMappings()
endif

I think this would please everyone. People wanting vim not to change a
single bit can, people wanting it to evolve can. Of course there's
still all the issues discussed above to solve, but at least I think
with this proposal things we'll get less resistance ;)

Thanks,
Philippe


Philippe Vaucher

unread,
Feb 25, 2011, 5:51:18 AM2/25/11
to vim_dev
> That said, I think there is a compromise. Vim has features, maybe the
> new input mechanism could simply be a feature, something like
> +enhanced_term_input (like we have for +python or +eval), and plugins
> could simply check with stuffs like:

With an option like :set term_input_type=backward_compatible by
default, or smth like that. Maybe this was already proposed tho.

Ben Schmidt

unread,
Feb 25, 2011, 7:32:31 AM2/25/11
to vim...@googlegroups.com
> So basically backward compatibility and memory efficiency are what
> hold vim back in 1970. You made a lot of good points and reasons for
> it to be so, but I'm always sad when good ideas are refused just
> because of old scenarios.

I don't think anybody has refused the idea, and I don't think anybody
wants to keep Vim in the 1970s. We all want this. In fact, people who
don't want this probably wouldn't get involved in the discussion--unless
we looked like we were going to introduce some horribly backwardly
incompatible change that would affect them nastily.

So I would encourage you not to view this discussion as opposition to
the idea. Quite the reverse. This discussion is actually an important
part of making this idea happen. We need to discuss these things so we
can do it the best way possible, without *needlessly* breaking backward
compatibility, without *needlessly* or *significantly* lowering
efficiency, and without *unnecessarily* wasting people's time.

> That said, I think there is a compromise.

Unless you saw something that I didn't see, I don't think there is any
resistance that necessitates a compromise.

I think it goes without saying that we'll need some kind of
compatibility mode, whether it's by means of a +feature, an 'option', or
just some carefully thought-out behaviours. Some of these issues have
already been raised and solutions brainstormed (including your recent
suggestions, as well as in earlier mails by me and others).

> Of course there's still all the issues discussed above to solve, but
> at least I think with this proposal things we'll get less resistance
> ;)

As I said, I don't think there is any resistance to the idea. We're just
discussing how to do it. As someone suggested earlier, it would be best
to discuss and draft some documentation for this stuff before doing the
hard implementation work. Having something solid will help, and a
checklist of issues/concerns along with their solutions.

So, to move it to the next stage, is anyone in a position to volunteer
to write up a more specific design (which probably needs to be written
with reference both to this email discussion and the Vim source code)?

Also, is anyone in a position to volunteer to help with implementation
once we have a design?

Ben.

Ben Schmidt

unread,
Feb 25, 2011, 7:41:53 AM2/25/11
to vim...@googlegroups.com, Stephen Lee
> This is my first time joining a discussion, I had a hard time just to
> open an account and send my suggestion. I use the default reply
> function and I don't mean to upset anyone. Hopefull now this reply is
> well-edited (I removed the quote).

Yes, it was a much better reply, thanks, Stephen. And welcome to the
list. You picked a good lively discussion to get involved in!

> How about checking the Scan Codes directly?
>
> <Enter> = 1c 9c
> <L-CTRL> = 1d 9d
> <M> = 32 b2
>
> <Tab> = 0f 8f
> <L-CTRL> = 1d 9d
> <I> = 17 97

We should be able to do it at a higher level than this, and avoid having
to deal with scan-code -> character mapping. Also, scan codes simply
aren't available when running in terminal mode. Every GUI will be
different, and different terminals will need different handling, but we
should be able to get something structured from the OS/window
manager/terminal which details the key pressed and its modifiers. I
don't think this part of things will be a problem. And since translation
from the GUI-specific form to Vim internal form will happen very
quickly, efficiency won't be an issue here either.

> About Paul's type:
>
>> Tab { 1, 0, KEY_TAB }
>> Ctrl-I { 0, MOD_CTRL, 'i' }
>> Ctrl-Shift-I { 0, MOD_CTRL, 'I' }
>
> I see is a switch type setting, it should work and require less
> changes but I really don't know how to differentiate if it is
> systemwise that<Tab> and<CTRL-I> gives the same 0x09.

GUIs should be able to get the information, and some terminals (e.g. via
libtermkey as has been discussed in another part of the thread).

The trouble with the structure is that it is big, and contrary to the
original assumption, it't not really useful to restrict its use to the
input queue. We need a solution that will work in other parts of Vim
efficiently--maybe a byte-stream solution, maybe an escape character
followed by a packed structure, maybe something we haven't thought of
yet. I think the byte-stream method will work well, and is closest to
what Vim already does so least work.

> Also about your question:
>
>> When there is one. Or a codepoint in some encoding. Yeah, your struct
>> does allow for that. The struct isn't inexpressive, it's just that it's
>> big, and if it needs to be used in a LOT of places (which seem to just
>> be increasing...input queue, mappings, registers, strings, buffers?!),
>> suddenly not just one thing is getting a bit bigger--a lot of things are
>> getting bigger--potentially a lot bigger! If every time you yank text
>> into a register it gets 12 times larger, that may cause some problems!
>
> Is it possilbe to swap those registers and histories onto hard disk so
> as to control the memory size? It has advantages of large capacity,
> infinite undo, and can be called to check the complete list.

It's not out of the question, but it would require significant changes
to a lot of Vim, I think, and is not what really how you would expect
them to work. Also, a massive chunk of disk space isn't all that much an
improvement than a massive chunk of main memory!

Cheers,

Ben.

Bram Moolenaar

unread,
Feb 25, 2011, 8:43:23 AM2/25/11
to Stephen Lee, vim_dev

> This is my first time joining a discussion, I had a hard time just to
> open an account and send my suggestion. I use the default reply
> function and I don't mean to upset anyone. Hopefull now this reply is
> well-edited (I removed the quote).
>
> About vim, I misunderstood that it was a keyboard problem. As Bram
> said it is internally set as <CTRL-I> = <Tab> and people requesting
> having structured input mechanism, all I can think of is to break
> apart all internal linkages and revert to the their original settings:
>
> I found that <CTRL-M> and <Enter> both have the same keymap code 0x0d,
> and both <Tab> and <CTRL-I> gives 0x09, maybe that is why vim gives
> the same output.

[...]

So the problem is that many users expect CTRL-M to have the same effect
as Enter, just like people use CTRL-[ instead of Esc. And a few people
would make the CTRL-M act different from Enter, and CTRL-[ different
from Esc.

First problem is to actually detect what key was pressed, in most
terminal emulators this is not possible. In the GUI we can. Changing
terminal emulators to support this and making this work with Vim is a
separate issue, I'll not go into that here.

Then we need a way to make the extra information available to be used in
mappings, without breaking it for users relying on the current way.

Some things that are no acceptable:
- Have a setting to enable "the new way". This will break existing
stuff and make users pull their hair out because they don't know this
setting exists. Forget it.
- Change the input queue from a stream of bytes to some list of structs.
This isn't adding any functionality and breaks all kinds of mapping
and termcode handling, register execution, etc. Forget it.

What we can do is extend the existing modifier byte sequence. This is a
bit tricky, but it should work. So we add a new byte sequence with the
raw key encoded, plus modifiers. Thus for CTRL-[ you get the [ key with
the CTRL modifier.

When this new modifier is not mapped, it is discarded and the following
translated codes are used normally, just like now. Thus without any
mappings that use the new modifier it is guaranteed to work as before.
When executing a register where the new modifiers are missing it works
just like before.

When a new modifier is mapped, it is replaced by what it's mapped to, AND
the following old style byte sequence is consumed.

An alternative is to let the termcap handling translate the new modifer
into the old style key. That works because mappings are applied both to
the raw escape sequences and on the key codes they are converted to.

We also need a way to specify mappings with the new modifier, perhaps
using a special modifier X, thus you could do:
:map <X-C-[> :echo CTRL-[<CR>
:map <X-C-I> :echo CTRL-I<CR>

--
Not too long ago, compress was something you did to garbage...

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Charles E Campbell Jr

unread,
Feb 25, 2011, 11:12:21 AM2/25/11
to vim...@googlegroups.com
Bram Moolenaar wrote:
> [snip]

> Then we need a way to make the extra information available to be used in
> mappings, without breaking it for users relying on the current way.
>
> Some things that are no acceptable:
> - Have a setting to enable "the new way". This will break existing
> stuff and make users pull their hair out because they don't know this
> setting exists. Forget it.
> - Change the input queue from a stream of bytes to some list of structs.
> This isn't adding any functionality and breaks all kinds of mapping
> and termcode handling, register execution, etc. Forget it.
>
> What we can do is extend the existing modifier byte sequence. This is a
> bit tricky, but it should work. So we add a new byte sequence with the
> raw key encoded, plus modifiers. Thus for CTRL-[ you get the [ key with
> the CTRL modifier.
>
[snip]

Another way to extend this might be to switch from a byte queue to a
short integer queue. The lower 8-bits would be as it is now; the upper
eight bits would encode modifier keys (shift, ctrl, alt, meta, whatever).

Regards,
Chip Campbell

Ben Schmidt

unread,
Feb 25, 2011, 5:12:08 PM2/25/11
to vim...@googlegroups.com
> Another way to extend this might be to switch from a byte queue to a short integer
> queue. The lower 8-bits would be as it is now; the upper eight bits would encode
> modifier keys (shift, ctrl, alt, meta, whatever).

I think sticking to the byte queue is best. So much in Vim relies on
text being represented in either a single byte encoding or UTF-8 (e.g.
when you use UCS-2 or UCS-4, Vim will use UTF-8 internally, I believe).
It will break less if we stick to a byte stream with escapes, as now,
IMHO.

Ben.

Philippe Vaucher

unread,
Feb 26, 2011, 7:57:17 AM2/26/11
to vim_dev
> So I would encourage you not to view this discussion as opposition to
> the idea. Quite the reverse. This discussion is actually an important
> part of making this idea happen. We need to discuss these things so we
> can do it the best way possible, without *needlessly* breaking backward
> compatibility, without *needlessly* or *significantly* lowering
> efficiency, and without *unnecessarily* wasting people's time.

Ah, sorry then. Yes I felt it a bit like a resistance to change. I've
unfortunately seen this happen too often... thanks for setting me back
on the right track :)


> So, to move it to the next stage, is anyone in a position to volunteer
> to write up a more specific design (which probably needs to be written
> with reference both to this email discussion and the Vim source code)?
> Also, is anyone in a position to volunteer to help with implementation
> once we have a design?

I'd volunteer but I think the new post from Bram doesn't let a lot of
room for work to do on the design part... I think he outlined a
specific task to do (extend the existing modifier byte sequence) and
put a big no on all the alternatives discussed.

All-in-all if this answers the needs we have I'm for it, and it looks
as a fairly easy task to do (compared to the struct rewrite).

Philippe

Philippe Vaucher

unread,
Feb 26, 2011, 9:20:16 AM2/26/11
to vim_dev
Apparently answering on the webpage of google groups likes to set the
old subject back, I changed it again.

Philippe

Bram Moolenaar

unread,
Feb 26, 2011, 9:23:51 AM2/26/11
to Charles E Campbell Jr, vim...@googlegroups.com

Charles Campbell wrote:

The current special-key encoding is a bit clumsy. On the other hand,
there are situations where the queue can get very long (recursive
execution of mappings or "1000@@"). Using 16 or 32 bits for every
character is not attractive.

One way to solve this is to use up to 128 bit integers, making sure that
the higher bits are usually zero, and then use variable length encoding.
ASCII characters should then still take only one byte.
http://code.google.com/apis/protocolbuffers/docs/encoding.html

Nevertheless, what gets encoded in registers must still be a byte
sequence. And we need to be able to store double-byte and multi-byte
characters. Thus I don't think this will make things simpler.

--
This is the polymorph virus! Follow these instructions carefully:
1. Send this message to everybody you know.
2. Format your harddisk.
Thank you for your cooperation in spreading the most powerful virus ever!

Paul LeoNerd Evans

unread,
Mar 7, 2011, 10:46:35 AM3/7/11
to vim...@googlegroups.com, Stephen Lee
On Fri, Feb 25, 2011 at 02:43:23PM +0100, Bram Moolenaar wrote:
> So the problem is that many users expect CTRL-M to have the same effect
> as Enter, just like people use CTRL-[ instead of Esc. And a few people
> would make the CTRL-M act different from Enter, and CTRL-[ different
> from Esc.

That's OK. For them it can be mapped. Or even made default.

> First problem is to actually detect what key was pressed, in most
> terminal emulators this is not possible. In the GUI we can. Changing
> terminal emulators to support this and making this work with Vim is a
> separate issue, I'll not go into that here.

Changing terminal emulators is a done task; xterm has supported this
since 2008. In fact I asked Thomas Dickey to clarify this for me, and he
informs me it's all present and working.

> Then we need a way to make the extra information available to be used in
> mappings, without breaking it for users relying on the current way.
>
> Some things that are no acceptable:
> - Have a setting to enable "the new way".

Really?

Is that what we'd have said right back at the beginning of Vim? On the
subject of the 'compatible setting:

"Adding a setting to enable all the vim-incompatible changes, is too
complex for users to handle."

> This will break existing
> stuff and make users pull their hair out because they don't know this
> setting exists. Forget it.

Right now we have users _all the time_ pulling their hair out _because_
such a setting does not exist. Including myself. I would much rather
have to support hoards of confused users in #vim by saying to them

"Yes, sorry your vim can't recognise that keypress now, but set this
setting and then it will"

instead of

"No, sorry your vim cannot recognise that keypress at all"

If you would prefer to support people by telling them it is not
possible, rather than telling them how to make it possible, please come
to #vim and explain that to them. Every week.

> - Change the input queue from a stream of bytes to some list of structs.
> This isn't adding any functionality and breaks all kinds of mapping
> and termcode handling, register execution, etc. Forget it.

OK, well that's fair. It was intended as an implementation detail in any
case. I hadn't realised it would be possible with the existing byte
queue.

As long as the two triplets of keypresses I suggested originally can all
be represented uniquely, and without reference to timing information in
the Escape vs Alt+ case, then I'm happy with whatever internal
implementation makes it happen.

> What we can do is extend the existing modifier byte sequence.

<snip long description>

That all sounds a bit long and complicated, and I'm not sure I see
whether it's necessarily any easier to implement than the arguably
much-neater and more forward-compatible queue-of-structures idea I had
originally. But again, that's all internal implementation details -
whatever makes it work, I'm happy with.

> We also need a way to specify mappings with the new modifier, perhaps
> using a special modifier X, thus you could do:
> :map <X-C-[> :echo CTRL-[<CR>
> :map <X-C-I> :echo CTRL-I<CR>

X for "eXtended"? Could work, though would be useful to check that
doesn't collide with any named modifiers on any existing system. Perhaps
some non-letter punctuation symbol? Though nothing suitable comes to
mind.

signature.asc

Philippe Vaucher

unread,
Mar 7, 2011, 10:55:09 AM3/7/11
to vim_dev
> What we can do is extend the existing modifier byte sequence.  This is a
> bit tricky, but it should work.  So we add a new byte sequence with the
> raw key encoded, plus modifiers.  Thus for CTRL-[ you get the [ key with
> the CTRL modifier.

Would this proposal allow us to map Tab, Ctrl-I, Ctrl-Shift-I, Alt+d,
Escape d, and é all independantly? I don't know what possibilities
extending the existing modifier byte sequence allows.

Philippe

Paul LeoNerd Evans

unread,
Mar 7, 2011, 10:58:02 AM3/7/11
to vim...@googlegroups.com
On Thu, Feb 24, 2011 at 05:25:17AM -0500, Benjamin R. Haskell wrote:
> The whole situation is pretty arbitrary. E.g. ./demo's output under
> a modifyOtherKeys:2 UXTerm, compressed to one row, for <Shift>+ the
> top row of my typical US 105-key or variant:
>
> <S-~> ! <S-@> # $ % <S-^> & * ( ) <S-_> + <Backspace>
>
> Why the Shift modifier on ~, @, ^, and _, but not !, #, $, etc?
> (Guessing hysterical raisins.)

That sounds like a buggy implementation. Due to all the vagueness of
what shift means in the context of Unicode (is ! really ! or is it
Shift-1) my specification called for explicitly ignoring the state of
the Shift key in the terminal, at the time the modified Unicode sequence
is created. If that isn't happening, I vote bug.

> Didn't realize ECMA-48 existed, though. Thanks.

It's essential in these matters. :)

Once you read the spec part for CSI, you realise how short and neat the
entire scheme really is, how extensible it is provided you actually have
a real CSI parser and not a dumb string-prefix matcher a.la.
termcap/terminfo/curses combinations.

> > modifyOtherKeys: 2
> >
> >Except that on its own now breaks most other applications, because
> >xterm is too keen to use the CSI encoding scheme on "normal"
> >keypresses, such as Ctrl-D. xterm doesn't yet send the correct and
> >useful sequences here, but it is not difficult to fix its code to
> >do so - I have been in occasional contact with Thomas Dickey on
> >this issue; I need to poke him again sometime soon to check he's
> >properly looking into it.
>
> Cool. Works as advertised (i.e. currently not in anything other
> than the `demo` program, but still interesting to see the ability is
> built in).

I am also recently informed by Thomas Dickey that

formatOtherKeys: 1

sends a slightly nicer CSI u rather than CSI 27~ encoding

> >What is needed is a hybrid scheme, whereby the simplest key
> >encoding is used in a nicer way; where Ctrl-D would send the
> >single 0x04 byte (so kernel's pty driver can recognise "here's an
> >EOF"), but Ctrl-Shift-D sends CSI 68;5 u to encode it correctly.
>
> How do <Tab> and <Ctrl-i> fit into this scheme? What's the
> "simplest key encoding"?

<Tab> 0x08 (ASCII HT)
<Ctrl-I> CSI 105;5u (i == 105 decimal in ASCII)
<Ctrl-Shift-I> CSI 73;5u (I == 73 decimal in ASCII)

Note that, due to what I said above, we have to ignore the shift key in
these CSI u sequences; the distinction between Ctrl-I and Ctrl-Shift-I
is simply that one is lowercase and one is capital.

> >>how would using libtermkey be anything less than a huge PITA
> >>when working on multiple systems that aren't so configured? Or
> >>is this a chicken-and-egg problem? (Nothing is configured to
> >>send anything other than 70's-style "<C-i>" == "<Tab>", so
> >>nothing ever will be.)
> >
> >Rightnow, xterm will send the same byte for Tab, Ctrl-I or
> >Ctrl-Shift-I. Setting modifyOtherKeys to 2 will send real
> >keypresses that it can tell, but now everything else cannot.
> >
> >Ultimately, yes. It is chicken and egg, in order for -everything-
> >to work in a happy shiney way. But even without terminal changes,
> >applications can benefit from using libtermkey now.
> >
> >So to answer your question on "how would using libtermkey be
> >anything less than a huge PITA" - it isn't. It already recognises,
> >via its terminfo-driven parser, every keypress that
> >terminfo/curses already recognises. It cannot be worse. It is
> >strictly better than, because of the full CSI parser I mentioned
> >above.
>
> Thanks. This paragraph combined with the first section about
> parsing unknown CSI sequences is plenty convincing. (trimmed rest)

Another thought occurs to me.

Run vim in xterm. It recognises Ctrl-Left, because vim handles
TERM=xterm specially to recognise this case.

Run vim in screen in xterm. It doesn't recognise Ctrl-Left, instead
interpreting it as Escape, 1; (ignored), 5C. Deletes 5 lines of text and
enters insert mode. Mass fail.

Run libtermkey's demo either in xterm or in screen-in-xterm. Observe
that Ctrl-Left works IN BOTH CASES. This is because it always applies
its CSI parser, so always understands xterm-alike CSI encoding of
Ctrl-Left (et.al.).

This fact -alone- should be enough to convince you. :)

signature.asc

Paul LeoNerd Evans

unread,
Mar 7, 2011, 11:02:20 AM3/7/11
to vim...@googlegroups.com, Stephen Lee
(Just a couple of points of extra clarity)

On Mon, Mar 07, 2011 at 03:46:35PM +0000, Paul LeoNerd Evans wrote:
> As long as the two triplets of keypresses I suggested originally can all
> be represented uniquely, and without reference to timing information in
> the Escape vs Alt+ case, then I'm happy with whatever internal
> implementation makes it happen.

The two triplets in question being

<Tab>
<Ctrl-I>
<Ctrl-Shift-I>

<Escape> C
<Alt+C>
é

Yes I am aware that on current byte-driven terminals there is no way
without using timing information to distinguish all these three cases in
the latter triplet here. This should not stop GUI systems from
distinguishing them however.

Specifically as to the terminal case, I have a solution to this, but it
requires the cooperation of terminals and applications to use libtermkey
or some equivalent code, so right now I don't want to distract the
argument. That is what derailed it last time when I tried this a couple
of years ago - I don't want that to happen again for now. :)

signature.asc

Benjamin R. Haskell

unread,
Mar 7, 2011, 11:24:23 AM3/7/11
to vim...@googlegroups.com, Stephen Lee
On Mon, 7 Mar 2011, Paul LeoNerd Evans wrote:

> On Fri, Feb 25, 2011 at 02:43:23PM +0100, Bram Moolenaar wrote:
>> First problem is to actually detect what key was pressed, in most
>> terminal emulators this is not possible. In the GUI we can.
>> Changing terminal emulators to support this and making this work with
>> Vim is a separate issue, I'll not go into that here.
>
> Changing terminal emulators is a done task; xterm has supported this
> since 2008. In fact I asked Thomas Dickey to clarify this for me, and
> he informs me it's all present and working.

Despite its popularity, xterm is not "most terminal emulators". I think
this is a great move on xterm's part (having a more-or-less consistent
set of escapes for modifiers). But, it's still a chicken-and-egg
problem. Once it's in xterm, it takes a while (if ever) to filter down
to derivatives. And even under xterm the setting that allows this to
work isn't the default.

I almost didn't send this, since I don't think the lack of terminal
support should be a show stopper (If any terminal supports it, Vim
should use that support), but claiming that it's a done deal because
xterm added a feature to support it a few years ago is naïve.

--
Best,
Ben

Benjamin R. Haskell

unread,
Mar 7, 2011, 11:29:06 AM3/7/11
to vim_dev, Stephen Lee
[drafted a while ago, but sending now that the thread got bumped; might
be a bit roughdraftish]

So, instead of enabling a single setting (usenewkeymaps or whatever),
the :map command needs a new "Yes, do what I'm telling you to do"
modifier? This seems totally backwards to me. The "new way" will cause
problems with old stuff. Why is that breakage unacceptable? This whole
suggestion seems like a major-version change anyway (not a point-version
change, where backwards compatibility is sacrosanct). The stuff it's
breaking was caused by having to do a workaround in the first place.

Consider:

" {intent} should do {rhs}
:map {lhs} {rhs}

Where {intent} describes the intended binding (what key should be
pressed to activate it), and {lhs} is what was actually typed. In the
following chart, {around} indicates that the only reason that particular
{lhs} was chosen was to work around the limitation of key mappings.
{new-works} indicates that using that same {lhs} would still work under
the "new way". (^x indicates entering a literal control character)

{intent} | {lhs} | {around} | {new-works} | Category
---------+---------+----------+-------------+---------
<Enter> | <C-m> | yes | no | Enter vs Ctrl+m
<Enter> | <Enter> | no | yes
<Enter> | ^M | yes | no(?)
<C-m> | <C-m> | no | yes
<C-m> | <Enter> | ? | no
<C-m> | ^M | yes | yes(?)
---------+---------+----------+-------------+---------
<Esc> | <C-[> | ? | no | Esc vs Ctrl+[
<Esc> | <Esc> | no | yes
<Esc> | ^[ | yes | no(?)
<C-[> | <C-[> | no | yes
<C-[> | <Esc> | ? | no
<C-[> | ^[ | yes | yes(?)
---------+---------+----------+-------------+---------
<Tab> | <C-i> | yes | no | Tab vs Ctrl-i
<Tab> | <Tab> | no | yes
<Tab> | ^I | yes | no(?)
<C-i> | <C-i> | no | yes
<C-i> | <Tab> | ? | no
<C-i> | ^I | yes | yes(?)
---------+---------+----------+-------------+---------
é | <M-i> | yes | no | é vs Alt/Meta-i
é | é | no | yes | (issue w/ 8th-bit meta)
<M-i> | <M-i> | no | yes
<M-i> | é | yes | no

Not sure if the chart is clear, but my point is:

I think that the people who expect, e.g., <C-i> to activate a <Tab>
mapping are the people who either:

1. Know what's going on behind the scenes, so are knowledgeable enough
to fix their mappings (fixing plugin mappings is a separate issue)

2. Were bitten by something that didn't work right, but found a
workaround and only use it due to inertia.

I highly doubt there are people who approach the :map command with the
*desire* that <C-i> or ^I would activate a :map whose {lhs} contains
<Tab> or that <Tab> would activate a <C-i> :map. This seems like a
clear case where backwards compatibility is undesirable.

The fact that plugins probably rely on this behavior is unfortunate, but
"Oh, that plugin's <Tab> binding is broken because Vim's key mapping
used to be broken, and so it was using the Ctrl-i workaround" seems like
a preferable position to be in.

--
Best,
Ben

Paul LeoNerd Evans

unread,
Mar 7, 2011, 1:38:18 PM3/7/11
to vim...@googlegroups.com, Stephen Lee
On Mon, Mar 07, 2011 at 11:24:23AM -0500, Benjamin R. Haskell wrote:
> >Changing terminal emulators is a done task; xterm has supported
> >this since 2008. In fact I asked Thomas Dickey to clarify this for
> >me, and he informs me it's all present and working.
>
> Despite its popularity, xterm is not "most terminal emulators". I
> think this is a great move on xterm's part (having a more-or-less
> consistent set of escapes for modifiers). But, it's still a
> chicken-and-egg problem. Once it's in xterm, it takes a while (if
> ever) to filter down to derivatives. And even under xterm the
> setting that allows this to work isn't the default.
>
> I almost didn't send this, since I don't think the lack of terminal
> support should be a show stopper (If any terminal supports it, Vim
> should use that support), but claiming that it's a done deal because
> xterm added a feature to support it a few years ago is naïve.

This is regrettable, yes. But not entirely true.

Until xterm added the extended CSI notation using the 2nd parameter for
modifiers in e.g. CSI 1;5C for Ctrl-Left, no terminal in the world did
it.

Now, apart from rxvt which uses its own scheme, prettymuch every
terminal does it, and does it -that way-. These things aren't completely
set in stone, but they do take a long long time to filter around.

I'm very keen to try to shepherd people into one standard and
consistently-designed way of doing things. That rxvt does modified keys
differently is regrettable, and a consequence of splintered reinvention
of wheels. I'm very keen to try not to let that happen any more.

It's to this end I tried initially writing this spec for modified
keypresses:

http://www.leonerd.org.uk/hacks/fixterms/

It comes a little whiney and amateur at the moment, because I'm no good
at the social engineering side of trying to motivate a world-full of
1970's-era engineers into trying to fix a big problem. :) But the
technical details are there, awaiting someone with better writing skills
than I to neaten it up a bit.

signature.asc

Philippe Vaucher

unread,
Mar 18, 2011, 5:38:45 AM3/18/11
to vim_dev

> > As long as the two triplets of keypresses I suggested originally can all
> > be represented uniquely, and without reference to timing information in
> > the Escape vs Alt+ case, then I'm happy with whatever internal
> > implementation makes it happen.
>
> The two triplets in question being
>
>   <Tab>
>   <Ctrl-I>
>   <Ctrl-Shift-I>
>
>   <Escape> C
>   <Alt+C>
>   é
>
> Yes I am aware that on current byte-driven terminals there is no way
> without using timing information to distinguish all these three cases in
> the latter triplet here. This should not stop GUI systems from
> distinguishing them however.


Can someone (Bram?) tell us if Bram's proposed solution would allow us
to map those triplets independently as wanted? I'm not familiar enough
about vim's byte queue system to tell if the modifier byte really
answers our needs here.

Thanks.
Philippe

Ben Schmidt

unread,
Mar 18, 2011, 6:15:18 AM3/18/11
to vim...@googlegroups.com
On 18/03/11 8:38 PM, Philippe Vaucher wrote:
>>> As long as the two triplets of keypresses I suggested originally can all
>>> be represented uniquely, and without reference to timing information in
>>> the Escape vs Alt+ case, then I'm happy with whatever internal
>>> implementation makes it happen.
>>
>> The two triplets in question being
>>
>> <Tab>
>> <Ctrl-I>
>> <Ctrl-Shift-I>
>>
>> <Escape> C
>> <Alt+C>
>> �

>>
>> Yes I am aware that on current byte-driven terminals there is no way
>> without using timing information to distinguish all these three cases in
>> the latter triplet here. This should not stop GUI systems from
>> distinguishing them however.
>
> Can someone (Bram?) tell us if Bram's proposed solution would allow us
> to map those triplets independently as wanted?

Of course it does. Bram wouldn't have suggested it if it wouldn't work
at all....

> I'm not familiar enough about vim's byte queue system to tell if the
> modifier byte really answers our needs here.

I don't think we need to keep the same byte-stream representation as we
currently have, and I don't think that's what Bram meant. I think we can
change it. As long as it remains a byte-stream representation.

And I don't think it will be possible to do what we need without at
least extending it. At the moment, keys are a fixed length of 3 bytes
(one of which is the 'escape byte' 0x80, and the other two of which have
restricted values). There simply isn't enough value-space to do what we
need, and it's possibly not very future-proof even if we can squeeze it
in. It's also a bit of a mess, IIRC. It would be better to give it an
overhaul. And it probably needs to use variable-length sequences.

A thought occurred to me a couple of days ago: would it be appropriate
to actually use CSI as the byte-stream representation? Or something like
it. It is a byte-stream representation, right? And it's extensible,
right, so it is less likely to need a big overhaul in future? Does it
allow for 'private use' stuff? Because Vim needs a way to represent a
bunch of other events as keystrokes (mouse clicks, scroll wheel, its
magic script-local meta-character thing, representing the CSI escape
character itself, etc.). Also, is it easy to skip over CSI sequences if
necessary with simple logic?

I think there are some nice/helpful things that could be done
design-wise though. For example, Vim is often used with enc=utf-8 these
days. I think it would be wise to, if possible, ensure all key sequence
representations are invalid UTF-8 in their entirety. Then they could be
put in buffers and yanked back into registers without ambiguity
(hopefully), i.e. there wouldn't be the ambiguity of whether the escape
character was part of a key sequence that should be yanked as such, or
part of the buffer text which should be escaped upon yanking.

See, Vim already deals with the CSI character in its byte-stream code,
and escapes it, so some aspects of this are already somewhat accounted
for. But I wonder when precisely it is escaped, and whether at times it
is escaped when it shouldn't be, or not when it should be. It would be
good to design to avoid this problem.

(I happened to be hacking on some Vim code recently where this came into
play, so I noted it with interest.)

Ben.

Paul LeoNerd Evans

unread,
Apr 4, 2011, 8:22:16 AM4/4/11
to vim...@googlegroups.com
On Fri, Mar 18, 2011 at 09:15:18PM +1100, Ben Schmidt wrote:
> A thought occurred to me a couple of days ago: would it be appropriate
> to actually use CSI as the byte-stream representation? Or something like
> it. It is a byte-stream representation, right? And it's extensible,
> right, so it is less likely to need a big overhaul in future? Does it
> allow for 'private use' stuff? Because Vim needs a way to represent a
> bunch of other events as keystrokes (mouse clicks, scroll wheel, its
> magic script-local meta-character thing, representing the CSI escape
> character itself, etc.). Also, is it easy to skip over CSI sequences if
> necessary with simple logic?

Sure. CSI is intended for just this. A CSI sequence is matched by a
regexp

CSI [\x20-\x2f]? [\x30-\x3f]* [\x40-\x7e]

Rougly "possibly-one punctuation charact