Dear Bram
graywh
libtermkey so that LeoNerd can fix terminal input and Gvim and
everyone will be happy. Thanks.
Erik Falor
Bram Moolenaar
Graywh wrote:
Send me a patch and I'll look into it.
If you want a more useful response, use your real name and include more
information about what you actually want, what system, etc.
--
hundred-and-one symptoms of being an internet addict:
195. Your cat has its own home page.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
Paul LeoNerd Evans
> Graywh wrote:
>
> > Please fix vim's input queue mechanism with key info structures a.la
> > libtermkey so that LeoNerd can fix terminal input and Gvim and
> > everyone will be happy. Thanks.
>
> Send me a patch and I'll look into it.
>
> If you want a more useful response, use your real name and include more
> information about what you actually want, what system, etc.
Hi Bram,
I would like this. For all systems. Everywhere.
This is a large undertaking. This isn't a "send me a patch" request.
I am talking about ripping out the byte-queue input system and replacing
it with a structured keypress queue. The lot. Everything. Remove the
"stuck in the 1970s all the world is a DEC-style glass virtual terminal"
input queue, and replace it with one that has structures; remembers
modifiers, etc.. With a structure we can properly represent "Ctrl-I" as
being the letter I with a control key, totally separate from the Tab.
Having done this, we can feed in the input events from a terminal in
EXACTLY the same way it currently works. We can take events out to match
them up to map/etc.. much neater. We can feed events in from GTK or
similar GUIs, without having to go near legacy 1970s terminal
technology, meaning that GTK (et.al) users can map Tab, Ctrl-I and
Ctrl-Shift-I, all independently of each other.
Having done this, I can then quite easily send you a patch to use
libtermkey if that is wanted, because then it will be able to recognise
Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
properly-configured xterm. But that is PURELY an extension mechanism. My
primary request is to fix the underlying queue mechanism.
This is an issue that has been dragging on for years now - 7 or 8 years
by my count. 7 or 8 years I have been sitting in #vim on Freenode; in
all that time almost every week we'll get someone or other asking how
they can map some key or other, that we have to keep explaining "No,
sorry, you can't. Not even in GTK.".
I am happy to sit and discuss it with you for potentially hours if
necessary. What I cannot do is "send a patch" - it's not a small simple
few-lines fix like that.
I think it is about time we ended this unfortunate situation, and
properly fixed the real underlying mechanism.
Yours, with my real name,
--
Paul "LeoNerd" Evans
leo...@leonerd.org.uk
ICQ# 4135350 | Registered Linux# 179460
http://www.leonerd.org.uk/
Ingo Karkat
> On Fri, Feb 04, 2011 at 07:14:29PM +0100, Bram Moolenaar wrote:
>>
>> Send me a patch and I'll look into it.
>>
>
I would love to have this, too, if only because after so many happy years with
Vim (a heartfelt thanks to Bram and all developers!), I'm running out of keys
for my custom mappings :-)
Though I'm too ignorant of the internals (and can therefore only offer to do
some testing), I agree that this probably cannot be done in a single patch. But,
with the Mercurial repository, it should be easy to create a feature branch that
allows devs to collaborate and test this, and only have this merged in after
it's done (hopefully in time for Vim 7.4 / 8.0?).
-- regards, ingo
Bram Moolenaar
Paul Evans wrote:
> I would like this. For all systems. Everywhere.
>
> This is a large undertaking. This isn't a "send me a patch" request.
>
> I am talking about ripping out the byte-queue input system and replacing
> it with a structured keypress queue. The lot. Everything. Remove the
> "stuck in the 1970s all the world is a DEC-style glass virtual terminal"
> input queue, and replace it with one that has structures; remembers
> modifiers, etc.. With a structure we can properly represent "Ctrl-I" as
> being the letter I with a control key, totally separate from the Tab.
>
> Having done this, we can feed in the input events from a terminal in
> EXACTLY the same way it currently works. We can take events out to match
> them up to map/etc.. much neater. We can feed events in from GTK or
> similar GUIs, without having to go near legacy 1970s terminal
> technology, meaning that GTK (et.al) users can map Tab, Ctrl-I and
> Ctrl-Shift-I, all independently of each other.
You can just keep the existing queue and make this work. It actually
already works for most keys, using a modifier sequence. Using
structures only makes it bigger.
Main problem is that this won't be backwards compatible, currently
CTRL-I is the same as Tab. I don't know how to make this work without
breaking backwards compatibilty.
> Having done this, I can then quite easily send you a patch to use
> libtermkey if that is wanted, because then it will be able to recognise
> Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
> properly-configured xterm. But that is PURELY an extension mechanism. My
> primary request is to fix the underlying queue mechanism.
>
> This is an issue that has been dragging on for years now - 7 or 8 years
> by my count. 7 or 8 years I have been sitting in #vim on Freenode; in
> all that time almost every week we'll get someone or other asking how
> they can map some key or other, that we have to keep explaining "No,
> sorry, you can't. Not even in GTK.".
Actually, in the GUI many key combinations do work. Not all though.
> I am happy to sit and discuss it with you for potentially hours if
> necessary. What I cannot do is "send a patch" - it's not a small simple
> few-lines fix like that.
>
> I think it is about time we ended this unfortunate situation, and
> properly fixed the real underlying mechanism.
>
> Yours, with my real name,
>
> --
> Paul "LeoNerd" Evans
--
A cow comes flying over the battlements, lowing aggressively. The cow
lands on GALAHAD'S PAGE, squashing him completely.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
Davertron
On Feb 17, 11:48 am, Paul LeoNerd Evans <leon...@leonerd.org.uk>
wrote:
> ICQ# 4135350 | Registered Linux# 179460http://www.leonerd.org.uk/
>
> signature.asc
> < 1KViewDownload
Paul LeoNerd Evans
> You can just keep the existing queue and make this work. It actually
> already works for most keys, using a modifier sequence. Using
> structures only makes it bigger.
OK, well which ever way works.. I'm not fussy about the internals of
implementation. So long as it is capable of knowing that Enter is not
Ctrl-M is not Ctrl-Shift-M, I am happy.
> Main problem is that this won't be backwards compatible, currently
> CTRL-I is the same as Tab. I don't know how to make this work without
> breaking backwards compatibilty.
:set oldterminalmode
:map <Ctrl-I> <Tab>
:map <Ctrl-M> <CR>
For that matter, make it default to this 1970s-like legacy behaviour and
if people want the ability to map newly-recognised keys,
:set newterminalmode
> Actually, in the GUI many key combinations do work. Not all though.
Well, then lets make them all work. Please?
Seriously - I am not the only person here. There is a constant stream of
users on #vim, every week you can guarantee someone will want to map
Ctrl-I or Shift-Space or Shift-Enter or some other key combination that
seems perfectly logical to any outside observer, to want to map. Every
week we have to tell them "no". I'm getting tired of it, when there
isn't any technical reason why it can't be done, only "it hasn't been
done this way so far".
So can we please have a plan? If you care deeply about retaining
1970s-like semantics, I'd be happy for a compromise solution. Some
boolean setting that defaults to a value whereby these aliases do occur,
where Ctrl-I and Ctrl-Shift-I -are- synonymous with Tab. But implemented
in a way where I or whichever knowledgable user can say "Yes vim, I know
that Ctrl-I used to mean Tab in the 1970s but I would like an extra
Ctrl key now and 'I' looks a good choice - let me have it". And then I
could toggle setting, and map it. Yes, I'm aware that not -every-
terminal can represent that. Yes I'm aware if I ever go use a real DEC
VT220 glass teletype I've now lost the ability to type it. But that
should be -my- problem - right now vim pointlessly restricts a whole
selection of possible key mappings from GTK or Mac OS X or Win32 or
proper xterms or all sorts of other places, simply for the
backward-looking legacy of retaining compatibility with what the
behaviour used to be in a time before many vim users were even born.
I think it's about time we opened this up a bit, and allowed users to
map the keypresses they want to.
Paul LeoNerd Evans
> already works for most keys, using a modifier sequence. Using
> structures only makes it bigger.
Oh, and I'm not sure I follow your logic here.
Yes, it would make it bigger. By maybe bytes. Consider
struct keypress {
uint16_t flags; /* Unicode or special */
uint16_t modifiers; /* A bitmask */
uint32_t codepoint;
};
that's 8 bytes. As compared a single byte keypress that's 7 bytes
bigger. Less so, compared to e.g. a 3 or 4 byte modifier prefixed
sequence. Not a massive growth here. Lets be really pessimistic and say
we're using 7 bytes extra. Lets now say we're really memory-constrained,
and can only afford a single extra kilobyte of memory. That's still 146
keypresses of queue we can afford. Is vim's queue that big? Can we
really afford no more than a single kilobyte?
Ofcourse, this was a simplistic structure. If we care more about saving
memory at a slight CPU overhead we could
struct keypress {
unsigned int is_special : 1;
unsigned int modifiers : 10;
unsigned int codepoint : 21;
};
that's 4 bytes, to store the same information, noting that Unicode is
only a 21bit code space, a Unicode/special flag, and storage space for
10 modifiers.
4 bytes.
Those variable-length-encoded prefixed keypresses are getting on for
that long anyway. Plus now we're in a fixed-length encoding, so perhaps
we've saved CPU time and associated code space to store the program used
to encode and decode them.
I'm not sure an argument of memory consumption can be made against using
this encoding scheme.
Plus, it's only the queue of incoming keypresses - that queue isn't
going to stay very big for very long.
Ben Schmidt
> going to stay very big for very long.
It's not just the input queue that's in question here, it's everywhere
in Vim where keypresses are represented. For instance, the right hand
sides of mappings are not primarily characters, but lists of keypresses.
They need the same amount of expressive power to work properly.
When macros are recorded, registers, which usually are primarily lists
of characters, are used to store keypresses. Likewise, for feedkeys() to
work, its input, a string, needs to be able to represent keypresses.
And I'm sure there are plenty more subtleties.
The bottom line, though, is that changing to a struct-based approach
could make the job absolutely huge, requiring reworking and/or
redesigning how maps, registers, etc. all work. And it might not even be
possible since, e.g. registers need to be able to do both characters and
keys. It would be much simpler to extend the current approach which uses
an 'escape mechanism' for representing special keys. Just hit qq and
type some arrow keys, q to end recording, and then inspect register q
and you'll see what's going on here. Vim uses a byte <80> followed by
two more bytes to represent special keys.
IMHO, what needs to be done is for this to be tidied up and made
consistent and documented so that these codes can be more readily
generated, interpreted, and even viewed and understood by users (e.g.
when I do :registers, it would be nice to see <S-Up> rather than
<80><fd>^D, when appropriate). Included in this is a clear specification
of what control characters mean, e.g. does ^I (<09>) mean tab or
control-I? And which of those keys can or does use the escape mechanism?
Then the behaviour of mappings needs to be defined--if there is a
mapping for ^I (<09>) and I push tab, will it be triggered? If in a
terminal which can't distinguish control-I and tab, and a ^I is
received, should the mapping for Tab or control-I be triggered? If
there's a mapping for ^I as well as Tab, which has precedence? All these
kinds of questions need clear answers, and sensible specifications and
design need to address them, avoid ambiguity, and take care to require
as little as possible work for users, plugin authors, etc. to update
their code and mappings.
Then the input code needs to be reworked in all the GUIs and in the
terminal handling to generate the appropriate internal codes,
consistently across all the different GUIs, etc., in line with that
specification/design.
Ben.
Adrien "Axioplase" Piérard
> in Vim where keypresses are represented.
> And I'm sure there are plenty more subtleties.
What I see here, if done, is a potential new major version of vim.
Such a new version may trade off backward compatibility for up-to-date
technology and/or design constraints.
Since the changes may be huge, I concur with the suggestion of
creating a branch, and having discussions *and* documentation on its
evolution.
You raised some very interesting questions, and I hope that other
people who know about the internals give their point of view too.
As far as memory usage is concerned, I am now very egoistic and
believe that I have enough RAM to run vim. If I don't, then I'll use
vi, or an old version of vim.
I used to collect and use old hardware, but I'm now getting my job
done and bringing home the bacon thanks to vim on machines with more
memory that I could think of not so long ago…
Anyway,
Cheers,
P!
--
Français, English, 日本語, 한국어
Stephen Lee
small suggestion:
Is it possible to put a parameter in .vimrc like:
set keymap=byte-queue, structured, etc..
(default is byte-queue)
so that both sets of mapping could be kept...
sorry if I see the problem too surface.
> /// sponsor Vim, vote for features --http://www.Vim.org/sponsor/\\\
> \\\ an exciting new programming language --http://www.Zimbu.org ///
Stephen Lee
the key mapping definition files like dvorak.vim, 101-keyboard.vim,
etc. and let people set in .vimrc?
It may be more flexible than just new or old term mode...
Philippe Vaucher
hard to make everyone happy on this subject by the mean of .vimrc
settings, like we have between vi and vim for :set compatible.
Wether we use structures or the byte 80 for special key is
implementation detail, I mean wether macros, mappings or registers
simply store a list of bytes or a list of structures shouldn't change
anything: you just need an explicit mapping between a displayed
keypress and a byte or a struct (<S-Up> can map internally to \x80\xFD
\xAA or { 1, MOD_SHIFT, KEY_UP } and it shouldn't matter).
Structures just look more sane as they're more generic/expressive and
accurate about the problem, they also are easier to expand on.
Philippe
Paul LeoNerd Evans
> FWIW, you have my vote for making this happen. I think it's not that
> hard to make everyone happy on this subject by the mean of .vimrc
> settings, like we have between vi and vim for :set compatible.
:set compatible does suggest the possibility for more similar options,
ways to optionally extend the featureset beyond its existing capability,
without upsetting people who rely on the current behaviour, limits and
all.
> Wether we use structures or the byte 80 for special key is
> implementation detail, I mean wether macros, mappings or registers
> simply store a list of bytes or a list of structures shouldn't change
> anything: you just need an explicit mapping between a displayed
> keypress and a byte or a struct (<S-Up> can map internally to \x80\xFD
> \xAA or { 1, MOD_SHIFT, KEY_UP } and it shouldn't matter).
>
> Structures just look more sane as they're more generic/expressive and
> accurate about the problem, they also are easier to expand on.
Another thought occurs to me: UTF-8 sequences vs. Alt keys.
I don't know if it's an accident of implementation, but I have never
been able to get them all working. Lets consider this example
é is U+00e9 in Unicode, encodes to two UTF-8 bytes C3 A9
C is U+0043 in Unicode, encodes to one UTF-8 byte 43
If vim simply uses an 8th-bit-high to indicate Alt+ prefix on simple
ASCII, then what does
C3
mean? Is it the first of two UTF-8 bytes, or is it Alt+C ?
Alternatively, perhaps we'll represent it as a two-byte sequence with an
<Escape> prefix. Now we've conflated someone typing <Escape> C from
someone typing Alt+C. Again, in the 1970s that may be what it was, but
these days it's far from clear from GUIs, that this behaviour is what's
wanted. Again, a simple 'set' option should default this:
:set noaltisescape (defaults to :set altisescape)
The reason I suggested a simple neat orthogonal structure representation
is that it can disambiguate all of the following cases in a clear
logical way; a way that can be easily fed from GUI events, and fed into
map tables or other internals.
Tab { 1, 0, KEY_TAB }
Ctrl-I { 0, MOD_CTRL, 'i' }
Ctrl-Shift-I { 0, MOD_CTRL, 'I' }
Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
Alt+C { 0, MOD_ALT, 'C' }
é { 0, 0, 0xe9 }
Paul LeoNerd Evans
> >Plus, it's only the queue of incoming keypresses - that queue isn't
> >going to stay very big for very long.
>
> It's not just the input queue that's in question here, it's everywhere
> in Vim where keypresses are represented. For instance, the right hand
> sides of mappings are not primarily characters, but lists of keypresses.
> They need the same amount of expressive power to work properly.
Yes. And the scheme I created is no less expressive than the byte
queues. In fact it is more expressive, able to disambiguate neatly
situations that cannot normally be represented - see also my other mail
on the thread a few minutes ago.
http://groups.google.com/group/vim_dev/msg/7bb4cc9b127cddb3
> When macros are recorded, registers, which usually are primarily lists
> of characters, are used to store keypresses. Likewise, for feedkeys() to
> work, its input, a string, needs to be able to represent keypresses.
>
> And I'm sure there are plenty more subtleties.
Yes - again, my point of these structures is that they exactly _do_
represent these keypresses.
> The bottom line, though, is that changing to a struct-based approach
> could make the job absolutely huge, requiring reworking and/or
> redesigning how maps, registers, etc. all work. And it might not even be
> possible since, e.g. registers need to be able to do both characters and
> keys.
But every character -can- be represented as a key - namely, the key that
generates it. Let me again repeat my structure:
struct keypress {
unsigned int is_special : 1;
unsigned int modifiers : 10;
unsigned int codepoint : 21;
};
Any character has a Unicode codepoint (cp). That's represented by
{ 0, 0, cp }
Any keypress has a symbolic key number, taken from some arbitrary
enumeration - I care not where.
{ 1, mod, key }
Now, we can also represent those modified Unicode keys such as Ctrl-I
and Shift-Space which previously were impossible:
{ 0, MOD_CTRL, 'i' } /* Ctrl-I */
{ 0, MOD_SHIFT, ' ' } /* Shift-Space */
> It would be much simpler to extend the current approach which uses
> an 'escape mechanism' for representing special keys. Just hit qq and
> type some arrow keys, q to end recording, and then inspect register q
> and you'll see what's going on here. Vim uses a byte <80> followed by
> two more bytes to represent special keys.
>
> IMHO, what needs to be done is for this to be tidied up and made
> consistent and documented so that these codes can be more readily
> generated, interpreted, and even viewed and understood by users (e.g.
> when I do :registers, it would be nice to see <S-Up> rather than
> <80><fd>^D, when appropriate). Included in this is a clear specification
> of what control characters mean, e.g. does ^I (<09>) mean tab or
> control-I? And which of those keys can or does use the escape mechanism?
OK - well, if you feel confident that the existing prefix-escape
mechanism can completely an unambiguously represent all these possible
keystrokes, then sure, that way might result in less code change
overall.
But do make quite sure it can represent them. Specifically consider the
two tricky special-cases I suggested - repeated here again:
Tab { 1, 0, KEY_TAB }
Ctrl-I { 0, MOD_CTRL, 'i' }
Ctrl-Shift-I { 0, MOD_CTRL, 'I' }
Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
Alt+C { 0, MOD_ALT, 'C' }
é { 0, 0, 0xe9 }
> Then the behaviour of mappings needs to be defined--if there is a
> mapping for ^I (<09>) and I push tab, will it be triggered? If in a
> terminal which can't distinguish control-I and tab, and a ^I is
> received, should the mapping for Tab or control-I be triggered? If
> there's a mapping for ^I as well as Tab, which has precedence?
I propose a pair of boolean settings, with the following defaults:
:set nomodifiedunicode
:set altisescape
Under these settings,
:map <Ctrl-I> ... vs :map <Tab> ... shall have the same effect, each
overwriting the effects of the other; last one wins.
Pressing the Tab key or Ctrl-I shall both invoke the last mapping
registered.
:map <Escape>C ... vs :map <Alt+C> ... shall behave similarly, each
overwrites the other. Typing either key sequence will invoke the
last map to be registered.
If a user decides "I want to use those extra Ctrl- keys", they can set
in their .vimrc
:set modifiedunicode
At this point,
:map <Ctrl-I> and :map <Tab> shall fill two -different-
slots of the mapping list, and typing either key will activate the
indicated mapping.
If a user further decides "I want the Alt modifier to not mean Escape
prefixing", then
:set noaltisescape
At which point,
:map <Escape>C and :map <Alt+C> shall also be different.
The only remaining ambiguity to be answered is, what happens in the
following case:
:set nomodifiedunicode
:map <Tab> ONE
:map <Ctrl-I> TWO
:set modifiedunicode
now press Tab or Ctrl-I
I don't have an easy answer for this case; no particular behaviour jumps
out at me as "obviously correct".
> All these
> kinds of questions need clear answers, and sensible specifications and
> design need to address them, avoid ambiguity, and take care to require
> as little as possible work for users, plugin authors, etc. to update
> their code and mappings.
>
> Then the input code needs to be reworked in all the GUIs and in the
> terminal handling to generate the appropriate internal codes,
> consistently across all the different GUIs, etc., in line with that
> specification/design.
I propose that the structure approach makes this simpler. It is now
clearly obvious how e.g. GTK should fill in these structures, as GDK
keypress events already have unicode / modifier bits or symbolic keys.
It's also obvious how to render such a structure as a string, possibly
wrapping <> around it, possibly prefixing C- or A- or whatever as
appropriate.
(( This sort of behaviour is already implemented by libtermkey, even
going so far as to support
char buffer[256];
termkey_snprint_key(tk, buffer, sizeof buffer, &key, TERMKEY_FORMAT_VIM);
for exactly this purpose. :) ))
Ben Schmidt
>>> going to stay very big for very long.
>>
>> It's not just the input queue that's in question here, it's everywhere
>> in Vim where keypresses are represented. For instance, the right hand
>> sides of mappings are not primarily characters, but lists of keypresses.
>> They need the same amount of expressive power to work properly.
>
> Yes. And the scheme I created is no less expressive than the byte
> queues. In fact it is more expressive, able to disambiguate neatly
> situations that cannot normally be represented - see also my other mail
> on the thread a few minutes ago.
Yeah, there was never anything lacking in your scheme in terms of
expressive power. It's just that it's not only the input queue that
would need to use it. It's a lot more.
>> When macros are recorded, registers, which usually are primarily lists
>> of characters, are used to store keypresses. Likewise, for feedkeys() to
>> work, its input, a string, needs to be able to represent keypresses.
>>
>> And I'm sure there are plenty more subtleties.
>
> Yes - again, my point of these structures is that they exactly _do_
> represent these keypresses.
Yes. But what happens when you then edit that macro by putting the
register into a buffer, changing it, and yanking it again? This is not
uncommonly done. How should the registers be stored in .viminfo? How do
you write the input to the feedkeys function as a string in vimscript?
Etc.. These are the kinds of issues I was trying to raise.
>> The bottom line, though, is that changing to a struct-based approach
>> could make the job absolutely huge, requiring reworking and/or
>> redesigning how maps, registers, etc. all work. And it might not even be
>> possible since, e.g. registers need to be able to do both characters and
>> keys.
>
> But every character -can- be represented as a key - namely, the key that
> generates it.
When there is one. Or a codepoint in some encoding. Yeah, your struct
does allow for that. The struct isn't inexpressive, it's just that it's
big, and if it needs to be used in a LOT of places (which seem to just
be increasing...input queue, mappings, registers, strings, buffers?!),
suddenly not just one thing is getting a bit bigger--a lot of things are
getting bigger--potentially a lot bigger! If every time you yank text
into a register it gets 12 times larger, that may cause some problems!
> Let me again repeat my structure:
>
> struct keypress {
> unsigned int is_special : 1;
> unsigned int modifiers : 10;
> unsigned int codepoint : 21;
> };
>
> Any character has a Unicode codepoint (cp). That's represented by
Well, a codepoint in whatever encoding you are using.
> { 0, 0, cp }
>
> Any keypress has a symbolic key number, taken from some arbitrary
> enumeration - I care not where.
>
> { 1, mod, key }
>
> Now, we can also represent those modified Unicode keys such as Ctrl-I
> and Shift-Space which previously were impossible:
>
> { 0, MOD_CTRL, 'i' } /* Ctrl-I */
> { 0, MOD_SHIFT, ' ' } /* Shift-Space */
Yeah. The benefit here is that different keyboard layouts can be
represented, e.g. keyboards which have single keys for accented
characters can represent those keys with modifiers, easily.
> OK - well, if you feel confident that the existing prefix-escape
> mechanism can completely an unambiguously represent all these possible
> keystrokes, then sure, that way might result in less code change
> overall.
I think with some careful design it could, which may well require some
reworking of how it's currently done. There is the issue, of course,
that <80> is a valid character in some encodings, too, and I don't know
if this is accounted for. It needs to be. So maybe the escape mechanism
needs to be a bit more formal (if it can be--but maybe there's no way
around this, or maybe that's a longer-term consideration). Definitely
the issues have to be bashed out. I still think this would be the
easiest and best way forward, though. But it's far from my call!
Regarding that whole meta issue that you raised earlier--I think, yes,
using an 8-bit-high representation for meta is completely out of the
question. Part of the input code should be transforming that to a proper
Vim-internal escape sequence for terminals that use it. Same for other
things. The input code should transform input into Vim's internal
representation which should be carefully designed not to be ambiguous,
etc., but which, nevertheless, is at its most fundamental, a byte stream
so it can be used in the input queue, mappings, registers, strings and
even buffers.
> But do make quite sure it can represent them. Specifically consider the
> two tricky special-cases I suggested - repeated here again:
>
> Tab { 1, 0, KEY_TAB }
> Ctrl-I { 0, MOD_CTRL, 'i' }
> Ctrl-Shift-I { 0, MOD_CTRL, 'I' }
>
> Escape C { 1, 0, KEY_ESCAPE }, { 0, 0, 'C' }
> Alt+C { 0, MOD_ALT, 'C' }
> é { 0, 0, 0xe9 }
Yes. This all has to be carefully accounted for.
Maybe the way to do it is to have printable characters represented just
as printable characters, 'default' control characters optionally just as
themselves (e.g. <09> for tab), to use when a terminal that can't
distinguish between this and Ctrl-I is used, and (whenever possible),
use <80>+flag_byte+character for control/special keys. One of the
flag_byte flags could be a 'special' flag, which means the 'character'
in the third byte would be some kind of mnemonic for the key (e.g. u for
the up arrow). When the key is not 'special' the character it produces
would be the third byte (and perhaps later for multibyte characters,
e.g. if ctrl is used with an accented character on keyboards that have a
specific key for this). How all this works depends a bit on how the
keyboard interfacing works--maybe it should just use keycodes if
characters are not readily obtainable. And maybe you need two flag bytes
if there are more than 7 modifier keys that need to be distinguished.
>> Then the behaviour of mappings needs to be defined--if there is a
>> mapping for ^I (<09>) and I push tab, will it be triggered? If in a
>> terminal which can't distinguish control-I and tab, and a ^I is
>> received, should the mapping for Tab or control-I be triggered? If
>> there's a mapping for ^I as well as Tab, which has precedence?
>
> I propose a pair of boolean settings, with the following defaults:
>
> :set nomodifiedunicode
> :set altisescape
>
> Under these settings,
>
> :map<Ctrl-I> ... vs :map<Tab> ... shall have the same effect, each
> overwriting the effects of the other; last one wins.
>
> Pressing the Tab key or Ctrl-I shall both invoke the last mapping
> registered.
>
> :map<Escape>C ... vs :map<Alt+C> ... shall behave similarly, each
> overwrites the other. Typing either key sequence will invoke the
> last map to be registered.
>
> If a user decides "I want to use those extra Ctrl- keys", they can set
> in their .vimrc
>
> :set modifiedunicode
>
> At this point,
>
> :map<Ctrl-I> and :map<Tab> shall fill two -different-
> slots of the mapping list, and typing either key will activate the
> indicated mapping.
What should Vim do in a terminal that can't distinuish between those
keys when this option is in effect? This is the same question as below,
really.
Maybe we need an annotation to the mapping, like <default> or something
that says "this mapping is acceptable to use when we're not sure if this
exact combo was pressed, but we got a code that 'could be it' in some
terminal."
> If a user further decides "I want the Alt modifier to not mean Escape
> prefixing", then
>
> :set noaltisescape
>
> At which point,
>
> :map<Escape>C and :map<Alt+C> shall also be different.
I think there might already be an option for this. Or something in
termcap? I seem to remember reading some stuff about it in the Vim
manual. I guess, though, the 'non-alt prefixing' of Vim probably means
to use 8-bit-high for meta, which we already said, is pretty useless.
I'm not sure two options are needed. Probably a single option suffices
to deal with both situations.
> The only remaining ambiguity to be answered is, what happens in the
> following case:
>
> :set nomodifiedunicode
> :map<Tab> ONE
> :map<Ctrl-I> TWO
> :set modifiedunicode
>
> now press Tab or Ctrl-I
>
> I don't have an easy answer for this case; no particular behaviour jumps
> out at me as "obviously correct".
I agree. If we go with the <default> annotation to mappings (preferably
with a better keyword), I guess that is the one that wins and takes
precedence. It may not even need to delete the other ones, so both slots
could remain active, but it would just run the 'default' one.
>> All these
>> kinds of questions need clear answers, and sensible specifications and
>> design need to address them, avoid ambiguity, and take care to require
>> as little as possible work for users, plugin authors, etc. to update
>> their code and mappings.
>>
>> Then the input code needs to be reworked in all the GUIs and in the
>> terminal handling to generate the appropriate internal codes,
>> consistently across all the different GUIs, etc., in line with that
>> specification/design.
>
> I propose that the structure approach makes this simpler. It is now
> clearly obvious how e.g. GTK should fill in these structures, as GDK
> keypress events already have unicode / modifier bits or symbolic keys.
> It's also obvious how to render such a structure as a string, possibly
> wrapping<> around it, possibly prefixing C- or A- or whatever as
> appropriate.
That's good for mappings, and I guess the \< escape in vimscript
strings, but it doesn't help for registers (which can end up in
buffers), etc..
Maybe it's a sensible 'middle point', though, for the input queue. The
GUI specific code could go into a structure like this, which generic
code can then translate into internal byte-stream representation as
appropriate, and then that byte-stream representation is the standard
throughout the rest of Vim. Basically the structure would only be used
as a mechanism to deal with the code that deals with hardware
keypresses, making it consistent between GUIs, platforms, etc..
> (( This sort of behaviour is already implemented by libtermkey, even
> going so far as to support
>
> char buffer[256];
> termkey_snprint_key(tk, buffer, sizeof buffer,&key, TERMKEY_FORMAT_VIM);
>
> for exactly this purpose. :) ))
Ben.
Philippe Vaucher
> Yes. But what happens when you then edit that macro by putting the
> register into a buffer, changing it, and yanking it again? This is not
> uncommonly done. How should the registers be stored in .viminfo? How do
> you write the input to the feedkeys function as a string in vimscript?
> Etc.. These are the kinds of issues I was trying to raise.
with some expansions e.g "<C-A><S-I><Tab>somechars<U-234>". I mean
this is simple key-as-text representation and then text-representation-
to-struct creation no?
For example, the user records a macro into register q that is then
displayed as "ihello<Tab>worlld<Esc><Left><Left>x". He then edits the
macro in a buffer, yanks it into register q, and when @q is executed
all it does is interpret the string?
We'll probably need some escape mechanism to differentiate between the
text "<Tab>" and the tab key, but this looks fairly easy to find no?
Wether it being a special byte before the <> block or some \ or
whatever.
Maybe I'm oversimplifying it but it'd help if you were more specific
about the "issues" the new design have. To me it looks like we just
need to agree on a new set of conventions by describing how the new
system would work for all the current (and future) macros/registers/
whatever scenarios.
I suggest maybe we make some kind of "new design" draft which would
answer all the issues raised (memory size/1970 support/macros
concerns) ?
Philippe
Milan Vancura
> register into a buffer, changing it, and yanking it again? This is not
> uncommonly done. How should the registers be stored in .viminfo? How do
> you write the input to the feedkeys function as a string in vimscript?
> Etc.. These are the kinds of issues I was trying to raise.
Hi.
Wouldn't be it same as now, only used more often? There is already a
possibility to write "<F4>" or "<S-Space>". The only problem is that, at least
the second, we can't press strongly enough to push it to vim :-)
But, on the other hand, you are right there still will be (and must be)
ambiguities. We can't do anything about that, in general - it's a user who must
decide how does he want to understand his keyboard.
For example: I, as Czech, have some Czech accented letters accessible via
modifier+key on my keyboard. To make the example more specific, think about
Mod5+s as "s with hook" (U0161) and redefined my keymap so Capslock key acts as
Mod5. It's up to me, and only me, if I define some vim mapping as
:map <Mod5+S> ...rhs...
- or -
:map <U0161> ...rhs...
Both do the same on my current keyboard but start to behave differently if I
change my keyboard setting in X window system, of course. If I switched to
another kind of Czech keyboard (called "typewriter one"), <U0161> appears at a
key of "number 3" and Mod5 would be on right Alt or not defined at all.
As I wrote above, we can't do anything about that, as far as I know.
Milan
--
Milan Vancura, Prague, Czech Republic, Europe
Stephen Lee
layouts instead of forcing any specific one to them.
In your case it seems you are talking about QWERTY and QWERTZ (from
wiki), so the following would be keyboard layouts in the new directory
called "Keyboard"
(General keyboard layouts)
QWERTY.vim
QWERTZ.vim
AZERTY.vim
QZERTY.vim
Dvorak.vim
Colemak.vim
JCUKEN.vim
Neo.vim
Turkish.vim
.
.
OR (Keyboard layout by Country Name)
and there would be 4 files:
QWERTYsequence.vim
QWERTZsequence.vim
QWERTYstructured.vim
QWERTZstructured.vim
OR (specific for your country)
CzechQWERTYsequence.vim
CzechQWERTZsequence.vim
CzechQWERTYstructured.vim
CzechQWERTZstructured.vim
OR simply (having advantage of adding new keyboard layouts in future
but disadvantage of difficult to find which is which when changing
them in command)
CzechLayoutA1.vim
CzechLayoutA2.vim
CzechLayoutB1.vim
CzechLayoutB2.vim
In .vimrc, there would be:
set keyboardlayout=....
Maybe a step forward to change the layout on-the-fly by the following
command (when changing keyboard setting in X window system):
:set keyboardlayout=...
I am not a programmer, but the concept is to make vim a converter and
convert keys on-the-fly:
Input ---> vim(search and map in the layout file) ---> output
The concept is inspired by the following plugins:
VimIM : Vim Input Method
ywvim : Another input method(IM) for VIM, supports all modes
and the Keyboard layout system in Windows because when inputting
Chinese we rely heavily on mapping different keys so as to generate
one Chinese character.
All we need would be desiging 2 sets of clear layout files for every
different kind of keyboard layout
Hope this help.
Stephen Lee
I just browsed through vim and found a directory called "keymap"
can we reuse and change or add different layout files in there?
Adrien "Pied" Piérard
2011/2/23 Stephen Lee <stephenl...@gmail.com>:
> Therefore I propose letting users choose their preferred keyboard
> layouts instead of forcing any specific one to them.
> In your case it seems you are talking about QWERTY and QWERTZ (from
> wiki), so the following would be keyboard layouts in the new directory
> called "Keyboard"
>
> (General keyboard layouts)
> QWERTY.vim
> ...
Hum, as a user of a home-made variant of dvorak-bépo (because my
keyboard, a kinesis, is *not* practical to use with standard bépo), I
hope it's going to be easy to set up and modify such a setting,
because you are not going to give an exhaustive list of all layouts…
Ben Fritz
On Feb 22, 8:55 pm, Adrien "Pied" Piérard <axiopl...@gmail.com> wrote:
> Hi.
>
> 2011/2/23 Stephen Lee <stephenletter...@gmail.com>:
> > Therefore I propose letting users choose their preferred keyboard
> > layouts instead of forcing any specific one to them.
> > In your case it seems you are talking about QWERTY and QWERTZ (from
> > wiki), so the following would be keyboard layouts in the new directory
> > called "Keyboard"
>
> > (General keyboard layouts)
> > QWERTY.vim
> > ...
>
> Hum, as a user of a home-made variant of dvorak-bépo (because my
> keyboard, a kinesis, is *not* practical to use with standard bépo), I
> hope it's going to be easy to set up and modify such a setting,
> because you are not going to give an exhaustive list of all layouts…
>
layouts. The proposal at hand is to replace Vim's input mechanism so
that ANY key, modified or otherwise, coming from ANY keyboard, will
work with Vim as a user would expect. Nobody is talking about forcing
specific keyboard layouts.
Benjamin R. Haskell
>> Graywh wrote:
>>
>>> Please fix vim's input queue mechanism with key info structures a.la
>>> libtermkey so that LeoNerd can fix terminal input and Gvim and
>>> everyone will be happy. Thanks.
>>
>> [...]
>
> I am talking about ripping out the byte-queue input system and replacing
> it with a structured keypress queue. [...]
>
> Having done this, I can then quite easily send you a patch to use
> libtermkey [...]
>
> Tab, Ctrl-I and Ctrl-Shift-I even over terminal from a
> properly-configured xterm. [...]
So, from other responses, it seems (to me) that the problem isn't
necessarily in the input queue, per se. "Special" keys are already
represented in some internal format (perhaps not-well-documented, since
it's internal, but still represented). (example of undocumented nature
is the K_IGNORE special key trick that stopped working when its keycode
was either changed or eliminated [1]).
Taking that as a given, I'm going to ignore that aspect of this issue.
My question is about libtermkey. I was curious enough to download it,
unpack it, `make` it, and run ./demo. And, surprisingly, given that it
seems to be the favored example, pressing <Tab>, then <Ctrl>+<i>, then
<Ctrl>+<Shift>+<i> gave me the output sequence:
<Tab>
<Tab>
<Tab>
So, I'm wondering: what does libtermkey bring to the table?
I tested under both rxvt-unicode (my preferred terminal emulator) and
uxterm (which, AFAIK, is a "properly-configured" xterm). If
"properly-configured" actually means "specially-configured", how would
using libtermkey be anything less than a huge PITA when working on
multiple systems that aren't so configured? Or is this a
chicken-and-egg problem? (Nothing is configured to send anything other
than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)
--
Best,
Ben
[1] http://groups.google.com/group/vim_use/browse_thread/thread/8c535e5cf2b35f63/319b3113716e52d5
Ben Schmidt
don't!) that is the convention on this list. I must admit, though, that
particularly after somebody else has already bottom-posted, top-posting
makes things a big mess. In future, please write your replies under
short, relevant extracts of the mail you are replying to.
> Therefore I propose letting users choose their preferred keyboard
> layouts instead of forcing any specific one to them.
Sorry, but this is not what we are discussing. Vim already has this
functionality. As you discovered, it can work through what it calls
'keymaps' (:help keymap). It also works with a number of different input
methods on different platforms, particularly when using Gvim.
> Maybe a step forward to change the layout on-the-fly by the following
> command (when changing keyboard setting in X window system):
>
> :set keyboardlayout=...
And there is even better than this already: Vim's 'iminsert' option
(:help iminsert). Of course, you can write mappings (as I have done) to
change the 'keymap' option between values, too.
> and there would be 4 files:
>
> QWERTYsequence.vim
> QWERTZsequence.vim
>
> QWERTYstructured.vim
> QWERTZstructured.vim
You are miles ahead of us, Stephen! It is indeed possible that this kind
of thing could evolve. In fact, the functionality for this part of it is
already there. What we need, though, is a way to get the structured
kepress information into vim in the first place, so that keymaps (and
more importantly, mappings) can refer to them.
Mappings are more important, because what we are discussing isn't really
about text input, but about shortcut keys. At the moment, Vim sometimes
can't tell the difference between Tab and Control-I (and a lot of other
things). And because Vim sometimes has no way of telling the difference,
at the moment, it doesn't even bother to notice the difference when it
*can* tell. We want to change that.
After we have changed that, then your idea of having multiple keyboard
layouts, some for when these keys can be differentiated, and some for
when they cannot, may become useful. I hope we won't need that idea,
though, but will be able to write one keymap file that works well in
both situations.
But that is for later.
First we need to get all the keys recognised as properly as possible.
Ben.
Ben Schmidt
>> Yes. But what happens when you then edit that macro by putting the
>> register into a buffer, changing it, and yanking it again? This is not
>> uncommonly done. How should the registers be stored in .viminfo? How do
>> you write the input to the feedkeys function as a string in vimscript?
>> Etc.. These are the kinds of issues I was trying to raise.
>
> Well, I don't see what the problem is? Just do it like today, maybe
> with some expansions e.g "<C-A><S-I><Tab>somechars<U-234>". I mean
> this is simple key-as-text representation and then text-representation-
> to-struct creation no?
>
> For example, the user records a macro into register q that is then
> displayed as "ihello<Tab>worlld<Esc><Left><Left>x". He then edits the
> macro in a buffer, yanks it into register q, and when @q is executed
> all it does is interpret the string?
OK, so you seem to be proposing changing the way the q and @ commands
work, so that instead of exposing internal key representation, they
record and play back <> notation. They would also have to represent < as
<lt> of course. The feedkeys() function would also need to be similarly
changed. It would be hard to know what to do about \<xxx> escapes in
strings (:help expr-string)--whether to make it a no-op or do the old
thing, or something else.
So this would be a backwards-incompatible change. It would not just
affect places where special keys are currently represented in internal
notation, but everywhere macros are used. Now there are currently
probably quite a lot of places where internal keys are represented, and
changing the internal representation as I suggest, may affect them; but
if the current representation is sparse enough, perhaps it can be done
in a backwards compatible way. There may be quite little impact, though,
as wherever possible, plugin authors use <> notation anyway; it is only
when they cannot do this that they resort to internal
representation--and even then, they often use \<xxx> escapes when they
can. There is no way your suggestion could be backwards compatible, I
don't think, and it could potentially affect a lot more macros (e.g. all
macros that insert xml and html, so use < and > a lot). It could break a
lot of plugins as well as home-grown code.
Now, that doesn't rule it out, and I think it is probably a way to solve
the problem. But I think it would need more careful thought than the
alternative option of changing the internal representation while keeping
it byte-stream based. I also think it would be much more work, because
it requires not just macro recording, but every problem involving keys
to be separately solved from scratch, and every one would probably be
backwards incompatible. And I think Vim and Bram have a history of
maintaining backwards compatibility (we still have :set compatible!) so
it may be that a more compatible change is preferred.
> Maybe I'm oversimplifying it but it'd help if you were more specific
> about the "issues" the new design have.
This would be easier if we actually had a solid 'new design'. We don't.
We just have a bunch of rough ideas. One of those ideas had a drafted
structure and suggestion that it be used only in the input queue, where
the memory inefficiency would not be a concern. I have pointed out that
there are many other problems to solve: right hand sides of mappings,
feedkeys(), registers for macro recordings (which indirectly affect
buffers, as registers must be able to be put and yanked), representation
in strings, and more (I just thought of the :normal command, and I'm
sure there is still more...). I pointed out that if you use the
structure to solve all these problems, memory efficiency is most
certainly a concern.
So I think I have been quite specific. I think I've been specific enough
for the purpose of this discussion.
> To me it looks like we just need to agree on a new set of conventions
> by describing how the new system would work for all the current (and
> future) macros/registers/ whatever scenarios.
Yep, this seems to sum up your approach nicely, which is basically
design and solve every key-related problem from scratch. The most
specific thing you have done is suggested a different way to using the
structure to solve at least the macro recording problem.
My response, to both that specific proposal and that general approach is
that I don't think it is a good way, because it is massively
backwards-incompatible and a lot of work.
> I suggest maybe we make some kind of "new design" draft which would
> answer all the issues raised (memory size/1970 support/macros
> concerns) ?
I suggested earlier a rough idea in response to the structure proposal
of extending/changing Vim's existing mechanism. That was also
springboarding off Bram's comment that he thought this could be done
using the existing mechanism. The existing mechanism is there for all to
see, so is already a much more well-developed design than any of the
alternatives. My rough idea of how it could be extended is in a previous
post, too. I think it would work, and be more efficient, less work, and
more compatible than the other directions we have explored.
But it's not zero work, and I'm not at the point where I can volunteer
to do work on it. Even putting together a careful detailed proposal is
probably not a contribution I can commit. I'm more than willing to
discuss and brainstorm ideas, though, and comment on any serious
proposal someone wants to make.
In the end, it's up to Bram as maintainer, and whoever's going to put
the (large amount of) time into implementing this, what goes ahead,
though.
But as a community member, I'd prefer it if it affected my code and
plugins as little as possible, and I consider it essential that Vim
doesn't become significantly less efficient because of this. And if it
were me considering an implementation, I'd certainly be looking for the
option that was the least work! YMMV.
Ben.
Paul LeoNerd Evans
> Taking that as a given, I'm going to ignore that aspect of this
> issue. My question is about libtermkey. I was curious enough to
> download it, unpack it, `make` it, and run ./demo. And,
> surprisingly, given that it seems to be the favored example,
> pressing <Tab>, then <Ctrl>+<i>, then <Ctrl>+<Shift>+<i> gave me the
> output sequence:
>
> <Tab>
> <Tab>
> <Tab>
>
> So, I'm wondering: what does libtermkey bring to the table?
It parses CSIs, the way that xterm et.al. send modified keys.
Start an old vim. Go into insert mode. Press Ctrl-Left.
xterm sends CSI 1;5D
Without a CSI parser, using only a 1970s-style terminfo-driven prefix
hunting algorithm, there is no terminfo key sequence matching this. So
vim gives up. It then sees Escape [ 1 ; 5 D. The upshot is that vim
leaves insert mode (Escape), does something unimportant with [ 1 ; then
deletes 5 lines of text (5D).
With a CSI parser such as libtermkey, this is recognised as a real CSI D
call, with the parameters 1 and 5. This is reported to the application
as Left (CSI D), with the Ctrl modifier (2nd param being 5).
The primary thing that libtermkey brings, is not blowing up and deleting
5 lines of your text just because xterm happened to send a valid CSI
sequence that isn't in the terminfo.
> I tested under both rxvt-unicode (my preferred terminal emulator)
rxvt uses its own, totally-incompatible encoding scheme for modified
keypresses. This scheme is different to any other terminal that has such
abilities (all the others follow xterm's CSI scheme). It is also
incompatible with ECMA-48 (whereas xterm extends it), inextensible
beyond Ctrl and Shift, and arbitrary (happening to pick three ASCII
characters seemingly at random, to represent the Shift, Ctrl and
Ctrl+Shift states).
> and uxterm (which, AFAIK, is a "properly-configured" xterm). If
> "properly-configured" actually means "specially-configured",
Specifically-configured means
modifyOtherKeys: 2
Except that on its own now breaks most other applications, because xterm
is too keen to use the CSI encoding scheme on "normal" keypresses, such
as Ctrl-D. xterm doesn't yet send the correct and useful sequences here,
but it is not difficult to fix its code to do so - I have been in
occasional contact with Thomas Dickey on this issue; I need to poke him
again sometime soon to check he's properly looking into it.
What is needed is a hybrid scheme, whereby the simplest key encoding is
used in a nicer way; where Ctrl-D would send the single 0x04 byte (so
kernel's pty driver can recognise "here's an EOF"), but Ctrl-Shift-D
sends CSI 68;5 u to encode it correctly.
> how
> would using libtermkey be anything less than a huge PITA when
> working on multiple systems that aren't so configured? Or is this a
> chicken-and-egg problem? (Nothing is configured to send anything
> other than 70's-style "<C-i>" == "<Tab>", so nothing ever will be.)
Rightnow, xterm will send the same byte for Tab, Ctrl-I or Ctrl-Shift-I.
Setting modifyOtherKeys to 2 will send real keypresses that it can tell,
but now everything else cannot.
Ultimately, yes. It is chicken and egg, in order for -everything- to
work in a happy shiney way. But even without terminal changes,
applications can benefit from using libtermkey now.
So to answer your question on "how would using libtermkey be anything
less than a huge PITA" - it isn't. It already recognises, via its
terminfo-driven parser, every keypress that terminfo/curses already
recognises. It cannot be worse. It is strictly better than, because of
the full CSI parser I mentioned above. Even if it cannot pass on that
information to its containing program, it can at least parse it. If
libtermkey was used by some application that has no concept of modifier
keys, then modifiers would just be ignored; i.e. that pressing Ctrl-Left
would simply be read as a Left k