bug#13471: 24.3.50; S-up does not work in xterm: <select> is undefined

[Sven Joachim]

Run "emacs -Q -nw" in an xterm (or some other modern terminal emulator
that sets $TERM to xterm) and press S-up. Rather than moving the cursor
and selecting text, Emacs beeps and informs you: <select> is undefined.

This seems to have been discussed a few times, e.g. in the threads
starting at

http://lists.gnu.org/archive/html/help-gnu-emacs/2011-05/msg00174.html
http://lists.gnu.org/archive/html/emacs-devel/2012-10/msg00267.html

The latter thread mentions in

http://lists.gnu.org/archive/html/emacs-devel/2012-10/msg00290.html

that the following change

--8<---------------cut here---------------start------------->8---
=== modified file 'src/term.c'
--- src/term.c 2013-01-11 02:40:58 +0000
+++ src/term.c 2013-01-17 08:49:20 +0000
@@ -1415,7 +1415,6 @@ term_get_fkeys_1 (void)
CONDITIONAL_REASSIGN ("F1", "k<", "f11");
CONDITIONAL_REASSIGN ("F2", "k>", "f12");
CONDITIONAL_REASSIGN ("%1", "kq", "help");
- CONDITIONAL_REASSIGN ("*6", "kU", "select");
#undef CONDITIONAL_REASSIGN
}
--8<---------------cut here---------------end--------------->8---

would fix that; I can confirm that, but don't know if it might break
anything.


In GNU Emacs 24.3.50.4 (i486-pc-linux-gnu, GTK+ Version 3.4.2)
of 2013-01-17 on turtle
Bzr revision: 111541 dman...@yandex.ru-20130117062940-mm7pgf1jxkbx1dsq
System Description: Debian GNU/Linux

Configured using:
`configure --build=i486-pc-linux-gnu --without-gconf
build_alias=i486-pc-linux-gnu --no-create --no-recursion'

Important settings:
value of $LC_COLLATE: C
value of $LANG: de_DE.UTF-8
locale-coding-system: utf-8-unix
default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
tooltip-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
line-number-mode: t
transient-mark-mode: t

Recent input:
ESC [ > 0 ; 2 7 8 ; 0 c ESC ] 1 1 ; r g b : f f f f
/ f f f f / f f f f ESC \ ESC [ 1 ; 2 A ESC x r e p
o r t - e m a c s - b u g RET

Recent messages:
("src/emacs")
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date tooltip ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

[dic...@his.com]

On Thursday, January 17, 2013 3:53:15 AM UTC-5, Sven Joachim wrote:
> --8<---------------cut here---------------start------------->8---
> === modified file 'src/term.c'
> --- src/term.c 2013-01-11 02:40:58 +0000
> +++ src/term.c 2013-01-17 08:49:20 +0000
> @@ -1415,7 +1415,6 @@ term_get_fkeys_1 (void)
> CONDITIONAL_REASSIGN ("F1", "k<", "f11");
> CONDITIONAL_REASSIGN ("F2", "k>", "f12");
> CONDITIONAL_REASSIGN ("%1", "kq", "help");
> - CONDITIONAL_REASSIGN ("*6", "kU", "select");
>
> #undef CONDITIONAL_REASSIGN
>
> }

A follow message says that there is a comment

/* IBM has their own non-standard dialect of terminfo.
If the standard name isn't found, try the IBM name. */

which is interesting: the code fragment is dealing with termcap
names (the first parameter is conventional termcap names, the
second is not). It is not dealing with terminfo (there is a
difference...). AIX has been terminfo-only for a long time
(I don't have access to a machine old enough to provide an
example of termcap in use - that would have to be from the
early 1990s.

I would suggest researching the history of this fragment,
and probably discarding the entire chunk as obsolete,
e.g., if it referred to AIX 3 or 4.

[dic...@his.com]

On Thursday, January 17, 2013 3:53:15 AM UTC-5, Sven Joachim wrote:

> --8<---------------cut here---------------start------------->8---
> === modified file 'src/term.c'
> --- src/term.c 2013-01-11 02:40:58 +0000
> +++ src/term.c 2013-01-17 08:49:20 +0000
> @@ -1415,7 +1415,6 @@ term_get_fkeys_1 (void)
> CONDITIONAL_REASSIGN ("F1", "k<", "f11");
> CONDITIONAL_REASSIGN ("F2", "k>", "f12");
> CONDITIONAL_REASSIGN ("%1", "kq", "help");
> - CONDITIONAL_REASSIGN ("*6", "kU", "select");
>
> #undef CONDITIONAL_REASSIGN
>
> }

[Andreas Schwab]

dic...@his.com writes:

> which is interesting: the code fragment is dealing with termcap
> names (the first parameter is conventional termcap names, the
> second is not). It is not dealing with terminfo

It is using the termcap(3) interface of terminfo.

Andreas.

--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[Andreas Schwab]

dic...@his.com writes:

> which is interesting: the code fragment is dealing with termcap
> names (the first parameter is conventional termcap names, the
> second is not). It is not dealing with terminfo

[Thomas Dickey]

On Thu, Jan 17, 2013 at 10:55:28AM +0100, Andreas Schwab wrote:
> dic...@his.com writes:
>
> > which is interesting: the code fragment is dealing with termcap
> > names (the first parameter is conventional termcap names, the
> > second is not). It is not dealing with terminfo
>

> It is using the termcap(3) interface of terminfo.

That addresses 1/3 of my comment. Looking at AIX 5, there is no termcap
file, the manpage for terminfo does not mention the names used for the
second parameter, and "strings" on the curses library shows no plausible
tables containing those strings.

(the other 2/3 of course dealt with determining when the change was
added, whether it makes sense now - or even if it was a suitable change
at the time)

--
Thomas E. Dickey <dic...@invisible-island.net>
http://invisible-island.net
ftp://invisible-island.net

[Thomas Dickey]

On Thu, Jan 17, 2013 at 05:06:29AM -0500, Thomas Dickey wrote:
> On Thu, Jan 17, 2013 at 10:55:28AM +0100, Andreas Schwab wrote:
> > dic...@his.com writes:
> >

> > > which is interesting: the code fragment is dealing with termcap
> > > names (the first parameter is conventional termcap names, the
> > > second is not). It is not dealing with terminfo
> >

> > It is using the termcap(3) interface of terminfo.

AIX, like the other vendor Unix's since no later than 1996,
does not have a termcap(3) manpage either: it documents
termcap as a compatibility detail in the terminfo manpage.

[Thomas Dickey]

On Thu, Jan 17, 2013 at 05:06:29AM -0500, Thomas Dickey wrote:
> On Thu, Jan 17, 2013 at 10:55:28AM +0100, Andreas Schwab wrote:
> > dic...@his.com writes:
> >

> > > which is interesting: the code fragment is dealing with termcap
> > > names (the first parameter is conventional termcap names, the
> > > second is not). It is not dealing with terminfo
> >

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> The code fragment in discussion is IBM specific,
> not related to ncurses.

But related to terminfo. Emacs doesn't depend on ncurses.

[Andreas Schwab]

[Thomas Dickey]

On Thu, Jan 17, 2013 at 12:00:27PM +0100, Andreas Schwab wrote:
> dic...@his.com writes:
>

> > I would suggest researching the history of this fragment,
> > and probably discarding the entire chunk as obsolete,
> > e.g., if it referred to AIX 3 or 4.
>

> You can browse the history here:
>
> http://git.savannah.gnu.org/cgit/emacs.git/log/src/term.c?ofs=440
>
> This has the last change in that area:
>
> http://git.savannah.gnu.org/cgit/emacs.git/commit/src/term.c?id=419cae6214d82904eafcaf06ce6b7762576c32eb

given the dates shown there, that chunk (and others) are overdue for deleting.

[Thomas Dickey]

On Thu, Jan 17, 2013 at 11:28:22AM +0100, Andreas Schwab wrote:
> Thomas Dickey <dic...@his.com> writes:
>
> > Looking at AIX 5, there is no termcap file,
>

> Why do you need a termcap file?

You don't _need_ a termcap file, but for instance the presence of
a termcap file on Solaris was the cause of analogous user confusion
leading to incorrect patches.

> > the manpage for terminfo does not mention the names used for the
> > second parameter,
>

> Does it support tgetstr?

There's a manpage for tgetstr, which states that it is used for
applications which use the termcap file.

However, tgetstr is implemented in the curses library,
which does not use termcap.

(I did ports to AIX 3 and 4 a while back, and from that and notes
am reasonably aware of their issues - it's been quite a while since
someone defended AIX 4 as being current - perhaps ten years).

> > and "strings" on the curses library shows no plausible tables
> > containing those strings.
>

> The termcap translation table can be arbitrarily encoded. libncurses
> doesn't contain those literal strings either.

If it's not in the AIX curses library, it won't be recognized by that
implementation - it relies upon a table, not upon an external user-supplied
definition of any kind whatsoever.

ncurses uses a different method, not used by any of the vendor unixes.
Furthermore, there is no point in bringing extensions which could be
defined by a user for ncurses into a topic which deals with "IBM terminfo".

[Thomas Dickey]

On Thu, Jan 17, 2013 at 10:55:28AM +0100, Andreas Schwab wrote:
> dic...@his.com writes:
>

> > which is interesting: the code fragment is dealing with termcap
> > names (the first parameter is conventional termcap names, the
> > second is not). It is not dealing with terminfo
>

> It is using the termcap(3) interface of terminfo.

That addresses 1/3 of my comment. Looking at AIX 5, there is no termcap
file, the manpage for terminfo does not mention the names used for the
second parameter, and "strings" on the curses library shows no plausible
tables containing those strings.

(the other 2/3 of course dealt with determining when the change was
added, whether it makes sense now - or even if it was a suitable change
at the time)

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> Looking at AIX 5, there is no termcap file,

Why do you need a termcap file?

> the manpage for terminfo does not mention the names used for the
> second parameter,

Does it support tgetstr?

> and "strings" on the curses library shows no plausible tables
> containing those strings.

The termcap translation table can be arbitrarily encoded. libncurses
doesn't contain those literal strings either.

[Thomas Dickey]

On Thu, Jan 17, 2013 at 11:28:22AM +0100, Andreas Schwab wrote:

> Thomas Dickey <dic...@his.com> writes:
>
> > Looking at AIX 5, there is no termcap file,
>
> Why do you need a termcap file?

You don't _need_ a termcap file, but for instance the presence of
a termcap file on Solaris was the cause of analogous user confusion
leading to incorrect patches.

> > the manpage for terminfo does not mention the names used for the
> > second parameter,
>
> Does it support tgetstr?

There's a manpage for tgetstr, which states that it is used for
applications which use the termcap file.

However, tgetstr is implemented in the curses library,
which does not use termcap.

(I did ports to AIX 3 and 4 a while back, and from that and notes
am reasonably aware of their issues - it's been quite a while since
someone defended AIX 4 as being current - perhaps ten years).

> > and "strings" on the curses library shows no plausible tables
> > containing those strings.
>
> The termcap translation table can be arbitrarily encoded. libncurses
> doesn't contain those literal strings either.

If it's not in the AIX curses library, it won't be recognized by that
implementation - it relies upon a table, not upon an external user-supplied
definition of any kind whatsoever.

ncurses uses a different method, not used by any of the vendor unixes.
Furthermore, there is no point in bringing extensions which could be
defined by a user for ncurses into a topic which deals with "IBM terminfo".

[Andreas Schwab]

dic...@his.com writes:

> I would suggest researching the history of this fragment,
> and probably discarding the entire chunk as obsolete,
> e.g., if it referred to AIX 3 or 4.

You can browse the history here:

http://git.savannah.gnu.org/cgit/emacs.git/log/src/term.c?ofs=440

This has the last change in that area:

http://git.savannah.gnu.org/cgit/emacs.git/commit/src/term.c?id=419cae6214d82904eafcaf06ce6b7762576c32eb

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> Furthermore, there is no point in bringing extensions which could be
> defined by a user for ncurses

IIUC ncurses officially supports the termcap names (just not those that
IBM terminfo is using).

[Thomas Dickey]

On Thu, Jan 17, 2013 at 11:49:17AM +0100, Andreas Schwab wrote:
> Thomas Dickey <dic...@his.com> writes:
>
> > Furthermore, there is no point in bringing extensions which could be
> > defined by a user for ncurses
>
> IIUC ncurses officially supports the termcap names (just not those that
> IBM terminfo is using).

yes - but your remark is nonresponsive and irrelevant - repeating it
only dilutes your comments.

The code fragment in discussion is IBM specific,
not related to ncurses.

[Stefan Monnier]

Hi Thomas,

I'm not familiar enough to follow your argument and/or suggestion and
I'm especially unable to relate your email's body with its subject.
Could you try to help me understand?


Stefan

[Thomas Dickey]

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> Furthermore, there is no point in bringing extensions which could be
> defined by a user for ncurses

IIUC ncurses officially supports the termcap names (just not those that
IBM terminfo is using).

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> Looking at AIX 5, there is no termcap file,

Why do you need a termcap file?

> the manpage for terminfo does not mention the names used for the
> second parameter,

Does it support tgetstr?

> and "strings" on the curses library shows no plausible tables
> containing those strings.

The termcap translation table can be arbitrarily encoded. libncurses
doesn't contain those literal strings either.

[Thomas Dickey]

On Thu, Jan 17, 2013 at 12:37:19PM -0500, Stefan Monnier wrote:
> Hi Thomas,
>
> I'm not familiar enough to follow your argument and/or suggestion and
> I'm especially unable to relate your email's body with its subject.
> Could you try to help me understand?

There are several things to discuss:

+ this thread started in Debian #698299
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=698299

+ the original report cited a few urls, including a previous
thread for Emacs:

http://stackoverflow.com/questions/10871745/shift-up-arrow-doesnt-highlight-text-emacs-iterm2
http://lists.gnu.org/archive/html/help-gnu-emacs/2011-05/msg00211.html
http://forums.vandyke.com/showpost.php?p=26580&postcount=25

+ Sven commented that removing the line with "kU" fixed the
reported problem.

+ I commented that the mailing-list thread discussing this had
quoted a comment from the Emacs source code indicating that
the chunk of code had been written for a "non-standard dialect
of terminfo".

I'd read that thread before, noting (to myself) only that
the comment was incorrect since the mnemonics in quotes
are all termcap. The first parameter (F1, F2, %1) is
a conventional termcap name (there are no standards for
termcap - it owes its existing convention to documentation
in the terminfo manual), while the others were unfamiliar
to me.

You can see the names in terminfo(5).

+ however, as a bug report against ncurses I pay more attention.
The problem is to infer where the problem is, and possible
causes.

+ Emacs uses the termcap interface provided by ncurses, which
means that Emacs calls tgetstr, tgetnum, tgetflag to obtain
string, numeric, boolean values. The termcap interface
matches only the first 2 characters of capability names.

(The reason for not requiring that the names have a trailing
null, I seem to recall being for compatibility with termcap
libraries which do this, leading to applications that rely
upon the behavior).

+ this behavior was commented on in a different thread:
http://lists.gnu.org/archive/html/emacs-devel/2012-10/msg00012.html

+ tgetstr looks for each name, starting with the predefined
termcap names, and then the extended names - unless one calls
the ncurses function
use_extended_names(FALSE);

That's been the behavior since 1999. The current data for the
extended keys for xterm dates from April 2005.

+ uses for the above features (and quirks) are long established.
That leads to considering how the problem arises in Emacs,
and whether Emacs needs the problematic code.

+ The thread from October suggests that the problem is that Emacs
is calling tgetstr with "kU" as a parameter. That will return
data for kUP (an extended key). Schwab comments that kR is
the same as kri - though there's a pitfall there. Quoting from
terminfo(5):

key_sf kind kF scroll-forward key
key_sr kri kR scroll-backward key
key_sleft kLFT #4 shifted left-arrow
key
key_sright kRIT %i shifted right-arrow
key

You may notice that terminfo(5) does not define a shifted up or
down-arrow. (Whether kind and kri should fill that role is arguable
- I decided they would, and in October 2005 modified xterm-new to
provided that connection but have found no prior art to establish
that connection in usage by others).

I retained kUP because it fits a pattern of names for extended keys.

So... if Emacs looks for "kU" and gets "kUP" data, and then sees
that it is the same string as an existing key kR, then it could
conclude that the assignment for "kU" was ineffective and that
"select" is not defined.

+ Emacs could of course solve the problem by calling
use_extended_names(FALSE);

I recall making that suggestion to someone working on Emacs a
few years ago - that's apparently not been done.

+ Alternatively (as suggested in the threads mentioned), one could
remove the problematic call from Emacs. That could be done if
it serves no useful purpose. From discussion and examining the
source code, it appears that the "non-standard terminfo" cited
in the comment is unlikely to be in use by anyone some 18
years later - it dates from 1994-10-14. AIX 4 was released in
1994; the code was for AIX 4 (early) or AIX 3 (late).
In either case it's old. AIX 5 is more than ten years old.

The entire chunk relating to that comment could be removed.

There is other obsolete code in term.c, e.g., the Teleray code,
the Hazeltine code, not to mention Emacs insistence on using
termcap names on a terminfo system :-)

+ Schwab's comments this morning seemed to be focused on retaining
the code because (a) it was related to terminfo and (b) ncurses
uses terminfo. My viewpoint on it is that the code is useful
only if there is a system whose users rely upon it, that a tie
to ncurses is pointless. The data in ncurses exposes some code
which is no longer useful.

> >>>>> "dickey" == dickey <dic...@his.com> writes:
>

[Andreas Schwab]

Thomas Dickey <dic...@his.com> writes:

> + Schwab's comments this morning seemed to be focused on retaining
> the code

I did nothing like that.

[dic...@his.com]

On Friday, January 18, 2013 9:12:20 AM UTC-5, Andreas Schwab wrote:
> Thomas Dickey <dic...@his.com> writes:
>
> > + Schwab's comments this morning seemed to be focused on retaining
>
> > the code
> I did nothing like that.

Then you might suggest to Stefan that removing the code would be an improvement.

[dic...@his.com]

On Friday, January 18, 2013 9:12:20 AM UTC-5, Andreas Schwab wrote:

> Thomas Dickey <dic...@his.com> writes:
>
> > + Schwab's comments this morning seemed to be focused on retaining
>
> > the code
> I did nothing like that.

[Juri Linkov]

>> > + Schwab's comments this morning seemed to be focused on retaining
>> > the code
>>
>> I did nothing like that.
>
> Then you might suggest to Stefan that removing the code would be an improvement.

Seeing a large group of users suffering from this problem
http://www.google.com/search?q=emacs+%22select+is+undefined%22
suggests to remove the code in the emacs-24 release branch.

[Stefan Monnier]

> Seeing a large group of users suffering from this problem
> http://www.google.com/search?q=emacs+%22select+is+undefined%22
> suggests to remove the code in the emacs-24 release branch.

AFAIK this is not a regression, and it's far from an obviously-safe fix,
so I don't think it qualifies for emacs-24.


Stefan

[Thomas Dickey]

thanks for your comment.

I had already resolved yesterday what changes I would make to ncurses.
That involved some more research to document the issue properly.
And some code change, as needed. Since you are not in a hurry,
it will percolate along.

Incidentally, I checked my archives, and determined that the bogus code
in Emacs did not reflect AIX 4. Even AIX 3 is questionable (1992),
since that feature did not appear in AIX 3's termcap file. It may have
been a customization on someone's personal system, but did not appear
in the IBM product within the past 20 years.

[Juri Linkov]

Version: 24.4

> AFAIK this is not a regression, and it's far from an obviously-safe fix,
> so I don't think it qualifies for emacs-24.

So I removed the obsolete IBM code now, for the next release.