I/O-interception layer

[Janis Papanagnou]

(First time since decades here. Had to clear that funny arabian spam first
to see whether this newsgroup is still active.)

This is a question related to roguelikes (specifically Nethack and Slashem)
on the _Unix_ platforms.

Background: I am playing NH-3.4.3 based Slashem and have no colors (for HPs,
messages, or menus) available.

Instead of fiddling a patch into the core code(s) I had the idea to create
an interception layer for the I/O to do the coloring so that this program
could be used more universally and independent of the actual roguelike (or
usable even for completely unrelated character oriented programs).
Basically I thought of a pty that would read a user definable config file
with patterns and associated colors and spawn the (Slashem-) process. I/O
would be connected appropriately from/to terminal and the child process.
Input would be simply passed through while output from the roguelike would
be compared against the patterns and colored as defined by the config file.

But a quick test (without pattern match, just transparent I/O) did not lead
to the desired effect, though. The roguelike typical one-key command input
required hitting the Enter-key reglarly, so I seem to have missed something.
(Seems like problems with char-by-char mode, that somehow a line-oriented
mode became active.)

Without going into details, my question would be whether someone has done
something similar (that works) or knows what has to be considered to make
such a function work in principle.

Janis

[oot...@hot.ee]

On Sunday, 5 January 2020 15:13:11 UTC+2, Janis Papanagnou wrote:
> (First time since decades here. Had to clear that funny arabian spam first
> to see whether this newsgroup is still active.)
>
> This is a question related to roguelikes (specifically Nethack and Slashem)
> on the _Unix_ platforms.
>
> Background: I am playing NH-3.4.3 based Slashem and have no colors (for HPs,
> messages, or menus) available.
>
> Instead of fiddling a patch into the core code(s) I had the idea to create
> an interception layer for the I/O to do the coloring so that this program
> could be used more universally and independent of the actual roguelike (or
> usable even for completely unrelated character oriented programs).
> Basically I thought of a pty that would read a user definable config file
> with patterns and associated colors and spawn the (Slashem-) process. I/O
> would be connected appropriately from/to terminal and the child process.
> Input would be simply passed through while output from the roguelike would
> be compared against the patterns and colored as defined by the config file.
>
> But a quick test (without pattern match, just transparent I/O) did not lead
> to the desired effect, though. The roguelike typical one-key command input
> required hitting the Enter-key reglarly, so I seem to have missed something.
> (Seems like problems with char-by-char mode, that somehow a line-oriented
> mode became active.)

Sounds like problem with I/O buffering so your input is for whatever reason
waiting in some buffer and not forwarded. Without precisely knowing what
you did it is just useless guesses and suggestions to debug.

> Without going into details, my question would be whether someone has done
> something similar (that works) or knows what has to be considered to make
> such a function work in principle.

Over 20 years ago when online multiplayer roguelikes (called MUDs) were
popular people wrote lot of such filters, clients and also bots to
terminals. But those MUDs had also line-entry-oriented interfaces not
key-press-oriented interfaces. Keyboard entry is not as trivial to
serialize as might appear at first glance.

[Janis Papanagnou]

Thanks for the confirmation. I am aware that there's no use guessing without
me posting the gory details of my pty implementation; I wanted to avoid that
by restricting myself first to the subsequent [assumed] simpler question...

>> Without going into details, my question would be whether someone has done
>> something similar (that works) or knows what has to be considered to make
>> such a function work in principle.
>
> Over 20 years ago when online multiplayer roguelikes (called MUDs) were
> popular people wrote lot of such filters, clients and also bots to
> terminals.

Yeah, that's what I assumed, so my hope for some hints in that direction.

> But those MUDs had also line-entry-oriented interfaces not
> key-press-oriented interfaces.

(I didn't knew that MUDs were line-oriented.) My impression was that bots
have been used, e.g. for pudding farming, "nethack-o-matic" bots, or other
monotoneous tasks, observable in telnet based Nethack sessions (like NAO).

> Keyboard entry is not as trivial to
> serialize as might appear at first glance.

(Two decades ago I also wrote line-buffered ptys that worked quite well.)

Something speciality might be necessary for the "key-press-oriented input",
though my hope would have been that with the standard Unix system I/O it
should be feasible. - Isn't Nethack's I/O also based on select/read/write
system calls?

Janis

[oot...@hot.ee]

The MUDs certainly were line oriented so people had to type "open door<CR>"
or "o d<CR>" for to open a door.
There is some demo about hooking Unix key presses
/usr/lib/ncurses/examples/demo_altkeys
that must work when there is SSH session or actual real terminal
(not X11 or console).

>
> > Keyboard entry is not as trivial to
> > serialize as might appear at first glance.
>
> (Two decades ago I also wrote line-buffered ptys that worked quite well.)
>
> Something speciality might be necessary for the "key-press-oriented input",
> though my hope would have been that with the standard Unix system I/O it
> should be feasible. - Isn't Nethack's I/O also based on select/read/write
> system calls?

I am unfamiliar with Nethack code but something that has been around
for so long can be configured to use all the standard ways of I/O
almost certainly.

[Elijah Stone]

You are looking for interhack.

https://taeb.github.io/interhack/

[Janis Papanagnou]

On 07.01.2020 04:06, Elijah Stone wrote:
> You are looking for interhack.
>
> https://taeb.github.io/interhack/

Heh, I was sure that someone did something like that for coloring
but this is even a hole system of useful stuff. Thanks!

Just one (important) point. I read in "3. How does it work?" that
"It's just a very specialized telnet client." (to access servers
like NAO). - I'd needed some simple process usable for local play.
Has someone experiences how (or if) interhack can be used that way?

Janis

[Janis Papanagnou]

I skimmed a bit more through the docs and perl code of Interhack;
it seems to be not only access-channel specific but also tailored
for Nethack. - Hmm.. - While it seems perfect as frontend if using
Nethack through a server like NAO it seems inpractical for other
software or other access methods (unless I missed something).

Still looking for program code that can simply be used as front-end
process to intercept I/O to (arbitrary) text-oriented programs.


To provide some detail about my own tries, I used cfmakeraw() to
obtain the character-wise I/O (which somehow did not work). It is
described as

cfmakeraw() sets the terminal to something like the
"raw" mode of the old Version 7 terminal driver: input
is available character by character, echoing is dis‐
abled, and all special processing of terminal input and
output characters is disabled. [...]

so it at least looks like it should do what I wanted.

Janis

[ais523]

Janis Papanagnou wrote:
> To provide some detail about my own tries, I used cfmakeraw() to
> obtain the character-wise I/O (which somehow did not work). It is
> described as
>
> cfmakeraw() sets the terminal to something like the
> "raw" mode of the old Version 7 terminal driver: input
> is available character by character, echoing is dis‐
> abled, and all special processing of terminal input and
> output characters is disabled. [...]
>
> so it at least looks like it should do what I wanted.

There are two potential layers of buffering involved. One is buffering
at the terminal, which can be disabled in a number of ways. IIRC,
cfmakeraw doesn't actually change the terminal settings, it simply
modifies an existing set of terminal settings to represent a raw-mode
version of those settings and leaves it up to you to apply them (with a
system call like tcsetattr). Note that it's probably a good idea to
remember the old terminal settings too (available via tcgetattr),
because if you don't restore the original terminal settings as your
program exits, the terminal will act weirdly after your program exits.

The other potential layer of buffering is stdio buffering, which is
involved in calls like putchar/getchar and printf/scanf. You can avoid
this either by exclusively using the kernel level system calls for
reading and writing (read/write), or by using setvbuf to entirely
disable stdio buffering on the file descriptors in question.

As a reference for this sort of terminal configuration, here's the code
that NetHack4 currently uses (omitting, for copyright-related reasons,
a small tweak required to make it work on BSD) to set up a terminal
(you'll need to adapt this for your own purposes, e.g. "fileno(ifile)"
and "fileno(ofile)" are likely to become hardcoded numbers, and "ti"
and "ti_orig" will need to be declared somewhere, with "ti_orig"
restored at the end of the program):

{{{
/* Set up the terminal control flags. Use whichever of ifile and ofile
happens to be a terminal. */
if (tcgetattr(fileno(ifile), &ti_orig) == -1)
tcgetattr(fileno(ofile), &ti_orig);

/*
* Terminal control flags to turn off:
*
* Input-related:
* ISTRIP bit 8 stripping (incompatible with Unicode)
* INPCK parity checking (ditto)
* PARENB more parity checking (ditto)
* IGNCR ignore CR (means there's a key we can't read)
* INLCR translate NL to CR (ditto)
* ICRNL translate CR to NL (ditto)
* IXON insert XON/XOFF into the stream (confuses the client)
* IXOFF respect XON/XOFF (causes the process to lock up on ^S)
* ICANON read a line at a time (means we can't read keys)
* ECHO echo typed characters (confuses our view of the screen)
* We also set ISIG based on raw_isig, and VMIN to 1, VTIME to 0,
* which gives select() the correct semantics; and we turn on CS8,
* to be able to read Unicode (although on Unicode systems it seems
* to be impossible to turn it off anyway).
*
* Output-related:
* OPOST general output postprocessing (portability nightmare)
* OCRNL map CR to NL (we need to stay on the same line with CR)
* ONLRET delete CR (we want to be able to use CR)
* OFILL delay using padding (only works on physical terminals)
*/
if (tcgetattr(fileno(ifile), &ti) == -1)
tcgetattr(fileno(ofile), &ti);
ti.c_iflag &= ~(ISTRIP | INPCK | IGNCR | INLCR | ICRNL | IXON | IXOFF);
ti.c_cflag &= ~PARENB;
ti.c_cflag |= CS8;
ti.c_lflag &= ~(ISIG | ICANON | ECHO);
ti.c_cc[VMIN] = 1;
ti.c_cc[VTIME] = 0;
if (!raw_isig)
ti.c_lflag |= ISIG;
ti.c_oflag &= ~(OPOST | OCRNL | ONLRET | OFILL);
if (tcsetattr(fileno(ifile), TCSADRAIN, &ti) == -1)
tcsetattr(fileno(ofile), TCSADRAIN, &ti);
}}}

--
ais523

[Janis Papanagnou]

On 07.01.2020 16:00, ais523 wrote:
> Janis Papanagnou wrote:
>> To provide some detail about my own tries, I used cfmakeraw() to
>> obtain the character-wise I/O (which somehow did not work). It is
>> described as
>>
>> cfmakeraw() sets the terminal to something like the
>> "raw" mode of the old Version 7 terminal driver: input
>> is available character by character, echoing is dis‐
>> abled, and all special processing of terminal input and
>> output characters is disabled. [...]
>>
>> so it at least looks like it should do what I wanted.
>
> There are two potential layers of buffering involved. One is buffering
> at the terminal, which can be disabled in a number of ways. IIRC,
> cfmakeraw doesn't actually change the terminal settings, it simply
> modifies an existing set of terminal settings to represent a raw-mode
> version of those settings and leaves it up to you to apply them (with a
> system call like tcsetattr).

Yes, using that function to set the data is exactly what I did.

> Note that it's probably a good idea to
> remember the old terminal settings too (available via tcgetattr),
> because if you don't restore the original terminal settings as your
> program exits, the terminal will act weirdly after your program exits.

Exactly. (I deliberately omitted all the surrounding code for the
post, because I thought it might not be appropriate here.)

>
> The other potential layer of buffering is stdio buffering, which is
> involved in calls like putchar/getchar and printf/scanf. You can avoid
> this either by exclusively using the kernel level system calls for
> reading and writing (read/write), or by using setvbuf to entirely
> disable stdio buffering on the file descriptors in question.

Yes, for I/O I use the read/write system calls only (in addition to
other system calls like open, close, fork, execvp, select, dup, ioctl).

I haven't considered using setvbuf because in the interception process
layer there's no stdio.h functions that I use. I thought I would be on
the safer side if I'll entirely stay on OS's system call level for the
intended purpose. (Okay, not entirely true; I used the C lib for the
program's error messages and memory allocation but not for the process
handling or process I/O which is in the focus here.)

>
> As a reference for this sort of terminal configuration, here's the code
> that NetHack4 currently uses (omitting, for copyright-related reasons,
> a small tweak required to make it work on BSD) to set up a terminal
> (you'll need to adapt this for your own purposes, e.g. "fileno(ifile)"
> and "fileno(ofile)" are likely to become hardcoded numbers, and "ti"
> and "ti_orig" will need to be declared somewhere, with "ti_orig"
> restored at the end of the program):
>
> {{{

> [snip code
> }}}
>

I'll have a look into it and try it in my environment. Thanks!

Janis

[Elijah Stone]

> Just one (important) point. I read in "3. How does it work?" that
> "It's just a very specialized telnet client." (to access servers
> like NAO). - I'd needed some simple process usable for local play.
> Has someone experiences how (or if) interhack can be used that way?

Per the FAQ (https://taeb.github.io/interhack/FAQ.html#devnull), if you
set the server address to 'ih_server', it'll dump you into a shell
instead of telnetting somewhere.

-E

[Nomen Nescio]

Janis Papanagnou <janis_pa...@hotmail.com> writes:

> (First time since decades here. Had to clear that funny arabian spam first
> to see whether this newsgroup is still active.)

Well, is it?

[ais523]

I'm still reading it. I just haven't had anything to say, and apparently
neither has anyone else.

--
ais523

[songbird]

well i do see messages here at times.

i once in a while poke at rogomatic/rogue
and try to figure out the bugs that i find.

unfortunately right now is a busy season
outside so i don't get much time where i can
concentrate and not be interrupted. i'm not
as smart as i used to be. :)

and this old code has some sphaghetti
sections and unions that give me a
headache...

if i turn the union into struct instead
at least i can get a more coherent picture of
what is going on, but it is slow debugging it
all right now. maybe i'll have it figured out
by next spring at the rate i'm going. arg!


songbird

[Janis Papanagnou]

On 20.06.2020 13:04, songbird wrote:
> [...]

>
> and this old code has some sphaghetti
> sections and unions that give me a
> headache...
>
> if i turn the union into struct instead
> at least i can get a more coherent picture of
> what is going on, but it is slow debugging it
> all right now. maybe i'll have it figured out
> by next spring at the rate i'm going. arg!

The application of 'union' that I've seen where
for two purposes. One was actually a low-level
hack where you store there one type of data and
access it by another type definition. The other
was to save memory; distinguished by some fixed
(always present) tag attribute the contents of
the associated data was determined. I seem to
recall Nethack code that had some properties
(of objects, or monsters, or both, don't recall)
depending on the actual type. So, say, if you
have 'union' data for a spellbook you could save
there the amount of time you already spent to
read it, or in case of a tin there could be the
type of contents stored. (Just a made up example
for illustration purpose, not sure where Nethack
actually uses it.)

Janis

[songbird]

Janis Papanagnou wrote:
...

> The application of 'union' that I've seen where
> for two purposes. One was actually a low-level
> hack where you store there one type of data and
> access it by another type definition. The other
> was to save memory; distinguished by some fixed
> (always present) tag attribute the contents of
> the associated data was determined. I seem to
> recall Nethack code that had some properties
> (of objects, or monsters, or both, don't recall)
> depending on the actual type. So, say, if you
> have 'union' data for a spellbook you could save
> there the amount of time you already spent to
> read it, or in case of a tin there could be the
> type of contents stored. (Just a made up example
> for illustration purpose, not sure where Nethack
> actually uses it.)

rogue uses it as either a representation of the
player or a monster or an item. each level has a
list of monsters on that level and items on that
level. the player has a list of items for their
pack.

the union isn't saving all that much memory but
back in the 80s it must have been seen as something
important because they did it. with hundreds of
people playing at a time it could have made a
difference.

i just hate that i'm rusty in looking at this
code i used to know it a lot better than i do now.
i've already debugged this particular chunk of code
once before and i'm annoyed at myself that i didn't
notice it still having a potential issue.

i have set up a testing framework where i can
use a specific seed for the random number generator
so that i can repeat runs of the code - that way
when i do find a bug i have a chance of figuring
out what is going on.


songbird

[David Damerell]

Quoting Janis Papanagnou <janis_pa...@hotmail.com>:
>access it by another type definition. The other
>was to save memory; distinguished by some fixed
>(always present) tag attribute the contents of
>the associated data was determined. I seem to
>recall Nethack code that had some properties
>(of objects, or monsters, or both, don't recall)
>depending on the actual type.

NetHack uses it all over the place for this purpose; it is a fairly
regular source of hilarious bugs.

Is there a reason
you're writing
very narrow posts?
--
David Damerell <dame...@chiark.greenend.org.uk> flcl?
Today is Oneiros, Presuary.
Tomorrow will be Mania, Presuary.

[songbird]

David Damerell wrote:
...re unions...

> NetHack uses it all over the place for this purpose; it is a fairly
> regular source of hilarious bugs.
>
> Is there a reason
> you're writing
> very narrow posts?

probably because i am. :)


songbird

[Janis Papanagnou]

Exactly, I oriented on songbird's line width. Curious; do you have any
objections about that?

IMO there's also nothing wrong with narrow posts (as opposed to wide
posts). (Though your exaggeration above I consider to be ridiculous.)
There had been advocates in the past that pointed to the fact that
narrow posts are better readable, and they supported that view with
reference to (printed) magazines and newspapers. Besides such results
from professional text presentation it's partly also a matter of taste
or just accustoming, I suppose.

For longer lines one should (client-size) use serif-fonts for better
readability (though nowadays it seems to be in vogue to use san serif
and proportional fonts with arbitrary line length). Just by the way.

On Usenet I use (for my own posts) typically around 70-78 characters
per line; just because that (old terminal width based) value had been
convention. (I similarly start using a width of 80 for coding programs,
but that's hard if you code in Java projects as I've noticed in the
past.) And for responses to posts with unlimited line lengths I often
just reformat the original text (with two key-strokes) before putting
my answer below.

To each his own.

Janis

> songbird
>

[songbird]

Janis Papanagnou wrote:
...

> On Usenet I use (for my own posts) typically around 70-78 characters
> per line; just because that (old terminal width based) value had been
> convention. (I similarly start using a width of 80 for coding programs,
> but that's hard if you code in Java projects as I've noticed in the
> past.) And for responses to posts with unlimited line lengths I often
> just reformat the original text (with two key-strokes) before putting
> my answer below.
>
> To each his own.

yes!

the old convention was 70-72 leaving a bit of room for others
to quote if desired. i just out of habit use a hard return in
my posts as i hate to have to scroll right to read a post with
unbroken lines.

since i don't run my posts through any kind of automatic
formatter to make them uniform or consistent it may vary
from post to post where i wrap. :)


songbird

[Auric__]

songbird wrote:

> since i don't run my posts through any kind of automatic
> formatter to make them uniform or consistent it may vary
> from post to post where i wrap. :)

I don't know much about slrn, but I wrote an automatic poster some years ago
(never released to avoid it becoming a spamming tool) that doesn't handle
wrapping (or much of anything else), so I wrote my posts in an external
editor which took care of it (and also things like spell check) and then
copied to my poster.

--
They had their chance and they chose to betray you.

[Nomen Nescio]

songbird <song...@anthive.com> writes:

> i hate to have to scroll right to read a post with
> unbroken lines.

You don't have to, slrn is able to wrap messages.

[David Damerell]

Quoting songbird <song...@anthive.com>:
> the old convention was 70-72 leaving a bit of room for others
>to quote if desired. i just out of habit use a hard return in
>my posts as i hate to have to scroll right to read a post with
>unbroken lines.

I think I would describe that as the current correct convention. Only
Google Groups people (usually replying to posts from 1995) don't linewrap.
:-)
--
David Damerell <dame...@chiark.greenend.org.uk>
And now, a seemingly inexplicable shot of a passing train.
Today is Aponoia, Presuary.
Tomorrow will be Epithumia, Presuary - a weekend.

[songbird]

funny that i'd never looked into it further, but
now that i have i've adjusted my .slrnrc file to do
just that. thanks!


songbird