VT100 Terminal Emulator Library

[Rett Berg]

Hey y'all,

I wrote a VT100 terminal emulator library for lua[1] that I think is pretty nifty. It lets you write text, fg color and bg color to separate text grids and then draw it all at once -- as well as handling asyncronous sizing. It all uses the LAP protocol so all operations can be done in lua's coroutines.

Check out the README or demo[2] for details on use

Best,

Rett

[1]: https://github.com/civboot/civlua/tree/main/lib/vt100

[2]: https://github.com/civboot/civlua/blob/main/lib/vt100/demo.lua

      https://raw.githubusercontent.com/civboot/civlua/main/lib/vt100/demo.lua

[Sainan]

I don't understand why you act like Lua coroutines are such a big problem that need this massive special handling. It's as simple as this:

-- this may block a bit, allow the scheduler to do other stuff

if coroutine.isyieldable() then

    coroutine.yield()

end

You can still 'return' from your functions as normal, so it's a really unintrusive way to introduce cooperative multitasking.

-- Sainan

[Rett Berg]

I'm not sure I understand the tone of your message. Lua coroutines are not a "big problem", they are wonderful. LAP is precisely what you say: a mechanism to tell the executor you are waiting and what you are waiting for (sleep, file, run ASAP, or forget since something else will re-awaken it)

[Sainan]

Well, just the part I don't understand is this "LAP protocol" stuff, and why you need such a huge library for it. (I solved real problems in less lines of code.)

-- Sainan

[Sean Conner]

It was thus said that the Great 'Sainan' via lua-l once stated:

> Well, just the part I don't understand is this "LAP protocol" stuff, and
> why you need such a huge library for it. (I solved real problems in less
> lines of code.)

There was a thread about this starting in late April. The only link I was
able to find on the Lua list archive [1] is:

https://groups.google.com/g/lua-l/c/QWXul3NUY1M/m/JCw4ZMm9AwAJ

but that's the last message in the thread---the start is at the top of the
page.

I don't agree with the LAP protocol itself, but I'll worry about it only
*IF* it becomes popular.

-spc

[1] Not to belator the inanimate equus pleonastically, but the Google
archives *SUCK*! The List-Archive: header is malformed, and the
interface is not friendly (in my opinion). I know it's probably a
non-starter to get to a better host, but man, does the archive suck
now.

[Sainan]

I vaguely remember seeing this, but didn't really pay much attention to it back then.

I mean, I suppose that in a typical coroutine cooperative multitasking scenario, the yielded values are unused, so it makes sense to try to make them have meaning; but I feel like this is just asking for trouble as all coroutine schedulers I've ever written simply discard yielded values (maybe raising a warning to you about this happening if you're lucky).

For cases like "forget about the current thread", you can just tell the scheduler directly, e.g. such that the entry referencing the current coroutine becomes nil. Then you can yield knowing that it will be your last.

Of course, now you're also relying on particular scheduler behaviour, but at least this behaviour is explicitly coded in so you're much more likely to get some sort of message in unexpected scenarios.

-- Sainan

[Sean Conner]

It was thus said that the Great 'Sainan' via lua-l once stated:

> I vaguely remember seeing this, but didn't really pay much attention to it
> back then.
>
> I mean, I suppose that in a typical coroutine cooperative multitasking
> scenario, the yielded values are unused, so it makes sense to try to make
> them have meaning; but I feel like this is just asking for trouble as all
> coroutine schedulers I've ever written simply discard yielded values
> (maybe raising a warning to you about this happening if you're lucky).

My coroutine scheduler also ignores the yielded values (not that I pass
any---I just checked). That's because the code doing the yield() has more
context about what it wants than the scheduler. Dealing with TCP is
different from TLS is different from a TTY. The scheduler doesn't need to
know all the gritty details, just that ths coroutine has yielded for some
reason. It's up to some other code that runs on the main coroutine (the one
receiving all the events from select()) to reschedule the appropriate
coroutine based upon the event.

-spc

[Sainan]

Interesting approach. I just poll. If nothing new came up -> yield. If something new did come up -> process it. Rinse & repeat.

-- Sainan

[Sean Conner]

It was thus said that the Great 'Sainan' via lua-l once stated:

> Interesting approach. I just poll. If nothing new came up -> yield. If
> something new did come up -> process it. Rinse & repeat.

First off, when I say/use "select()" I actually mean one of epoll_wait()
(Linux), kqueue() (BSD derived), poll() (most any other POSIX system) or
select() (as a last ditch resort) [1]. And second, yes, I do the select().
The main event loop looks like (some details removed [2]):

local function eventloop(done_f)
if done_f() then return end

local timeout = -1 -- wait indefinitely by default
local now = clock.get('monotonic')

while #TOQUEUE > 0 do -- check the timeout queue
local co = TOQUEUE[1]
if co.trigger then
timeout = co.awake - now -- calculate new timeout
if timeout > 0 then -- if positive, wait at least this long
break
end
schedule(co.co,unpack(co))
end
TOQUEUE:remove()
end

if #RUNQUEUE > 0 then -- run queue, if not empty
timeout = 0 -- do a quick poll
end

SOCKETS:wait(timeout)
for event in SOCKETS:events() do
event.obj(event) -- run function associated with socket/fd
end

while #RUNQUEUE > 0 do
-- run items in run queue,
-- deal with dead coroutines
end
return eventloop(done_f) -- and repeat
end

The function associated with the socket/fd deals with dealing with the
actual event (like a listenening socket will create a new coroutine; a
connected socket will read data and schedule a waiting coroutine etc.).

-spc

[1] https://github.com/spc476/lua-conmanorg/blob/master/src/pollset.c

[2] For the full gory details:
https://github.com/spc476/lua-conmanorg/blob/9b5462b66a26288766dc891802c5138e1ccfb143/lua/nfl.lua#L160

[Sainan]

Oh boy, I sure do hope there's no unexpected problems when trying to run this high-level language code that works on my Linux machine on a Windows machine. :P

-- Sainan

[Denis Dos Santos Silva]

checkout:

https://www.tset.de/termfx/

https://www.tset.de/term/

https://www.tset.de/lpty/

[Rett Berg]

Sorry y'all, I wasn't subscribed to this thread so missed a large amount of the conversation

> Well, just the part I don't understand is this "LAP protocol" stuff, and why you need such a huge library for it. (I solved real problems in less lines of code.)

The code is here: https://github.com/civboot/civlua/blob/main/lib/lap/lap.lua

It currently totals 409 lines, so not "huge" by most people's imagination I think...

It does three main things:

• 4 global variables all starting with LAP_ -- these are actually the entire protocol.
• ~250 lines: Provides common functionality and types such as Send/Recv channel, schedule(), error formatting, etc. These aren't essential to the protocol but are nice and provide a reference for how the protocol works.
• ~150 lines: the reference lap executor with error handling and user-helping checking and logging. Also not necessary but also nice.
• ~10 lines: support for protocol to switch the global lua state (i.e. the io module) from sync <-> async mode 

The latter is especially important, as nearly all of the libraries I'm writing can now work in either synchronous or asynchronous code. Everything in vt100 can as well except the input() and resize() methods.

I don't like the approach of fiddling with some kind of global executor (or even worse an executor variable) as it is dependent on the user writing such an executor. If you know the internals of your function you can "drive" a LAP coroutine as simply as:

https://github.com/civboot/civlua/blob/e4d49823a688f66e2e2b4023366dde960a639c50/lib/vt100/vt100/testing.lua#L54

    local inTh = co.create(function() t:input(r) end)

    while t._waiting do
      T.assertEq({true, 'poll', 0, fd.sys.POLLIN}, {ds.resume(inTh)}) -- always yields "poll" on stdin
    end

My approach gives flexibility -- the executor just has to support all the possible yielded values, with the simplest possible supporting being to just adding the coroutine to LAP_READY (aka run it again on the next executor loop).

> For cases like "forget about the current thread", you can just tell the scheduler directly, e.g. such that the entry referencing the current coroutine becomes nil.

In LAP you just yield "forget" for this case. I used to forget on nil/false but as you say, I was accidentally yielding and not getting an error. Now coroutine.resume should only return nil if the coroutine is dead (function done) -- and it throws an error otherwise.

> That's because the code doing the yield() has more

context about what it wants than the scheduler. Dealing with TCP is
different from TLS is different from a TTY. The scheduler doesn't need to
know all the gritty details, just that ths coroutine has yielded for some
reason. It's up to some other code that runs on the main coroutine (the one
receiving all the events from select()) to reschedule the appropriate
coroutine based upon the event.

I don't understand. You want your executor to poll a LIST of values which are associated with coroutines -- therefore the logic of handling this behavior belongs in the scheduler -- not in the coroutine. The coroutine just says it wants the scheduler to poll on a specific fileno with a specific pollcode (i.e. POLLIN).

I don't want the coroutine code to know any details about polling except it needs a poll done... actually, it seems you are doing something similar here:

SOCKETS:wait(timeout)
for event in SOCKETS:events() do
event.obj(event) -- run function associated with socket/fd
end

I guess there are other cases I want to support besides poll, including "nice()" (yielding but wanting to be run ASAP) and sleeps don't need to use polls (a pure-lua binary heap is just fine IMO). 

Best,

Rett

[Sean Conner]

It was thus said that the Great Rett Berg once stated:

> Sorry y'all, I wasn't subscribed to this thread so missed a large amount of
> the conversation

Sainan:

> > For cases like "forget about the current thread", you can just tell the
> > scheduler directly, e.g. such that the entry referencing the current
> > coroutine becomes nil.

I've found that if I want to "forget about the current thread" (assuming
that "current thread" is "coroutine that is running" is to just return. The
coroutine is now dead, the "executor" knows it dead because it does nothing
to schedule a dead coroutine. That's how I handle it. I do NOT handle the
"this coroutine wants to kill/cancel that other coroutine" because a) I
haven't had the need for that, and b) how does "this coroutine" get a
reference to "that other coroutine"?

> In LAP you just yield "forget" for this case. I used to forget on nil/false
> but as you say, I was accidentally yielding and not getting an error. Now
> coroutine.resume should only return nil if the coroutine is dead (function
> done) -- and it throws an error otherwise.

I would think you would want to avoid resuming a dead coroutine.

Me:

> > That's because the code doing the yield() has more context about what it
> > wants than the scheduler. Dealing with TCP is different from TLS is
> > different from a TTY. The scheduler doesn't need to know all the gritty
> > details, just that ths coroutine has yielded for some reason. It's up to
> > some other code that runs on the main coroutine (the one receiving all
> > the events from select()) to reschedule the appropriate coroutine based
> > upon the event.
>
> I don't understand. You want your executor to poll a LIST of values which
> are associated with coroutines -- therefore the logic of handling this
> behavior belongs in the scheduler -- not in the coroutine. The coroutine
> just says it wants the scheduler to poll on a specific fileno with a
> specific pollcode (i.e. POLLIN).

Here's the flow I have: I have the main coroutine (the one Lua starts
with before any coroutine are created). This is executed (sans some error
checking):

SOCKETS:wait(timeout);
for event in SOCKETS:events() do
event.obj(event)

end

So yes, I have a call to select() [1], and the returned events have data
associated with them---in this case, it's a function that is run in the main
coroutine, called from the "executor" when the appropriate event comes in.
For a TCP listening socket, this code is:

nfl.SOCKETS:insert(sock,'r',function()
local conn,remote,err = sock:accept()

if not conn then
syslog('error',"sock:accept() = %s",errno[err])
return
end

conn.nonblock = true
conn.nodelay = true
local ios,packet_handler = create_handler(conn,remote)
ios.__co = nfl.spawn(mainf,ios)
nfl.SOCKETS:insert(conn,'r',packet_handler)
end)

The socket is wrapped around a Lua-like file object ((it has functions
o:read(), o:write(), o:flush(), o:lines(), o:seek(), o:setvbuf() and
o:close()), and then a function is added to handle read events for this new
socket. If there's data for the connected socket, the event is handled by
the main coroutine in this code:

function(event)
assert(not (event.read and event.write))

if event.hangup then
ios._eof = true
nfl.schedule(ios.__co,"")
return
end

if event.read then
local _,packet,err = ios.__socket:recv()
if packet then
ios._eof = #packet == 0
nfl.schedule(ios.__co,packet)
else
if err ~= errno.EAGAIN then
syslog('error',"socket:recv() = %s",errno[err])
nfl.schedule(ios.__co,false,errno[err],err)
end
end
end

if event.write then
nfl.SOCKETS:update(ios.__socket,'r')
nfl.schedule(ios.__co,true)
end
end

Note: the variable ios is an upvalue and is the Lua-file like object of the
socket.

My "executor" doesn't need to know the details of how to handle the
event---it just passes the event off to some other code. So for a connect
socket, if we get a 'hangup' event (EPOLLHUP, POLLHUP, whatever), we
schedule the coroutine with no data; for a read event, we schedule the
coroutine with the packet and for write, we reset the event trigger to
readonly, and schedule the coroutine to resume. I notice one big difference
between our "executors" is that I can schedule a coroutine with data; yours
doesn't allow for that at all.

And keep in mind, all this code dealing with events from select() are
running on the main coroutine, along with the "executor". There are details
in the coroutines as well. For instance, a coroutine calls:

data = conn:read("*l") -- read a line of data

If there's no data already buffered, the conn:read() routine will eventually
get to some code that does (effectivel):

readdata = coroutine.yield()

that will pause the coroutine until a read event (see above) delivers a
packet and reschedules the coroutine to resume.

> I guess there are other cases I want to support besides poll, including
> "nice()" (yielding but wanting to be run ASAP)

Yup, I can do that in a coroutine:

nfl.schedule(coroutine.running())
coroutine.yield()

I don't have that as a function because I just didn't have a need for it,
and no one has asked for one yet.

> and sleeps don't need to use
> polls (a pure-lua binary heap is just fine IMO).

That's the one type of "event" my "executor" handles. To sleep:

nfl.timeout(3) -- wait 3 seconds
coroutine.yield()

Again, no function for that because I haven't had a need for it. I use
the timeouts mostly for dealing with functions that might block too long:

local data = {}
nfl.timeout(5) -- wait 5 seconds for a connection
local conn = tcp.connect("example.com",'gopher')
nfl.timeout(0) -- cancel timeout

if conn:write("foobar\r\n") then
for line in conn:lines() do
table.insert(data,conn:read("*l"))
end
end
conn:close()

And I use a pure-Lua binary heap for the timeout queue.

-spc

[1] I'm using "select()" as the designator for a series of functions
that all do the same thing---wait for I/O on a file descriptor.
This could be epoll_wait(), kqueue(), poll() of select(), depending
upon the operating sytsem.

[Rett Berg]

s> it does nothing

to schedule a dead coroutine.  That's how I handle it.  I do NOT handle the
"this coroutine wants to kill/cancel that other coroutine" because a) I
haven't had the need for that, and b) how does "this coroutine" get a
reference to "that other coroutine"?

That's not the problem being solved. The problem being solved is throwing an error when a LAP coroutine yields nil/false (a common mistake).

Eventually I'll probably put the check behind a SAFETY flag or something so it doesn't affect runtime performance in production but fails fast in tests.

> My "executor" doesn't need to know the details of how to handle the
event

I'm a bit confused of what "event means" -- this is a data packet sent by one co to another? Yes, you wouldn't want the scheduler to know anything about that. Lap provides Send/Recv channel ends for sending events cross thread -- but you can also simply share a table or something if you know what you are doing.

> I notice one big difference
between our "executors" is that I can schedule a coroutine with data; yours
doesn't allow for that at all.

I'm not sure what this means. Who has the data? Here is the actual implementation of my LAP scheduler that handles polling

https://github.com/civboot/civlua/blob/9809209c45a4ddbc84f5fe19d25d0335cdd21d67/lib/civix/civix.lua#L192

here's the PollList object used to track fileno and poll events

https://github.com/civboot/civlua/blob/9809209c45a4ddbc84f5fe19d25d0335cdd21d67/lib/fd/fd.lua#L193

Here's the C code that converts it from the pollList+timeout to a list of filenos that are ready

https://github.com/civboot/civlua/blob/9809209c45a4ddbc84f5fe19d25d0335cdd21d67/lib/fd/fd.c#L557

The PL (polllist) object supports all the normal polling "events" bitmap -- so it supports POLLIN, POLLOUT and other forms of "polls that have data". I would think you could build a nearly-pure Lua TCP implementation on it (with the addition of a l_socket() function in fd.c or an addon library)

I think we have very similar approaches, the primary difference is that you interact with a "global executor" and I modify a few global tables and yield to the calling executor. I actually started by doing an approach similar to yours but I wanted to utilize the yield for (I think?) a small performance improvement, especially if someone (eventually) implemented the LAP scheduler in C.

Possibly one design goal (I'm not sure) is that I'm trying to keep (within reason) as much of the logic in Lua (and as little in C) as possible. That's why fd.c works the way it does -- it allows Lua code to choose whether to have a blocking or non-blocking file handle, even for disk files. The files simply return a code if the file would block -- all the LAP yielding/etc is done in pure Lua (never in C).

Best,

Rett

[Sean Conner]

It was thus said that the Great Rett Berg once stated:
>

> I think we have very similar approaches, the primary difference is that you
> interact with a "global executor" and I modify a few global tables and
> yield to the calling executor.

That sounds about right.

-spc