Switching to "continuable" style.

[Tim Caswell]

Back in the early days of nodejs, we had an interface known as node
promises. They were basically event emitters that emitted "value" and
"error" events. This was terrible, so we removed them in favor of
callbacks. During this transition I proposed and pushed for a style
known as "continuable". I even wrote a howtonode article about it at
<http://howtonode.org/do-it-fast>

Basically the change in luvit would be to change all functions that
accept a callback as the last arg to instead return a curried version
of the function that accepts the same callback.

Before:

-- simple async function with optional arg
local function doIt(arg1, arg2, callback)
if type(arg2) == "function" and type(callback) == "nil" then
callback = arg2
arg2 = "default value"
end
callback(nil, arg1 .. ":" .. arg2)
end

doIt("Hello", function (err, result)
p(err, result)
end)

After:

-- simple async function with optional arg
local function doIt(arg1, arg2) return function (callback)
if not arg2 then arg2 = "default value" end
callback(nil, arg1 .. ":" .. arg2)
end end

doIt("Hello")(function (err, result)
p(err, result)
end)

As you can see this adds a little boiler plate when defining an async
function, but it's not too bad. But it's way more powerful. It's a
cross between the raw callback and a promise or monad. Also handling
optional args is much since the callback is never misaligned.

The real power comes from the fact that you can use the returned
function from the first call and do things with it. Powerful flow
control libraries (like Do in the article) can be written. Also
something I realized yesterday is this makes working with coroutines
really nice!

createFiber(function (wait)

-- Auto suspend the thread and resume with the callback's results
local err, result = wait(doIt("Hello", "World"))

-- Same, but true flag means to auto-handle the err parameter
local result = wait(doIt("Hello"), true)

end)

Unlike the current fiber module that requires wrapping every function
before using in every coroutine that uses it, this requires no
wrapping at all. The wait function works on any of the returned
continuables from any new style async function.

Also I've been feeling out the syntax in various languages. It's
really pretty in coffeescript, and I imagine also in moonscript.
https://gist.github.com/1b47cba9f3a066147966


My question is how much of a burden would this be for everyone to port
their code to this format? Is it a deal breaker or an annoyance. Do
you like the idea or think it's stupid?

-Tim Caswell

[Tim Caswell]

I added an example that shows continuables combined with the new proposed fiber module combined with the new proposed stream API.  It's beautiful!

https://github.com/luvit/luvit/blob/master/examples/stream/test.lua#L34-43

[Vladimir Dronnikov]

I like the way they made stuff coroutine-aware in
https://github.com/grrrwaaa/luauv

--dvv

[Julien Ammous]

Hi,

I am keeping a close look at luvit and ideas like this one really makes me want to try it out :)

This project is very promising !

[Brian Maher]

Sorry I didn't see this thread earlier, but I've been thinking about coroutines in the context of event loops also.  I think I've got an idea that doesn't require you to rewrite APIs and would be rather trivial to implement.

The basic idea is to run everything in the context of a coroutine, so your event loop callbacks look something like this:

function onevent(event, co)

    local res = {coroutine.resume(co)}

    if type(res[0]) == "function" then

        res[0](coroutine.wrap(co), unpack(res, 2))

    end

end

This would allow you to make your program flow appear as though it is doing "blocking" calls:

    local http = require('http')

    local req = http.request({

        host = "luvit.io",

        port = 80,

        path = "/"

    }, function (res)

        while ( true ) do

            -- Ideally you could encapsulate this in an API

            -- that "looks like it blocks" and appears as an

            -- iterator so it can be used in a for loop.

            local chunk = coroutine.yield(function(co)

                res:once('data', co)

                res:once('end', co)

            end)

            if chunk then

                p("ondata", {chunk=chunk})

            else

                res:destroy()

            end

         end

    end)

    req:done()

I think that would be a bit more intuitive for the uninitiated.

...another related thing that looks interesting is node's domain API:

http://nodejs.org/api/domain.html

It seems like that would make it easier to do "cleanup" when errors occur in your application.

Cheers,

-Brian

[agentzh]

Hello!

On Sat, Oct 27, 2012 at 8:40 AM, Brian Maher wrote:
> Sorry I didn't see this thread earlier, but I've been thinking about
> coroutines in the context of event loops also. I think I've got an idea
> that doesn't require you to rewrite APIs and would be rather trivial to
> implement.
>
> The basic idea is to run everything in the context of a coroutine, so your
> event loop callbacks look something like this:
>

The nginx lua (aka ngx_lua) module already does this by running
everything in a boilerplate Lua coroutine for each request, for
example, the cosocket API:

http://wiki.nginx.org/HttpLuaModule#ngx.socket.tcp
http://wiki.nginx.org/HttpLuaModule#ngx.socket.udp

The cosocket APIs are mostly compatible with the LuaSocket API by
design. And the Redis, Memcached, MySQL client libraries based on this
API are very compact and elegant Lua:

https://github.com/agentzh/lua-resty-redis
https://github.com/agentzh/lua-resty-memcached
https://github.com/agentzh/lua-resty-mysql

And everything is 100% nonblocking out of the box :)

To do truly concurrent processing in a single request handler, ngx_lua
also implements the "light thread" API:

http://wiki.nginx.org/HttpLuaModule#ngx.thread.spawn

The "light threads" in ngx_lua are essentially Lua coroutines
automatically scheduled by ngx_lua (and the Nginx event model per se),
so the Lua programmer does not have to explicitly schedule the Lua
coroutines themselves, which can quickly become very tedious for
real-world applications.

All existing synchronous but nonblocking APIs provided by ngx_lua work
with the "light threads" out of the box.

I'm sure Luvit can do something similar to avoid NodeJS's callback
nightmare in the Lua world ;) We do have native support for coroutines
in Lua, which is really a big advantage compared to some other
languages like JavaScript ;)

Best regards,
-agentzh

[Tim Caswell]

I'm fully aware that lua has coroutines built-in. This is the reason
I ported nodejs to lua. However, I don't feel that callbacks are evil
and something to be avoided at all costs. I want to be able to use
either callbacks *or* coroutines in my code as an app developer. Both
have their strengths, and the lua language is good at both. The nice
thing about this modified callback format I propose is that it's
simple while allowing both types of usage.

[Julien Ammous]

I also thing both have their uses.

Running every request in a coroutine is what I am currently doing on Ruby with EventMachine and it works well

but that does not mean I never use callbacks directly too, in Ruby they are called fibers but they look to be the same beast

underneath.

[Pierre-Yves Gérardy]

On Friday, August 31, 2012 5:11:58 AM UTC+2, Tim Caswell wrote:

Back in the early days of nodejs, we had an interface known as node
promises.  They were basically event emitters that emitted "value" and
"error" events.  This was terrible, so we removed them in favor of
callbacks.  

Sorry for 1) more grave digging and 2) being off topic, but I'd like to know what the problem was with promises. They felt opaque for a while, then it finally clicked, and they look like an elegant solution for async code. In practice, what were the pain points?

-- Pierre-Yves

[Tim Caswell]

The things that were called promises in node weren't really good promises.  Proper promises are indeed an elegant abstraction.  But they are a pretty heavy abstraction and aren't for everyone.  This proposal I made have most of the benefits of full promises, without are super simple and easy to understand and implement.  Also combined with coroutines, they are just as powerful as full promises in terms of error handling and propagation.

[Pierre-Yves Gérardy]

Thanks. 

I was thinking of relying on promises for my next JS endeavour (in Angular), and was afraid of some hidden defects.

Lua is sooo much cleaner than JS. TCO, coroutines, metamethods, tables as real hash maps, weak tables (for keys and values), multiple return values... The JS guys are slowly getting to feature parity, but their hand is IMO to heavy.

-- Pygy