Re: [lua-nginx-module] "attempt to yield" error using smtp module of luasockets (#155)

[agentzh]

Hello!

On Sun, Aug 26, 2012 at 3:26 AM, clubpetey wrote:
>
> I am trying to use LUA to send a email via SMTP. I've load the smtp modules from the luasockets package, but during the send() function I recieve the following error:
>
> 2012/08/26 10:00:57 [error] 2733#0: *5 lua handler aborted: runtime error: attempt to yield across metamethod/C-call boundary
> stack traceback:
> [C]: in function 'send'
> /usr/local/nginx/conf/lua/refer-friend.lua:60: in function while sending to client, client: 127.0.0.1,
>
> I've already changed the messages socket "create" function to ngx.socket.tcp as I read that ngx.socket is compatible with luasockets, I believe the error is due to the smtp library calling socket.protect() I've read in other places that protect() can yield. Here's the code of the send function:
>
> send = socket.protect(function(mailt)
> local s = open(mailt.server, mailt.port, mailt.create)
> local ext = s:greet(mailt.domain)
> s:auth(mailt.user, mailt.password, ext)
> s:send(mailt)
> s:quit()
> return s:close()
> end)
>
> Is this a bug? or expected behavior? If it's expected, is there a way around it?
>

Okay, I've just coded up a complete code sample that enforces the
socket.smtp library shipped with LuaSocket 2.0.2 to use ngx_lua's TCP
cosocket API instead of LuaSocket's own.

The nginx.conf snippet is like this:

location = /t {
content_by_lua_file html/send.lua;
}

And then in the html/send.lua file, we first extend ngx_lua's
ngx.socket API to the extend that socket.smtp will be happy enough:

local socket = ngx.socket

socket._VERSION = "ngx_lua cosocket"

socket.protect = function (f)
local rm = table.remove
return function (...)
local rets = {pcall(f, ...)}
if rets[1] then
rm(rets, 1);
return unpack(rets)
else
local err = rets[2]
if type(err) == "table" then
return nil, err[1]
else
return error(err)
end
end
end
end

socket.sink = function (name, sock)
if name ~= 'keep-open' then
return error(name .. " not supported")
end
return setmetatable({}, {
__call = function (self, chunk, err)
if chunk then return sock:send(chunk)
else return 1 end
end
})
end

socket.newtry = function (f)
return function (...)
local args = {...}
if not args[1] then
if f then
pcall(f)
end
return error({args[2]})
end
return ...
end
end

socket.try = socket.newtry()

socket.skip = function (d, ...)
local args = {...}
local rm = table.remove
for i = 1, d do
rm(args, 1)
end
return unpack(args)
end

And then we make Lua believe that the "socket" module is already
loaded (and it is actually our ngx.socket instead of LuaSocket's):

package.loaded.socket = ngx.socket

Then we load LuaSocket's socket.smtp module and use it as usual:

local smtp = require("socket.smtp")
local mime = require("mime")
local ltn12 = require("ltn12")

source = smtp.message{
headers = {
from = "agentzh <age...@gmail.com>",
to = "foo <f...@bar.com>",
subject = "Hello, guy!"
},
body = {
preamble = "preamble",
[1] = { body = mime.eol(0, "Hello, how is going?") },
}
}

r, e = smtp.send{
from = "<sic...@example.com>",
rcpt = "<ful...@example.com>",
source = source,
server = "127.0.0.1",
port = 25,
}

if not r then
ngx.say("failed to send: ", e)
end

ngx.say("done!")

And then using an HTTP client like "curl" to access the location /t
defined above:

$ curl localhost/t
done!

And now I can get the message on my local SMTP server listening on the
port 25, as follows:

From: <sic...@example.com>
To: <ful...@example.com>
Message:
mime-version: 1.0
content-type: multipart/mixed; boundary="2608201217115879420==00001"
from: agentzh <age...@gmail.com>
date: Mon, 27 Aug 2012 00:11:58 -0000
x-mailer: ngx_lua cosocket
to: foo <f...@bar.com>
subject: Hello, guy!

preamble

--2608201217115879420==00001
content-type: text/plain; charset="iso-8859-1"

Hello, how is going?
--2608201217115879420==00001--

So it works as expected. But I'd rather implement a lua-resty-smtp
library from scratch atop ngx_lua's cosocket API instead of using the
socket.smtp library which is not implemented in the simplest and
fastest way.

Best regards,
-agentzh

[JT Archie]

This is an example that I was looking for.

Your last comment states that this is not "the simplest and fastest way," but my argument would be that its a standard way. I am relatively new to the Lua community, but it seems LuaSockets is the default go to for HTTP request handling, SMTP, etc

If each application built for OpenResty had to implement its own lua-resty-protocol, that pretty much feature locks you into OpenResty as your framework, forever. Is this what is expected with OpenResty?

When trying to handle external HTTP requests to 3rd party servers from OpenResty, I have to setup a proxy-endpoint to those 3rd party servers. I'd rather just call out to the HTTP server with LuaSocket and have it do its magic. It would be great to implement the non-blocking socket implementation for LuaSocket so it integrated flawlessly with the nginx runtime.

Does all this make sense? I apologize, its late, but I needed to write this before I went to bed. :)

[agentzh]

Hello!

On Mon, Nov 5, 2012 at 9:04 PM, JT Archie wrote:
> This is an example that I was looking for.
>
> Your last comment states that this is not "the simplest and fastest way,"
> but my argument would be that its a standard way. I am relatively new to the
> Lua community, but it seems LuaSockets is the default go to for HTTP request
> handling, SMTP, etc
>

You take me wrong.

I'm not saying using ngx_lua cosocket is not the simplest and fastest
way, but rather the implementation of the SMTP module is LuaSocket is
not efficient in its own right.

> If each application built for OpenResty had to implement its own
> lua-resty-protocol, that pretty much feature locks you into OpenResty as
> your framework, forever. Is this what is expected with OpenResty?
>

OpenResty provides a special ecosystem (and also extension) for Lua.
To fully take advantage of all the APIs provided by ngx_lua, you need
to write special libraries like the lua-resty-* series which are
amazingly short if you even bother looking at the Lua source.

If you want to keep compatibility with other Lua environment, you can
surely limit yourself with the common subset of the ngx_lua cosocket
API and the LuaSocket library. And that's also why I keep API
compatibility with LuaSocket in the first place.

And...you surely won't expect most of the NodeJS library will work at
client side in a common web browser, right? The same thing applies to
the existing lua-resty-* libraries which are designed and optimized
specifically for OpenResty.

> When trying to handle external HTTP requests to 3rd party servers from
> OpenResty, I have to setup a proxy-endpoint to those 3rd party servers. I'd
> rather just call out to the HTTP server with LuaSocket and have it do its
> magic. It would be great to implement the non-blocking socket implementation
> for LuaSocket so it integrated flawlessly with the nginx runtime.
>

We already have. Check out the cosocket API provided by ngx_lua:

http://wiki.nginx.org/HttpLuaModule#ngx.socket.tcp

But I'm not sure if the http client library provided by LuaSocket can
work out of the box by "monkey patching" as demonstrated in one of my
earlier emails in this thread because the cosocket API just implements
a common subset of the LuaSocket API.

Also, there's a lua-resty-http library contributed by our users, which
may have a lot of rough edges at the moment:

https://github.com/liseen/lua-resty-http

And I'm going to take it, polish it to a good state, and include it in
the OpenResty bundle by default.

BTW, personally I'm always very willing to reinvent all the common
client drivers in the lua-resty-* series and I'd ensure that the code
is in a really good shape and always keep performance in mind.

I could have borrowed code from the existing Lua libraries written by
others but I usually don't. That's because very few 3rd-party Lua
libraries in the wild are really carefully written (at least to me),
including LuaSocket itself (no offense to other Lua library
contributors), just as in many other dynamic language communities.

In short, I love only serious code (be it Lua or C) that makes things
really work and work well.

Best regards,
-agentzh