"attempt to yield across C-call boundary" when doing Redis IO from FFI callback

[Michal Cichra]

I'm using http://luapower.com/expat to parse XML and during some callbacks I need to write to redis.

This fails with "attempt to yield across C-call boundary" error.

The stack trace looks kind of like:

./lua/concurredis.lua:68: in function 'write'=
[C]: in function 'XML_Parse'
./lua/expat.lua:123: in function <./lua/expat.lua:104>
[C]: in function 'xpcall'

Is there a way to get around this?

[Yichun Zhang (agentzh)]

Hello!

Well, xpcall's callback function is invoked from within a C function
of the Lua/LuaJIT VM which does not support yielding. Better put your
redis accessing code outside the xpcall() call, for example,

local saved_err = nil
xpcall(my_code, function (err) saved_err = err end)
if saved_err then
-- use lua-resty-redis to write saved_err to redis.
end

It is fine, however, to access redis in the "my_code" function though
(but not the xpcall callback function).

Regards,
-agentzh

[Michal Cichra]

Hi!

Even when I don't use xpcall it fails. I made a minimal example https://gist.github.com/mikz/92e7adcc2ef0424360b2

and I run it by `resty test.lua`. `resty` util uses pcall, but even when i run it without it fails with:

2014/10/06 16:22:46 [error] 47946#0: lua entry thread aborted: runtime error: attempt to yield across C-call boundary

stack traceback:

coroutine 0:

[C]: in function 'XML_Parse'

test.lua:25: in function 'gen'

init_worker_by_lua:32: in function <init_worker_by_lua:30>, context: ngx.timer

Because in our project we run user's code, they can do IO. They can write to database and do other things. Logging is implemented by IO.

This feels especially weird because Nginx tries to be non blocking, so all the XML parsing, JSON stream parsing will have to be implemented as callbacks (expat, yajl). 

I'm using latest openresty rc2 with luajit.

nginx version: openresty/1.7.4.1rc2 (no pool)

built by clang 5.1 (clang-503.0.40) (based on LLVM 3.4svn)

TLS SNI support enabled

configure arguments: --prefix=/usr/local/Cellar/ngx_openresty/1.7.4.1rc2/nginx --with-debug --with-cc-opt='-I/private/tmp/ngx_openresty-pLugzY/ngx_openresty-1.7.4.1rc2/build/luajit-root/usr/local/Cellar/ngx_openresty/1.7.4.1rc2/luajit/include/luajit-2.1 -DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2' --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.56 --add-module=../xss-nginx-module-0.04 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.26 --add-module=../form-input-nginx-module-0.10 --add-module=../encrypted-session-nginx-module-0.03 --add-module=../srcache-nginx-module-0.28 --add-module=../ngx_lua-0.9.12 --add-module=../ngx_lua_upstream-0.02 --add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03 --add-module=../memc-nginx-module-0.15 --add-module=../redis2-nginx-module-0.11 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05 --with-ld-opt='-Wl,-rpath,/usr/local/Cellar/ngx_openresty/1.7.4.1rc2/luajit/lib -L/private/tmp/ngx_openresty-pLugzY/ngx_openresty-1.7.4.1rc2/build/luajit-root/usr/local/Cellar/ngx_openresty/1.7.4.1rc2/luajit/lib' --with-http_ssl_module --with-pcre --with-pcre-jit --sbin-path=/usr/local/Cellar/ngx_openresty/1.7.4.1rc2/bin/openresty --conf-path=/usr/local/etc/openresty/nginx.conf --pid-path=/usr/local/var/run/openresty.pid --lock-path=/usr/local/var/openresty/nginx.lock --with-http_gunzip_module --with-dtrace-probes

[Yichun Zhang (agentzh)]

Hello!

On Mon, Oct 6, 2014 at 7:26 AM, Michal Cichra wrote:
> 2014/10/06 16:22:46 [error] 47946#0: lua entry thread aborted: runtime
> error: attempt to yield across C-call boundary
> stack traceback:
> coroutine 0:
> [C]: in function 'XML_Parse'

This XML_Parse is the culprit. It is a C function which calls your
user callback requiring yielding (for nonblocking IO and etc). You
need to ensure the caller of your user callback to be a Lua function
(as well as the caller's caller, and etc).

>
> Because in our project we run user's code, they can do IO. They can write to
> database and do other things. Logging is implemented by IO.
>

This is fine as long as the user code's caller is implemented as a Lua
function (and none of the caller's ancestor is a C function).

> This feels especially weird because Nginx tries to be non blocking, so all
> the XML parsing, JSON stream parsing will have to be implemented as
> callbacks (expat, yajl).
>

In the model used by ngx_lua, we actually avoid the callback style
wherever possible. We use the iterative style which is callback free.
For example, the cosocket API is synchronous but 100% nonblocking.
Also, the lua-resty-upload library implements a multipart request body
streaming parser without introducing a callback-based API:

https://github.com/openresty/lua-resty-upload#readme

Regards,
-agentzh

[Michal Cichra]

Hi!

So you are saying, that callback driven C libraries like expat and yajl are not compatible with OpenResty/LuaJIT ?

Maybe would be worth having this documented in the readme of the ngx_lua. 

https://github.com/openresty/lua-nginx-module#lua-coroutine-yieldingresuming does not describe it properly IMHO.

And is there some plan to overcome this limitation?

So to implement streaming XML/JSON parser, we have to write one in pure Lua, right? Do you know about some projects doing this?

Thanks!

Michal

[Aapo Talvensaari]

On Tuesday, October 14, 2014 10:19:52 AM UTC+3, Michal Cichra wrote:

So you are saying, that callback driven C libraries like expat and yajl are not compatible with OpenResty/LuaJIT ?

I also do like to hear more about this. On lua-resty-hoedown there is also a call back based renderer:

https://github.com/bungle/lua-resty-hoedown/blob/master/lib/resty/hoedown/document.lua#L51

There are to default renderers, html and html toc -renderer that are C-based:

https://github.com/bungle/lua-resty-hoedown/blob/master/lib/resty/hoedown/html.lua#L42-L43

But if I want to make a Lua based renderer (or overwrite some handlers on html-renderer), it might not work on OpenResty?

Regards

Aapo

[Yichun Zhang (agentzh)]

Hello!

On Tue, Oct 14, 2014 at 12:19 AM, Michal Cichra wrote:
> So you are saying, that callback driven C libraries like expat and yajl are
> not compatible with OpenResty/LuaJIT ?

C callbacks are problematic and suboptimal at least for the LuaJIT VM.
It means calling into Lua VM from external C code. We should stay away
from them whenever possible.

Try always calling into C from within Lua (via FFI) instead of the
other way around. Calling the Lua code (and the LuaJIT VM) from within
external C can never be JIT compiled because of the unavoidable VM
side-effects (as opposed to FFI's side-effect-free approach for JITted
Lua code).

This is not specific to OpenResty but also applies to any LuaJIT apps
in general.

> Maybe would be worth having this documented in the readme of the ngx_lua.
> https://github.com/openresty/lua-nginx-module#lua-coroutine-yieldingresuming
> does not describe it properly IMHO.

We could. Though it's more related to LuaJIT in general. Patches welcome :)

> And is there some plan to overcome this limitation?
>

This limitation is in LuaJIT 2, not in OpenResty :) And for
performance reasons, it may not really worth it because even if we
implement "C coroutines" in LuaJIT (as in coco [1]), it is really slow
as compared to the FFI approach (both piratically and theoretically).

You can check out LuaJIT's author and Coco's author, Mike Pall's reply
below on this topic:

http://comments.gmane.org/gmane.comp.lang.lua.luajit/1774

> So to implement streaming XML/JSON parser, we have to write one in pure Lua,
> right? Do you know about some projects doing this?
>

No really. The C library just needs to expose an iterator-style API
(or "pull style") as in libc's epoll/poll/select. (And we're talking
about C libraries doing streaming parsing here rather than nonblocking
I/O though).

For example, my sregex library [2] exposes a "pull style" C API for
streaming regex matching (and substitution) which is friendly to
LuaJIT both in functionality and performance (via FFI). This is also
suggested by Mike Pall in the thread mentioned above.

Regards,
-agentzh

[1] http://coco.luajit.org/
[2] https://github.com/openresty/sregex