openresty + file I/O + semaphores

[Andrew Melo]

Hi all,

I'm working to develop an application with openresty to perform both data movement (upload/download files) and remote data access (allowing clients to perform byte-range reads of files over HTTPS).

I'll preface by saying I'm a novice to OpenResty, so I might be looking to the wrong solution. That being said, we need to do some processing of content to do some checksumming-type calculations and later to report to various monitoring streams, so we can't simply delegate the file I/O to nginx. 

From my understanding the regular file I/O functions in lua are blocking, so we should avoid using them in OpenResty (otherwise the event loop will block). I have some knowledge of libuv from my C days, so I attempted to use the bindings at https://github.com/luvit/luv to rewrite some code to be non-blocking. This particular bit of code needs to process the data in-order, so I use a semaphore to make sure a new read doesn't occur until the previous read callback is completed: (this is abridged, minimal reproducer linked below)

local more_bytes = true
local req = uv.fs_open(path, 'r', tonumber('644', 8), function(err, fd)
local read_sem, err = semaphore.new()
while more_bytes do
local read_complete = false
uv.fs_read(fd, 1024, nil, function(err, data)
if (data ~= nil) then

                           more_bytes = false

end
if data then
--  DO REAL WORK
end
read_sem:post(1)
end)
-- This blocks until the previous read_sem.post(1) is executed
local ONE_YEAR_SECS = 31536000
read_sem:wait(ONE_YEAR_SECS)
end

        cb_complete = true
end)
while not cb_complete do

        -- I don't know a better way to get the uv event loop to run within OpenResty...
uv.run("nowait")
ngx.sleep(0.000001)
end

This, however, fails with "Uncaught Error: /usr/local/openresty/lualib/ngx/semaphore.lua:160: attempt to yield across C-call boundary" on the read_sem_wait() call. I'm writing because I'm unsure of a better wayto implement this. I would think that if I drop the read_sem:post() call above, then there wouldn't be a crossing of a C boundary (e.g. there is not post() -> wait() transfer of control), then the error would go away, but that doesn't work either.

So, I guess my questions are:

a) Is there a better way to do large file I/O within OpenResty that still allows lua code to see the bytes moving across and gets a chance to perform computation

b) Is there a way to add the libuv polling handle to nginx' event loop so I don't have to pump it "manually" from the lua code?

c) Is there a canonical way to run/debug OpenResty code? I currently start a docker container with OpenResty with my source trees bind-mounted inside, but it would be nice to be able to do some sort of unit-testing of code where the ngx module is stubbed out. Is there already an existing framework that exists for that?

I've put a minimal reproducer here: https://gist.github.com/PerilousApricot/77a721c85ea6f558af856872ce7d4454 to test, you can use the supplied dockerfile to build an image with the same dependencies.

Thanks for all your help!

Andrew

[Andrew Melo]

Hello,

I'm still looking for a solution here, and I'm wondering if there
exists code/example of how to add an FD to the nginx epoll descriptor?
That way, I could use libuv to do file I/O and nginx would be able to
wake up when new file events come in. I've looked at a few different
--resty libs and haven't seen an example of this.

Thanks!
Andrew

> --
> You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/3_Uk37uBEPo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/openresty-en/66f544fb-d06b-42fe-b2a7-60c5c885ad9en%40googlegroups.com.

[Junlong li]

I don't know if inotify  + lua-resty-shell can meet your needs.
Use the inotify to monitor the changes, and read the changes using lua-pipe to read the change messages.