asynchronous lua-resty-exec/lua-resty-shell

[Nelson, Erik - 2]

I'm trying to asynchronously execute programs in openresty, and lua-resty-exec and lua-resty shell have nice nonblocking interfaces along the lines of

 

local exec = require'resty.exec'

local prog = exec.new('/tmp/exec.sock')

local res, err = prog('uname')

 

I know that prog() in this case doesn’t block the web werver, but it does block progression the request.

 

I’m hoping to do something along the lines of this pseudocode.

 

location /run_program {

    content_by_lua_block {

        local prog = require'resty.exec'.new('/tmp/exec.sock')

        local handle = async_prog(‘long_running_program')  -- I don’t want to wait until the subprocess exits here

        ngx.say(‘started a long running program at handle ’ .. handle)

    }

}

 

location /check_program_status {

    content_by_lua_block {

        local handle = ngx.req.get_uri_args().handle

         local res, err = async_prog_wait(handle)  -- also don’t block here, just status like waitpid() with WNOHANG

        ngx.say(‘long running program status is ’ .. res)

    }

}

 

Anyone have any ideas?  Or a suggestion for a better way to do it?

 

Thanks

 

Erik

---

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.

[John Regan]

Hi there! Author of lua-resty-exec here, long-running processes are tricky, but do-able.

lua-resty-exec keeps a connection open while a program is running, and you can't pass connections to different contexts (you'll get a "attempt yield across c boundary" error).

So, in my init_worker_by_lua_block, I fire up a "process manager" in an nginx timer - this basically spawns a background thread, like

init_worker_by_lua_block {
  processmgr = require('process_manager').new()
  ngx.timer.at(0,function()
    processmgr:run()
  end)
}

That "process manager"'s run function connects to some notification system - in my case, I use redis. Any other pub/sub type of system could work, like https://github.com/slact/ngx_lua_ipc, or maybe the process manager just continuously polls for new jobs from some queue system, whatever.

Then when I want to fire up some long-running, background program, I fire off a notification with details on what program to run.

The background "process manager" picks up the notification, and it calls the "prog" function in an nginx thread, like

local handle = ngx.thread.spawn(function()
  local res = prog('long_running_program')
  call_some_useful_function_when_done
end)

and then stores that handle in a table, and said table is keyed to something based on the program arguments or some other property (ie, maybe you're keeping track of users so you key off of that).

There's a few ways to check the status - I *think* you can call "coroutine.status" on the thread handle, or that might not work, I'm not really sure. Everything I do is event-based, so when my program exits, I broadcast a notification that it's done and things react appropriately. Some other ideas:

Have your ngx.thread.spawn function set/unset something in shared memory, like:

local handle = ngx.thread.spawn(function()
  ngx.shared.progs.set('some-key',true)
  local res = prog('long_running_program')
  ngx.shared.progs.set('some-key',nil)
end)

Then you can call "ngx.shared.progs.get('some-

key')" to check if the process is still running or not.

Hope that helps!

[Nelson, Erik - 2]

Those are some useful ideas, I appreciate the response!  In the meantime I hacked together an asynchronous version using luvit, which was a bit of fun.  https://luvit.io/

 

But I’ll review and see if your suggestion is more appropriate, I’d rather not maintain my one-off code if I don’t have to.