Nginx lua module - threads hang process with no error

[Sam Lawrence]

Hi,

I am running lua-ngx-module with Nginx v1.4.x with some Lua code that spawns off threads to get data concurrently, ie: it simulates ngx.location.capture_multi.  However when running in a loop of hundreds or thousands of requests the parent request hangs at a random position without any error message in the Nginx error log.

Given the following nginx server config

location /_parallel/ {

     content_by_lua_file "/etc/nginx/conf.d/multi_request.lua";

}

And contents of multi_request.lua being

-- maximum number of threads to spawn at one time

local max_threads = 24

local function crc_request(uri)

     local crc_res = {}

     crc_res["uri"] = uri

--     crc_res["raw"] = ngx.location.capture(uri)

     return crc_res

end

local function send_requests(buffer)

     local threads = {}

     

     for req_id, req_uri in ipairs(buffer) do

          table.insert( threads, ngx.thread.spawn( crc_request, "/_getcrc" .. req_uri ) )

     end

     

     -- wait for all threads to return

     for tid = 1, #threads do

          local ok, res = ngx.thread.wait(threads[tid])

          

          if not ok then

               ngx.say(tid, ": failed to run: ", res)

          else

               ngx.say(tid, ": uri: ", res["uri"])

         --      ngx.say(tid, ": status: ", res["raw"].status)

         --      ngx.say(tid, ": body: ", res["raw"].body)

          end

     end

end

-- create some URI's to open

local files = {}

for inc = 1,2000,1 do

     table.insert(files, "/imgs/cat" .. inc .. ".jpg" )

end

-- count number of files

local total_files = 0

for i, file in ipairs(files) do

     total_files = total_files + 1

end

-- check how many threads to open at one time with maximum being max_threads

local open_threads = max_threads

if total_files < max_threads then

     open_threads = total_files

end

-- buffer which holds maximum number of files for max_threads

local buffer = {}

local buffer_count = 0

-- for each file get the crc 

for id, file in ipairs(files) do   

     

     if buffer_count < open_threads then

          table.insert(buffer, file)

          buffer_count = buffer_count + 1

     end

     

     if buffer_count == open_threads then

          send_requests( buffer )

          buffer = {}

          buffer_count = 0

     end

end

-- send any remaining requests

ngx.say("sending remaining requests")

send_requests( buffer )

-- end message

ngx.say("closing")

In summary the code above generates a list of 2000 URI's then spawns 24 concurrent processes to make a subrequest for each one and loops until finished.  The above code is then run with

curl -v http://localhost/_parallel/

it will start output the following

1: uri: /_getcrc/imgs/cat1.jpg

2: uri: /_getcrc/imgs/cat2.jpg

3: uri: /_getcrc/imgs/cat3.jpg

4: uri: /_getcrc/imgs/cat4.jpg

5: uri: /_getcrc/imgs/cat5.jpg

6: uri: /_getcrc/imgs/cat6.jpg

7: uri: /_getcrc/imgs/cat7.jpg

8: uri: /_getcrc/imgs/cat8.jpg

...

But it will randomly hang before completing output for the full loop of 2000 files.  Sometimes it hangs in the lower and upper hundreds and early thousands.

Do you have any ideas of how to debug this further?

Thanks,

Sam

[Yichun Zhang (agentzh)]

Hello!

On Thu, Oct 17, 2013 at 7:18 PM, Sam Lawrence wrote:
> I am running lua-ngx-module with Nginx v1.4.x with some Lua code that spawns
> off threads to get data concurrently, ie: it simulates
> ngx.location.capture_multi. However when running in a loop of hundreds or
> thousands of requests the parent request hangs at a random position without
> any error message in the Nginx error log.
>

Thank you for the test case! It reveals two bugs in ngx_lua's light
thread implementation. I've just fixed them respectively by the
following patches:

https://github.com/chaoslawful/lua-nginx-module/commit/0f3c249
https://github.com/chaoslawful/lua-nginx-module/commit/350e38a

Now with the latest master, your test case works for me without problems :)

Thanks!
-agentzh