resty.http connection timeout

[Gordon Madarm]

I use lua-resty-http to perform an external API query. Many times I need to run the script twice. The first time the request times out with the error below, the second time I get the expected response. I tried increasing the resty.http timeout value several times via set_timeout() to no avail. In a situation where the remote host resets the connection, how can I gracefully handle that error?

2014/07/16 07:36:13 [error] 10742#0: *2176281 lua tcp socket read timed out,
client: 1.2.3.4, server: acme.com, request: "GET /script
HTTP/1.1", host: "acme.com"
2014/07/16 07:36:13 [error] 10742#0: *2176281 lua entry thread aborted: runtime
error: /usr/local/src/lua-resty-http/lib/resty/http.lua:601: bad argument #1 to
'pairs' (table expected, got nil)
stack traceback:
coroutine 0:
        [C]: in function 'pairs'
        /usr/local/src/lua-resty-http/lib/resty/http.lua:601: in function
'__index'
        /opt/openresty/nginx/conf/script.lua:171: in function 'HTTPClient'
        /opt/openresty/nginx/conf/script.lua:327: in function 'main'
        /opt/openresty/nginx/conf/script.lua:349: in function
</opt/openresty/nginx/conf/script.lua:1>, client: 1.2.3.4, server:
acme.com, request: "GET /script HTTP/1.1", host: "acme.com"
2014/07/16 07:36:13 [info] 10742#0: *2176281 recv() failed (104: Connection
reset by peer) while sending to client, client: 1.2.3.4, server:
acme.com, request: "GET /script HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php5-fpm.sock", host: "acme.com"

thanks,
-G

[Vladislav Manchev]

Which of the 3 or even 4 lua-resty-http libraries are you using?

A self-contained code example is always useful to illustrate what you're trying to do and thus get better help from the list.

Anyhow - it sounds like you should rather use pcall/xpcall in these cases to catch the error.

Best,
Vladislav

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gordon Madarm]

On Wednesday, July 16, 2014 3:29:33 PM UTC+3, Vladislav Manchev wrote:

Which of the 3 or even 4 lua-resty-http libraries are you using?

Hi Vladislav,

I'm using: https://github.com/pintsized/lua-resty-http

A self-contained code example is always useful to illustrate what you're trying to do and thus get better help from the list.

The issue is in the function below. As per the error log, line 171 corresponds to: Response = r:read_body()

function HTTPClient(APIIP, TargetIP)
    local http = require "resty.http"
    local URI1= "/"
    local URI2 = "/check/" .. TargetIP .. "/"
    local Host = "www.acme2.com"
    local Response = ""
    local httpc = http.new()
    local HTTPClientScore = 0

    httpc:set_timeout(2000)
    httpc:connect(APIIP, 80)

    ngx.log(ngx.ERR, "Connected to ", Host)

    local res, err = httpc:request_pipeline{
        {
            path = URI1,
            method = "POST",
            headers = {
                ["Host"] = Host,
                ["Content-Type"] = "application/x-www-form-urlencoded"
            },
            body = "ip=" .. TargetIP,
        },
        {
            path = URI2,
            headers = {
                ["Host"] = Host,
            }, 
        }
    }

    if err then
        ngx.log(ngx.ERR, "Error on connection", err)
    end

    if res == nil then
        ngx.log(ngx.ERR, "No response received from acme2.com: ", err)
    else
        for i,r in ipairs(res) do
            Response = r:read_body()
        end
        httpc:set_keepalive()
        return Response
    end
end

thanks,
- G

[Vladislav Manchev]

I meant your own piece of code where you're trying to use lua-resty-http.

Also, take a look at the two commits from today - https://github.com/pintsized/lua-resty-http/commits/master

Seems that James fixed the problem you're experiencing and added a note in the docs.

Still not a bad idea to take a look at pcall/xpcall for future reference.


Best,

Vladislav

[James Hurst]

Hi,

This is a definitely a timeout error, though the error you're getting is because I failed to test for it (this only applies to pipelined requests). I've made this a bit better: https://github.com/pintsized/lua-resty-http/commit/5bc745ff510d90b5e76dc75121cdbf8fd3cea466

...so it no longer just bails on pairs(), and will log the socket error, and have updated the docs to show you must test the status for a reasonable value before you access the other fields.

That is, you should do this:

for i,r in ipairs(res) do

    if r.status then
        Response = r:read_body()

    else

        -- There was an error reading this response.

    end
end

But obviously the real issue is because the connection is timing out, so you'll want to investigate / fix that separately.

I would also add, pipelining probably doesn't really buy you much. You could just do two requests normally, and use keepalives. YMMV.

To explain the issue a bit further: Pipelined requests are kinda clumsy, in that the responses simply appear one after the other on the socket. So you have to read everything fully before moving on. In addition, I wanted to make sure the streaming API in this module was still valid for pipelined requests. So the interface is little bit "magic", in that the "responses" table which you get back from "request_pipeline", is just a table of handles, and I use a metamethod to actually read the response as you access the fields themselves. So long as you access the responses in order, it will read status and headers on first "touch" of a field, and the body only when you ask for it (though you *must* ask for it in order to move on).

I toyed with having a "read_response" method, but the trouble is you might want to stream the actual body, and even merge in trailers afterwards, so the interface just looks weird.

James.

--
James Hurst

ja...@pintsized.co.uk