resty-http and cookies

[Gordon Madarm]

I'm using resty-http (https://github.com/pintsized/lua-resty-http) to proxy requests to a backend server. I want to mirror all headers from the proxied response to the client. Using the example code from the documentation, I can access *most* of the headers via the res.headers table but some headers, like Set-Cookie are not there for some reason. How can I forward all proxied response headers including cookies?

thanks,
- G

[Jon Keys]

On Tuesday, September 16, 2014 4:05:39 PM UTC-4, Gordon Madarm wrote:

I'm using resty-http (https://github.com/pintsized/lua-resty-http) to proxy requests to a backend server. I want to mirror all headers from the proxied response to the client. Using the example code from the documentation, I can access *most* of the headers via the res.headers table but some headers, like Set-Cookie are not there for some reason. How can I forward all proxied response headers including cookies?

I've run into similar problems myself (https://groups.google.com/d/msg/openresty-en/HXDHB6q8CRs/cgJvcMQNh4sJ)

Just curious, why not use ngx.location.capture (http://wiki.nginx.org/HttpLuaModule#ngx.location.capture) with proxy_pass (e.g. code can be seen here: https://groups.google.com/d/msg/openresty-en/1qmVwQwTUw4/ZT2vbRrpzjkJ)? You can automatically forward pretty much everything with these tools.

There are contexts where ngx.location.capture won't work of course (e.g. log_by_lua). In this case you can always use the ngx.var.* variables. 

E.g.

-- for list of builtin headers see: https://github.com/nginx/nginx/blob/b72398c9a386f5100c857869af10a910d97549fe/src/http/ngx_http_variables.c#L281

function builtinRespHeaders(include_ext)

    local h = {}

    local content_type = ngx.var.sent_http_content_type

    local content_length = ngx.var.sent_http_content_length

    if content_type then

        h["content-type"] = content_type

    end

    if content_length then

        h["content-length"] = content_length

    end

    if include_ext == true then

        local location = ngx.var.sent_http_location

        local last_modified = ngx.var.sent_http_last_modified

        local connection = ngx.var.sent_http_connection

        local keep_alive = ngx.var.sent_http_keep_alive

        local transfer_encoding = ngx.var.sent_http_transfer_encoding

        local cache_control = ngx.var.sent_http_cache_control

        if location then

            h["location"] = location

        end

        if last_modified then

            h["last-modified"] = last_modified

        end

        if connection then

            h["connection"] = connection

        end

        if keep_alive then

            h["keep-alive"] = keep_alive

        end

        if transfer_encoding then

            h["transfer-encoding"] = transfer_encoding

        end

        if cache_control then

            h["cache-control"] = cache_control

        end

    end

    return h

end

 

thanks,
- G

[James Hurst]

On 16 September 2014 21:05, Gordon Madarm <gma...@gmail.com> wrote:

I'm using resty-http (https://github.com/pintsized/lua-resty-http) to proxy requests to a backend server. I want to mirror all headers from the proxied response to the client. Using the example code from the documentation, I can access *most* of the headers via the res.headers table but some headers, like Set-Cookie are not there for some reason. How can I forward all proxied response headers including cookies?

Hi,

You shouldn't have any problem accessing all headers using lua-resty-http. Can you post example code and your server response in a gist? You may need to pay attention to the case of the header key, since HTTP header fields are technically case insensitive?

--
James Hurst

ja...@pintsized.co.uk

[Aapo Talvensaari]

On Tuesday, September 16, 2014 11:31:07 PM UTC+3, Jon Keys wrote:

Just curious, why not use ngx.location.capture

I asked this about a month ago:

https://groups.google.com/d/msg/openresty-en/6OwhJioFVoo/LTxMumhbFIUJ

See the answers from Hamish and agentzh.

"IMHO, the subrequest route is a dead end due to the limited 

implementation in the nginx core. For example, it's VERY hard to do 
streaming processing with a subrequest or aborting a running 
subrequest safely without terminating the whole main request. I 
already have no plan to add fancy features to the existing subrequest 
API but I'll still maintain this thing to ensure it works as specified 

:)" -- agentzh 

[Gordon Madarm]

Hi James,

The code calls the following function which proxies the request. As shown below, at first I tried using 'ngx.header.set_cookie = res.headers["Set-Cookie"]' to set the cookie. This worked for setting the content type and length but not the cookie set in the server response. When that didn't work I added a debugging loop to log all of the response headers and saw that the Set-Cookie response header is not there, despite the fact that the back end server sets it (as shown via a tcpdump). Having said that, the response body is proxied correctly. Any ideas how I can proxy the Set-Cookie header? An example response header is show below as well.

function ProxyRequest(Host, URI, Params)
    local http = require "resty.http"
    local httpc = http.new()
    local UserAgent = "Openresty"
    httpc:set_timeout(2000)
    httpc:connect(Host, 80)

    local res, err = httpc:request{
        method = "POST",
        path = URI,
        body = Params,
        headers = {
            ["Host"] = Host,
            ["User-Agent"] = UserAgent,
            ["Content-Type"] = "application/x-www-form-urlencoded"
        },
    }
    if not res then
        ngx.say("failed to request: ", err)
        return
    end

    for k,v in pairs(res.headers) do
        -- Log all headers here - note the Set-Cookie header is missing for some reason
        ngx.log(ngx.ERR, "Header: ", k, " ", v)
    end

    ngx.header.content_type = res.headers["Content-Type"]
    ngx.header.content_length = res.headers["Content-Length"]
    ngx.header.set_cookie = res.headers["Set-Cookie"]
   
    local reader = res.body_reader
      repeat
        local chunk, err = reader(8192)
        if err then
          ngx.log(ngx.ERR, err)
          break
        end
        if chunk then
          ngx.say(chunk)
        end
      until not chunk

      local ok, err = httpc:set_keepalive()
      if not ok then
        ngx.say("failed to set keepalive: ", err)
        return
      end
end

-------------------------------------server response headers--------------------------------------
HTTP/1.1 200 OK
Server: nginx/1.4.1
Date: Wed, 17 Sep 2014 07:38:30 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.3.10-1~lucid+2uwsgi2
Set-Cookie: login=test%2Ftest
Content-Length: 5117
--------------------------------------------------------------------------------------------------------------

thanks,
- G

[James Hurst]

On 17 September 2014 09:47, Gordon Madarm <gma...@gmail.com> wrote:

Hi James,

The code calls the following function which proxies the request. As shown below, at first I tried using 'ngx.header.set_cookie = res.headers["Set-Cookie"]' to set the cookie. This worked for setting the content type and length but not the cookie set in the server response. When that didn't work I added a debugging loop to log all of the response headers and saw that the Set-Cookie response header is not there, despite the fact that the back end server sets it (as shown via a tcpdump). Having said that, the response body is proxied correctly. Any ideas how I can proxy the Set-Cookie header? An example response header is show below as well.

Hmm, strange. The following example works fine here...

https://gist.github.com/pintsized/7b488c5c97eaa48feb33

I tried it with both a single cookie and setting multiple cookies in case that was relevant, but no problems at all. Can you test with my simpler version at your end?

--
James Hurst

ja...@pintsized.co.uk

[Jon Keys]

Wow. Thanks for sharing. This really is a shock to me.

As far as I know this is currently the only supported (i.e. baked in to OpenResty bundle) way to make outbound HTTP requests. Further, it is the only way to make outbound HTTPS requests in the latest stable OpenResty bundle.

Perhaps this will change in the next release or two.

[James Hurst]

On 17 September 2014 14:24, Jon Keys <jon....@gmail.com> wrote:

"IMHO, the subrequest route is a dead end due to the limited 

implementation in the nginx core. For example, it's VERY hard to do 
streaming processing with a subrequest or aborting a running 
subrequest safely without terminating the whole main request. I 
already have no plan to add fancy features to the existing subrequest 
API but I'll still maintain this thing to ensure it works as specified 

:)" -- agentzh 

Wow. Thanks for sharing. This really is a shock to me.

As far as I know this is currently the only supported (i.e. baked in to OpenResty bundle) way to make outbound HTTP requests. Further, it is the only way to make outbound HTTPS requests in the latest stable OpenResty bundle.

Perhaps this will change in the next release or two.

Yeah, though it's probably worth adding that efforts were made to introduce streaming to the subrequest API, but there just turned out to be no nice way to do it, and the cosocket approach is just cleaner and, surprisingly to me at first, more efficient.

My lua-resty-http module has SSL support if you're using the latest RC of OpenResty thanks to SSL cosockets, so yes, this will shortly be available in a stable release.

Going forward, I'd be happy for my module to be included and tested as part of OpenResty releases, if that was deemed appropriate. Yichun has high standards though ;)

Regards,

--
James Hurst

ja...@pintsized.co.uk

[Gordon Madarm]

Thanks James, that worked!