access_by_lua Basic Authentication Debugging

[david.ri...@enquora.com]

I'm implementing http basic authentication against an upstream key/value store (couchdb in this case) in a server-level access_by_lua_file:

local cjson = require "cjson"
local bcrypt = require "bcrypt"
-- If login name and password are provided with http request, validate them and return 200 status on success
if ngx.var.remote_user and ngx.var.remote_passwd then
    
    -- Fetch user record using remote_user as key
    local fetched_result = ngx.location.capture("/active_users" .. "?reduce=false&startkey=%22" .. ngx.var.remote_user .. "%22&endkey=%22" .. ngx.var.remote_user .. "%22")
    local responseJSON = cjson.decode(fetched_result.body)
    
    -- If one, and only one, matching user profile is returned
    -- verify remote_passwd against hashed password using bcrypt
    local matched_user_profiles = 0
    for _ in pairs(responseJSON['rows']) do matched_user_profiles = matched_user_profiles + 1 end
    if matched_user_profiles == 1 then
        -- local password_hashed = responseJSON['rows'][1]['value']
        
        -- Don't verify password during debugging, just return
        return
    end
end
ngx.header.www_authenticate = [[Basic realm="Enquora Data Services"]]
ngx.exit(401)

Accessing any protected uri returns a 500 server error, but nothing in the nginx logs.

Commenting out code lines indicates the problem occurs at the for _ pairs line that counts the table rows in the decoded json response.

I'm reasonably certain the sub-request url is correct, but anything could be wrong.

I'm at a loss for ways to debug this.

Suggestions?

[Yichun Zhang (agentzh)]

Hello!

On Sat, Oct 19, 2013 at 10:03 AM, <david.ri...@enquora.com> wrote:
>
> Accessing any protected uri returns a 500 server error, but nothing in the
> nginx logs.
> Commenting out code lines indicates the problem occurs at the for _ pairs
> line that counts the table rows in the decoded json response.

Are you sure there are no error logs in nginx's error.log file?

Because some people have troubles in reading nginx error logs, here
are some suggestions:

1. ensure that you're looking at nginx's error.log file, not access.log file.
2. ensure that you don't use a high log level like "crit" or "alert"
in your error_log directive in nginx.conf. See also

http://nginx.org/en/docs/ngx_core_module.html#error_log

Your problem looks like a Lua exception, so you SHOULD get detailed
error messages (at the [error] level) in your nginx's error.log file.

Regards,
-agentzh

[david.ri...@enquora.com]

Turns out the sub-request is hitting the couchdb server (it appears in the logs), but it is being returned 401.

The identical url, copied and pasted into browser or cURL, returns just fine - there is no authentication requirement on the upstream server.

I need to get at the raw request, including headers, but am having trouble doing that still.

Still open to any specific suggestions for debugging non-trivial apps in openresty.

[Yichun Zhang (agentzh)]

Hello!

On Sat, Oct 19, 2013 at 12:06 PM, david.richardson wrote:
>
> I need to get at the raw request, including headers, but am having trouble
> doing that still.

You can get the raw downstream request by the following API functions:

https://github.com/chaoslawful/lua-nginx-module#ngxreqraw_header
https://github.com/chaoslawful/lua-nginx-module#ngxreqget_body_data

If that's what you want.

For nginx upstream request information, you can enable Nginx's debug
logs to get the details: http://nginx.org/en/docs/debugging_log.html

> Still open to any specific suggestions for debugging non-trivial apps in
> openresty.
>

Always keep an eye on your nginx's error.log file (note, it's not
access.log file).

And you're recommended to put custom output code in your key Lua code
path via the `print` or `ngx.log` API functions:

https://github.com/chaoslawful/lua-nginx-module#print
https://github.com/chaoslawful/lua-nginx-module#ngxlog

Best regards,
-agentzh

[david.ri...@enquora.com]

Thks Yichun,

The log destination was misconfigured - this is a development system and we lost track of things. It seems that effective management of multiple apps is going to require some thought.

I finally broke down and found an http sniffer (http://www.tuffcode.com is the only good Mac citizen that I've found, although it gives up features to Charles and WireShark) to diagnose the problem - the authentication request headers are being inherited by the sub-request. Since the upstream data source (couchdb or elasticsearch) is un-authenticated, this is a problem. A 401 is returned and there is no JSON content to parse.

I'd really like to remove all http headers from ngx.location.capture and start from scratch with my own, as well as ensure that HTTP 1.1 is used. Amongst other issues, we really would prefer the upstream data source log requests using an easily identifiable internal user-agent. Ditto for all the other header junk passed on from the browser that initiated the request - it isn't wanted or needed.

What is the proper place/method to remove all these headers and explicitly add just our own the ngx.location.capture request?

Thinking ahead, although this is currently running on a safe internal network, ultimately we should probably authenticate the ngx.location.capture phase using a client certificate. Is anyone doing anything like this?

david

[Yichun Zhang (agentzh)]

Hello!

On Sun, Oct 20, 2013 at 11:25 AM, david.richardson wrote:
>
> What is the proper place/method to remove all these headers and explicitly
> add just our own the ngx.location.capture request?
>

Well, don't use ngx.location.capture(), just use Brian Akins's
lua-resty-http-simple library instead:

https://github.com/bakins/lua-resty-http-simple

The only limitation right now is the lack of support for https. But I
think we'll have that soon :)

Best regards,
-agentzh

[david.ri...@enquora.com]

That looks useful for the future.

For now, problem solved by adding following to the upstream /location:

location /active_users {

proxy_pass_request_headers off;

proxy_set_header Header value;

ngx.location.capture works just as well with another host and port as it does with a sub-request for the current one.

Found by reading documentation on github project site rather than Nginx lua module docs.

Thanks for the pointer to that, Yichun.

Case closed!

david