nginx decodes uri for upstream,

[RJoshi]

Hi,

  I am making good progress on my prototype to replace Oracle OSB with nginx.   One issue I found is  nginx decodes special characters like %2F with / which changes the URI for rest resource and results into not found.  Is there a way to fix it?  Apache has AllowEncodedSlashes.  Does nginx support similar feature?

e.g

/customers/ps%2Fhabcdefghi%3D/credit-cards

is replaced with 

/customers/ps/habcdefghi=/credit-cards

location /customers {

  rewrite ^/customers/(.*) /credit-cards/$1 break;

 proxy_pass https://upstream_backend;

}

[Yichun Zhang (agentzh)]

Hello!

On Fri, Sep 5, 2014 at 2:16 PM, RJoshi wrote:
> I am making good progress on my prototype to replace Oracle OSB with
> nginx. One issue I found is nginx decodes special characters like %2F
> with / which changes the URI for rest resource and results into not found.
> Is there a way to fix it? Apache has AllowEncodedSlashes. Does nginx
> support similar feature?
>

Well, one solution is to use rewrite_by_lua to replace that "rewrite"
directive and preserve everything you need from the original unparsed
URI (via ngx.var.request_uri).

Regards,
-agentzh

[Rohit Joshi]

Yes, I did tried that and set it as proxy pass upstream URL but getting 502 Bad Gateway error.

I will look into details and see what is causing that.

Thanks,
Rohit

> --
> You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/a9Gk4CDHPZA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Yichun Zhang (agentzh)]

Hello!

On Fri, Sep 5, 2014 at 4:08 PM, Rohit Joshi wrote:
> Yes, I did tried that and set it as proxy pass upstream URL but getting 502 Bad Gateway error.
>

It should work. Basically, you parse the stuff in ngx.var.request_uri
yourself in Lua, and then set the new URI via ngx.req.set_uri() (no
redirect set) and the new querystring via ngx.req.set_uri_args().

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Fri, Sep 5, 2014 at 5:50 PM, Yichun Zhang (agentzh) wrote:
> It should work. Basically, you parse the stuff in ngx.var.request_uri
> yourself in Lua, and then set the new URI via ngx.req.set_uri() (no
> redirect set) and the new querystring via ngx.req.set_uri_args().
>

Also, you should ensure your nginx is at least 1.7.1 because earlier
nginx cores have a bug in URI args handling. The latest OpenResty
formal release, 1.7.2.1, should be fine.

Regards,
-agentzh

[Rohit Joshi]

It did work.

Problem was it does not resolving domain when rewrite_by_lua is used.

I have to add resolver /DNS server entry.

If I don't use rewrite_by_lua, proxy_pass works fine without resolver.

Appreciated your help.

[Yichun Zhang (agentzh)]

Hello!

On Sat, Sep 6, 2014 at 4:00 AM, Rohit Joshi wrote:
> Problem was it does not resolving domain when rewrite_by_lua is used.
>
> I have to add resolver /DNS server entry.
>
> If I don't use rewrite_by_lua, proxy_pass works fine without resolver.
>

That's strange. I don't see why you need to configure "resolver" for
this. Maybe you unnecessarily use nginx variables in the server part
of the target url in "proxy_pass"?

The following standalone example emulating your use case works for me
without configuring "resolver":

upstream upstream_backend {
server localhost:8080;
# we could also enable http connection pool here by
configuring "keepalive" and etc...
}

server {
listen 8080;

location /customers {
set $new_uri '';
rewrite_by_lua '
local url = ngx.var.request_uri
local new_uri, n, err = ngx.re.sub(url,
"^/customers/", "/credit-cards/", "jo")
if new_uri then
ngx.var.new_uri = new_uri
return
end
if err then
ngx.log(ngx.ERR, "failed to do regex substitution: ", err)
end
ngx.exit(500)
';
proxy_pass http://upstream_backend$new_uri;
}

# emulated backend service:
location /credit-cards {
content_by_lua '
ngx.say("received raw uri in the backend: ",
ngx.var.request_uri)
';
}
}

Then accessing location /customers using your test request:

$ curl 'localhost:8080/customers/ps%2Fhabcdefghi%3D/credit-cards'
received raw uri in the backend:
/credit-cards/ps%2Fhabcdefghi%3D/credit-cards

$ curl 'localhost:8080/customers/ps%2Fhabcdefghi%3D/credit-cards?a=1&b=3'
received raw uri in the backend:
/credit-cards/ps%2Fhabcdefghi%3D/credit-cards?a=1&b=3

That's what you expected, right? ;)

Regards,
-agentzh

[RJoshi]

Thanks agentzh.  I think this is due to I am looking up upstream domain based on mapping results and setting into a variable.

e.g 

content_by_lua '

    local domain =  get the domain based on service from json config

    ngx.var.upstream = domain .. new_uri

';

 proxy_pass $upstream;

[Yichun Zhang (agentzh)]

Hello!

On Sat, Sep 6, 2014 at 7:33 PM, RJoshi wrote:
> e.g
> content_by_lua '
> local domain = get the domain based on service from json config
> ngx.var.upstream = domain .. new_uri
> ';
>
> proxy_pass $upstream;
>

I don't think you can use content_by_lua and proxy_pass in the same
location. Typo?

Assuming you mean rewrite_by_lua above, then you have 2 choices:

1. Configure the resolver directive to point to your nameservers:

http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

2. If you have a limited set of possible domains that can be known in
advance, then you can define various upstream {} blocks for each
possible domain. This way you can still avoid configuring "resolver"
and enjoy connection pools and etc. For example,

upstream foo.com {
server localhost:8080;
}

http {
location /customers {
set $new_uri '';
set $upstream '';

rewrite_by_lua '
local url = ngx.var.request_uri
local new_uri, n, err = ngx.re.sub(url,
"^/customers/", "/credit-cards/", "jo")
if new_uri then
ngx.var.new_uri = new_uri

ngx.var.upstream = "foo.com"

return
end
if err then
ngx.log(ngx.ERR, "failed to do regex substitution: ", err)
end
ngx.exit(500)
';

proxy_pass http://$upstream$new_uri;

}

# emulated backend service:
location /credit-cards {
content_by_lua '
ngx.say("received raw uri in the backend: ",
ngx.var.request_uri)
';
}
}

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Sat, Sep 6, 2014 at 8:57 PM, Yichun Zhang (agentzh) wrote:
>
> http {

Sorry, typo here. This line should be

server {
listen 8080;

Side note: because I always use the Test::Nginx [1] test scaffold to
test such configuration snippets, the "bridge" between http block
configurations and server block configurations is usually untested and
subject to typos :P Other parts of the configuration are tested :)

Regards,
-agentzh

[1] http://search.cpan.org/perldoc?Test%3A%3ANginx

[RJoshi]

Yes, it was a typo. I am using rewrite_by_lua_file. 

Problem is resolved by using your solution.  Thanks.

   upstream foo.com { 
        server localhost:8080; 
    } 

   ngx.var.upstream = "foo.com" 

proxy_pass http://$upstream$new_uri; 

Thanks,

Regards