declaring nginx variables for ngx.location.capture_multi subrequests

[Gavin Rolleston]

Hey! 

I need to take an incoming request and send it up to multiple upstreams at once. I've sorted out a way to do this using ngx.location.capture_multi and nginx variables, but I don't really like that I'm using a dummy location to declare the $pool variable. The code below effectively captures what I'm trying to do (I've omitted parts that aren't relevant to this discussion, and IRL I'm building the subrequests and unpacking the responses dynamically to accommodate several pools).

server {

  upstream pool_0 {

    server "127.0.0.1:26157;

  }

  upstream pool_1 {

    server "127.0.0.1:26158;

  }

  rewrite_by_lua '

    if ngx.req.get_headers()["X-Flag"] then

      -- this is a subrequest

      if ngx.var.pool == 0 then

        target_upstream = pool_0

      elseif ngx.var.pool == 1 then

        target_upstream = pool_1

      end

    else

      -- this is the original request

      ngx.req.set_header("X-Flag", 1)

      ngx.req.read_body()

      resps = { ngx.location.capture_multi( {{ngx.var.request_uri, vars = {pool=x}}, {ngx.var.request_uri, {pool=y}} }) }

      -- unpack responses, find the successful one and send it back to client

    end

  ';

  location / {

    set $target_upstream foo;

    proxy_pass http://$target_upstream

  }

  location /dummy_for_variable_declaration {

    internal;

    set $pool -1;

  }

Note that the above code works because nginx declares variables when it encounters a "set" while loading the config file. However, the subrequest doesn't enter the /dummy_for_variable_declaration location, so its "set" directive doesn't get executed. If I declare $pool anywhere else, then the value I assign to $pool for the subrequest gets overwritten in the subrequest rewrite phase.

Is there another/better way to do this? Again, I don't like that I'm using this dummy location.

Thanks!

-Gavin

[Yichun Zhang (agentzh)]

Hello!

On Wed, Apr 1, 2015 at 1:32 PM, Gavin Rolleston wrote:
> Note that the above code works because nginx declares variables when it
> encounters a "set" while loading the config file. However, the subrequest
> doesn't enter the /dummy_for_variable_declaration location, so its "set"
> directive doesn't get executed. If I declare $pool anywhere else, then the
> value I assign to $pool for the subrequest gets overwritten in the
> subrequest rewrite phase.

I'm not sure if you really want to put rewrite_by_lua in the server {}
block level because that way all the containing location {} blocks
will inherit it by default and both your main request and all the
subrequests will run this rewrite_by_lua in their own lifetime. Better
put your rewrite_by_lua in a separate rewrite.conf file and then only
"include" it in locations you really need.

Regards,
-agentzh

[Gavin Rolleston]

Thanks for the words of caution agentz! However, we actually need

rewrite_by_lua to run on both the main request and the subrequest. We

have other use cases that depend on it, for example, to insert headers

and to route to upstreams. Same for rewrite_by_lua being inherited by

all location {} blocks.

Warm regards,

-Gavin

[Yichun Zhang (agentzh)]

Hello!

On Thu, Apr 2, 2015 at 1:56 PM, Gavin Rolleston wrote:
> Thanks for the words of caution agentz! However, we actually need
> rewrite_by_lua to run on both the main request and the subrequest. We
> have other use cases that depend on it, for example, to insert headers
> and to route to upstreams. Same for rewrite_by_lua being inherited by
> all location {} blocks.
>

Okay, then I'd suggest you use the set_if_empty directive provided by
ngx_set_misc (which is also enabled in OpenResty by default):

https://github.com/openresty/set-misc-nginx-module#set_if_empty

Just write

set_if_empty $pool -1;

on your server {} level or above.

Well, you may also want to check out other directives in ngx_set_misc.
They're handy :)

FWIW, I'm considering changing the running phase of rewrite_by_lua*
configured directly on the server {} level or above. Eventually I'd
like it to run in the server-rewrite phase just like the standard
"rewrite" directive. But even then, you'll still need the set_if_empty
trick because subrequests run from the server-rewrite phase as well.

Regards,
-agentzh

[Gavin Rolleston]

Hello Agentzh!

> FWIW, I'm considering changing the running phase of rewrite_by_lua* 

> configured directly on the server {} level or above. Eventually I'd 

> like it to run in the server-rewrite phase just like the standard 

> "rewrite" directive

We sometimes use empty rewrite_by_lua directives in the location {} level to prevent server {} level rewrite_by_lua directives from running for that specific location.

Would running server-level rewrite_by_lua directives in the server-rewrite phase cause this technique to no longer work? I'm concerned that with this proposed/potential change both the location {} level rewrite_by_lua and the server {} level rewrite_by_lua directives would run for a request.

Thanks,

-Gavin 

[Yichun Zhang (agentzh)]

Hello!

On Thu, Jul 30, 2015 at 7:23 AM, Gavin Rolleston wrote:
> We sometimes use empty rewrite_by_lua directives in the location {} level to
> prevent server {} level rewrite_by_lua directives from running for that
> specific location.
>

Yes, that's a common technique.

> Would running server-level rewrite_by_lua directives in the server-rewrite
> phase cause this technique to no longer work?

Correct. server-rewrite phase always runs *before* the request binds
any locations. So your location-wise configurations won't affect it.
This is similar to the standard "rewrite" directive in the nginx core.

Given that so many people are already relying on the existing
behaviour, I may introduce another server_rewrite_by_lua directive to
avoid breaking backward compatibility.

Regards,
-agentzh