passing the request body transparently to ngx.location.capture

[avi...@adallom.com]

Hello,

Currently, the only API to pass a request body to the ngx.location.capture is to pass the entire request body as a Lua string in the options dictionary.

This means that if the main request needs to pass its own request body to the subrqeuest, it should first read the entire request body to the memory (using the various API the Lua module offers), and then pass it as an argument for ngx.location.catpure.

I want to write a patch that enables the main request to pass its body to the subrequest transparently, meaning without reading it first.

The API should be rather simple, first of all, it means that the main request shouldn't read the body at all and that the lua_need_request_body directive should be set to "off". Then, the API would look something like that:

result = ngx.location.capture("subrequest_location", {parent_request_body = true}).

Does anyone have any suggestions regarding how such a patch can be implemented?

I started off by not setting the subrequest's headers_in.content_len to zero when no body is given if the "parent_request_body" parameter is set to True), but setting it to the correct content length, but this works partially (meaning that only some of the request body is read or none at all).

Best Regards

[Yichun Zhang (agentzh)]

Hello!

On Thu, Jul 25, 2013 at 4:35 PM, <avi...@adallom.com> wrote:
> Hello,
>
> Currently, the only API to pass a request body to the ngx.location.capture
> is to pass the entire request body as a Lua string in the options
> dictionary.
>
> This means that if the main request needs to pass its own request body to
> the subrqeuest, it should first read the entire request body to the memory
> (using the various API the Lua module offers), and then pass it as an
> argument for ngx.location.catpure.
>

No, that's not the case. To quote the official documentation for
ngx.location.capture:

"When the body option is not specified, the POST and PUT subrequests
will inherit the request bodies of the parent request (if any)."

Please see http://wiki.nginx.org/HttpLuaModule#ngx.location.capture
for more details :)

> I want to write a patch that enables the main request to pass its body to
> the subrequest transparently, meaning without reading it first.

No, it's required that only the main request reads the request body.
One should always ensure that no subrequest is trying to *read* the
request body because the behavior is undefined. IIRC, recent Nginx
core explicitly checks this case.

It's fine that we don't read the request body into the Lua land in
your main request's Lua handler. But we should never never let the
subrequest actually read the request body from the system socket
receive buffers.

> The API should be rather simple, first of all, it means that the main
> request shouldn't read the body at all and that the lua_need_request_body
> directive should be set to "off". Then, the API would look something like
> that:
>
> result = ngx.location.capture("subrequest_location", {parent_request_body =
> true}).
>
> Does anyone have any suggestions regarding how such a patch can be
> implemented?

I'd rather accept a new option that makes the subrequest inherit its
parent's already-read request bodies even if the request method is not
POST nor PUT.

Best regards,
-agentzh

[avi...@adallom.com]

Well, I guess that the problem isn't with the Lua module, but with Nginx; for cases in which the request body is very big, it must be read entirely into the memory/disk before handling it (which makes the reverse proxy be significantly slower for file uploads, for example).

Once they solve this issue, I think they should solve it for subrequests as well, as this issue is very similar to the reverse proxy issue.

[Yichun Zhang (agentzh)]

Hello!

On Mon, Jul 29, 2013 at 1:04 AM, aviram wrote:
> Well, I guess that the problem isn't with the Lua module, but with Nginx;
> for cases in which the request body is very big, it must be read entirely
> into the memory/disk before handling it (which makes the reverse proxy be
> significantly slower for file uploads, for example).
> Once they solve this issue, I think they should solve it for subrequests as
> well, as this issue is very similar to the reverse proxy issue.
>

Yes, Nginx's built-in request body reader always requires full
buffering (either in RAM or on disk).

And that was why I introduced an alternative request body reader in
ngx_lua that supports non-buffered reading:

http://wiki.nginx.org/HttpLuaModule#ngx.req.socket

Regards,
-agentzh