Re: Need help for Nginx lua gzip response.

[Yichun Zhang (agentzh)]

Hello!

On Mon, Oct 14, 2013 at 6:51 AM, Bhargav Trivedi wrote:
> I am using Nginx as reverse proxy to apache, apache sends gzip response. I
> am trying to decompress this response using lua-zlib and then modify the
> response content. After that I am compressing the response again using
> lua-zlib but some how I could not get the complete HTML content in the
> browser.
>
> Below lua code is used with body_filter_by_lua_file ,
>
> if (string.match(ngx.var.sent_http_content_type, 'text/html') ~= nil) then
> local zlib = require("zlib")
> local stream = zlib.inflate()
> ngx.arg[1] = stream(ngx.arg[1])
> ngx.arg[1] = string.gsub(ngx.arg[1], "(\r?\n)%s*\r?\n", "%1")
> ngx.arg[1] = string.gsub(ngx.arg[1], "(\r?\n)%s*", "%1")
> local deflate_stream = zlib.deflate()
> ngx.arg[1] = deflate_stream(ngx.arg[1], 'full')
> end
>
> I am not getting where do I make mistake because of that some part of the
> page (footer part) is missing in response.
>

Could you please check if your backend server sends a Content-Length header?

One common mistake is that the user forgets to clear the
Content-Length response header in header_filter_by_lua when he
actually changes the length of the response in body_filter_by_lua. To
quote the official documentation:

“When the Lua code may change the length of the response body, then it
is required to always clear out the Content-Length response header (if
any) in a header filter to enforce streaming output, as in

location /foo {
# fastcgi_pass/proxy_pass/...

header_filter_by_lua 'ngx.header.content_length = nil';
body_filter_by_lua 'ngx.arg[1] = string.len(ngx.arg[1]) .. "\\n"';
}
”

I'm cc'ing the openresty-en mailing list:
https://groups.google.com/group/openresty-en And you're highly
recommended to post such questions there in the future :)

Thanks!
-agentzh

[bha...@aum.bz]

Hi,

I tried with clearing Content-Length response header but it could not help.

Thanks,
Bhargav

[Yichun Zhang (agentzh)]

Hello!

On Tue, Oct 15, 2013 at 5:50 AM, bhargav wrote:
> I tried with clearing Content-Length response header but it could not help.
>

I've had a closer look at your Lua code and it seems that your way of
decompressing the data stream is just wrong. You initialize a new zlib
stream instance for *every* data chunk in the response body but you
should really really initialize the stream only once for each response
and process each data chunk in the response body stream with exactly
the same zlib stream instance. decompressing a data stream is a
stateful process anyway.

Be aware that the body_filter_by_lua* code operates on data chunks, so
for a big enough response data stream, your Lua code will run multiple
times, once for each data chunk. This is exactly an example of
streaming processing.

You can save your current zlib stream instance in ngx.ctx so that all
your later invocations of body_filter_by_lua on the same response data
stream can share the same decompressing state.

BTW, assigning to ngx.arg[1] is expensive. You should use a local Lua
variable to hold the intermediate string processing results and only
assign the final value to ngx.arg[1].

Best regards,
-agentzh

[bha...@aum.bz]

Hi,

Thanks for sharing your Nginx knowledge with us and I really appreciate your help for this issue.

As you suggested I tried with initializing zlib stream with ngx.ctx in access_by_lua_file and used it in body_filter_by_lua* code.

After making this change, I am getting this error in log.
failed to run body_filter_by_lua*: IllegalState: calling inflate function when stream was previously closed
stack traceback:
        [C]: in function 'stream'


I have also made the change in code such that it uses local variable for string processing and final result will be assigned to ngx.arg[1] .

Thanks,
Bhargav

[benjamin pottier]

Are you calling stream() with no args? Check https://github.com/brimworks/lua-zlib/commit/a22bef6682c2ee0844c94762a0b440956aef821c and the linking issues.

[bha...@aum.bz]

Here is the code which I have used

local inflate_body = ngx.ctx.inflate_stream(ngx.arg[1], "finish")
inflate_body = /*Modify inflate_body*/
local deflate_stream = ngx.ctx.zlib.deflate(6,31)
ngx.arg[1] = deflate_stream(inflate_body, "finish")

[Yichun Zhang (agentzh)]

Hello

On Wed, Oct 16, 2013 at 1:40 AM, <bha...@aum.bz> wrote:
> As you suggested I tried with initializing zlib stream with ngx.ctx in
> access_by_lua_file and used it in body_filter_by_lua* code.
>
> After making this change, I am getting this error in log.
> failed to run body_filter_by_lua*: IllegalState: calling inflate function
> when stream was previously closed
> stack traceback:
> [C]: in function 'stream'
>

Please provide your Lua code. It seems that you don't get the lifetime
of your zlib object right.

Regards,
-agentzh

[bha...@aum.bz]

Hi,

We've used access_by_lua_file  to initialize

ngx.ctx.zlib = require("zlib")
ngx.ctx.inflate_stream = ngx.ctx.zlib.inflate()

And after that we have used body_filter_by_lua_file with below code

local inflate_body = ngx.ctx.inflate_stream(ngx.arg[1], "finish")

inflate_body = string.gsub(inflate_body, "SEARCH", "REPLACE")

local deflate_stream = ngx.ctx.zlib.deflate(6,31)
ngx.arg[1] = deflate_stream(inflate_body, "finish")

Thanks,
Bhargav

[Yichun Zhang (agentzh)]

Hello!

On Wed, Oct 16, 2013 at 8:28 PM, <bha...@aum.bz> wrote:
> We've used access_by_lua_file to initialize
>
> ngx.ctx.zlib = require("zlib")
> ngx.ctx.inflate_stream = ngx.ctx.zlib.inflate()
>

You don't have to do the initialization in the access phase. You're
just holding the related resources (mostly memory) for longer time
than needed.

You can initialize in header_filter_by_lua, and only when the backend
(Apache in your case) returns the "Content-Encoding: gzip" response
header.

> And after that we have used body_filter_by_lua_file with below code
>
> local inflate_body = ngx.ctx.inflate_stream(ngx.arg[1], "finish")

"finish" argument for inflate_stream is a no-op. It only accepts 1
single argument. Please refer to lua-zlib's documentation.

> inflate_body = string.gsub(inflate_body, "SEARCH", "REPLACE")
>
> local deflate_stream = ngx.ctx.zlib.deflate(6,31)
> ngx.arg[1] = deflate_stream(inflate_body, "finish")
>

It's totally wrong to pass the "finish" argument (i.e., setting
Z_FINISH) to deflate_stream for every data chunk. You should only do
that when the current chunk is the last chunk in the response body
stream (indicated by ngx.arg[2]).

No wonder you're getting the IllegalState error. This error is
actually documented in lua-zlib's README.

I suggest you take a little more time reading the documentation of the
software you're using, including ngx_lua's and lua-zlib's. It will
save you a lot of time and troubles :)

BTW, you don't really need to do the gzip compression in
body_filter_by_lua yourself. You can just let the standard ngx_gzip
module do the compression work for you because ngx_gzip runs after
body_filter_by_lua (and ngx_gzip is more efficient than your Lua
code).

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

I missed another two mistakes in your Lua code. Here we go:

On Wed, Oct 16, 2013 at 8:28 PM, <bha...@aum.bz> wrote:

> We've used access_by_lua_file to initialize
>
> ngx.ctx.zlib = require("zlib")
> ngx.ctx.inflate_stream = ngx.ctx.zlib.inflate()
>

Forgot to mention another two mistake:

You should not save the zlib module table into ngx.ctx in the first
place. It's useless. The streaming processing state is actually
stored in the upvalues of the closures generated by zlib.inflate() and
zlib.deflate(). So you should save the closures into ngx.ctx instead.

>
> local deflate_stream = ngx.ctx.zlib.deflate(6,31)

This line is totally wrong too. You should have saved the return value
of zlib.deflate(), i.e., the deflate closure, into ngx.ctx. Here
you're just allocating brand new stream object at *every* data chunk,
which defeat the purpose of streaming processing.

Have you noticed that your way of handling deflate is very different
from your way of handling inflate? The former is just wrong.

But I've already pointed out (in my previous mail) that you should
really use ngx_gzip do the compression here for you. You don't need to
call deflate yourself.

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Wed, Oct 16, 2013 at 11:34 PM, Yichun Zhang (agentzh) wrote:
>
> You don't have to do the initialization in the access phase. You're
> just holding the related resources (mostly memory) for longer time
> than needed.
>
> You can initialize in header_filter_by_lua, and only when the backend
> (Apache in your case) returns the "Content-Encoding: gzip" response
> header.
>

Just for the reference, the following self-contained example for doing
gzip inflate with lua-zlib in body_filter_by_lua has been tested on my
side:

lua_package_cpath "/path/to/lua-zlib/?.so;;";
gzip on;
gzip_min_length 1;

server {
listen 8080;

location = /t {
proxy_http_version 1.1;
proxy_pass http://127.0.0.1:$server_port/apache;

header_filter_by_lua '
if ngx.header.content_encoding == "gzip" then
local zlib = require "zlib"
ngx.ctx.inflate = zlib.inflate()
ngx.header.content_length = nil
ngx.header.content_encoding = nil
end
';

body_filter_by_lua '
local inflate = ngx.ctx.inflate
if not inflate then
return
end
local s = ngx.arg[1]

if s ~= "" then
local inflated, eof = inflate(s)
print("inflated: [", inflated, "]") -- for debugging
if inflated ~= "" then
inflated = string.gsub(inflated, "\\n", "")
local new = "{" .. string.upper(inflated)
.. "}\\n"
ngx.arg[1] = new
else
ngx.arg[1] = nil
end
end
';
}

# for mocking the apache backend
location = /apache {
default_type text/html;
content_by_lua '
for i = 1, 10 do
ngx.say(i .. "hello world" .. i)
ngx.flush()
ngx.sleep(0.2)
end
';
}
} # server

Here we use ngx_lua to mock the Apache backend service, /apache, in
the same Nginx server instance. The main entry point is /t, where it
uses proxy_pass to talk to /apache over HTTP 1.1 and do the gzip
decompression and data modification in place in body_filter_by_lua.

Here we use nginx's standard ngx_gzip module to do the gzip
(re)compression for us right after every time body_filter_by_lua is
run.

Here is the test result using curl:

$ curl -i --compressed localhost:8080/t
HTTP/1.1 200 OK
Server: nginx/1.4.2
Date: Thu, 17 Oct 2013 22:46:03 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Content-Encoding: gzip

{1HELLO WORLD1}
{2HELLO WORLD2}
{3HELLO WORLD3}
{4HELLO WORLD4}
{5HELLO WORLD5}
{6HELLO WORLD6}
{7HELLO WORLD7}
{8HELLO WORLD8}
{9HELLO WORLD9}
{10HELLO WORLD10}

Note the "Content-Encoding: gzip" response header, which indicates
that the final response is indeed gzip compressed.

And then, we can check nginx's error.log file for the debugging logs
genereated by the print() function in our body_filter_by_lua (the
output is edited a bit to save some space here):

$ grep inflated logs/error.log
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [1hello world1
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [2hello world2
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [3hello world3
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [4hello world4
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [5hello world5
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [6hello world6
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [7hello world7
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [8hello world8
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [9hello world9
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10:
inflated: [10hello world10
[notice] 11550#0: *4 [lua] [string "body_filter_by_lua"]:10: inflated: []

Please note how the response body data from /apache is inflated by our
body_filter_by_lua chunk by chunk.

Best regards,
-agentzh

[bha...@aum.bz]

Hi,

Thanks to review my lua code and providing the example code.

I am new with nginx-lua so it will take some time for me to understand. But I really appreciate your knowledge and help for this issue.

After checking your example, now I got clear picture where do I mistake and it seems that chunk processing is the main cause of the issues here. Actually trying multiple things together could be creating the trouble in solving the issue for me :).

I was using ngx_gzip initially but to resolve the issue I gave up a try with zlib deflate.

Best Regards,
Bhargav