[ANN] OpenResty 1.9.3.2 RC1 now available for testing

[Yichun Zhang (agentzh)]

Hi guys,

I've just kicked out OpenResty 1.9.3.2 RC1 for testing:

https://openresty.org/download/ngx_openresty-1.9.3.2rc1.tar.gz

The highlight of this release is the new *_by_lua_block {} directives
added in the ngx_http_lua module.

For example, instead of writing

content_by_lua '
ngx.say("Hello, OpenResty\'s world!\\n")
';

We can just write

content_by_lua_block {
ngx.say("Hello, OpenResty's world!\n")
}

No ugly special character escaping is needed in the Lua source any
more in the latter form :)

Changes since the last (formal) release, 1.9.3.1:

* upgraded LuaNginxModule to 0.9.17.

* feature: implemented "*_by_lua_block {}" directives for all
the existing *_by_lua directives so that we no longer have
to escape special characters while inlining Lua source
inside the "nginx.conf" file.

* feature: initial fixes when being used with the new
"ngx_http_v2" module since nginx 1.9.5. thanks itpp16 for
the patches.

* bugfix: the user specified "./configure"'s "--with-cc-opt"
and "--with-ld-opt" might override the
"LUAJIT_INC"/"LUAJIT_LIB" and "LUA_INC"/"LUA_LIB"
environment settings. thanks Julian Gonggrijp for the
report.

* bugfix: setting builtin request header "Upgrade" via
ngx.req.set_header and etc might not take effect with some
builtin nginx modules.

* bugfix: setting builtin request headers "Depth",
"Destination", "Overwrite", and "Date" via
ngx.req.set_header() and etc might not take effect at least
with ngx_http_dav_module. thanks Igor Clark for the report.

* bugfix: fixed typos due to copy&paste mistakes in some error
messages.

* bugfix: fixed one "-Wmaybe-uninitialized" warning when
compiling with "gcc -Os".

* bugfix: use of shared dicts resulted in (unwanted)
registrations of shared dict metatables on all the
lightuserdata in the Lua space. thanks helloyi for the
report and patch.

* bugfix: if a 3rd-party module calls
"ngx_http_conf_get_module_srv_conf" to fetch its current
"srv_conf" construct in its "merge_srv_conf" callback, then
use of init_worker_by_lua might lead to segmentation faults
(the same also applied to merge_loc_conf). thanks chiyouhen
for the report and patch.

* bugfix: the "if_unmodified_since" "shortcut" field in
"ngx_http_headers_in_t" was first added in nginx 0.9.2.

* bugfix: ngx.req.clear_header/ngx.req.set_header: we did not
update the shortcut fields in "ngx_http_headers_in_t" added
since nginx 1.3.3 which may confuse other nginx modules
accessing them.

* bugfix: setting "Content-Type" response values including ";
charset=xxx" via the ngx.header API might bypass the MIME
type checks in other nginx modules like ngx_gzip. thanks
Andreas Fischer for the report.

* bugfix: typo fixes in some debug logging messages. thanks
doujiang for the patch.

* optimize: fixed the hash-table initial sizes of the cosocket
metatables. thanks ops-dev-cn for the patch.

* tests: removed the useless "use lib" directives from the
Perl test files. thanks Markus Linnala for the report.

* doc: various typo fixes from Lance Li.

* doc: ngx.exit was not disabled within the
header_filter_by_lua* context.

* doc: a code example misses a "return". thanks YuanSheng Wang
for the patch.

* doc: typo fix from Tatsuya Hoshino.

* upgraded LuaUpstreamNginxModule to 0.04.

* feature: "upstream.get_servers(server_name)" now returns the
server name (if any) as well, which can be the domain name
if the user puts it in "nginx.conf". thanks Hung Nguyen for
the request.

* upgraded HeadersMoreNginxModule to 0.27.

* bugfix: setting (builtin) request headers "Upgrade",
"Depth", "Destination", "Overwrite", and "Date" might not
take effect in standard nginx modules like ngx_http_proxy
and ngx_http_dav.

* bugfix: when the response header "Content-Type" contains
parameters like "; charset=utf-8", the "-t MIME-List"
options did not work as expected at all. thanks Joseph
Bartels for the report.

* bugfix: clearing input headers "If-Unmodified-Since",
"If-Match", and "If-None-Match" did not clear the builtin
"shortcut" fields in "ngx_http_headers_in_t" which might
confuse other nginx modules like
"ngx_http_not_modified_filter_module". The first header gets
"shortcuts" fields since nginx 0.9.2 while the latter two
since nginx 1.3.3.

* upgraded IconvNginxModule to 0.12.

* bugfix: HTTP 0.9 requests would turn "iconv_filter" into a
bad unrecoverable state leading to "iconv body filter
skiped" error upon every subsequent request. thanks
numberlife for the report. also introduced some coding style
fixes.

* bugfix: lowered the error log level for HTTP 0.9 requests
from "error" to "warn" to prevent malicious clients from
flooding the error logs.

* upgraded LuaRestyRedisLibrary to 0.21.

* bugfix: the "attempt to call local new_tab (a table value)"
error might happen when LuaJIT 2.0 was used and a local Lua
module named "table.new" was visible. thanks Michael Pirogov
for the report.

* doc: fixed code examples to check redis pipelined requests'
return values more strictly. some commands (like hkeys and
smembers) may return empty tables, which may result in "nil
res[1]" values. thanks Dejiang Zhu for the patch.

* upgraded LuaJIT to v2.1-20151028:
https://github.com/openresty/luajit2/tags

* imported Mike Pall's latest changes:

* limit number of arguments given to "io.lines()" and
"fp:lines()".

* ARM64: fix "__call" metamethod handling for tail calls.

* FFI: Do not propagate qualifiers into subtypes of
complex.

* feature: parse binary number literals ("0bxxx").

* fix NYICF error message.

* properly handle OOM in "trace_save()".

* ARM64: add support for saving bytecode as object files.

* ARM64: fix ELF bytecode saving.

* feature: parse Unicode string escape "\u{XX...}".

* FFI: add "ssize_t" declaration.

* fix unsinking check.

* feature: add "collectgarbage("isrunning")".

* flush symbol tables in "jit.dump" on trace flush.

Feedback welcome!

Hopefully we can cut the 1.9.3.2 formal release in the next few days
so that we can move onto the new nginx 1.9.6 core ASAP :)

Thanks!
-agentzh

[Igor Clark]

Hello agentzh! This is awesome, thanks. The *_by_lua_block directives in particular are going to be super helpful for me, very good for prototyping and trying things out alongside nginx location / config work.

I DL'd and built 1.9.3.2-RC2 to try it out, and while the blocks are working perfectly, I have what I think is another DAV-related issue. When making requests to the DAV module via ngx.location.capture(), MOVE and DELETE requests seem to return proper HTTP codes on failure, but 0 on success.

It's not new to RC2, so I guess I should have reported it with the other DAV issue - but it didn't occur to me that it might be on the lua side rather than the DAV side, or on my side. (Which it still might be!) But it's still happening in RC2, and I can't work out what I'm doing wrong, if I am. So hopefully it's helpful to report.

Here's the config I made to test it out:

user vagrant;

worker_processes  1;

events {

    worker_connections  1024;

}

http {

    server {

        listen 80; 

        default_type text/plain;

        set $file1 /file1.txt;

        set $file2 /file2.txt;

        location /lua-dav/move {

            content_by_lua_block {

                ngx.req.set_header( "Destination", ngx.var.file2 )

                local mv_res = ngx.location.capture(

                    ngx.var.file1, { method = ngx.HTTP_MOVE }

                )

                ngx.say(

                    "MOVE " .. ngx.var.file1 .. " -> " .. ngx.var.file2 ..

                    " response status: " .. mv_res.status

                ) 

                ngx.say( ngx.location.capture( "/" ).body )

            }

        }

        location /lua-dav/delete {

            content_by_lua_block {

                for _,v in ipairs( { ngx.var.file1, ngx.var.file2 } ) do

                    local del_res = ngx.location.capture(

                        v, { method = ngx.HTTP_DELETE }

                    )

                    ngx.say( "DELETE " .. v .. " response status: " .. del_res.status )

                    ngx.say( ngx.location.capture( "/" ).body )

                end

            }

        }

        location / {

            root /home/vagrant/dav_root;

            dav_methods MOVE PUT DELETE;

            create_full_put_path on;

            autoindex on;

            autoindex_format json;

        }

    }

}

Here's how I tested it, and the output I got:

vagrant@debian-7-9:~/dav_root$ ls -l

total 0

vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/move

MOVE response status: 404

[

]

vagrant@debian-7-9:~/dav_root$ touch file1.txt

vagrant@debian-7-9:~/dav_root$ ls -l

total 0

-rw-r--r-- 1 vagrant vagrant 0 Oct 29 22:27 file1.txt

vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/move

MOVE /file1.txt -> /file2.txt response status: 0

[

{ "name":"file2.txt", "type":"file", "mtime":"Thu, 29 Oct 2015 22:27:50 GMT", "size":0 }

]

vagrant@debian-7-9:~/dav_root$ ls -l

total 0

-rw-r--r-- 1 vagrant vagrant 0 Oct 29 22:27 file2.txt

vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/delete

DELETE /file1.txt response status: 404

[

{ "name":"file2.txt", "type":"file", "mtime":"Thu, 29 Oct 2015 22:27:50 GMT", "size":0 }

]

DELETE /file2.txt response status: 0

[

]

vagrant@debian-7-9:~/dav_root$ ls -l

total 0

vagrant@debian-7-9:~/dav_root$ 

Incidentally/separately, I also made this config to try to make lua PUT the file in the first place:

        location /lua-dav/put {

            content_by_lua_block {

                local body_content = "blah blah blah"

                local body_length = string.len( body_content )

                ngx.req.set_header( "Content-type", "text/plain" )

                ngx.req.set_header( "Content-length", body_length )

                local put_res = ngx.location.capture(

                    ngx.var.file1, { method = ngx.HTTP_PUT, body = body_content }

                )

                ngx.say( "PUT response status: " .. put_res.status )

                ngx.say( ngx.location.capture( "/" ).body )

            }

        }

But that reliably returned 500. Is that because the DAV module still can't read the request body as it's in a subrequest, as per https://github.com/openresty/lua-nginx-module/issues/375#issuecomment-46104177 ?

Anyway - hope it's not just me being dim!

Thanks again for openresty, it's amazing.

Cheers

Igor

[Yichun Zhang (agentzh)]

Hello!

On Fri, Oct 30, 2015 at 6:45 AM, Igor Clark wrote:
> Hello agentzh! This is awesome, thanks. The *_by_lua_block directives in
> particular are going to be super helpful for me, very good for prototyping
> and trying things out alongside nginx location / config work.
>

Great! Glad you like it :)

> I DL'd and built 1.9.3.2-RC2 to try it out, and while the blocks are working
> perfectly,

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Yeah, I do understand the frustration with so many little digits in
the version number :)

> I have what I think is another DAV-related issue. When making
> requests to the DAV module via ngx.location.capture(), MOVE and DELETE
> requests seem to return proper HTTP codes on failure, but 0 on success.
>

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Best regards,
-agentzh

[Igor Clark]

Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

 

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

    https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

 

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!

Igor

[Lord Nynex]

I highly recommend busted for testing apps and apis. 

http://olivinelabs.com/busted/

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Igor Clark]

Thanks Lord Nynex, that looks tight for unit testing. I'll have a good look into it. I'm particularly interested in integration tests for API endpoints, which take into account data changes, so unit testing doesn't always seem to fit, but maybe the "async" section will help.

Any other recos for integration testing tools also v welcome :-)

Cheers!

[Lord Nynex]

I'm not sure what the actual limitation here is. If you want to use Test::Nginx::Socket to write integration tests, go for it it's your software, but what exactly do you think you can't do busted?

Here is a framework specific example of using busted for both unit testing and integration testing with their use_test_server and request http://leafo.net/lapis/reference/testing.html#using-the-test-server

Personally if I was doing integration testing (i am) to look for 'data changes' (i am) I would use busted (again, i am) because I think lua helpers are really cool and it's easy to cjson decode a json response and validate the table as opposed to messing with JSON::XS. Further, the distinction between 'unit testing' and 'integration test' kina blurs into just testing at that point. 

[Igor Clark]

That's super helpful, thanks. Honestly I think the limitation is probably my (comparative) Lua & (absolute) Busted ignorance. What you describe sounds like what I want. I'll spend some time digging into it. Appreciate it!

You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/4dW-3dNM_mM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.

[Yichun Zhang (agentzh)]

Hi Igor

On Fri, Oct 30, 2015 at 6:18 PM, Igor Clark wrote:
>
> Excellent, tested and working here too, thanks very much :-)
>

Thanks for your feedback.

> Great. Really helpful to see the simple tests integrated like that too,
> helps me understand what Test::Nginx::Socket is up to. Which seems really
> cool.

Glad you like it.

> Does it make sense to use that testing framework for web apps/APIs
> written with OpenResty, or is it only really meant for testing the
> bundle/app server itself?

It's good for both. For example, the Lua CDN and Lua SSL's business
systems at CloudFlare use Test::Nginx::Socket at the bottom to drive
their test suite, in a similar declarative fashion, enjoying all the
advanced testing features of Test::Nginx::Socket (memory leak testing,
hup reload mode, mockeagain testing, valgrind checks, systemtap
integration, you name it) out of the box.

The idea of organizing test cases as simple data and spec rather than
hard-coded code is very powerful. We also have tools to make the tests
update themselves easily and automatically at our will.

For higher level testing, you may want to extend the DSL offered by
Test::Nginx::Socket by subclassing it. You can invent and de-sugar
your own "sections" down to existing "sections" or implement your new
fancy sections (like those manipulating external resources, like
memcached, dns servers, and databases) yourself. For example, the test
scaffold driving ngx_stream_echo module's test suite is using this
approach:

https://github.com/openresty/stream-echo-nginx-module/tree/master/t/

Simple and powerful.

> Is there a preferred method for doing integration
> tests on apps/APIs if so?
>

Personally I'd recommend my own Test::Nginx approach :) But I also
encourage you investigate other solutions to pick up the one you like
most.

Oh, I surely need to write more detailed tutorials on
Test::Nginx::Socket to demonstrate its power quickly accumulated over
the years (since 2009).

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Fri, Oct 30, 2015 at 6:45 AM, Igor Clark wrote:

> Incidentally/separately, I also made this config to try to make lua PUT the
> file in the first place:
>

[...]

>
>
> But that reliably returned 500. Is that because the DAV module still can't
> read the request body as it's in a subrequest, as per
> https://github.com/openresty/lua-nginx-module/issues/375#issuecomment-46104177
> ?
>

The ngx_dav module assumes that it reads the client request body
itself and it requires the body to be always put on the file system
during the read. So if we fake this in ngx.location.capture, then we
probably shouldn't use ngx_dav's PUT method in the first place (since
there's no gain here, really, as compared to writing to the file
directly ourselves, in Lua).

Regards,
-agentzh

[Igor Clark]

Thanks agentzh, very helpful to know it's not only used for the underlying module development. The declarative approach definitely appeals.

Cheers!

[Igor Clark]

Hi!

On Monday, November 2, 2015 at 2:48:05 AM UTC, agentzh wrote:

 

The ngx_dav module assumes that it reads the client request body
itself and it requires the body to be always put on the file system
during the read. So if we fake this in ngx.location.capture, then we
probably shouldn't use ngx_dav's PUT method in the first place (since
there's no gain here, really, as compared to writing to the file
directly ourselves, in Lua).

 

Ah, OK, got it. I've organised it so clients PUT directly to ngx_dav anyway, and with this info it obviously makes sense just to leave it that way.

Thanks :-)