Re: Scheduling lua tasks in Nginx and installing modules

[dat...@openresty.com]

Hello!

You can use https://github.com/openresty/lua-nginx-module#ngxtimerevery to create recurring tasks in ngx_lua. Based on your use case you will most likely want to setup the timer in the init_worker phase.

You can replace your rewrite directive using rewrite_by_lua and proxy_pass to arbitrary URI using variables inside rewrite_by_lua. There is no need to "generate rewrite rules" in this case.

On Thursday, November 9, 2017 at 12:18:36 PM UTC-8, Brandon wrote:

Hello, I am new to both nginx and lua, but have some questions and am

trying to get orientated 

I have been using these two sources for my documentation so far, but I'm
still having some trouble.
https://github.com/openresty/lua-nginx-module#ngxredirect
https://openresty-reference.readthedocs.io/en/latest/Directives/#rewrite_by_lua
and

I have a use case where I want to be able to define how nginx rewrites a
URL from the content of a file stored somewhere, but I'm not sure how to
actually program this or properly work with Lua in an embedded environment.

Right now I have clients that pull down assets with a curl command like
this. Internally Nginx rewrites /prod/linux/ to /12345/ and that works.

    location / {
      rewrite /prod/linux/(.*) /12345/$1 permanent;
      proxy_pass https://bucket.s3.amazonaws.com;

    }

curl -L https://me.com:8443/prod/linux/deploy.sh > deploy.sh

I have a lot of these nginx servers that may be serving different
content to different types of clients and I want to be maintaining this
mapping of client type to asset version, the /12345/ I rewrite to, in a file. 

I was thinking of the mapping being formatted like this:

gce-2017-10-10,12345
azure-2017-10-10,12345
gce-2017-10-20,54321
azure-2017-10-20,98765

I want to keep a file defining this mapping on S3 and have nginx pull it
down, read it and generate appropriate rewrite rules every hour. I see
that there is a module for helping with s3 access. How can I install
such modules in the embedded lua-nginx environment?
https://github.com/gcr/lua-s3

How can I have a job like this running internal to Nginx on a regular
schedule without being in response to some access event?

Am I being an idiot and doing this wrong entirely?

I appreciate any direction someone may have 

[Yichun Zhang (agentzh)]

Hello!

On Fri, Nov 10, 2017 at 9:29 PM, <bra...@rabb.it> wrote:
> --broken code from below in init_by_lua
>
> init_by_lua error:
> ...nstributor/include.d/lua-resty-s3/libs/resty/s3_auth.lua:14: no request
> object found
>

init_by_lua* runs in the nginx master process while cosockets cannot
run in that context (yet).

>
> --broken code from below in init_worker_by_lua
> Nov 11 05:14:16 server.dockerd[1968]: 2017/11/11 05:14:16 [alert] 1#1:
> cache manager process 50 exited on signal 11 (core dumped)
> Nov 11 05:14:16 server.dockerd[1968]: 2017/11/11 05:14:16 [alert] 1#1:
> worker process 51 exited on signal 11 (core dumped)
>

I need the full backtrace from the gdb command "bt full" when
inspecting these core dumps. See

https://www.nginx.com/resources/admin-guide/debug/#backtrace

for more details.

BTW, please make sure you use the latest OpenResty release. See

https://openresty.org/en/download.html

>
>
> --broken code
>
> init_worker_by_lua_block {
>
> local cjson = require("cjson")
> local awss3 = require "resty.s3"
>
>
> local s3 = awss3:new("key", "secret", "bunstributor", {timeout=1000*10})
> }

You need to run the cosocket code in a timer handler (created by
ngx.timer.at or ngx.timer.every). The cosocket API is disabled in the
init_worker_by_lua* itself as well (just like init_by_lua*). But it's
available in the timer handler.

Please read the official documentation more carefully.

Regards,
Yichun

[Robert Paprocki]

This looks like the segfault is coming from a third party module (ngx_aws_auth). Best to contact the author of the module to have them take a look. 

On Nov 12, 2017, at 12:27, bra...@rabb.it wrote:

I see that you've previously mentioned that ngx.timer.at is not supported in init_by_lua. That leaves me to using the init_worker_by_lua block.  

https://github.com/openresty/lua-nginx-module/issues/774

This is where I am getting the core dump problem.

init_worker_by_lua '
  local function hi(premature)
    ngx.log(ngx.ERR, "hi")
  end   ngx.timer.at(0, hi)
';



/usr/local/openresty/bin/openresty -V
nginx version: openresty/1.11.2.5
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)
built with OpenSSL 1.0.2k  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.31 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.06 --add-module=../srcache-nginx-module-0.31--add-module=../ngx_lua-0.10.10 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.32 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.18 --add-module=../redis2-nginx-module-0.14 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.14 --add-module=../rds-csv-nginx-module-0.07 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --with-openssl=/tmp/openssl-1.0.2k --with-pcre=/tmp/pcre-8.40 --add-module=/tmp/openresty-1.11.2.5/ngx_aws_auth --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-pcre-jit --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads



GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/openresty/bin/openresty...done.


warning: core file may not match specified executable file.
[New LWP 78]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: master process /usr/local/openresty/bin/openresty -c /ngin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_http_add_variable (cf=cf@entry=0x7ffe5328a9a0,
    name=name@entry=0x9ec160 <ngx_http_aws_auth_vars>, flags=1)
    at src/http/ngx_http_variables.c:387
387 src/http/ngx_http_variables.c: No such file or directory.
(gdb)

On Saturday, November 11, 2017 at 7:39:48 PM UTC-8, bra...@rabb.it wrote:

I'm terribly sorry for having such dumb questions here.

The code below is almost verbatim copy and pasted from the nginx.timer.at documentation https://github.com/openresty/lua-nginx-module#ngxtimerat

Even in the context of ngx.timer.at I am seeing that I cannot issue requests. Is this what is meant as the work around explained here?
https://github.com/openresty/lua-nginx-module#cosockets-not-available-everywhere

 [alert] 1#1: lua_code_cache is off; this will hurt performance in /nginx.conf:106
Nov 12 03:36:03 dockerd[1968]: nginx: [alert] lua_code_cache is off; this will hurt performance in /nginx.conf:106
Nov 12 03:36:03 dockerd[1968]: 2017/11/12 03:36:03 [error] 1#1: init_by_lua error: init_by_lua:23: no request
Nov 12 03:36:03 dockerd[1968]: stack traceback:
Nov 12 03:36:03 dockerd[1968]:         [C]: in function 'at'
Nov 12 03:36:03 dockerd[1968]:         init_by_lua:23: in main chunk
Nov 12 03:36:03 dockerd[1968]: nginx: [error] init_by_lua error: init_by_lua:23: no request
Nov 12 03:36:03 dockerd[1968]: stack traceback:
Nov 12 03:36:03 dockerd[1968]:         [C]: in function 'at'
Nov 12 03:36:03 dockerd[1968]:         init_by_lua:23: in main chunk

 

http {
  lua_socket_pool_size 100;
  lua_socket_connect_timeout 10ms;
  lua_socket_read_timeout 10ms;
  lua_package_path '/data/bunstributor/include.d/lua-resty-s3/libs/?.lua;;';


  lua_shared_dict bunstributor_mapping 10m;


  init_by_lua_block {

    local cjson = require("cjson")

    local awss3 = require "resty.s3"

    local delay = 5

    local handler

    handler = function (premature)

     -- do some routine job in Lua just like a cron job

      local s3 = awss3:new(key, secret, "bunstributor", {timeout=1000*10})

      local ok, response = s3:get("/test1/bunstributor_mapping.txt")

      local succ, err, forcible = bunstributor_mapping.set(bunstributor_mapping, cjson.decode(response))

      if premature then

        return

      end

      local ok, err = ngx.timer.at(delay, handler)

      if not ok then

        ngx.log(ngx.ERR, "failed to create the timer: ", err)

        return

      end

    end

    local ok, err = ngx.timer.at(delay, handler)

    if not ok then

      ngx.log(ngx.ERR, "failed to create the timer: ", err)

      return

    end

  }
}

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.