Nginx+Lua

[Alexander Litvak]

Hello!

I was trying to get to know Openresty, as it would possibly be required on my job, but ran into a problem. I need to be able to lookup text value from memcached(by the domain) and, if not found, retrieve it from specified location(url). Then I want to replace some text in the page(the response body) with the said string from memcached or downloaded. 

I have two scripts in lua, one I call from content_by_lua and other from the body_filter_by_lua in nginx.conf file within the same location. I also use nginx variable to pass a string from one script to another. My problem is: I can't use memcached inside of body_filter_by_lua and I can't use ngx.arg API in the content_by_lua, but when I use both the response I get is just an empty page. 

Logs are all clean.

Found no info on the issue, hope there is some kind of workaround!

Thanks in advance!

[Robert Paprocki]

Can you paste your code and minimal config to show a reproducible example of your problem?

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[al...@chameleonx.com]

On Saturday, March 26, 2016 at 10:16:59 PM UTC+3, rpaprocki wrote:

Can you paste your code and minimal config to show a reproducible example of your problem?

Absolutely!

Nginx.conf part: 

        location / {
            root   html;
            index  index.html index.htm;
         set $replaceContent '';
    content_by_lua_file /home/cxadmin/work/fromMemcached.lua;
    body_filter_by_lua_file /home/cxadmin/work/fromUrlReplace.lua;
        }

Everything else in .conf stayed default.

fromMemcached.lua :

-- Load Memcached module

local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
-- Instantiate Memcached object
local memc, err = memcached:new()
if not memc then
    ngx.log(ngx.ERR, "Failed to instantiate memc")
else
ngx.log(ngx.ERR, "memc successfully instantiated")
-- Set Memcached timeout
memc:set_timeout(1000)
-- Connect to Local Memcached
local ok, err = memc:connect("10.0.0.4", 11211)
if not ok then
   ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ERR, "Connected to local memcached")
-- Try to retrieve from Memcached
fromMemc, flags, err = memc:get(memcachedKey)
if err then
   ngx.log(ngx.NOTICE, "Not found in Local Memcached: ", err)
end
if not fromMemc then
memc:close()
ngx.log(ngx.ERR, "memc closed")
return
end
ngx.log(ngx.NOTICE, "Value found and passed to $replaceContent", fromMemc)
ngx.var.replaceContent = fromMemc
memc:close()
ngx.log(ngx.ERR, "memc closed")
end
end

fromUrlReplace.lua:

if not ngx.var.replaceContent then
ngx.log(ngx.NOTICE, "Retrieveing sript from URL")
-- Key will later be more complex
local http = require("socket.http")
local ltn12 = require("ltn12")
-- Request from URL
local t = {}
http.request {
url = "https://s3-us-west-1.amazonaws.com/cxeasylistcust/luatest.txt",
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
dtext = table.concat(t)
replaceString = dtext .. "</head>"
else
ngx.log(ngx.NOTICE, "Using content from Memcached")
replaceString = ngx.var.replaceContent .. "</head>"
end
ngx.log(ngx.NOTICE, "replace string: ", replaceString)
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replaceString)

Everything is pretty much straight forward. fromMemcached.lua looks the value up in the memcached, fromUrlReplace.lua tries to download the data from a url if it wasn't found in memcached and injects it into the response body.

[al...@chameleonx.com]

Also I should probably mention I work on Ubuntu 14.04, just in case it matters 

[Itamar Gilad]

You could fetch the data in the content_by_lua / access_by_lua stages and store it in the request context via the ngx.ctx.* table, and use it later on when needed. 

Also, instead of making the replacements in the response body in lua-land using *_filter_by_lua*, I suggest checking out the excellent replace-filter nginx module, which uses the sregex library (both by agentzh).

In case you need to use the matching result groups in your replacements, you may want to do that by setting an ngx.var.* in lua (since passing dollar signs in the nginx config file is no fun).

Best of luck!

-Itamar

--

[Alexander Litvak]

Thanks for the advice!

For some reason I now only get empty page in response even with replace_filter marked out.

However adding ngx.say("Something") at the bottom of the script does return Something in response...

It seems I'm touching something I don't suppose to. Do I?

Now the .conf portion looks like this:

location / {
resolver 8.8.8.8;

            root   html;
            index  index.html index.htm;

   content_by_lua_file '/home/cxadmin/Work/filter.lua';
   replace_filter '</head>' 'ngx.ctx.replaceScript' ig;
        }

filter.lua:

local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host

ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
local isScriptFromMemcached = false

memc, err = memcached:new()

-- Try to retrieve script from Memcached

if not memc then
    ngx.log(ngx.ERR, "Failed to instantiate memc: ")
else

ngx.log(ngx.ALERT, "memc successfully instantiated")
memc:set_timeout(1000)
local ok, err = memc:connect("localhost", 11211)
if not ok then
   ngx.log(ngx.ERR, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ALERT, "Connected to local memcached, key: ", memcachedKey)
local scriptMemc, flags, err = memc:get(memcachedKey)
if err then
   ngx.log(ngx.ALERT, "No appropriate script found in Local Memcached: ", err)
end
if scriptMemc then
ngx.log(ngx.ALERT, "Script found in memcached")
ngx.ctx.replaceScript = scriptMemc .. "</head>"
isScriptFromMemcached = true
else
ngx.log(ngx.ALERT, "Script not found in memcached")
end
end
end
-- If script wasn't found in Memcached - retrieve script directly from CX
if not isScriptFromMemcached then
ngx.log(ngx.ALERT, "Retrieveing sript from CX")

local http = require("socket.http")
local ltn12 = require("ltn12")

local t = {}
http.request {
url = "http://blabla.com/luatest.txt",

method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string

scriptCX = table.concat(t)
ngx.log(ngx.ALERT, "Script from CX: ", scriptCX)
ngx.ctx.replaceScript = scriptCX .. "</head>"
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
ngx.log(ngx.ALERT, "scrip: ", scriptCX)
-- local ok, err = memc:set(memcachedKey, scrip, 900)
if not err then ngx.log(ngx.ALERT, "Script stored in Memcached, ok: ", ok) 
else ngx.log(ngx.ERR, "Failed to store the script in Memcached, err: ", err)
end
end
memc:close()
ngx.log(ngx.ALERT, "memc closed")
ngx.say(ngx.ctx.replaceScript)

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/bZi5GgqiuLs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.