nginx lua modify ngx.var.request_uri to increase cache HIT ratio

[c0nw...@googlemail.com]

So I use Nginx's fastcgi_cache and It occurred to me that different files unnecessarily was being cached even though they are the same page because of the request URI arguments being in a different order.

Here is a example :

All of these are the exact same URL :

https://example.com/a?color=red&day=friday&word=hi
https://example.com/a?color=red&word=hi&day=friday
https://example.com/a?day=friday&color=red&word=hi
https://example.com/a?day=friday&word=hicolor=red
https://example.com/a?word=hi&color=red&day=friday
https://example.com/a?word=hi&day=friday&color=red

If the request URI was ordered correctly like so :

https://example.com/a?color=red&day=friday&word=hi

We would achieve a much much higher cache HIT ratio.

My Cache Key :

fastcgi_cache_key "$session_id_value$scheme$host$request_uri$request_method$cache_request_body";


What is the best way to go about achieving this I have done things in Lua before.

ngx.var.request_uri

But I am unsure what directive should be used to modify the request before it gets passed onto the upstream.

https://github.com/openresty/lua-nginx-module#directives

Then also the little more complex matter when it comes down to some kind of sorting method / algorithm for the arguments and query strings in the request.


Thank's for reading.

[tokers]

Hello!

You can't modify the variable $request_uri because it is readonly. As an alternative scheme, the variable $args  is changeable, i think you can use the method  ngx.req.get_uri_args(which organizes the $args as a table) to sort your args(by the key), and replace the $request_uri in your cache key by the combine of $uri and $args.

[c0nw...@googlemail.com]

Can't we also do this

Nginx create empty var :

set $new_request_uri '';

Lua :

ngx.var.new_request_uri = ngx.var.request_uri --make our empty var contents the same as the request URI contents to then be modified.

And where my Cache key is change from this :

fastcgi_cache_key "$session_id_value$scheme$host$request_uri$request_method$cache_request_body";

To this :

fastcgi_cache_key "$session_id_value$scheme$host$new_request_uri$request_method$cache_request_body";

And we could also change our fastcgi_params :

From this :

fastcgi_param  REQUEST_URI        $request_uri;

To this :

fastcgi_param  REQUEST_URI        $new_request_uri;

Leaving just the requirement for the LUA code to modify the "ngx.var.new_request_uri" to sort / order the arguments in it into the correct order.

[tokers]

Hello

Yeah, that's what i mean.

[c0nw...@googlemail.com]

Do you know what directive i should use because of the execution order of things some *_lua directives execute before others.

Can you help with the Lua code to modify and orde ngx.var.new_request_uri

[tokers]

Do you know what directive i should use because of the execution order of things some *_lua directives execute before others.

I think you can use the directive rewrite_by_lua_file, because the fastcgi module work on the content phase.

Can you help with the Lua code to modify and orde ngx.var.new_request_uri

I have already said, ngx.var.request_uri is the one from the http request line(raw), it is inconvenient if you want to fetch the args, alternatively, you can get your args table by the method ngx.req.get_uri_args, then sort these keys(arg_name) using table.sort like the following code(just for demonstration):

local args_table = ngx.req.get_uri_args
local args_name = {}

for name, _ in pairs(args_table) do
    table.insert(args_name, name) -- maybe you would like incase-sensitive.
end

table.sort(args_name)

local newargs = {}

for _, name in ipairs(args_name) do
    table.insert(newargs, name .. "=" .. args_table[name])
end

ngx.var.args = table.concat(newargs, "&")

Then you construct your ngx.var.new_request_uri by the combination of ngx.var.uri and ngx.var.newargs.

[c0nw...@googlemail.com]

Wow awesome thanks so much :)

I can also use a variable i was reading about called  $is_args

local is_args = ngx.var.is_args
if is_args then
--if arguements exist then lets sort and order them
end

http://nginx.org/en/docs/http/ngx_http_core_module.html#var_is_args    “?” if a request line has arguments, or an empty string otherwise

[c0nw...@googlemail.com]

Thanks again for your help previously thought I should let you know I have been struggling to get your code to work as intended for a better cache HIT ratio.

For example this code works at outputing the args and information supplied in the URL.

Header GET requests :

location = /test_GET_args {
     content_by_lua_block {
         local args = ngx.req.get_uri_args()
         for key, val in pairs(args) do
             if type(val) == "table" then
                 ngx.say(key, ": ", table.concat(val, ", "))
             else
                 ngx.say(key, ": ", val)
             end
         end        
     }
}

Header POST requests :

location = /test_POST_args {
     content_by_lua_block {
         ngx.req.read_body()
         local args, err = ngx.req.get_post_args()
         if not args then
             ngx.say("failed to get post args: ", err)
             return
         end
         for key, val in pairs(args) do
             if type(val) == "table" then
                 ngx.say(key, ": ", table.concat(val, ", "))
             else
                 ngx.say(key, ": ", val)
             end
         end
     }
}

The code you supplied has a few issues as follows :

It is trying to put a table into a string.

The following function was missing its braces

ngx.req.get_uri_args

was missing

ngx.req.get_uri_args()

Throwing out function errors.

And I am having a hurdle of

bad argument #1 to 'pairs' (table expected, got string)

I have only tested briefly and will work at it more tomorrow to try and fix and get it functioning completely in ordering the arguments supplied for both GET and POST requests.

On Saturday, 8 July 2017 09:25:47 UTC+1, tokers wrote:

[c0nw...@googlemail.com]

I eliminated all the errors and the only problem left is there is no output....But I can't see and don't know why :(

location = /test_GET_args {
    content_by_lua_block {
       
local args_table = ngx.req.get_uri_args()
local args_name = {}

for key, val in pairs(args_table) do

    if type(val) == "table" then

        --ngx.say(key, ": ", table.concat(val, ", "))
    else
        --ngx.say(key, ": ", val)
        table.insert(args_name, name)
    end

end

table.sort(args_name)

local newargs = {}
for _, name in ipairs(args_name) do
    table.insert(newargs, name .. "=" .. args_table[name])
end

local output = table.concat(newargs, "&")
if output then
ngx.say(" FINISH " .. output .. "")
ngx.var.args = output --set the args to be the output
end

--Check the args order has been optimized
local args_table = ngx.req.get_uri_args()

for key, val in pairs(args_table) do

    if type(val) == "table" then
        ngx.say(key, ": ", table.concat(val, ", "))
    else
        ngx.say(key, ": ", val)
    end
end
   
    }
}

The ngx.var.args should be the URL supplied but ordered neatly when checked at the end.

But instead it does not output anything...


So this URL :


/test_GET_args?m=hello&a=3&b=42&c=1&z=2&y=4&x=5

Should become this and be confirmed in the output too :

/test_GET_args?a=3&b=42&c=1&m=hello&x=5&y=4&z=2

But all I get in my output is "FINISH"

ngx.say(" FINISH " .. output .. "")

And the following Check should output everything that has been ordered but does not output anything...

--Check the args order has been optimized
local args_table = ngx.req.get_uri_args()

for key, val in pairs(args_table) do

    if type(val) == "table" then
        ngx.say(key, ": ", table.concat(val, ", "))
    else
        ngx.say(key, ": ", val)
    end
end

I don't know what is wrong because there are no errors anymore...

On Saturday, 8 July 2017 09:25:47 UTC+1, tokers wrote:

[tokers]

Hello!

I browsed through your code, and i think the devil is in here

for key, val in pairs(args_table) do

    if type(val) == "table" then

        --ngx.say(key, ": ", table.concat(val, ", "))
    else
        --ngx.say(key, ": ", val)

        table.insert(args_name, name) -- table.insert(args_name, key)
    end

end

[c0nw...@googlemail.com]

Ah yes thank you <3!!.

I made it this now what works but still needs some very minor tweaks / changes.

location = /testargs {

     content_by_lua_block {

local args_table = ngx.req.get_uri_args()
local args_name = {}

for name, value in pairs(args_table) do
    if type(value) == "table" then
        --ngx.say(name, ": ", table.concat(value, ", "))
       
        --This needs fixing incase multiple arguements share the same name. (Example multiple of the same arguement name with different values)
        --&argument1=cake&arguement1=cheese&arguement1=frog
    else

        table.insert(args_name, name) -- maybe you would like incase-sensitive.
    end

end

table.sort(args_name) --Sort the table into order

local newargs = {}
for _, name in ipairs(args_name) do
    table.insert(newargs, name .. "=" .. args_table[name])
end

local output = table.concat(newargs, "&")
if output then

ngx.say(" FINISH " .. output .. " END ")

ngx.var.args = output --set the args to be the output
end




--Check the args order has been optimized
local args_table = ngx.req.get_uri_args()
for key, val in pairs(args_table) do
    if type(val) == "table" then
        ngx.say(key, ": ", table.concat(val, ", "))
    else
        ngx.say(key, ": ", val)
    end
end
         
     }
}

The only dilemmas / bugs optimizations needed are the following.

Input :

localhost/test_GET_args?lol=1&lol2=lollypop&argument1=cake&arguement1=cheese&arguement1=frog

Output :

argument1=cake&lol=1&lol2=lollypop

It seems to be loosing multiple arguements of the same name due to this.

if type(value) == "table" then

else

I would of expected the output to be this. (With the arguments with the same name "arguement1" still existing in the output but ordered by their values)
argument1=cake&arguement1=cheese&arguement1=frog&lol=1&lol2=lollypop


[c0nw...@googlemail.com]

I put everything into a function to make it readable and tidy now too :)

local function sort_args(input)

    local args_table = input
    local args_name = {}

    for name, value in pairs(args_table) do

        if type(value) == "table" then
            --ngx.say(name, ": ", table.concat(value, ", "))
       
            --This needs fixing incase multiple arguements share the same name. (Example multiple of the same arguement name with different values)
            --&argument1=cake&arguement1=cheese&arguement1=frog
        else

            table.insert(args_name, name) -- maybe you would like incase-sensitive.
        end

    end

    table.sort(args_name) --Sort the table into

order

    local newargs = {}

    for _, name in ipairs(args_name) do
        table.insert(newargs, name .. "=" .. args_table[name])
    end

    local output = table.concat(newargs, "&")

    return output --set the args to be the output

end

--GET requests

if ngx.HTTP_GET then

    local get_uri_args = ngx.req.get_uri_args()
    ngx.var.args = sort_args(get_uri_args)

    ngx.say(" GET : FINISH " .. ngx.var.args .. " END ")

end

--END GET requests


--POST requests

local function sort_POST_body(input)
end

if ngx.HTTP_POST then

    ngx.req.read_body()
    local get_POST_args, POST_args_err = ngx.req.get_post_args()
    if not get_POST_args then
        ngx.say("failed to get post args: ", POST_args_err)
        return
    end

    if get_POST_args then --Sort the URL args
        ngx.var.args = sort_args(get_POST_args)
        ngx.say(" POST : FINISH " .. ngx.var.args .. " END ")
    end
   
    --Sort the BODY args

end

--END POST requests

The only things to order now are multiple arguments of the same name. "argument1=hello&argument1=world&argument1=lovely"

And to also make sure that POST requests supplied DATA gets ordered correctly too.

POST Data URL sort :

curl -X POST -d "foo=bar&bar=baz&bar=blah&assemble=put" http://localhost:80/test_GET_args

POST Form Data CURL :

curl -H "Content-Type: application/json" -X POST -d "{"username":"xyz","password":"xyz"}" http://localhost:80/test_GET_args

On Monday, 10 July 2017 09:44:20 UTC+1, tokers wrote:

[c0nw...@googlemail.com]

Hello again tokers <3 :)

I am having a bit of trouble with the incase sensative thing you mentioned in a nulled / commented out statement. "-- maybe you would like incase-sensitive."

I wanted to make the arguement names lower case. The values can be what user supplied as for the sake of
"?SEARCH=Hello"
"?Search=Hello"
"?SeArCh=Hello"

All should output this :
"?search=Hello"

lua_args_sort.lua:26: attempt to concatenate a nil value

The culprit is this :

table.insert(args_name, string.lower(name)) -- maybe you would like incase-sensitive.

How can I make arg names lower case ?

On Monday, 10 July 2017 09:44:20 UTC+1, tokers wrote:

[tokers]

Hello!

Firstly, i must say sorry for my poor engish. -.-!

For the error log you paste:

lua_args_sort.lua:26: attempt to concatenate a nil value

Well, i think you should code more carefully. the keys in args_table is all in lowercase, so when you accessing them, you also need to use the lowercase form.

for _, name in ipairs(args_name) do

    table.insert(newargs, name .. "=" .. args_table[name]) -- args_table[string.lower(name)]
end

[c0nw...@googlemail.com]

Hey and thanks tokers :) that is perfectly ok no need to apologize your English is fantastic and understandable for me :)

I have hit a problem with try_files with this. Here is the full code :


Here is my code :

##HTTP BLOCK
map $request_body $cache_request_body {
default $request_body;
}

map $request_uri $cache_request_uri {
default $request_uri;
}
##END HTTP BLOCK
##etc all default Nginx stuff
##INSIDE SERVER BLOCK

# Support Clean (aka Search Engine Friendly) URLs | https://docs.joomla.org/Nginx#Configure_Nginx
location / {
    try_files $uri $uri/ /index.php?$args;
}


location ~ \.php$ {
#rewrite_by_lua_block { ##tried this one too also did not work
access_by_lua_block {

local function sort_args(input)
    local args_table = input
    local args_name = {}

    local newargs = {}

    for name, value in pairs(args_table) do
        if type(value) == "table" then

            for k, v in pairs(value) do
                table.insert(newargs, name .. "=" .. value[k])
            end
        else

            table.insert(args_name, name) -- maybe you would like incase-sensitive.
        end

    end

    for _, name in ipairs(args_name) do
        table.insert(newargs, name .. "=" .. args_table[name])
    end
   

    table.sort(newargs) --Sort the table into order

    local output = table.concat(newargs, "&")

    return output --set the args to be the output
end

--GET requests

--if ngx.HTTP_GET then

    local get_uri_args = ngx.req.get_uri_args()
    local args_output = sort_args(get_uri_args)
   
    local uri = (ngx.var.uri or "")
    local is_args = (ngx.var.is_args or "")
    local args = (args_output or "")
   
    --Set the cache URI key
    ngx.var.cache_request_uri = string.lower(uri .. is_args .. args) --make all lowercase for a higher cache hit ratio
   
    --ngx.exec(ngx.var.cache_request_uri) --nulled this out because this caused the following error |   rewrite or internal redirection cycle while internally redirecting to "/index.php",

--end

--END GET requests
--]]

} #END LUA BLOCK


#etc fastcgi caching settings and key
fastcgi_cache_key "$scheme$host$cache_request_uri$request_method$cache_request_body";

#fastcgi stuff
fastcgi_pass  127.0.0.1:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi.conf;

} #END PHP BLOCK

##END INSIDE SERVER BLOCK

Now the problem is the following.

I request a URL like this :

/index.php?option=com_details

It works flawlessly and all is great. (direct access to the location PHP block it seems)



If i request the URL for the same page what is a SEF url that goes through "try_files" i get problems.

/details

This will show me the websites homepage for some reason.

Even URL's like this :

/details&help=me

/details&contact=page

All show the websites homepage i can't figure out how to get it to work I don't know what to do to get it to work as I intend displaying their rightful pages instead of the homepage each time, at the same time running the Lua code set my "cache_request_uri" correctly so my fastcgi_cache_key matches and has a higher cache HIT ratio.

So the Lua code works great its just fixing the INTERNAL try_files issue now.

Thank's in advance if you can assist me "AGAIN" <3