Modify response from upstream using Lua code before caching it?

[ShmulikB]

Hi guys,

I'm wondering if there is a way to achieve this:

I'd like to modify the response from the upstream server before the cache module handles it, preferably using Lua code.

For instance, I'd like to change the Cache-Control header value, which will result in different expiration time in the cache itself.

Or add/remove headers before caching them.

The only method I've found so far, which isn't this efficient, is to use two different locations:

The first does the caching, and fetches from the second location as upstream.

The second location does a plain proxy without caching, and then I can modify anything in the response using the 'header_filter_by_lua_block' directive.

But to do so, I need the first location to proxy_pass from the second location via a real http request, which has its performance penalty (at least I didn't find a way to use an internal location instead).

Is there any other way to do it?

Or are there plans to add a lua hook after getting the response header from the upstream?

Any suggestion will be appreciated!

Thanks in advance,

ShmulikB

[Kshitij Joshi]

Are you using https://github.com/openresty/srcache-nginx-module?

You could have 

location  / {

  set $cacheControlValue "";

  header_filter_by_lua_file /usr/local/openresty/lualib/header_content.lua;

  add_header Cache-Control $cacheControlValue;

}

and in your header_content.lua file, you can have

if something then

  ngx.var.cacheControlValue = "x"

else 

  ngx.var.cacheControlValue = "y"

end

This will override the original cache-control send before caching the response and sending it to the client. 

Another thing you could do is you could route srcache_store through a new location. 

srcache_store PUT /store-modify-response key=$escaped_key;

location /store-modify-response {

  internal;

  content_by_lua_file /usr/local/openresty/lualib/modify_response.lua;

}

in your modify_response.lua, you can have 

local html_content = ngx.var.echo_request_body:lower();

-- modify whatever you want here

-- html_content will give you all the response headers as well as the body which is completely mutable

html_content = string.gsub(html_content, "cache%-control","")

local put_cache_arg = "cache_duration=" .. cache_duration .. "&" .. key

-- Finaly you will do this.

ngx.exec('/store-put-cache', put_cache_arg)

-- you would have another location block

location /store-put-cache {

  internal;

  set $default_cache_duration 604800;

  redis_connect_timeout 750ms;

  redis_send_timeout 750ms;

  redis_read_timeout 750ms;

  set_unescape_uri $cache_duration $arg_cache_duration;

  set_unescape_uri $key $arg_key;

  redis2_query set $key $echo_request_body;

  redis2_query expire $key $cache_duration;

  redis2_pass $redishost;

}

[Shmulik Bibi]

Thank you for the detailed answer!

I was not aware of the srcache module, I'll dive into the details, maybe it's exactly what I need.

From first glance, I think this also comes with the overhead of passing the content itself multiple times (while copying it each time). 

Since I'm caching big files, I'd like to avoid it if possible. So I won't sure it will work here. Also, I'm still not sure how to use srcache with Nginx's proxy_cache. Or maybe it should be used as a replacement?

As for the first suggestion - using a variable "cacheControlValue":

This will send the modified value to the client, but it will not modify the cached header. And the caching module would still calculate the expiration based on the original header.

I did not mention it explicitly in my first post, my apologies - the cache module I'm using is Nginx's built-in cache module, over the proxy module (i.e. proxy_cache).

Any other suggestions are of course very welcome!

Thanks,

Shmulik

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/AH0XV1ixeU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/9b5da02a-bd34-4742-9e01-8a901bcbbdf4n%40googlegroups.com.

[monyrasta lima]

well you deceive people saying that you will pay and you never paid this is false advertising

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/25834f8e-8857-40c0-890b-5dba5181ec2dn%40googlegroups.com.

[Shmulik Bibi]

Sorry, I'm not following 🤔

To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/CAHYBxcK3gwm7kxsdqRfQ-XFkQUXQnhsp_JY_nRMjJGzKq0_pnA%40mail.gmail.com.

[monyrasta lima]

Ok no problem thanks 

To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/CADTAdvunvKqmGmScs-M40O7mKgRNJobWooWte5EBFsRZN1VFxw%40mail.gmail.com.

[Kshitij Joshi]

Ah Sorry. I didn't realize that you were using the Nginx's built-in cache module.  srcache uses something like redis or memcache as a cache. it can be used with proxy_pass. So it's use case is different.

If you are using proxy_cache and using the server to cache content, the comments on this thread https://forum.nginx.org/read.php?2,283347,283347 might help you. Looks like it has the same problem you're trying to solve.

However, all the solutions only modify the response before they send data to the client. So this problem will not be solved. "Nginx would still calculate the expiration based on the original header."

I'm not sure if this would work https://forum.nginx.org/read.php?2,283347,283367#msg-283367 but you can give it a try.

[Shmulik Bibi]

Great, thank you very much!

From these resources it seems that indeed the only way to affect the actual cached headers is using a double proxy location (one fetching from the other).

I appreciate the very fast and professional response, thanks a lot!

You received this message because you are subscribed to the Google Groups "openresty-en" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/a958b37b-70ae-45da-954c-f0233b3c6a33n%40googlegroups.com.

[Kshitij Joshi]

No problem! All the best.

To view this discussion on the web visit https://groups.google.com/d/msgid/openresty-en/CADTAdvtZpM6D18xMsPbgM398wpOOGvHGavQBh_cmtULx2G6RRA%40mail.gmail.com.

--

Regards,

Kshitij Joshi.