Error in getting data from redis - Using redis cluster with srcache module

[Diptamay Sanyal]

Hello 

For some reason I cannot get the srcache module to work well with HttpRedis2Module module on a redis cluster. srcache works well with HttpRedisModule and HttpRedis2Module on a single redis node (the HttpRedisModule doesn't work with a redis cluster using set_hashed_upstream, sadly)

I am using OpenResty 1.2.7.1 and Redis 2.6.11. I am trying to cache my backend server JSON responses.

The urls I am calling are like:

http://api.dev:9090/collections/511193ba839e376d42913319/items

http://api.dev:9090/collections?edition=us&collection=home

I keep getting errors like this when I try to fetch

2013/03/21 08:56:12 [error] 9951#0: *1 srcache_fetch: cache sent invalid status line while sending to client, client: 127.0.0.1, server: api.*, request: "GET /collections?edition=us&collection=home HTTP/1.1", subrequest: "/redis-get", upstream: "redis2://127.0.0.1:6379", host: "api.dev:9090"

2013/03/21 09:11:09 [error] 9951#0: *10 srcache_fetch: cache sent truncated response body while sending to client, client: 127.0.0.1, server: api.*, request: "GET /collections/511193ba839e376d42913319/items HTTP/1.1", subrequest: "/redis-get", upstream: "redis2://127.0.0.1:6379", host: "api.dev:9090"

My nginx config is as follows:

daemon off;

pid {{scratch_directory}}/nginx.pid;

error_log {{scratch_directory}}/nginx_error.log;

events {

    worker_connections  1024;

}

http {

    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    upstream api_server {

        server 127.0.0.1:8001;

    }

    upstream redis_tesla {

        server 127.0.0.1:6379;

        keepalive 512;

    }

    upstream redis_leaf {

        server 127.0.0.1:6380;

        keepalive 512;

    }

    upstream_list redis_cluster redis_tesla redis_leaf;

    proxy_read_timeout 120;

    proxy_connect_timeout 120;

    server {

        listen 9090;

        client_max_body_size 10M;

        server_name api.*;

        access_log   {{scratch_directory}}/api_access.log;

        error_log    {{scratch_directory}}/api_error.log;

        location / {

            if ($request_method !~ ^(GET|HEAD)$ ) {

                return 405;

            }

            proxy_redirect off;

            # Send appropriate headers through

            proxy_set_header X-Real-IP $remote_addr;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_set_header Host $http_host;

            rewrite ^/api/dev/(.*)$ /$1 break;

            # set expiration headers for site calls to be short

            expires +120s;

            # Only cache valid HTTP responses

            srcache_store_statuses 200 301 302;

            # only allow pull methods

            srcache_methods GET HEAD;

            set $key $request_method$request_uri;

            set_escape_uri $escaped_key $key;

            srcache_fetch GET /redis-get key=$escaped_key;

            srcache_store PUT /redis-set key=$escaped_key&exptime=300;

            proxy_pass http://api_server;

        }

        location = /redis-get {

            internal;

            set_unescape_uri $key $arg_key;

            redis2_query get $key;

            set_hashed_upstream $backend redis_cluster $key;

            redis2_pass $backend;

        }

        location = /redis-set {

            internal;

            set_unescape_uri $exptime $arg_exptime;

            set_unescape_uri $key $arg_key;

            set_hashed_upstream $backend redis_cluster $key;

            redis2_query set $key $echo_request_body;

            redis2_query expire $key $exptime;

            redis2_pass $backend;

        }

        # block common, unnecessary requests

        location /favicon.ico {

            access_log off;

            error_log off;

            log_not_found off;

            deny all;

        }

        location /robots.txt {

            access_log off;

            error_log off;

            log_not_found off;

            deny all;

        }

    }

}

Any help would be greatly appreciated.

Thanks

-Diptamay

[agentzh]

Hello!

On Thu, Mar 21, 2013 at 6:14 AM, Diptamay Sanyal wrote:
> For some reason I cannot get the srcache module to work well with
> HttpRedis2Module module on a redis cluster. srcache works well with
> HttpRedisModule and HttpRedis2Module on a single redis node (the
> HttpRedisModule doesn't work with a redis cluster using set_hashed_upstream,
> sadly)
>

Well, I can patch ngx_redis to support nginx variables in its
redis_pass directive if you really want it. Otherwise, you may want to
contact the author of ngx_redis directly.

Another option is to use ngx_lua + lua-resty-redis to serve your
location /redis-get.

> location = /redis-get {
> internal;
>
> set_unescape_uri $key $arg_key;
> redis2_query get $key;
> set_hashed_upstream $backend redis_cluster $key;
> redis2_pass $backend;
> }
>

You cannot use ngx_redis2 in location /redis-get here because
ngx_redis2 always returns the raw redis response which is not what
srcache_fetch expects. That is why ngx_redis is used in the sample
configuration in the documentation of ngx_srcache:

http://wiki.nginx.org/HttpSRCacheModule#Caching_with_Redis

Best regards,
-agentzh

[Diptamay Sanyal]

Thanks!

I will try the ngx_lua + lua-resty-redis first. Is there any example that I can look at?

-Diptamay

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/N8eHtJb17ew/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[agentzh]

Hello!

On Thu, Mar 21, 2013 at 4:24 PM, Diptamay Sanyal <dipt...@gmail.com> wrote:
> I will try the ngx_lua + lua-resty-redis first. Is there any example that I
> can look at?
>

Just look at the official documentation:

https://github.com/agentzh/lua-resty-redis

There is sample code.

Best regards,
-agentzh

[agentzh]

Hello!

On Thu, Mar 21, 2013 at 4:04 PM, agentzh wrote:
>
> Well, I can patch ngx_redis to support nginx variables in its
> redis_pass directive if you really want it. Otherwise, you may want to
> contact the author of ngx_redis directly.
>

Okay, I've just prepared a small patch for ngx_redis 0.3.6 to allow
use of Nginx variables in the redis_pass directive:

https://raw.github.com/agentzh/ngx_openresty/master/patches/ngx_http_redis-0.3.6-variables_in_redis_pass.patch

If you're using the ngx_openresty bundle, you can directly try out the
1.2.7.3rc5 prerelease, which contains this patch:

http://openresty.org/download/ngx_openresty-1.2.7.3rc5.tar.gz

I'm cc'ing the author of ngx_redis, Sergey A. Osokin to see if he'd
include this patch in the mainstream version :)

Hi, Sergey! Please check out the complete discussion here if you need
more context:

https://groups.google.com/group/openresty-en/browse_thread/thread/37c787b496f5edec

Best regards,
-agentzh

[Diptamay Sanyal]

This is awesome! Thanks a lot! 

I shall check it out and let you know how it goes.

[Diptamay Sanyal]

This worked perfectly! Thanks a ton!

Would love this to go mainstream.

-Diptamay