Avoiding cache stampedes using srcache

[Diptamay Sanyal]

Hi 

Is there a way I could configure srcache to avoid cache stampedes the way HttpProxyModule does using "proxy_cache_use_stale"?

Thanks

-Diptamay

[agentzh]

Hello!

On Fri, Mar 22, 2013 at 5:25 PM, Diptamay Sanyal wrote:
> Is there a way I could configure srcache to avoid cache stampedes the way
> HttpProxyModule does using "proxy_cache_use_stale"?
>

You can consider using longer expiration in the cache storage backend
(be it redis or memcached), and do all the special handling in some
Lua injected by rewrite_by_lua in the srcache_fetch subrequest's
location to do the necessary coordination and dispatching. See

http://wiki.nginx.org/HttpLuaModule#rewrite_by_lua

The beauty of srcache is that it can be easily customized by adding
magic to the subrequests.

Best regards,
-agentzh

[Diptamay Sanyal]

Thanks! I shall check it out.

--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/Q6n8yg_rSdg/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Brian Akins]

On Mar 22, 2013, at 9:06 PM, agentzh <age...@gmail.com> wrote:

> You can consider using longer expiration in the cache storage backend
> (be it redis or memcached),

That's what I do, using setnx+expire in redis using Lua. I played with doing it in shared dictionaries, but the "transactions" and Lua scripting in redis make it a better fit, IMO.

[James Hurst]

On 23 March 2013 00:25, Diptamay Sanyal <dipt...@gmail.com> wrote:

Is there a way I could configure srcache to avoid cache stampedes the way HttpProxyModule does using "proxy_cache_use_stale"?

You might want to check out Ledge[1], even if just to see an example of this in action. It can be configured to serve stale content, revalidate stale content in the background, collapse concurrent similar requests for cacheable content into a single origin fetch, as well as two modes of offline operation - ORIGIN_MODE_BYPASS which will just error if it can't serve from cache, and ORIGIN_MODE_AVOID which is like Squid's offline mode, where it'll only try the origin if it has no cache no matter how expired it might be.

This is all achieved with ngx_lua and Redis. Specifically the collapsed forwarding feature uses transactions and a small amount of Lua scripting in Redis[2] to get around possible races. Hitting it with thousands of concurrent requests for the same expired yet (previously known to be) cacheable content results in a single origin fetch every time.

[1] https://github.com/pintsized/ledge 

[2] https://github.com/pintsized/ledge/blob/master/lib/ledge/ledge.lua#L724-L774

[Brian Akins]

On Mar 23, 2013, at 9:41 AM, James Hurst <ja...@riverratrecords.com> wrote:

> You might want to check out Ledge[1],

Ledge has matured a bit since I originally looked at it. I'll have to give it a try as I think I may be in danger of recreating it (poorly) on my own.

--Brian

[Diptamay Sanyal]

James

Ledge definitely look interesting. Is there any way it can deal with a cluster of redis hosts and can hash and pick the host depending on the request uri out of the box? Or is there a way I could pass the name of an "upstream" server list to the function rather than a host/port through config?

Thanks

Diptamay

--

[James Hurst]

Hi Diptamay,

On 24 March 2013 01:31, Diptamay Sanyal <dipt...@gmail.com> wrote:

James

Ledge definitely look interesting. Is there any way it can deal with a cluster of redis hosts and can hash and pick the host depending on the request uri out of the box?

The genearl idea is that you work "with the grain" in your Nginx config, meaning anything you wish to do differently for different URIs, you catch in location blocks.

location / {

  # Use the default host/port, or one you've globally specified during init_by_lua.

  content_by_lua 'ledge:run()';

}

location /special_uri {

  # Use a different redis host.

  content_by_lua '

    ledge:config_set("redis_host", x.x.x.x)

    ledge:run()

  ';

}

This effectively allows you shard cache data based on URIs. You could also change the host (and any other config option) based on arbitrary conditionals within a location block.

Or is there a way I could pass the name of an "upstream" server list to the function rather than a host/port through config?

No, but what are you trying to achieve exactly? You probably don't need to load balance Redis (and remember there's no master-master replication yet), but failover is important to some. I'm in the process of refactoring the config portions slightly, and adding Redis Sentinel support for automatic failover. This should be in next week.

James.

[Diptamay Sanyal]

Hi James

I am actually sharding my data across a redis cluster (all in-memory mode with no slaves aka memcached style). I actually am using a special build of openresty where Yichun patched the redis module to work with a list of upstream servers. Look here https://groups.google.com/forum/?fromgroups=#!topic/openresty-en/N8eHtJb17ew

So my config looks like:

1) I use request uri to shard my data across a cluster of redis upstream servers. Each redis upstream server is a pair. I am on AWS, so 1 in each pair is in 2 different availability zones.

2) I then use ip_hash to decide which server in the redis upstream pair to pick. So even if 1 zone goes down, I have a replica (sort of) then and the ip_hash would ensure that my cache is completely not cold. And by doing this, I am kinda simulating, a Master-Master scenario for the same uri using client ip and then adding sharding on top with my request uri.

This configuration approach is actually working very well. I just need to find a way to add anti-cache stampedes. 

upstream api_server {

    ip_hash;

    server 10.46.169.88:6379;

    server 10.191.191.1:6379;

    keepalive 1024;

}

upstream redis_6379 {

    ip_hash;

    server 10.46.169.88:6379;

    server 10.191.191.1:6379;

    keepalive 1024;

}

upstream redis_6380 {

    ip_hash;

    server 10.46.169.88:6380;

    server 10.191.191.1:6380;

}

upstream_list redis_cluster redis_6379 redis_6380;

server {

    listen 80;

server_name site.*;

    # site traffic from the browser to the API should come through here

    location /api/dev {

        if ($request_method !~ ^(GET|HEAD)$ ) {

            return 405;

        }

        # tell nginx not to modify the response for URLs from the proxied process.

        proxy_redirect off;

        # Send appropriate headers through

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Host $http_host;

        # set expiration headers for site calls to be short

        expires +10s;

        # Only cache valid HTTP responses

        srcache_store_statuses 200 301 302;

        # only allow pull methods

        srcache_methods GET HEAD;

        set $key $request_method$request_uri;

        set_escape_uri $escaped_key $key;

        srcache_fetch GET /redis-get $key;

        srcache_store PUT /redis-set key=$escaped_key&exptime=300;

        rewrite ^/api/dev/(.*)$ /$1 break;

        proxy_pass http://api_server;

    }

    location = /redis-get {

        internal;

        set $redis_key $args;

        set_hashed_upstream $backend redis_cluster $redis_key;

        redis_pass $backend;

    }

    location = /redis-set {

        internal;

        set_unescape_uri $exptime $arg_exptime;

        set_unescape_uri $key $arg_key;

        set_hashed_upstream $backend redis_cluster $key;

        redis2_query set $key $echo_request_body;

        redis2_query expire $key $exptime;

        redis2_pass $backend;

    }

}

-Diptamay

--

[James Hurst]

On 25 March 2013 02:32, Diptamay Sanyal <dipt...@gmail.com> wrote:

Hi James

I am actually sharding my data across a redis cluster (all in-memory mode with no slaves aka memcached style). I actually am using a special build of openresty where Yichun patched the redis module to work with a list of upstream servers. Look here https://groups.google.com/forum/?fromgroups=#!topic/openresty-en/N8eHtJb17ew

Ah I see. Sorry, I missed that thread. 

 

So my config looks like:

1) I use request uri to shard my data across a cluster of redis upstream servers. Each redis upstream server is a pair. I am on AWS, so 1 in each pair is in 2 different availability zones.

2) I then use ip_hash to decide which server in the redis upstream pair to pick. So even if 1 zone goes down, I have a replica (sort of) then and the ip_hash would ensure that my cache is completely not cold. And by doing this, I am kinda simulating, a Master-Master scenario for the same uri using client ip and then adding sharding on top with my request uri.

This configuration approach is actually working very well. I just need to find a way to add anti-cache stampedes. 

I guess the problem you have is that srcache is intended as a transparent cache for location blocks, so it'll do exactly as told, but can't (and probably shouldn't) on its own handle complex possibilities such as serving known stale content under a given scenario.

If protecting your origins from traffic spikes is vital to you, the only thing I can suggest off the top of my head is to rethink the architecture slightly and put an OpenResty / Ledge / Redis node at each of your AWS locations, and then loadbalance / shard them using upstream and HttpProxyModule, instead of sharding to pure Redis instances. Then Ledge can help you with managing upstream better. Granted this is more complicated, but then caching is more complicated than it looks on first glance ;)

There might be another way, perhaps someone else here has a better suggestion...

[Brian Akins]

In one project, I have a local redis server per nginx box. Every cache
write also publishes to a pub/sub channel. A script on each box
subscribes to the channels on all the other boxes and does "poor
man's" replication of objects. It's a little more than that (ie,
deciding what to copy, etc), but that's it in a nutshell. This is
much simpler than trying to do sharding, failover, etc. Of course,
this really only works if your hot set fits in the footprint of a
single redis instance.

[Diptamay Sanyal]

Thats an interesting idea, having a nginx fronting redis on each node. For now I am gonna roll as is, and deal with this stampeding later. 

Thanks

-Diptamay

--

[Diptamay Sanyal]

So the cache stampeding ended up being a blocker during performance tests with redis caching. But the good thing is most probably I would be able to hold the hot set in memory on a single redis instance. So 

Brian, 

I would be super grateful if could you give me some details on how you exactly did the poor man's replication If you could legally share it that is?

Thanks

Diptamay