Best Redis Config

[Michael Rowe]

I'm wondering if anyone can provide some guidance on the best settings for configuring redis, particularly to limit its memory usage. Here is my situation and how I've configured it.

I've got a Canvas server using Redis and I just noticed that the redis service stopped a couple days ago with the following error:

Background saving terminated by signal 9

I believe this is related to not having enough memory available to do a background save of the redis database, which is up to about 1.1 GB. So, I tried changing some of the settings in redis.conf.

maxmemory 800mb

maxmemory-policy volatile-lru

This was a bad idea cause it gave all my canvas user errors:

ActionView::TemplateError (ERR command not allowed when used memory > 'maxmemory') on line #46 of app/views/layouts/application.html.erb

I tried changing the maxmemory-policy to allkeys-lru and that seemed to accomplish limiting redis' memory usage without giving me the Canvas errors.

Any thoughts? Is there a better way or other things I should be considering?

[Cody Cutrer]

Michael,

Canvas has two uses for Redis - as a normal in-memory cache that can
lose data with the only consequence being the app slowing down while
the cache refills. This one is configured in cache_store.yml. The
other use is as a high-performance semi-volatile data store for
certain features (search the code for Canvas.redis to see where it is
used), configured in redis.yml. It's perfectly fine for the caching
redis instance to be set to allkeys-lru (especially since Canvas
doesn't set an expiration policy for every cache key it uses), but if
you set that for the data redis instance (or if you use the same
instance for both), it's possible that some features won't work
correctly sometimes when the cache is full. The optimal configuration
(and how Instructure has it configured) is to have a separate redis
instance for caching and data, with the eviction policy set to
allkeys-lru on the caching instance, and volatile-lru on the data
instance. Note that you can run both instances on the same server, and
you can also set the maxmemory lower for the data instance (especially
if you have PageViews disabled, or configured to not go to redis
first).

Hope you understood that,

Cody Cutrer
Software Engineer
Instructure

> --
>
>
>

[Michael Rowe]

Thanks Cody, that helps. I noticed that I do not have a redis.yml file configured (just the example). Does that mean I am currently only using redis for the cache_store or will the data storage default to use the same redis config as the cache_store.yml.

Also, (to save me a little research) do I need a completely separate instance of redis running from a different config file or can I use the same instance and just a different database?

[Cody Cutrer]

Michael,

The data redis is disabled unless you have a redis.yml with valid
settings. You can put your server settings in redis.yml, and simply
specify "redis" in cache_store.yml, and it will inherit from
redis.yml. You can use the same instance and a different database,
but I'm not sure if you can specify different eviction policies per
database.

Cody Cutrer
Software Engineer
Instructure

> --
>
>
>

[Christopher Bennell]

This is great information, thanks Cody. 

Form what I understand, it's easy enough to run multiple instances of redis on the same server, with different configs, with little overhead: http://chrislaskey.com/blog/342/running-multiple-redis-instances-on-the-same-server/

[Christopher Bennell]

Cody, is there any any to flush the cache data and keep the data-store data? If I'm migrating from one instance for both types of data to two instances, I will want to copy my redis dump file, so I don't lose any data. But I don't want to have to a duplicate of the cache data in my new data-store. Are there key prefixes I can use to flush the cache? 

Thanks. 

On Wednesday, September 5, 2012 11:27:41 PM UTC-4, Michael Rowe wrote:

[Cody Cutrer]

Christopher,

Unfortunately I don't know of an easy way to split the cache.
Searching through the code for Canvas.redis, it looks like the
important key prefixes to not delete in the data redis are oauth2*,
page_view_queue, active_users*, service:timeouts:*, and
login_attempts:*.

Cody Cutrer
Software Engineer
Instructure

> --
>
>
>