Redis configuration with Canvas

[Andres Jordan]

Hello Canvas LMS experts,

Hope you guys can help me out with an issue we are having.   First of, our setup constitutes of all canvas components running in a single server with plenty of CPU, RAM and storage.  We have just recently started experiencing some random timeouts/latency, I noticed that the redis-rdb-bgsave process is using up all the memory and swap space on the server.  While looking through the redis-server.log, there are no errors shown but I see a lot of the below.

1301:C 23 Aug 12:42:00.554 * DB saved on disk

21301:C 23 Aug 12:42:02.498 * RDB: 934 MB of memory used by copy-on-write

1685:M 23 Aug 12:42:05.094 * Background saving terminated with success

1685:M 23 Aug 12:43:06.010 * 10000 changes in 60 seconds. Saving...

1685:M 23 Aug 12:43:08.422 * Background saving started by pid 21946

21946:C 23 Aug 12:57:03.793 * DB saved on disk

21946:C 23 Aug 12:57:05.678 * RDB: 852 MB of memory used by copy-on-write

1685:M 23 Aug 12:57:08.121 * Background saving terminated with success

1685:M 23 Aug 12:58:09.103 * 10000 changes in 60 seconds. Saving...

1685:M 23 Aug 12:58:11.566 * Background saving started by pid 22603

22603:C 23 Aug 13:11:58.770 * DB saved on disk

22603:C 23 Aug 13:12:00.661 * RDB: 1334 MB of memory used by copy-on-write

1685:M 23 Aug 13:12:03.196 * Background saving terminated with success

1685:M 23 Aug 13:13:04.042 * 10000 changes in 60 seconds. Saving...

1685:M 23 Aug 13:13:06.535 * Background saving started by pid 23458

23458:C 23 Aug 13:26:24.714 * DB saved on disk

23458:C 23 Aug 13:26:26.663 * RDB: 955 MB of memory used by copy-on-write

1685:M 23 Aug 13:26:29.260 * Background saving terminated with success

1685:M 23 Aug 13:27:30.052 * 10000 changes in 60 seconds. Saving...

1685:M 23 Aug 13:27:32.383 * Background saving started by pid 25127

25127:C 23 Aug 13:41:09.278 * DB saved on disk

25127:C 23 Aug 13:41:11.179 * RDB: 527 MB of memory used by copy-on-write

1685:M 23 Aug 13:41:13.714 * Background saving terminated with success

1685:M 23 Aug 13:42:14.008 * 10000 changes in 60 seconds. Saving...

1685:M 23 Aug 13:42:16.299 * Background saving started by pid 27140

I believe all this started because we had the server crash a while ago and Redis was not running for about a month.  Now that I have started it Redis is trying to process all the backlog.  See rdb files below. 

/var/lib/redis# ls -lt -h

total 83G

-rw-rw---- 1 redis redis  36G Aug 23 14:08 temp-29298.rdb

-rw-rw---- 1 redis redis  41G Aug 23 13:55 dump.rdb

-rw-rw---- 1 redis redis 2.4G Aug 23 00:22 temp-1683.rdb

-rw-rw---- 1 redis redis 2.1G Aug 22 22:44 temp-8334.rdb

-rw-rw---- 1 redis redis 2.5G Jul 20 01:00 temp-8735.rdb

-rw-rw---- 1 redis redis 185M Apr 17 03:47 temp-30556.rdb

 

While looking through this Canvas group, there are a lot of different suggestions about flushing the Redis cached data and disabling "saving Redis to disk in redis.conf" by commenting out the save lines (i.e.: save 900 1).  My questions about doing this:

• How do I know if Redis is being used only for caching purposes?
• Will flushing the Redis cache have major effects on the canvas application?
• Will disabling the "saving Redis to disk" break anything in canvas?

Forgive my ignorance in some of these things, I am fairly new to canvas and I am trying to wrap my head around all this.

[Graham Ballantyne]

Hi Andres,

Redis in Canvas is used for a few things:

- view caching

- session management

- oauth

It's primarily used in a caching role. We don't have save enabled on our three redis servers. As for your other two questions:

> Will flushing the Redis cache have major effects on the canvas application?

Yes; you'll likely kill all your current user sessions and your users may have to log in again (caveat: we use CAS in front of Canvas, and when we flush redis users have to log in again; I'm betting it's the same if you're using Canvas internal auth but I'm not 100% sure). However; it's sometimes necessary; we try to do it during off hours if we need to do it.

> Will disabling the "saving Redis to disk" break anything in canvas?

Nope, it shouldn't.

There's a few good guides on Redis in production; I've used http://shokunin.co/blog/2014/11/11/operational_redis.html with good results; http://tech.trivago.com/2017/01/25/learn-redis-the-hard-way-in-production/ and https://blog.sensible.io/2013/08/20/setting-up-redis-for-production-environment.html are also good.

Good luck!

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Graham Ballantyne
IT Services
Simon Fraser University

[Cody Cutrer]

Andres,

One quick clarification - if you're NOT using CAS, then users will not be logged out. CAS users are logged out because we have to keep server-side state in order to implement Single Log Out from the CAS Server. All other session details are stored in (encrypted) cookies, so other auth types are not effected by clearing Redis.

Cody Cutrer

Software Engineer

Instructure

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Graham Ballantyne
IT Services
Simon Fraser University

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-users+unsubscribe@googlegroups.com.

[Graham Ballantyne]

Thanks for the clarification, Cody – I figured as much.

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Graham Ballantyne
IT Services
Simon Fraser University

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andres Jordan]

thanks guys that clarifies things for me, just one last thing about clearing the cache I have seen two ways of doing it I am just curious which is best.

etc/init.d/redis-server stop

cat /dev/null > /var/lib/redis/dump.rdb

/etc/init.d/redis-server start

or

FLUSHDB command by redis-cli 

[Cody Cutrer]

It depends on how much is in your cache. flushdb tends to be easier to do, but can cause bad side effects if your cache is very large and blocks other requests while redis is doing internal bookkeeping. If it's critical, your best bet is to `kill -9` the redis server, then start it anew (and assuming dumps are not configured, you don't need to delete the dump from disk).

Cody Cutrer

Software Engineer

Instructure

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-users+unsubscribe@googlegroups.com.

[Andres Jordan]

So I disabled "saving to disk" and restarted Redis.  And my swap went all the way down and the performance on the server seems to have improved.  However,  I notice that Redis is using up pretty much all the memory available.

Is there a configuration that I might need to change for Redis? any suggestions?

[Graham Ballantyne]

Using all available memory is Redis’ core competency, and generally what you want on a dedicated machine. In your case where you're running everything on one box, though, it isn’t great ;) 

You should take a look at the redis MAXMEMORY config option. It defaults to unlimited in 64bit systems.  https://redis.io/topics/lru-cache

-- 

Graham Ballantyne 

IT Services 

Simon Fraser University 

On Aug 24, 2017, at 09:14, Andres Jordan <ayor...@gmail.com> wrote:

So I disabled "saving to disk" and restarted Redis.  And my swap went all the way down and the performance on the server seems to have improved.  However,  I notice that Redis is using up pretty much all the memory available.

Is there a configuration that I might need to change for Redis? any suggestions?

--

[Andres Jordan]

So I just looked over the redis.conf and trough redis-cli INFO.  And by the looks of it the maxmemory and maxmemory_policy are set to default (results below from redis-cli INFO).

# Memory

used_memory:124181547104

used_memory_human:115.65G

used_memory_rss:124049670144

used_memory_rss_human:115.53G

used_memory_peak:124182183744

used_memory_peak_human:115.65G

total_system_memory:135074127872

total_system_memory_human:125.80G

used_memory_lua:53248

used_memory_lua_human:52.00K

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

mem_fragmentation_ratio:1.00

mem_allocator:jemalloc-4.0.3

Based on the links you guys have provided, it seems that the best route to go with is set a maxmemory limit and set maxmemory-policy to allkeys-lru

[Shawn Iverson]

allkeys-lru is what I use.

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Shawn Iverson, CETL

Director of Technology

Rush County Schools

765-932-3901 x271

iver...@rushville.k12.in.us

[Graham Ballantyne]

Same here. 

To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

[Andres Jordan]

Guys thanks for all the help I took all your different suggestions and ended up building a separate server to run REDIS and now my canvas site is FLYING!!!!!!

To note for other people with similar issues, settings changed in redis.conf on the new REDIS server.

• Set maxmemory to about 80% of what is available.
• Set the maxmemory_policy to allkeys-lru
• Disabled saving "Redis to disk"