Canvas - Extremely Slow

[kr...@novaware.com.au]

Hi Guys,

I am having some major issues with a Canvas installation being extremely slow and has been this way for awhile now. It doesn't seem to be a memory issue at all, we are currently running on a 8GB VM with 5GB free.

Does anyone have any experience on where the issue may lay?

I appreciate any help! :)

[Graham Ballantyne]

Is redis running? Without caching a Canvas will be very slow. 

Ant pythons interesting in your logs?

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Graham Ballantyne]

Err, that should have been "anything interesting in your logs?" Ducking autocorrect. 

On Mar 26, 2014, at 21:23, kr...@novaware.com.au wrote:

--

[Kristopher Pickering]

Redis seems to be up and running fine. It seems to be running smoother now but not snappy by any means. Does SSL have a big effect on the speed or am I imagining things?

--

---
You received this message because you are subscribed to a topic in the Google Groups "Canvas LMS Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/canvas-lms-users/hDvRdigXVjw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to canvas-lms-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Kristopher Pickering Manager

P  03 8689 0056  |  M  0420 946 323  |  E kr...@novaware.com.au |  W  www.novaware.com.au

This communication may contain confidential or copyright information of Novaware (ABN 88 129 345 162).
If you are not an intended recipient, you must not keep, forward, copy, use, save or rely on this communication, and any such action is unauthorised and prohibited. If you have received this communication in error, please reply to this email to notify the sender of its incorrect delivery, and then delete both it and your reply.

[Graham Ballantyne]

We terminate SSL at our load balancer, but even if you're not doing something like that SSL shouldn't have that much of an impact.

Canvas has never been particularly memory-bound for us; most of our VMs are 4-core, 6GB machines with lots of free RAM. Where things seem to get bogged down for us is CPU; we've seen times where every machine in our cluster (8 VMs serving the web domain traffic; there's a bunch more for the files domain and delayed jobs) will peg out at 80%+ CPU for sustained periods. We ended up narrowing that down to a specific account that was enrolled in many thousands of courses (most of them inactive courses imported from our old WebCT installation). Whenever that account tried to log in (and it usually couldn't), Canvas would go nuts and become unresponsive.

That's not to say that you're having the exact same issue; it's just to say that performance problems are often difficult to track down. We only found the cause of that one by looking through the logs, running strace on the ruby processes to see what they were actually doing, noticing that there was a huge amount of traffic back and forth between the app server and our redis cluster (transferring tens of megabytes of cached data per request for this account) and seeing a common user ID in the requests.

If you can give some more information on your setup we might be able to help more. A couple of questions for you:

- You said "we are currently running on a 8GB VM with 5GB free". Are you running on a single VM? Is that VM also handling redis, database and delayed jobs, or are those on their own machines? If you're only running one server, come to my Instructurecon talk on scaling open-source Canvas ;)

- How many active users, courses, etc do you have? Any idea as to your numbers of concurrent users (we use Google Analytics to get a rough idea for the latter).

And a question for the Instructurefolk -- I asked on IRC a week or so ago but only got crickets: do you use anything like New Relic to instrument your production servers? Like I said, we only tracked down the root cause of our "account with too many courses" problem by one of our very experienced systems guys running strace on individual ruby threads and being able to actually interpret the strace output. 

-- 

Graham Ballantyne

IT Services

Simon Fraser University

[Steve Hillman]

Redis definitely has a huge impact on performance, so make sure Canvas is using it and that it's operating properly. Other things to look for:

- If you use Phusion Passenger to handle the connection from Apache/Nginx to Rails, check the passenger queue (as root, "passenger-status"). It should always be 0. If it's non-zero, it means you have requests waiting on Rails processes to be free. If that's the case, it probably means your pool of processes is too small for the number of requests you're getting. Each Rails process can only handle one request at a time.

- Do you always ensure there are some idle Rails processes waiting for requests? (similar to the point above, but from a different angle) - starting up a new process is very CPU intensive, so if Passenger has to spin up a new process to handle a request, this can add multiple seconds to the time it takes to handle that request. 

- Rails logs how long each requests takes to process. The numbers are in milliseconds and most requests should rarely take more than 200-300ms. If they do, something is making the Rails processes bog down (note that these times don't include the amount of time the requests waited in the Apache/Passenger queue)

Of course, there's all the standard system-level things to check for too -- are your CPUs maxed out? Are you slamming the disk and making your processes wait for disk I/O?

---

Steve Hillman        IT Architect
hil...@sfu.ca       IT Services
778-782-3960         Simon Fraser University

[Kristopher Pickering]

Currently everything is running on a single VM. Which includes redis, database and delayed jobs. The talk sounds great for the talk, see if I can fly from here in Australia haha.

Active users would be around 70, never more then that. But it doesn't seem to be tied to the amount of users that are currently connected at all.

Steve, I will have a look at those things and see if they can help me narrow it down!

[Charles Wood]

I try not to push my consulting services on this list, but I am willing to jump on the server with you in tow to troubleshoot if you need me to. Just contact me offlist.

Charles Max Wood

Lead Rails/iPhone Developer

Intentional Excellence Productions - http://iephq.com

Show Host

Ruby Rogues - http://rubyrogues.com

Javascript Jabber - http://javascriptjabber.com

Freelancers' Show - http://freelancersshow.com

iPhreaks Show - http://iphreaksshow.com

[Christopher Bennell]

You might also want to make sure that Rails isn't running in development mode; this would make it run much more slowly. This is controlled via a passenger setting in the apache configuration. The default for this setting is "production", so I doubt that this is the case. Just covering all of the bases. 

[Kristopher Pickering]

Thanks Charles! I'll keep you in mind in the future :)

[Kristopher Pickering]

Thanks for all the helps guys, I've seen Canvas have huge improvements recently with some tweaks I used from your comments. :)