Upgrading Self hosted Canvas LMS to latest version

[Anand Srinivasan]

Hello everyone!

What is the correct way to upgrade a self-hosted Canvas LMS to the latest version? We've tried to follow Upgrading · instructure/canvas-lms Wiki · GitHub, but have been encountering issues, especially in db:migrate.

To migrate DB, the following command was run, albeit with errors:

/var/canvas$ sudo RAILS_ENV=production bundle exec rake db:migrate
rake aborted!
Zeitwerk::NameError: wrong constant name Backbone-ext inferred by Module from directory

  /var/canvas/app/coffeescripts/backbone-ext

Possible ways to address this:

  * Tell Zeitwerk to ignore this particular directory.
  * Tell Zeitwerk to ignore one of its parent directories.
  * Rename the directory to comply with the naming conventions.
  * Modify the inflector to handle this case.


After this, tried again by modifying config/application.rb to have the following line:

config.autoloader = :classic, which was followed by a new error:

/var/canvas$ sudo RAILS_ENV=production bundle exec rake db:migrate
rake aborted!
NoMethodError: undefined method `ignore' for nil:NilClass

What is it that we're doing wrong? Any help is greatly appreciated here! Full log is attached herewith.

[Graham Ballantyne]

How old of a Canvas installation are you upgrading from?

We never overwrite the previous release; we deploy into individual release directories and move a “current” symlink that points to the active release. We also deploy every two weeks so that we’re not making huge changes at one time. 

– 
Graham Ballantyne 
Senior Systems Engineer | IT Services 
Simon Fraser University | Strand Hall 1001 
8888 University Dr., Burnaby, B.C. V5A 1S6

On Feb 21, 2022, at 09:18, Anand Srinivasan <writet...@gmail.com> wrote:

Hello everyone!

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/canvas-lms-users/fce3a23b-172f-4c0b-a8a1-4fabe1546456n%40googlegroups.com.

[Anand Srinivasan]

Hi Graham!

We deployed around 8 to 9 months ago, and this is our first upgrade attempt since then.

[Jacob Burroughs]

I will note that `app/coffeescripts/backbone-ext` no longer exists and hasn't existed since https://github.com/instructure/canvas-lms/commit/4243097b211ff97a7e02b28ffa24032ca49bd1e8 , so I'm thinking your upgrade process left old files around somehow.

[Graham Ballantyne]

I'm also seeing a Zeitwerk error in the attached logfile; it's something that Instructure started using internally and didn't really give a heads-up about. We had to figure out how to disable it in our environment by passing a   CANVAS_ZEITWERK=0 environment variable to Canvas when building, and to the app itself (both the web app and delayed jobs daemon). How you do that will depend on your environment. For building, it's just another variable to pass to the rake command (RAILS_ENV=production CANVAS_ZEITWERK=0 bundle exec....). For the web app, we use Apache with Passenger, so we're setting it in the Apache vhost config. For the jobs daemon, we manage that with systemd, so it's set in the systemd service file.

Graham Ballantyne 
Senior Systems Engineer —  IT Services 
Simon Fraser University

-- 

--- 
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/canvas-lms-users/546929cb-cd5c-4e5f-ab1a-daf7b011d95bn%40googlegroups.com.

[Anand Srinivasan]

Thanks a lot! We had indeed overwritten the old release files (in a Live-like setup), which could explain the left out folders, including `app/coffeescripts/backbone-ext`. Our existing deployment is on Ubuntu 20 LTS, with Apache, Passenger, Redis, psql 12.9, ...

I've a couple of follow-through questions:

1. @Graham: If we deploy into individual release directories once every two weeks into a backup folder, storage will be taken up eventually on the server. So I presume you delete the previous release files once they are old enough? (including logs)
2. Is disabling Zeitwerk something that's necessary to upgrade self-hosted canvas... If so, how to exactly do it in the Apache vhost config (canvas.conf..?) and systemd service file?

Drafted a rough sequence of actions to attempt the upgrade once again. Looking forward to your valuable feedback on this:

Upgrade procedure

1. Checkout the latest stable release branch to the active release folder in '/var/..'
2. Point the symlink '/var/canvas' to the the active release folder
3. Move the configuration files to the active release folder, as well as all attachments from '../tmp/files'
4. Do the following in the active release folder:
   1. bundle install
   2. yarn install
   3. Compile / generate assets (basically everything in  Generate Assets - Production Start · canvas-lms Wiki  )
   4. Transfer ownership of all the config (*.yml) files and assign read permissions to the canvasuser (or www-data)
   5. Migrate the DB
   6. Update notification types?  Notification types - Upgrading · canvas-lms Wiki
5. Restart all services

[Nico López]

Just my 2 cents: you could use symlinks for the user files (and even for the config files) as well. In our case we have the user files in a separate volume, so it's easier to do the backups / vm snapshots / volume snapshots, then using symlinks was a must.

[Graham Ballantyne]

This outlines our upgrade process: https://gist.github.com/grahamb/98e72098d5a96915421b#package-upgrade-process. It's several years old and some of the mechanisms have changed (e.g. we no longer use Capistrano), but the basics are the same. In general, we:

- Build a Canvas release on a separate build box (check out repos, bundle & npm install, compile assets) and generate an artifact (a tarball of that release). This ensures we're deploying the same built Canvas to all our environments. 

- Deploy that artifact through our various test/stage/production environments (previously via Capistrano, now with Ansible).

- Code is pushed to servers (we have 20 app servers, two jobs servers) into a release directory (e.g. /var/rails/canvas/releases/20220217181417-sfu-release-2022-02-17-5 for our current release)

- We use a shared NFS mount for the Canvas files storage and symlink it into the release directory

- We rebuild brand assets on one of the jobs servers, copy those to another shared NFS mount, and symlink into public/dist/brandable_css

- Database migrations are run on one of our management (jobs) servers

- We have a "current" symlink (/var/rails/canvas/current) that we use as the Apache DocumentRoot (/var/rails/canvas/current/public to be precise). Passenger picks that up as the app root. As part of the deploy, we re-point that symlink at the new release.

- We have template config files for each environment and copy them into the config directory.

The Ansible playbook we use to deploy is here; tasks are executed in sequence.

1. @Graham: If we deploy into individual release directories once every two weeks into a backup folder, storage will be taken up eventually on the server. So I presume you delete the previous release files once they are old enough? (including logs)

We keep five releases on disk (current and four previous). I don't think we've ever rolled back to one in nearly 10 years. For logs, we have Rails logging to syslog. Syslog on each individual server is forwarded to our central logging service, no logs on local disk on those machines (technically they're in /var/log/messages I guess but that gets rotated by syslog). As well, the app servers forward the Canvas syslog facility to one of our Canvas management (jobs) servers which aggregates all of the web and jobs logs into one log file on disk; it's the firehose of all Canvas app logs. We rotate that file daily, but it is also forwarded to our central logging service, where we've got at least a year, probably more, of archived logs if we ever need them.

1. Is disabling Zeitwerk something that's necessary to upgrade self-hosted canvas... If so, how to exactly do it in the Apache vhost config (canvas.conf..?) and systemd service file?

It shouldn't be; we ran into issues with it because of some customizations we've done, and I've been told that Zeitwerk is going to be required so it's something we'll have to address on our end.

– 

Graham Ballantyne 
Senior Systems Engineer —  IT Services 
Simon Fraser University

To view this discussion on the web visit https://groups.google.com/d/msgid/canvas-lms-users/061ac704-5c42-4825-a391-fdeac622cff5n%40googlegroups.com.

[Anand Srinivasan]

Thanks Graham, for the posted upgrade process!

We are facing an error while trying to compile assets:

/var/canvas$ sudo RAILS_ENV=production bundle exec rake canvas:compile_assets
Could not find json-2.6.1 in any of the sources
Run `bundle install` to install missing gems.

Running 'bundle install' doesn't solve the problem. Any leads..?

[Anand Srinivasan]

We had to settle finally with modifying the gem requirement to json-2.5.1, in app.rb. Not sure if it was the proper way, but was able finally upgrade successfully in a Live like environment. Thank you :)

[Nico López]

Hey Graham, would you mind sharing with us the size of your redis servers (cpu/ram) and how did you do the sizing of it (or how would you estimate it)? Same for the database server. 

You know you are our guru for "big implementations" :)

[Graham Ballantyne]

Redis: 4 VMs each with 2 vCPUs and 12GB RAM

Postgres: 4 VMs (1 primary, 3 replication secondaries, 1 watchdog). Primary and secondaries are 8 vCPU, 64GB RAM each. The secondaries are configured using repmgr for automatic failover; the witness is there to prevent a split-brain condition (we have the secondaries split between two different datacentres).

– 
Graham Ballantyne 
Senior Systems Engineer —  IT Services 
Simon Fraser University

To view this discussion on the web visit https://groups.google.com/d/msgid/canvas-lms-users/4c2474db-bc9a-4834-9681-7a93d72659e5n%40googlegroups.com.

[Nico López]

Thanks!