How do I run automated jobs on separate node?

[Boris Kheyfets]

Hi all,

Production start notes say:

it is advisable to dedicate one of your app servers to be just a job server. You can do this by simply skipping the Apache steps on one of your app servers, and then only on that server follow these automated jobs setup instructions.

If I understand it correctly, this means I have to install canvas on two nodes. How do I set up canvas automated jobs node? If I am to set up it the same way as app node — how do I let it know to do the automated jobs for app node canvas?

Thanks in advance.

[Christopher Bennell]

A jobs node runs the "canvas_init" service. A web node runs apache. A single server can have both roles. 

The section of the production documentation titled "Automated Jobs" describes setting up and running the jobs service. You can skip this section for nodes that aren't running jobs, and you can skip the "Apache Configuration" section for nodes that aren't service web requests. So to set up a jobs node, follow the same steps as your web node, but don't set up apache. Then turn off the canvas_init service on your web node. 

This setup requires that some resources be external to the nodes themselves. Your database server needs to be accessible to all nodes, as does your file storage. There are lots of discussions on these boards about this kind of configuration, particularly posts by Graham Ballantyne. There are also videos from InstructureCon that you might find useful - search for the talks about open source. 

[Boris Kheyfets]

Thanks, Christopher.

As I understand it, on web node DocumentRoot (apache) [or root (nginx)] is actually located on remote machine, that is app machine. And that app machine has canvas installed, which in its turn might use posgresql, redis and cassandra all from remote machines.

[Christopher Bennell]

Web nodes don't have their documents roots on remote servers. Your web nodes have local copies of the application, and are running web servers. 

You can set up a load balancer in front of a number of web nodes, maybe this is what you're referring to?

[Boris Kheyfets]

I just can't understand if app is on web node, how can it jobs run on separate node? Probably I should really watch the InstructureCon talks you mentioned. (I spotted some on youtube)

--

---
You received this message because you are subscribed to a topic in the Google Groups "Canvas LMS Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/canvas-lms-users/pKuHQZGh7wo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to canvas-lms-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Graham Ballantyne]

Boris.

This is our architecture diagram for the setup we use at SFU. https://dl.dropboxusercontent.com/u/33854/Scaling%20Open%20Source%20Cavnas%20for%20Fun%20and%20Not%20Much%20Profit.key%20copy.pdf It's from my 2014 InstructureCon talk on scaling open-source Canvas.

App servers are the ones serving HTTP requests to Canvas (running apache). File servers are handling requests to our files domain (we split them up for performance reasons when we first started; it probably isn't required now but it still works so we haven't changed it). Both app and file pools are behind our load balancer. They're all configured the same and have local installs of Canvas.

We have two servers for running delayed jobs (the "management nodes" on the diagram. As Christopher mentioned, you want to run the canvas_init service on your delayed job node(s) - that service manages the pool of delayed job processors. We actually have apache running on those nodes, but they're not in any of our load balancer pools, so they never receive any traffic.

Your database, and file storage, need to be accessible to both your app and delayed job nodes. For files, if you're using S3, then this isn't an issue as long as your S3 config is set up right on both servers. If you're running local storage, like we do, you'll need to have some way of mounting that space on all your servers. We use a NFS export from our NetApp storage.

--

---
You received this message because you are subscribed to the Google Groups "Canvas LMS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to canvas-lms-use...@googlegroups.com.

[Boris Kheyfets]

Hi Graham,

thanks for the link to the video. Does your setup diagram means you have 22 nodes? (each with 4 Xeon proc and 8 Gb RAM, source)

[Graham Ballantyne]

Correct. 8 app nodes, 7 file nodes, two management (delayed jobs, enrollment scripts, etc) and five hot spares.

[Nick Kokkos]

Hi Graham, may I ask you a question?

From the figure you have posted, you have one master database Postgres only. That is, you don't have any slave databases running, right? So basically, you  have multiple nodes for the app and files and one node for database and everyone connects to the same database(no slaves?). Is this correct?

[Graham Ballantyne]

That's correct - we have one database server. It's currently a single point of failure, but one the powers that be are willing to live with. Our database has never been a bottleneck, and we have adequate snapshotting that I think if we suffered a catastrophic crash we'd lose at most an hour's worth of data (possibly as little as 15 minutes).