Multiple application servers installation questions

834 views
Skip to first unread message

Trương Hoàng Dũng

unread,
Mar 31, 2012, 11:19:19 AM3/31/12
to canvas-l...@googlegroups.com
Could you please explain more about the installation with multiple application servers ? It's like how to set up a produciton environment on two virtual machines with small RAM, and how can we set up load balancing between them ? Another case, is that how to set up the queing jobs on seperate server ?
Thank you for reading.

Christopher Bennell

unread,
Apr 2, 2012, 9:53:58 AM4/2/12
to canvas-l...@googlegroups.com
In our experience, there aren't really any major differences between a standalone server and multiple load-balanced servers. The application servers are configured in the same way.

There is no special configuration needed for the delayed jobs, they work across servers out-of-the-box. If you want to have a separate server that is only running the delayed jobs, do a normal production install, but skip the apache web server stage. 

We use nginx for load balancing. It's very lightweight and easy to configure. So our DNS points to canvas.myschool.com, which is the load balancer running nginx, which is a reverse proxy to the application servers. 

The other consideration is storage: it has to be shared across the app servers. The most straightforward way to do this is to use Amazon S3. That's not an option for us for a few reasons, so we set up shared storage on a local SAN, using iSCSI and OFS2.

We also use Puppet, Capistrano and a local standalone Git repo to reduce some of the complexity associated with managing multiple servers and frequent updates.

dres

unread,
Apr 2, 2012, 10:56:33 AM4/2/12
to canvas-l...@googlegroups.com
Could you possible share your puppet conf?

Christopher Bennell

unread,
Apr 2, 2012, 3:53:47 PM4/2/12
to canvas-l...@googlegroups.com
I've been thinking about posting it. It's kind of specific to our environment but could, of course, be modified. It's needs a bit of work to get it in shape to share but I'll try to get it posted soon. 

revskill

unread,
Apr 2, 2012, 7:32:32 PM4/2/12
to canvas-l...@googlegroups.com
Thank you very much sir. We really need the nginx configuration for load balancing https and reverse proxy. If you could , please get more details on:
- The nginx configuration on smart load balancing with sticky session.
- The local SAN share procedure on app server ( mount, symlink,.. ? ) , i'm sorry but i have only little Linux experience so far.
- I heard of MySQL proxy, but i think MySQL replication is also good. I love the feature read-write splitting of MySQL proxy but still no try it.



2012/4/3 Christopher Bennell <tunfor...@gmail.com>

I've been thinking about posting it. It's kind of specific to our environment but could, of course, be modified. It's needs a bit of work to get it in shape to share but I'll try to get it posted soon. 



On Monday, April 2, 2012 10:56:33 AM UTC-4, dres wrote:
Could you possible share your puppet conf?



--
TRUONG HOANG DUNG
Librarian Researcher
Information and Library Centre
Mobile: 0121.411.5322
Email: dun...@hpu.edu.vn

Hai Phong Private University

Christopher Bennell

unread,
Apr 3, 2012, 11:22:42 AM4/3/12
to canvas-l...@googlegroups.com
Nginx load balancing is ridiculously easy:
  
server {
    server_name canvas.myschool.com;
    listen 443 default_server ssl;
    location / {
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Ssl on;
      proxy_set_header X-Forwarded-Proto https;
      proxy_pass http://canvas;
    }

upstream canvas {
}


There are many options for changing the balancing behaviour. More documentation for the proxy_pass module here: wiki.nginx.org/HttpProxyModule. You don't need sticky sessions because sessions are stored in the DB.

We have a MySQL cluster on the local network. It's not really my area, so I can't speak to the specifics here. 

We mount the SAN at /mnt and symlink tmp/files to /mnt/canvas. This step is part of the capistrano deploy. The iSCSI setup is a bit involved, may not be the most straightforward way to do it. 


On Monday, April 2, 2012 7:32:32 PM UTC-4, Trương Hoàng Dũng wrote:
Thank you very much sir. We really need the nginx configuration for load balancing https and reverse proxy. If you could , please get more details on:
- The nginx configuration on smart load balancing with sticky session.
- The local SAN share procedure on app server ( mount, symlink,.. ? ) , i'm sorry but i have only little Linux experience so far.
- I heard of MySQL proxy, but i think MySQL replication is also good. I love the feature read-write splitting of MySQL proxy but still no try it.



I've been thinking about posting it. It's kind of specific to our environment but could, of course, be modified. It's needs a bit of work to get it in shape to share but I'll try to get it posted soon. 



On Monday, April 2, 2012 10:56:33 AM UTC-4, dres wrote:
Could you possible share your puppet conf?



--
TRUONG HOANG DUNG
Librarian Researcher
Information and Library Centre
Mobile: 0121.411.5322

Trương Hoàng Dũng

unread,
Apr 8, 2012, 10:04:47 AM4/8/12
to canvas-l...@googlegroups.com
I've installed Canvas, but i have a problem with delayed_jobs. The delayed_jobs.log say: undefined method 'logger' for Rails::Module (NoMethodError).
Do you know how to solve this problem ?
Thank you very much for reading.

Regis Millet

unread,
Apr 10, 2014, 8:39:08 AM4/10/14
to canvas-l...@googlegroups.com
Hi! 

I know it's been a while since the first question was asked, but as we encounter a problem with Nginx and load balancing, we prefer to post our question on a topic related to this subject. 

So... We deployed two canvas applications on two different physical servers. They both use the same database (a postgresql one), same redis, same delayed jobs, same S3 bucket. All their configurations are identical. We managed to make the load balancing work on our Nginx server and it was redirecting to one server or the other. 

But! We encounter a strange behavior: it seems that sometimes some pages were not fully loading, such as the calendar, discussions, people... (in fact pages that involve an ajax request, apparently). Sometimes the page were loading correctly sometimes they were not. 

I tried to study the logs, no errors appeared while I was browsing those pages. I also tried to use both applications by directly accessing them using their "ip:port" address, and everything worked fine, even for the pages that were not loading. 

I did not find any errors anywhere... Did someone have the same problem when using load balancing with Nginx?

Also... as we store our files on amazon S3, do we still need to "share" the tmp folder across our applications? 

Did we miss some configurations? We used the same one as Christopher Benell but added ip_hash to the upstream module. Was it a bad idea?

I am ready to give any information needed to help me find the solution to this strange problem. Also I'm ready to write a how to on load balancing as soon as mine is working. 

Thanks!

Yannick Petit

unread,
Apr 18, 2014, 7:04:18 AM4/18/14
to canvas-l...@googlegroups.com
Hello all,

I'm also very interested by your insights on Régis questions. Any profesional load balancers around ?

Many thanks,

Yannick

Christopher Bennell

unread,
Apr 21, 2014, 10:34:49 AM4/21/14
to canvas-l...@googlegroups.com
We've never seen any of the problems you've mentioned. I would suggest figuring out exactly what's going wrong with your page loads. Are you getting 404 errors on any JS or CSS files? If so, maybe something went wrong with the "compile assets" step. Are you getting 500 errors on ajax requests? If so, the error report ID will be included in the JSON response, which would be a solid clue. 

Why are you using IP hash? It shouldn't be necessary.

Michael Rowe

unread,
Apr 22, 2014, 8:29:38 AM4/22/14
to canvas-l...@googlegroups.com
I use Amazon Web Services to utilize multiple application servers. I am still fine-tuning it, but here's my basic setup:

EC2 - Cloud Server Instances, each instance also handles delayed jobs (also supports auto-scaling)
Load Balancer - distributes traffic to EC2 instances and handles SSL
RDS - Postgres Database Server
ElastiCache - Redis Datastore (I currently also run Redis on each EC2 instance for the cache store)
S3 - File storage
SES - "Simple Email Service" to send email
VPC - Virtual Private Cloud creates a private network for all the services to connect to each other

It took me a while to figure out all the ins and outs of AWS, but once you get it down it's not too bad. I believe Instructure also uses AWS, but I'm not sure how there setup is.

Christopher Bennell

unread,
Apr 22, 2014, 11:23:02 AM4/22/14
to canvas-l...@googlegroups.com
What is the reasoning behind running Redis on your app servers? Doesn't having separate instances of Redis means that your cache isn't shared across your servers? I would think that it would be more efficient in terms of memory consumption to have one central cache. Just curious :)

Michael Rowe

unread,
Apr 22, 2014, 2:02:35 PM4/22/14
to canvas-l...@googlegroups.com
The main reason at this point is cost. The ElastiCache server prices go up pretty quickly the more storage you get. My understanding is the data store is small, but has some semi-volatile session data which is why I have it in ElastiCache. I am using persistent connections through the load balancer so having the cache store on each server instance, while maybe not 100% efficient, is still pretty effective I think.

If anyone has more details on the data store vs. the cache, I'd be glad to see if there would be a better setup.

mani

unread,
Apr 28, 2014, 12:13:25 PM4/28/14
to canvas-l...@googlegroups.com
Hi Michael,

Could you please consider writing a doc similar to the production start on how to setup canvas on AWS so everyone here can benefit from your experience?

Regards,

Regis Millet

unread,
May 6, 2014, 8:15:25 AM5/6/14
to canvas-l...@googlegroups.com
Hi Christopher and many thanks for your answer. 

In fact I already thought about errors with Ajax requests. But I don't receive any errors. I've tracked all the requests using the Chrome's console and did not find any aborted request. 
That's why I was asking for help as I found this problem is very peculiar.

I was using ip_hash because of what the Nginx documentation says: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash

It seems to redirect each client to the same server according to its IP. I thought that it woulld improve the performance of my load balancing. 

Do you think this may be the origin of the bug? I should try the configuration by removing the ip_hash. 

Thanks again,
Regis. 

Christopher Bennell

unread,
May 7, 2014, 9:38:29 AM5/7/14
to canvas-l...@googlegroups.com
I don't think that this would cause the problem you're seeing, but I don't think it would enhance performance. I may be wrong here, but I think the difference between using ip_hash or not would be pretty negligible. Canvas store session info in the database, so as long as your DB, redis and storage are shared across servers, it shouldn't really make a difference. But you could always try to turn it off and see what happens. 

Regis Millet

unread,
Jul 16, 2014, 12:58:57 PM7/16/14
to canvas-l...@googlegroups.com
Ok, we just tried again to put load balancing in place using Nginx. 

We figured out that we indeed had an issue while loading some JS files, here it is:

net::ERR_INCOMPLETE_CHUNKED_ENCODING

Then I've checked the NGinx logs, and found an error: 

failed(13: Permission denied) while reading upstream
Thanks to that I googled the error and found this helpful article: 


I think it solved the problem, I'll let you know if it did not. 

Regis Millet

unread,
Jul 17, 2014, 9:49:17 AM7/17/14
to canvas-l...@googlegroups.com
Hello all :)

I seem to have a brand new problem with load balancing (I think it is related to it as I never had the problem before). 

Sometimes, when I try to access an external application which uses the oauth authentication, I get this error from the javascript console: 

Refused to display 'https://myapp.org/login/oauth2/confirm' in a frame because it set 'X-Frame-Options' to 'SAMEORIGIN'.

Of course I'm loading this application from the same webdomain (myapp.org) within a task in a course. It means that the browser doesn't recognize that the page "oauth2/confirm" is from the same domain... it strange isn't it? 
Did anyone already encounter this issue? 

I think it may be related to the "different ips" used by the load balancing. I saw that we can add "@embedable = false" in the pseudonym_sessions "oauth2_confirm" action to prevent the application from adding this header, but I am wondering if it is a good idea. 

If anyone could help, that would be much appreciated :)
Thanks! 

Joel Lopez

unread,
Jan 4, 2017, 8:03:20 PM1/4/17
to Canvas LMS Users
Hi,

I'm having trouble getting the SES email working on my EC2 setup.  Did you have to open ports?  Are you using plain, login or cram_md5 authentication?  How'd you set it up?  Thanks.
Reply all
Reply to author
Forward
0 new messages