Page refresh with nginx load balancer

[Phuoc Do]

Hello,

Our app is reaching a point where we need to scale horizontally. We have been using AWS and decide to go with nginx load balancer. We basically have two nodejs servers running behind nginx load balancer. I managed to compile nginx with stick session module.

https://code.google.com/p/nginx-sticky-module/wiki/Documentation

It is running ok except one problem. Occasionally, meteor page continuously reloads. This generates request flood to our server. It doesn't happen all the time. When I reboot meteor server, it is back to normal. I haven't figured out how to reproduce the problem. I found meteor has one page reload here:

https://github.com/meteor/meteor/blob/0bb6b0bb4ce9d0e9a83aabdcad36792b5d381c2c/packages/reload/reload.js#L159

It seems like the problem is with meteor and nginx stick session. I know that nginx sticky session uses cookie. I wonder if that can conflict with meteor session tracking. Any insight would be appreciated.

Thanks,

Phuoc Do

[Jonas Aschenbrenner]

I had the problem in the past too. I think I solved it by:

1. Restart the meteor app after an update was deployed

2. Add the reload and autoupdate packes to .meteor/packages. The are included in the standard-app-packages package but I don't use it and I forgot to add them.

[Phuoc Do]

Hi Jonas,

Thanks. It seems like reload package is causing my app to reload the page. I wonder why adding it to .meteor/packages can fix the issue.

Phuoc

[Phuoc Do]

Hi Jonas, just gave your suggestion. I still get page refresh.

[Phuoc Do]

I found that meteor is trying to call reload. When I run locally, reload is never called. It seems to think that there is new code available on server. reload is called twice. I took 2 screenshots below. In the first time, function returns without refresh window. In the second time, reload is called from _maybeMigrate. Then page refreshes with location.reload.

https://drive.google.com/folderview?id=0B0ayFgn4x5Sccjd1QTdYdDNkNXc&usp=sharing

The code hasn't changed. Could it be that load balancer interferes with hot code reload check?

Phuoc

[Phuoc Do]

I've got nginx setup locally to debug the issue. It looks like my current code version is not the same as autoupdateVersion (see reload_code_version.png). So meteor reload kicks in to sync the code.

I check route cookies (see reload_route_cookies.png) and there are several of them. Browser is loading files from different servers. I suppose the problem is with multiple route cookies being assigned by nginx sticky session module. Maybe meteor doesn't submit all cookies with file request?

[Andrew Mao]

Because sticky uses cookies, it might break things. We had a much lesser problem using ip_hash, although equal balancing is not as guaranteed.

[Phuoc Do]

Hi Andrew, nginx sticky module warns not to use ip_hash. Is it working for you?

https://code.google.com/p/nginx-sticky-module/wiki/Documentation

I found a way to work around by setting AUTOUPDATE_VERSION environment variable. But that will disable hot code reload.

[Phuoc Do]

I changed sticky session to set cookie only on "/" path instead of letting browser decide. This seems to work. Thanks Arunoda for the suggestion. Here's my nginx config if anyone is interested:

https://gist.github.com/dnprock/53fa74153d29557aa699

[Andrew Mao]

I don't think you can use sticky with ip_hash at the same time. it is one or the other.

[Arunoda Susiripala]

It's possible receive a lot of requests from a single IP (due to transparent proxies and etc..)

So, there might be issues with load distributions.

Cookie based sticky sessions are the solution for that. Yes, it does fail over. Meteor trying to re-connet when the connection drops. 

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

On Fri, Jul 25, 2014 at 1:26 PM, Danijel Cole <dan.ba...@gmail.com> wrote:

Hmm...interesting idea. But are there any issues with using ip_hash to horizontally scale a Meteor site? 

Also any thoughts on what happens if a server node goes down...will subsequent meteor client request gracefully fail over to another meteor instance? 

Thanks 

--
You received this message because you are subscribed to the Google Groups "meteor-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-core...@googlegroups.com.
To post to this group, send email to meteo...@googlegroups.com.
Visit this group at http://groups.google.com/group/meteor-core.
For more options, visit https://groups.google.com/d/optout.

[Adrian Lanning]

For those exploring other options, HAProxy has a nice balancing option, 'leastconn':

  leastconn   The server with the lowest number of connections receives the
              connection. Round-robin is performed within groups of servers
              of the same load to ensure that all servers will be used. Use
              of this algorithm is recommended where very long sessions are
              expected, such as LDAP, SQL, TSE, etc... but is not very well
              suited for protocols using short sessions such as HTTP. This
              algorithm is dynamic, which means that server weights may be
              adjusted on the fly for slow starts for instance.

"This only applies when no persistence information is available, or when a connection is redispatched to another server",  which means this only kicks in when no sticky session cookie is found.

Source:  http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#balance

[Stephan Hochhaus]

There is a similar function in nginx called least_conn

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn

[Harshit Daga]

Hi Guys,

I am also facing similar issue. Request are going to multiple servers causing page continuous page reloads/refresh.

Config file : http://pastebin.com/ezxAqRiK

Server Nginx : http://pastebin.com/1TRicbyd

I am using DDP_DEFAULT_CONNECTION_URL since cloudflare is currently not supporting websockets. 

Thanks ,

 HD