Running Couchbase Server + Sync Gateway on AWS

[Steve]

I have setup a Couchbase Server and Sync Gateway on Amazon Web Services (AWS).

Want to understand the recommended server and gateway setup when running a small number of Couchbase Servers, say between 3 and 10 servers?

For example, should I setup a Sync Gateway in front of each Couchase Server (or each cluster)? Or should I setup one or more Sync Gateways in front of only a subset of the total Couchbase Servers?

Where can I find info on the above in the Couchbase documentation?

Thanks
Steve

[ajres]

There are a couple of articles related to production deployment:

http://developer.couchbase.com/mobile/develop/guides/sync-gateway/deployment/index.html

http://developer.couchbase.com/mobile/develop/guides/sync-gateway/nginx/index.html

But nothing specifically about recommended server topologies.

On our demo cluster we have three CouchBase server instances in a single cluster, with two Sync Gateway instances connecting to the CBS cluster and we have a single nginx instance load balancing HTTP(S) requests across the two Sync Gateway instances.

Andy

[Steve]

OK, thanks Andy. I think I read in the doco each Sync Gateway can handle about 5000 users.

As I am hosting my Couchbase Server on AWS I was hoping to use their load balancer offering, though alternatively I could use Nginx I assume.

Thanks
Steve

[Christoph Berlin]

We have an ELB setup with Sync Gateway and DB - and its working fine. ELB is pretty limited when it comes to fine tuning and overall features but it works out of the box vs Nginx is much more sophisticated but requires maintenance as well as more EC2 instances which cost money. So it really comes down what you want...

Only one slight disadvantage is ELB's inability to handle long polling properly. It times out after 60sec which is configurable up to 1hr but still it drops the connection. Couchbase however reconnects fine whenever needed...

Our setup:

- 2 sync gateway behind an ELB in a public subnet

- 3 databases in a private subnet (call us old-fashioned :) )

[Todd Freese]

Trying to decide between ELB and Nginx as well.

Can you elaborate more on the ELB long polling issue? 

What issues does this cause with CBL on iOS?

Todd

[Christoph Berlin]

Well I am pretty sure that Andy, Jens or the team would be better suited to provide a holistic answer but it works fine for us. What the ELB does is to terminate any connection after a certain time no matter whether the client trying to keep it alive, by default 60sec. That is usually not an issue for typical use cases but long-poll applications such as Couchbase Sync Gateway or stuff like XMPP have a problem with that. They don't like being terminated.

Couchbase Mobile however does a good job of reconnecting when needed - we don't see any errors. But it is also important to keep in mind that our solution is very light on sync - most data resides on the mobile end.

Nginx on the other is a lot more sophisticated but also more complex to maintain.

Of course this all could be wrong if the CB team has a different opinion...

Thanks

[Sebastien ARBOGAST]

I'm trying to set up a Couchbase Sync Gateway cluster with Traun Leyden's tutorial: http://tleyden.github.io/blog/2014/12/15/running-a-sync-gateway-cluster-under-coreos-on-aws/

When I connect directly to the server where Sync Gateway is deployed, everything works fine, replication works both ways.

But when I connect to the ELB using a custom URL, push replication works fine, but pull replication fails with the following SyncVerbose log:

2015-01-11 20:13:24.926 MyApp[18492:4459783] Logging Sync, SyncVerbose to NSLog

2015-01-11 20:13:24.926 MyApp[18492:4459783] ### Couchbase Lite 1.0-Debug (unofficial) ###

2015-01-11 20:13:24.950 MyApp[18492:4459783] Sync: CBLReplication[to http://elb.myserver.com/mydb]: offline, progress = 0 / 0, err: (null)

2015-01-11 20:13:24.952 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb] STARTING ...

2015-01-11 20:13:24.991 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:unreachable (00), suspended=0

2015-01-11 20:13:24.994 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:reachable (3G) (40003), suspended=0

2015-01-11 20:13:24.994 MyApp[18492:4459783] Sync: CBLReplication[from http://elb.myserver.com/mydb]: offline, progress = 0 / 0, err: (null)

2015-01-11 20:13:24.994 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Going online

2015-01-11 20:13:25.001 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb] Progress: set active = 1

2015-01-11 20:13:25.001 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=1 (batch=0, net=1), online=1)

2015-01-11 20:13:25.002 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: GET _local/79b9c21cdbfadc67d1b19e68ad364d3485e171d9

2015-01-11 20:13:25.021 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=1 (batch=0, net=1), online=1)

2015-01-11 20:13:25.023 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb] STARTING ...

2015-01-11 20:13:25.058 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:reachable (3G) (40003), suspended=0

2015-01-11 20:13:25.065 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Going online

2015-01-11 20:13:25.069 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb] Progress: set active = 1

2015-01-11 20:13:25.069 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=1 (batch=0, net=1), online=1)

2015-01-11 20:13:25.071 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb]: GET _local/c8b86535d45db2aa1aa609149a640bacc7e4a407

2015-01-11 20:13:25.072 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=1 (batch=0, net=1), online=1)

2015-01-11 20:13:25.098 MyApp[18492:4459783] Sync: CBLReplication[to http://elb.myserver.com/mydb]: active, progress = 0 / 0, err: (null)

2015-01-11 20:13:25.100 MyApp[18492:4459783] Sync: CBLReplication[from http://elb.myserver.com/mydb]: active, progress = 0 / 0, err: (null)

2015-01-11 20:13:27.566 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Replicating from lastSequence=(null)

2015-01-11 20:13:27.571 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb] starting ChangeTracker: mode=3, since=(null)

2015-01-11 20:13:27.572 MyApp[18492:4459867] SyncVerbose: CBLWebSocketChangeTracker[0x17419c220 mydb]: GET //elb.myserver.com/mydb/_changes?feed=websocket

2015-01-11 20:13:27.576 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Replicating from lastSequence=6

2015-01-11 20:13:27.580 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: Received 0 revs

2015-01-11 20:13:27.581 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb] Progress: set active = 0

2015-01-11 20:13:27.581 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=0 (batch=0, net=0), online=1)

2015-01-11 20:13:27.582 MyApp[18492:4459783] Sync: CBLReplication[to http://elb.myserver.com/mydb]: idle, progress = 0 / 0, err: (null)

2015-01-11 20:13:28.008 MyApp[18492:4459867] CBLWebSocketChangeTracker[0x17419c220 mydb]: Connection error #1, retrying in 2.0 sec: The operation couldn’t be completed. HTTP/1.1 502 BAD_GATEWAY

2015-01-11 20:13:30.015 MyApp[18492:4459867] SyncVerbose: CBLWebSocketChangeTracker[0x17419c220 mydb]: GET //elb.myserver.com/mydb/_changes?feed=websocket

2015-01-11 20:13:30.516 MyApp[18492:4459867] CBLWebSocketChangeTracker[0x17419c220 mydb]: Connection error #2, retrying in 4.0 sec: The operation couldn’t be completed. HTTP/1.1 502 BAD_GATEWAY

2015-01-11 20:13:30.584 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:reachable (02), suspended=0

2015-01-11 20:13:30.591 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:reachable (02), suspended=0

2015-01-11 20:13:30.592 MyApp[18492:4459867] SyncVerbose: CBLWebSocketChangeTracker[0x17419c220 mydb]: GET //elb.myserver.com/mydb/_changes?feed=websocket

2015-01-11 20:13:30.916 MyApp[18492:4459867] CBLWebSocketChangeTracker[0x17419c220 mydb]: Connection error #3, retrying in 8.0 sec: nodename nor servname provided, or not known

2015-01-11 20:13:30.917 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:unreachable (00), suspended=0

2015-01-11 20:13:30.917 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: Going offline

2015-01-11 20:13:30.917 MyApp[18492:4459867] Sync: Stopping 0 remote requests

2015-01-11 20:13:30.918 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=1 (batch=0, net=1), online=0)

2015-01-11 20:13:30.918 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb]: ChangeTracker stopped; error=(null)

2015-01-11 20:13:30.918 MyApp[18492:4459867] Sync: CBL_Puller[http://elb.myserver.com/mydb] Progress: set active = 0

2015-01-11 20:13:30.919 MyApp[18492:4459867] SyncVerbose: CBL_Puller[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=0 (batch=0, net=0), online=0)

2015-01-11 20:13:30.919 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Reachability state = <elb.myserver.com>:unreachable (00), suspended=0

2015-01-11 20:13:30.919 MyApp[18492:4459783] Sync: CBLReplication[from http://elb.myserver.com/mydb]: offline, progress = 0 / 0, err: (null)

2015-01-11 20:13:30.919 MyApp[18492:4459867] Sync: CBL_Pusher[http://elb.myserver.com/mydb]: Going offline

2015-01-11 20:13:30.919 MyApp[18492:4459867] Sync: Stopping 0 remote requests

2015-01-11 20:13:30.920 MyApp[18492:4459867] SyncVerbose: CBL_Pusher[http://elb.myserver.com/mydb]: postProgressChanged (0/0, active=0 (batch=0, net=0), online=0)

2015-01-11 20:13:30.923 MyApp[18492:4459783] Sync: CBLReplication[to http://elb.myserver.com/mydb]: offline, progress = 0 / 0, err: (null)

The lines that end in 502 BAD GATEWAY seem to indicate that there's a problem with the pull websocket. Did you guys had to do anything to make ELB and Sync Gateway work together?

Note that I'm using the master version of both Sync Gateway and Couchbase-Lite iOS.