Couchbase Timeout on first hit from syncgateway

[Michael Mayer]

We have a mobile application that uses the Sync Gateway to keep documents up to date.  That same application also has a web site that can write data. In order to keep everything synced properly, the website access the coucbase data via the Sync Gateway also.  The first time we attempt to write to the sync gateway from the website, we receive the following errors in the log.

10:23:45.618608 HTTP+: #154:     --> 201   (62.1 ms)
10:23:45.641556 HTTP:  #155: PUT /<coubhbasebucket>/groupdata::15::2015::03::09?rev=  (ADMIN)
10:23:45.785649 HTTP+: #155:     --> 201   (144.1 ms)2015/03/09 10:23:49 go-couchbase: call to Do("groupdata::08::2015::03::09") in github.com/couchbaselabs/go-couchbase.(*Bucket).casNext took 18.9097081s
10:23:49.848933 WARNING: Couldn't interpret error type *net.OpError, value WSARecv tcp 10.10.3.39:61617: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. -- base.ErrorAsHTTPStatus() at error.go:63
10:23:49.852932 HTTP: #143:     --> 500 Internal error: WSARecv tcp ##.##.#.##:61617: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.  (18918.7 ms)

We have set up a ping to perform a GET against the sync gateway every 30 minutes in an attempt to keep the connection awake but this does not seem to have resolved the put issue.  We do not receive timeouts any more on the GET's.  Is there anything we can do to prevent the first PUT from failing?  Something we can keep alive on the Couchbase server?

[Jens Alfke]

On Mar 9, 2015, at 8:30 AM, Michael Mayer <mi...@themayerhouse.com> wrote:

10:23:45.785649 HTTP+: #155:     --> 201   (144.1 ms)2015/03/09 10:23:49 go-couchbase: call to Do("groupdata::08::2015::03::09") in github.com/couchbaselabs/go-couchbase.(*Bucket).casNext took 18.9097081s
10:23:49.848933 WARNING: Couldn't interpret error type *net.OpError, value WSARecv tcp 10.10.3.39:61617: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. -- base.ErrorAsHTTPStatus() at error.go:63

The Sync Gateway is having serious problems communicating with Couchbase Server — the errors above are from the SDK that the Gateway uses to talk to the database. 

The first shows an extremely long time to service a request (19 sec!), and the second looks like a timeout establishing a TCP connection.

I think something’s seriously wrong with either the network connectivity from SG to the database cluster, or with the cluster itself.

—Jens

[ajres]

Are the Sync Gateway and Couchbase Server nodes on the same LAN, or do you have a network device (firewall ?) in between that could be terminating connections?.

[Traun Leyden]

Also, is the couchbase server under heavy load?

On Mon, Mar 9, 2015 at 8:38 AM, ajres <an...@couchbase.com> wrote:

Are the Sync Gateway and Couchbase Server nodes on the same LAN, or do you have a network device (firewall ?) in between that could be terminating connections?.

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/896b1d29-dbd2-4d7a-aa4e-e0bda7ca59b5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Mayer]

I will check with our network team and see if they can see any problems.  Our Sync Gateway is in the DMZ and the CB Cluster is behind the firewall.  The CB cluster is definitely not under heavy load.  It is just bizarre to me that after the first put everything seems to work fine.  I will let you know what the network team finds.

Thanks for the suggestions.

[Michael Mayer]

Can someone confirm exactly what ports I should open through the firewall from the Sync Gateway to the Couchbase cluster?

[Michael Mayer]

I have confirmed we have the following ports open:
8091
8092
11209
11210
11211

The network team can not think of any reason there would be a delay on the network as experienced in the error message.  They indicated it would either be blocked or not in the firewall rules.  I have not been able to recreate the issue today but I will try again tomorrow. It only seems to happen the first time we issue a PUT to the sync gateway.  Are there any couchbase configuration issues you can think of that would be causing this.  Our environment is currently under VERY low utilization.

[Jens Alfke]

On Mar 9, 2015, at 12:29 PM, Michael Mayer <mi...@themayerhouse.com> wrote:

Are there any couchbase configuration issues you can think of that would be causing this.

Probably your best option is to go ask on the Couchbase Server forums. There are people there with deeper knowledge of Couchbase Server configuration and troubleshooting.

—Jens