Generic connection issue when executing view

[Dump Hole]

Hello,

I am using Couchbase Community Edition 3.0.1 build 1444
I built a Ruby-Sinatra app with a Couchbase backend.

This error is occuring only on this specific instance of Couchbase that I have on AWS.  When tested on my development machine, everything seems to work okay.  This is the error I got from the stacktrace:

Couchbase::Error::Network - failed to execute HTTP request, Generic network failure. Enable detailed error codes (via LCB_CNTL_DETAILED_ERRCODES, or via `detailed_errcodes` in the connection string) and/or enable logging to get more information (key="_design/printers/_view/by_cpu_id?connection_timeout=75000&limit=1", error=0x10):

/../.bundle/ruby/2.1.0/gems/couchbase-1.3.10/lib/couchbase/view.rb:492:in `continue'
/../.bundle/ruby/2.1.0/gems/couchbase-1.3.10/lib/couchbase/view.rb:492:in `fetch_sync'
/../.bundle/ruby/2.1.0/gems/couchbase-1.3.10/lib/couchbase/view.rb:378:in `fetch'
/../.bundle/ruby/2.1.0/gems/couchbase-1.3.10/lib/couchbase/view.rb:205:in `first'


What is going on here and what does it mean by detailed_errcodes in the connection string?  This problem seems to occur only when there's actually a document found.  If I used a different key, which I know does not exist, it properly returns not found.

[Mark Nunberg]

As far as the detailed error codes go, this is a hint that you might be able to gain better information about the specific error code by enabling this feature.

I do not think the current version of the Ruby client exposes the connection string feature, so you can accomplish the same aim by setting LCB_OPTIONS="detailed_errcodes=1” in your environment (Sergey Avseyev, who maintains the Ruby client, might have a better idea of how to enable these extended codes).

Once the detailed error codes have been enabled, you should be able to see more details about why your request failed. There is also an option of logging (http://docs.couchbase.com/developer/c-2.4/logging.html).

Mark

--
You received this message because you are subscribed to the Google Groups "Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to couchbase+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dump Hole]

I added the LCB_OPTIONS.  It now says Timeout

Couchbase::Error::Timeout - failed to get value, Client-Side timeout exceeded for operation. Inspect network condition
s or increase the timeout (key="printer_1", error=0x17)

I can still access it through through port 8091 on my browser, and it works fine.

Could it be my AWS instance is too small?

[Dump Hole]

FYI, it didn't take long to timeout.  It displayed that message within 2-3 seconds.  I would assume timeout is set to 75 secs as specified by the connection_timeout parameter.

[Mark Nunberg]

It would indeed seem you are having issues with the AWS instance. In this case you seem to be timing out when performing a key-value operation (it seems you’re doing a get) which is an entirely different issue than what you were seeing before.

Adding logging would certainly help disambiguate most issues, in addition to LCB_OPTIONS, set LCB_LOGLEVEL=5 in your environment (it will log to stderr).

On Jan 15, 2015, at 9:33 AM, Dump Hole <dumpst...@gmail.com> wrote:

FYI, it didn't take long to timeout.  It displayed that message within 2-3 seconds.  I would assume timeout is set to 75 secs as specified by the connection_timeout parameter.

[Dump Hole]

Hi Mark!

Thank you for helping me.  Got more results back after enabling the LCB_LOGLEVEL=5

12931ms [I0] {9907} [DEBUG] (lcbio_mgr - L:416) <1.2.3.4:8092> (HE=0x39efd90) Creating new connection because none are available in the pool
12931ms [I0] {9907} [DEBUG] (lcbio_mgr - L:321) <1.2.3.4:8092> (HE=0x39efd90) Starting connection on I=0x39ef520
12931ms [I0] {9907} [INFO] (connection - L:441) <1.2.3.4:8092> (SOCK=0x39f02b0) Starting. Timeout=75000000us
12931ms [I0] {9907} [TRACE] (connection - L:258) <1.2.3.4:8092> (SOCK=0x39f02b0) Got event handler for new connection
12931ms [I0] {9907} [TRACE] (connection - L:305) <1.2.3.4:8092> (SOCK=0x39f02b0) Scheduling asynchronous watch for socket.
12961ms [I0] {9907} [TRACE] (connection - L:258) <1.2.3.4:8092> (SOCK=0x39f02b0) Got event handler for new connec
tion
12961ms [I0] {9907} [INFO] (connection - L:116) <1.2.3.4:8092> (SOCK=0x39f02b0) Connected
12961ms [I0] {9907} [DEBUG] (lcbio_mgr - L:271) <1.2.3.4:8092> (HE=0x39efd90) Received result for I=0x39ef520,C=0
x39f02b0; E=0x0
12961ms [I0] {9907} [DEBUG] (lcbio_mgr - L:223) <1.2.3.4:8092> (HE=0x39efd90) Assigning R=0x39efb20 SOCKET=0x39f0
2b0
12961ms [I0] {9907} [DEBUG] (ioctx - L:101) <1.2.3.4:8092> (CTX=0x39f0c80,unknown) Pairing with SOCK=0x39f02b0
12997ms [I0] {9907} [DEBUG] (ioctx - L:151) <1.2.3.4:8092> (CTX=0x39f0c80,mgmt/capi) Destroying. PND=0,ENT=1,SORC
=1
12997ms [I0] {9907} [INFO] (lcbio_mgr - L:491) <1.2.3.4:8092> (HE=0x39efd90) Placing socket back into the pool. I
=0x39ef520,C=0x39f02b0
12997ms [I0] {9907} [DEBUG] (lcbio_mgr - L:416) <1.2.3.4:11210> (HE=0x39f2bb0) Creating new connection because no
ne are available in the pool
12997ms [I0] {9907} [DEBUG] (lcbio_mgr - L:321) <1.2.3.4:11210> (HE=0x39f2bb0) Starting connection on I=0x39ef4c0
12997ms [I0] {9907} [INFO] (connection - L:441) <1.2.3.4:11210> (SOCK=0x2fc13d0) Starting. Timeout=2500000us
12997ms [I0] {9907} [TRACE] (connection - L:258) <1.2.3.4:11210> (SOCK=0x2fc13d0) Got event handler for new conne
ction
12997ms [I0] {9907} [TRACE] (connection - L:305) <1.2.3.4:11210> (SOCK=0x2fc13d0) Scheduling asynchronous watch f
or socket.
15500ms [I0] {9907} [ERROR] (server - L:476) <NOHOST:NOPORT> (SRV=0x392ab00,IX=0) Got error for connection! (OS=0)
15500ms [I0] {9907} [WARN] (server - L:374) <NOHOST:NOPORT> (SRV=0x392ab00,IX=0) Failing command (pkt=0x39f1590, opaqu
e=0, opcode=0x0) with error 0x17
15500ms [I0] {9907} [TRACE] (confmon - L:292) Start refresh requested
15500ms [I0] {9907} [ERROR] (connection - L:118) <1.2.3.4:11210> (SOCK=0x2fc13d0) Failed: lcb_err=0x17, os_errno=
0
15500ms [I0] {9907} [DEBUG] (lcbio_mgr - L:271) <1.2.3.4:11210> (HE=0x39f2bb0) Received result for I=0x39ef4c0,C=
(nil); E=0x17

Couchbase::Error::Timeout - failed to get value, Client-Side timeout exceeded for operation. Inspect network condition
s or increase the timeout (key="printer_1", error=0x17)

I just upgraded the instance to something slightly more beefy, but still got the error.  This time it's a brand new CB instance, not crowded with any documents.

[Mark Nunberg]

Judging from the logs, the problem seems to be establishing the actual TCP connection.

In other words, for this particular error this may not be so much an issue of resources as it is an issue of accessibility.

I forget if there’s a reference manual or something for AWS, but you should make sure that the hostnames you give to your cluster (i.e. the nodes in the cluster) should also be reachable from within the client. I’m guessing that ‘1.2.3.4’ is really a redacted version of the actual IP the client is trying to reach.

Can you make sure you can ping that address? Can you verify the given port is open?

[Dump Hole]

Oh dear, I feel so stupid now.  I forgot to open port 11210.  I opened just port 8091 and 8092.

Mark, thank you for your help!