Cassandra port forwarding: TSocket read 0 bytes

Svend Vanderveken

unread,

Jan 14, 2014, 2:48:47 PM1/14/14

to vagra...@googlegroups.com

Hi all,

(not sure if this is a Cassandra or Vagrant difficulty, so I'm starting here, sorry in advance if it's not appropriate)

I am failing to connect with cqlsh to a Cassandra DB installed on Ubuntu 12.04 running on virtual-box 4.1.16 created/started through Vagrant 1.4.2.

I have this issue with both Cassandra 1.2 and Cassandra 2.

From within the box, all is fine, Cassandra is running and I can access it:

"

vagrant@precise64:~$ cqlsh

Connected to Test Cluster at localhost:9160.

[cqlsh 3.1.7 | Cassandra 1.2.11 | CQL spec 3.0.0 | Thrift protocol 19.36.1]

Use HELP for help.

"

I configured this port forwarding:

"

config.vm.network :forwarded_port, guest: 6627, host: 6627 # storm Nimbus Thrift port

config.vm.network :forwarded_port, guest: 7777, host: 7777 # storm UI

config.vm.network :forwarded_port, guest: 2181, host: 2181 # zk

config.vm.network :forwarded_port, guest: 7000, host: 7000 # Cassandra cluster

config.vm.network :forwarded_port, guest: 9160, host: 9160 # Cassandra Thrift client

config.vm.network :forwarded_port, guest: 9042, host: 9042 # Cassandra CQL client

config.vm.network :forwarded_port, guest: 7199, host: 7199 # Cassandra jmx

"

as well as this:

"

config.vm.provider :virtualbox do |vb|

vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]

end

"

Port forwarding for 7777 is successful: it's an http server I can access from a browser on my host.

Starting cqlsh on the host machine however fails like this

"

$./cqlsh

Connection error: TSocket read 0 bytes

$./cqlsh localhost 9160

Connection error: TSocket read 0 bytes

"

It seems localhost 9160 is ok though since using another (incorrect) port yields another error:

"

(alabs)Svends-iMac:bin svend$ ./cqlsh localhost 1234

Connection error: Could not connect to localhost:1234

"

A quick google mentions this is a "cassandra server side error", but no message appear on the cassandra log during my connection attempt.

I suspect it might be related to this Cassandra config param:

"

listen_address: localhost

"

But if so, what value to use instead? If not, where to look for error information?

Thanks a lot in advance for you help!

Svend Vanderveken

unread,

Jan 15, 2014, 6:08:59 AM1/15/14

to vagra...@googlegroups.com

Some progress:

checking tcp connections before and after a successful cqlsh from within the guest box reveals that cqslh actually uses two ports: 9160 + a dynamic one (44500 in the example bellow)

before cqlsh connection:

"

vagrant@precise64:/var/log/supervisor$ sudo netstat -anltp

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 556/rpcbind

tcp 0 0 0.0.0.0:60625 0.0.0.0:* LISTEN 1488/java

tcp 0 0 127.0.0.1:9042 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 646/sshd

tcp 0 0 127.0.0.1:7000 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:48354 0.0.0.0:* LISTEN 1488/java

tcp 0 0 127.0.0.1:9160 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:52488 0.0.0.0:* LISTEN 602/rpc.statd

tcp 0 0 10.0.2.15:22 10.0.2.2:51471 ESTABLISHED 1929/sshd: vagrant

tcp 0 0 10.0.2.15:22 10.0.2.2:51435 ESTABLISHED 876/sshd: vagrant [

tcp6 0 0 :::111 :::* LISTEN 556/rpcbind

tcp6 0 0 :::41585 :::* LISTEN 1485/java

tcp6 0 0 :::22 :::* LISTEN 646/sshd

tcp6 0 0 :::40509 :::* LISTEN 602/rpc.statd

tcp6 0 0 :::7777 :::* LISTEN 1486/java

tcp6 0 0 :::6627 :::* LISTEN 1487/java

tcp6 0 0 :::2181 :::* LISTEN 1485/java

tcp6 0 0 127.0.0.1:2181 127.0.0.1:58557 ESTABLISHED 1485/java

tcp6 0 0 127.0.0.1:58557 127.0.0.1:2181 ESTABLISHED 1487/java

tcp6 0 0 127.0.0.1:2181 127.0.0.1:58555 ESTABLISHED 1485/java

tcp6 0 0 10.0.2.15:7777 10.0.2.2:51459 ESTABLISHED 1486/java

tcp6 0 0 127.0.0.1:58555 127.0.0.1:2181 ESTABLISHED 1480/java

"

after cqlsh connection:

"

vagrant@precise64:/var/log/supervisor$ sudo netstat -anltp

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 556/rpcbind

tcp 0 0 0.0.0.0:60625 0.0.0.0:* LISTEN 1488/java

tcp 0 0 127.0.0.1:9042 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 646/sshd

tcp 0 0 127.0.0.1:7000 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:48354 0.0.0.0:* LISTEN 1488/java

tcp 0 0 127.0.0.1:9160 0.0.0.0:* LISTEN 1488/java

tcp 0 0 0.0.0.0:52488 0.0.0.0:* LISTEN 602/rpc.statd

tcp 0 0 127.0.0.1:9160 127.0.0.1:44500 ESTABLISHED 1488/java

tcp 0 0 10.0.2.15:22 10.0.2.2:51471 ESTABLISHED 1929/sshd: vagrant

tcp 0 32 10.0.2.15:22 10.0.2.2:51435 ESTABLISHED 876/sshd: vagrant [

tcp 0 0 127.0.0.1:44500 127.0.0.1:9160 ESTABLISHED 2054/python

tcp6 0 0 :::111 :::* LISTEN 556/rpcbind

tcp6 0 0 :::41585 :::* LISTEN 1485/java

tcp6 0 0 :::22 :::* LISTEN 646/sshd

tcp6 0 0 :::40509 :::* LISTEN 602/rpc.statd

tcp6 0 0 :::7777 :::* LISTEN 1486/java

tcp6 0 0 :::6627 :::* LISTEN 1487/java

tcp6 0 0 :::2181 :::* LISTEN 1485/java

tcp6 0 0 127.0.0.1:2181 127.0.0.1:58557 ESTABLISHED 1485/java

tcp6 0 0 127.0.0.1:58557 127.0.0.1:2181 ESTABLISHED 1487/java

tcp6 0 0 127.0.0.1:2181 127.0.0.1:58555 ESTABLISHED 1485/java

tcp6 0 0 10.0.2.15:7777 10.0.2.2:51459 ESTABLISHED 1486/java

tcp6 0 0 127.0.0.1:58555 127.0.0.1:2181 ESTABLISHED 1480/java

Svend Vanderveken

unread,

Jan 15, 2014, 6:13:44 AM1/15/14

to vagra...@googlegroups.com

So finally, answering my own question (in case somebody needs that info someday),

One way to obtain a single node Cassandra within a Vagrant created Ubuntu box is:

forget about port forwarding in the Vagrant file:

"

# config.vm.network :forwarded_port, guest: 6627, host: 6627 # storm Nimbus Thrift port

# config.vm.network :forwarded_port, guest: 7777, host: 7777 # storm UI

# config.vm.network :forwarded_port, guest: 2181, host: 2181 # zk

# config.vm.network :forwarded_port, guest: 7000, host: 7000 # Cassandra cluster

# config.vm.network :forwarded_port, guest: 9160, host: 9160 # Cassandra Thrift client

# config.vm.network :forwarded_port, guest: 9042, host: 9042 # Cassandra CQL client

# config.vm.network :forwarded_port, guest: 7199, host: 7199 # Cassandra jmx

"

activate host-only networking:

"

config.vm.network :private_network, ip: "192.168.33.10"

"

WIthin the box, make Cassandra listen to CQLSH connection on the above IP (cassandra.yaml)

"

rpc_address: 192.168.33.10

"

Svend

Philip G

unread,

Mar 26, 2014, 6:21:04 PM3/26/14

to vagra...@googlegroups.com

This is an old message, but I wanted to share the solution I just discovered: set rpc_address to 0.0.0.0

It appears to be an issue with how vbox does it NAT forwarding; it forwards between the host IP and the local IP. When cassandra is set to bind to localhost, VBox is unable to see the port is open, and unable to forward. Since it's just a local vm; I don't see any problems with forwarding.

Another potential solution could be to set the guest ip to 127.0.0.1. It may work. But I'm not concerned on testing it myself.

Hope this help you and anybody else that may stumble upon it through Google (like I have).

Philip

James Oliver

unread,

May 14, 2014, 7:40:30 PM5/14/14

to vagra...@googlegroups.com

Thanks a bunch for this! Saved me a few hours of debugging.

Yannis Gonianakis

unread,

Feb 9, 2017, 9:25:35 AM2/9/17

to Vagrant

Thanks a lot for this answer!

Reply all

Reply to author

Forward