Connection Refused Localhost

[Shehaaz Saif]

Goal: 

To connect locahost via REST to Cassandra

Settings: 

virgil.yaml and Cassandra.yaml were left un-touched.

Solutions tried: 

Not a permissions issues or substituting localhost to 0.0.0.0

I followed the getting started guide and this is the console output:

~ bin/virgil -host localhost

  ...

Starting virgil against remote cassandra server [localhost:9160]

  ...

Exception in thread "main" org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)

at com.hmsonline.virgil.pool.VirgilConnection.<init>(VirgilConnection.java:37)

at com.hmsonline.virgil.pool.ConnectionPool.createConnection(ConnectionPool.java:36)

at com.hmsonline.virgil.pool.ConnectionPool.initializePool(ConnectionPool.java:30)

at com.hmsonline.virgil.CassandraStorage.<init>(CassandraStorage.java:79)

at com.hmsonline.virgil.cli.VirgilCommand.createCassandraStorage(VirgilCommand.java:64)

at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:73)

at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:1)

at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:70)

at com.yammer.dropwizard.cli.Command.run(Command.java:112)

at com.yammer.dropwizard.AbstractService.run(AbstractService.java:160)

at com.hmsonline.virgil.VirgilService.main(VirgilService.java:19)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)

at java.net.Socket.connect(Socket.java:546)

at org.apache.thrift.transport.TSocket.open(TSocket.java:178)

... 12 more

Thank you very much!!! 

I have been stuck on this the whole night :(

[Brian O'Neill]

Shehaaz,

Tough one… are you able to connect to Cassandra locally using other clients?  (cassandra-cli? Hector?)

Can you verify that you can get to that port using telnet?

http://kb.acronis.com/content/7503

-brian

---

Brian O'Neill

Lead Architect, Software Development

Health Market Science

The Science of Better Results

2700 Horizon Drive • King of Prussia, PA • 19406

M: 215.588.6024 • @boneill42  •  

healthmarketscience.com

This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.

 

--
You received this message because you are subscribed to the Google Groups "Virgil-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virgil-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Shehaaz Saif]

Thank you Brian for the response! :)

I was able to connect with the Cassandra Java Driver. I made a little java program and it was able to connect to the database....I decided on your client because I wanted to talk to the database using REST. 

I didn't know about Telnet until now...I just gave a go...and I get 

"telnet: Unable to connect to remote host: Connection refused" 

So if it definitely a port blocking/firewall issue. I will keep researching. Do you have any suggestions?

Once again Thank you very much!!! I was losing hope...I think we narrowed down the problem! 

- Shehaaz

[Brian O'Neill]

Shehaaz,

The Cassandra Java Driver uses the native protocol, which runs on a different port.  

What do you have in cassandra.yaml for the following settings?

lisa@icube:~/tools/apache-cassandra-1.2.2/conf-> grep rpc *

cassandra.yaml:start_rpc: true

cassandra.yaml:rpc_address: localhost

cassandra.yaml:rpc_port: 9160

-brian

-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/

[Shehaaz Saif]

Hi Brian,

Actually "start_rpc: true" was missing in the cassandra.yaml in Virgil.  I added that line and I still get the same error.

The cassandra.yaml in Cassandra 1.2.3 (not Virgil) had "start_rpc: true" and it was able to listen to localhost:9160.

Why is Cassandra.yaml in  (.../Virgil/target/virgil-1.0.2-SNAPSHOT/conf) different? What else could I try?

Thank you very much for you time! 

- Shehaaz

[Brian O'Neill]

The cassandra.yaml in Virgil is used for the embedded server.  In this case, you are running against a remote server (even though it is on your localhost), so it should have no effect.  

Only the cassandra.yaml in $CASSANDRA_HOME/conf matters.

Let me confirm, you are running cassandra 1.2.3?

You start cassandra with, "bin/cassandra -f" (or w/o -f)

But you cannot telnet to localhost 9160?

What is listen_address configured to in your cassandra.yaml?

[Shehaaz Saif]

Hi Brian,

I start Cassandra with "bin/cassandra -f" and 

In $CASSANDRA_HOME/conf/cassandra.yaml I have: listen_address: localhost  

but I can't telnet into localhost 9160

When you say:

"Only the cassandra.yaml in $CASSANDRA_HOME/conf matters."

How does Virgil find the cassandra.yaml in $CASSANDRA_HOME/conf? I put it in the $PATH to see if it checks it there, but I get the same error. 

SO I started the server using "cassandra -f" and then did bin/virgil -host localhost...I get the error below...that means that there is another process bound to the same port.

ERROR [2013-04-23 16:04:42,916] com.yammer.dropwizard.cli.ServerCommand: Unable to start server, shutting down

! java.net.BindException: Address already in use

Thank you

- Shehaaz

[Brian O'Neill]

Can you try setting listen_address to an actual IP on your machine?

Then try telnetting to that address (@9160).

(You could also try 127.0.0.1)

That address is dependent on how you have your hosts file configured on your machine.

Virgil doesn't need access to the yaml file in $CASSANDRA_HOME/conf.

[Shehaaz Saif]

Hi Brian!

Command: cassandra -f

Machine I.P: fails.

127.0.0.1: it works

Telnet:

Localhost/127.0.01: 

Trying 127.0.0.1...

telnet: Unable to connect to remote host: Connection refused

Machine I.P:

Trying 132.205.253.68...

telnet: Unable to connect to remote host: Connection timed out

This is what my HOST file looks like...

Host File (/etc/host)

127.0.0.1 localhost

127.0.1.1 shehaaz-VirtualBox

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

Thank you Brian! honestly you are awesome!

- Shehaaz 

[Brian O'Neill]

Your hosts file looks good.

Let's narrow it down. 

Are you familiar with netcat?

http://linux.die.net/man/1/nc

Shutdown Cassandra. Then, in one terminal window type:

"nc -l 9160"

In another one, try your telnet.  It should connect, and anything you type in the telnet window should show up in the other terminal.

If that works, then it is a Cassandra configuration issue.  If that doesn't work, then it points to a firewall/connectivity issue.

[Shehaaz Saif]

I was able to talk from one terminal to the next! So it is a Cassandra Config issue

What is the next step? I think we are close :)

Thank you very much Brian! 

- Shehaaz

[Brian O'Neill]

Excellent. 

And you said you already set rpc_address to 0.0.0.0?

(that really sounds like it is the likely culprit)

Can you send me the last few lines of your startup log?

It should look something like this:

INFO 14:08:52,584 Starting listening for CQL clients on localhost/127.0.0.1:9042...

 INFO 14:08:52,644 Binding thrift service to localhost/127.0.0.1:9160

 INFO 14:08:52,701 Using TFramedTransport with a max frame size of 15728640 bytes.

 INFO 14:08:52,714 Using synchronous/threadpool thrift server on localhost : 9160

I've highlighted the important lines.

[Shehaaz Saif]

Good Morning Brian! 

> ~/Desktop/apache-cassandra-1.2.4 $ bin/cassandra -f

 (....)

 INFO 10:13:59,492 Node localhost/127.0.0.1 state jump to normal

 INFO 10:13:59,518 Startup completed! Now serving reads.

 INFO 10:13:59,636 Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) or nodetool (enablebinary) to start it

 INFO 10:13:59,669 Binding thrift service to localhost/127.0.0.1:9160

 INFO 10:13:59,804 Using TFramedTransport with a max frame size of 15728640 bytes.

 INFO 10:13:59,828 Using synchronous/threadpool thrift server on localhost : 9160

 INFO 10:13:59,833 Listening for thrift clients...

The Cassandra is able to listen....Below is the virgil output

>  ~/Desktop/virgil/target/virgil-1.0.2-SNAPSHOT $ bin/virgil -host localhost

(...)

Starting virgil against remote cassandra server [localhost:9160]

(...)

Exception in thread "main" org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)

at com.hmsonline.virgil.pool.VirgilConnection.<init>(VirgilConnection.java:37)

at com.hmsonline.virgil.pool.ConnectionPool.createConnection(ConnectionPool.java:36)

at com.hmsonline.virgil.pool.ConnectionPool.initializePool(ConnectionPool.java:30)

at com.hmsonline.virgil.CassandraStorage.<init>(CassandraStorage.java:79)

at com.hmsonline.virgil.cli.VirgilCommand.createCassandraStorage(VirgilCommand.java:64)

at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:73)

at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:1)

at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:70)

at com.yammer.dropwizard.cli.Command.run(Command.java:112)

at com.yammer.dropwizard.AbstractService.run(AbstractService.java:160)

at com.hmsonline.virgil.VirgilService.main(VirgilService.java:19)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)

at java.net.Socket.connect(Socket.java:546)

at org.apache.thrift.transport.TSocket.open(TSocket.java:178)

... 12 more 

 

#I attached both outputs in .txt format and also cassandra.yaml (apache-cassandra-1.2.4/conf) 

 

 Thank you Brian! I hope I gave you all the info :)

- Shehaaz

[Brian O'Neill]

That looks good.

And the telnet continues to fail right?

Are you able to connect via cassandra-cli?

[Shehaaz Saif]

Yes Telnet fails and also cassandra-cli (which I didn't use until now)

Here is what it looks like:

shehaaz@shehaaz-VirtualBox ~/Desktop/apache-cassandra-1.2.4 $ bin/cassandra-cli

org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)

at org.apache.cassandra.cli.CliMain.connect(CliMain.java:70)

at org.apache.cassandra.cli.CliMain.main(CliMain.java:246)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)

at java.net.Socket.connect(Socket.java:546)

at org.apache.thrift.transport.TSocket.open(TSocket.java:178)

... 3 more

Exception connecting to 127.0.0.1/9160. Reason: Connection refused.

Welcome to Cassandra CLI version 1.2.4

Type 'help;' or '?' for help.

Type 'quit;' or 'exit;' to quit.

[default@unknown] 

- Shehaaz

[Shehaaz Saif]

Actually cassandra-cli works if the server (bin/cassandra -f) is running 

shehaaz@shehaaz-VirtualBox ~/Desktop/apache-cassandra-1.2.4 $ bin/cassandra-cli

Connected to: "Test Cluster" on 127.0.0.1/9160

Welcome to Cassandra CLI version 1.2.4

Type 'help;' or '?' for help.

Type 'quit;' or 'exit;' to quit.

[default@unknown] 

[Brian O'Neill]

Hmm, maybe that is the disconnect….

You need the cassandra server to be running in order to run virgil the way you are running it.  (remotely)

Start cassandra with "bin/cassandra -f".

With that running, start virgil with "bin/virgil -host localhost"

Is that what you were doing?

Virgil is also capable of running Cassandra in an embedded mode, which does not require you to run Cassandra in a separate process.  But, with the "-host" parameter Virgil is a client to the Cassandra server.

[Shehaaz Saif]

I just tried that and I got this...Why is it trying 0.0.0.0:8080? interesting 

WARN  [2013-04-24 14:49:16,026] org.eclipse.jetty.util.component.AbstractLifeCycle: FAILED InstrumentedBlocki...@0.0.0.0:8080 FAILED: java.net.BindException: Address already in use

! java.net.BindException: Address already in use

! at sun.nio.ch.Net.bind(Native Method)

! at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137)

! at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.open(BlockingChannelConnector.java:122)

! at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:311)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.doStart(BlockingChannelConnector.java:80)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at org.eclipse.jetty.server.Server.doStart(Server.java:272)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at com.yammer.dropwizard.cli.ServerCommand.run(ServerCommand.java:61)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:76)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:1)

! at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:70)

! at com.yammer.dropwizard.cli.Command.run(Command.java:112)

! at com.yammer.dropwizard.AbstractService.run(AbstractService.java:160)

! at com.hmsonline.virgil.VirgilService.main(VirgilService.java:19)

WARN  [2013-04-24 14:49:16,099] org.eclipse.jetty.util.component.AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@b7141a: java.net.BindException: Address already in use

! java.net.BindException: Address already in use

! at sun.nio.ch.Net.bind(Native Method)

! at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137)

! at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.open(BlockingChannelConnector.java:122)

! at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:311)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.doStart(BlockingChannelConnector.java:80)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at org.eclipse.jetty.server.Server.doStart(Server.java:272)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at com.yammer.dropwizard.cli.ServerCommand.run(ServerCommand.java:61)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:76)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:1)

! at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:70)

! at com.yammer.dropwizard.cli.Command.run(Command.java:112)

! at com.yammer.dropwizard.AbstractService.run(AbstractService.java:160)

! at com.hmsonline.virgil.VirgilService.main(VirgilService.java:19)

ERROR [2013-04-24 14:49:16,121] com.yammer.dropwizard.cli.ServerCommand: Unable to start server, shutting down

! java.net.BindException: Address already in use

! at sun.nio.ch.Net.bind(Native Method)

! at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:137)

! at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.open(BlockingChannelConnector.java:122)

! at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:311)

! at org.eclipse.jetty.server.nio.BlockingChannelConnector.doStart(BlockingChannelConnector.java:80)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at org.eclipse.jetty.server.Server.doStart(Server.java:272)

! at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)

! at com.yammer.dropwizard.cli.ServerCommand.run(ServerCommand.java:61)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:76)

! at com.hmsonline.virgil.cli.VirgilCommand.run(VirgilCommand.java:1)

! at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:70)

! at com.yammer.dropwizard.cli.Command.run(Command.java:112)

! at com.yammer.dropwizard.AbstractService.run(AbstractService.java:160)

! at com.hmsonline.virgil.VirgilService.main(VirgilService.java:19)

[Brian O'Neill]

Excellent.  You are almost there.

There must be something else running on 8080.  Do you have tomcat installed and running?

If you find that and kill it, virgil should startup.

Alternatively, you can edit the conf/virgil.yaml to change the port.  (http:port)

[Shehaaz Saif]

shehaaz@shehaaz-VirtualBox ~ $ netstat -nap | grep 8080

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

tcp6       0      0 :::8080                 :::*                    LISTEN      -               

unix  3      [ ]         STREAM     CONNECTED     8080     -                   @/tmp/.X11-unix/X0

So basically This is blocking 8080?

On Wednesday, 24 April 2013 11:09:37 UTC-4, Shehaaz Saif wrote:

I just tried that and I got this...Why is it trying 0.0.0.0:8080? interesting 

WARN  [2013-04-24 14:49:16,026] org.eclipse.jetty.util.component.AbstractLifeCycle: FAILED InstrumentedBlockingChannelConnec...@0.0.0.0:8080 FAILED: java.net.BindException: Address already in use

[Brian O'Neill]

Yep.

[Shehaaz Saif]

How do I change org.eclipse.jetty.util.component.AbstractLifeCycle port from 8080? is there some config xml file?

- Shehaaz

[Brian O'Neill]

In $VIRGIL_HOME/conf/virgil.yaml, edit the port setting.

[Shehaaz Saif]

OMG!!! IT WORKED!!!! THANK YOU BRIAN!!! You are the BESTEST!! You don't understand how happy I am right now!! THANK YOU! THANK YOU! I learned so much about Unix networking in the last few days because of you! :)

- Shehaaz

[Brian O'Neill]

My pleasure Shehaaz. =)