Connection refused, but Cassandra is listening on port and using CLI works

[Kevin]

Can someone help me debug this problem?

Before I explain my particular problem, i'd like to point out that it
seems as though it may be a compatibility issue between my Windows
setup (from which i'm running the program from) and my Ubuntu server
(where lone Cassandra node is located). That hypothesis comes from me
reading this (which is basically the problem I have):

http://stackoverflow.com/questions/3168261/cant-connect-to-cassandra-nodecontext-killing-all-pooled-connections-for-sessi

As for my specific problem:

Pelops cannot even seem to connect to my Cassandra node at port 9160.
I can connect remotely to my server and verify that Cassandra is
listening on that port by connecting to it through the CLI. Here is
the part of the stacktrace pertinent to the problem:

org.scale7.cassandra.pelops.pool.CommonsBackedPool - A
TTransportException was thrown while attempting to create a connection
to 'XXX.XXX.X.XXX'. This node will be suspended for 10000ms. Trying
another node...

org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused: connect
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:
81)
at org.scale7.cassandra.pelops.Connection.open(Connection.java:
93)
at org.scale7.cassandra.pelops.pool.CommonsBackedPool
$ConnectionFactory.makeObject(CommonsBackedPool.java:891)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:
1190)
at
org.scale7.cassandra.pelops.pool.CommonsBackedPool.getConnectionExcept(CommonsBackedPool.java:
316)
at
org.scale7.cassandra.pelops.Operand.tryOperation(Operand.java:73)
at
org.scale7.cassandra.pelops.Selector.getColumnCount(Selector.java:278)
at
org.scale7.cassandra.pelops.Selector.getColumnCount(Selector.java:88)

I only have one node so it does this a couple of times before giving
me this:

org.scale7.cassandra.pelops.pool.CommonsBackedPool - Failed to get a
connection within the maximum allowed wait time. Try increasing the
either the number of allowed connections or the max wait time.

Is this a Pelops-specific problem? I hope not, I really like its
simplicity. In any case, if someone could take a minute or two to help
me out I'd be very grateful.

[Dominic Williams]

Hi,

This isn't a Pelops problem. The underlying Thrift transport (the TCP network protocol used between the client and server) cannot connect to the server.

Either you are being blocked by a firewall on the Windows machine, or your Cassandra database hasn't been configured to listen on all interfaces e.g. 0.0.0.0

Hope this helps

[Dan Washusen]

So from the windows machine cassandra-cli can connect to the node (e.g. cassandra-cli -host XXX.XXX.X.XXX -port 9160)?   Can your windows machine connect to the Cassandra instance with telnet (e.g. telnet XXX.XXX.X.XXX 9160)?

Pelops isn't doing anything remotely tricky when it comes to making connections to Cassandra so it seems very unlikely it's Pelops specific.  Check it out yourself:

https://github.com/s7/scale7-pelops/blob/0.7.x/src/main/java/org/scale7/cassandra/pelops/Connection.java

I've just had a quick look at the Hector connection code and it looks pretty much the same:

https://github.com/rantav/hector/blob/master/core/src/main/java/me/prettyprint/cassandra/connection/HThriftClient.java

The cassandra-cli code looks pretty much the same as well:

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cli/CliMain.java#L57

So yeah, I agree with Dominic that this isn't a Pelops problem but if you create a test case that illustrates the issue I'll have a bit more of a look...

-- 
Dan Washusen

Make big files fly

visit digitalpigeon.com

[Dan Washusen]

I just read that stackoverflow link you included, he says he has Cassandra configured to listen on 0.0.0.0 which isn't correct (http://wiki.apache.org/cassandra/FAQ#cant_listen_on_ip_any).  Just wanted to confirm that you are not using Cassandra this way...?  

-- 
Dan Washusen

Make big files fly

visit digitalpigeon.com

[Kevin]

Sorry for the late reply. The problem is fixed.

I resolved it by setting the rpclisten address of the node (in
Cassandra.yaml) to the IP address of the node (instead of localhost).

I tested setting it to 0.0.0.0, and that seems to work as well, but I
was kind of hesitant to set it to the default gateway. I didn't
completely understand what the implications of doing so were.

If someone would mind telling me (so I could decide to change it back
or not), I'd be grateful.

Thanks guys!

On Apr 1, 9:54 pm, Dan Washusen <d...@reactive.org> wrote:
> I just read that stackoverflow link you included, he says he has Cassandra configured to listen on 0.0.0.0 which isn't correct (http://wiki.apache.org/cassandra/FAQ#cant_listen_on_ip_any). Just wanted to confirm that you are not using Cassandra this way...?
>
> --
> Dan Washusen
> Make big files fly
> visit digitalpigeon.com
>
> On Saturday, 2 April 2011 at 12:43 PM, Dan Washusen wrote:
>
> So from the windows machine cassandra-cli can connect to the node (e.g. cassandra-cli -host XXX.XXX.X.XXX -port 9160)?  Can your windows machine connect to the Cassandra instance with telnet (e.g. telnet XXX.XXX.X.XXX 9160)?
>
>
>
>
>
> > Pelops isn't doing anything remotely tricky when it comes to making connections to Cassandra so it seems very unlikely it's Pelops specific. Check it out yourself:

> >https://github.com/s7/scale7-pelops/blob/0.7.x/src/main/java/org/scal...

>
> > I've just had a quick look at the Hector connection code and it looks pretty much the same:

> >https://github.com/rantav/hector/blob/master/core/src/main/java/me/pr...

>
> > The cassandra-cli code looks pretty much the same as well:

> >https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/ca...

>
> > So yeah, I agree with Dominic that this isn't a Pelops problem but if you create a test case that illustrates the issue I'll have a bit more of a look...
>
> > --
> > Dan Washusen
> > Make big files fly
> > visit digitalpigeon.com
>
> > On Saturday, 2 April 2011 at 6:59 AM, Dominic Williams wrote:
> > > Hi,
>
> > > This isn't a Pelops problem. The underlying Thrift transport (the TCP network protocol used between the client and server) cannot connect to the server.
>
> > > Either you are being blocked by a firewall on the Windows machine, or your Cassandra database hasn't been configured to listen on all interfaces e.g. 0.0.0.0
>
> > > Hope this helps
>

> > > On 1 April 2011 18:03, Kevin <klawso...@gmail.com> wrote:
> > > >  Can someone help me debug this problem?
>
> > > >  Before I explain my particular problem, i'd like to point out that it
> > > >  seems as though it may be a compatibility issue between my Windows
> > > >  setup (from which i'm running the program from) and my Ubuntu server
> > > >  (where lone Cassandra node is located). That hypothesis comes from me
> > > >  reading this (which is basically the problem I have):
>

> > > >http://stackoverflow.com/questions/3168261/cant-connect-to-cassandra-...

>
> > > >  As for my specific problem:
>
> > > >  Pelops cannot even seem to connect to my Cassandra node at port 9160.
> > > >  I can connect remotely to my server and verify that Cassandra is
> > > >  listening on that port by connecting to it through the CLI. Here is
> > > >  the part of the stacktrace pertinent to the problem:
>
> > > >  org.scale7.cassandra.pelops.pool.CommonsBackedPool - A
> > > >  TTransportException was thrown while attempting to create a connection
> > > >  to 'XXX.XXX.X.XXX'. This node will be suspended for 10000ms. Trying
> > > >  another node...
>
> > > > org.apache.thrift.transport.TTransportException:
> > > >  java.net.ConnectException: Connection refused: connect
> > > > at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
> > > > at
> > > >  org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:
> > > >  81)
> > > > at org.scale7.cassandra.pelops.Connection.open(Connection.java:
> > > >  93)
> > > > at org.scale7.cassandra.pelops.pool.CommonsBackedPool
> > > >  $ConnectionFactory.makeObject(CommonsBackedPool.java:891)
> > > > at

> > > >  org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKey­edObjectPool.java:
> > > >  1190)
> > > > at
> > > >  org.scale7.cassandra.pelops.pool.CommonsBackedPool.getConnectionExcept(Comm­onsBackedPool.java:

[Dan Washusen]

Not sure I understand what you're asking... If you set Cassandra to listen on localhost (the default) then you won't be able to connect to it from another machine.  You do NOT want to set it to 0.0.0.0 (read the FAQ entry for details)... 
-- 
Dan Washusen