Cassandra Connection refused

[Pablo Abbate]

Hi guys, 

I'm using an embedded Titan 0.5.1 and a Cassandra 2.1.0 Cluster. I'm receiving this error while trying to connect from the map reduce job (hadoop 2.4.0.2.1.5.0-695).

I notice 2 things. The first one, is that the same configuration parameters is working from Gremlin (I even have the same warning for the index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.hostname property, but the connection success)

The second one, is that Closed Thrift connection pooler is logged after the warning. So why the connection is being closed?

Any ideas? Thanks!

2014-10-30 08:10:44,258 WARN [main] com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration: Local setting index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.hostname=[10.209.5.31, 10.209.5.32, 10.209.4.204, 10.209.5.42] (Type: GLOBAL) is overridden by globally managed value ([10.209.5.31, 10.209.5.32, 10.209.4.204, 10.209.5.42]).  Use the ManagementSystem interface instead of the local configuration to control this setting.

2014-10-30 08:10:44,292 INFO [main] com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager: Closed Thrift connection pooler.

2014-10-30 08:10:44,295 INFO [main] com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration: Generated unique-instance-id=0adff0a65904-hdp31

2014-10-30 08:10:44,812 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager

at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:55)

at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:425)

at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:366)

at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:130)

at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1718)

at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:105)

at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:91)

at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:71)

at com.mypackage.TitanConnectionImpl.connect(TitanConnectionImpl.java:77)

at com.mypackage.TitanGraphDaoImpl.createWithoutStoringTitanConnection(TitanGraphDaoImpl.java:260)

at com.mypackage.TitanGraphDaoImpl.createAndStoreTitanConnection(TitanGraphDaoImpl.java:266)

at com.mypackage.VertexInsertJob$VertexInsertReducer.setup(VertexInsertJob.java:143)

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:44)

... 19 more

Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:218)

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.<init>(CassandraThriftStoreManager.java:196)

... 24 more

Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

at org.apache.thrift.transport.TSocket.open(TSocket.java:185)

at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeRawConnection(CTConnectionFactory.java:88)

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:52)

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:21)

at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:215)

... 25 more

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at org.apache.thrift.transport.TSocket.open(TSocket.java:180)

... 31 more

These are my configuration properties.

storage.backend=cassandrathrift 

storage.hostname=10.223.224.195,10.223.241.210,10.223.240.166,10.223.241.115,10.223.224.32,10.223.241.182

storage.cassandra.thrift.frame-size=800 

storage.cassandra.read-consistency-level=ONE 

storage.connection-timeout=20000

storage.lock.wait-time=3000

storage.cassandra.replication-factor=3

storage.batch-loading=true

cache.db-cache=true 

cache.db-cache-time=600000

ids.block-size=300000

ids.authority.wait-time=5000

cache.tx-cache-size=500000 

query.fast-property=true

storage.cassandra.keyspace=d4421046_efa2_4b8f_86b0_7cdce9b8067a

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.backend=elasticsearch

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.hostname=10.209.5.31,10.209.5.32,10.209.4.204,10.209.5.42

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.index-name=d4421046_efa2_4b8f_86b0_7cdce9b8067a

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.elasticsearch.cluster-name=esdev1

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.elasticsearch.client-only=true

index.d4421046_efa2_4b8f_86b0_7cdce9b8067a.elasticsearch.local-mode=false

[Matthias Broecheler]

Hi Pablo,

is this on ec2 or some other cloud provider? This has happened to me before when the network settings didn't allow access and hence a connection could not be established.

HTH,

Matthias

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/4f17a359-dace-4eb3-aab8-fe0437253c8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matthias Broecheler
http://www.matthiasb.com

[Pablo Abbate]

Mathias, 

It's on Rackspace, yes. You know how the problem was fixed? Just changing the connector from cassandrathrift to cassandra. 

I'm not sure why this could happen, but I'm behind the schedule so I don't have the time right now to figure out what's going on ...

Thanks for your response anyway.

Pablo,