Connecting to cassandra cluster issues

262 views

Skip to first unread message

daniel robert pilch

unread,

Nov 20, 2015, 4:44:24 PM11/20/15

to Aurelius

Hi all,

When trying to connect to my cassandra cluster I am having some issues:

02  [main] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Configuring Gremlin Server from conf/gremlin-server/gremlin-server.yaml
458  [main] INFO  org.apache.tinkerpop.gremlin.server.util.MetricManager  - Configured Metrics ConsoleReporter configured with report interval=180000ms
461  [main] INFO  org.apache.tinkerpop.gremlin.server.util.MetricManager  - Configured Metrics CsvReporter configured with report interval=180000ms to fileName=/tmp/gremlin-server-metrics.csv
560  [main] INFO  org.apache.tinkerpop.gremlin.server.util.MetricManager  - Configured Metrics JmxReporter configured with domain= and agentId=
563  [main] INFO  org.apache.tinkerpop.gremlin.server.util.MetricManager  - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
1186 [main] INFO  com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager  - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
1196 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.1
1196 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.2
1197 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.3
1451 [main] INFO  com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager  - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
1452 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.1
1452 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.2
1452 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - AddHost: 10.0.0.3
1707 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - RemoveHost: 10.0.0.1
1709 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - RemoveHost: 10.0.0.2
1709 [main] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - RemoveHost: 10.0.0.3
29232 [main] INFO  com.thinkaurelius.titan.diskstorage.util.BackendOperation  - Temporary exception during backend operation [getConfiguration]. Attempting backoff retry.
com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:114)
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:78)
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getSlice(AstyanaxKeyColumnValueStore.java:67)
        at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:91)
        at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:88)
        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:133)
        at com.thinkaurelius.titan.diskstorage.util.BackendOperation$1.call(BackendOperation.java:147)
        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:144)
        at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:88)
        at com.thinkaurelius.titan.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:93)
        at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1338)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
        at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
        at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$27(GraphManager.java:50)
        at java.util.LinkedHashMap$LinkedEntrySet.forEach(Unknown Source)
        at org.apache.tinkerpop.gremlin.server.GraphManager.<init>(GraphManager.java:48)
        at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:94)
        at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:88)
        at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:290)
Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from
        at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:48)
        at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83)
        at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
        at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4.execute(ThriftColumnFamilyQueryImpl.java:538)
        at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxKeyColumnValueStore.getNamesSlice(AstyanaxKeyColumnValueStore.java:112)
        ... 27 more
31836 [RingDescribeAutoDiscovery] WARN  com.netflix.astyanax.impl.RingDescribeHostSupplier  - Failed to get hosts from titan_test via ring describe.  Will use previously known ring instead

Here is my 'titan-cassandra.properties' settings for cassandra:

storage.backend=cassandra
storage.hostname=10.0.0.1,10.0.0.2,10.0.0.3
storage.cassandra.keyspace=titan_test
storage.cassandra.read-consistency-level=ONE
storage.cassandra.write-consistency-level=ONE
storage.cassandra.replication-factor=2
storage.cassandra.replication-strategy-class=org.apache.cassandra.locator.NetworkTopologyStrategy
storage.cassandra.replication-strategy-options=titan_test,2
storage.password=cassandra
storage.username=cassandra
storage.cassandra.astyanax.max-cluster-connections-per-host=50
storage.cassandra.astyanax.max-connections-per-host=50
storage.cassandra.astyanax.local-datacenter=titan_test

Has anyone ever seen this behaviour? I don't understand why the nodes are added and then immediately removed. All cassandra hosts are connectible from the titan host.