titan not working well with distrbuted cassandra

[David Montgomery]

Hi,

Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=cassandra.do.development.sf.com(107.170.235.192):9160, latency=10000(10000), attempts=1]Timed out waiting for connection

On digital ocean I have titan, ES and cassarda running on separate servers.  On cassandra, titan has access to the following ports {'9160':'tcp','7000':'tcp','7199':'tcp','8888':'tcp','61620':'tcp','61621':'tcp','80':'tcp','23':tcp} yet I have have connection issues per the below.

In Cassandra.yaml the list_address is the public facing address.  Per the docs I did not have to set broadcast_address given I have list_address set to pub facing address.

How do I resolve?



Here is my titan config file-> conf/titan-server.properties

storage.backend=cassandra
#storage.backend=cassandrathrift
storage.hostname=cassandra.do.development.sf.com
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.5

storage.index.search.backend=elasticsearch
storage.index.search.client-only=true
storage.index.search.hostname=es.do.development.sf.com




 g = TitanFactory.open('conf/titan-server.properties')

TitanFactory.open('conf/titan-server.properties')
Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager
Display stack trace? [yN] y
java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager
    at com.thinkaurelius.titan.diskstorage.Backend.instantiate(Backend.java:355)
    at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:367)
    at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:311)
    at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:121)
    at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1163)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:75)
    at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
    at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:29)
    at com.thinkaurelius.titan.core.TitanFactory$open.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at groovysh_evaluate.run(groovysh_evaluate:56)
    at groovysh_evaluate$run.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    at groovysh_evaluate$run.call(Unknown Source)
    at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
    at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
    at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
    at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
    at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
    at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
    at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
    at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:57)
    at com.thinkaurelius.titan.tinkerpop.gremlin.Console.<init>(Console.java:70)
    at com.thinkaurelius.titan.tinkerpop.gremlin.Console.main(Console.java:96)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.thinkaurelius.titan.diskstorage.Backend.instantiate(Backend.java:344)
    ... 54 more
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend
    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:530)
    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:267)
    ... 59 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=cassandra.do.development.sf.com(107.170.235.192):9160, latency=10000(10000), attempts=1]Timed out waiting for connection
    at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:218)
    at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:185)
    at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:66)
    at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:67)
    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)
    at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
    at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:246)
    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:525)
    ... 60 more

[Dan LaRocque]

On 04/15/2014 03:18 AM, David Montgomery wrote:
> Hi,
>
> Caused by:
> com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
> PoolTimeoutException:
> [host=cassandra.do.development.sf.com(107.170.235.192):9160,
> latency=10000(10000), attempts=1]Timed out waiting for connection
>
> On digital ocean I have titan, ES and cassarda running on separate
> servers. On cassandra, titan has access to the following ports
> {'9160':'tcp','7000':'tcp','7199':'tcp','8888':'tcp','61620':'tcp','61621':'tcp','80':'tcp','23':tcp}
> yet I have have connection issues per the below.
>
> In Cassandra.yaml the list_address is the public facing address. Per
> the docs I did not have to set broadcast_address given I have
> list_address set to pub facing address.
>
> How do I resolve?
>
>

Hi,

1. Do you mean listen_address?

2. "Timed out waiting for connection" suggests that port 9160 is
probably getting blocked. Can you open a socket to 9160 from your
Titan client (e.g. run nmap or `telnet 107.170.235.192 9160`)? If nmap
reports 9160 is filtered or telnet times out, then there's probably an
issue with with the host or network firewalls.

thanks,
Dan