[titan-server-0.4.1] Exception in rexstitan.log : Graph not persisted

[Puneet Arora]

Hi All,

I am facing the following Exception. Kindly, inform me the cause for it.

Also, attached are the config files.

PROBLEM 1:

1. I start the titan server using "sudo ./bin/titan.sh -c cassandra-es start".
2. login to resxter console and run the following commands:
• rexster[groovy]>  g = rexster.getGraph("graph")
  
• ==>titangraph[cassandra:null]
  
• rexster[groovy]>  g.addVertex([name:'marko'])
  
• ==>v[160004]
  
3. Stop the server using "sudo ./bin/titan.sh stop".
• It does not kill the titan+rexster server and gives the following exception in the rexstitan log and goes in a loop
• I have to kill the process to stop it.
4. The graph is not persisted, all the nodes and edges are lost when I again start the server.

EXCEPTION:

59828 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - JVM shutdown hook called

59828 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - Calling ShutdownListener: com.tinkerpop.rexster.Application$2@223e5bd

68045 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.AbstractMapRexsterApplication  - Shutting down [graph] - [titangraph[cassandra:null]]

90894 [RingDescribeAutoDiscovery] WARN  com.netflix.astyanax.impl.RingDescribeHostSupplier  - Failed to get hosts from titan via ring describe.  Will use previously known ring instead

120665 [JVM Shutdown Hook] INFO  com.thinkaurelius.titan.diskstorage.util.BackendOperation  - Temporary storage exception during backend operation [BufferMutation]. Attempting incremental retry

com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend

    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:259)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:99)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:96)

    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.flushInternal(BufferTransaction.java:96)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.commit(BufferTransaction.java:121)

    at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.commit(ExpectedValueCheckingTransaction.java:167)

    at com.thinkaurelius.titan.diskstorage.BackendTransaction.commit(BackendTransaction.java:87)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:315)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:245)

    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:324)

    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1002)

    at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:117)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdown(StandardTitanGraph.java:104)

    at com.tinkerpop.rexster.server.AbstractMapRexsterApplication.stop(AbstractMapRexsterApplication.java:66)

    at com.tinkerpop.rexster.Application.stop(Application.java:153)

    at com.tinkerpop.rexster.Application$2.shutdown(Application.java:166)

    at com.tinkerpop.rexster.server.ShutdownManager.runShutdownHandlers(ShutdownManager.java:180)

    at com.tinkerpop.rexster.server.ShutdownManager.shutdown(ShutdownManager.java:144)

    at com.tinkerpop.rexster.server.ShutdownManager$ShutdownHookHandler.run(ShutdownManager.java:349)

    at java.lang.Thread.run(Thread.java:744)

Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from

    at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:30)

    at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83)

    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:485)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:79)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:123)

    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:257)

    ... 21 more

120894 [RingDescribeAutoDiscovery] WARN  com.netflix.astyanax.impl.RingDescribeHostSupplier  - Failed to get hosts from titan via ring describe.  Will use previously known ring instead

138383 [JVM Shutdown Hook] INFO  com.thinkaurelius.titan.diskstorage.util.BackendOperation  - Temporary storage exception during backend operation [BufferMutation]. Attempting incremental retry

com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend

    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:259)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:99)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction$1.call(BufferTransaction.java:96)

    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.flushInternal(BufferTransaction.java:96)

    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferTransaction.commit(BufferTransaction.java:121)

    at com.thinkaurelius.titan.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.commit(ExpectedValueCheckingTransaction.java:167)

    at com.thinkaurelius.titan.diskstorage.BackendTransaction.commit(BackendTransaction.java:87)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:315)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph$2.call(StandardTitanGraph.java:245)

    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:62)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:324)

    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1002)

    at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.shutdown(TitanBlueprintsGraph.java:117)

    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.shutdown(StandardTitanGraph.java:104)

    at com.tinkerpop.rexster.server.AbstractMapRexsterApplication.stop(AbstractMapRexsterApplication.java:66)

    at com.tinkerpop.rexster.Application.stop(Application.java:153)

    at com.tinkerpop.rexster.Application$2.shutdown(Application.java:166)

    at com.tinkerpop.rexster.server.ShutdownManager.runShutdownHandlers(ShutdownManager.java:180)

    at com.tinkerpop.rexster.server.ShutdownManager.shutdown(ShutdownManager.java:144)

    at com.tinkerpop.rexster.server.ShutdownManager$ShutdownHookHandler.run(ShutdownManager.java:349)

    at java.lang.Thread.run(Thread.java:744)

Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from

    at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:30)

    at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83)

    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:256)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:485)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$000(ThriftKeyspaceImpl.java:79)

    at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:123)

    at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.mutateMany(AstyanaxStoreManager.java:257)

    ... 21 more

PROBLEM 2:

1. I start the titan server using "sudo ./bin/titan.sh -c cassandra-es start".
2. login to resxter console and run the following commands:
• rexster[groovy]>  g = rexster.getGraph("graph")
• ==>titangraph[cassandra:null]
• rexster[groovy]>  g.addVertex([name:'marko'])
• ==>v[200004]
• rexster[groovy]> g.commit()
• ==>null
3. Stop the server using "sudo ./bin/titan.sh stop".
• It KILLED  the titan+rexster server and 
• BUT, gives the following exception in the rexstitan log
4. The graph is not persisted, all the nodes and edges are lost when I again start the server.

EXCEPTION:

 69528 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - JVM shutdown hook called

69529 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - Calling ShutdownListener: com.tinkerpop.rexster.Application$2@52fa0572

78237 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.AbstractMapRexsterApplication  - Shutting down [graph] - [titangraph[cassandra:null]]

78253 [JVM Shutdown Hook] INFO  org.elasticsearch.node  - [Bora] stopping ...

78286 [JVM Shutdown Hook] INFO  org.elasticsearch.node  - [Bora] stopped

78287 [JVM Shutdown Hook] INFO  org.elasticsearch.node  - [Bora] closing ...

78293 [JVM Shutdown Hook] INFO  org.elasticsearch.node  - [Bora] closed

78294 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - ShutdownListener com.tinkerpop.rexster.Application$2@52fa0572 complete

78294 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - Calling ShutdownListener: ShutdownListener [bindHost=/127.0.0.1, port=8183]

78294 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - ShutdownListener ShutdownListener [bindHost=/127.0.0.1, port=8183] complete

78294 [JVM Shutdown Hook] INFO  com.tinkerpop.rexster.server.ShutdownManager  - Calling ShutdownListener: JVM Shutdown Hook Remover

78294 [JVM Shutdown Hook] WARN  com.tinkerpop.rexster.server.ShutdownManager  - ShutdownListener JVM Shutdown Hook Remover threw an exception, continuing with shutdown

78295 [ShutdownListener-127.0.0.1:8183] INFO  com.tinkerpop.rexster.server.ShutdownManager  - Caught SocketException on shutdownSocket, assuming close() was called: Socket closed

I am unable to understand the cause of these exceptions or the procedure to rectify them.

Kindly, help on this.

Thanks

Puneet

[Matthias Broecheler]

Hi Puneet,

the line:

titangraph[cassandra:null]

in your rexster console indicates that Cassandra isn't starting correctly and hence the subsequent error where it fails to connect to it. Can you do a titan.sh status call to see if all the components are running correctly? Do you have another version of cassandra running lcoally?

Cheers,

Matthias

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Matthias Broecheler
http://www.matthiasb.com

[Puneet Arora]

Hi Matthias,

No other Cassandra process if running on my machine. I check it by "ps -ef | grep cassandra".

When I check the status, it shows that the processes are running with the following output.

~/titan-server-0.4.1$ sudo ./bin/titan.sh -c cassandra-es status

Titan + Rexster (com.tinkerpop.rexster.Application) is running with pid 20957

Cassandra (org.apache.cassandra.service.CassandraDaemon) is running with pid 20646

One weird thing that happened was, even when I stopped the titan server using " sudo ./bin/titan.sh -c cassandra-es stop"

It kills the Cassandra process and unable to kill the rexster process.

Now, when I do  "ps -ef | grep rexster" , it shown the REXSTER process running.

LOG:

pld6@pld6-virtual-machine:~/titan-server-0.4.1$ sudo ./bin/titan.sh -c cassandra-es stop

Killing Titan + Rexster (pid 20957)...

Killing Cassandra (pid 20646)...

pld6@pld6-virtual-machine:~/titan-server-0.4.1$ sudo ./bin/titan.sh -c cassandra-es stop

Killing Titan + Rexster (pid 20957)...

Cassandra (org.apache.cassandra.service.CassandraDaemon) not found in the java process table

pld6@pld6-virtual-machine:~/titan-server-0.4.1$ ps -ef | grep rexster

root     20948     1  0 16:16 pts/3    00:00:00 /bin/bash ./bin/rexster.sh -s -wr public -c ../conf/rexster-cassandra-es.xml

root     20957 20948  2 16:16 pts/3    00:00:18 /usr/lib/jvm/jdk1.7.0_45/bin/java -server -Xms128m -Xmx512m -Dtitan.logdir=../log -Dcom.sun.management.jmxremote.port=7299 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false com.tinkerpop.rexster.Application -s -wr public -c ../conf/rexster-cassandra-es.xml

pld6     21168 21159  1 16:26 pts/4    00:00:01 /usr/lib/jvm/jdk1.7.0_45/bin/java -server -Xms32m -Xmx512m -cp ./bin/../conf:./bin/../lib/activation-1.1.jar:./bin/../lib/akiban-persistit-3.3.0.jar:./bin/../lib/ant-1.8.3.jar:./bin/../lib/ant-launcher-1.8.3.jar:./bin/../lib/antlr-2.7.7.jar:./bin/../lib/antlr-3.2.jar:./bin/../lib/antlr-runtime-3.2.jar:./bin/../lib/asm-3.1.jar:./bin/../lib/asm-analysis-3.1.jar:./bin/../lib/asm-commons-3.1.jar:./bin/../lib/asm-tree-3.1.jar:./bin/../lib/asm-util-3.1.jar:./bin/../lib/astyanax-cassandra-1.56.37.jar:./bin/../lib/astyanax-core-1.56.37.jar:./bin/../lib/astyanax-recipes-1.56.37.jar:./bin/../lib/astyanax-thrift-1.56.37.jar:./bin/../lib/avro-1.4.0-cassandra-1.jar:./bin/../lib/avro-ipc-1.5.3.jar:./bin/../lib/blueprints-core-2.4.0.jar:./bin/../lib/cassandra-all-1.2.2.jar...............................Let me know if you want to get the full output.

Thanks for your help.

Thanks

Puneet Arora

[Dan LaRocque]

Hi Puneet,

What is Cassandra writing to log/cassandra.log?

You could also try invoking titan.sh with the -v option, as in "sudo
./bin/titan.sh -v -c cassandra-es start", to get additional output on
the console.

Also, it's normally unnecessary to run titan.sh (in the default config)
with root privileges, but it shouldn't cause problems either.

thanks,
Dan

On 12/15/2013 04:36 PM, Puneet Arora wrote:
> Hi Matthias,
>
> No other Cassandra process if running on my machine. I check it by "ps
> -ef | grep cassandra".
>
> When I check the status, it shows that the processes are running with
> the following output.
>

> ~/titan-server-0.4.1$ *sudo ./bin/titan.sh -c cassandra-es status*

> Titan + Rexster (com.tinkerpop.rexster.Application) is running with
> pid 20957
> Cassandra (org.apache.cassandra.service.CassandraDaemon) is running
> with pid 20646
>
>
> One weird thing that happened was, even when I stopped the titan server

> using "* sudo ./bin/titan.sh -c cassandra-es stop"*

> It kills the Cassandra process and unable to kill the rexster process.

> Now, when I do "ps -ef *| grep rexster*" , it shown the REXSTER process
> running.
>
>
> *_/LOG:/_*
> pld6@pld6-virtual-machine:~/titan-server-0.4.1$*sudo ./bin/titan.sh -c
> cassandra-es stop*

> Killing Titan + Rexster (pid 20957)...
> Killing Cassandra (pid 20646)...
>

> pld6@pld6-virtual-machine:~/titan-server-0.4.1$*sudo ./bin/titan.sh -c
> cassandra-es stop*

> Killing Titan + Rexster (pid 20957)...

> Cassandra (org.apache.cassandra.service.CassandraDaemon) *not found in
> the java process table*
> *
> *
> pld6@pld6-virtual-machine:~/titan-server-0.4.1$*ps -ef | grep rexster*

> root 20948 1 0 16:16 pts/3 00:00:00 /bin/bash
> ./bin/rexster.sh -s -wr public -c ../conf/rexster-cassandra-es.xml
> root 20957 20948 2 16:16 pts/3 00:00:18
> /usr/lib/jvm/jdk1.7.0_45/bin/java -server -Xms128m -Xmx512m
> -Dtitan.logdir=../log -Dcom.sun.management.jmxremote.port=7299
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> com.tinkerpop.rexster.Application -s -wr public -c

> ../conf/rexster-*cassandra*-es.xml

> pld6 21168 21159 1 16:26 pts/4 00:00:01
> /usr/lib/jvm/jdk1.7.0_45/bin/java -server -Xms32m -Xmx512m -cp

> ./bin/../conf:./bin/../lib/activation-1.1.jar:./bin/../lib/akiban-persistit-3.3.0.jar:./bin/../lib/ant-1.8.3.jar:./bin/../lib/ant-launcher-1.8.3.jar:./bin/../lib/antlr-2.7.7.jar:./bin/../lib/antlr-3.2.jar:./bin/../lib/antlr-runtime-3.2.jar:./bin/../lib/asm-3.1.jar:./bin/../lib/asm-analysis-3.1.jar:./bin/../lib/asm-commons-3.1.jar:./bin/../lib/asm-tree-3.1.jar:./bin/../lib/asm-util-3.1.jar:./bin/../lib/astyanax-*cassandra*-1.56.37.jar:./bin/../lib/astyanax-core-1.56.37.jar:./bin/../lib/astyanax-recipes-1.56.37.jar:./bin/../lib/astyanax-thrift-1.56.37.jar:./bin/../lib/avro-1.4.0-*cassandra*-1.jar:./bin/../lib/avro-ipc-1.5.3.jar:./bin/../lib/blueprints-core-2.4.0.jar:./bin/../lib/*cassandra*-all-1.2.2.jar...............................Let

> me know if you want to get the full output.
>
>
> Thanks for your help.
>
> Thanks
> Puneet Arora
>
>

--
Dan LaRocque
http://www.thinkaurelius.com

[Puneet Arora]

Hi Dan,

I am unable to find anything in the Cassandra logs, except the exception in rexstitan.log.

Please find attached all the required logs, maybe I missed something.

Kindly, inform if we require any other logs to debug the issue.

I am stuck with this issue and unable to proceed. 

I really appreciate your help on this.

Thanks

Puneet Arora

[Dan LaRocque]

On 12/16/2013 02:09 PM, Puneet Arora wrote:
> Hi Dan,
>
> I am unable to find anything in the Cassandra logs, except the exception
> in rexstitan.log.
> Please find attached all the required logs, maybe I missed something.
>
> Kindly, inform if we require any other logs to debug the issue.
>
> I am stuck with this issue and unable to proceed.
> I really appreciate your help on this.
>

Hi Puneet,

This line in Cassandra's log is unusual:

INFO [main] 2013-12-16 13:31:50,795 ThriftServer.java (line 90) Binding
thrift service to /0.0.0.0:9160

listen_address and rpc_address in the conf/cassandra.yaml file shipped
with 0.4.1 are set to "localhost". Here's how that logline normally looks:

INFO [main] 2013-12-16 12:34:56,789 ThriftServer.java (line 90) Binding
thrift service to localhost/127.0.0.1:9160

Have you altered conf/cassandra.yaml or conf/rexster-cassandra-es.xml?
Unless you have altered these files, I suspect the host's DNS resolver
is misconfigured. Please send the output of the following two commands
from the same host that runs titan.sh. Neither command requires root
privilege.

1. getent hosts localhost
2. host localhost

Looking closely at the Astyanax exception earlier in the thread, I see
that it too refers to the address 0.0.0.0, which also makes me suspect
the resolver.

thanks,
Dan

[Puneet Arora]

Hi Dan,

Yes I have changed the cassandra.yaml in that I replaced rpc_address:localhost to rpc_address:0.0.0.0 .

I think it was stated in the documentation, moreover I was facing and exception, if I don't have it that way. 

I tried having the rpc_address as machine Ethernet address <192.168.0.195> but was giving an Exception, therefore I chose to use Wildcard 0.0.0.0

A copy of both the config file are provided in my first post on this thread, maybe you can have a look at it.

The OUTPUT:

:~/titan-server-0.4.1$ getent hosts localhost

127.0.0.1       localhost

:~/titan-server-0.4.1$ host localhost

localhost.cse.buffalo.edu has address 127.0.0.1

Let me know if you want me to send you the logs with rpc_address:localhost

Also, let me tell you this, I am performing a performance analysis of Neo4j and Titan+Cassandra. 

Therefore I require the setup to be a cluster with 4-5 machines on same network. 

Using titan server 0.4.1, I am able to setup the cluster and connect the machines.

Thanks

Puneet

[Dan LaRocque]

On 12/16/2013 02:55 PM, Puneet Arora wrote:
> Hi Dan,
>
> Yes I have changed the cassandra.yaml in that I replaced

> rpc_address:localhost torpc_address:0.0.0.0 .

> I think it was stated in the documentation, moreover I was facing and
> exception, if I don't have it that way.
> I tried having the rpc_address as machine Ethernet address
> <192.168.0.195> but was giving an Exception, therefore I chose to use
> Wildcard 0.0.0.0

What exception, exactly? Can you please send the full trace?

Also, listen_address in your cassandra.yaml is 192.168.0.194, so I'm
going to assume that either .194 or .195 was a typo.

[Puneet Arora]

Hi Dan,

yes that was a typo.

Let me walk you through the configurations.
I have 2 machines, lets name it as MACHINE_194 and MACHINE_195.

CONFIGURATION RPC ETHERNET:

MACHINE_194:

rpc_address: 192.168.0.194
listen_address: 192.168.0.194
seeds: "192.168.0.194"

MACHINE_195:

rpc_address: 192.168.0.195
listen_address: 192.168.0.195
seeds: "192.168.0.194"

LOG FILE  : <>_ethernet.log

CONFIGURATION RPC LOCALHOST:

MACHINE_194:

rpc_address: localhost
listen_address: 192.168.0.194
seeds: "192.168.0.194"

MACHINE_195:

rpc_address: localhost
listen_address: 192.168.0.195
seeds: "192.168.0.194"

LOG FILE  : <>_localhost.log

I think both the logs show the same exception.

Thanks
Puneet

[Puneet Arora]

Hi All,

It would be great if you could help me on this. I am stuck with this, and unable to proceed with my Performance analysis of Titan + Cassandra and Neo4j.

I am currently trying my luck with titan-0.4.0.

Looking forward for your help.

Thanks

Puneet Arora

[Marko Rodriguez]

Hi Puneet,

It seems like we have hit a dead end as no one is responding to this thread at the haste by which you desire.

Given that you are new to the technology, perhaps you could start off simple. Download Titan 0.4.1 and test/play/learn using only a single machine.

From there, as you get more comfortable with the technology, you can start adding more machines and grow your cluster.

Good luck with your endeavors,

Marko.

http://thinkaurelius.com

[Puneet Arora]

Hi Marko,

Yes, I am new to Titan, and need to learn a lot more about it.

But have spend almost every hour from past 15 days in reading, understanding and configuring Titan+Cassandra on a Cluster of 5 machines.

I have learnt Gremlin, working with Rexster server, configuring Cassandra and various mode of Cassandra as backend, Titan Blueprint API, Connecting java programs using RexPro and querying the sever using Gremlin. And have been following this community from past 2-3 months.

That said, yes I have hit a dead end and have tried everything to resolve this problem but was unsuccessful.

After your email, I also tried using a fresh copy of titan-server-0.4.0 and start a fresh, but facing the same issue and Unable to proceed due to same Exception. 

Moreover, I am facing this exception even if I do not connect to cluster i.e. on single machine.

STEPS TO REPRODUCE THE EXCEPTION:

1. Grab a fresh copy of titan-server-0.4.0
2. Unzip it.
3. run the server "sudo ./bin/titan.sh -c cassandra-es start"

1. login to resxter console and run the following commands:

• rexster[groovy]>  g = rexster.getGraph("graph")
  

• ==>titangraph[cassandra:null]             // my output
  

• rexster[groovy]>  g.addVertex([name:'marko'])
  

• ==>v[160004]                             // my output
  

1. Stop the server using "sudo ./bin/titan.sh stop".

1. wait for 30-40 s to let the rexster server to be stopped
2. check the rexstitan.log 
3. RESULT : I get the Exception (Exception stated in my first post)

This make me believe that maybe I am missing a small detail, that could be easily identified by Experts. 

Therefore, I came to get help from the Experts.

I would be of great help if people from community could help me on this.

I think that what Google groups are made for.

Looking forward for your help

Thanks

Puneet Arora

[Matthias Broecheler]

Hi Puneet,

are you running this on OS X? Which version? We have encountered an issue in the titan.sh script for some versions of OS X. That might explain the issue. Otherwise, the steps you are following seem to be identical to the documentation.

Thanks,

Matthias

[Matthias Broecheler]

BTW, another minor point: the fact that the vertex you are adding is assigned id 160004 indicates that there is other data in the database you are loading into. Make sure to clean out existing data. This is not the source of the problem but I thought it worth pointing out since that might cause problems down the road due to data conflicts.

[Puneet Arora]

Hi Matthias,

Thanks for your reply.

I am not using OS X. I am running it on Ubuntu 12.04 VM's.

In you earlier reply you pointed that titangraph[cassandra:null] means Rexster is not able to connect to Cassandra.

There are no issues or Exceptions present in Cassandra.log. The only exception I am facing is in Rexster.log. 

Also, id 160004 indicates that there are other elements in the database, but I am unable to fetch any of them through rexster.

Kindly,let me know if you require a copy of logs.

Thanks

Puneet

[Matthias Broecheler]

Mmh, that's bizarre. We are not able to reproduce your issue on Linux. Are you using Titan 0.4.1?

[Stephen Mallette]

I was unable to recreate it with 0.4.1 with ubuntu 12.  Things seemed to work fine.  

> In you earlier reply you pointed that titangraph[cassandra:null] means Rexster is not able to connect to Cassandra.

matthias, i've noticed that "null" and it always shows that way, I think.  It doesn't seem to have an effect on proper operations with Rexster and Titan.  I guess I should have dug into it sooner.  Anyway, I'm not sure why it does show that way since it really is just the result of the Gremlin ScriptEngine calling a toString on the Graph instance.  Any idea why that toString call would get mangled this way on the Titan side in that context?

Stephen

[Puneet Arora]

Hi Stephen,

Hi Matthias,

I understand that you guys are not able to replicate the issue.

Therefore I have created a screen recording to display the steps I am following and the Exception.

I understand that you will be busy with your schedule but, at your convenience kindly take out 5 mins to see the video.

LINK : http://youtu.be/BmLq6mgWb70

PROCEDURE:

1. I have unziped a fresh downloaded titan-server0.4.1 file
   
2. started titan server along with the cassandra.log and rexstitan.log
   
3. added vertex through rexster terminal.
   
4. Stop the server And got Exception in rexstitanl.log.
   

PS: I had to modify the titan.sh script to stop the server as "jps" was not available in the path.

Hoping to end this problem and find a solution.

Thanks

Puneet Arora

[Puneet Arora]

Hi All,

I have found the problem. and the solution to it.

PROBLEM :

when we do ./bin/titan.sh stop it kills the Rexster server first, and then Cassandra.

But Rexster takes time to shutdown and close. So the script was killing cassandra first, meanwhile Rexster was still in its shutdown process, which was causing the exception.

SOLUTION:

in the titan.sh script i added  SLEEP 60

Now when rexster get killed it waits for 60 seconds in order to complete the rexster shutdown process successfully.

And then after it kills Cassandra.

On my process in setting up titan+cassandra I made 2 changes to the titan.sh script, which made the shell script work for me.

Maybe, these changes are not required for others but they worked for me.

Change 1: (jps was not available at /usr/bin as I installed sun jdk using tar)

local p=`${JAVA_HOME}/bin/jps -l | grep "$2" | awk '{print $1}'`

Change 2: (solution to the exception at the beginning of this thread)

stop() {

    kill_class 'Titan + Rexster' com.tinkerpop.rexster.Application 

    sleep 60

    kill_class Cassandra org.apache.cassandra.service.CassandraDaemon

}

Thanks Matthias, Thanks Stephen, Thanks Dan and Thanks Marko.

Maybe I would have kept looking for the problem in wrong place if you would have not suggested to start at fresh with single machine.

That said, now I can continue with my Performance Analysis of Titan+Cassandra and Neo4j.

Will keep you guys posted with the results. And also, maybe I would require your help in performance tuning Titan+Cassandra later on.

Thanks again !

Have a great day !

Puneet Arora

[Matthias Broecheler]

Hi Puneet,

glad to hear you were able to sort things out. We will see if we can improve the script to make things more robust. Thanks for the pointers.

Definitely consult the mailing list for tips and tricks around performance tuning.

Cheers,

Matthias

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.