Suddenly slow commits

[Collin Scangarella]

Hello,

I have an application which relies on titan (which uses cassandra). Yesterday our commit times went from milliseconds to minutes without warning (and without any changes to our application - we hadn't deployed any new code nor changed any infrastructure). Now all write requests are timing out and we cannot identify why. It is a three node cluster with a 3X replication and Quorum consistency.

Here's our titan configuration:

gremlin.graph=com.thinkaurelius.titan.core.TitanFactory

storage.backend=cassandra

storage.hostname=3.IP.Addresses

index.search.backend=elasticsearch

index.search.elasticsearch.interface=TRANSPORT_CLIENT

index.search.hostname=IP.Address

schema.default=none

And our cassandra.yaml.

We've tried the following: 

1. Restarting the application
2. Compacting the nodes.
3. Force closing all open titan instances.
4. Reducing the gc_grace_seconds and compacting the nodes.
5. Repairing the cluster.

All TitanTransaction commit calls take over one minute - regardless of whether or not any information was changed. Additionally there is no logging about being unable to get a lock nor exceptions. Does anyone have any ideas about what might be causing this behaviour?

Collin

[Collin Scangarella]

I'd like to make a correction - it looks like commits that do not change anything happen in a normal duration:

//setup, read data
gremlin> start = new Date()

==>Thu Mar 16 22:50:54 UTC 2017

gremlin> tx.commit()

==>null

gremlin> (new Date().getTime() - start.getTime())/1000

==>0.386

While those that do make changes take a lot longer:

//setup, write data
gremlin> start = new Date()

==>Thu Mar 16 22:52:16 UTC 2017

gremlin> tx.commit()

==>null

gremlin> (new Date().getTime() - start.getTime())/1000

==>60.401

Interestingly it always takes just over sixty seconds, never 61 or more.

Thanks,

Collin

[Nkuli Stunna]

Hi as a side note what are you using as application. I am new to titan and seem to have a though time getting it to work.