Titan on Cassandra: From 1 datacenter/SimpleStrategy to 2 datacenters/NetworkTopologyStrategy

[Christophe Billiard]

Hello all,

We have a running titan 0.5.2 cluster, we want to add a new datacenter.
We have to change storage.cassandra.replication-strategy-class and storage.cassandra.replication-strategy-options.

It is very easy it directly with cqlsh, hence we have:
cqlsh> SELECT * FROM system.schema_keyspaces  ;

 keyspace_name | durable_writes | strategy_class                                       | strategy_options
---------------+----------------+------------------------------------------------------+------------------
        system |           True |           org.apache.cassandra.locator.LocalStrategy |               {}
         titan |           True | org.apache.cassandra.locator.NetworkTopologyStrategy |    {"DC1":"3","DC2":"3"}
 system_traces |           True | org.apache.cassandra.locator.NetworkTopologyStrategy |    {"DC1":"3"}

But it doesn't seem to be possible to change titan conf, attempting to do so gives the following warning:
[warn] c.t.t.g.c.GraphDatabaseConfiguration - Local setting storage.cassandra.replication-strategy-class=org.apache.cassandra.locator.NetworkTopologyStrategy (Type: FIXED) is overridden by globally managed value (org.apache.cassandra.locator.SimpleStrategy).  Use the ManagementSystem interface instead of the local configuration to control this setting.

And trying to modifying it with the management system gives (without any surprise since it is "FIXED"):
scala> mgmt.set("storage.cassandra.replication-strategy-class","org.apache.cassandra.locator.NetworkTopologyStrategy")
java.lang.IllegalArgumentException: Cannot change the fixed configuration option: root.storage.cassandra.replication-strategy-class
  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:148)
  at com.thinkaurelius.titan.graphdb.database.management.ManagementSystem$1.verifyModification(ManagementSystem.java:146)
  at com.thinkaurelius.titan.diskstorage.configuration.UserModifiableConfiguration.set(UserModifiableConfiguration.java:84)
  at com.thinkaurelius.titan.graphdb.database.management.ManagementSystem.set(ManagementSystem.java:1117)
  ... 43 elided

It is not very clear if the configuration is really applied to titan.

The only thing I can say for sure is that the reads/writes of my app do not respect the LOCAL_QUORUM
And I believe this to be the reason.
(set with
  storage.cassandra.write-consistency-level = LOCAL_QUORUM
  storage.cassandra.read-consistency-level = LOCAL_QUORUM
)

Any help would be very appreciated

[Ted Wilmes]

Maybe you've tried this, but if you're using the astyanax driver, you can set your local dc with storage.cassandra.astyanax.local-datacenter.  That might straighten things out.

http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_cassandra_astyanax

--Ted