Schema changes Not getting propogated

[mayank pesswani]

Hi,

I have a use case :

1: frequently Create UDT, Create Table(using udt),  Perform Operation, Delete UDT and Delete table.

2: Perform step 1 in multiple region.

This leads to keyspace in inconsistent state with below error message

Unknown type <UDT Name>. keyspace is no longer usable.

Observation:

nodetool describecluster  : has multiple schema versions in output.

on performing any operation on table it is showing unconfigured table <table name>,

while  UDT doesn't exists.

Configuration In Use : 

Cassandra 3.11 version

driver version

<dependency>

    <groupId>com.datastax.cassandra</groupId>

    <artifactId>cassandra-driver-core</artifactId>

    <version>3.3.0</version>

</dependency>

Any suggestion How to use it.

[mayank pesswani]

Any suggestion How to fix and why it is occuring.

Any suggestion How to fix it.

[Andrew Tolbert]

Hi Mayank,

With regards to:

nodetool describecluster  : has multiple schema versions in output.

Eventually, if you keep running that command, does it ever resolve to only one schema version?  If it doesn't, that sounds like a problem on the C* cluster.  It should eventually coalesce into one schema version, although it shouldn't usually take longer than a minute but it might depend on your cluster topology.   

After making a schema change with the driver, it will by default keep checking the schema versions until the schema agrees across the cluster.  If agreement is not met within 10 seconds, the driver will return.  You can read more about that process here.

2: Perform step 1 in multiple region.

Just to be clear, are you making that same schema change concurrently on multiple clients?  If that is the case that could be the problem.  You should avoid making concurrent schema changes in general, and you should definitely avoid making the *same* schema change concurrently, as that could cause weird behavior such as this.   Schema changes should be done infrequently and when they are done they should only be done from a single client to avoid these kinds of issues.

Thanks,

Andy

[mayank pesswani]

Hi Andy,

Thanks for reply. here i have few inputs & queries.

With regards to:

nodetool describecluster  : has multiple schema versions in output.

    Eventually, if you keep running that command, does it ever resolve to only one schema version?  If it doesn't, that sounds like a problem on the C* cluster.  It should eventually coalesce into one schema version, although it shouldn't usually take longer than a minute but it might depend on your cluster topology.   

After making a schema change with the driver, it will by default keep checking the schema versions until the schema agrees across the cluster.  If agreement is not met within 10 seconds, the driver will return.  You can read more about that process here.

          It's never getting resolved, even after 12 hours. I am able to see 6 different schema version in my 3 region (5 nodes each region) cluster.

2: Perform step 1 in multiple region.

Just to be clear, are you making that same schema change concurrently on multiple clients?  If that is the case that could be the problem.  You should avoid making concurrent schema changes in general, and you should definitely avoid making the *same* schema change concurrently, as that could cause weird behavior such as this.   Schema changes should be done infrequently and when they are done they should only be done from a single client to avoid these kinds of issues.

        Yeah the schema changes are frequent from different clients. I hope by *same* schema you mean by same keyspace. if that's, yes multiple clients are modifying same keyspace.

Q: Is it something related to 3.3.0 driver. Are there any changes in 3.4.0 version w.r.t schema?

Q: what is the suggested way for these scenario. As we are using UDT as well  and being multi region cluster the client making schema changes will be different , in worst case might be they are modifying same schema?

Thanks,

Mayank

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

--

Regards,

Mayank Pesswani
9711823725

[Andrew Tolbert]

Yeah the schema changes are frequent from different clients. I hope by *same* schema you mean by same keyspace. if that's, yes multiple clients are modifying same keyspace.

I mean the same schema change statement.  I.e. Are you running the same CQL command to make a schema change on multiple clients concurrently?  I would be worried about doing that as I'm not sure how that might be managed when schema versions are in conflict among your cluster.

 

Q: Is it something related to 3.3.0 driver. Are there any changes in 3.4.0 version w.r.t schema?

I very much doubt it, the driver itself doesn't manage schema changes, it just sends the requests to a C* coordinator and C* handles it.

Q: what is the suggested way for these scenario. As we are using UDT as well  and being multi region cluster the client making schema changes will be different , in worst case might be they are modifying same schema?

I would not recommend making numerous schema changes among a number of clients in your cluster.  While I wouldn't say it is required, but preferable, that schema changes would be few and far between and only executed from one client at a time.  If you could avoid the possibly of concurrent schema changes, I think you will have an easier time.

The ideal process is:

1. Make a schema change.

2. Check ResultSet.getExecutionInfo().isSchemaInAgreement() on the returned result from the schema change.  You can use withMaxSchemaAgreementSeconds to increase the wait time on waiting for agreement.  If isInSchemaInAgreement is false, either call Cluster.getMetadata.checkSchemaAgreement() until it returns true or throw an Exception.

3. When the schema agrees, go to 1 to make your next change.

This is the safest way to ensure your schema doesn't get in an inconsistent state where nodes in a cluster don't agree.  If you are only making one schema change at a time, it's much less likely to get in this situation.  At worst you'll have two separate schema versions in the cluster (which might be easier to diagnose).

Thanks,

Andy

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.