Data cannot be retrieved after populating data in freshly created cluster

[Dmitry Buykin]

I created two-nodes cluster with single keyspace (replication_factor=2), 256 tokens, both nodes are seeds, in the same DC.

Started two applications with datastax java driver com.datastax.cassandra:cassandra-driver-dse:2.1.5 : 

- data-loader which populates the ~200K rows into cassandra (only inserts to C*)

    - LoadBalancingPolicy = RoundRobinPolicy

    - ConsistencyLevel = ALL

    - connected to both nodes

- data-service which selects data and exposes through REST interface for clients (only selects from C*)

    - LoadBalancingPolicy = RoundRobinPolicy

    - ConsistencyLevel = ONE

    - connected to both nodes

    

These two applications started simultaneously, data-loader is starting loading from legacy database, populating data (~200K) into C* and finished after ~12 minutes. Both applications are connected to both nodes (found in logs), data duplicated on two nodes.

After this step data cannot be retrieved through REST API (select returns empty dataset from C*), but could be found on both C* nodes using cqlsh console. After some delay (1-4 hrs) all data become available in data-service.

After some investigation I found that restarting data-service applications (reconnecting to cassandra cluster) helps to get results immediately after populating.

I think it's related to caching empty tokens in datastax java driver which seems to be updated really slow.

Please check the issue.

Regards, Dmitry.

[Dmitry Buykin]

Keyspace created using the next script:

CREATE KEYSPACE test with replication ={'class':'SimpleStrategy','replication_factor':2};

CREATE TABLE instrument (
  date ascii,
  tenor ascii,
  ccy ascii,
  datetime timestamp,
  content varchar,
  PRIMARY KEY ((date, tenor, ccy), datetime)
) WITH CLUSTERING ORDER BY (datetime DESC) AND caching = 'ALL';

[Andrew Tolbert]

Hi Dmitry,

Is there any indication on the 'data-service' side that queries are successful and returning no data or if they are failing (with some kind of exception, do you have any kind of logs)?

Whenever there is a schema change, the driver should detect it and rebuild it's local interpretation of both the schema and the token map granted it has an active control connection.   Even if the driver doesn't detect the change and is out of date with regards to the token map and schema (which is bound to the keyspace and it's replication strategy) it should still be able to query data, it just may not do it in the most efficient way (i.e. it may choose a coordinator node that doesn't own the data, but the coordinator will request the data from a node that does).

Thanks,

Andy

[Dmitry Buykin]

Hi Andrew,

I found a root cause in connectivity of environments, so it's not related to java driver. Sorry for bothering you.

Kind regards, Dmity.

[Denis Makarskiy]

Hi Dmitry,

Could you share the cause, because it seems to me I have the same problem.

After I filled huge table with data using Spark job, another job failed to read from this table - Cassandra falls with OOM exception.

Regards,

Denis

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Dmitry Buykin]

Hi Denis, 

My case was not related to any Cassandra exceptions, it was connectivity problem between the client and Cassandra nodes.

Cassandra was quite stable for our performance profile.

Regards, Dmitry.