WriteConf for TTL/writetime not honored on delete/updates (appending collections)

12 views
Skip to first unread message

Will Fraley

unread,
Aug 15, 2024, 1:08:19 PM8/15/24
to DataStax Spark Connector for Apache Cassandra
When attempting to set the TTL when saving a CQL collection in "append" mode as shown here, the TTL writeconf is not honored and the TTL is not set.

.saveToCassandra(
  keyspaceName = config.cassandra.keyspace,
  tableName = config.cassandra.table,
  columns = SomeColumns(
    "id",
    "set_collection_col".append),
    writeConf = WriteConf(ttl = TTLOption.constant(config.cassandra.writeTtl.toInt)))


This is the same issue that was originally described in this thread:
https://groups.google.com/a/lists.datastax.com/g/spark-connector-user/c/unKbCYZbIXo/m/sIa9OdbzAQAJ

It was subsequently submitted and fixed in the following Jira and associated pr:
https://datastax-oss.atlassian.net/browse/SPARKC-505
https://github.com/datastax/spark-cassandra-connector/pull/1142/files

It appears that the last version having the fix from SPARKC-505 is 2.4.3:

https://github.com/datastax/spark-cassandra-connector/blob/v2.4.3/spark-cassandra-connector/src/main/scala/com/datastax/spark/connector/writer/TableWriter.scala#L92

However, as of 2.5.0 it appears to have been reverted:
https://github.com/datastax/spark-cassandra-connector/blob/v2.5.0/connector/src/main/scala/com/datastax/spark/connector/writer/TableWriter.scala#L102

I'm guessing this was a miss, but looking to see if there was a specific reason this was reverted.  I've patched my local connector codebase for now and it's working well, but hoping to get this back in if that's the right path forward.

Thank you!

Will
Reply all
Reply to author
Forward
0 new messages