Cassandra 2.0.5 timestamp Fields: Cannot read property 'ser' of undefined

[pmo...@gmail.com]

We are having a problem with Helenus that just started happening with Cassandra. We were fine with 2.0.3 but either the move to 2.0.5 or something else in the table construction changed. The problem appears very similar to the ones here:

https://github.com/simplereach/helenus/pull/88

There is a patch here:
https://github.com/retailnext/helenus/commit/8b9f80e7283ffd2b73128f200edf975e1012618e
and here
https://github.com/retailnext/helenus/commit/dbe076d55bf91fcc88a585fcfd09d4e9acc2b239

Unfortunately, the root cause seems very different. Any field with a timestamp seems to be failing.

I can work around the problem by not selecting the field but, obviously, that's not a good solution since we actually need the data.

For example:
- Select * on the table below fails
- SELECT customer_id, verification_field_hash FROM customers WHERE provider_id = ? AND customer_identifier = ?

works fine.

Here is the table, any thoughts on how I could alter the table but still use timestamps?

CREATE TABLE customers (
provider_id uuid,
customer_id uuid,
customer_identifier text,
verifiable_until timestamp,
verification_field_hash text,
PRIMARY KEY (provider_id, customer_id)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};

Here is exception:
2014-02-16T23:51:32.403Z - error: uncaughtException: Cannot read property 'ser' of undefined date=Sun Feb 16 2014 23:51:32 GMT+0000 (UTC), pid=15360, uid=0, gid=0, cwd=/, execPath=/usr/bin/node, version=v0.10.25, argv=[/usr/bin/node, /opt/docsforce/acvs-node/app.js, /usr/local/etc/docsforce.json, 3000, /opt/docsforce/acvs-node/acvs.log], rss=51863552, heapTotal=38338816, heapUsed=22793872, loadavg=[0.03466796875, 0.14697265625, 0.17041015625], uptime=875.053585379, trace=[column=48, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/marshal/index.js, function=getSerializer, line=239, method=null, native=false, column=20, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/marshal/index.js, function=new Marshal, line=278, method=null, native=false, column=26, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/row.js, function=new Row, line=40, method=null, native=false, column=19, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/connection.js, function=onReturn, line=450, method=null, native=false, column=7, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/connection.js, function=onReturn, line=387, method=null, native=false, column=11, file=client._reqs.(anonymous function) (/opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js, function=null, line=80, method=null, native=false, column=12, file=/opt/docsforce/acvs-node/node_modules/helenus/lib/cassandra/Cassandra.js, function=Object.CassandraClient.recv_execute_cql3_query, line=8343, method=CassandraClient.recv_execute_cql3_query, native=false, column=37, file=/opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js, function=null, line=83, method=null, native=false, column=9, file=/opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/transport.js, function=, line=70, method=null, native=false, column=17, file=events.js, function=Socket.EventEmitter.emit, line=95, method=EventEmitter.emit, native=false], stack=[TypeError: Cannot read property 'ser' of undefined, at getSerializer (/opt/docsforce/acvs-node/node_modules/helenus/lib/marshal/index.js:239:48), at new Marshal (/opt/docsforce/acvs-node/node_modules/helenus/lib/marshal/index.js:278:20), at new Row (/opt/docsforce/acvs-node/node_modules/helenus/lib/row.js:40:26), at onReturn (/opt/docsforce/acvs-node/node_modules/helenus/lib/connection.js:450:19), at onReturn (/opt/docsforce/acvs-node/node_modules/helenus/lib/connection.js:387:7), at client._reqs.(anonymous function) (/opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:80:11), at Object.CassandraClient.recv_execute_cql3_query (/opt/docsforce/acvs-node/node_modules/helenus/lib/cassandra/Cassandra.js:8343:12), at /opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:83:37, at Socket.<anonymous> (/opt/docsforce/acvs-node/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/transport.js:70:9), at Socket.EventEmitter.emit (events.js:95:17)]

[pmo...@gmail.com]

Ok, in a nutshell
- timestamps worked IF we upgraded to 2.x from 1.2.8
- timestamps DON'T work if you start from 2.x and create your tables there

I suspect something in the upgrade from 1.2.8 to 2.x keeps the 1.2.8 data structures intact and the driver keeps working. So, basically, if you want to use Helenus, you have to create your tables in 1.2.8 then move to 2.x and you can really only use those tables.

At least, that's what is working for me.

Another thing could be a setting in the cassandra.yaml. In the move from 1.2.8 I used my 1.2.8 yaml file for 2.x and did a minimum number of changes (removed 4 keys), this means there are a ton of things that are not changed for 2.x ... so, if it's not data formats in 2.x that are making helenus unusable, it is one of the new options in the the yaml that probably leads to a format change or thrift change.

[Russell Bradberry]

Interesting.  Did you try the patches you showed above. They were authored a while ago so I doubt they were created to address this issue.  I'll look into what may be different in 2.0.5. Would it be possible to create a test that demonstrates the breakage? 

[pmo...@gmail.com]

I will see if I can build a quick test on 2.0.5 that wasn't migrated, I kept one of our clusters here to work with ...

It should be as simple as:
CREATE TABLE customers (
customer_id uuid,
verifiable_until timestamp,
PRIMARY KEY (customer_id)
)

select verifiable_until from customers;

Let me see if this fails ... it will be a few hours though, we are on a firedrill at the moment ...