Cassandra 3.2 crashing at cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)

[Saurabh Gupta]

Hello There,

Issue:

Cassandra crashing in every 3-4 hrs with:

# J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x00002b7d3d417fb4 [0x00002b7d3d417c80+0x334]

COnfiguration:

-Xms8G

-Xmx16G

-apache-cassandra-3.2 with Java - 1.8.0_161-b12.

hs_err_pid.log:

# Problematic frame:

# J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x00002b7d3d417fb4 [0x00002b7d3d417c80+0x334]

#

---------------  T H R E A D  ---------------

Current thread (0x00002b7d3a1033e0):  JavaThread "SharedPool-Worker-1" daemon [_thread_in_Java, id=32216, stack(0x00002b7e4085f000,0x00002b7e408a0000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000014914c69

Registers:

RAX=0x0000000000000001, RBX=0x0000000000000000, RCX=0x000000009f1fbef0, RDX=0x00000004f8fdf798

RSP=0x00002b7e4089e4b0, RBP=0x0000000000000001, RSI=0x0000000014907800, RDI=0x0000000000000000

R8 =0x000000000000d469, R9 =0x0000000000000000, R10=0x00000004a41764c8, R11=0x0000000000000000

R12=0x0000000000000000, R13=0x0000000000000000, R14=0x000000000000d469, R15=0x00002b7d3a1033e0

RIP=0x00002b7d3d417fb4, EFLAGS=0x0000000000010283, CSGSFS=0x0000000000000033, ERR=0x0000000000000004

  TRAPNO=0x000000000000000e

[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00002b7e4085f000,0x00002b7e408a0000],  sp=0x00002b7e4089e4b0,  free space=253k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

J 8283 C2 org.apache.cassandra.db.marshal.TimestampType.compareCustom(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I (6 bytes) @ 0x00002b7d3d417fb4 [0x00002b7d3d417c80+0x334]

J 12970 C2 org.apache.cassandra.db.Slice$Bound.compareTo(Lorg/apache/cassandra/db/ClusteringComparator;Ljava/util/List;)I (119 bytes) @ 0x00002b7d3e0291c0 [0x00002b7d3e028900+0x8c0]

J 16245 C2 org.apache.cassandra.db.Slices$ArrayBackedSlices.intersects(Ljava/util/List;Ljava/util/List;)Z (46 bytes) @ 0x00002b7d3e619cfc [0x00002b7d3e619b20+0x1dc]

J 18878 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;Z)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (822 bytes) @ 0x00002b7d3ebcabf4 [0x00002b7d3ebc7be0+0x3014]

J 9377 C2 org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; (219 bytes) @ 0x00002b7d3d80cde8 [0x00002b7d3d80c0a0+0xd48]

J 14198 C2 org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(Lorg/apache/cassandra/net/MessageIn;I)V (328 bytes) @ 0x00002b7d3c8bcbd0 [0x00002b7d3c8bca20+0x1b0]

J 9731 C2 org.apache.cassandra.net.MessageDeliveryTask.run()V (187 bytes) @ 0x00002b7d3d158d60 [0x00002b7d3d158bc0+0x1a0]

J 18999% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ 0x00002b7d3eaa10ec [0x00002b7d3eaa0960+0x78c]

j  java.lang.Thread.run()V+11

v  ~StubRoutines::call_stub

V  [libjvm.so+0x695ae6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056

V  [libjvm.so+0x695ff1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321

V  [libjvm.so+0x696497]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47

V  [libjvm.so+0x731cb0]  thread_entry(JavaThread*, Thread*)+0xa0

V  [libjvm.so+0xa7eaa3]  JavaThread::thread_main_inner()+0x103

V  [libjvm.so+0xa7ebec]  JavaThread::run()+0x11c

V  [libjvm.so+0x92da28]  java_start(Thread*)+0x108

C  [libpthread.so.0+0x7e25]  start_thread+0xc5

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

  0x00002b7da57924a0 JavaThread "MemtableReclaimMemory:52" daemon [_thread_blocked, id=117880, stack(0x00002b7d917ff000,0x00002b7d91840000)]

  0x00002b7d39f6a9e0 JavaThread "PerDiskMemtableFlushWriter_0:52" daemon [_thread_blocked, id=117879, stack(0x00002b7e4ea94000,0x00002b7e4ead5000)]

  0x00002b7d39d0f520 JavaThread "MemtablePostFlush:53" daemon [_thread_blocked, id=117878, stack(0x00002b7e407dd000,0x00002b7e4081e000)]

  0x00002b7df31a9150 JavaThread "MemtableFlushWriter:52" daemon [_thread_blocked, id=117877, stack(0x00002b7e406d9000,0x00002b7e4071a000)]

  0x00002b7e53e60110 JavaThread "RMI TCP Connection(1795)-127.0.0.1" daemon 

:

:

lot of threads in BLOCKED status

Other Threads:

  0x00002b7d38de5ea0 VMThread [stack: 0x00002b7d8208d000,0x00002b7d8218d000] [id=32098]

  0x00002b7d38fa9de0 WatcherThread [stack: 0x00002b7d88ee9000,0x00002b7d88fe9000] [id=32108]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:

 garbage-first heap   total 8388608K, used 6791168K [0x00000003c0000000, 0x00000003c0404000, 0x00000007c0000000)

  region size 4096K, 785 young (3215360K), 55 survivors (225280K)

 Metaspace       used 40915K, capacity 42044K, committed 42368K, reserved 1087488K

  class space    used 4429K, capacity 4646K, committed 4736K, reserved 1048576K

Heap Regions: (Y=young(eden), SU=young(survivor), HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, TS=gc time stamp, PTAMS=previous top-at-mark-start, NTAMS=next top-at-mark-start)

AC   0  O    TS     0 PTAMS 0x00000003c0400000 NTAMS 0x00000003c0400000 space 4096K, 100% used [0x00000003c0000000, 0x00000003c0400000)

AC   0  O    TS     0 PTAMS 0x00000003c0800000 NTAMS 0x00000003c0800000 space 4096K, 100% used [0x00000003c0400000, 0x00000003c0800000)

AC   0  O    TS     9 PTAMS 0x00000003c0800000 NTAMS 0x00000003c0800000 space 4096K, 100% used [0x00000003c0800000, 0x00000003c0c00000)

AC   0  O    TS    11 PTAMS 0x00000003c0c00000 NTAMS 0x00000003c0c00000 space 4096K, 100% used [0x00000003c0c00000, 0x00000003c1000000)

AC   0  O    TS    11 PTAMS 0x00000003c1000000 NTAMS 0x00000003c1000000 space 4096K, 100% used [0x00000003c1000000, 0x00000003c1400000)

AC   0  O    TS    11 PTAMS 0x00000003c1400000 NTAMS 0x00000003c1400000 space 4096K, 100% used [0x00000003c1400000, 0x00000003c1800000)

:

:

lot of such messages

Has anyone faced similar issues ?