ERROR "at java.util.Collections$UnmodifiableMap.get(Collections.java:1456"

[Rahul S]

We are getting the following error in loop:

2020-11-04 11:55:11,569 ERROR [Thread-61961] CassandraDaemon.java:241 uncaughtException Exception in thread Thread[Thread-61961,5,main]
java.lang.StackOverflowError: null
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	. . . (many lines)

2020-11-04 11:55:11,569 WARN  [elasticsearch[11.11.11.11][http_server_worker][T#3]] Netty4HttpServerTransport.java:575 exceptionCaught caught exception while handling client http traffic, closing connection [id: 0xdedfabe7, L:/11.11.11.11:9200 - R:/11.11.11.11:34024]
java.lang.StackOverflowError: null
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
	at java.util.Collections$UnmodifiableMap.get(Collections.java:1456)
    . . . (many lines)
   

Server Configs:
3 DC Cluster
DC-1 - 6 Cassandra Only 
DC-2 - 6 Cassandra Only
DC-3 - 9 (Elasticsearch is enabled only in this DC alone

Version: Elassandra 6.8.4.7
Deployed on AWS
Instance types: m5.2xlarge	(8 core	32 GiB)
OS: RHEL
Java: Open JDK Version 1.8.0.265

As a result of this error the DC-3 nodes will first show very high response times for elasticsearch queries and eventually nodes shutdown. 
When we restart the nodes everything will be back to normal again.


Process grep output:
cassand+  2715     1  3 Nov04 ?        00:48:10 java -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -XX:StringTableSize=1000003 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -Xms16G -Xmx16G -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Des.search_strategy_class=RackAwareSearchStrategy -Dcom.sun.management.jmxremote.access.file=/usr/lib/jvm/zulu-8/jre/lib/management/jmxremote.access -Xmn800M -XX:+UseCondCardMark -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -Dcassandra.jmx.remote.port=7199 -Dcom.sun.management.jmxremote.rmi.port=7199 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -Djdk.io.permissionsUseCanonicalPath=true -Des.distribution.flavor=oss -Des.distribution.type=rpm -Djava.awt.headless=true -XX:OnOutOfMemoryError=kill -9 %p -Dlogback.configurationFile=/etc/cassandra/logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=/var/lib/cassandra -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/ant-1.7.0.jar:/usr/share/cassandra/lib/ant-launcher-1.7.0.jar:/usr/share/cassandra/lib/antlr-3.5.2.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-all-3.11.6.1.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/cassandra-thrift-3.11.6.1.jar:/usr/share/cassandra/lib/commons-cli-1.3.1.jar:/usr/share/cassandra/lib/commons-codec-1.10.jar:/usr/share/cassandra/lib/commons-lang3-3.4.jar:/usr/share/cassandra/lib/commons-logging-1.2.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/elasticsearch-6.8.4.7.jar:/usr/share/cassandra/lib/elasticsearch-cli-6.8.4.7.jar:/usr/share/cassandra/lib/elasticsearch-core-6.8.4.7.jar:/usr/share/cassandra/lib/elasticsearch-launchers-6.8.4.7.jar:/usr/share/cassandra/lib/elasticsearch-secure-sm-6.8.4.7.jar:/usr/share/cassandra/lib/elasticsearch-x-content-6.8.4.7.jar:/usr/share/cassandra/lib/fastutil-6.5.7.jar:/usr/share/cassandra/lib/guava-19.0.jar:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/hibernate-validator-4.3.0.Final.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.7.1.jar:/usr/share/cassandra/lib/httpclient-4.5.2.jar:/usr/share/cassandra/lib/httpcore-4.4.5.jar:/usr/share/cassandra/lib/jackson-core-2.8.11.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.13.jar:/usr/share/cassandra/lib/jackson-dataformat-cbor-2.8.11.jar:/usr/share/cassandra/lib/jackson-dataformat-smile-2.8.11.jar:/usr/share/cassandra/lib/jackson-dataformat-yaml-2.8.11.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.13.jar:/usr/share/cassandra/lib/javassist-3.20.0-GA.jar:/usr/share/cassandra/lib/java-version-checker-6.8.4.7.jar:/usr/share/cassandra/lib/javax.inject-1.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jboss-logging-3.1.0.CR2.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jmxremote_optional-repackaged-5.0.jar:/usr/share/cassandra/lib/jna-4.5.1.jar:/usr/share/cassandra/lib/joda-time-2.10.1.jar:/usr/share/cassandra/lib/jopt-simple-5.0.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jts-core-1.15.0.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-api-2.11.1.jar:/usr/share/cassandra/lib/log4j-core-2.11.1.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/log4j-to-slf4j-2.11.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.8.jar:/usr/share/cassandra/lib/logback-core-1.1.8.jar:/usr/share/cassandra/lib/lucene-analyzers-common-7.7.2.jar:/usr/share/cassandra/lib/lucene-backward-codecs-7.7.2.jar:/usr/share/cassandra/lib/lucene-core-7.7.2.jar:/usr/share/cassandra/lib/lucene-grouping-7.7.2.jar:/usr/share/cassandra/lib/lucene-highlighter-7.7.2.jar:/usr/share/cassandra/lib/lucene-join-7.7.2.jar:/usr/share/cassandra/lib/lucene-memory-7.7.2.jar:/usr/share/cassandra/lib/lucene-misc-7.7.2.jar:/usr/share/cassandra/lib/lucene-queries-7.7.2.jar:/usr/share/cassandra/lib/lucene-queryparser-7.7.2.jar:/usr/share/cassandra/lib/lucene-sandbox-7.7.2.jar:/usr/share/cassandra/lib/lucene-spatial3d-7.7.2.jar:/usr/share/cassandra/lib/lucene-spatial-7.7.2.jar:/usr/share/cassandra/lib/lucene-spatial-extras-7.7.2.jar:/usr/share/cassandra/lib/lucene-suggest-7.7.2.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.5.jar:/usr/share/cassandra/lib/netty-all-4.1.32.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.4.jar:/usr/share/cassandra/lib/plugin-classloader-6.8.4.7.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.25.jar:/usr/share/cassandra/lib/snakeyaml-1.17.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/spatial4j-0.7.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/t-digest-3.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/validation-api-1.0.0.GA.jar org.apache.cassandra.service.ElassandraDaemon


We have nested fields in the index as well:


"foo_bar": {
     "type": "nested",
     "cql_collection": "singleton",
     "cql_udt_name": "foo_bar",
     "properties": {
         "x": {
             "type": "date",
             "index": false,
             "cql_collection": "singleton"
         },
         "y": {
             "type": "keyword",
             "index": false,
             "cql_collection": "singleton"
         },
         "z": {
             "type": "keyword",
             "index": false,
             "cql_collection": "singleton"
         }

[Vincent Royer]

Hi Rahul,

Could you check for OOM or StackOverflow errors in your logs.

On possible cause is StackOverflow error, may need to increase the Stack size with  -Xss2m  or more.

- Vincent

--
You received this message because you are subscribed to the Google Groups "Elassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elassandra+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elassandra/CAP1y0yYFGAYCsM1Hee-cAGAqCXYNOwbeYo0yrYcOx18jVaExSQ%40mail.gmail.com.