Hazelcast 3.7 error with Zing JVM

[Joan Balagueró]

Hello,

After upgrading from Hz 3.6.2 to Hz 3.7 with Zing JVM, Tomcat log shows these errors:

01-Sep-2016 09:56:07.485 WARNING [localhost-startStop-1] com.hazelcast.config.AbstractXmlConfigHelper.null Name of the hazelcast schema location incorrect using default

01-Sep-2016 09:56:09.230 WARNING [localhost-startStop-1] com.hazelcast.config.AbstractXmlConfigHelper.null Name of the hazelcast schema location incorrect using default

01-Sep-2016 09:56:15.978 WARNING [hz._hzInstance_1_ventusproxy01.partition-operation.thread-5] com.hazelcast.util.JVMUtil.null Failed to read HotSpot specific configuration

 javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: VM option "UseCompressedOops" does not exist

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:821)

        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)

        at com.hazelcast.util.JVMUtil.isHotSpotCompressedOopsOrNull(JVMUtil.java:69)

        at com.hazelcast.util.JVMUtil.isCompressedOops(JVMUtil.java:46)

        at com.hazelcast.util.JVMUtil.<clinit>(JVMUtil.java:39)

        at com.hazelcast.internal.serialization.impl.HeapData.getHeapCost(HeapData.java:96)

        at com.hazelcast.map.impl.BinaryMapSizeEstimator.calculateSize(BinaryMapSizeEstimator.java:54)

        at com.hazelcast.map.impl.recordstore.StorageImpl.calculateHeapCost(StorageImpl.java:144)

        at com.hazelcast.map.impl.recordstore.StorageImpl.put(StorageImpl.java:75)

        at com.hazelcast.map.impl.recordstore.StorageImpl.put(StorageImpl.java:41)

        at com.hazelcast.map.impl.recordstore.DefaultRecordStore.putIfAbsent(DefaultRecordStore.java:946)

        at com.hazelcast.map.impl.operation.PutIfAbsentOperation.run(PutIfAbsentOperation.java:34)

        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:181)

        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:122)

        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102)

Caused by: java.lang.IllegalArgumentException: VM option "UseCompressedOops" does not exist

        at sun.management.HotSpotDiagnostic.getVMOption(HotSpotDiagnostic.java:73)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:281)

        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)

        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)

        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)

        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)

        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)

        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)

        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)

        at javax.management.StandardMBean.invoke(StandardMBean.java:405)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)

        ... 14 more

01-Sep-2016 09:56:15.982 WARNING [hz._hzInstance_1_ventusproxy01.partition-operation.thread-5] com.hazelcast.util.JVMUtil.null obj1

 java.lang.NoSuchFieldException: obj1

        at java.lang.Class.getField(Class.java:1584)

        at com.hazelcast.util.JVMUtil.isCompressedOopsOrNull(JVMUtil.java:93)

        at com.hazelcast.util.JVMUtil.isCompressedOops(JVMUtil.java:52)

        at com.hazelcast.util.JVMUtil.<clinit>(JVMUtil.java:39)

        at com.hazelcast.internal.serialization.impl.HeapData.getHeapCost(HeapData.java:96)

        at com.hazelcast.map.impl.BinaryMapSizeEstimator.calculateSize(BinaryMapSizeEstimator.java:54)

        at com.hazelcast.map.impl.recordstore.StorageImpl.calculateHeapCost(StorageImpl.java:144)

        at com.hazelcast.map.impl.recordstore.StorageImpl.put(StorageImpl.java:75)

        at com.hazelcast.map.impl.recordstore.StorageImpl.put(StorageImpl.java:41)

        at com.hazelcast.map.impl.recordstore.DefaultRecordStore.putIfAbsent(DefaultRecordStore.java:946)

        at com.hazelcast.map.impl.operation.PutIfAbsentOperation.run(PutIfAbsentOperation.java:34)

        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:181)

        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:122)

        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102)

Should I be worried about these messages, or they are only related to HZ server monitoring?

Thanks.

Joan.

[gam...@gmail.com]

Joan,

Please provide hazelcast configs and JVM startup parameters.
Thanks

[Joan Balagueró]

ZING JVM PARAMETERS

# Set Zing jvm parameters.

JAVA_OPTS="     -Xmx16g  \

                -Xms16g  \

                -verbose:gc \

                -XX:+PrintGCDetails  \

                -XX:+PrintGCDateStamps  \

                -XX:+PrintGCApplicationStoppedTime \

                -Xloggc:/var/log/ventus/gc.$(date -u +%Y-%m-%d-%H).log  \

                -XX:ARTAPort=50000 \

                -XX:+KeepSafepointsInCountedLoops \

                -XX:+ConcurrentDeflation \

                -XX:ReservedCodeCacheSize=768m \

                -XX:+DisableExplicitGC \

                -XX:+UnlockExperimentalVMOptions \

                -XX:+UseZingMXBeansExperimentalVersion \

                -Djava.awt.headless=true \

                -Duser.language=es \

                -Duser.country=ES \

                -Dfile.encoding=UTF-8 \

                -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true  \

                -Djava.net.preferIPv4Stack=true \

                -noverify \

                -Dcom.sun.management.jmxremote=true \

                -Djava.rmi.server.hostname=192.168.1.70 \

                -Dcom.sun.management.jmxremote.port=38921 \

                -Dcom.sun.management.jmxremote.ssl=false \

                -Dcom.sun.management.jmxremote.authenticate=false \

                -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true"

HAZELCAST XML CONFIG

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.7.xsd"

           xmlns="http://www.hazelcast.com/schema/config"

           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    

<!-- Properties -->

<properties>

<property name="hazelcast.mancenter.enabled">false</property> 

<property name="hazelcast.memcache.enabled">false</property> 

<property name="hazelcast.rest.enabled">false</property> 

<property name="hazelcast.logging.type">log4j</property> 

<property name="hazelcast.merge.next.run.delay.seconds">86400</property>

<property name="hazelcast.log.state">false</property>

<property name="hazelcast.jmx">false</property>

<property name="hazelcast.jmx.detailed">false</property>

<property name="hazelcast.health.monitoring.level">NOISY</property>

<property name="hazelcast.health.monitoring.delay.seconds">60</property>

</properties>

    <!-- Cache -->

<map name="Cache*">

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

    <!-- Indexes -->

<map name="Index*">

<time-to-live-seconds>0</time-to-live-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

    <!-- Load Balancer -->

<map name="BalancerGlobalServers">

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

<near-cache>

<time-to-live-seconds>0</time-to-live-seconds>

<max-idle-seconds>0</max-idle-seconds>

<eviction-policy>NONE</eviction-policy>

            <invalidate-on-change>true</invalidate-on-change>

</near-cache>

</map>

<map name="BalancerSession">

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

<near-cache>

<time-to-live-seconds>0</time-to-live-seconds>

<max-idle-seconds>0</max-idle-seconds>

<eviction-policy>NONE</eviction-policy>

            <invalidate-on-change>true</invalidate-on-change>

</near-cache>

</map>

<map name="BalancerMonitoring">

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

<map name="BalancerCallSessions">

<max-idle-seconds>3600</max-idle-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

<map name="BalancerClientSessions">

<max-idle-seconds>3600</max-idle-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

    <!-- Quotas -->

<map name="QuotaMap">

<time-to-live-seconds>0</time-to-live-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

    <!-- Monitoring -->

<map name="AlertMap">

<time-to-live-seconds>0</time-to-live-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

<map name="MonitorDisconnection">

<time-to-live-seconds>0</time-to-live-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

<!-- Lock and await -->

<map name="LWMap*">

<time-to-live-seconds>0</time-to-live-seconds>

<backup-count>0</backup-count>

<eviction-policy>NONE</eviction-policy>

</map>

</hazelcast>

[Vassilis Bekiaris]

Hi Joan,

thanks for reporing this; these warnings are related to Hazelcast attempt to estimate memory cost, first by attempting (and failing) at reading the Hotspot-specific UseCompressedOops option, then by attempting to read the size of a reference. I think logging these at warning level is a bit scarier than it should be. The second warning looks a bit suspicious to me so we would need to investigate this more. Would you like to open a github issue to reduce the log level and investigate the second stack trace at https://github.com/hazelcast/hazelcast/issues ? These warnings do not affect the functionality of your maps.

Cheers!
Vassilis

[Joan Balagueró]

Issue opened.

Thanks,

Joan.