hazelcast client -> cluster setup & unable to fetch all data via keyset (only getting keys from one node and not the other)

[Ali]

I am using hazelcast v3.6.  I have a Java client connected to a two node test cluster on amazon AWS (not using AWS specific config in the hazelcast server xml config).

I tried adding 30 objects via the hazelcast client into the test map.  I am able to fetch the objects from Hazelcast using the map.get call.  However, when I try to get the keySet for the map (not the localKeySet but the keySet), I am only getting half of the keys.  I tried getting the localKeySet and it gave me the exception copied below.  Server and client configs (relevant parts) are copied below.  I have not added any entry listeners or mapstore calls yet for this setup.   

The calls I am making on the maps:
//test Map is the IMap being referred to in the server config file. 
testMap.keySet();  // returns half the values  (I get the same result when I call values() on the map)
testMap.localKeySet();   //throws the exception

Exception in thread "main" java.lang.UnsupportedOperationException: Locality is ambiguous for client!!!
    at com.hazelcast.client.proxy.ClientMapProxy.localKeySet(ClientMapProxy.java:1172)

I tried getting the EntryView on the key-value pair that I am not getting back from the keySet call and that seems to return fine i.e. I can get the key/value pair and print the statistics out.  Calling size on the IMap gives me the count of 30 but the keySet being returned only has half the values (size() returns 15).  Any thoughts on what I might have misconfigured or misinterpreted to get to this state whereby the hazelcast client does not seem to be able to get the full keyset (and localKeySet is returning an exception). 

The two server nodes seem to have formed a cluster (one of the nodes was restarted 5 minutes after the initial start and was able to connect to the cluster).   

Server config (relevant part):

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <network>
        <port auto-increment="true" port-count="100">5701</port>
        <outbound-ports>
            <ports>0-5900</ports>
        </outbound-ports>
        <join>
            <multicast enabled="false">
            </multicast>
            <tcp-ip enabled="true">
<!-- IP addresses switched in config -->
                <member>101.102.103.104</member>
        <member>101.102.103.105</member>
             </tcp-ip>
        </join>
        <interfaces enabled="true">
            <interface>101.102.103.104</interface>
        </interfaces>
        <ssl enabled="false" />
        <socket-interceptor enabled="false" />
        <symmetric-encryption enabled="false">
        </symmetric-encryption>
    </network>
   
    <partition-group enabled="false"/>


    <map name="testMap">
        <async-backup-count>1</async-backup-count>
<!-- mistakenly added the near-cache part in the server config -->
        <near-cache>
            <max-size>5000</max-size>
            <invalidate-on-change>true</invalidate-on-change>
        </near-cache>
    </map>

  hazelcast client config (relevant part):

<hazelcast-client xsi:schemaLocation="http://www.hazelcast.com/schema/client-config hazelcast-client-config-3.6.xsd"
                  xmlns="http://www.hazelcast.com/schema/client-config"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <group>
        <name>dev</name>
        <password>dev-pass</password>
    </group>


    <properties>
        <property name="hazelcast.client.shuffle.member.list">true</property>
        <property name="hazelcast.client.heartbeat.timeout">60000</property>
        <property name="hazelcast.client.heartbeat.interval">5000</property>
        <property name="hazelcast.client.event.thread.count">5</property>
        <property name="hazelcast.client.event.queue.capacity">1000000</property>
        <property name="hazelcast.client.invocation.timeout.seconds">120</property>
    </properties>


    <network>
        <cluster-members>
            <address>101.102.103.104</address>
            <address>101.102.103.105</address>
        </cluster-members>
        <smart-routing>true</smart-routing>
        <redo-operation>true</redo-operation>
        <connection-timeout>60000</connection-timeout>
        <connection-attempt-period>3000</connection-attempt-period>
        <connection-attempt-limit>2</connection-attempt-limit>
        <socket-options>
            <tcp-no-delay>false</tcp-no-delay>
            <keep-alive>true</keep-alive>
            <reuse-address>true</reuse-address>
            <linger-seconds>3</linger-seconds>
            <timeout>-1</timeout>
            <buffer-size>32</buffer-size>
        </socket-options>
    </network>

[Jaromir Hamala]

Hi Ali,

I could not reproduce the issue. Can you share logs from both members and the client? Debug/Finest settings would help a great deal.

Cheers,

Jaromir