Re: Unable to get hazelcast nodes to work in cluster :(

[Enes Akar]

Firewalls (mostly in windows) can cause problems.

Can you send the logs?

On Thu, Aug 30, 2012 at 5:11 AM, <atin...@gmail.com> wrote:

I have 2 java app's for which I am trying to communicate/share some information using hazelcast. But for some reason I am not sure why hazelcast started from these 2 instances are not able to discover each other

This is my config
<hz:hazelcast id="hazelCastInstance">
        <hz:config>
            <hz:properties>
                <hz:property name="hazelcast.merge.first.run.delay.seconds">5</hz:property>
                <hz:property name="hazelcast.merge.next.run.delay.seconds">5</hz:property>
            </hz:properties>
            <hz:network port="5701" port-auto-increment="true">
                <hz:join>
                    <hz:multicast enabled="true"/>
                </hz:join>
            </hz:network>

            <hz:map name="configuration">
                <hz:map-store enabled="true" implementation="mongomapstore"
                    write-delay-seconds="0">
                </hz:map-store>
            </hz:map>

            <hz:map name="routes">
                <hz:map-store enabled="true" implementation="mongomapstore"
                    write-delay-seconds="0">
                </hz:map-store>
            </hz:map>

        </hz:config>
    </hz:hazelcast>

At this point I am totally stuck unable to figure out what I might be doing wrong, any help will be awesome.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/FzZEQs2Wj6EJ.
To post to this group, send email to haze...@googlegroups.com.
To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.

[Enes Akar]

You should enable log4j logging:

System.setProperty("hazelcast.logging.type", "log4j");

On Thu, Aug 30, 2012 at 9:02 PM, <atin...@gmail.com> wrote:

I am running this on mac with firewall turned off. I am not really sure how to get the logs though. I tried with the log4j.props file as

log4j.rootLogger=DEBUG, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender

log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
com.hazelcast= DEBUG 

but did not saw any logs for hazelcast. 

Apologies if I am missing something really basic here. 


Also I am using the hazelcast instance that is created here to make all the calls rather than using the static Hazelcast. methods.

<bean id="hazelCastHelper" class="com.b2b.hazelcast.HazelCastHelper">
		<property name="configurationMapInstance" ref="configurationMap"></property>
		<property name="routesMapInstance" ref="routesMap"></property>
		<property name="deployedRouteTopic" ref="deployedRoutesTopic"></property>
		<property name="activatedRouteTopic" ref="activatedRoutesTopic"></property>
		<constructor-arg ref="hazelCastInstance"></constructor-arg>
</bean>

is how I get my hazelcastInstance.

Appreciate the help. 

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/r-DiaG1DyT0J.

[Mehmet Dogan]

What local socket IP address is selected by Hazelcast, an IPv4 address or IPv6 address?

Can you test by disabling IPv6, by setting -Djava.net.preferIPv4Stack=true ?

@mmdogan

On Fri, Aug 31, 2012 at 7:51 PM, oket <ok...@shaw.ca> wrote:

I have had a similar experience on a Centos 5 VM with Virtualbox on a Win 7 host.
Though it appeared that multicast was functional, ifconfig showed it as up on the interfaces and no errors occurred, the cluster would not connect its members.

I put together a program to test multicast and determined that it was not connecting.
I switched discovery over to TCP/IP and got my cluster going.

I have not determined what the networking problem was, and have put that investigation on the shelf for a while, while I continue draining the swamp in other areas.

The validate code is pretty simple, and can be used as:

            validate = new ValidateMulticast("224.2.2.3");
            canMulticast = validate.isMulticastWorking();

With the support Class of:

package com.yourNameInThisSpace.net;

import java.net.DatagramPacket;
import java.net.InetAddress;
import java.net.MulticastSocket;
import java.net.UnknownHostException;

/**
 * A Class to validate that Multicast works on your system.
 */
public class ValidateMulticast {

    /**
     * the address we are validating.
     */
    private InetAddress multicastAddress = null;
    /**
     * true if we have tested, and multicast is working.
     */
    private boolean multicastWorking = false;
    /**
     * a Thread to read the multicast data on.
     */
    private Thread readThread = null;
    /**
     * true if we have already run a validation on this address.
     */
    private boolean tested = false;

    /**
     * construct a null Multicast validator.
     */
    public ValidateMulticast() {
    }

    /**
     * construct an initialized Multicast validator.
     *
     * @param address
     */
    public ValidateMulticast(String address) throws UnknownHostException {
        setMulticastAddress(address);
    }

    /**
     * return the address that we are supposed to multicast on.
     *
     * @return
     */
    public InetAddress getMulticastAddress() {
        return multicastAddress;
    }

    /**
     * return true if Multicast works on the supplied address.
     *
     * @return
     */
    public boolean isMulticastWorking() {
        if (!tested) {
            test();
        }
        return multicastWorking;
    }

    /**
     * specify the Multicast address to use.
     *
     * @param address
     * @throws UnknownHostException
     */
    public void setMulticastAddress(String address) throws UnknownHostException {
        multicastAddress = InetAddress.getByName(address);
        tested = false;
    }

    /**
     * test multicast. We create a reader thread and a writer, starting the
     * reader first. If the reader gets what we are writing, then we are good.
     * If it does not, then multicast is not working for us.
     */
    private void test() {
        tested = false;
        multicastWorking = false;
        readThread = new Thread(new Runnable() {
            public void run() {
                try {
                    InetAddress group = InetAddress.getByName("224.2.2.3");
                    MulticastSocket s = new MulticastSocket(6789);
                    s.joinGroup(group);
                    byte[] buf = new byte[1000];
                    DatagramPacket recv = new DatagramPacket(buf, buf.length);
                    s.receive(recv);
                    tested = true;
                    multicastWorking = true;
                    s.leaveGroup(group);
                } catch (Exception e) {
                    tested = true;
                    multicastWorking = false;
                }
            }
        });
        readThread.setDaemon(true);
        readThread.start();
        try {
            InetAddress group = InetAddress.getByName("224.2.2.3");
            MulticastSocket s = new MulticastSocket(6789);
            s.joinGroup(group);
            String msg = "Hello wind";
            DatagramPacket hi = new DatagramPacket(msg.getBytes(), msg.length(), group, 6789);
            for (int i = 0; !tested && i < 5; i++) {
                synchronized (this) {
                    try {
                        this.wait(100);
                    } catch (Exception e) {
//                        e.printStackTrace();
                    }
                }
                System.out.println("Send: " + i);
                s.send(hi);
            }
        } catch (Exception e) {
//            e.printStackTrace();

        }
    }
}

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/G4FdXGAim3sJ.

[oket]

On Friday, 31 August 2012 10:58:33 UTC-6, Mehmet Dogan wrote:

What local socket IP address is selected by Hazelcast, an IPv4 address or IPv6 address?

Can you test by disabling IPv6, by setting -Djava.net.preferIPv4Stack=true ?

@mmdogan

 
The definition made no change in the multicast validation test, which does work on the host windows system.

Hazelcast is selecting IPv4 adresses.


It would appear that Hazelcast is also trying IPV4, from the following logging, but interestingly enough it is often returning 127.0.0.1 as a host, rather than the 192.168.56.51 it finds at other times, which certainly does confuse its determination of who is Master. All of the Hazelcast instances are running in the same VM, with the same cluster:

        h1 = Hazelcast.newHazelcastInstance(config1);
        h1.addInstanceListener(this);
        HazelcastInstance h2 = Hazelcast.newHazelcastInstance(config1);


INFO: Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.56.51]
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.AddressPicker
INFO: Prefer IPv4 stack is true.
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.AddressPicker
INFO: Picked Address[192.168.56.51]:5704, using socket ServerSocket[addr=/0.0.0.0,localport=5704], bind any local is true
Aug 31, 2012 11:30:46 AM com.hazelcast.system
INFO: [192.168.56.51]:5704 [devEnv] Hazelcast Community Edition 2.2 (20120723) starting at Address[192.168.56.51]:5704
Aug 31, 2012 11:30:46 AM com.hazelcast.system
INFO: [192.168.56.51]:5704 [devEnv] Copyright (C) 2008-2012 Hazelcast.com
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.PartitionManager
INFO: [192.168.56.51]:5703 [dev] Initializing cluster partition table first arrangement...
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.LifecycleServiceImpl
INFO: [192.168.56.51]:5704 [devEnv] Address[192.168.56.51]:5704 is STARTING
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.56.51]:5704 [devEnv] Connecting to possible member: Address[192.168.56.51]:5701
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.56.51]:5704 [devEnv] Connecting to possible member: Address[192.168.56.51]:5702
Aug 31, 2012 11:30:46 AM com.hazelcast.impl.TcpIpJoiner
INFO: [192.168.56.51]:5704 [devEnv] Connecting to possible member: Address[192.168.56.51]:5703
Aug 31, 2012 11:30:46 AM com.hazelcast.nio.SocketAcceptor
INFO: [192.168.56.51]:5703 [dev] 5703 is accepting socket connection from /192.168.56.51:44074
Aug 31, 2012 11:30:46 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5703 [dev] 5703 accepted socket connection from /192.168.56.51:44074
Aug 31, 2012 11:30:46 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5704 [devEnv] 44074 accepted socket connection from /192.168.56.51:5703
Aug 31, 2012 11:30:46 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5704 [devEnv] 44367 accepted socket connection from /192.168.56.51:5701
Aug 31, 2012 11:30:46 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5704 [devEnv] 34324 accepted socket connection from /192.168.56.51:5702
Aug 31, 2012 11:30:47 AM com.hazelcast.impl.Node
INFO: [192.168.56.51]:5704 [devEnv] ** setting master address to Address[127.0.0.1]:5701
Aug 31, 2012 11:30:47 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5704 [devEnv] 60314 accepted socket connection from /127.0.0.1:5701
Aug 31, 2012 11:30:47 AM com.hazelcast.impl.Node
INFO: [192.168.56.51]:5704 [devEnv] ** setting master address to Address[192.168.56.51]:5703
Aug 31, 2012 11:30:47 AM com.hazelcast.cluster.ClusterManager
INFO: [192.168.56.51]:5704 [devEnv]

Members [3] {
    Member [127.0.0.1]:5701
    Member [127.0.0.1]:5702
    Member [192.168.56.51]:5704 this
}

Aug 31, 2012 11:30:48 AM com.hazelcast.nio.ConnectionManager
INFO: [192.168.56.51]:5704 [devEnv] 49400 accepted socket connection from /127.0.0.1:5702
Aug 31, 2012 11:30:48 AM com.hazelcast.impl.PartitionManager
WARNING: [192.168.56.51]:5704 [devEnv] Received a PartitionRuntimeState, but its sender doesn't seem master! => Sender: Address[192.168.56.51]:5701, Master: Address[192.168.56.51]:5703! (Ignore if master node has changed recently.)
Aug 31, 2012 11:30:49 AM com.hazelcast.impl.PartitionManager
WARNING: [192.168.56.51]:5704 [devEnv] Received a PartitionRuntimeState, but its sender doesn't seem master! => Sender: Address[192.168.56.51]:5701, Master: Address[192.168.56.51]:5703! (Ignore if master node has changed recently.)
Aug 31, 2012 11:30:50 AM com.hazelcast.impl.PartitionManager

[Mehmet Dogan]

Do you use the same configuration for all instances? If yes, Hazelcast should not pick/bind 127.0.0.1 address (according to logs; TCP-IP config addresses: [192.168.56.51]).

@mmdogan

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hazelcast/-/Yn-ygIGW3FAJ.

[Michael....@online.de]

Hello,

I have the same issue.
I'm using hazelcast 3.0.2, and the "customer" GettingStarted.java code.

It's working fine with Fedora 14 Linux (IP v4) and Windows 7.
A node running Fedora 19 (supports IPv6) can not connect. I tried OpenJDK as well as the lastest Oracle Java (1.7.0_40).

These are my observations:
- the program outputs "INFO: Prefer IPv4 stack is true" as the first line. Despite, I see ServerSocket[addr=/0.0.0.0.0.0.0.0,localport=5701] in the output.
- I add the -DpreferIPv4Stack JVM option, thast line changes to ServerSocket[addr=/0.0.0.0,localport=5701], and netstat also shows the LISTEN on the IPv4 address
- in both cases, the multicast test listed above returns "false" (it returns "true" on the other Linux machine)

So far I have not yet been able to solve this, and I would not like to disable IPv6 completely on that machine.

(The Multicast test ping to 224.0.0.1 works on both machines, and receives answers from both machines, so I assume the network is set up correctly).

Any suggestions?

Best regards,
    Michael