Vertx hazelcast implementation

[Krishnaprasad Narayanan]

Hallo all,

I have two servers that communicates messages over the TCP / IP interface using the Hazelcast cluster. I have followed the example posted in this page to set up the Hazelcast cluster.

Each of the server contains one verticle in which one of them is a sender while the other is a receiver. When the verticles are executed with in the eclipse IDE using the -cluster and -cluster-host option, I am able to see the messages in the receiver verticle and the corresponding acknowledgement on the sender verticle.

When I execute the same from the command line, I am only able to see the messages related to the successful deployment of verticles. I am not able to see the messages on the receiver and sender side.

Can anybody help me in resolving the problem?

Thanks,

Krishnaprasad

[Arnold Schrijver]

If you enable logging on hazelcast, you should be able to see whether it finds all nodes on the cluster. Depending on what you find, you may have to do some tweaking in the cluster config.

[Krishnaprasad Narayanan]

May I know how should I enable logging on hazelcast cluster?

[Arnold Schrijver]

It depends on what logger you use. Vert.x uses the standard JUL logger by default, but you can use other loggers with it. I personally use logback, in which case you add something like this in your logback.xml:

    <logger name="com.hazelcast" level="INFO" />

Or even DEBUG level...

[Krishnaprasad Narayanan]

I added logback.xml in the src/main/resources specifying DEBUG as the level for com.hazelcast. I still do not see any messages displayed on the stdout / conosole when I run the program. Do I have to change any property file or add flags while running vertx to see the DEBUG messages on the console / command line? By messages I mean the DEBUG information which is displayed inside the Eclipse IDE are not shown when the application is run on the command line.

[Arnold Schrijver]

Yes, it once again depends on how you are running Vertx, but you have to pass a system property. I run my verticles like this from a shadow jar:

java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -Dhazelcast.config=./cluster.xml -Dlogback.configurationFile=./logback.xml -jar MyShadowJar.jar -conf MyConf.json -cluster

There is more mention on this in the JavaDoc (http://vertx.io/docs/apidocs/io/vertx/core/logging/Logger.html) and in older posts to this forum.

[Arnold Schrijver]

Oh, BTW, the above will get you log output from hazelcast. But it doesn't follow the logback format you have configured unless you add a property to your cluster config properties section:

<property name="hazelcast.logging.type">slf4j</property>

[Krishnaprasad Narayanan]

I have generated a FAT jar and I am following the command which you had mentioned. The only difference is I dont have a json file due to which I am not specifying the -conf flag. I am neither any messages nor any exceptions in the console.

Is the json file mandatory to run it from the command line?

[Anil]

No its not mandatory. its customized properties used in your project.

[Arnold Schrijver]

No the conf is not required. This is just to pass config to your app. Should be working now. Do you see other log messages besides hazelcast ones?
Do you have the correct sequence of args on the java command? The system properties must come before the -jar arg

You could add an appender that logs to file. Something like this:

    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>build/logs/my-log.log</file>
        <append>false</append>
        <encoder>
            <pattern>%date %-5level %class{64}:%line - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="FILE" />
        <appender-ref ref="STDOUT" />
    </root>

[Krishnaprasad Narayanan]

Finally, it has started to work from the command line. I am noticing all the logs which are displayed in the Eclipse IDE on the command line.

However, I am noticing a strange behavior. I am using gradle to build the application. I have generated two jars in which one of them is a FAT jar while the other is a shadow jar.

When I apply java -jar <shadow jar> -cluster -cluster-host <local IP address of the server>, I notice that the public IP address of the server is used in the Hazelcast clustering. If I use the command java -jar <FAT jar> -cluster -cluster-host <local IP address>, I notice that the public IP address is getting translated to a local IP address which is binded to one of the network interfaces in the server.

Is this the ideal behavior? I also tried setting the public IP address with in the program using the java API (setClusterAddress) of ClusterManager but that does not seem to work.

May I know how should I force the program to pick up the public IP address of the server?

[Arnold Schrijver]

Glad that that part works! I don't know about your current problem. It might be best to post a new article for this issue and provide a bit of an example with it for clarification.

[Krishnaprasad Narayanan]

Thanks Arnold for your inputs. Those were helpful. Can somebody help me with the following questions

a) How can I specify a network interface (like eth0 / eth1) either in the default-cluster.xml or programmatically within the implementation of Hazelcast?

b) When I run a docker container with vertx  having hazelcast cluster embedded inside it, do I have to specify the -p option such as 5701 and so on....?

[Clement Escoffier]

Hi,

On 6 mai 2016, at 11:02, Krishnaprasad Narayanan <krish...@gmail.com> wrote:

Thanks Arnold for your inputs. Those were helpful. Can somebody help me with the following questions

a) How can I specify a network interface (like eth0 / eth1) either in the default-cluster.xml or programmatically within the implementation of Hazelcast?

You don’t set the network interface name but the IP assigned to this interface.

b) When I run a docker container with vertx  having hazelcast cluster embedded inside it, do I have to specify the -p option such as 5701 and so on….?

No, at least I never did it and it worked. 

Clement

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/98086b75-beb8-4967-bdb4-4841cda81ec8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Krishnaprasad Narayanan]

Hi Clement,

As I mentioned in my earlier post, when I run java -jar <FAT jar> -cluster -cluster-host < IP address of the hoster>, I noticed that this IP address is getting resolved to a local IP which is associated to another network interface and not accessible. I have also set this in the default-cluster.xml but it does not seem to work. Is there a way where I can statically assign this address in the program or outside of the source code?

Krishnaprasad

[Tim Fox]

If your fatjar is ignoring -cluster-host at the command line this probably means that instead of using Starter you've implemented your own custom startup.

Can you share you build/code? Then we can easily see if this is the case.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5614a7b3-ec14-4435-9067-cbc9e57e81dd%40googlegroups.com.

[Arnold Schrijver]

PS on b) I can confirm that the -p option with docker is not required. Should work perfectly without.

[Krishnaprasad Narayanan]

Hi Tim.

I have added the Examples.java which sends messages every 10 secs. In the receiver side, I have a similar file which listens on the same topic and sends an acknowledgement to the sender.

Further, I have attached the build.gradle file. The rest of the source code related to the Hazelcast implementation is the same as the one that is available in this github page.

Let me know if I am missing something.

Krishnaprasad 

[Krishnaprasad Narayanan]

I missed the build.gradle file in my last post. Attached it here.

[Tim Fox]

Just looking at your code...

You're programmatically creating Vertx and ignoring the arguments to the main so no wonder the -cluster-host does nothing ;)

There's no need to do any of this programmatically. A much simpler approach is to use the Starter class which will parse the command line arguments for you and will start up vert.x and the cluster manager for you.

I suggest you follow this simple example as a template to get you started:

https://github.com/vert-x3/vertx-examples/tree/master/gradle-verticles/gradle-verticle

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/63cead4f-0d97-4b80-9153-e8e67747eda4%40googlegroups.com.

[Tim Fox]

There's also a clustered event bus example in the same examples repo that does pretty much what you're trying to do. You could just copy that.

https://github.com/vert-x3/vertx-examples/tree/master/core-examples/src/main/java/io/vertx/example/core/eventbus/pointtopoint

Please also read the README:

https://github.com/vert-x3/vertx-examples/blob/master/core-examples/README.adoc

[Krishnaprasad Narayanan]

After modularizing the code following the links mentioned below, the hazelcast cluster is started on both the sender and the receiver, the two verticles (sender and receiver) are deployed, however the messages are not exchanged. I have attached the zip files which consists of the source code, build.gradle and the default-cluster.xml of both the sender and the receiver. The eventing mechanism is a publish subscribe model and the sender and the receiver are hosted on two different containers. 

The command with which I execute the program is java -jar <name of the sender FAT jar> -cluster -cluster-host <sender containers IP address> for sender while for the receiver it is java -jar <name of the receiver FAT jar> -cluster -cluster-host <receiver containers IP address>.

Can somebody help me to fix this problem?

[Krishnaprasad Narayanan]

I am using the default-cluster.xml which is placed inside the src/main/resources folder. When I extracted the contents of FAT jar, I found that there is a wrong reference to this XML file. 

May I know in which folder should I place this xml file so that it can be embedded in the FAT jar during the build?

[Clement Escoffier]

Hi,

src/main/resources is the right directory, but it should be named cluster.xml and not default-cluster.xml

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/73858f93-853f-473b-aed3-7b36a7c38fc0%40googlegroups.com.

[Tim Fox]

The file should be called cluster.xml not default-cluster.xml.

http://vertx.io/docs/vertx-hazelcast/java/#_configuring_this_cluster_manager

In a standard Maven build you can put this in src/main/resources and it will end up in the top of the jar, but this depends on how you have configured your build. I suggest copying the Maven verticle build from the examples repo.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/73858f93-853f-473b-aed3-7b36a7c38fc0%40googlegroups.com.

[Krishnaprasad Narayanan]

I renamed the cluster.xml. I extracted the FAT jar and the xml was present in the top most directory after the extraction.

I have attached the output of the sender and receiver verticles. As I mentioned in my previous post, the sender and receiver verticles are hosted on two different containers. I still observe that the sender and receiver verticles are deployed but no messages are published / exchanged. 

May I know am I missing something?

[Tim Fox]

If you post your example, along with instructions on how to run it, to GitHub I will take a look.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/dd59f7cf-6149-4c34-a422-2009da9ccc27%40googlegroups.com.

[Krishnaprasad Narayanan]

Sorry for posting the implementation here.

I have attached the zip files which contains the source code for the sender and the receiver. The FAT jar is generated using the gradle command. The sender and the receiver are hosted on two containers. Each of the attachments contains a build.gradle, cluster.xml and a src folder which has two packages: java and resources. Resources folder contains the cluster.xml file. The Java directory contains two folders: example and io. The .java files in the io directory was taken from the github: https://github.com/vert-x3/vertx-hazelcast/tree/master/src.

Building sender and receiver

-----------------------------------------

Generate the FAT jar - execute gradle --info clean assemble from the top directory. 

Generate the shadow jar - execute gradle --info shadowJar from the top directory.

The JARS are persisted in the build/libs directory. 

Run the sender on container 1: java -jar <name of the sender FAT jar> -cluster -cluster-host <IP address of the container hosting the sender (172.17.0.4)>

Run the receiver that is hosted on container 2: java -jar <name of the receiver FAT jar> -cluster -cluster-host <IP address of the container hosting the receiver (172.17.0.5)>

Let me know if I am missing anything or if i have to provide any further details.

Hi,

        <span style="color:rgb(0

[Krishnaprasad Narayanan]

One question. May I know should I use the FAT jar or the shadow jar to run the sender and receiver?

[Tim Fox]

Please push this to GitHub, then it's a lot easier for everyone to work with :)

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/acfd998b-44f8-4c4b-9986-26a04964d366%40googlegroups.com.

[Krishnaprasad Narayanan]

Please find the source code with a description in the README file in the Github here.

[Tim Fox]

Thanks, I've taken an initial look.

It strikes me the example does exactly the same thing as the clustered event bus example that we provide in the examples repo:

https://github.com/vert-x3/vertx-examples/tree/master/core-examples/src/main/java/io/vertx/example/core/eventbus/pubsub

We know this example works as it's tested on each release.

The README is here, which explains how to run it:

https://github.com/vert-x3/vertx-examples/blob/master/core-examples/README.adoc

Any reason why you don't just copy that and run it?

It seems your version also unnecessarily complicates things by adding an Examples class which has a main method - you don't need to do this, as mentioned in other posts.

Also, you seem to have copied all of the Hazelcast cluster manager classes into your project. This is also not necessary, you just need to add the dependency to vertx-hazelcast.

I suggest you start with the example we provide. Run it, and convince yourself that it works. That will help you understand how to run Vert.x verticles and clustering, and then use that example as a template for your application.

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4346375a-0cb8-4eea-ac3c-b9920e420c4c%40googlegroups.com.

[Tim Fox]

Also, I strongly recommend you read the main examples README:

https://github.com/vert-x3/vertx-examples

This tells you how the examples are structured and how to run them in the IDE - that's the easiest way to run them - right click your main class to run your sender, then right click your other main class to run your receiver.

[Krishnaprasad Narayanan]

Finally, it is working :) I removed the additional hazelcast source code from the sender and receiver packages. I merged the Hazelcast client side implementation with the publisher and subscriber verticles. 

Thanks for the help.