About alternatives and documentation

[Gerard W]

Hi,

We are in the process of developing an application running on multiple JVMs which need to communicate to each other. For the specific issue of messaging between JVMs we have considered Chronicle Engine as one of three alternatives (being the others JGroups and Akka).

My first question is: is it right to consider Chronicle Engine as an alternative to JGroups and Akka?

About the documentation, I could not find the following:

• Tutorials or examples as a first approach to the tool (how to install, configure and use)
  
• Introductory documentation with an overview describing the building blocks and how they work together
  
• Reference documentation
  

Specifically, I would like to know:

• How to join to a cluster, identify and keep track of the other members in the cluster, and receive notifications when a new member joins, or an existing member leaves or crashes
  
• How to send messages to all members or single members and receive messages from members in the cluster
  

I would appreciate you guiding me in the right direction.

Thanks.

[Diego Marcelo]

Hello.
I have the same question, I need to use a framework to send and receive messages between JVMs in a cluster.
I find that Chronicle has many features to fulfill my requeriments, but the documentation is poor, and I cannot find a way to run it.
Can you help me too?
Regards

[Rob Austin]

is the Custer a cluster of JVM’s on the same machine or between machine/servers.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Lawrey]

Hello,

  While we can add and remove nodes from a cluster we don't emphasis this functionality. 

We assume your system will be designed to maintain a consistently low latency by having plenty of head room to support your worst bursts of activity.

Aay you have an average rate of 10,000 events per second over a day, so you design your system to handle 100,000 events per second, with short burst of say 1000 per milli-second. This way you can have good latencies 99% or 99.9% of the time.

As a result, you might have 90% - 99% head room compared to your typical loads so adding another node on an ad hoc basis isn't generally needed. It something you would do on a planned basis if you start to see the utilisation increase. Similarly, removing a node will reduce head room but you would generally only do this on a failure (which you will want to restore quickly), or a planned basis (e.g. not very often)

For redundancy you might want 2 or 3 servers in one region in active-active configuration e.g. Europe, Ideally each server could handle your entire load in a worst case scenario. When they are all running you could handle about 30,000 events per second sustained, with peaks of 300K event per second when all nodes are available.

You really need to have very high loads or expensive event handling (such as some complex event processing) to be considering needing multiple server for anything except redundancy.  Given you can buy 22 core machines at a reasonable price these days, that's a lot of process power.

Note: we have a client with a cluster of 6 servers, which takes peaks of 30 million events per second and has over 100 TB of data stored which can the accessed directly by any JVM (clustering is not required to access all the data). The events come via non-reliable UDP packets so if the system stops for even a short time they start losing data which is why they also need to be able to process messages at a very consistent rate.

I know these design assumptions are not true of many system, but hopefully this will give an idea of where we are coming from.

Regards,

   Peter.

On 21 September 2016 at 06:00, Rob Austin <rob.a...@higherfrequencytrading.com> wrote:

is the Custer a cluster of JVM’s on the same machine or between machine/servers.

On 21 Sep 2016, at 13:59, Diego Marcelo <dmse...@gmail.com> wrote:

Hello.
I have the same question, I need to use a framework to send and receive messages between JVMs in a cluster.
I find that Chronicle has many features to fulfill my requeriments, but the documentation is poor, and I cannot find a way to run it.
Can you help me too?
Regards

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Gerard W]

Rob:

By "cluster" we mean a group of JVMs that can be on the same machine and/or on different machines.

Peter:

Thanks for your detailed response. Anyway, I will appreciate your answers to my questions in the first post:

• Is it right to consider Chronicle Engine as an alternative to JGroups and Akka?
  
• Can you provide tutorials/examples/documents on how to properly use Chronicle Engine?

Regards,

Gerard

El miércoles, 21 de septiembre de 2016, 11:32:13 (UTC-3), Peter Lawrey escribió:

Hello,

  While we can add and remove nodes from a cluster we don't emphasis this functionality. 

We assume your system will be designed to maintain a consistently low latency by having plenty of head room to support your worst bursts of activity.

Aay you have an average rate of 10,000 events per second over a day, so you design your system to handle 100,000 events per second, with short burst of say 1000 per milli-second. This way you can have good latencies 99% or 99.9% of the time.

As a result, you might have 90% - 99% head room compared to your typical loads so adding another node on an ad hoc basis isn't generally needed. It something you would do on a planned basis if you start to see the utilisation increase. Similarly, removing a node will reduce head room but you would generally only do this on a failure (which you will want to restore quickly), or a planned basis (e.g. not very often)

For redundancy you might want 2 or 3 servers in one region in active-active configuration e.g. Europe, Ideally each server could handle your entire load in a worst case scenario. When they are all running you could handle about 30,000 events per second sustained, with peaks of 300K event per second when all nodes are available.

You really need to have very high loads or expensive event handling (such as some complex event processing) to be considering needing multiple server for anything except redundancy.  Given you can buy 22 core machines at a reasonable price these days, that's a lot of process power.

Note: we have a client with a cluster of 6 servers, which takes peaks of 30 million events per second and has over 100 TB of data stored which can the accessed directly by any JVM (clustering is not required to access all the data). The events come via non-reliable UDP packets so if the system stops for even a short time they start losing data which is why they also need to be able to process messages at a very consistent rate.

I know these design assumptions are not true of many system, but hopefully this will give an idea of where we are coming from.

Regards,

   Peter.

On 21 September 2016 at 06:00, Rob Austin <rob.a...@higherfrequencytrading.com> wrote:

is the Custer a cluster of JVM’s on the same machine or between machine/servers.

On 21 Sep 2016, at 13:59, Diego Marcelo <dmse...@gmail.com> wrote:

Hello.
I have the same question, I need to use a framework to send and receive messages between JVMs in a cluster.
I find that Chronicle has many features to fulfill my requeriments, but the documentation is poor, and I cannot find a way to run it.
Can you help me too?
Regards

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.

[Peter Lawrey]

Hello Gerand,

   I wouldn't consider JGroups or Akka to be replacement for one another. They are very different solutions.  Similarly Chronicle Engine is more of an API and Data Gateway which supports clustering queues and key-value stores.

We have our documentation here https://github.com/OpenHFT/Chronicle-Engine, there are many examples in our unit tests. 

Without more information, I would reconsider the assumption you need more than one machine. While slower solutions might need many machines we have found only most demanding applications need more than one.

It would be helpful if you could say a) why you believe multiple machines are essential and how they will be used, b) what performance requirement you have in terms of throughput and latency and c) what documentation you would like to see.

Regards,

   Peter.

To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicle+unsubscribe@googlegroups.com.

[Gerard W]

Hello Peter,

As you rightly say, JGroups and Akka are quite different, but still we have to consider them to evaluate their pub-sub abilities, and we would  like to do the same with Chronicle.

As I can see now, we should not consider Chronicle Engine alone, but a conjunction of Chronicle Queue + Cchronicle Engine, am I right on this?

a) why you believe multiple machines are essential and how they will be used

a) The application we are developing will serve hundreds of thousands of users, and we need to distribute them on multiple JVMs to achieve load balancing. We will run as many JVMs per physical server as optimal. We want to communicate event messages between JVMs that could be or not on the same physical server. These messages will be classified into different topics. Every JVM must be able to subscribe and publish to multiple topics, and each time a message is received from another JVM, a callback on the receiving JVM must be invoked to give the appropriate treatment.

b) what performance requirement you have in terms of throughput and latency and 

We are evaluating the available APIs to choose the one with the lowest latency and response time. Our worst case scenario estimates the circulation of 10,000 messages per second, weighing at most 1MB each.

---

c) what documentation you would like to see.

We would like to see documentation on the appropriate configuration and API, to achieve the functionality as expressed in a). 

A tutorial will be highly appreciated.

Regards

Gerard