What does Erlang do in RabbitMQ?

[Oğuzhan]

What does Erlang do in RabbitMQ? Is it cluster manager? Who decides which slave is going to be master. 

[Michael Klishin]

Erlang is the primary programming language used in RabbitMQ server and core plugins.

There are no master or follower nodes in Erlang or RabbitMQ. Queues (classic mirrored and quorum) have master replicas.

Each queue type decides how leader election and promotion work. The docs should provide an overview. In case of quorum queues the mechanism is dictated by the well documented Raft consensus protocol.

On Tue, 11 Feb 2020 at 20:45, Oğuzhan <kayao...@windowslive.com> wrote:

What does Erlang do in RabbitMQ? Is it cluster manager? Who decides which slave is going to be master. 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b29d2ca0-eb8d-4ff3-92f2-b4f501074b6e%40googlegroups.com.

--

Staff Software Engineer, Pivotal/RabbitMQ

[Oğuzhan]

You know wrong bro in this document it says master and slave nodes https://www.rabbitmq.com/ha.html 

11 Şubat 2020 Salı 20:48:50 UTC+3 tarihinde Michael Klishin yazdı:

Erlang is the primary programming language used in RabbitMQ server and core plugins.

There are no master or follower nodes in Erlang or RabbitMQ. Queues (classic mirrored and quorum) have master replicas.

Each queue type decides how leader election and promotion work. The docs should provide an overview. In case of quorum queues the mechanism is dictated by the well documented Raft consensus protocol.

On Tue, 11 Feb 2020 at 20:45, Oğuzhan <kayao...@windowslive.com> wrote:

What does Erlang do in RabbitMQ? Is it cluster manager? Who decides which slave is going to be master. 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b29d2ca0-eb8d-4ff3-92f2-b4f501074b6e%40googlegroups.com.

[Gavin M. Roy]

On Tue, Feb 11, 2020 at 1:11 PM Oğuzhan <kayao...@windowslive.com> wrote:

You know wrong bro in this document it says master and slave nodes https://www.rabbitmq.com/ha.html 

Uh, you're wrong. If you don't understand that's ok, but your response is uneducated and rude.

The RabbitMQ cluster automatically manages HA queues across the nodes that are clustered. It uses Erlang's VM and clustering to communicate between nodes. Queues are processes in the Erlang VM and there will be a master node for a queue, but that will differ from queue to queue. There are no master nodes in a cluster, they are all peers.

Michael and the rest of the RabbitMQ time are very patient with the support requests that are made on this list. I don't have to be. Don't be an ass. If you were not intentionally an ass, then I apologize, but calling out someone who is helping you as wrong is rude. He was not wrong.

Gavin

[Oğuzhan]

ok I am just here to learn something and do my job. I am just trying to understand. from rabbtimq document it says master queues and mirrored queues right? in ha-mode also there is a master queue and mirror queues. when I select ha-mode exactly and give ha-params 2. it takes replicas of master queue and mirror queue I hope you are reading. I am talking about this master and mirrored queue. But when master queue dies, mirrrored queue becomes master queue right? You say I think Erlang's VM decides mirrored qeueu to becomes master queue right? Because this document says this. I know all nodes are peers. Because they must be sync. In kafka zookeeper does the job when node is down the other node becomes master queue.

11 Şubat 2020 Salı 21:17:33 UTC+3 tarihinde Gavin M. Roy yazdı:

[Oğuzhan]

You mean when I start first consumer to cluster, it creates a.k.a the master queue. if I made a policy for this ha-mode exactly ha-params 1 ha-sync-mode automatic then it creates a mirror queue right? But I couldnt understand about Erlang's VM. Is there any document about this? 

11 Şubat 2020 Salı 21:17:33 UTC+3 tarihinde Gavin M. Roy yazdı:

[Wesley Peng]

If you have setup HA mode, erlang cluster will do the queue leader choices for you. The backend theory is Raft distribution algorithm.

https://raft.github.io/

Regards 

12 февраля 2020 г., 02:55:28,от "Oğuzhan" <kayao...@windowslive.com>:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5258764f-e49c-47c2-b177-c1950633abc1%40googlegroups.com.

[Michael Klishin]

> You know wrong bro in this document it says master and slave nodes https://www.rabbitmq.com/ha.html

I'm a RabbitMQ core team member (have been for close to 7 years and contributor for about 10 years), so I have some idea about how things work internally.

There are no leader and follower nodes in Erlang. All nodes are equal as far as Erlang is concerned (we will ignore C and hidden nodes for

the sake of this discussion since RabbitMQ cluster members are neither).

[1] explains what is replicated in a RabbitMQ cluster. There are no special nodes in RabbitMQ. Again, all nodes are equal.

What is not equal is queue replicas. They do have a leader and 0 or more followers. Classic mirrored queues use a homegrown

leader election algorithm while quorum queues [3] rely on Raft's leader election part.

[2] explains that any client can connect to any node. That should give you some idea that indeed, in most ways nodes are not

special.

[4], which I happened to be a contributor to, clearly explains that by default a queue resides on a single node. Technically there is a master

replica but there are no followers, so again, all nodes are semantically identical for any practical purpose unless replication comes into play.

When a cluster is formed [5] or restarted [6], nodes temporarily become "special" because at least one has to serve as a seed or recovery

point. Again, by most definitions and given the transient nature of this "designation" this node is not a "leader", just the very first node to have

started.

HTTP API and management plugin used to have a dedicated cluster node. That's no longer the case as of 3.6.7 [7], every node collects

and serves its own metrics unless it has to aggregate. When it does reach out to peers for their data to aggregate, there are no leaders

or followers.

I also should point out that your arrogant attitude is not appreciated on this list. If you are so incredibly smart and know better then why are you on this list asking

questions? Just go ahead and read the source. Continue behaving like this and you will end up banned in no time.

1. https://www.rabbitmq.com/clustering.html#overview-what-is-replicated

2. https://www.rabbitmq.com/clustering.html#clustering-and-clients

3. https://www.rabbitmq.com/quorum-queues.html

4. https://www.rabbitmq.com/ha.html#what-is-mirroring

5. https://www.rabbitmq.com/cluster-formation.html

6. https://www.rabbitmq.com/clustering.html#restarting

7. https://www.rabbitmq.com/management.html#clustering

On Tue, Feb 11, 2020 at 9:31 PM Oğuzhan <kayao...@windowslive.com> wrote:

Are you kidding to me? Please tell me hwo did you understand I wrote this rudely? I writing you know wrong this document says it and I am sharing you should be calm and dont be ass to me I am calm and cool. But please explain to me how did you assumed I am rude? You are rude beqacuase you are thinking in that way. Please be kind to everybody before reading one command. Maybe my english is bad. You are just not thinking I am trying to do tons of works and somebody pissed me off. Please dont be ass. Think twice.

11 Şubat 2020 Salı 21:17:33 UTC+3 tarihinde Gavin M. Roy yazdı:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/77d6cbfa-087f-48c3-bc44-ec3811c50255%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Oğuzhan]

Thank you both for your kind answers. One last question I have. I made a rabbitmq cluster there are 3 nodes. I can consume from external machines from every node but I want to consume from cluster. Because for example when I consume from x node when x node is down I cant consume. How can I get cluster IP or Erlang IP or something else when node is down I can get data from another node. I need to write IP to pika.BlockingConnection. what should I need to write? Thanks for your time.

12 Şubat 2020 Çarşamba 03:44:06 UTC+3 tarihinde Michael Klishin yazdı:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/77d6cbfa-087f-48c3-bc44-ec3811c50255%40googlegroups.com.

[Wesley Peng]

You can setup a TCP proxy in front of the cluster, such as LVS or haproxy or commercial LB devices.

Regards 

12 февраля 2020 г., 15:05:49,от "Oğuzhan" <kayao...@windowslive.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d92798dc-2ba4-4ee2-84fa-c4872e092e5e%40googlegroups.com.

[Oğuzhan]

What do you suggest for this issue? 

12 Şubat 2020 Çarşamba 10:17:28 UTC+3 tarihinde Wesley Peng yazdı:

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d92798dc-2ba4-4ee2-84fa-c4872e092e5e%40googlegroups.com.

[Michael Klishin]

There is no such as thing as a "cluster IP" or "Erlang IP". An IP is an address of a single host and if that host is not available

or the RabbitMQ node on it is not running, inbound TCP connection will fail.

With some client libraries you can provide a list of hosts and it will try to connect to them in sequence before one of the endpoints succeeds [1].

Or, as Wesley Peng points out, connect to a proxy or load balancer (which itself will have to have redundancy at some point) and it will

connect to an upstream that is up.

RabbitMQ does not care what load balancer you use as long as you are aware of its potential side effects, negative and positive [2][3][4].

HAproxy is one popular open source option.

1. https://www.rabbitmq.com/api-guide.html#address-array

2. https://www.rabbitmq.com/networking.html#intermediaries

3. https://www.rabbitmq.com/heartbeats.html#tcp-proxies

4. https://www.rabbitmq.com/ha.html#master-migration-data-locality

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d92798dc-2ba4-4ee2-84fa-c4872e092e5e%40googlegroups.com.