Migrate RabbitMQ Server to RabbiitMQ Kubernetes

[AndresCo]

HI,

im experimenting with a migration of an old instance of RabbitMQ 3.9.x to RAbbitMQ 3.11 or 3.12 running on Kubernetes vit RabbitMQ Cluster Operator. The version is OSS

I´d like to reduce downtime at much as i can. 

is it the use of Federated Queues a good idea? 

In this case i can use blue-green deployment, but i will need to change the connection settiins of  ALL the consumers to the new cluster, and after that, all producers. There is a load balancer in front, but a dont think i can route traffic from consumers/producers.

I have been also experimenting with adding a kubernetes pod to the external rabbitmq cluster, but in/out traffic in kubernetes can be tricky. At least, im not able to make it work.

any suggetsion or experience?

[RR]

Believe I may be in the same spot as you are, not positive, if not let me know and I'll repost. I am migrating from a simple container of RabbitMQ 3.9 that was working just fine in a minikube setting. However, have since migrated to a full on Kubernetes install - it no longer works. My understanding is that I would need to use the RabbitMQ Cluster Operator. 

Am looking for the best practice approach to implementing RabbitMQ within my new infrastructure. Any guidance is much appreciated, thanks

[Michal Kuratczyk]

You don't have to use the Operator, but it's the recommended way.

We really can't help based on "it no longer works".

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1267cecc-31d6-4d8f-856b-8472c2bec211n%40googlegroups.com.

--

Michał

RabbitMQ team

[AndresCo]

Hi,

As Michal mentioned, i recommend using the Operator.

What im trying to reach is a node communication between external RabbiitMQ and a pod in K8s.

From the POD, trying to join an external MQ i can see:

"

attempted to contact: ['rab...@disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local']"

while im trying to connect to the external hosts.

This value: .rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local is taken from some value in the operator.

More Info:

 rabbitmqctl join_cluster rabbit@disrmqs11
Clustering node rab...@rabbitmq-sandbox2-server-0.rabbitmq-sandbox2-nodes.rabbitmq-system with rabbit@disrmqs11
Error: unable to perform an operation on node 'rab...@disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rab...@disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rab...@disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local']

rab...@disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local:
  * unable to connect to epmd (port 30369) on disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local: nxdomain (non-existing domain)


Current node details:
 * node name: 'rabbitmqcli...@rabbitmq-sandbox2-server-0.rabbitmq-sandbox2-nodes.rabbitmq-system'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Bq7acsmwkVTUJU580vSnmw==

[Michal Kuratczyk]

RabbitMQ requires node names to be valid DNS names: https://www.rabbitmq.com/clustering.html#hostname-resolution-requirement

Therefore, you'd need to make it so that disrmqs11.rabbitmq-sandbox2-nodes.rabbitmq-system.svc.cluster.local resolves to a valid

name both inside the Kubernetes cluster (that's almost certainly already true) and outside of it. You will also need a few ports

to be open. It may get tricky to set up a cluster with nodes both inside and outside Kubernetes, if that's what you are trying to do.

A blue-green upgrade/migration is a much safer option.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/a6edfaca-39a5-4aba-8f2c-20168a1523a7n%40googlegroups.com.

--

Michał

RabbitMQ team

[Andres Colodrero]

Hi and thanks for the answer.

Regarding the first approach, im opening ports and services (nodePort) for the first stage of the migration. But as you mention, maybe is not a standard way to migrate a cluster and i can get into other problems.

Blue-green deployment means that i need to change configuration for all the consumers first (we have >100 consumers) and redeploy those apps.. and them the producers.

I wonder if in this case (2 clusters) can set up federated queues, Stop Cluster A, Change DNS and Start Cluster B. In that case, producers and consumers just get a short timeout (most clients are .Net, i guess they can handle the changes).

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/t_1Fq5Q4DLo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0sXoB%2B00G%3Ds5fSV0txn7m6huCGxAyZx%3DFOgO64sSms%3Dmw%40mail.gmail.com.

[Michal Kuratczyk]

There are perfectly valid variations of the process based on the constraints. If you can afford a bit of downtime, you can certainly simplify the blue-green process.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAB8GGzDgHXbRcodhx1%3DGPY2Rsrv-buO6HDXmZmPVyiFJJMk%3Dow%40mail.gmail.com.

--

Michał

RabbitMQ team

[AndresCo]

Hi Michal,

I have been testing federation:

1. Cluster A. A windows cluster with default ports (AMQP 5672)

2. Cluster B. New K8s cluster. I created a nodeport for out communication to k8s 30672. BY now it is a single node.

Im migrating to Cluster B, so this is my downstream server.

For upstream server, cluster B i check communication

telnet server1.clusterb 30672 -> works

So as you see, i cant use 5672 on Kubernetes, but i dont want to change the default port on our current environment.

can i add a extra port for Federation? in this case both nodes can connect via 30672

or should i set a load balancer in from to map port 5672->30672?

[Michal Kuratczyk]

It's more of a Kubernetes/networking question than a RabbitMQ question.

Personally I would rather not use NodePorts but indeed a load balancer or some other ingress.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3ffd560c-6359-4b50-9541-c3dc9d83dbf9n%40googlegroups.com.

--

Michał

RabbitMQ team

[Andres Colodrero]

Well,im just running tests and for production i wont use nodeport.

But looks like i will need a load balancer in front for testing..

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3ffd560c-6359-4b50-9541-c3dc9d83dbf9n%40googlegroups.com.

--

Michał

RabbitMQ team

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/t_1Fq5Q4DLo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0u%3DAGGSt7bmsd2FH_3Tzt7ZUUpPZpi9%3DHWZXZXtURLmfw%40mail.gmail.com.