Kubernetes scaling HPA based off metrics in Prometheus from RabbitMQ Queues

[Chad Upton]

The goal here is very simple, but I've spent the better part of the last 2 weeks trying different solutions and options - none seem to pan out for production use yet.

Goals:

• Use prometheus-rabbitmq-exporter plugin: https://github.com/deadtrickster/prometheus_rabbitmq_exporter - check this is working
• Use prometheus service monitor to gather those metrics - check this is working
• Use helm template to define the queue the HPA pod should scale off of - syntax is close but probably incorrect.
• Use the queue metrics gathered in prometheus to scale a pod based on queue count ( NOT THE RABBIT CLUSTER ) - not working!

Details:

AWS EKS - Latest 1.13 I believe ( this changes the HPA syntax in 

Deployments are done via helm templates

Rabbitmq is clustered to the namespace and is on version 3.7.17.

Metrics are listed as such in prometheus:

Element	Value
rabbitmq_core_queue_messages{endpoint="stats",instance="10.3.89.178:15672",job="rabbitmq",namespace="us-east-1-stg",pod="rabbitmq-2",queue="queue_1_staging",service="rabbitmq",vhost="/"}	555
rabbitmq_core_queue_messages{endpoint="stats",instance="10.3.89.178:15672",job="rabbitmq",namespace="us-east-1-stg",pod="rabbitmq-2",queue="queue_2_staging",service="rabbitmq",vhost="/"}	123
rabbitmq_core_queue_messages{endpoint="stats",instance="10.3.89.178:15672",job="rabbitmq",namespace="us-east-1-stg",pod="rabbitmq-2",queue="queue_3_staging",service="rabbitmq",vhost="/"}	10

I can do a query like:

rabbitmq_core_queue_messages{queue="queue_1_staging"}

This will return the specific queue I would like to scale a pod off of.

The issues that I found so far.

• A custom rule for the prometheus operator to use to consolidate or group the queue metrics from multiple pods into a useable metric
• A kubectl get --raw request url that returns the above value in prometheus ( I must be dense because I've read way to many docs on this )
• The helm syntax to use to actually get this to work

An example of the HPA syntax I've tried to get something like this to work.

#########################################################

# HPA

#########################################################

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

  name: "some-job"

  labels:

    application: "some-job"

    namespace: "us-east-1-stg"

    helm.sh/chart: {{ include "..chart" . }}

spec:

  scaleTargetRef:

    apiVersion: apps/v1beta2

    kind: Deployment

    name: "some-job"

  minReplicas: 1

  maxReplicas: 20

  metrics:

  - type: Object

      object:

        metricName: rabbitmq_core_queue_messages

        target:

          apiVersion: custom.metrics.k8s.io/v1beta1

          name: "queue_1_staging"

          namespace: "us-east-1-stg"

        targetValue: 1

I know the syntax is probably way off, but I really haven't found any documentation that can narrow it down.

I also know that other projects like: Keda https://github.com/kedacore/keda exist and provide a much simpler solution to do this, it isn't production ready yet.

Any help with this would be greatly appreciated, I've basically hit a brick wall.

Thanks!

[Michael Klishin]

Automatic pod scaling is more of a Kubernetes topic. Worth mentioning that

1. 3.8.0 is out with a new Prometheus plugin: https://www.rabbitmq.com/prometheus.html

2.  3.8.1 or 3.8.2 will ship with `rabbitmq-queues rebalance` and `rabbitmq-upgrade post_upgrade` which will be useful after the new pod is added.

How exactly you should execute "post scaling" hooks, I'm not sure but at least there will be a way to rebalance queue masters which I've figured is our end goal?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/052701e8-e373-4001-857c-6c30506c0ddd%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Rinat Shigapov]

We successfully scale our rabbitmq worker pools using 2 HPA metrics. On test environment we even scale from zero pods using changes in HPA - https://github.com/kubernetes/kubernetes/pull/74526

To export metrics from Prometheus to K8S API we use https://github.com/DirectXMan12/k8s-prometheus-adapter

вторник, 13 августа 2019 г., 18:00:17 UTC+3 пользователь Chad Upton написал: