aten_detector crashing together with memory spikes

[Johan Rhodin]

Hi,

In two different clusters (Both 3.8.2 with Erlang 22.0.7) we've seen aten_detector crashing right before OOM. Can it be related or is it just a symptom of high memory pressure in general? None of these clusters are using quorum queues. Similar case reported in [1].

Unfortunately prometheus scraping is timing out so might have to go back to some more basic memory logging and try to log the highest erlang mailbox numbers and associated processes.

Mar 24 18:33:25 RABBITNAME-02 rabbitmq-server[23831]: eheap_alloc: Cannot allocate 1581921408 bytes of memory (of type "old_heap").

See attached for log.

NB: Channel churn is very high. Simliar to [2], maybe.

[Karl Nilsson]

We have seen this a few times and suspect it is a symptom rather than a cause. We added a bit more logging in 3.8.3 (I think) would hopefully would give us a better idea.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_742256D2-2DCFD7-2D4ADA-2D858C-2D494EA5F02AED-2540cloudamqp.com&d=DwIFaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=IN9u8rsUkejqEICYaP4x2-95QZYcNaKCou5rJnx7AcE&e= .

Steep memory spikes:

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_742256D2-2DCFD7-2D4ADA-2D858C-2D494EA5F02AED-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=IN9u8rsUkejqEICYaP4x2-95QZYcNaKCou5rJnx7AcE&e= .



--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_742256D2-2DCFD7-2D4ADA-2D858C-2D494EA5F02AED-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=IN9u8rsUkejqEICYaP4x2-95QZYcNaKCou5rJnx7AcE&e= .
[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msg_rabbitmq-2Dusers_CNXPvwHpbgQ_t1bxlGRrAQAJ&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=h30PnON6twlDyIr22jtmujIrf_QXy92qiYrVCVQLqLQ&e=
[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msg_rabbitmq-2Dusers_EIo6J4vWupQ_G71-2DCEQ6GwAJ&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=4oNvzYx6DaVl1LETMxPUCjTYJhL-P4VLcPJ1FbnRiDs&e=

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_742256D2-2DCFD7-2D4ADA-2D858C-2D494EA5F02AED-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=opi9G2FkNu0cXYiVF-A6mfH0OOvfYv6Ki1WLkdLIPcA&m=-Gjf-vbyVryS68N3K0tLqIjgUEVodRLVnu3lJSXOUXs&s=IN9u8rsUkejqEICYaP4x2-95QZYcNaKCou5rJnx7AcE&e= .

--

Karl Nilsson

Pivotal/RabbitMQ