queue showing in management interface and rabbitmqctl - but not accessible

[Olivier Van Dam]

I have a peculiar issue in our RabbitMQ cluster, a queue that shows up with messages in the management console (as synchronized), shows up also through rabbitmqctl (but without message count there).

Attempts at getting the messages from the queue fail (404 - not found)

RabbitMQ 3.6.5, Erlang 19.1 Cluster 2 nodes - queues replicated (exactly 2 - manual sync mode)

I'm currently checking the logs (I only have about 40 days of them though) trying to track down when the issue occurred.

Possible "candidate" related to this:

** Reason for termination == 

** {function_clause,

       [{rabbit_variable_queue,d,

            [{delta,29,1,29}],

            [{file,"src/rabbit_variable_queue.erl"},{line,1096}]},

        {rabbit_variable_queue,maybe_deltas_to_betas,2,

            [{file,"src/rabbit_variable_queue.erl"},{line,2479}]},

        {rabbit_variable_queue,fetch_from_q3,1,

            [{file,"src/rabbit_variable_queue.erl"},{line,2423}]},

        {rabbit_variable_queue,queue_out,1,

            [{file,"src/rabbit_variable_queue.erl"},{line,1382}]},

        {rabbit_variable_queue,fetch,2,

            [{file,"src/rabbit_variable_queue.erl"},{line,611}]},

        {rabbit_priority_queue,fetch,2,

            [{file,"src/rabbit_priority_queue.erl"},{line,299}]},

        {rabbit_mirror_queue_master,fetch,2,

            [{file,"src/rabbit_mirror_queue_master.erl"},{line,359}]},

        {rabbit_amqqueue_process,fetch,2,

            [{file,"src/rabbit_amqqueue_process.erl"},{line,674}]}]}

Cluster has 399 queues, this is the only one with this issue.

The queue nature is to store events that have failed processing, no automatic processing is in place (meaning no process listens to the queue by default)

Basic get fails (tried with pika, as well as management console).

Next "test" will be to see if additional messages matching the binding in place would reach the queue or not.

Any help (beyond the very reasonable upgrade to latest, which I cannot do for the moment, due to management constraints) would be greatly appreciated,

be it in terms of pointers to general documentation, erlang debugging tips or other.

Kind regards,

  Olivier

[Olivier Van Dam]

One missing information in original mail, OS is Centos 7.2

[Diana Parra Corbacho]

You're running an unsupported version of RabbitMQ [1]. Please upgrade as many of those issues have been solved since then.

[1] https://www.rabbitmq.com/versions.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0f7e6bf9-a195-4295-9e92-f391a658bfd0%40googlegroups.com.

--

Diana Corbacho

Pivotal / RabbitMQ

[Olivier Van Dam]

Hi Diana,

As stated in my original mail, I do understand this is the main recommandation, 

but I cannot get the production cluster down on short notice, management has indicated that timeline for upgrade will need to be planned in the next 2 months.

So in the mean time, I need to try and stabilize the version in place.

Could you (or anyone else) point me to information regarding the issue prior to upgrades that resolved it,

so that I can see approaches to stabilize the situation, possibly recover the messages that are "stuck"?

Thanks in advance,

  Olivier.  

On Thursday, October 24, 2019 at 11:54:12 AM UTC+2, Diana Corbacho wrote:

You're running an unsupported version of RabbitMQ [1]. Please upgrade as many of those issues have been solved since then.

[1] https://www.rabbitmq.com/versions.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/0f7e6bf9-a195-4295-9e92-f391a658bfd0%40googlegroups.com.

[Luke Bakken]

Hi Olivier,

It appears you have hit a similar issue to what was reported here:

https://groups.google.com/d/topic/rabbitmq-users/7kpZAWnSks0/discussion

Somehow the data files that store your queue messages have become corrupted. I was unable to resolve the other user's issue. The dmesg file provided by that user indicated that the operating system's out-of-memory killer abruptly killed the Erlang VM process which (presumably) lead to file corruption.

If you could check your system's dmesg and /var/log/messages files for those events it could shed some light on this.

Thanks,

Luke

[Olivier Van Dam]

Hi Luke,

Thanks for the pointers!

I'll be checking all mnesia files more in details, but was able to at least "reset" by deleting the queues and recreating them:

sudo rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"/">>, queue, <<"failing_queue">>)), rabbit_amqqueue:delete_crashed(Q).'

I do feel there is an issue with the management plugin on that cluster as I tried creating a shovel (queue --> exchange - original length) that terminated, emptying the source queue but never publishing the content on the target exchange (amq.fanout, 2 bindings to queues).

But acknowledging the version we're currently using, I'll simply script future tasks till we have completed the upgrade.

Thanks to all the RabbitMQ team still, it's an amazing product. 

Kind regards,

  Olivier.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e362d08d-7e12-427b-9b99-ad31e8790380%40googlegroups.com.