Some messages remains unacknowledged from consumer which is giving reached_max_restart_intensity error.

[Swapnil Bhusari]

I have upgraded rabbitmq-server from 3.8.9 to 3.9.8 also tried upgrading on 3.8.25.

Getting following crash report repeatedly after every 1-2 seconds.

 

/crash.log:

2021-11-25 11:31:34 =ERROR REPORT====

** Generic server <0.32413.4079> terminating

** Last message in was {'$gen_cast',{method,{'basic.publish',0,<<>>,<<"amq.rabbitmq.reply-to.g1hkABZyYWJiaXRAZGwzNjBnMTAtMDEtdjAzAAAqeQAAAABhnGrt.TVTqLlu7vlQzO0tsyq11wA==">>,false,false},{content,60,none,<<252,0,30,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110,59,99,104,97,114,115,101,116,61,117,116,102,45,56,5,85,84,70,45,56,0,0,1,234,22,120,45,105,116,114,112,45,115,101,114,118,105,99,101,45,99,111,110,116,101,120,116,83,0,0,1,173,123,34,116,101,110,97,110,116,34,58,34,34,44,34,115,101,114,118,105,99,101,73,100,34,58,34,115,101,114,118,105,99,101,46,105,116,114,112,46,116,101,99,104,46,97,119,115,46,99,111,109,109,111,110,34,44,34,115,101,114,118,105,99,101,73,110,115,116,97,110,99,101,73,100,34,58,34,65,55,49,50,57,55,69,55,51,65,66,65,52,50,53,65,57,69,70,51,53,52,67,57,54,54,66,52,56,56,54,66,34,44,34,108,111,103,82,101,115,111,117,114,99,101,66,117,110,100,108,101,34,58,34,99,111,109,46,118,114,116,115,46,105,116,114,112,46,116,101,99,104,110,111,108,111,103,121,46,97,119,115,46,117,116,105,108,46,108,111,99,97,108,101,46,97,119,115,34,44,34,101,118,101,110,116,82,101,115,111,117,114,99,101,66,117,110,100,108,101,34,58,34,99,111,109,46,118,114,116,115,46,105,116,114,112,46,116,101,99,104,110,111,108,111,103,121,46,97,119,115,46,117,116,105,108,46,108,111,99,97,108,101,46,97,119,115,34,44,34,116,97,115,107,82,101,115,111,117,114,99,101,66,117,110,100,108,101,34,58,110,117,108,108,44,34,115,101,114,118,105,99,101,72,111,115,116,34,58,34,100,108,51,54,48,103,49,48,45,48,49,45,118,48,51,46,118,120,105,110,100,105,97,46,118,101,114,105,116,97,115,46,99,111,109,34,44,34,104,111,115,116,73,100,34,58,34,79,83,95,85,85,73,68,58,123,48,48,48,49,48,48,53,48,45,53,54,56,100,45,55,99,97,102,45,48,48,48,48,45,48,48,48,48,48,48,48,48,48,48,48,48,125,34,44,34,114,109,73,100,34,58,34,55,101,50,50,100,54,49,56,45,56,52,97,101,45,52,49,102,52,45,57,99,100,55,45,54,55,57,100,51,97,49,56,52,54,49,56,34,125,27,120,45,105,116,114,112,45,115,101,114,118,105,99,101,45,114,101,115,112,111,110,115,101,99,111,100,101,73,0,0,0,200,2,0,32,54,54,48,68,70,65,55,67,55,70,55,70,52,57,48,68,57,48,69,50,70,68,69,57,70,51,68,70,48,48,55,68>>,rabbit_framing_amqp_0_9_1,[<<"json_message">>]},flow}}

** When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,1,<0.31840.4079>,<0.31670.4079>,<0.31840.4079>,<<"127.0.0.1:40096 -> 127.0.0.1:5672">>,undefined,{user,<<"guest">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,<<"src.common.q.info">>,<0.3918.3968>,[{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"basic.nack">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],none,1,134217728,1800000,#{},1000000000},{lstate,<0.911.3969>,false},none,2,{1,{[{1,<<"amq.ctag-jr4dExNXbGkS_dHrkPJgg">>,1637820094484,{<0.796.0>,64}}],[]}},#{<0.796.0> => {resource,<<"/">>,queue,<<"src.common.q.info">>}},{state,#{<0.796.0> => #Ref<0.284499808.3633840131.207393>},erlang},#{<<"amq.ctag-jr4dExNXbGkS_dHrkPJgg">> => {{amqqueue,{resource,<<"/">>,queue,<<"src.common.q.info">>},true,false,none,[],<0.796.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>},rabbit_classic_queue,#{}},{false,1,false,[]}}},#{<0.796.0> => {1,{<<"amq.ctag-jr4dExNXbGkS_dHrkPJgg">>,nil,nil}}},{set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[<0.796.0>],[],[],[],[],[],[],[],[]}}},{state,none,5000,undefined},false,1,{unconfirmed,{0,nil},#{},#{}},[],[],none,flow,[],#{},#Ref<0.284499808.3576954887.30932>}

** Reason for termination ==

** {{case_clause,<<100,0,22,114,97,98,98,105,116,64,100,108,51,54,48,103,49,48,45,48,49,45,118,48,51,0,0,42,121,0,0,0,0,97,156,106,237>>},[{pid_recomposition,from_binary,1,[{file,"src/pid_recomposition.erl"},{line,42}]},{rabbit_direct_reply_to,decode_reply_to_v2,2,[{file,"src/rabbit_direct_reply_to.erl"},{line,69}]},{rabbit_channel,deliver_reply,2,[{file,"src/rabbit_channel.erl"},{line,303}]},{rabbit_exchange,'-route/2-lc$^0/1-0-',2,[{file,"src/rabbit_exchange.erl"},{line,414}]},{rabbit_exchange,route,2,[{file,"src/rabbit_exchange.erl"},{line,415}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1370}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]}

2021-11-25 11:31:34 =CRASH REPORT====

  crasher:

    initial call: rabbit_channel:init/1

    pid: <0.32413.4079>

    registered_name: []

    exception exit: {{{case_clause,<<100,0,22,114,97,98,98,105,116,64,100,108,51,54,48,103,49,48,45,48,49,45,118,48,51,0,0,42,121,0,0,0,0,97,156,106,237>>},[{pid_recomposition,from_binary,1,[{file,"src/pid_recomposition.erl"},{line,42}]},{rabbit_direct_reply_to,decode_reply_to_v2,2,[{file,"src/rabbit_direct_reply_to.erl"},{line,69}]},{rabbit_channel,deliver_reply,2,[{file,"src/rabbit_channel.erl"},{line,303}]},{rabbit_exchange,'-route/2-lc$^0/1-0-',2,[{file,"src/rabbit_exchange.erl"},{line,414}]},{rabbit_exchange,route,2,[{file,"src/rabbit_exchange.erl"},{line,415}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1370}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

    ancestors: [<0.29938.3968>,<0.29305.4079>,<0.32095.4079>,<0.29459.4079>,<0.1836.0>,<0.1835.0>,<0.1834.0>,<0.1832.0>,<0.1831.0>,rabbit_sup,<0.278.0>]

    message_queue_len: 0

    messages: []

    links: [<0.29938.3968>]

    dictionary: [{guid,{{2198883681,1693869178,2047299175,3223789394},0}},{channel_operation_timeout,15000},{process_name,{rabbit_channel,{<<"127.0.0.1:40096 -> 127.0.0.1:5672">>,1}}},{rand_seed,{#{jump => #Fun<rand.3.47293030>,max => 288230376151711743,next => #Fun<rand.5.47293030>,type => exsplus},[218854499923923642|140883450362971118]}},{guid_secure,{{1,'rabbit@dl360g10-01-v03',#Ref<0.284499808.3633840131.207386>},0}},{credit_flow_default_credit,{400,200}},{{credit_to,<0.31840.4079>},199},{{xtype_to_module,topic},rabbit_exchange_type_topic},{topic_permission_cache,[{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.x.y.aa.#.complete">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.a.x">>},#{routing_key => <<"src.a.request.aa.storage.#">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.rm.domain.#">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.public.r.*.*">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.x.y.r.protect">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.x.y.aw.#.complete">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.a.x">>},#{routing_key => <<"src.a.request.aw.storage.#">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.validations.x">>},#{routing_key => <<"src.validation.request.a.nc.vc.add">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.validations.x">>},#{routing_key => <<"src.validation.request.a.com.vc.reg">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.validations.x">>},#{routing_key => <<"src.a.request.vc.val.#">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.x.validations.x">>},#{routing_key => <<"src.a.request.vc.configureprotection.validation">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read},{{resource,<<"/">>,topic,<<"src.events.x">>},#{routing_key => <<"src.event.public.upgrade.success">>,variable_map => #{<<"username">> => <<"guest">>,<<"vhost">> => <<"/">>}},read}]},{permission_cache_can_expire,false},{permission_cache,[{{resource,<<"/">>,exchange,<<>>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.aw.common.q.info">>},#{},read},{{resource,<<"/">>,queue,<<"src.tech.aa.storage.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.aw.storage.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.vc.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.obs.copy.q.info">>},#{},write},{{resource,<<"/">>,exchange,<<"src.events.x">>},#{},read},{{resource,<<"/">>,exchange,<<"src.x.a.x">>},#{},read},{{resource,<<"/">>,queue,<<"src.tech.aw.hostnetwork.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.aa.hostnetwork.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.aw.datagateway.validation.q.info">>},#{},write},{{resource,<<"/">>,queue,<<"src.tech.aw.datagateway.q.info">>},#{},write}]},{msg_size_for_gc,1555}]

    trap_exit: true

    status: running

    heap_size: 2586

    stack_size: 28

    reductions: 6441090

  neighbours:

2021-11-25 11:31:34 =SUPERVISOR REPORT====

     Supervisor: {<0.29938.3968>,rabbit_channel_sup}

     Context:    child_terminated

     Reason:     {{case_clause,<<100,0,22,114,97,98,98,105,116,64,100,108,51,54,48,103,49,48,45,48,49,45,118,48,51,0,0,42,121,0,0,0,0,97,156,106,237>>},[{pid_recomposition,from_binary,1,[{file,"src/pid_recomposition.erl"},{line,42}]},{rabbit_direct_reply_to,decode_reply_to_v2,2,[{file,"src/rabbit_direct_reply_to.erl"},{line,69}]},{rabbit_channel,deliver_reply,2,[{file,"src/rabbit_channel.erl"},{line,303}]},{rabbit_exchange,'-route/2-lc$^0/1-0-',2,[{file,"src/rabbit_exchange.erl"},{line,414}]},{rabbit_exchange,route,2,[{file,"src/rabbit_exchange.erl"},{line,415}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1370}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]}

     Offender:   [{pid,<0.32413.4079>},{id,channel},{mfars,{rabbit_channel,start_link,[1,<0.31840.4079>,<0.31670.4079>,<0.31840.4079>,<<"127.0.0.1:40096 -> 127.0.0.1:5672">>,rabbit_framing_amqp_0_9_1,{user,<<"guest">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,[{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"basic.nack">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],<0.3918.3968>,<0.911.3969>]}},{restart_type,intrinsic},{shutdown,70000},{child_type,worker}]

 

2021-11-25 11:31:34 =SUPERVISOR REPORT====

     Supervisor: {<0.29938.3968>,rabbit_channel_sup}

     Context:    shutdown

     Reason:     reached_max_restart_intensity

     Offender:   [{pid,<0.32413.4079>},{id,channel},{mfars,{rabbit_channel,start_link,[1,<0.31840.4079>,<0.31670.4079>,<0.31840.4079>,<<"127.0.0.1:40096 -> 127.0.0.1:5672">>,rabbit_framing_amqp_0_9_1,{user,<<"guest">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,[{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"basic.nack">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],<0.3918.3968>,<0.911.3969>]}},{restart_type,intrinsic},{shutdown,70000},{child_type,worker}]

 

One of the queue gets stuck with no consumer's response.

Few more details:

No of nodes: 1
OS: Linux

[Karl Nilsson]

Which Erlang version are you using?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3b7e22f3-b67d-41b8-99c5-d801d946b4b2n%40googlegroups.com.

--

Karl Nilsson

[Swapnil Bhusari]

hi Karl,

Thank you for your reply.

I have tried erlang 23.2.4 with rabbitmq version 3.9.8 and 3.8.25.
I have also tried erlang version 23.3.4.7 with  rabbitmq version 3.8.25(Referred logs are from this setup) and got these errors with all these combinations.

Prior to this I was using rabbitmq version 3.8.9 and erlang version 23.2.4 and everything was working on that setup.

Thanks and regards,
Swapnil Bhusari

[Loïc Hoguin]

Hello!

 

I think this is a legitimate issue. Please open a ticket with the crash logs as well as the Erlang/RabbitMQ versions you were using before upgrading (and if the messages have been queued for a long time, any version you were using that may have been involved).

 

The problem is that the pid_recompose module does not support the old ATOM_EXT format https://www.erlang.org/doc/apps/erts/erl_ext_dist.html#atom_ext--deprecated-

 

Cheers,

 

-- 

Loïc Hoguin

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1fa67f80-ef62-4af1-8fea-62900ce8c8a4n%40googlegroups.com.

[Luke Bakken]

Hello,

In addition to the useful information that Loïc requested would you please export the definitions from this environment? (https://www.rabbitmq.com/definitions.html#export)

It looks like you're using the "direct reply-to" feature of RabbitMQ (https://www.rabbitmq.com/direct-reply-to.html). Could you please confirm that you are using that feature as well as let us know which client library you're using? Is there anything "unique" about how you are using it?

Thanks -

Luke 

On Friday, November 26, 2021 at 1:21:58 AM UTC-8 lho...@vmware.com wrote:

Hello!

I think this is a legitimate issue. Please open a ticket with the crash logs as well as the Erlang/RabbitMQ versions you were using before upgrading (and if the messages have been queued for a long time, any version you were using that may have been involved).

The problem is that the pid_recompose module does not support the old ATOM_EXT format https://www.erlang.org/doc/apps/erts/erl_ext_dist.html#atom_ext--deprecated-

Cheers,

Loïc Hoguin 

[Swapnil Bhusari]

Hi  Loïc,  Luke ,

Thank you for your reply, I had opened a ticket as you asked: https://github.com/rabbitmq/rabbitmq-server/issues/3809 but it got closed with reply

"This is very similar to #3751. Direct reply-to value encoding has changed in 3.8.13. If a client tried to use a previously computed value, e.g. because it was stored, its decoding will fail.

Since direct reply-to "identifiers" are extremely short lived, the vast majority of environments will never run into this.
Those that do must avoid using those older response identifiers."

But I am not using direct reply-to identifier. I am using spring-amqp-2.3.11 as client and I have also set AmqpTemplate.setUseDirectReplyToContainer(false) in my code.

Captured message which got stuck in my queue:
{

   "headers": {

.............

.............

   }

  "properties": {

    "content_encoding": "UTF-8",

    "content_type": "application/json",

    "delivery_mode": 2,

    "exchange": "x.src.tl.x",

    "message_id": "660DFA7C7F7F490D90E2FDE9F3DF007D",

    "priority": 0,

    "reply_to": "amq.rabbitmq.reply-to.g1hkABZyYWJiaXRAZGwzNjBnMTAtMDEtdjAzAAAqeQAAAABhnGrt.TVTqLlu7vlQzO0tsyq11wA==",

    "routing_key": "x.tl.rq.aw.cm.dc"

  }

}

Please mention if any more details are required.

Thanks and Regards,
Swapnil Bhusari

[Luke Bakken]

https://github.com/rabbitmq/rabbitmq-server/issues/3809#issuecomment-982676412

You can see the message has a "reply_to" property. Even if your application does not use reply-to, perhaps some other application does.

Luke

[Swapnil Bhusari]

hi Luke,

Thank you for your replay.

Wanted to confirm one more thing, would any messages sent using RPC pattern cause such issues even when Direct reply-to is not used? We are using spring-amqp sendAndReceive() to send the message.

Thanks and Regards,
Swapnil Bhusari

[Swapnil Bhusari]

Hi Luke,

Is the following scenario supported by RabbitMQ?

- Environment running RabbitMQ version 3.8.9

- Client sends a message which is in the Queue and not yet processed

- Environment is upgraded to RabbitMQ version 3.9.8

- Server now processes this message from the queue and returns a response

Are above steps expected to work:

For messages using reply-to (RPC style messages)

For messages not using reply-to

Note that we are using RabbitMQ cluster with single node only which we are upgrading to newer RabbitMQ versions.

Thanks and regards,
Swapnil Bhusari

[Luke Bakken]

Hello,

Since the topic of this discussion has strayed from the original message, please ask a new question next time.

Yes, those steps should work for non reply-to messages. Be sure your exchanges and queues are durable, and you have published persistent messages.

I will ask the team about the reply-to scenario.

Luke

[Luke Bakken]

Hello  Swapnil,

Direct Reply-To does not use "real" queues - https://www.rabbitmq.com/direct-reply-to.html

Because of this, you should stop your RPC applications prior to an upgrade and restart them after an upgrade. After the upgrade the reply-to value will no longer be valid because the associated channel will be closed when RabbitMQ stops.

Thanks,

Luke

[Gary Russell]

You should ask questions about Spring on Stack Overflow, not here.

The reply will be discarded in that case because the client side will have timed out (and/or failed because the broker was restarted).

The unexpected reply will be logged (no corresponding request waiting).

This is true, regardless of the replyTo mechanism used by the sendAndRecieve() operation.

There are 3 modes:

Direct reply_to using an internal listener container within the template.

Named reply queue (using an external container with the RabbitTemplate being the message listener)

Temporary reply queue.

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Swapnil Bhusari <swapnil....@gmail.com>
Sent: Wednesday, December 1, 2021 8:34 AM

To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [Suspected Spam] Re: [rabbitmq-users] Some messages remains unacknowledged from consumer which is giving reached_max_restart_intensity error.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7e6440f6-d5d5-4707-8319-b1a2ab7b4f11n%40googlegroups.com.

[guy]

Hi Guys,

I'm experiencing the same thing as:  "Some messages remains unacknowledged from consumer which is giving reached_max_restart_intensity error"
Is that issue is been solved? which version?

I'm using
RabbitMQ 3.8.27
Erlang 24.3.2

Thanks