Mysterious Consumer Cancelled – suspecting corrupting message
170 views
Skip to first unread message
Oytun Tez
Aug 20, 2019, 10:28:10 AM8/20/19
to rabbitm...@googlegroups.com
Hi there,
We've been witnessing some weird ConsumerCa.... exceptions in our Apache Flink application.
Java client version: 4.2.0
I am attaching the termination/crash log here. It is very cryptic, I couldn't understand what the error is really talking about. I suspect our DLX queue (this error is not thrown in dlx queue though, this is a different queue), or I had read somewhere the number of ctags or redelivery tags could cause some issues.
Can you enlighten me? Meanwhile, I am trying to find the suspected message among thousands of other messages.
---
Oytun Tez
M O T A W O R D
The World's Fastest Human Translation Platform.
Oytun Tez
Aug 20, 2019, 10:31:14 AM8/20/19
to rabbitm...@googlegroups.com
Oops, the full name of the exception is: ConsumerCancelledException
This is thrown in the QueueingConsumer class, which Flink binding uses. I read the whole binding library, it didn't look strange – I have implemented RabbitMQ in multiple environments before with advanced-ish configurations.
---
Oytun Tez
M O T A W O R D
The World's Fastest Human Translation Platform.
Oytun Tez
Aug 20, 2019, 1:47:59 PM8/20/19
to rabbitmq-users
And now, I can't even start processing messages... My connection is almost immediately shut down:
rabbitmq_1 | 2019-08-20 17:44:03.595 [debug] <0.1222.0> Supervisor {<0.1222.0>,rabbit_channel_sup} started rabbit_channel:start_link(1, <0.1011.0>, <0.1225.0>, <0.1011.0>, <<"172.18.0.1:41982 -> 172.18.0.5:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"ipm">>,[ipm],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentica...">>,...},...], <0.1172.0>, <0.1226.0>) at pid <0.1231.0>
rabbitmq_1 | 2019-08-20 17:44:03.614 [error] <0.907.0> Error on AMQP connection <0.907.0> (172.18.0.1:41954 -> 172.18.0.5:5672, vhost: '/', user: 'ipm', state: running), channel 1:
rabbitmq_1 | {{{function_clause,
rabbitmq_1 | [{rabbit_variable_queue,d,
rabbitmq_1 | [{delta,5783552,299,0,5783552}],
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,1184}]},
rabbitmq_1 | {rabbit_variable_queue,maybe_deltas_to_betas,2,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,2656}]},
rabbitmq_1 | {rabbit_variable_queue,fetch_from_q3,1,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,2579}]},
rabbitmq_1 | {rabbit_variable_queue,queue_out,1,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,1481}]},
rabbitmq_1 | {rabbit_variable_queue,collect_by_predicate,3,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,1701}]},
rabbitmq_1 | {rabbit_variable_queue,fetch_by_predicate,4,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,1660}]},
rabbitmq_1 | {rabbit_variable_queue,fetchwhile,4,
rabbitmq_1 | [{file,"src/rabbit_variable_queue.erl"},{line,679}]},
rabbitmq_1 | {rabbit_priority_queue,fetchwhile,4,
rabbitmq_1 | [{file,"src/rabbit_priority_queue.erl"},{line,291}]}]},
rabbitmq_1 | {gen_server2,call,
rabbitmq_1 | [<0.424.0>,
rabbitmq_1 | {basic_consume,false,<0.959.0>,<0.955.0>,false,0,
rabbitmq_1 | <<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>,false,[],
rabbitmq_1 | {'basic.consume_ok',<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>},
rabbitmq_1 | <<"ipm">>},
rabbitmq_1 | infinity]}},
rabbitmq_1 | [{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,335}]},
rabbitmq_1 | {rabbit_channel,'-basic_consume/8-fun-0-',9,
rabbitmq_1 | [{file,"src/rabbit_channel.erl"},{line,1576}]},
rabbitmq_1 | {rabbit_misc,with_exit_handler,2,[{file,"src/rabbit_misc.erl"},{line,509}]},
rabbitmq_1 | {rabbit_channel,basic_consume,8,
rabbitmq_1 | [{file,"src/rabbit_channel.erl"},{line,1573}]},
rabbitmq_1 | {rabbit_channel,handle_method,3,
rabbitmq_1 | [{file,"src/rabbit_channel.erl"},{line,1295}]},
rabbitmq_1 | {rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,567}]},
rabbitmq_1 | {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1056}]},
rabbitmq_1 | {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
rabbitmq_1 | 2019-08-20 17:44:03.615 [warning] <0.907.0> Non-AMQP exit reason '{{{function_clause,[{rabbit_variable_queue,d,[{delta,5783552,299,0,5783552}],[{file,"src/rabbit_variable_queue.erl"},{line,1184}]},{rabbit_variable_queue,maybe_deltas_to_betas,2,[{file,"src/rabbit_variable_queue.erl"},{line,2656}]},{rabbit_variable_queue,fetch_from_q3,1,[{file,"src/rabbit_variable_queue.erl"},{line,2579}]},{rabbit_variable_queue,queue_out,1,[{file,"src/rabbit_variable_queue.erl"},{line,1481}]},{rabbit_variable_queue,collect_by_predicate,3,[{file,"src/rabbit_variable_queue.erl"},{line,1701}]},{rabbit_variable_queue,fetch_by_predicate,4,[{file,"src/rabbit_variable_queue.erl"},{line,1660}]},{rabbit_variable_queue,fetchwhile,4,[{file,"src/rabbit_variable_queue.erl"},{line,679}]},{rabbit_priority_queue,fetchwhile,4,[{file,"src/rabbit_priority_queue.erl"},{line,291}]}]},{gen_server2,call,[<0.424.0>,{basic_consume,false,<0.959.0>,<0.955.0>,false,0,<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>,false,[],{'basic.consume_ok',<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>},<<"ipm">>},infinity]}},[{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,335}]},{rabbit_channel,'-basic_consume/8-fun-0-',9,[{file,"src/rabbit_channel.erl"},{line,1576}]},{rabbit_misc,with_exit_handler,2,[{file,"src/rabbit_misc.erl"},{line,509}]},{rabbit_channel,basic_consume,8,[{file,"src/rabbit_channel.erl"},{line,1573}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1295}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,567}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1056}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}'
rabbitmq_1 | 2019-08-20 17:44:03.626 [error] <0.424.0> ** Generic server <0.424.0> terminating
rabbitmq_1 | ** Last message in was {basic_consume,false,<0.959.0>,<0.955.0>,false,0,<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>,false,[],{'basic.consume_ok',<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>},<<"ipm">>}
rabbitmq_1 | ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"ipm-events">>},true,false,none,[],<0.424.0>,[],[],[],[{vhost,<<"/">>},{name,<<"DLX">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"dead-letter-exchange">>,<<"dead-letter-exchange">>}]},{priority,1}],undefined,[],undefined,live,0,[],<<"/">>,#{user => <<"rmq-internal">>}},none,false,rabbit_priority_queue,{passthrough,rabbit_variable_queue,{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,5767168,634,0,5783552},{12,{[{msg_status,5756985,<<99,228,91,231,4,157,51,96,203,31,176,80,114,8,212,14>>,{basic_message,{resource,<<"/">>,exchange,<<"default-exchange">>},[<<"events.projects.98499.en-US.147963">>],{content,60,{'P_basic',undefined,undefined,[{<<"content-length">>,longstr,<<"221">>},{<<"timestamp_in_ms">>,long,1566238880066}],2,undefined,undefined,undefined,undefined,undefined,1566238880,undefined,undefined,undefined,undefined},none,none,[<<"{\"type\":\"LOCK_STRING_FROM_REMOTE\",\"payload\":{\"phraseId\":1109799,\"currentUser\":\"user0004511\",\"onlyRemote\":true,\"vendorId\":4511,\"userId\":4511},\"event\":\"LOCK_STRING_FROM_REMOTE\",\"_queuedAt\":1566238879,\"sender_id\":\"jzipj1wg\"}">>]},<<99,228,91,231,4,157,51,96,203,31,176,80,114,8,212,14>>,true},true,true,false,true,queue_index,{message_properties,undefined,false,221}}],[{msg_status,5756974,<<121,255,160,50,222,229,37,178,65,15,13,0,205,179,88,30>>,{basic_message,{resource,<<"/">>,exchange,<<"default-exchange">>},[<<"events.projects.98500.zh-CN.147964">>],{content,60,{'P_basic',undefined,undefined,[{<<"content-length">>,longstr,<<"223">>},{<<"timestamp_in_ms">>,long,1566238861859}],2,undefined,undefined,undefined,undefined,undefined,1566238861,undefined,undefined,undefined,undefined},none,none,[<<"{\"type\":\"LOCK_STRING_FROM_REMOTE\",\"payload\":{\"phraseId\":1109890,\"currentUser\":\"user0017932\",\"onlyRemote\":true,\"vendorI...">>]},...},...},...]}},...}},...}
rabbitmq_1 | ** Reason for termination ==
rabbitmq_1 | ** {function_clause,[{rabbit_variable_queue,d,[{delta,5783552,299,0,5783552}],[{file,"src/rabbit_variable_queue.erl"},{line,1184}]},{rabbit_variable_queue,maybe_deltas_to_betas,2,[{file,"src/rabbit_variable_queue.erl"},{line,2656}]},{rabbit_variable_queue,fetch_from_q3,1,[{file,"src/rabbit_variable_queue.erl"},{line,2579}]},{rabbit_variable_queue,queue_out,1,[{file,"src/rabbit_variable_queue.erl"},{line,1481}]},{rabbit_variable_queue,collect_by_predicate,3,[{file,"src/rabbit_variable_queue.erl"},{line,1701}]},{rabbit_variable_queue,fetch_by_predicate,4,[{file,"src/rabbit_variable_queue.erl"},{line,1660}]},{rabbit_variable_queue,fetchwhile,4,[{file,"src/rabbit_variable_queue.erl"},{line,679}]},{rabbit_priority_queue,fetchwhile,4,[{file,"src/rabbit_priority_queue.erl"},{line,291}]}]}
rabbitmq_1 | 2019-08-20 17:44:03.715 [error] <0.424.0> CRASH REPORT Process <0.424.0> with 0 neighbours exited with reason: no function clause matching rabbit_variable_queue:d({delta,5783552,299,0,5783552}) line 1184 in gen_server2:terminate/3 line 1172
rabbitmq_1 | 2019-08-20 17:44:03.721 [info] <0.907.0> closing AMQP connection <0.907.0> (172.18.0.1:41954 -> 172.18.0.5:5672, vhost: '/', user: 'ipm')
rabbitmq_1 | 2019-08-20 17:44:03.724 [error] <0.423.0> Supervisor {<0.423.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"ipm-events">>},true,false,none,[],<0.2097.0>,[],[],[],[{vhost,...},...],...}, recovery, <0.422.0>) at <0.424.0> exit with reason no function clause matching rabbit_variable_queue:d({delta,5783552,299,0,5783552}) line 1184 in context child_terminated
rabbitmq_1 | 2019-08-20 17:44:03.771 [debug] <0.423.0> Supervisor {<0.423.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"ipm-events">>},true,false,none,[],<0.2097.0>,[],[],[],[{vhost,...},...],...}, recovery, <0.422.0>) at pid <0.1235.0>
rabbitmq_1 | 2019-08-20 17:44:03.772 [error] <0.959.0> ** Generic server <0.959.0> terminating
rabbitmq_1 | ** Last message in was {'$gen_cast',{method,{'basic.consume',0,<<"ipm-events">>,<<>>,false,false,false,false,[]},none,noflow}}
rabbitmq_1 | ** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.907.0>,<0.954.0>,<0.907.0>,<<"172.18.0.1:41954 -> 172.18.0.5:5672">>,rabbit_reader,{lstate,<0.955.0>,false},{{[],[]},[]},1,{[],[]},{user,<<"ipm">>,[ipm],[{rabbit_auth_backend_internal,none}]},<<"/">>,<<"ipm-events">>,#{},{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},erlang},#{},#{},{set,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<0.925.0>,{state,fine,5000,#Ref<0.1993627454.2369519618.257458>},false,1,{{0,nil},{0,nil}},[],[],{{0,nil},{0,nil}},[{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"basic.nack">>,bool,true},{<<"publisher_confirms">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],none,0,none,flow,[{rabbit_timestamp_interceptor,undefined}]}
rabbitmq_1 | ** Reason for termination ==
rabbitmq_1 | ** {{{function_clause,[{rabbit_variable_queue,d,[{delta,5783552,299,0,5783552}],[{file,"src/rabbit_variable_queue.erl"},{line,1184}]},{rabbit_variable_queue,maybe_deltas_to_betas,2,[{file,"src/rabbit_variable_queue.erl"},{line,2656}]},{rabbit_variable_queue,fetch_from_q3,1,[{file,"src/rabbit_variable_queue.erl"},{line,2579}]},{rabbit_variable_queue,queue_out,1,[{file,"src/rabbit_variable_queue.erl"},{line,1481}]},{rabbit_variable_queue,collect_by_predicate,3,[{file,"src/rabbit_variable_queue.erl"},{line,1701}]},{rabbit_variable_queue,fetch_by_predicate,4,[{file,"src/rabbit_variable_queue.erl"},{line,1660}]},{rabbit_variable_queue,fetchwhile,4,[{file,"src/rabbit_variable_queue.erl"},{line,679}]},{rabbit_priority_queue,fetchwhile,4,[{file,"src/rabbit_priority_queue.erl"},{line,291}]}]},{gen_server2,call,[<0.424.0>,{basic_consume,false,<0.959.0>,<0.955.0>,false,0,<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>,false,[],{'basic.consume_ok',<<"amq.ctag-MT52zGeW4OCFuYGYOKoXHw">>},<<"ipm">>},infinity]}},[{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,335}]},{rabbit_channel,'-basic_consume/8-fun-0-',9,[{file,"src/rabbit_channel.erl"},{line,1576}]},{rabbit_misc,with_exit_handler,2,[{file,"src/rabbit_misc.erl"},{line,509}]},{rabbit_channel,basic_consume,8,[{file,"src/rabbit_channel.erl"},{line,1573}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1295}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,567}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1056}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
rabbitmq_1 | 2019-08-20 17:44:03.773 [error] <0.959.0> CRASH REPORT Process <0.959.0> with 0 neighbours exited with reason: {{function_clause,[{rabbit_variable_queue,d,[{delta,5783552,299,0,5783552}],[{file,"src/rabbit_variable_queue.erl"},{line,1184}]},{rabbit_variable_queue,maybe_deltas_to_betas,2,[{file,"src/rabbit_variable_queue.erl"},{line,2656}]},{rabbit_variable_queue,fetch_from_q3,1,[{file,"src/rabbit_variable_queue.erl"},{line,2579}]},{rabbit_variable_queue,queue_out,1,[{file,"src/rabbit_variable_queue.erl"},{line,1481}]},{rabbit_variable_queue,collect_by_predicate,3,[{file,"src/rabbit_variable_queue.e..."},...]},...]},...} in gen_server2:call/3 line 335 in gen_server2:terminate/3 line 1172
rabbitmq_1 | 2019-08-20 17:44:03.783 [error] <0.952.0> Supervisor {<0.952.0>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.907.0>, <0.954.0>, <0.907.0>, <<"172.18.0.1:41954 -> 172.18.0.5:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"ipm">>,[ipm],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentica...">>,...},...], <0.925.0>, <0.955.0>) at <0.959.0> exit with reason {{function_clause,[{rabbit_variable_queue,d,[{delta,5783552,299,0,5783552}],[{file,"src/rabbit_variable_queue.erl"},{line,1184}]},{rabbit_variable_queue,maybe_deltas_to_betas,2,[{file,"src/rabbit_variable_queue.erl"},{line,2656}]},{rabbit_variable_queue,fetch_from_q3,1,[{file,"src/rabbit_variable_queue.erl"},{line,2579}]},{rabbit_variable_queue,queue_out,1,[{file,"src/rabbit_variable_queue.erl"},{line,1481}]},{rabbit_variable_queue,collect_by_predicate,3,[{file,"src/rabbit_variable_queue.e..."},...]},...]},...} in gen_server2:call/3 line 335 in context child_terminated
rabbitmq_1 | 2019-08-20 17:44:03.784 [error] <0.952.0> Supervisor {<0.952.0>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.907.0>, <0.954.0>, <0.907.0>, <<"172.18.0.1:41954 -> 172.18.0.5:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"ipm">>,[ipm],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"exchange_exchange_bindings">>,bool,true},{<<"connection.blocked">>,bool,true},{<<"authentica...">>,...},...], <0.925.0>, <0.955.0>) at <0.959.0> exit with reason reached_max_restart_intensity in context shutdown
rabbitmq_1 | 2019-08-20 17:44:03.835 [info] <0.913.0> closing AMQP connection <0.913.0> (172.18.0.1:41958 -> 172.18.0.5:5672, vhost: '/', user: 'ipm')
Oytun Tez
Aug 20, 2019, 2:09:33 PM8/20/19
to rabbitmq-users
I can confirm this error message even when I use management panel to get the messages from the queue – so definitely not caused by how I connect to RMQ from our Java application.
Oytun Tez
Aug 20, 2019, 3:40:11 PM8/20/19
to rabbitmq-users
As a no-Erlang person, I am now trying to use observer to take a look at the mnesia tables. It is not working out for me very well yet.
Can someone hold my hand here? I can see there is this message that RMQ server cannot use, but I can't see anything else about it.
Luke Bakken
Aug 20, 2019, 3:40:30 PM8/20/19
to rabbitmq-users
Hello,
What version of RabbitMQ and Erlang are you using, and on what operating system? Do you see anything interesting in the RabbitMQ logs prior to this error? Can you share details about your environment (clustered?) including the full log file?
Thanks -
Luke
Oytun Tez
Aug 20, 2019, 4:41:51 PM8/20/19
to rabbitmq-users
HI Luke,
RabbitMQ: 3.7.17
Erlang: 22.0.7
Env: Official RabbitMQ Docker container, not clustered, single server. I am using a Docker volume for mnesia directory (in production, we use AWS EFS volume for mnesia directory and I copied this local mnesia from production to debug the issue – I receive identical errors both on prod and local docker)
The logs are super clean, I am attaching the container logs from boot to crash. I didn't see anything weird here. I feel that this happens only with this specific message that is next in the queue. I have come down from 1xxx messages to 636 messages n the queue without issue, just to debug the issue and stuck with the message 636.
Oytun Tez
Aug 21, 2019, 9:18:56 AM8/21/19
to rabbitmq-users
Hi everyone,
Any ideas? This is a production issue.
Karl Nilsson
Aug 21, 2019, 9:53:25 AM8/21/19
to rabbitm...@googlegroups.com
It looks like somehow the delta calculation has become incorrect and when it hits this function that asserts the sum of the first two numbers in the delta structure are smaller than the last it fails.
It may be that restarting the queue (the easiest way would be to restart the broker) could "fix" it unless you have already tried that.
I'm interested in the steps that may have led to this scenario.
Cheers
Karl
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2d9df25e-590c-4964-9dd8-d5b614b40f00%40googlegroups.com.
Karl Nilsson
Pivotal/RabbitMQ
Karl Nilsson
Aug 21, 2019, 9:55:48 AM8/21/19
to rabbitm...@googlegroups.com
This assumes, of course, that your messages are durable and persistent. Else they'll be lost.
Karl Nilsson
Pivotal/RabbitMQ
Luke Bakken
Aug 21, 2019, 10:31:14 AM8/21/19
to rabbitmq-users
Hi Oytun,
There is no need to bump a discussion. Please remember that the people who assist you on this list do so in their free time, and we generally try to respond within a week to all questions. If you require urgent support, please see these pages:
Thanks,
Luke
Oytun Tez
Aug 21, 2019, 10:54:46 AM8/21/19
to rabbitm...@googlegroups.com
Hi Luke & Karl,
I am almost sure we tried rebooting several times (messages are persistent) – but let me retry and see what happens.
Excuse for the bump :( this error feels very low level, so I have my hands tied.
---
Oytun Tez
M O T A W O R D
The World's Fastest Human Translation Platform.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/20177eab-e261-44ec-83e7-598293468c78%40googlegroups.com.
Oytun Tez
Aug 21, 2019, 10:56:40 AM8/21/19
to rabbitm...@googlegroups.com
On my local, I did restart/rebuild the container multiple times by the way. That didn't change anything.
I can't really foresee what led to this situation, I have 90K events waiting to be processed in my queue right now. Locally, I filtered it down to a specific message (because I can't really pull any other messages from the queue because of this error). But this narrowing down will be difficult in production.
---
Oytun Tez
M O T A W O R D
The World's Fastest Human Translation Platform.
Oytun Tez
Aug 21, 2019, 2:20:42 PM8/21/19
to rabbitmq-users
Hi there,
I can confirm restarting did not change anything. Any ideas how to overcome this? I have no idea what the error means at all or what might have caused it.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
Luke Bakken
Aug 21, 2019, 2:51:24 PM8/21/19
to rabbitmq-users
Hi Oytun,
The RabbitMQ team discussed this issue on today's sync-up call. Unfortunately, we're mystified by what could have caused it. The closest I could find are the following -
Without a way to reproduce this scenario it will be difficult to fix. If you could send an archive of the RabbitMQ data directory privately I could at least see the error upon startup and may be able to provide you with a workaround. If you're concerned about data privacy PM me anyway ("reply privately to author") and we'll figure it out.
Thanks,
Luke
Oytun Tez
Aug 21, 2019, 2:55:14 PM8/21/19
to rabbitmq-users
Hi Luke!
Oh yes, I memorized those issue pages :D but couldn't figure it out... I am suspecting a producer may be sending a weird message, but I have no idea about erlang ecosystem, so couldn't get `observer` up and running to take a look at the mnesia database.
I am sending the mnesia directory now in private.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages