Upgrade rabbitmq cluster from 3.8.2 -> 3.8.8

[oleg9301 xz]

OS debian 10
rabbimq 3.8.2
erlang 22.1.8

In test cluster(10 test queue) after update all ok, on every step.
(apt-get install rabbitmq-server=3.8.8-1)

In prod(1082 queue) after update 1 node to 3.8.8 i have next log



2020-09-08 18:09:57.142 [info] <0.6384.6> queue 'N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb' in vhost 'fiscal-processing': terminating with function_clause in state follower
2020-09-08 18:09:57.144 [error] <0.6384.6> ** State machine 'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb' terminating
** Last event = {{timeout,tick},tick_timeout}
** When server state  = {follower,"ra_server_proc:format_status/2 crashed"}
** Reason for termination = error:function_clause
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{rabbit_fifo,messages_total,[{rabbit_fifo,{cfg,'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb',{resource,<<"fiscal-processing">>,queue,<<"N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb">>},{64000,64000},{rabbit_quorum_queue,dead_letter_publish,[undefined,undefined,{resource,<<"fiscal-processing">>,queue,<<"N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb">>}]},{rabbit_quorum_queue,become_leader,[{resource,<<"fiscal-processing">>,queue,<<"N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb">>}]},undefined,undefined,competing,undefined,undefined,undefined},#{},undefined,2,{0,{[],[]}},1,#{},{rabbit_fifo_index,#{},undefined,undefined},{0,{[],[]}},#{{<<"amq.ctag-gcn9UtWX-CDBYf2g8ZIlzw">>,<15774.18834.394>} => {consumer,#{ack => true,args => [],prefetch => 1,username => <<"Account:1">>},#{},0,1,0,simple_prefetch,auto,up}},{[{<<"amq.ctag-gcn9UtWX-CDBYf2g8ZIlzw">>,<15774.18834.394>},{<<"amq.ctag-gcn9UtWX-CDBYf2g8ZIlzw">>,<15771.3508.28>},{<<"amq.ctag-gcn9UtWX-CDBYf2g8ZIlzw">>,<15774.26042.7494>},{<<"amq.ctag-S5BAFGs9329UaE4z57uDyw">>,<15771.23476.6379>}],[{<<"amq.ctag-K1nBWDMkDKKSTeXaRWxUhQ">>,<15771.1198.6365>}]},{[],[]},0,0,[],0,0}],[{file,"src/rabbit_fifo.erl"},{line,809}]},{rabbit_fifo,eval_gc,3,[{file,"src/rabbit_fifo.erl"},{line,673}]},{rabbit_fifo,handle_aux,6,[{file,"src/rabbit_fifo.erl"},{line,663}]},{ra_machine,handle_aux,7,[{file,"src/ra_machine.erl"},{line,321}]},{ra_server,handle_aux,4,[{file,"src/ra_server.erl"},{line,1316}]},{ra_server_proc,handle_effect,5,[{file,"src/ra_server_proc.erl"},{line,1051}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{ra_server_proc,handle_effects,5,[{file,"src/ra_server_proc.erl"},{line,975}]}]
2020-09-08 18:09:57.145 [error] <0.6384.6> CRASH REPORT Process 'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb' with 0 neighbours crashed with reason: no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb',{resource,...},...},...}) line 809
2020-09-08 18:09:57.146 [error] <0.21685.0> Supervisor {<0.21685.0>,ra_server_sup} had child 'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb' started with ra_server_proc:start_link(#{await_condition_timeout => 30000,broadcast_time => 100,cluster_name => 'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb',...}) at <0.6384.6> exit with reason no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb',{resource,...},...},...}) line 809 in context child_terminated
2020-09-08 18:09:57.632 [info] <0.8589.6> queue 'N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb' in vhost 'fiscal-processing': detected a new leader {'fiscal-processing_N3AelNduf4uxgatx2mNdn5k290kJ2RRv05DE8UnKlM40lD1cSb','rabbit@nREPLACE-rmq1'} in term 32
2020-09-08 18:09:59.379 [info] <0.7366.6> queue 'fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM' in vhost 'fiscal-processing': terminating with function_clause in state follower
2020-09-08 18:09:59.380 [error] <0.7366.6> ** State machine 'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM' terminating
** Last event = {{timeout,tick},tick_timeout}
** When server state  = {follower,"ra_server_proc:format_status/2 crashed"}
** Reason for termination = error:function_clause
** Callback mode = [state_functions,state_enter]
** Stacktrace =
**  [{rabbit_fifo,messages_total,[{rabbit_fifo,{cfg,'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM',{resource,<<"fiscal-processing">>,queue,<<"fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM">>},{64000,64000},{rabbit_quorum_queue,dead_letter_publish,[undefined,undefined,{resource,<<"fiscal-processing">>,queue,<<"fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM">>}]},{rabbit_quorum_queue,become_leader,[{resource,<<"fiscal-processing">>,queue,<<"fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM">>}]},undefined,undefined,competing,undefined,undefined,undefined},#{13 => {172,{2,{basic_message,{resource,<<"fiscal-processing">>,exchange,<<"fiscal_processing.exchange">>},[<<"connector:5">>],{content,60,{'P_basic',<<"text/plain">>,undefined,[],2,0,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},<<184,0,10,116,101,120,116,47,112,108,97,105,110,0,0,0,0,2,0>>,rabbit_framing_amqp_0_9_1,[<<"61">>]},<<7,235,221,216,109,195,74,88,95,39,20,223,71,38,107,158>>,true}}}},13,14,{0,{[],[]}},13,#{},{rabbit_fifo_index,#{172 => ''},172,172},{0,{[],[]}},#{},{[],[]},{[],[]},2,0,[],2,1}],[{file,"src/rabbit_fifo.erl"},{line,809}]},{rabbit_fifo,eval_gc,3,[{file,"src/rabbit_fifo.erl"},{line,673}]},{rabbit_fifo,handle_aux,6,[{file,"src/rabbit_fifo.erl"},{line,663}]},{ra_machine,handle_aux,7,[{file,"src/ra_machine.erl"},{line,321}]},{ra_server,handle_aux,4,[{file,"src/ra_server.erl"},{line,1316}]},{ra_server_proc,handle_effect,5,[{file,"src/ra_server_proc.erl"},{line,1051}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{ra_server_proc,handle_effects,5,[{file,"src/ra_server_proc.erl"},{line,975}]}]
2020-09-08 18:09:59.381 [error] <0.7366.6> CRASH REPORT Process 'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM' with 0 neighbours crashed with reason: no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM',{resource,...},...},...}) line 809
2020-09-08 18:09:59.383 [error] <0.19993.0> Supervisor {<0.19993.0>,ra_server_sup} had child 'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM' started with ra_server_proc:start_link(#{await_condition_timeout => 30000,broadcast_time => 100,cluster_name => 'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM',...}) at <0.7366.6> exit with reason no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_fAfAn6bnb9tPsca1mNSjEd32smCmEVxCMbM1sj3mU3s2yF9wKM',{resource,...},...},...}) line 809 in context child_terminated

[oleg9301 xz]

load config from prod to test
prod) curl -H "Accept:application/json" -u admin:password "http://localhost:15672/api/definitions"

test) curl -i -u admin:password -H "content-type:application/json" -X POST --data @/home/user/backup.json http://localhost:15672/api/definitions

after that i upgrade test cluster from 3.8.2 -> 3.8.8 and there is no problem.

вторник, 8 сентября 2020 г. в 18:23:23 UTC+3, oleg9301 xz:

[oleg9301 xz]

I install 3 new server to production cluster,

OS debian 10 with latest uptade.

rabbitmq 3.8.2

Erlang 22.3.4.9 latest 22.x in repo

join to old 3 nodes(6 nodes)

then i migrate queue from old nodes to new nodes.

after that i shutdown 3 old nodes.

But when i try to update rabbitmq on new node(Erlang 22.3.4.9) I have the same error:

2020-09-10 12:07:25.895 [error] <0.18221.7> CRASH REPORT Process 'fiscal-processing_S6a055D7VaF0ebvLEJgw81CMkL3x4fKGpNmElCdvDNu77spjB8' with 0 neighbours crashed with reason: no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_S6a055D7VaF0ebvLEJgw81CMkL3x4fKGpNmElCdvDNu77spjB8',{resource,...},...},...}) line 809
2020-09-10 12:07:25.896 [error] <0.12467.0> Supervisor {<0.12467.0>,ra_server_sup} had child 'fiscal-processing_S6a055D7VaF0ebvLEJgw81CMkL3x4fKGpNmElCdvDNu77spjB8' started with ra_server_proc:start_link(#{await_condition_timeout => 30000,broadcast_time => 100,cluster_name => 'fiscal-processing_S6a055D7VaF0ebvLEJgw81CMkL3x4fKGpNmElCdvDNu77spjB8',...}) at <0.18221.7> exit with reason no function clause matching rabbit_fifo:messages_total({rabbit_fifo,{cfg,'fiscal-processing_S6a055D7VaF0ebvLEJgw81CMkL3x4fKGpNmElCdvDNu77spjB8',{resource,...},...},...}) line 809 in context child_terminated

вторник, 8 сентября 2020 г. в 19:47:38 UTC+3, oleg9301 xz:

[oleg9301 xz]

020-09-10 12:12:54.919 [info] <0.7297.13> queue 'kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m' in vhost 'fiscal-processing': terminating with function_clause in state leader
2020-09-10 12:12:54.922 [error] <0.7297.13> ** State machine 'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m' terminating

** Last event = {{timeout,tick},tick_timeout}

** When server state  = {leader,"ra_server_proc:format_status/2 crashed"}

** Reason for termination = error:function_clause

** Callback modules = [ra_server_proc]

** Callback mode = [state_functions,state_enter]
** Stacktrace =

**  [{rabbit_fifo,messages_ready,[{rabbit_fifo,{cfg,'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m',{resource,<<"fiscal-processing">>,queue,<<"kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m">>},{64000,64000},{rabbit_quorum_queue,dead_letter_publish,[undefined,undefined,{resource,<<"fiscal-processing">>,queue,<<"kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m">>}]},{rabbit_quorum_queue,become_leader,[{resource,<<"fiscal-processing">>,queue,<<"kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m">>}]},undefined,undefined,competing,undefined,undefined,undefined},#{},undefined,2,{0,{[],[]}},1,#{},{rabbit_fifo_index,#{},undefined,undefined},{0,{[],[]}},#{{<<"amq.ctag-vTGQ59VIO-QhGfqJqetN9w">>,<15868.23314.165>} => {consumer,#{ack => true,args => [{<<"x-priority">>,signedint,1}],prefetch => 1,username => <<"Account:1">>},#{},0,1,0,simple_prefetch,auto,up}},{[{<<"amq.ctag-vTGQ59VIO-QhGfqJqetN9w">>,<15868.23314.165>},{<<"amq.ctag-VxoJ2XNx35aYpydvwsBiVA">>,<15868.8630.103>},{<<"amq.ctag-KpyjuL51uQejJYoj-5I7Fw">>,<15868.8637.59>},{<<"amq.ctag-KpyjuL51uQejJYoj-5I7Fw">>,<15868.22512.58>},{<<"amq.ctag-KpyjuL51uQejJYoj-5I7Fw">>,<18765.26038.104>},{<<"amq.ctag-L7K-NtRvlp7YrpPdSJx7jg">>,<18766.9559.32>},{<<"amq.ctag-L7K-NtRvlp7YrpPdSJx7jg">>,<18765.16883.22>},{<<"amq.ctag-XRy_IZWj133juUhS33itbQ">>,<18765.10016.22>},{<<"amq.ctag-RhV2wnqM0Zezt3wL2XkLbw">>,<18765.3185.22>},{<<"amq.ctag-S8sZXOUz6VCAeQtjJlpjqw">>,<18765.29165.21>},{<<"amq.ctag-S9_xrD-za5WRKq46ccP88A">>,<18765.21981.21>},{<<"amq.ctag-40kbGKPUXs9JQln073EIIA">>,<18765.15312.21>},{<<"amq.ctag-hmjtoqufNK5C0yVfcuBK5Q">>,<18765.8196.21>},{<<"amq.ctag-fcqf34OHsYThavH_EiaA5Q">>,<18765.77.21>},{<<"amq.ctag-AMFACtoSwNICSQAfzodaNw">>,<18765.22979.20>},{<<"amq.ctag--TtbGLqBxQGXJykIrYAx4A">>,<18765.14672.20>},{<<"amq.ctag-0c4jHiZC_ifaJcRjHFEq6g">>,<18765.7545.20>},{<<"amq.ctag-ZHHqbRFGekZ81fglP7XTHA">>,<18765.171.20>},{<<"amq.ctag-hOrtlyusxpQ2k7pxrwCbiA">>,<18765.25184.19>},{<<"amq.ctag-WSKInhryllUwAMU_jXPMhw">>,<18765.18167.19>},{<<"amq.ctag-WQRydM2xiI_AstwPsPT-Hw">>,<18765.10656.19>},{<<"amq.ctag-WPdlyowH-vgUtoRqv2ZzVg">>,<18765.3205.19>},{<<"amq.ctag-w7Nmtj1ozbGUBmrjLFcfrA">>,<18765.28817.18>},{<<"amq.ctag-a1H-04jX9SOkHMUkbsSGzQ">>,<18765.21694.18>},{<<"amq.ctag-34DpqDOq2tsV5tNHmXtsWA">>,<18765.14536.18>},{<<"amq.ctag-SzkHJWuL2A1TB8-OvIosKg">>,<18765.7419.18>},{<<"amq.ctag-WaT6ZXwihsaPWW-ULFo7Nw">>,<18765.151.18>},{<<"amq.ctag-nD6eADYZPF3cbCXSlznVXQ">>,<18765.25586.17>},{<<"amq.ctag-hGcrAjpCNizWLMFbHHgS1g">>,<18765.17897.17>},{<<"amq.ctag-xDX9B7tAp0LVXsqPaYCGbg">>,<18765.10660.17>},{<<"amq.ctag-Zd1f4aJTboK9QjmPpqgumA">>,<18765.2148.17>},{<<"amq.ctag-cEEusroVtDrv-r_V50gULg">>,<18765.27896.16>},{<<"amq.ctag-3zo-EExE4bLA2WaM7sX2tA">>,<18765.19574.16>},{<<"amq.ctag-FIL6FfnAzRFt0oZoFDlC_Q">>,<18765.12559.16>},{<<"amq.ctag-wP3si2Cel2yP2b_gj_8OBg">>,<18765.2430.16>},{<<"amq.ctag-aQnjklMeHBEGyrobVgy6bg">>,<18765.28643.15>},{<<"amq.ctag-SHJLRMYKe_UD7hlL4uMVYg">>,<18765.21211.15>},{<<"amq.ctag-H0ENzMnqV5NH769AZPwtNA">>,<18765.12147.15>},{<<"amq.ctag-tes6SFp9DdXGbWxxpJUFSw">>,<18765.4824.15>},{<<"amq.ctag-mQTqeNK3BxgFc7CU1UQnww">>,<18765.26972.14>},{<<"amq.ctag-bVvWGUjsNQOlARmTHtaE9g">>,<18765.18000.14>},{<<"amq.ctag-oiOX6Gs7XnvXZN79z-gJ6g">>,<18765.10871.14>},{<<"amq.ctag-iFcwQZIaxO6ZJWmrqCnNTw">>,<18765.3732.14>},{<<"amq.ctag-W8bDISCrPZO1V5JQmYOHfg">>,<18765.29283.13>},{<<"amq.ctag-XRbSydZE_4ZZeuLUAQGefA">>,<18765.21817.13>},{<<"amq.ctag-auYbOgxDon5WgRcQqlr1fA">>,<18765.14449.13>},{<<"amq.ctag-xeeJ1KLU2K3OfzUSApKA_Q">>,<18765.7281.13>},...],...},...}],...},...]
2020-09-10 12:12:54.923 [error] <0.7297.13> CRASH REPORT Process 'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m' with 0 neighbours crashed with reason: no function clause matching rabbit_fifo:messages_ready({rabbit_fifo,{cfg,'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m',{resource,...},...},...}) line 801
2020-09-10 12:12:54.924 [error] <0.13866.0> Supervisor {<0.13866.0>,ra_server_sup} had child 'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m' started with ra_server_proc:start_link(#{await_condition_timeout => 30000,broadcast_time => 100,cluster_name => 'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m',...}) at <0.7297.13> exit with reason no function clause matching rabbit_fifo:messages_ready({rabbit_fifo,{cfg,'fiscal-processing_kfUnB03ltwx6RS1CGymPgccfGmTMgMuys9pTmuVNvKwLa1D06m',{resource,...},...},...}) line 801 in context child_terminated

четверг, 10 сентября 2020 г. в 12:08:20 UTC+3, oleg9301 xz:

[Karl Nilsson]

Does this error occur on the node you upgraded? Does it persist after all nodes have been upgraded?

---

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of oleg9301 xz <oleg...@gmail.com>
Sent: 08 September 2020 4:23 PM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: [rabbitmq-users] Upgrade rabbitmq cluster from 3.8.2 -> 3.8.8

 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b7e4b877-8daa-4fa4-93da-b26371229859n%40googlegroups.com.

[oleg9301 xz]

It does persist after all node upgrade.

i made another 3 VM with new cluster(3.8.8), and did backup restore definitions. And with script help carry messages from old to new cluster.

понедельник, 14 сентября 2020 г. в 19:15:28 UTC+3, Karl Nilsson: