bgpd will crash after configuring BMP

284 views
Skip to first unread message

Guan-Yu Shen

unread,
Jun 27, 2022, 2:15:59 AM6/27/22
to sonicproject

hello all,
I configure bgp BMP mirror and remove router bgp in vtysh, the bgpd will crash immediately, does anyone have idea about this issue? The FRR configuration and log is as follows,

---------------------------------------------------------------------------------

All daemons are running

Hello, this is FRRouting (version 7.5.1-sonic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

sonic#
sonic#
sonic#
sonic#
sonic#
sonic# show running-config
Building configuration...

Current configuration:
!
frr version 7.5.1-sonic
frr defaults traditional
hostname sonic
log syslog
log facility local4
log commands
no service integrated-vtysh-config
!
debug bgp neighbor-events
debug bgp graceful-restart
!
password zebra
enable password zebra
!
line vty
!
end
sonic#
sonic#
sonic#
sonic# configure
sonic(config)# router bgp 100
sonic(config-router)# bgp router-id 1.1.1.1
sonic(config-router)# bmp mirror buffer-limit 4294967294
sonic(config-router)# bmp targets server1
sonic(config-bgp-bmp)# bmp connect server-1 port 65535 min-retry 100 max-retry 200
sonic(config-bgp-bmp)# bmp listener 0.0.0.0 port 65535
sonic(config-bgp-bmp)# ip access-list test
sonic(config-bgp-bmp)# ipv6 access-list test-ipv6
sonic(config-bgp-bmp)# bmp stats interval 100
sonic(config-bgp-bmp)# bmp monitor ipv4 unicast pre-policy
sonic(config-bgp-bmp)# bmp mirror
sonic(config-bgp-bmp)# no bmp mirror
sonic(config-bgp-bmp)# bmp mirror
sonic(config-bgp-bmp)#
sonic(config-bgp-bmp)#
sonic(config-bgp-bmp)# exit
sonic(config-router)# exit
sonic(config)# no router bgp
sonic(config)# exit
Warning: closing connection to bgpd because of an I/O error!
Warning: connecting to bgpd...failed!
sonic#

---------------------------------------------------------------------------------

Jun 27 05:41:04.241120 sonic NOTICE bgp#bgpd[52]: vty[??]@> enable
Jun 27 05:55:31.875878 sonic NOTICE bgp#bgpd[52]: vty[??]@# do write terminal
Jun 27 05:56:11.148492 sonic NOTICE bgp#bgpd[52]: vty[??]@# configure
Jun 27 05:56:12.468815 sonic NOTICE bgp#bgpd[52]: vty[??]@(config)# router bgp 100
Jun 27 05:56:12.470309 sonic DEBUG bgp#bgpd[52]: bgp_peer_gr_init called ..
Jun 27 05:56:12.470309 sonic DEBUG bgp#bgpd[52]: [BGP_GR] Peer state changed --to--> : 4 : !
Jun 27 05:56:12.470309 sonic DEBUG bgp#bgpd[52]: bgp_global_gr_init called ..
Jun 27 05:56:21.652441 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-router)# bgp router-id 1.1.1.1
Jun 27 05:56:27.869031 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-router)# bmp mirror buffer-limit 4294967294
Jun 27 05:56:33.877584 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-router)# bmp targets server1
Jun 27 05:56:38.877968 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp connect server-1 port 65535 min-retry 100 max-retry 200
Jun 27 05:56:38.994115 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:56:42.885644 sonic WARNING bgp#bgpd[52]: message repeated 17 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:56:42.885644 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp listener 0.0.0.0 port 65535
Jun 27 05:56:42.976698 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:56:47.040753 sonic WARNING bgp#bgpd[52]: message repeated 18 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:56:47.040753 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# ip access-list test
Jun 27 05:56:47.093990 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:56:53.510420 sonic WARNING bgp#bgpd[52]: message repeated 29 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:56:53.510420 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# ipv6 access-list test-ipv6
Jun 27 05:56:53.536392 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:56:57.471475 sonic WARNING bgp#bgpd[52]: message repeated 18 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:56:57.471475 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp stats interval 100
Jun 27 05:56:57.637019 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:01.543106 sonic WARNING bgp#bgpd[52]: message repeated 18 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:01.543106 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp monitor ipv4 unicast pre-policy
Jun 27 05:57:01.718300 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:07.223841 sonic WARNING bgp#bgpd[52]: message repeated 25 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:07.223841 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp mirror
Jun 27 05:57:07.316962 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:29.967605 sonic WARNING bgp#bgpd[52]: message repeated 105 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:29.967605 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# no bmp mirror
Jun 27 05:57:30.073900 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:35.496000 sonic WARNING bgp#bgpd[52]: message repeated 25 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:35.496000 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# bmp mirror
Jun 27 05:57:35.641212 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:47.040647 sonic WARNING bgp#bgpd[52]: message repeated 53 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:47.040647 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-bgp-bmp)# exit
Jun 27 05:57:47.193856 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:50.696427 sonic WARNING bgp#bgpd[52]: message repeated 16 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:50.696427 sonic NOTICE bgp#bgpd[52]: vty[??]@(config-router)# exit
Jun 27 05:57:50.696854 sonic NOTICE bgp#bgpd[52]: vty[??]@(config)# end
Jun 27 05:57:50.697433 sonic NOTICE bgp#bgpd[52]: vty[??]@# configure
Jun 27 05:57:50.831208 sonic WARNING bgp#bgpd[52]: bmp[server-1]: hostname resolution failed: Domain name not found
Jun 27 05:57:54.841770 sonic WARNING bgp#bgpd[52]: message repeated 18 times: [ bmp[server-1]: hostname resolution failed: Domain name not found]
Jun 27 05:57:54.841770 sonic NOTICE bgp#bgpd[52]: vty[??]@(config)# no router bgp
Jun 27 05:57:54.842501 sonic DEBUG bgp#bgpd[52]: bgp_fsm_change_status : vrf default(0), Status: Deleted established_peers 0
Jun 27 05:57:54.842501 sonic DEBUG bgp#bgpd[52]: Static announcement went from Idle to Deleted
Jun 27 05:57:54.878954 sonic CRIT bgp#BGP[52]: Received signal 11 at 1656309474 (si_addr 0x30, PC 0x7f504991f6d6); aborting...
Jun 27 05:57:54.880575 sonic CRIT bgp#BGP[52]: zlog_signal+0xdd 7f504eca3e4d 7ffdd0849a40 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f504ec49000)
Jun 27 05:57:54.881884 sonic CRIT bgp#BGP[52]: PBKDF2_SHA256+0x484 7f504eccaaa4 7ffdd0849b70 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f504ec49000)
Jun 27 05:57:54.882985 sonic CRIT bgp#BGP[52]: __restore_rt+0 7f504dce40e0 7ffdd0849cc0 /lib/x86_64-linux-gnu/libpthread.so.0 (mapped at 0x7f504dcd3000)
Jun 27 05:57:54.883495 sonic CRIT bgp#BGP[52]: ---- signal ----
Jun 27 05:57:54.883495 sonic CRIT bgp#BGP[52]: _init+0x3876 7f504991f6d6 7ffdd084a270 /usr/lib/x86_64-linux-gnu/frr/modules/bgpd_bmp.so (mapped at 0x7f5049919000)
Jun 27 05:57:54.884860 sonic CRIT bgp#BGP[52]: thread_call+0x60 7f504ecda060 7ffdd084a2f0 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f504ec49000)
Jun 27 05:57:54.886084 sonic CRIT bgp#BGP[52]: frr_run+0xe0 7f504eca2760 7ffdd084a490 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f504ec49000)
Jun 27 05:57:54.886477 sonic CRIT bgp#BGP[52]: main+0x2f4 55e5c4596a04 7ffdd084a6b0 /usr/lib/frr/bgpd (mapped at 0x55e5c455c000)
Jun 27 05:57:54.887524 sonic CRIT bgp#BGP[52]: __libc_start_main+0xf1 7f504d9542e1 7ffdd084a720 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7f504d934000)
Jun 27 05:57:54.887922 sonic CRIT bgp#BGP[52]: _start+0x2a 55e5c4598cba 7ffdd084a7e0 /usr/lib/frr/bgpd (mapped at 0x55e5c455c000)
Jun 27 05:57:54.888988 sonic CRIT bgp#BGP[52]: ? 0 7ffdd084a7e8
Jun 27 05:57:54.889034 sonic CRIT bgp#BGP[52]: in thread bmp_stats scheduled from bgpd/bgp_bmp.c:1298
Jun 27 05:57:54.889310 sonic INFO bgp#supervisord: bgpd core_handler: showing active allocations in memory group libfrr
Jun 27 05:57:54.889599 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Buffer : 4 * 24
Jun 27 05:57:54.889599 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Host config : 5 * (variably sized)
Jun 27 05:57:54.889599 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Command Tokens : 11561 * 72
Jun 27 05:57:54.889792 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Command Token Text : 8275 * (variably sized)
Jun 27 05:57:54.889792 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Command Token Help : 8275 * (variably sized)
Jun 27 05:57:54.889792 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Command Argument Name : 1890 * (variably sized)
Jun 27 05:57:54.889842 sonic INFO bgp#supervisord: bgpd core_handler: memstats: RCU thread : 3 * 128
Jun 27 05:57:54.889842 sonic INFO bgp#supervisord: bgpd core_handler: memstats: RCU sequence barrier : 1 * 32
Jun 27 05:57:54.889842 sonic INFO bgp#supervisord: bgpd core_handler: memstats: FRR POSIX Thread : 6 * (variably sized)
Jun 27 05:57:54.889883 sonic INFO bgp#supervisord: bgpd core_handler: memstats: POSIX sync primitives : 6 * (variably sized)
Jun 27 05:57:54.889883 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Graph : 40 * 8
Jun 27 05:57:54.889914 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Graph Node : 13630 * 32
Jun 27 05:57:54.889914 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Hash : 523 * (variably sized)
Jun 27 05:57:54.889958 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Hash Bucket : 2225 * 32
Jun 27 05:57:54.889958 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Hash Index : 262 * (variably sized)
Jun 27 05:57:54.889984 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Interface : 43 * 272
Jun 27 05:57:54.890001 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Connected : 2 * 48
Jun 27 05:57:54.890001 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Link List : 114 * 40
Jun 27 05:57:54.890192 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Link Node : 338 * 24
Jun 27 05:57:54.890258 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Temporary memory : 2 * (variably sized)
Jun 27 05:57:54.890258 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Module loading name : 2 * (variably sized)
Jun 27 05:57:54.890258 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Northbound Node : 126 * 1184
Jun 27 05:57:54.890283 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Northbound Configuration : 2 * 16
Jun 27 05:57:54.890283 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Prefix : 2 * 48
Jun 27 05:57:54.890402 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Privilege information : 3 * (variably sized)
Jun 27 05:57:54.890402 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Skip List : 2 * 56
Jun 27 05:57:54.890402 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Skip Node : 4 * 160
Jun 27 05:57:54.890402 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Stream : 5 * (variably sized)
Jun 27 05:57:54.890430 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Route table : 21 * 56
Jun 27 05:57:54.890430 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Thread : 10 * 168
Jun 27 05:57:54.890470 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Thread master : 15 * (variably sized)
Jun 27 05:57:54.890470 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Thread Poll Info : 8 * 8388608
Jun 27 05:57:54.890543 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Thread stats : 17 * 72
Jun 27 05:57:54.890543 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Typed-hash bucket : 1 * (variably sized)
Jun 27 05:57:54.890543 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Typed-heap array : 1 * 576
Jun 27 05:57:54.890543 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Vector : 27347 * 16
Jun 27 05:57:54.890543 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Vector index : 27347 * (variably sized)
Jun 27 05:57:54.890617 sonic INFO bgp#supervisord: bgpd core_handler: memstats: VRF : 1 * 200
Jun 27 05:57:54.890617 sonic INFO bgp#supervisord: bgpd core_handler: memstats: VRF bit-map : 3 * 8
Jun 27 05:57:54.890617 sonic INFO bgp#supervisord: bgpd core_handler: memstats: VTY : 2 * (variably sized)
Jun 27 05:57:54.890640 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Work queue : 2 * 152
Jun 27 05:57:54.890640 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Work queue name string : 2 * (variably sized)
Jun 27 05:57:54.890640 sonic INFO bgp#supervisord: bgpd core_handler: memstats: YANG module : 4 * 48
Jun 27 05:57:54.890666 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Zclient : 2 * 3456
Jun 27 05:57:54.890812 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Redistribution instance IDs : 6 * 2
Jun 27 05:57:54.890812 sonic INFO bgp#supervisord: bgpd core_handler: memstats: log thread-local buffer : 3 * 9752
Jun 27 05:57:54.890812 sonic INFO bgp#supervisord: bgpd core_handler: showing active allocations in memory group logging subsystem
Jun 27 05:57:54.890812 sonic INFO bgp#supervisord: bgpd core_handler: memstats: syslog target : 2 * 56
Jun 27 05:57:54.890812 sonic INFO bgp#supervisord: bgpd core_handler: showing active allocations in memory group bgpd
Jun 27 05:57:54.890860 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Mac Hash Entry : 9 * 16
Jun 27 05:57:54.890860 sonic INFO bgp#supervisord: bgpd core_handler: memstats: Mac Hash Entry Interface String: 43 * (variably sized)
Jun 27 05:57:54.890877 sonic INFO bgp#supervisord: bgpd core_handler: memstats: BGP table : 21 * 40
Jun 27 05:57:54.890877 sonic INFO bgp#supervisord: bgpd core_handler: memstats: community-list handler : 1 * 120
Jun 27 05:57:54.890893 sonic INFO bgp#supervisord: bgpd core_handler: memstats: BGP EVPN Multihoming Information: 1 * 40
Jun 27 05:57:54.890893 sonic INFO bgp#supervisord: bgpd core_handler: showing active allocations in memory group rfapi
Jun 27 05:57:54.890909 sonic INFO bgp#supervisord: bgpd core_handler: showing active allocations in memory group BMP (BGP Monitoring Protocol)
Jun 27 05:57:54.890946 sonic INFO bgp#supervisord: bgpd core_handler: memstats: BMP listener : 1 * 64
Jun 27 05:57:56.999614 sonic INFO bgp#supervisord 2022-06-27 05:57:56,075 INFO exited: bgpd (terminated by SIGABRT (core dumped); not expected)
Jun 27 05:58:11.520254 sonic ERR monit[639]: 'bgpd' process is not running

flyse...@gmail.com

unread,
Jun 27, 2022, 2:36:32 AM6/27/22
to sonicproject
Do you add extra option for bgpd for enabling the BMP? 
You can check the following steps

1. enter bgp container
docker exec -it bgp bash

2. add extra option in /etc/frr/daemon like the following
bgpd_options=" -A 127.0.0.1 -M snmp -M bmp" 

3.restart the frr service
service frr restart

After that you can use BMP configuration in FRR.

Regards,
Jimi


dogdog...@gmail.com 在 2022年6月27日 星期一下午2:15:59 [UTC+8] 的信中寫道：

Guan-Yu Shen

unread,
Jun 27, 2022, 3:04:37 AM6/27/22
to sonicproject
hi  Jimi,
     thanks for your replying, I have checked bgpd extra option,
     the BMP option is enabled. Btw my frr is version 7.5.1-sonic.

root@sonic:/etc/frr#
root@sonic:/etc/frr#
root@sonic:/etc/frr#
root@sonic:/etc/frr# ps aux | grep bgpd
frr         52  0.0  0.0 472436 15480 pts/0    Sl   06:27   0:00 /usr/lib/frr/bgpd -A 127.0.0.1 -M snmp -M bmp
root       127  0.0  0.0  11108   936 pts/1    S+   06:59   0:00 grep bgpd


Best Regards,
Kevin



flyse...@gmail.com 在 2022年6月27日 星期一下午2:36:32 [UTC+8] 的信中寫道：

flyse...@gmail.com

unread,
Jun 27, 2022, 3:08:51 AM6/27/22
to sonicproject
Hi Kevin, 
I remember that 202012 version can be worked, but there's some issue for 202006 version.  
You can check if that is work for later version. 

Regards,
Jimi

dogdog...@gmail.com 在 2022年6月27日 星期一下午3:04:37 [UTC+8] 的信中寫道：

Guan-Yu Shen

unread,
Jun 27, 2022, 11:34:52 PM6/27/22
to sonicproject
hi Jimi,
          my frr version 7.5.1  is released after 202012.
          I traced the code and found that bgp is crashed becasue of memory overflow.
          After fixing some problems, the bgpd still crash after configuring BMP option. 
   

flyse...@gmail.com 在 2022年6月27日 星期一下午3:08:51 [UTC+8] 的信中寫道：
Reply all
Reply to author
Forward
0 new messages