IS-IS & Segment Routing test report

[Olivier Dugeon]

Dear all,

I progress in setting up Segment Routing under SONIC. I'm using latest master image on a Wedge-32bf which comes with Barefoot SDK version 9.9 (with IS-IS trap support and MPLS profile for tofino1). I activated IS-IS with Segment Routing configuration and connected our Wedge-32bf to a Juniper MX and Cisco ASR routers. IS-IS collects correctly Segment Routing information and send its prefix-SID. Juniper and Cisco routers setup MPLS table accordingly and Sonic box too. I could verify with 'ip route' and 'ip -M route' that prefix-SID are correctly configured. I could ping a neighbor router and verify with tcpdump that packets are correctly labeled.

 But I got a last problem. Ping the Loopback of the Sonic box from a neighbor router doesn't work and I can't capture any packet with tcpdump.

Thus, I check the configuration on the tofino (with bfshell; bf_switchd; show mpls all) and discover that the MPLS table is empty. I check the log and didn't find any errors, but no trace of MPLS label configuration.

After investigating, it seems that MPLS labels and IP prefix with MPLS encapsulation i.e. prefix-SID are not propagated from FRR to fpmsyncd.

I browse both fpmsyncd and FRR fpm code and it seems that the problem is on FRR side. Indeed, I launch both zebra and fpmsyncd with debug log, and only IP route (AF_INET or AF_INET6) are sent by FRR to fpmsyncd through fpm. Fpmsyncd wait for AF_MPLS message, but not one is coming. It is also not clear if IP prefix sent by FRR contains the prefix SID (i.e. MPLS label) or not.

As MPLS is now supported in SONiC, how it is expected to work ? How MPLS labels are expected to be configured in the data plane (i.e. on the ASIC) through SAI ? Is there a special command ? Is fpmsyncd ready, but a patch is needed on FRR side ? Note, that I also upgrade to FRR 8.2.2 (following this PR https://github.com/Azure/sonic-buildimage/pull/10691) and the result is similar.

Thanks for your help.

Regards

Olivier

[Olivier Dugeon]

Dear all,

I finally success to setup IS-IS with Segment Routing within SONiC. First of all, I would like to warmly thank the Intel/Barefoot team who helped me a lot to achieve this result.

The configuration is as below:

[admin@sonic ~]: show version

SONiC Software Version: SONiC.master.95332-dirty-20220502.215123
Distribution: Debian 11.3
Kernel: 5.10.0-8-2-amd64
Build commit: 4ec3af86a
Build date: Mon May  2 21:58:15 UTC 2022
Built by: AzDevOps@sonic-build-workers-001H5W

Platform: x86_64-accton_wedge100bf_32x-r0
HwSKU: montara
ASIC: barefoot
ASIC Count: 1
Serial Number: AJ19015468
Model Number: N0PCH7632002S

On top of that, I install libopenSAI recompiled from P4 SDE after applying a patch from Intel (to overcome a problem with duplicate entry per same nexthop). I also need to patch fpmsyncd (inspired from https://github.com/Azure/sonic-swss/pull/1765) in order to use dplane_fpm_nl module from Zebra as fpm module is not propagate MPLS entries. Finally, I modified supervisord configuration of the bgp docker in order to:

1) launch zebra with '-M dplane_fpm_nl' option instead of '-M fpm' option,

2) launch fpmsyncd with 'l dplane' (new option)

3) start isisd like for ospfd.

Regarding zebra configuration, you just need to add 'fpm connection 127.0.0.1' to let the dplane_fpm_nl connects to fpmsyncd. For isisd, I just add standard configuration with segment-routing as follow:

interface Ethernet126
ip address 10.194.77.194/31
ip router isis test
isis circuit-type level-2-only
isis hello-multiplier 3
isis network point-to-point
link-params
 enable
exit-link-params
exit
!
interface Loopback0
ip address 10.194.77.238/32
ip router isis test
isis circuit-type level-2-only
isis passive
exit
!
router isis test
is-type level-2-only
net 47.0023.0000.0000.0000.0000.0000.0000.0ac2.4dee.00
lsp-timers gen-interval 2 refresh-interval 900 max-lifetime 1200
topology ipv6-unicast
mpls-te on
mpls-te router-address 10.194.77.238
segment-routing on
segment-routing global-block 50000 59999
segment-routing node-msd 8
segment-routing prefix 10.194.77.238/32 index 238 no-php-flag
exit

The device is connected to a network formed with Cisco and Juniper routers all running IS-IS with Segment-Routing.

And voilà:

sonic# sh mpls table
Inbound Label  Type        Nexthop        Outbound Label   
----------------------------------------------------------
15000          SR (IS-IS)  10.194.77.186  implicit-null    
15001          SR (IS-IS)  10.194.77.195  implicit-null    
50235          SR (IS-IS)  10.194.77.195  40235            
50238          SR (IS-IS)  lo             -                
50241          SR (IS-IS)  10.194.77.186  30241            
50242          SR (IS-IS)  10.194.77.186  30242            
50244          SR (IS-IS)  10.194.77.186  30244            
50246          SR (IS-IS)  10.194.77.186  30246            
50247          SR (IS-IS)  10.194.77.186  30247            
50252          SR (IS-IS)  10.194.77.186  30252            
50253          SR (IS-IS)  10.194.77.186  30253            
50254          SR (IS-IS)  10.194.77.186  implicit-null   

[admin@sonic ~]: ip -M route
15000 via inet 10.194.77.186 dev Ethernet124 proto isis  
15001 via inet 10.194.77.195 dev Ethernet126 proto isis  
50235 as to 40235 via inet 10.194.77.195 dev Ethernet126 proto isis  
50238 dev lo proto isis  
50241 as to 30241 via inet 10.194.77.186 dev Ethernet124 proto isis  
50242 as to 30242 via inet 10.194.77.186 dev Ethernet124 proto isis  
50244 as to 30244 via inet 10.194.77.186 dev Ethernet124 proto isis  
50246 as to 30246 via inet 10.194.77.186 dev Ethernet124 proto isis  
50247 as to 30247 via inet 10.194.77.186 dev Ethernet124 proto isis  
50252 as to 30252 via inet 10.194.77.186 dev Ethernet124 proto isis  
50253 as to 30253 via inet 10.194.77.186 dev Ethernet124 proto isis  
50254 via inet 10.194.77.186 dev Ethernet124 proto isis

bf_switch:0> show mpls all

 =====================================================================================================================
 oid     label       packet_action  nexthop_rif_handle  num_pop  pop_ttl_mode   pop_qos_mode   counter_handle  id      
 =====================================================================================================================
 1       15000       FORWARD        nexthop.4           1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 2       50254       FORWARD        nexthop.4           1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 3       50235       FORWARD        nexthop.6           1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 4       15001       FORWARD        nexthop.3           1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 5       50253       FORWARD        nexthop.20          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 6       50241       FORWARD        nexthop.14          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 7       50244       FORWARD        nexthop.16          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 8       50246       FORWARD        nexthop.17          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 9       50247       FORWARD        nexthop.18          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 10      50242       FORWARD        nexthop.15          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 11      50252       FORWARD        nexthop.19          1        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS    
 12      50238       TRAP           0                   0        UNIFORM_MODEL  UNIFORM_MODEL  0               PKTS

bf_switch:0> show nexthop handle 19
   nexthop_handle: 0x21000000000013
   Attributes:
     internal_object: False
     type: MPLS
     handle: rif.12
     dest_ip: 10.194.77.186
     rw_type: NONE
     tunnel_vni: 0
     srv6_sidlist_id: 0
     tunnel_dest_ip_handle: 0
     tunnel_dest_ip_index: 0
     labelop: SWAP
     mpls_encap_qos_mode: UNIFORM_MODEL
     mpls_encap_exp: 0
     mpls_encap_ttl_mode: UNIFORM_MODEL
     mpls_encap_ttl: 0
     mac_address: 00:00:00:00:00:00
     port_lag_handle: 0
     labelstack:
       element_type: UINT32
       list_element(s):
         30252

I just need to add manually the entry 50238 in the Tofino MPLS table as orchagent filter configuration on Loopback interface. This entry is necessary because the prefix-sid 10.194.77.238/32 with index 238 is advertise with the 'no-php' flag i.e. the router doesn't request Penultimate Hop Popping thus, direct neighbour routers delivered labelled packets and SONiC will perform the pop. Of course, if 'no-php' flag is set there is no need for this configuration.

MPLS traffic are correctly forwarded by the Tofino as well as labels are PUSH or POP accordingly to the routing table.

Next steps will be to industrialize the modifications:

• Merge openSAI patch for P4 SDE (I hope that Intel/Barefoot team could release a new version with the path soon)
• Merge fpmsyncd modification. As an FRR contributor and maintainer, I create an issue to tackle the FPM problem with MPLS (see https://github.com/FRRouting/frr/issues/11189) and we are looking to how enhance FPM to handle MPLS. But, the way VRFids are convey to fpmsyncd is not very good and could cause potential problem. We need to understand if VRFids are needed for MPLS table.
• Merge bgp docker modification to start automatically isisd
• Look to orchagent to determine how we could handle MPLS configuration on Loopback interface

I'll be glad to present in detail this work if you are interested.

Best regards

Olivier