vxlan encapsulation with ttl=0

[Abdelmuhaimen Seaudi]

Hi

I am testing p2p vxlan using sonic community 202012 on edgecore as8535-54x.

The ping is not working, and i found with packet capture I am receiving vxlan traffic with ttl=0.

Why is the hardware encapsulating the ARP in VXLAN packet with TTL=0 ?

Or is this something I can correct/configure in the sonic ?

16:32:52.234085 IP (tos 0x0, id 2994, offset 0, flags [none], proto UDP (17), length 96)

    4.4.4.4.61446 > 2.2.2.2.4789: [no cksum] VXLAN, flags [I] (0x08), vni 50

ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.50.18 tell 192.168.50.11, length 46

16:32:52.234183 IP (tos 0xc0, ttl 64, id 16733, offset 0, flags [none], proto ICMP (1), length 124)

    10.3.4.3 > 4.4.4.4: ICMP time exceeded in-transit, length 104

IP (tos 0x0, id 2994, offset 0, flags [none], proto UDP (17), length 96)

    4.4.4.4.61446 > 2.2.2.2.4789: [no cksum] VXLAN, flags [I] (0x08), vni 50

ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.50.18 tell 192.168.50.11, length 46

Abdelmuhaimen Seaudi, CCIE # 25265
Email: ase...@gmail.com

Skype: aseaudi

LinkedIn: www.linkedin.com/in/aseaudi

Telephone: +2012 84644 733

[Christian Svensson]

Do you see the same issue with IPv6?

--
You received this message because you are subscribed to the Google Groups "sonicproject" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonicproject...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonicproject/CABFmVg7P4rpYkaRP%3DcR_Qq5FKYjNLf3mojF%2B6GSewQxPh0oqMg%40mail.gmail.com.

[Abdelmuhaimen Seaudi]

Hello

IPv6 packets are VXLAN encapsulated with TTL 64, do you know why this is happening ?

21:40:56.806808 IP (tos 0x0, ttl 64, id 61067, offset 0, flags [none], proto UDP (17), length 122)

    4.4.4.4.40641 > 2.2.2.2.4789: [udp sum ok] VXLAN, flags [I] (0x08), vni 50

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001::1 > ff02::1:ff00:2: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001::2

  source link-address option (1), length 8 (1): f8:8e:a1:e0:72:11

    0x0000:  f88e a1e0 7211

21:40:57.830709 IP (tos 0x0, ttl 64, id 61141, offset 0, flags [none], proto UDP (17), length 122)

    4.4.4.4.40641 > 2.2.2.2.4789: [udp sum ok] VXLAN, flags [I] (0x08), vni 50

IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001::1 > ff02::1:ff00:2: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001::2

  source link-address option (1), length 8 (1): f8:8e:a1:e0:72:11

    0x0000:  f88e a1e0 7211

Abdelmuhaimen Seaudi, CCIE # 25265
Email: ase...@gmail.com

Skype: aseaudi

LinkedIn: www.linkedin.com/in/aseaudi

Telephone: +2012 84644 733

[Christian Svensson]

Hi,

No - no idea, but we too had issues when doing IPv4 between a SONiC and an Arista switch. We never got ARP to work, only IPv6. I think we managed to send IPv4 with static ARP, and we concluded it must be an incompatibility between EOS and SONiC with ARP suppression for EVPN. We never tried it again, but we filed https://github.com/kamelnetworks/sonic/issues/9 ("ARP over EVPN blackholed from Sonic to Arista switch") for our own records.

I think you are on the right track on finding what the actual issue we hit back then was. Good job! :-).

Maybe you can upload some PCAPs with ARP, IPv4 w/ static ARP, and IPv6 to the Github issue you are discussing this with Broadcom in (https://github.com/Azure/sonic-buildimage/issues/10050)?

That way people can look in e.g. Wireshark to debug a bit further.

Regards,

[abdelmuhai...@orange.com]

What about Sonic and Sonic, did you manage to run IPv4 between them ?

If yes, which image version did you use ?

Abdel-Muhaimen Seaudi

Orange Innovation Egypt

Email: abdelmuhai...@orange.com

Mobile: +2012 84644 733

---

From: Christian Svensson [chri...@cmd.nu]
Sent: Sunday, March 06, 2022 9:08 PM
To: Abdelmuhaimen Seaudi
Cc: sonicproject
Subject: Re: [SONiC] vxlan encapsulation with ttl=0

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

[Christian Svensson]

We never tried between two SONiC switches, at the time of testing we only had one SONiC Trident3 switch available for testing.

[Abdelmuhaimen Seaudi]

I changed the VXLAN tunnel attribute in orchagnet from the default UNIFROM_MODEL to PIPE_MODEL with TTL = 64, and now ARP and Ping is working over the P2P Vxlan Tunnel.

        attr.id = SAI_TUNNEL_ATTR_ENCAP_TTL_MODE;
        attr.value.s32 = SAI_TUNNEL_TTL_MODE_PIPE_MODEL;
        tunnel_attrs.push_back(attr);

        attr.id = SAI_TUNNEL_ATTR_ENCAP_TTL_VAL;
        attr.value.u8 = 64;
        tunnel_attrs.push_back(attr);

Abdelmuhaimen Seaudi, CCIE # 25265
Email: ase...@gmail.com

Skype: aseaudi

LinkedIn: www.linkedin.com/in/aseaudi

Telephone: +2012 84644 733

[Christian Svensson]

Awesome, great find! Reading the descriptions in saitunnel.h this fix makes sense. I guess somewhere a default for non-IP traffic is never set and defaults to 0. Hopefully Broadcom can comment on this.