[stratum-dev] Is there a limit on the # of ECMP groups stratum_bmv2 can handle?

[A Sydney]

Hi Stratum folks,

# Context:                          

I have created a pipeline that enables IPv4-ECMP (P4 snippet shown in attached file action_selector*) and I am able to build and push p4info.txt and bmv2.json to the switches. 

# Issue:

When I attempt to add groups, the first one moves to the "ADDED" state. When I attempt to add a second, the first group and the second both move to the "PENDING_ADD_RETRY" state.

# ONOS logs:

At the same time, P4RuntimeClientImpl throws the error "Error while performing READ on device:spine2...Unexpected error in RPC handling". Then ONOS spirals into this cycle of attempting to reinstall all flows on the switch (See debug.log for details.).

# Logs on one of the bmv2 switches:

I then see the following error on the switch:

[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/repeated_field.h:1694] CHECK failed: (index) < (current_size_):

# Question:

Have you ever come across this one?  Are there any obvious mistakes that I'm running into (Perhaps my P4 ECMP definition attached is incorrect?)? Any suggestions on how to debug further?

Thanks!

-Syd

[Maximilian Pudelko]

Hi Syd,

I have not observed this error before. Stratum-bmv2 is backed by PI (https://github.com/p4lang/PI).

All P4RT messages are passed to it nearly unmodified, so the issue must lie there.

Max

_______________________________________________
stratum-dev mailing list
strat...@lists.stratumproject.org
https://lists.stratumproject.org/listinfo/stratum-dev

[Maximilian Pudelko]

Hi Syd,

After some investigation I found the root cause to this issue: https://github.com/stratum/stratum/pull/687

I expect this to get merged soon.

Max

[A Sydney]

Hi Stratum folks,

                         @Max, thanks a bunch! I updated the package in and indeed, this particular error has been fixed. 

However, I'm getting this new error where LLDP pkt-ins are causing a buffer underflow exception in ONOS. When I disable the lldpprovider at ONOS, the error is gone. Can you kindly take a look at the following log and provide feedback (i.e. Have you seen this one before, is stratum the source of the issue, or should I take it up with ONOS folks)?

https://pastebin.com/SuS2ZhtV

Cheers!
-Syd

[Maximilian Pudelko]

Hi Syd,

That's an issue on the ONOS side, more specifically the fabric-interpreter: https://github.com/opennetworkinglab/onos/blob/master/pipelines/fabric/impl/src/main/java/org/onosproject/pipelines/fabric/impl/behaviour/FabricInterpreter.java#L262-L265

We fixed something similar in our internal version very recently. In short, byte strings returned by

Stratum are not zero-padded and ONOS has to handle them accordingly.

Here is a snippet of the new code:

        if (packetMetadata.isPresent()) {
            try {
                ImmutableByteSequence portByteSequence = packetMetadata.get()
                        .value().fit(P4InfoConstants.INGRESS_PORT_BITWIDTH);
                short s = portByteSequence.asReadOnlyBuffer().getShort();
                ConnectPoint receivedFrom = new ConnectPoint(deviceId, PortNumber.portNumber(s));
                ByteBuffer rawData = ByteBuffer.wrap(packetIn.data().asArray());
                return new DefaultInboundPacket(receivedFrom, ethPkt, rawData);
            } catch (ImmutableByteSequence.ByteSequenceTrimException e) {
                throw new PiInterpreterException(format(
                        "Malformed metadata '%s' in packet-in received from '%s': %s",
                        P4InfoConstants.INGRESS_PORT, deviceId, packetIn));
            }
        } else {
            throw new PiInterpreterException(format(
                    "Missing metadata '%s' in packet-in received from '%s': %s",
                    P4InfoConstants.INGRESS_PORT, deviceId, packetIn));
        }

Feel free to submit this as a fix to ONOS.

Max

On Wed, May 12, 2021 at 9:54 AM A Sydney via stratum-dev <strat...@lists.stratumproject.org> wrote:

Hi Stratum folks,

                         @Max, thanks a bunch! I updated the package in and indeed, this particular error has been fixed. 

However, I'm getting this new error where LLDP pkt-ins are causing a buffer underflow exception in ONOS. When I disable the lldpprovider at ONOS, the error is gone. Can you kindly take a look at the following log and provide feedback (i.e. Have you seen this one before, is stratum the source of the issue, or should I take it up with ONOS folks)?

https://pastebin.com/SuS2ZhtV

Cheers!
-Syd

On Tue, May 11, 2021 at 11:38 PM Maximilian Pudelko <m...@opennetworking.org> wrote:

[A Sydney]

I'll take this one up with ONOS folks, but in the meantime, I'm attempting to insert this snippet in my codebase for a quick test. 

It appears that ONOS has a FabricConstants.java class (as opposed to P4InfoConstants.java) which does have "INGRESS_PORT" as follows:

'''

    public static final PiPacketMetadataId INGRESS_PORT =
            PiPacketMetadataId.of("ingress_port");

'''

I'm assuming that I can replace your "P4InfoConstants.INGRESS_PORT" with "FabricConstants.INGRESS_PORT"? But what about but P4InfoConstants.INGRESS_PORT_BITWIDTH? Can you kindly share how it is defined?

Thanks,

-Syd

[Maximilian Pudelko]

Sure.

We parse the p4info file of the fabric pipeline and look for the `controller_packet_metadata` fields.

Currently it is 9 bits:

  metadata {
    id: 1
    name: "ingress_port"
    bitwidth: 9
  }

Max

[A Sydney]

So I've updated my pipeconf class with this info, and the error no longer occurs.

Thanks!

-Syd