ASTF TCP; Not receiving packets on server side

[Darius Grassi]

Hi,

I'm currently trying to use TRex's ASTF automation API to test the performance of a programmable switch pipeline. My current issue is that I see TRex generate packets and get received and sent by the switch, but the packets are not being received on the ASTF servers.

I'd like to simply loop traffic through the switch and back to the server it is generating from, so the traffic is received on the ASTF servers again. For my ASTF tests, I'm currently using the http_eflow.py profile to generate traffic, since I did not want to use any of the pcap examples.

Do you have any idea what I may be missing that would cause ASTF servers not to receive traffic?

I also have a few questions regarding the ASTF Python API:

1) Why does it no longer use the 'port' style that the STL API uses to specify which ports to send traffic to? Is there a different way in the ASTF API that I can accomplish this? (The only way I've found now is to alter the 'interfaces' section of my TRex config file)

2) My current tests generate 64 packets only, and I'm not sure why. In what part of the ASTF API can I specify, and increase, the number of packets to a more realistic stream?

Please let me know what output would be helpful so I can provide it for debugging.

[Besart Dollma]

Hi Darius, 

- First, you can capture traffic in TRex per port. So I would suggest capturing Rx traffic in the port that is meant to receive and verify that the traffic is indeed incoming. It can be anything from the configuration of the TRex port to the configuration of the Switch port. Another thing that can help us here is the output of portattr -a and the topology.

- Regarding ports, in ASTF the ports are coupled in (C,S) for example, port (0, 1), (2, 3) and so on. You can read about it  in the doc. You can enable or disable ports from a specific profile using the client mask.

- Please provide your code so that we can understand why the limitation. 

Thanks, 

[Darius Grassi]

Hi,

Thanks, this information has given me a much better understanding of things.

First, a few updates:

- I successfully used the client mask to enable specific ports for ASTF

- I enabled traffic capturing on the expected TRex Rx port, and did not see any traffic captured.

- I tried out capturing traffic with portattr -a with TRex in interactive mode, and posted my findings below.

------------------------------------------------------

Here are my results from running my profile with interactive TRex in ASTF mode:

NOTE: At the very start, 32 clients are started to match the 32 servers, but they drop to 0 after a few seconds. However, traffic continues to send.

------------------------------------------------------

Here is the output of portattr -a while the above traffic was sending:

On the switch, I see the same amount of traffic received as TRex says its sending, on the same ports. However, on my configured port (port 1, same port 1 as TRex) to send the traffic back on the switch, I only see 64 packets sent from the switch. With the current information I have, my hypothesis is that you are correct that this is either an issue with my TRex config file, or with my switch configuration.

------------------------------------------------------

My code below:

1) Trex config file

- version: 2

  port_limit: 4

  interfaces: ['5e:00.0', '5e:00.1', '3b:00.1', '3b:00.0']

  port_bandwidth_gb: 40

  c: 16

  port_info:

    - src_mac: 00:00:00:00:00:01

      dest_mac: 00:90:fb:71:64:8a

    - src_mac: 00:00:00:00:00:02

      dest_mac: 00:90:fb:71:64:8a

    - src_mac: 00:00:00:00:00:03

      dest_mac: 00:90:fb:71:64:8a

    - src_mac: 00:00:00:00:00:04

      dest_mac: 00:90:fb:71:64:8a

  memory:

    mbuf_64     : 1024

    mbuf_128    : 512

    mbuf_256    : 256

    mbuf_512    : 128

    mbuf_1024   : 256

    mbuf_2048   : 128

    traffic_mbuf_64     : 1024

    traffic_mbuf_128    : 512

    traffic_mbuf_256    : 256

    traffic_mbuf_512    : 128

    traffic_mbuf_1024   : 256

    traffic_mbuf_2048   : 128

    dp_flows    : 4096

  platform:

    master_thread_id: 0

    rx_thread_id: 17

    dual_if:

      - socket: 0

        threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]

      - socket: 1

        threads: [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]

------------------------------------------------------

2) TCP profile (http_eflow.py)

# we can send either Python bytes type as below:

http_req = b'GET /3384 HTTP/1.1\r\nHost: 22.0.0.3\r\nConnection: Keep-Alive\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\r\nAccept: */*\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate, compress\r\n\r\n'

# or we can send Python string containing ascii chars, as below:

class Prof1():

    def __init__(self):

        pass  # tunables

    def create_profile(self, size, loop, mss):

        http_response = 'HTTP/1.1 200 OK\r\nServer: Microsoft-IIS/6.0\r\nContent-Type: text/html\r\nContent-Length: 32000\r\n\r\n<html><pre>'+('*'*size*1024)+'</pre></html>'

        bsize = len(http_response)

        # Configure TCP client program

        prog_c = ASTFProgram()

        prog_c.send(http_req)

        prog_c.recv(bsize*loop)

        # Configure TCP server program

        prog_s = ASTFProgram()

        prog_s.recv(len(http_req))

        #prog_s.set_var("var1",10); # not used var, just for the example

        prog_s.set_var("var2",loop); # set var 0 to 10 

        prog_s.set_label("a:");

        prog_s.send(http_response)

        prog_s.jmp_nz("var2","a:") # dec var "var2". in case it is *not* zero jump a: 

        # Showcase TCP tuning

        info = ASTFGlobalInfo()

        if mss:

          info.tcp.mss = mss

        info.tcp.initwnd = 2  # start big

        info.tcp.no_delay = 1   # to get fast feedback for acks

        info.tcp.rxbufsize = 1024*1024  # 1MB window 

        info.tcp.txbufsize = 1024*1024  

        #info.tcp.do_rfc1323 =0

        # Define IP generators

        ip_gen_c = ASTFIPGenDist(ip_range=["16.0.0.0", "16.0.0.255"], distribution="seq")

        ip_gen_s = ASTFIPGenDist(ip_range=["48.0.0.0", "48.0.255.255"], distribution="seq")

        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"),

                           dist_client=ip_gen_c,

                           dist_server=ip_gen_s)

        temp_c = ASTFTCPClientTemplate(program=prog_c, ip_gen=ip_gen, cps=1, limit=1)

        temp_s = ASTFTCPServerTemplate(program=prog_s)  # using default association

        template = ASTFTemplate(client_template=temp_c, server_template=temp_s)

        profile = ASTFProfile(default_ip_gen=ip_gen, 

                              templates=template,                              

                              default_c_glob_info=info,

                              default_s_glob_info=info)

        return profile

    def get_profile(self, size=1, loop=10, mss=0, **kwargs):

        return self.create_profile(size, loop, mss)

def register():

    return Prof1()

------------------------------------------------------

3) ASTF automation API client Python code

# Test specs

TRAFFIC_MULT = 1.0

TEST_DURATION = 5

CAPTURE_LIMIT = 100

# TCP tunables

SIZE = 1

LOOP = 10

WIN = 32

MSS = 0

RECEIVER_PORT = [1]

class AstfTcpFlow(...):

    """

    """

    def runTest(self):

        self.trex_client.reset()

        self.trex_client.clear_stats()

        # IGNORE: switch port shaping configuration

        self.setup_basic_forwarding_to_1g()

        capture = self.trex_client.start_capture(

            rx_ports=RECEIVER_PORT, limit=CAPTURE_LIMIT

        )

        passed = True

        try:

            # load ASTF profile

            profile_path = 'http_eflow.py'

            profile_tunables = {'size': SIZE, 'loop': LOOP, 'mss': MSS}

            self.trex_client.load_profile(profile = profile_path, tunables = profile_tunables)

            print("Injecting with multiplier of '%s' for %s seconds" % (TRAFFIC_MULT, TEST_DURATION))

            self.trex_client.start(mult = TRAFFIC_MULT, duration = TEST_DURATION, client_mask=1)

            self.trex_client.wait_on_traffic()

            output = "/tmp/tcp-recv-port-{}.pcap".format(

                datetime.now().strftime("%Y%m%d-%H%M%S")

            )

            self.trex_client.stop_capture(capture["id"], output)

            stats = self.trex_client.get_stats()

            # use this for debug info on all the stats

            pprint(stats)

            if self.trex_client.get_warnings():

                print('\n\n*** test had warnings ****\n\n')

                for w in self.trex_client.get_warnings():

                    print(w)

            client_stats = stats['traffic']['client']

            server_stats = stats['traffic']['server']

            tcp_client_sent, tcp_server_recv = client_stats.get('tcps_sndbyte', 0), server_stats.get('tcps_rcvbyte', 0)

            tcp_server_sent, tcp_client_recv = server_stats.get('tcps_sndbyte', 0), client_stats.get('tcps_rcvbyte', 0)

            udp_client_sent, udp_server_recv = client_stats.get('udps_sndbyte', 0), server_stats.get('udps_rcvbyte', 0)

            udp_server_sent, udp_client_recv = server_stats.get('udps_sndbyte', 0), client_stats.get('udps_rcvbyte', 0)

            assert (tcp_client_sent == tcp_server_recv), 'Too much TCP drops - clients sent: %s, servers received: %s' % (tcp_client_sent, tcp_server_recv)

            assert (tcp_server_sent == tcp_client_recv), 'Too much TCP drops - servers sent: %s, clients received: %s' % (tcp_server_sent, tcp_client_recv)

            assert (udp_client_sent == udp_server_recv), 'Too much UDP drops - clients sent: %s, servers received: %s' % (udp_client_sent, udp_server_recv)

            assert (udp_server_sent == udp_client_recv), 'Too much UDP drops - servers sent: %s, clients received: %s' % (udp_server_sent, udp_client_recv)

        except TRexError as e:

            passed = False

            print(e)

        except AssertionError as e:

            passed = False

            print(e)

        if passed:

            print('\nTest has passed :-)\n')

        else:

            print('\nTest has failed :-(\n')

            sys.exit(1)

------------------------------------------------------

Is there any discernible reason why my automation code only generates 64 packets per port, whereas my interactive testing appears to naturally and freely generate traffic?

Are there any blatant issues with my code here that could be causing packets to not be received back at the server?

Thank you for all your help!

Best,

Darius

[Darius Grassi]

Apologies, I made a small mistake: 

• When I run TRex in interactive ASTF mode with my profile, lots of traffic is sent, but all packets are dropped at the switch. When I run TRex via my ASTF automation API, only 64 packets are sent to the switch, and all 64 packets are sent back by the switch, and then are never received by the server. Can this strange behaviour be described at all?

- Darius

[Darius Grassi]

Apologies again for the continual updates!

• I added the parameter to my start(...) function in my automation code for latency_pps=1000, and now the traffic generated by my script is identical to that generated by the interactive client.

Since my automation code contains switch configuration functions, this explains why I was seeing drops at the switch when I ran in interactive mode. I no longer see these at the switch.

Switch counters:

My only issue is that these frames sent back by the switch are not being received back by the server. My question is, is there anything in my code or results that points to why packets would be generated at the server and routed through the switch, but never received back at the server?

Thanks and apologies again,

Darius

[Besart Dollma]

Hello,

I am happy to see that you are understanding more of TRex and getting better at it.

My guess would be that the packets are simply not getting to TRex. Is there any other device between your switch and the TRex server port that might drop the packets?

My initial guess would be that the destination MAC address is incorrect. As such that TRex would drop the packets. However you should still see these packets in capture since capture turns on promiscuous. 

Unfortunately, as it seems at the moment, the packets are simply not getting to TRex, hence you will need to debug your topology. A simple thing to do, to verify your TRex configuration is correct is to temporally remove the switch and try a simple loopback connection. 

Thanks,