trex with astf and linux interfaces

[Alexander Shalimov]

Hi all,

 

I have two VMs with NAT in the middle. I run one trex as just --astf, the second one with --astf --astf-server-only. Trex instances use linux interfaces. Run http_simple example.

 

There are issues:

1. As I start traffic generation, my ssh connection breaks. Is it expected?

2. It looks Trex and Linux fight for incoming packets. One SYN leads for RST answer, others with correct SYN/ACK+HTTP (see pic below). What I did wrong?

 

 

[Besart Dollma]

Hi, 

1. Not expected, shouldn't happen. It seems like the amount of traffic is overwhelming your machine.

2. Please provide the topology with IPv4 addresses so we can understand better your PCAP.

3. Also provide your TRex configuration.

Thanks

[Alexander Shalimov]

Hi,

 

Thanks for your reply.

 

The topology is:

 

VM1 (internal ip 10.7.7.28, no public ip, trex astf client, sending traffic to 130.193.32.6) --> NAT (changing VM1's internal IP/port to public one 84.201.191.207) --> VM2 (internal ip 10.0.1.21, public ip 130.193.32.6, trex astf server).

 

Pcap comes from VM1.

 

VMs have 4 cores. Ubuntu 20.04.

 

Trex is used from wget --no-cache https://trex-tgn.cisco.com/trex/release/latest --no-check-certificate

 

Could you please verify the configuration below?

 

VM1's:

 

trex config:

- port_limit      : 2

  version         : 2

  interfaces    : ["eth0", "dummy"]

  c : 1

  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.

    - ip : 10.7.7.29

      default_gw : 10.7.7.1

 

astf/my_http_simple.py:

from trex.astf.api import *

 

class Prof1():

    def __init__(self):

        pass

 

    def get_profile(self, **kwargs):

        # ip generator

        ip_gen_c = ASTFIPGenDist(ip_range=["10.7.7.28", "10.7.7.28"], distribution="seq")

        ip_gen_s = ASTFIPGenDist(ip_range=["130.193.32.6", "130.193.32.6"], distribution="seq")

        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"),

                           dist_client=ip_gen_c,

                           dist_server=ip_gen_s)

 

        return ASTFProfile(default_ip_gen=ip_gen,

                            cap_list=[ASTFCapInfo(file="../avl/delay_10_http_browsing_0.pcap",

                            cps=2.776)])

 

def register():

    return Prof1()

 

Commands:

sudo ./t-rex-64 -i --astf

sudo ./trex-console

start -f astf/my_http_simple.py -m 1 -d 1000 -l 1000

 

VM2's:

 

trex config:

- port_limit      : 2

  version         : 2

  interfaces    : ["dummy", "eth0"]

  c : 1

  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.

    - ip : 1.1.1.2 #trash

      default_gw: 1.1.1.1 #trash

    - ip : 10.0.1.21

      default_gw : 10.0.1.1

 

astf/my_http_simple.py:

from trex.astf.api import *

 

class Prof1():

    def __init__(self):

        pass

 

    def get_profile(self, **kwargs):

        # ip generator

        ip_gen_c = ASTFIPGenDist(ip_range=["10.7.7.28", "10.7.7.28"], distribution="seq")

        ip_gen_s = ASTFIPGenDist(ip_range=["10.0.1.21", "10.0.1.21"], distribution="seq")

        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"),

                           dist_client=ip_gen_c,

                           dist_server=ip_gen_s)

 

        return ASTFProfile(default_ip_gen=ip_gen,

                            cap_list=[ASTFCapInfo(file="../avl/delay_10_http_browsing_0.pcap",

                            cps=2.776)])

 

def register():

    return Prof1()

 

Commands:

sudo ./t-rex-64 -i --astf --astf-server-only

sudo ./trex-console

start -f astf/my_http_simple.py -m 1000 -d 1000

 

 

Btw, ssh connection breaks even with "-m 1" right after the start command.

 

 

Thanks!

 

20.02.2021, 10:16, "Besart Dollma" <besi7...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/3172cf29-e348-42e3-a71b-c47f192a8176n%40googlegroups.com.

[Besart Dollma]

Ok some unclear points. 

The IP you have given the TRex port in VM1 is 10.7.7.29 and in the script is 10.7.7.28. That would explain RST.

Secondly, your profiles don't make much sense. Your client should only send client traffic , replaying the pcap might not be a good idea since it might contain server responses.

I suggest you write a simple profile that sends in the client some HTTP request and in the server write a traffic that sends an OK when the request is received. From there you can continue to more complicated stuff.

Thirdly, please remove the ip 1.1.1.2 and its default gateway from vm2 trex_config. It might confuse the machine.

Finally, I don't understand what you are trying to achieve in the server. Your external IP 130.193.32.6 is not bound to TRex. Is kernel forwarding the packets from 130.193.32.6 to 10.0.0.1.21 and vice versa? Also are you sure that eth0 owns 10.0.0.21?

Lastly, do you have routes? How is the traffic supposed to get back to 10.7.7.28?

Thanks

[Alexander Shalimov]

Hi,

 

The IP you have given the TRex port in VM1 is 10.7.7.29 and in the script is 10.7.7.28. That would explain RST.

Oh, good catch. I've fixed trex_cfg,yaml with correct ip 10.7.7.28. Nothing changed unfortunately. See attached pcap for one tcp port.

 

Secondly, your profiles don't make much sense. Your client should only send client traffic , replaying the pcap might not be a good idea since it might contain server responses.

I suggest you write a simple profile that sends in the client some HTTP request and in the server write a traffic that sends an OK when the request is received. From there you can continue to more complicated stuff.

 

Actually, I don't really understand your point. According to docs, separate client and server modes should fixed the problem of separate pcaps. The documentation says that the client sends only clients request, server - only responses. And trex understands this from pcap itself. Is this not correct?

 

Thirdly, please remove the ip 1.1.1.2 and its default gateway from vm2 trex_config. It might confuse the machine.

When I remove the ip/gateway for dummy ports, I immediately get warning "Port 1 destination is unresolved". Not sure if it is a proble since nothing changed in my problem.

 

Finally, I don't understand what you are trying to achieve in the server. Your external IP 130.193.32.6 is not bound to TRex. Is kernel forwarding the packets from 130.193.32.6 to 10.0.0.1.21 and vice versa? Also are you sure that eth0 owns 10.0.0.21?

Lastly, do you have routes? How is the traffic supposed to get back to 10.7.7.28?

My goal is to emulate http server in VM2. I want to test NAT in the middle with high http load. All routing is done by Cloud itself.

For example, what happened for the session from the pcap:

VM1 sends packets 10.7.7.28:42121 -> 130.193.32.6:80. The Cloud knows that the destination is outside the user's subnet, thus the packet goes to NAT. NAT changes the packet to 84.201.191.207:57575 -> 130.193.32.6:80. When the packet comes to VM2, The Cloud changes external IP 130.193.32.6 to local VM2's ip 10.0.1.21. And VM2's Linux sees the packet 84.201.191.207:57575 -> 10.0.1.21:80. The response will go back through the same procedure:  VM2 sends 10.0.1.21:80 -> 84.201.191.207:57575; the Cloud fixes external IP: 130.193.32.6:80 -> 84.201.191.207:57575; The NAT does reverse translation: 130.193.32.6:80 -> 10.7.7.28:42121.

 

How do you suggest to reach the goal?

 

Thanks! Appreciate your help.

 

20.02.2021, 15:13, "Besart Dollma" <besi7...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/eed8c169-07e1-42e5-8673-df4ce0ae360en%40googlegroups.com.

[Besart Dollma]

Hi Alex, 

Now I get it. From the docs:

The server side learn the information from the traffic. The server side responds to src/dst IP traffic with the opposite ips (e.g. SYN from 10.0.0.1→40.0.0.1 will be answered by TRex 40.0.0.1→10.0.01) and it is not related to the client pool — as there could be NAT that change inside/outside. The server only cares about the TCP/UDP destination port to associate the port with the right L7 template.

In VM2, you are sending traffic back to the IP behind the NAT.  The server doesn't know this address, can't have it in its flow table.

[Vijay Ram]

Hi Alex,

Were you able to fix the TCP RST issue? I have exactly same topology (client/server with linux kernel interfaces) as like yours and with HTTP script.

I observed the same TCP RST and so I added below

Client:

iptables -A OUTPUT -p tcp --destination-port 80

Server:

iptables -A INPUT -p tcp --destination-port 80

With this TCP RST resolved for me but I observed huge TCP DUP ACK and re-transmissions. 

I even added 

ethtool  --offload eth1 tx off rx off sg off gso off tso off

But still DUP ACKs observed. If you have fixed issue in some other way please share?

thx,

Vijayram

[Bidhov Bizar]

Hi,

    I too faced the same issue. I solved it using the following way

1. Make sure there is no other interface active in your namespace that is connecting to the router.

    Do the following

    bash$ ip route

    and see how many interfaces are currently active and willing to connect to the same network. If you see more than one. Change your network such a way that there is just one interface eth1 in your case.

2 Your interface got swapped. Just do the following

1. In server cfg change from

interfaces    : ["dummy", "eth1"]

to

interfaces    : [ "eth1" , "dummy"]

2.

In client cfg side

interfaces    : [ "eth1" , "dummy"]

to

interfaces    : ["dummy", "eth1"]

3. Add some garbage ip and gateway in the configuration file if your interface order is

interfaces    : ["dummy", "eth1"]

eg:

interfaces    : ["dummy", "eth1"]

          - ip         : 192.168.12.78

            default_gw : 192.168.12.65

          - ip         : 192.168.12.78

            default_gw : 192.168.12.65

Even though first interface is dummy it will still look for some values.

4. You can remove --astf-server-only and --astf-client-mask 0x1 from your command its ok if you dont use this and use dummy.

Regards,

Bidhov Bizar