NAT testing, TRex originated traffic treated differently than 'real' traffic

jba...@gmail.com

unread,

Jun 29, 2017, 1:55:55 PM6/29/17

to TRex Traffic Generator

Hi there!

I'm using TRex to load test NAT performance.

I'm running into some interesting issues that I think may be a fault of the DUT, but wondering if anyone else has seen anything similar.

I have my DUT (Ubiquiti EdgeRouter Pro) configured to NAT between two interfaces. When i connect real clients, everything works fine, as expected. Tested with TCP/UDP/ICMP flows

When I run traffic through it with TRex, only UDP packets get translated, TCP are routed through untouched (no NAT translation takes place).

I experimented with the --checksum-offload flag, enabling it broke UDP entirely, the DUT wouldn't even pass the traffic, there was no effect on TCP. I've also tried with and without --learn-mode (1, 2, and 3), there was no effect.

The pcaps I'm using to generate the flows are some of the included http and dns samples that ship with TRex.

I'm not discounting that this could be something faulty with the DUT, but Ubiquiti products have been out a while, I feel like they should have caught this in QA? Especially since my real clients work correctly.
I'm wondering if TRex does something non-standard to the traffic that I'm not seeing, or if the samples were created with some weird TCP/IP options or flags. Maybe there's a runtime option I should try?

My next steps are to
1. Create my own sample pcap files with traffic that works.
2. craft packets with Scapy that match what i'm capturing on the wire and massaging them until it works.

Thanks much! Really enjoying the tool so far!

-Jeremy

hanoh haim

unread,

Jun 29, 2017, 2:59:21 PM6/29/17

to TRex Traffic Generator, jba...@gmail.com

Hi,

Are you sure your DUT does not terminate the TCP sessions? A simple way to verify it is to capture a Linux TCP session in both sides of the NAT, in case there is a diff between the sessions (same L7 not the same L4) you will need to wait a bit for our upcoming feature

See here

https://communities.cisco.com/community/developer/trex/blog/2017/06/20/trex-upcoming-stateful-scalable-tcp-support?mobileredirect=true

Thanks,

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6864c105-0d9b-458b-95bd-55ffe9010d32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

Jeremy

unread,

Jul 4, 2017, 4:57:40 PM7/4/17

to hanoh haim, TRex Traffic Generator

(resending because I forgot to include the mailing list)

Hi Hanoh,

Thank you for the response! It does not appear that the my DUT is terminating TCP. After more testing with Scapy it seems that this DUT validates all IP and TCP checksums, and upon further examination my setup was not handling --checksum-offload correctly, it would do TCP/UDP but not IP header checksums. Now I have a setup (trex in a esxi VM with direct passthrough to the network cards) that is handling this correctly.

Now my DUT is NATing my first SYN packets correctly, I can see the counters incrementing that the packets are received on the 'server' interface, but I can't get Trex to send the SYN/ACK. I've tried all of the various --learn-mode options. In '--learn-mode 1' I see the ACK number in the SYN in captures on both sides of the DUT, so I know it isn't getting stomped on in translation.

I've included the summary output below. If I'm parsing it correctly:

- 27 SYNs sent

- No packets were dropped

- 30 packets received (some extra random stuff emitted from the DUT)

- 27 NAT timeouts

So Trex appears to be seeing everything, but isn't recognizing the incoming SYNs as the ones that it sent out, so the SYN/ACKs are never being sent. Am I missing something here?

---

port : 0

------------

opackets : 27

obytes : 1728

ipackets : 0

ibytes : 0

Tx : 811.19 bps

port : 1

------------

opackets : 0

obytes : 0

ipackets : 30

ibytes : 1920

Tx : 0.00 bps

Cpu Utilization : 2.2 % 0.0 Gb/core

Platform_factor : 1.0

Total-Tx : 811.19 bps NAT time out : 27 (0 in wait for syn+ack)

Total-Rx : 862.84 bps NAT aged flow id: 0

Total-PPS : 1.58 pps Total NAT active: 0 (0 waiting for syn)

Total-CPS : 1.71 cps Total NAT opened: 27

Expected-PPS : 102.71 pps

Expected-CPS : 2.78 cps

Expected-BPS : 767.80 Kbps

Active-flows : 0 Clients : 1224 Socket-util : 0.0000 %

Open-flows : 27 Servers : 56 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

summary stats

--------------

Total-pkt-drop : 0 pkts

Warning : number of rx packets exceeds 101% of tx packets!

Total-tx-bytes : 1728 bytes

Total-tx-sw-bytes : 0 bytes

Total-rx-bytes : 1920 byte

Total-tx-pkt : 27 pkts

Total-rx-pkt : 30 pkts

Total-sw-tx-pkt : 0 pkts

Total-sw-err : 0 pkts

Total ARP sent : 7 pkts

Total ARP received : 2 pkts

---

commandline: sudo ./t-rex-64-debug -f ~/gnet_test/http_simple.yaml -d 10 --arp-refresh-period 5 --checksum-offload --learn-mode 1

yaml:

- duration : 0.1

generator :

distribution : "seq"

clients_start : "10.200.0.1"

clients_end : "10.200.4.200"

servers_start : "55.55.55.0"

servers_end : "55.55.55.55"

clients_per_gb : 201

min_clients : 101

dual_port_mask : "1.0.0.0"

tcp_aging : 0

udp_aging : 0

cap_ipg : true

cap_info :

- name: /home/ubuntu/gnet_test/delay_10_http_browsing_0.pcap

cps : 2.776

ipg : 10000

rtt : 10000

w : 1

Thanks!

Jeremy

On Thu, Jun 29, 2017 at 11:59 AM, hanoh haim <hhaim...@gmail.com> wrote:

Hi,
Are you sure your DUT does not terminate the TCP sessions? A simple way to verify it is to capture a Linux TCP session in both sides of the NAT, in case there is a diff between the sessions (same L7 not the same L4) you will need to wait a bit for our upcoming feature
See here

https://communities.cisco.com/community/developer/trex/blog/2017/06/20/trex-upcoming-stateful-scalable-tcp-support?mobileredirect=true

Thanks,
Hanoh

On Thu, 29 Jun 2017 at 13:55 <jba...@gmail.com> wrote:

Hi there!

I'm using TRex to load test NAT performance.

I'm running into some interesting issues that I think may be a fault of the DUT, but wondering if anyone else has seen anything similar.

I have my DUT (Ubiquiti EdgeRouter Pro) configured to NAT between two interfaces. When i connect real clients, everything works fine, as expected. Tested with TCP/UDP/ICMP flows

When I run traffic through it with TRex, only UDP packets get translated, TCP are routed through untouched (no NAT translation takes place).

I experimented with the --checksum-offload flag, enabling it broke UDP entirely, the DUT wouldn't even pass the traffic, there was no effect on TCP. I've also tried with and without --learn-mode (1, 2, and 3), there was no effect.

The pcaps I'm using to generate the flows are some of the included http and dns samples that ship with TRex.

I'm not discounting that this could be something faulty with the DUT, but Ubiquiti products have been out a while, I feel like they should have caught this in QA? Especially since my real clients work correctly.
I'm wondering if TRex does something non-standard to the traffic that I'm not seeing, or if the samples were created with some weird TCP/IP options or flags. Maybe there's a runtime option I should try?

My next steps are to
1. Create my own sample pcap files with traffic that works.
2. craft packets with Scapy that match what i'm capturing on the wire and massaging them until it works.

Thanks much! Really enjoying the tool so far!

-Jeremy

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6864c105-0d9b-458b-95bd-55ffe9010d32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jeremy

unread,

Jul 4, 2017, 4:59:11 PM7/4/17

to hanoh haim, TRex Traffic Generator

Additionally, when I add '--learn-verify' and disable NAT on my DUT, I see 'NAT learn errors: 0' but still don't see the SYNACK packets being sent. I only see the SYNACKs and the rest of the flow when I remove '--learn-mode X'

-Jeremy

hanoh haim

unread,

Jul 5, 2017, 3:06:39 AM7/5/17

to Jeremy, TRex Traffic Generator

Hi Jeremy,

You didn't mention the NIC type, running TRex with -v 7 will tell us.

To get specific packets to the software we need the RX packets to match a hardware filter. For some cards we use a TTL filter (TTL>128) and for some TOS LSB bit (depends on NIC capability) see here for CX4/Mellanox TOS filter example

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_trex_specific_implementation_details

I suspect that the DUT change one of the fields and it does not match our HW filter, can you check that.

A simple way to check it is by running without NAT and "-l 10" and check if latency packets are seen.

Another way is with your scapy script

thanks,

Hanoh

Jeremy

unread,

Jul 6, 2017, 2:16:59 AM7/6/17

to hanoh haim, TRex Traffic Generator

Does this help?

driver name : net_ixgbe

min_rx_bufsize : 1024

max_rx_pktlen : 15872

max_rx_queues : 128

max_tx_queues : 64

max_mac_addrs : 128

rx_offload_capa : 0x4f

tx_offload_capa : 0xbf

rss reta_size : 512

flow_type_rss : 0x38d34

port 0 desc: Ethernet Connection X552/X557-AT 10GBASE-T

port 1 desc: Ethernet Connection X552/X557-AT 10GBASE-T

I tried again with a different setup (Intel 82576 Gigabit x4, on a baremetal install, no esxi), I'm seeing the same issues.

--

With NAT enabled (and --learn-mode 1) I'm seeing a TTL of 255 as packets enter the DUT. (all packets, latency tests(SCTP) and test traffic). On both cards the Latency tests fail with NAT enabled

--

With NAT disabled and the following command: sudo ./t-rex-64-debug -f ~/gnet_test/http_simple.yaml -d 5 --arp-refresh-period 5 -l 10

I can only see the latency packets on the 82576 card, not the X552/X557-AT card.

maximum-latency : 88 usec

average-latency : 55 usec

latency-any-error : OK

On both:

- The TOS LSB is set to 1

- SCTP: TTL 255

- Test traffic: TTL 128

Thanks!

Jeremy

hanoh haim

unread,

Jul 6, 2017, 6:38:01 AM7/6/17

to Jeremy, TRex Traffic Generator

It means that your DUT when NAT enabled change the TTL, could you verify it?

The TTL should be 254 .

You can try with "--software" switch but it will work in lower speed.

Thanks,

Hanoh

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/6864c105-0d9b-458b-95bd-55ffe9010d32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.