Trouble setting up NAT use case

VaibhaV Sharma

unread,

Mar 16, 2016, 2:22:30 AM3/16/16

to TRex Traffic Generator

Hello,

Thanks for this excellent tool. I am still getting a hang of how this works so pardon my ignorance.

I am trying to simulate stateful traffic running through a linux firewall / NAT device. I am running two servers connected to a Cisco 2960 switch. The switch has two VLANs -

900 - Client interfaces (Private) - 16.0.0.0/8

910 - Server interfaces (Public) - 48.0.0.0/8

One server has Trex with two interfaces for traffic (one on client and one on server vlan). The other server is a linux machine with NAT setup between two interfaces.

To get started, I am trying to use http_simple yaml to run traffic through the linux machine. Routed setup seems to work OK but when I switch the linux machine to do NAT and enable learn-mode on trex command line, I noticed a few things -

* There is a huge number of bad chunk length packets

* Even with forwarding blocked on the linux router/nat, both trex ports generate packets as if the client side was requesting that data. This is by design I guess?

Would appreciate any clues to get a working setup.

Thanks,

--

VaibhaV Sharma

unread,

Mar 16, 2016, 3:14:50 AM3/16/16

to TRex Traffic Generator

More info on this.

The moment I enable --learn-mode, trex starts generating huge number of these packets on the client interface -

00:12:11.532040 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

00:12:11.533036 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

00:12:11.534035 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

00:12:11.535036 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

00:12:11.536032 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

00:12:11.537033 IP 16.0.0.1.32836 > 48.0.0.1.80: sctp (1) [Bad chunk length 0]

Any ideas?

Thanks,

--

VaibhaV

hanoh haim

unread,

Mar 16, 2016, 3:21:16 PM3/16/16

to VaibhaV Sharma, TRex Traffic Generator

Hi

There are two option for NAT learning.

Mode ==2 , we add IPv4 option to the syn packet. Exactly what you see.

This option is interpreted by WireShark.

Cisco ASA does not allow IPv4 option so we add mode==1 that convey the info on the SYN packet ack field

I would check the learn mode when NAT is not configured on the Linux ,if it does switch to mode 1.

Let us know if it work,

Thanks

Hanoh

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/08eccf99-a57a-46d7-915c-053a09589be9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

חנוך‎

unread,

Mar 16, 2016, 4:50:37 PM3/16/16

to VaibhaV Sharma, TRex Traffic Generator

Hi,

Looking again in bigger screen, this are the latency packets. You can reduce the rate by adding -l 1

Or change the type to icmp ( see the manual)

Thanks

Hanoh

Sent from my iPad

VaibhaV Sharma

unread,

Mar 17, 2016, 1:24:39 AM3/17/16

to TRex Traffic Generator, vaib...@gmail.com

Thanks for the pointers.

I added -l1 and changed it to use ICMP and the output looks much easier to deal with. I have also tried both nat mode 1 and 2 but I still have NAT issues.

I logged linux iptables and it looks like the linux kernel on the NAT device is treating all trex generated packets as invalid and hence dropping them. Any ideas? -

Mar 16 22:21:24 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.1 DST=48.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:25 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.2 DST=48.0.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:25 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.3 DST=48.0.0.3 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:26 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.4 DST=48.0.0.4 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:26 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.5 DST=48.0.0.5 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:26 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.6 DST=48.0.0.6 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:27 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.7 DST=48.0.0.7 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:27 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.8 DST=48.0.0.8 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 22:21:27 vstest-26-219 kernel: INVALID packet: IN=enp24s0f0 OUT= MAC=00:15:17:86:bc:5c:00:1b:21:b4:77:08:08:00 SRC=16.0.0.9 DST=48.0.0.9 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=29742 DF PROTO=TCP SPT=1024 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

--

VaibhaV

hanoh haim

unread,

Mar 17, 2016, 7:32:35 AM3/17/16

to VaibhaV Sharma, TRex Traffic Generator

Dear VaibhaV,

Could you send the TRex command line and the output of TRex window when NAT is *not* configured on the linux?

Please add "-l 100" to the command line and set it to be ICMP

I can't see the DROP reason. want to be sure that the drop is due to NAT configuration or maybe something else

thanks

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/0d69670f-63eb-4400-a3f5-7c0c5b3da9f6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ido

unread,

Mar 17, 2016, 11:12:37 AM3/17/16

to TRex Traffic Generator, vaib...@gmail.com

We send non zero number in the TCP ACK field of the SYN packet.

.Although it is not legal, firewalls we dealt with did not care

Maybe IPtables is more strict in this case. Just an idea. I did not see this in documentation.

בתאריך יום חמישי, 17 במרץ 2016 בשעה 13:32:35 UTC+2, מאת Hanoch Haim:

VaibhaV Sharma

unread,

Mar 17, 2016, 5:00:48 PM3/17/16

to TRex Traffic Generator, vaib...@gmail.com

Here is the commandline and output without NAT enabled (quite long) -

./t-rex-64 -f cap2/http_simple.yaml -c 1 -d 10 -m 1 -l 1

=======

==================

interface sum

==================

------------------------

per core stats core id : 1

------------------------

per core per if stats id : 1

------------------------

port 0, queue id :0 - client

----------------------------

port 1, queue id :0 - server

----------------------------

==================

generators

==================

normal

-------------

min_delta : 10 usec

cnt : 696

high_cnt : 0

max_d_time : 0 usec

sliding_average : 0 usec

precent : 0.0 %

histogram

-----------

m_total_bytes : 941.31 Kbytes

m_total_pkt : 1.04 Kpkt

m_total_open_flows : 28.00 flows

m_total_pkt : 1036

m_total_open_flows : 28

m_total_close_flows : 28

m_total_bytes : 963900

==================

latency

==================

Cpu Utilization : 0.0 %

if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window

| , , check, , latency(usec),latency (usec) ,(usec) ,

----------------------------------------------------------------------------------------------------------------

0 | 12, 12, 0, 0, 50 , 163, 18 | 114 0 143 0 111 0 139 0 126 0 163 0 132

1 | 12, 12, 0, 0, 50 , 171, 13 | 115 0 153 0 139 0 129 0 127 0 108 0 142

cpu : 0.0 %

port 0

-----------------

counter

-----------

m_tx_pkt_ok : 12

m_pkt_ok : 12

-----------

min_delta : 10 usec

cnt : 12

high_cnt : 12

max_d_time : 163 usec

sliding_average : 50 usec

precent : 100.0 %

histogram

-----------

h[100] : 12

jitter : 18

port 1

-----------------

counter

-----------

m_tx_pkt_ok : 12

m_pkt_ok : 12

-----------

min_delta : 10 usec

cnt : 12

high_cnt : 12

max_d_time : 171 usec

sliding_average : 50 usec

precent : 100.0 %

histogram

-----------

h[100] : 12

jitter : 13

rx_checker is disabled

---------------

port : 0

------------

opackets : 404

obytes : 32684

ipackets : 656

ibytes : 936944

Tx : 15.13 Kbps

port : 1

------------

opackets : 656

obytes : 936944

ipackets : 404

ibytes : 32684

Tx : 437.33 Kbps

Cpu Utilization : 0.0 % 10.0 Gb/core

Platform_factor : 1.0

Total-Tx : 452.46 Kbps

Total-Rx : 452.46 Kbps

Total-PPS : 61.68 pps

Total-CPS : 1.60 cps

Expected-PPS : 102.71 pps

Expected-CPS : 2.78 cps

Expected-BPS : 764.51 Kbps

Active-flows : 0 Clients : 255 Socket-util : 0.0000 %

Open-flows : 28 Servers : 65535 Socket : 0 Socket/Clients : 0.0

drop-rate : 0.00 bps

summary stats

--------------

Total-pkt-drop : 0 pkts

Total-tx-bytes : 969628 bytes

Total-tx-sw-bytes : 1584 bytes

Total-rx-bytes : 969628 byte

Total-tx-pkt : 1060 pkts

Total-rx-pkt : 1060 pkts

Total-sw-tx-pkt : 24 pkts

Total-sw-err : 0 pkts

maximum-latency : 171 usec

average-latency : 50 usec

latency-any-error : OK

[root@vstest-26-220 v1.90]#

=======

--

VaibhaV

VaibhaV Sharma

unread,

Mar 17, 2016, 5:03:02 PM3/17/16

to TRex Traffic Generator, vaib...@gmail.com

That could be it. Any way around this?

For NAT, wouldn't it be better to define NAT config and some parameters in the config rather than trex trying to autodetect? That way, at least trex will know what NAT'd IP to expect. Ports can be dynamic.

--

VaibhaV

hanoh haim

unread,

Mar 17, 2016, 11:56:35 PM3/17/16

to VaibhaV Sharma, TRex Traffic Generator

Hi,

You need to understand why iptable drop the first SYN packet. It might be due a rule/not enough flows

See here how to debug it more:

http://serverfault.com/questions/309691/why-is-our-firewall-ubuntu-8-04-rejecting-the-final-packet-fin-ack-psh-wit

As Ido said with mode==1 we set the ACK value of the SYN packet with our flow id (not zero). But I can't be sure this is your problem.

I would check another thing:

Could you run the same command that works with

1. simple_http

2. *without* --learn

3. With NAT configuration

This mode should not work as TRex won't learn the translation *but* you would be able to see if the first SYN is being dropped by iptable.

If the packet is dropped there is something else with iptable

Thanks

Hanoh

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c3f1d554-d697-4eda-82b7-5cccb7677892%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ido

unread,

Mar 18, 2016, 4:38:04 PM3/18/16

to TRex Traffic Generator

Hi VaibhaV,

Would help if you could really try to understand as Hanoch explained if the issue is really the non zero ACK in the SYN

Just to answer your last question.

In the general cast, we can't have configuration file to know the NAT translation, since the device doing the NAT chooses the addresses/ ports translation.

You are correct if the mapping is static.

So far, we did not encounter issues with our "info passing" method inside the SYN ACK.

If you determine this is really an issue, we will think of some solution

Ido
בתאריך יום רביעי, 16 במרץ 2016 בשעה 08:22:30 UTC+2, מאת VaibhaV Sharma:

Reply all

Reply to author

Forward