iptables NAT forwarding adding 75-100ms

[Mike Lovell]

I have a strange occurrence of lag on my local networking. I have a
cable modem that plugs into a Debian server, then that Debian server is
plugged into a switch that all other machines in the house connect to.

So something like:

wan0 -> wan
eth0 -> lan

The relevant forwarding/NAT rules are:


#iptables -A FORWARD -i wan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#iptables -A FORWARD -i eth0 -o wan0 -j ACCEPT

#iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE


So pretty standard boring NAT.

Lag is occurring (between 75ms and 100ms) on all forwarding rules. Apart
from the lag they function fine (no connectivity issues).


Ping: LAN Machine -> Debian Router = ~0.7ms
Ping: Debian Router -> Google = ~20ms
Ping: LAN Machine -> Google = ~121ms !!!


The Debian server has plenty of free RAM, the load is showing as low,
it's (at this time) entirely dedicated to routing - Why is it
introducing 100ms of lag into forwarded traffic???

Anyone else seen similar to this???


I get great speed from LAN machines, just high latency.

~ Mike

[ein]

On 04/30/2012 12:42 AM, Mike Lovell wrote:
> I have a strange occurrence of lag on my local networking. I have a
> cable modem that plugs into a Debian server, then that Debian server is
> plugged into a switch that all other machines in the house connect to.
>
> So something like:
>
> wan0 -> wan
> eth0 -> lan
>
> The relevant forwarding/NAT rules are:
>
>
> #iptables -A FORWARD -i wan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> #iptables -A FORWARD -i eth0 -o wan0 -j ACCEPT
>
> #iptables -t nat -A POSTROUTING -o wan0 -j MASQUERADE
>
>
> So pretty standard boring NAT.

How many FORWARD rules u have?
Do above rules are in beginning of FORWARD chain? If no, please switch
them as far of begin as u can. Is lag time changed?

> Lag is occurring (between 75ms and 100ms) on all forwarding rules. Apart
> from the lag they function fine (no connectivity issues).

How do u check that?

> Ping: LAN Machine -> Debian Router = ~0.7ms
> Ping: Debian Router -> Google = ~20ms
> Ping: LAN Machine -> Google = ~121ms !!!

Wrong! Please 'ping' nearest machine after your router for example your
ISP's gateway or ISP's DNS servers. Please have in mind that your ISP's
router have more important things to do, than respond to ICMP echo
request messages.

> The Debian server has plenty of free RAM, the load is showing as low,
> it's (at this time) entirely dedicated to routing - Why is it
> introducing 100ms of lag into forwarded traffic???

What version of Debian is it?
How much forward traffic u have?

> I get great speed from LAN machines, just high latency.

Do you have some QoS at this machine?

[ein]

On 04/30/2012 11:06 AM, ein wrote:
>> So pretty standard boring NAT.
>
> How many FORWARD rules u have?
> Do above rules are in beginning of FORWARD chain? If no, please switch
> them as far of begin as u can. Is lag time changed?

I meaned as close of the FORWARD begin as possible. :)

[Chris Davies]

In comp.os.linux.networking Mike Lovell <mike....@null.local> wrote:
> Ping: LAN Machine -> Debian Router = ~0.7ms
> Ping: Debian Router -> Google = ~20ms
> Ping: LAN Machine -> Google = ~121ms !!!

Are you pinging by name or by address? If it's by name can you please
repeat the tests with IP addresses. (This will eliminate any delay caused
by slow DNS lookups.)

Chris

[Richard Kettlewell]

The time= field in ping output measures the round trip of the ICMP
packet; it doesn't include the DNS lookup.

--
http://www.greenend.org.uk/rjk/

[Alastair Black]

Bad connection on an RJ-45 connector
between eth0 and LAN switch?

Also, do *not* rely on autosense in
switchers for any crossover/patch cable
detection. If you know it ought to be a
crossover, use a crossover cable.

Alastair

[Mike Lovell]

On 2012-04-30, ein <e...@no.spam> wrote:
> How many FORWARD rules u have?
> Do above rules are in beginning of FORWARD chain? If no, please switch
> them as far of begin as u can. Is lag time changed?

Just rules for the NAT, that's it:

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
LOG all -- anywhere anywhere LOG level warning prefix "IPv4-forward "

> How do u check that?
>
>> Ping: LAN Machine -> Debian Router = ~0.7ms
>> Ping: Debian Router -> Google = ~20ms
>> Ping: LAN Machine -> Google = ~121ms !!!
>
> Wrong! Please 'ping' nearest machine after your router for example your
> ISP's gateway or ISP's DNS servers. Please have in mind that your ISP's
> router have more important things to do, than respond to ICMP echo
> request messages.

Interesting, much better:

--- 68.86.118.57 ping statistics ---
10 packets transmitted, 9 received, 10% packet loss, time 9017ms
rtt min/avg/max/mdev = 28.711/57.428/109.753/28.626 ms

>> The Debian server has plenty of free RAM, the load is showing as low,
>> it's (at this time) entirely dedicated to routing - Why is it
>> introducing 100ms of lag into forwarded traffic???
>
> What version of Debian is it?
> How much forward traffic u have?

squeeze

Even if I firewall all other traffic but a single test machine, I still
get the latency problem there.

>> I get great speed from LAN machines, just high latency.
>
> Do you have some QoS at this machine?

No


So, based on the better ping to the next hop after my router, why (how)
would something be distinguishing between the router carrying out the
ping, and something behind the router?

Can I mask it from doing so (if that's the problem)?


Thanks,

~ Mike

[Mike Lovell]

On 2012-04-30, Alastair Black <abqab...@example.net> wrote:
> Bad connection on an RJ-45 connector
> between eth0 and LAN switch?
>
> Also, do *not* rely on autosense in
> switchers for any crossover/patch cable
> detection. If you know it ought to be a
> crossover, use a crossover cable.

But ping/connectivity *to* the router (eth0) from the LAN is fine. The
problem only occurs when going through NAT/FORWARDING to something
behind it.

Although it appear (from a previously suggested test) that it *Doesn't*
occur on the first hop after my router.

So something strange is happening...

--
Jews, Christians & Muslims
The content of your posts will show how much you
really believe God is looking over your shoulder

[Rick Jones]

Chris Davies <chris-...@roaima.co.uk> wrote:
> Are you pinging by name or by address? If it's by name can you
> please repeat the tests with IP addresses. (This will eliminate any
> delay caused by slow DNS lookups.)

Is there a ping utility out there which puts a DNS lookup in the
timing between the sending of the ICMP Echo Request and the receipt of
the ICMP Echo Reply?

Sure, DNS lookups could delay sending the first or subsequent ICMP
Echo Requests, but any ping utility that sticks just about anything
between the sending of the request and the receipt of the reply is
doing something it shouldn't.

rick jones
--
firebug n, the idiot who tosses a lit cigarette out his car window
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

[Chris Davies]

Richard Kettlewell <r...@greenend.org.uk> wrote:
> The time= field in ping output measures the round trip of the ICMP
> packet; it doesn't include the DNS lookup.

Ooops, yes. I was thinking of the (really old) ping that (wrongly)
used to do this, despite knowing that recent ping utilities do it
right. I'm going to blame transient brain-fade on this one and hope
no-one objects :-/

Chris

[Pascal Hambourg]

Hello,

Mike Lovell a écrit :

> On 2012-04-30, ein <e...@no.spam> wrote:
>>
>>> Ping: LAN Machine -> Debian Router = ~0.7ms
>>> Ping: Debian Router -> Google = ~20ms
>>> Ping: LAN Machine -> Google = ~121ms !!!
>>
>> Wrong!

What is wrong ?

>> Please 'ping' nearest machine after your router for example your
>> ISP's gateway or ISP's DNS servers. Please have in mind that your ISP's
>> router have more important things to do, than respond to ICMP echo
>> request messages.

Indeed. So why do you suggest to ping the ISP's gateway, which is a
router ? By the way, AFAICS, the OP did ping his own Debian router and
a Google server, not any ISP's router.

> Interesting, much better:

What is much better ?

> --- 68.86.118.57 ping statistics ---

What is that address ?

> 10 packets transmitted, 9 received, 10% packet loss, time 9017ms
> rtt min/avg/max/mdev = 28.711/57.428/109.753/28.626 ms

The average RTT is still ~60 ms.

> So, based on the better ping to the next hop after my router,

Better than what ?

> why (how)
> would something be distinguishing between the router carrying out the
> ping, and something behind the router?

In the old days some ISPs have been known to try to detect multiple
hosts behind NAT with variations of the TTL in packets.

> Can I mask it from doing so (if that's the problem)?

You can rewrite the TTL with iptables. Note that traceroute won't work
any more.

[Mike Lovell]

On 2012-05-01, Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:
>> Interesting, much better:
>
> What is much better ?

Pinging the next hop as opposed to Google

>> --- 68.86.118.57 ping statistics ---
>
> What is that address ?
>
>> 10 packets transmitted, 9 received, 10% packet loss, time 9017ms
>> rtt min/avg/max/mdev = 28.711/57.428/109.753/28.626 ms
>
> The average RTT is still ~60 ms.
>
>> So, based on the better ping to the next hop after my router,
>
> Better than what ?

Sorry, maybe that wasn't a good output.

I just pinging:

te-1-2-ur03.nchicago.il.chicago.comcast.net

Which is the next hop, from my Debian router and a machine in the LAN.

Router avg 20.327
LAN avg 21.623

>> why (how)
>> would something be distinguishing between the router carrying out the
>> ping, and something behind the router?
>
> In the old days some ISPs have been known to try to detect multiple
> hosts behind NAT with variations of the TTL in packets.

Interesting

>> Can I mask it from doing so (if that's the problem)?
>
> You can rewrite the TTL with iptables. Note that traceroute won't work
> any more.

My friend, you are a genius...


sudo iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 56
sudo iptables -t mangle -A OUTPUT -j TTL --ttl-set 56


~$ ping -c 10 www.google.com
PING www.google.com (74.125.225.83) 56(84) bytes of data.
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=1
ttl=55 time=35.7 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=2
ttl=55 time=12.8 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=3
ttl=55 time=14.7 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=4
ttl=55 time=15.5 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=5
ttl=55 time=15.3 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=6
ttl=55 time=14.6 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=7
ttl=55 time=18.9 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=8
ttl=55 time=14.7 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=9
ttl=55 time=12.6 ms
64 bytes from ord08s07-in-f19.1e100.net (74.125.225.83): icmp_req=10
ttl=55 time=180 ms

--- www.google.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 12.615/33.538/180.324/49.347 ms



So looks like those little darlings were to blame after all!

~ Mike

[Pascal Hambourg]

Mike Lovell a écrit :

> On 2012-05-01, Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:
>>> Interesting, much better:
>> What is much better ?
>
> Pinging the next hop as opposed to Google

Better ping a host than a router. As ein wrote, "routers have more

important things to do, than respond to ICMP echo request messages".

> My friend, you are a genius...

I wish I were...

> sudo iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 56
> sudo iptables -t mangle -A OUTPUT -j TTL --ttl-set 56

Note that the TTL of forwarded packets is decremented by 1 after the
PREROUTING chains, so you shouldn't set the same TTL in OUTPUT if you
want the same TTL when locally-generated and forwarded packets leave
your router.

Or you can set the TTL in POSTROUTING for both locally-generated and
forwarded packets.

[Mike Lovell]

Yer you're right, I jumped the gun. It's gone back up now, must have
been a fluke. It was lower right after coming up from a reboot.

So this would achieve the same TTL?

iptables -t mangle -A PREROUTING -i eth0 -j TTL --ttl-set 57

[Pascal Hambourg]

Mike Lovell a écrit :

I guess so. Or just :

iptables -t mangle -A POSTROUTING -o wan0 -j TTL --ttl-set 56

[Mike Lovell]

On 2012-05-01, Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:
> I guess so. Or just :
>
> iptables -t mangle -A POSTROUTING -o wan0 -j TTL --ttl-set 56

Gah I definitely celebrated too soon.

Must have been (possibly) related to rebooting the Debian router, it's
returned to the same delay as before now (even with the TTL changes).

There's something else fishy here.

[Pascal Hambourg]

Mike Lovell a écrit :

>
> Must have been (possibly) related to rebooting the Debian router, it's
> returned to the same delay as before now (even with the TTL changes).

Can you reproduce this ?

[Ian Zimmerman]

Mike> why (how) would something be distinguishing between the router
Mike> carrying out the ping, and something behind the router?

Pascal> In the old days some ISPs have been known to try to detect
Pascal> multiple hosts behind NAT with variations of the TTL in packets.

Mike> Can I mask it from doing so (if that's the problem)?

Pascal> You can rewrite the TTL with iptables. Note that traceroute
Pascal> won't work any more.

Why the need for iptables? Wouldn't it be simpler and safer to just
tweak the value in /proc/sys/net/ipv4/ip_default_ttl on each host
according to its hop distance from the router?

--
Ian Zimmerman
gpg public key: 1024D/C6FF61AD
fingerprint: 66DC D68F 5C1B 4D71 2EE5 BD03 8A00 786C C6FF 61AD
http://www.gravatar.com/avatar/c66875cda51109f76c6312f4d4743d1e.png
Rule 420: All persons more than eight miles high to leave the court.

[Pascal Hambourg]

Ian Zimmerman a écrit :

>
> Pascal> You can rewrite the TTL with iptables. Note that traceroute
> Pascal> won't work any more.
>
> Why the need for iptables?

Because it is simple and effective.

> Wouldn't it be simpler and safer to just
> tweak the value in /proc/sys/net/ipv4/ip_default_ttl on each host
> according to its hop distance from the router?

Not simpler, as this has to be done on all hosts instead of one router.
Besides, it works only for Linux hosts, and some packets may not use the
default TTL.

[ein]

On 05/01/2012 02:21 AM, Pascal Hambourg wrote:
> Hello,
>
> Mike Lovell a écrit :
>> On 2012-04-30, ein <e...@no.spam> wrote:
>>>
>>>> Ping: LAN Machine -> Debian Router = ~0.7ms
>>>> Ping: Debian Router -> Google = ~20ms
>>>> Ping: LAN Machine -> Google = ~121ms !!!
>>>
>>> Wrong!
>
> What is wrong ?

IMHO Mike's way to attempt to diagnose the problem.

>>> Please 'ping' nearest machine after your router for example your
>>> ISP's gateway or ISP's DNS servers. Please have in mind that your ISP's
>>> router have more important things to do, than respond to ICMP echo
>>> request messages.
>
> Indeed. So why do you suggest to ping the ISP's gateway, which is a
> router ? By the way, AFAICS, the OP did ping his own Debian router and
> a Google server, not any ISP's router.

Correlation is simple: more hops = more things can change while
connection or even while transmitting different packets to the same
machine in this connection. Each ICMP echo request message can go over
Internet with different path.

In my country small ISPs are usesd to queue everything and decide for u
what traffic is more important. ICMP "connections" are at the bottom of
list.

>> Interesting, much better:
>
> What is much better ?

I still think the OP ISP is doing something nasty.

Mike: can u connect to your (while observing the latency problem)
router's WAN some other device for which u have access to?
It will divide the problem to smaller parts. It will answer u that the
problem is actually in forwarding at debian router.

br.

[ein]

On 04/30/2012 08:33 PM, Rick Jones wrote:
> Chris Davies <chris-...@roaima.co.uk> wrote:
>> Are you pinging by name or by address? If it's by name can you
>> please repeat the tests with IP addresses. (This will eliminate any
>> delay caused by slow DNS lookups.)
>
> Is there a ping utility out there which puts a DNS lookup in the
> timing between the sending of the ICMP Echo Request and the receipt of
> the ICMP Echo Reply?

root@linux:~# time ping -c1 google.ch
PING google.ch (173.194.35.152) 56(84) bytes of data.
64 bytes from muc03s01-in-f24.1e100.net (173.194.35.152): icmp_req=1
ttl=49 time=48.5 ms

--- google.ch ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 48.550/48.550/48.550/0.000 ms

real 0m0.139s
user 0m0.001s
sys 0m0.001s
root@linux:~# time ping -c1 173.194.35.152
PING 173.194.35.152 (173.194.35.152) 56(84) bytes of data.
64 bytes from 173.194.35.152: icmp_req=1 ttl=49 time=47.1 ms

--- 173.194.35.152 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 47.155/47.155/47.155/0.000 ms

real 0m0.048s
user 0m0.000s
sys 0m0.001s

:D

[Rick Jones]

That is the overall difference in how long the ping command takes to
run, but there is no difference in the RTTs of the actual ICMP Echo
Request/Reply pairs. IE, there is a DNS lookup before the ICMP Echo
Request is sent, and one after the ICMP Echo Reply is received, but
they are outside the timestamping inside ping to measure the RTTs.

Roughly speaking:

DNS lookup name
Get starting timestamp
Send ICMP Echo Request
...time passes...
Receive ICMP Echo Reply
Get ending timestamp
DNS lookup source IP of ICMP Echo Reply

So, DNS delays can cause even a well-written ping utility to take
longer to execute overall, but they will not cause the RTTs reported
by that utility to be longer.

rick jones
--
oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag

[Ralph Spitzner]

Pascal Hambourg wrote:
>
>> --- 68.86.118.57 ping statistics ---
>
> What is that address ?

Comcast Cable Communications, Inc. JUMPSTART-2 (NET-68-80-0-0-1)
68.80.0.0 - 68.87.255.255
Comcast Cable Communications, Inc. COMCAST-16 (NET-68-86-64-0-1)
68.86.64.0 - 68.86.127.255

As told by `whois` :-)

To the OP: It's always a good idea to ping an address you
a) know
b) know the distance to
c) maybe use `traceroute -n` instead

As the ghoul for instance has many IP addresses for the same
domai name, so the DNS is not the problem but just by pinging
e.g. google.com you're probably pinging a dozen different IP's...

No offense, just my 0.02 Drachmen :-P

-rasp



--
See why I hate Windows users?
All pain, no gain.
-Howard Chu

[Ralph Spitzner]

Mike Lovell wrote:
[....]

>
> Yer you're right, I jumped the gun. It's gone back up now, must have
> been a fluke. It was lower right after coming up from a reboot.

That's a hint.
I'm on 'Kabel Deutschland' and get about a zillion ARP requests
a day, sometimes this even f**ks up the cable modem (EPC3212).
I usually just 'reboot' the modem then....

[Wolfram Gloger]

Mike Lovell <mike....@null.local> writes:

> The relevant forwarding/NAT rules are:
>
>
> #iptables -A FORWARD -i wan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

I find this rule strange. Did you get it from any standard cookbook?
It will necessitate connection tracking for any packet routed back to
your LAN. It's quite improbable that this can account for 100ms
delay, but could you just as an experiment replace this by:

iptables -A FORWARD -i wan0 -o eth0 -j ACCEPT

Regards,
Wolfram.