Same tcp connections twice in ipfstat -t output

43 views
Skip to first unread message

Rene van Hoek

unread,
Jun 15, 2008, 4:10:04 PM6/15/08
to
Hello,

I am using IPF v4.1.28 on FreeBSD7. The firewall is working stable and
does what is is supposed to do. So no problems there.

The following however, I don't expect: In the ipfstat -t output I see
the same connections (source-ip, port <--> destination-ip, port) twice.
For example (part of output ipfstat -t):

Source IP Destination IP ST PR #pkts
#bytes ttl
80.60.81.93,1363 195.86.22.59,587 B/6 tcp 173
202746 0:13
80.57.132.26,60464 195.86.22.53,22 4/4 tcp 2393 147824
119:59:59
80.60.81.93,1363 195.86.22.59,587 B/6 tcp 88
101445 0:13
80.57.132.26,60477 195.86.22.59,22 4/4 tcp 1077 64400
119:59:47 (*)
77.162.155.20,49808 195.86.22.50,80 4/4 tcp 203 54140
119:59:17
77.162.155.20,49807 195.86.22.50,80 4/4 tcp 173 45966
119:59:16
80.57.132.26,56603 195.86.22.50,80 4/4 tcp 429
45716 96:09:25
78.171.174.130,1675 195.86.22.54,80 4/4 tcp 145
45292 90:04:42
85.147.196.239,54166 195.86.22.52,80 4/4 tcp 95 34286
119:57:45
83.82.139.218,51157 195.86.22.50,80 B/4 tcp 153
33210 0:12
80.57.132.26,60477 195.86.22.59,22 4/4 tcp 540 32296
119:59:47 (*)

Marked with * is twice.

The output of ipfstat is:

IP states added:
1862533 TCP
523994 UDP
0 ICMP
49403681 hits
9612162 misses
0 bucket full
0 maximum rule references
0 maximum
0 no memory
1231 bkts in use
2496 active
523940 expired
1860091 closed
State logging enabled

State table bucket statistics:
1231 in use
49% hash efficiency
1.89% bucket usage
0 minimal length
4 maximal length
2.028 average length

TCP Entries per state
0 1 2 3 4 5 6 7 8 9 10
11
0 0 24 0 1017 556 12 0 10 0 332
491

In this output I see that 1231 buckets are in use. Does that mean that
there are 1231 connections for which state-informattion is kept in
memory?
I see that there are 2496 'active'. Does that mean that there are 2496
hashes which point too the 1231 connections? Is that the (1231/2496) =
49% hash efficiency?

So does ipfstat -t takes the hash-entries and shows the information
found in the buckets? Does that explain why the output of ipfstat -t
shows connections twice?
Is this behavior by design or should I worry about it?

Greetings,

Rene van Hoek

Darren Reed

unread,
Jun 15, 2008, 11:55:14 PM6/15/08
to

No. It is hash table terminology.

> I see that there are 2496 'active'. Does that mean that there are 2496
> hashes which point too the 1231 connections? Is that the (1231/2496) =
> 49% hash efficiency?

No and yes.

> So does ipfstat -t takes the hash-entries and shows the information
> found in the buckets? Does that explain why the output of ipfstat -t
> shows connections twice?
> Is this behavior by design or should I worry about it?

hmmm... so it could be the mechanism used to get state entries out
of the kernel is walking through a very active list and that it changes
between the first and the n-th, displaying an entry twice.

Darren

Rene van Hoek

unread,
Jun 16, 2008, 1:42:37 AM6/16/08
to

Hi,

I took the output of ipfstat -sl, to see the current states. I see the
same source-ip, port <--> destination-ip, port connections twice. For
example:

82.35.175.131 -> 213.201.199.243 pass 0x40004502 pr 6 state 11/4
tag 0 ttl 575536
1201 -> 80 d422a986:43c21a31 65535<<0:65535<<0
cmsk 0000 smsk 0000 s0 d422a8f2/43c14142
FWD:ISN inc 0 sumd 0
REV:ISN inc 0 sumd 0
forward: pkts in 21 bytes in 1000 pkts out 22 bytes out 1048
backward: pkts in 40 bytes in 57143 pkts out 40 bytes out 57143
pass out quick keep state IPv4
pkt_flags & 0(10000) = 1000, pkt_options & ffffffff = 0, ffffffff = 0
pkt_security & ffff = 0, pkt_auth & ffff = 0
is_flx 0x1 0x1 0x1 0x1
interfaces: in X[em0],X[bge0] out X[bge0],X[bridge0]
Sync status: not synchronized
82.35.175.131 -> 213.201.199.243 pass 0x40008502 pr 6 state 11/4
tag 0 ttl 575536
1201 -> 80 d422a986:43c21a31 65535<<0:65535<<0
cmsk 0000 smsk 0000 s0 d422a8f2/43c14142
FWD:ISN inc 0 sumd 0
REV:ISN inc 0 sumd 0
forward: pkts in 1 bytes in 48 pkts out 22 bytes out 1048
backward: pkts in 40 bytes in 57143 pkts out 40 bytes out 57143
pass in quick keep state IPv4
pkt_flags & 0(10000) = 1000, pkt_options & ffffffff = 0, ffffffff = 0
pkt_security & ffff = 0, pkt_auth & ffff = 0
is_flx 0x1 0x1 0x1 0x1
interfaces: in X[em0],X[bridge0] out X[bridge0],X[em0]
Sync status: not synchronized

These are the same connection listed twice. What I notice is the
different list of interfaces in the two states. My ifconfig output is
as follows:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=198<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:15:17:75:ab:84
inet 195.86.22.53 netmask 0xfffffff0 broadcast 195.86.22.63
media: Ethernet autoselect (100baseTX <half-duplex>)
status: active
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
ether 00:15:17:75:ab:85
media: Ethernet autoselect
status: no carrier
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1e:c9:bb:7f:fd
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:1e:c9:bb:7f:fe
media: Ethernet autoselect (none)
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500
ether 32:39:9f:e0:10:a3
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

Interface em0 is connected too the internet. bge0 is through a Cisco
switch connected to our servers.

Rene van Hoek

unread,
Jun 16, 2008, 2:05:21 AM6/16/08
to

On Jun 16, 2008, at 7:42 AM, Rene van Hoek wrote:

>
> On Jun 16, 2008, at 5:55 AM, Darren Reed wrote:
>
>> Rene van Hoek wrote:
>>>

<cut>

The output of sysctl net.link.bridge is as follows:

net.link.bridge.ipfw: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1
net.link.bridge.pfil_bridge: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_onlyip: 1

According to the FreeBSD manual page IF_BRIDGE(4) (quote):

'net.link.bridge.pfil_member Set to 1 to enable filtering on the
incoming
and outgoing member interfaces, set to 0 to
disable it.

net.link.bridge.pfil_bridge Set to 1 to enable filtering on the
bridge
interface, set to 0 to disable it.

'

Should I set net.link.bridge.pfil_bridge to 0 and
net.link.bridge.pfil_member to 1 ?

Greetings,

Rene van Hoek

Rene van Hoek

unread,
Jun 16, 2008, 9:54:31 AM6/16/08
to
On Jun 16, 2008, at 8:05 AM, Rene van Hoek wrote:

>
> On Jun 16, 2008, at 7:42 AM, Rene van Hoek wrote:
>
>>
>> On Jun 16, 2008, at 5:55 AM, Darren Reed wrote:
>>
>>> Rene van Hoek wrote:
>>>>
>
> <cut>
>>

>> <cut>


> The output of sysctl net.link.bridge is as follows:
>
> net.link.bridge.ipfw: 0
> net.link.bridge.log_stp: 0
> net.link.bridge.pfil_local_phys: 0
> net.link.bridge.pfil_member: 1
> net.link.bridge.pfil_bridge: 1
> net.link.bridge.ipfw_arp: 0
> net.link.bridge.pfil_onlyip: 1
>
> According to the FreeBSD manual page IF_BRIDGE(4) (quote):
>
> 'net.link.bridge.pfil_member Set to 1 to enable filtering on the
> incoming
> and outgoing member interfaces, set to 0 to
> disable it.
>
> net.link.bridge.pfil_bridge Set to 1 to enable filtering on the
> bridge
> interface, set to 0 to disable it.
>
> '
>
> Should I set net.link.bridge.pfil_bridge to 0 and
> net.link.bridge.pfil_member to 1 ?
>
> Greetings,
>
> Rene van Hoek
>

Hi,

I setup a test-environment to diagnose this issue.

A FreeBSD 7 RELEASE system with IP Filter: v4.1.28, configured as
bridge:

evil client 192.168.0.105 <-----> xl0 - firewall - xl1 <----> server
192.168.0.207

The network-interfaces configuration is:

$ ifconfig
xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=9<RXCSUM,VLAN_MTU>
ether 00:01:03:2a:5a:7b
inet 192.168.0.50 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
xl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=9<RXCSUM,VLAN_MTU>
ether 00:0b:db:25:8b:48
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0
mtu 1500


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384

inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4


inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500

ether b6:7c:78:79:32:55


id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

member: xl1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: xl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

With $ sysctl net.link.bridge


net.link.bridge.ipfw: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1
net.link.bridge.pfil_bridge: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_onlyip: 1

When I now do a ssh connection from 192.168.0.105, the ipfstat -t
output shows two connections with same source-ip,port <--> dest ip,port:

Source IP Destination IP ST PR #pkts
#bytes ttl

192.168.0.105,51128 192.168.0.50,22 4/4 tcp 2736 164096
119:59:59
192.168.0.105,51154 192.168.0.207,22 4/4 tcp 61 6808
119:59:44
192.168.0.105,51154 192.168.0.207,22 4/4 tcp 32 3500
119:59:44

With $ sysctl net.link.bridge


net.link.bridge.ipfw: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1

net.link.bridge.pfil_bridge: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_onlyip: 1


When I now do a ssh connection from 192.168.0.105, the ipfstat -t
output shows one connection with same source-ip,port <--> dest ip,port:

Source IP Destination IP ST PR #pkts
#bytes ttl

192.168.0.105,51128 192.168.0.50,22 4/4 tcp 2877 173220
119:59:59
192.168.0.105,51155 192.168.0.207,22 4/4 tcp 54 6456
119:59:43


So, that is expected behavior :-)

I think it is wise to include this in the documentation or FAQ or
something. It is pretty easy to miss on a busy firewall.

Reply all
Reply to author
Forward
0 new messages