freebsd-pf Digest, Vol 269, Issue 5
freebsd-p...@freebsd.org
freeb...@freebsd.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
or, via email, send a message with subject or body 'help' to
freebsd-p...@freebsd.org
You can reach the person managing the list at
freebsd-...@freebsd.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of freebsd-pf digest..."
Today's Topics:
1. sending mail with attachments always fails (FreeBSD/pf)
(Victor Lyapunov)
2. Re: sending mail with attachments always fails (FreeBSD/pf)
(olli hauer)
3. Re: sending mail with attachments always fails (FreeBSD/pf)
(Victor Lyapunov)
4. Re: sending mail with attachments always fails (FreeBSD/pf)
(Michael Proto)
5. Re: sending mail with attachments always fails (FreeBSD/pf)
(Michael Proto)
6. Re: sending mail with attachments always fails (FreeBSD/pf)
(olli hauer)
7. Re: sending mail with attachments always fails (FreeBSD/pf)
(David DeSimone)
8. Re: sending mail with attachments always fails (FreeBSD/pf)
(Victor Lyapunov)
----------------------------------------------------------------------
Message: 1
Date: Sat, 21 Nov 2009 21:06:50 +0600
From: Victor Lyapunov <fullbla...@gmail.com>
Subject: sending mail with attachments always fails (FreeBSD/pf)
To: freeb...@freebsd.org
Message-ID:
<6c51dbb10911210706g349...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi all,
I have production network with FreeBSD box acting as firewall. The
problem emerge as soon as users send mail with attachments. (Sending
mail without attachments always succeeds). Basically, when a user
tries to send a message, only part of it transmitted before connection
is interrupted and sending fails. The problem persists only when pf is
enabled.
My ruleset:
scrub in all fragment reassemble
block drop on em0 all
pass inet proto tcp from 192.168.0.0/24 to any port = smtp flags S/SA keep state
pass inet proto tcp from 192.168.0.0/24 to any port = pop3 flags S/SA keep state
pass inet proto tcp from 192.168.0.0/24 to any port = imap flags S/SA keep state
pass inet proto tcp from 192.168.0.0/24 to any port = smtps flags S/SA
keep state
pass inet proto tcp from 192.168.0.0/24 to any port = pop3s flags S/SA
keep state
pass proto udp from any to any port = domain keep state
This is what i get from pfctl -si just after #/etc/rc.d/pf start
# pfctl -si
Status: Enabled for 0 days 00:00:09 Debug: Urgent
State Table Total Rate
current entries 0
searches 0 0.0/s
inserts 0 0.0/s
removals 0 0.0/s
Counters
match 0 0.0/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
After I try to send some mail with attachments a couple of times(which
always fail), i get this from pfctl -si:
Status: Enabled for 0 days 00:02:58 Debug: Urgent
State Table Total Rate
current entries 48
searches 1313 7.4/s
inserts 131 0.7/s
removals 83 0.5/s
Counters
match 152 0.9/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 22 0.1/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
Any suggestions/ideas would be appreciated,
Best regards,
Victor
FreeBSD router 7.2-RELEASE FreeBSD 7.2-RELEASE #4: Sun May 3 23:29:04
2009 root@router:/usr/obj/usr/src/sys/GENERIC i386
------------------------------
Message: 2
Date: Sat, 21 Nov 2009 18:27:30 +0100
From: olli hauer <oha...@gmx.de>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: Victor Lyapunov <fullbla...@gmail.com>
Cc: freeb...@freebsd.org
Message-ID: <4B082302...@gmx.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Victor Lyapunov wrote:
> Hi all,
>
> I have production network with FreeBSD box acting as firewall. The
> problem emerge as soon as users send mail with attachments. (Sending
> mail without attachments always succeeds). Basically, when a user
> tries to send a message, only part of it transmitted before connection
> is interrupted and sending fails. The problem persists only when pf is
> enabled.
>
> My ruleset:
> scrub in all fragment reassemble
> block drop on em0 all
> pass inet proto tcp from 192.168.0.0/24 to any port = smtp flags S/SA keep state
> pass inet proto tcp from 192.168.0.0/24 to any port = pop3 flags S/SA keep state
> pass inet proto tcp from 192.168.0.0/24 to any port = imap flags S/SA keep state
> pass inet proto tcp from 192.168.0.0/24 to any port = smtps flags S/SA
> keep state
> pass inet proto tcp from 192.168.0.0/24 to any port = pop3s flags S/SA
> keep state
> pass proto udp from any to any port = domain keep state
>
[...]
Is this only for client submitting (imap) or even for incoming mails
from outside via smtp?
What about outgoing traffic from the machine?
Try the following in pf.conf to see why this happens
set loginterface pflog0
set block-policy drop
set skip on lo0
block drop in log on em0 all
pass out log on em0 all
Now use tcpdump to see which rule drops the traffic
#> tcpdump -net -i pflog0
--
olli
------------------------------
Message: 3
Date: Sun, 22 Nov 2009 00:07:14 +0600
From: Victor Lyapunov <fullbla...@gmail.com>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: olli hauer <oha...@gmx.de>, freeb...@freebsd.org
Message-ID:
<6c51dbb10911211007x4ea...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Thanks for your answer, olli.
As i send mail not from my freebsd server, but rather from the clients
on the local network, so here's what i did:
my pf.conf:
set loginterface pflog0
set block-policy drop
set skip on lo0
block drop log on em0 all
pass log inet proto tcp from 192.168.0.0/24 to any port {smtp, pop3,
imap, smtps, pop3s} flags S/SA keep state
pass log proto udp from any to any port = domain keep state
# tcpdump -net -i pflog0
Now i went to a windows computer and tried to send an email with
attachment to gmail.com (sending failed at 2%)
here's what i got in my pflog:
rule 4/0(match): pass in on em0: (tos 0x0, ttl 128, id 19860, offset
0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
209.85.129.111.465: [|tcp]
rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
209.85.129.111.465: tcp 28 [bad hdr length 0 - too short, < 20]
2 packets captured
2 packets received by filter
0 packets dropped by kernel
Again, everything works just fine when pf is disabled.
And there's no problems for incoming traffic, only outgoing traffic
gets corrupted( that is, dropped in the middle of transmitting data).
Any ideas?
2009/11/21 olli hauer <oha...@gmx.de>:
> Victor Lyapunov wrote:
>>
>> Hi all,
>>
>> I have production network with FreeBSD box acting as firewall. The
>> problem emerge as soon as users send mail with attachments. (Sending
>> mail without attachments always succeeds). Basically, when a user
>> tries to send a message, only part of it transmitted before connection
>> is interrupted and sending fails. The problem persists only when pf is
>> enabled.
>>
>> My ruleset:
>> scrub in all fragment reassemble
>> block drop on em0 all
>> pass inet proto tcp from 192.168.0.0/24 to any port = smtp flags S/SA keep
>> state
>> pass inet proto tcp from 192.168.0.0/24 to any port = pop3 flags S/SA keep
>> state
>> pass inet proto tcp from 192.168.0.0/24 to any port = imap flags S/SA keep
>> state
>> pass inet proto tcp from 192.168.0.0/24 to any port = smtps flags S/SA
>> keep state
>> pass inet proto tcp from 192.168.0.0/24 to any port = pop3s flags S/SA
>> keep state
>> pass proto udp from any to any port = domain keep state
>>
>
> [...]
>
> Is this only for client submitting (imap) or even for incoming mails from
> outside via smtp?
>
> What about outgoing traffic from the machine?
>
> Try the following in pf.conf to see why this happens
>
> set loginterface pflog0
> set block-policy drop
> set skip on lo0
>
> block drop in log on em0 all
> pass out log on em0 all
>
> Now use tcpdump to see which rule drops the traffic
> #> tcpdump -net -i pflog0
>
> --
> olli
>
------------------------------
Message: 4
Date: Sat, 21 Nov 2009 13:23:58 -0500
From: Michael Proto <mi...@jellydonut.org>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: Victor Lyapunov <fullbla...@gmail.com>
Cc: freeb...@freebsd.org
Message-ID:
<1de79840911211023n165...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On Sat, Nov 21, 2009 at 1:07 PM, Victor Lyapunov
<fullbla...@gmail.com> wrote:
> rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
> 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
> 209.85.129.111.465: �tcp 28 [bad hdr length 0 - too short, < 20]
This looks to be your problem-- bad hdr length 0. I don't know enough
of what mailer(s) you're using to relay this message outbound, but
since port 465 is smtp over TLS/SSL are you sure your smtp encryption
is working correctly? I often see these types of errors with other
TLS/SSL apps when one side is expecting an encrypted connection and
the other is not (correctly) providing it.
Have you tried using unencrypted smtp on port 25? Does that work?
-Proto
------------------------------
Message: 5
Date: Sat, 21 Nov 2009 13:27:08 -0500
From: Michael Proto <mi...@jellydonut.org>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: Victor Lyapunov <fullbla...@gmail.com>
Cc: freeb...@freebsd.org
Message-ID:
<1de79840911211027mbc0...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On Sat, Nov 21, 2009 at 1:23 PM, Michael Proto <mi...@jellydonut.org> wrote:
> On Sat, Nov 21, 2009 at 1:07 PM, Victor Lyapunov
> <fullbla...@gmail.com> wrote:
>
>> rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
>> 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
>> 209.85.129.111.465: �tcp 28 [bad hdr length 0 - too short, < 20]
>
> This looks to be your problem-- bad hdr length 0. I don't know enough
> of what mailer(s) you're using to relay this message outbound, but
> since port 465 is smtp over TLS/SSL are you sure your smtp encryption
> is working correctly? I often see these types of errors with other
> TLS/SSL apps when one side is expecting an encrypted connection and
> the other is not (correctly) providing it.
>
> Have you tried using unencrypted smtp on port 25? Does that work?
>
Er... wait, I just re-read that you said things work fine with pf
disabled, so my theory about bad encryption probably isn't very
accurate. Are you still using a scrub rule? Have you tried disabling
it? If pf is seeing a "bad hdr length" error it might be dropping the
packet due to scrubbing. Of course, this could also mean that TSO is
enabled on your ethernet interface and bpf just isn't seeing the tcp
header at all, so my whole theory might be moot.
-Proto
------------------------------
Message: 6
Date: Sat, 21 Nov 2009 20:25:17 +0100
From: olli hauer <oha...@gmx.de>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: Victor Lyapunov <fullbla...@gmail.com>
Cc: freeb...@freebsd.org
Message-ID: <4B083E9D...@gmx.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Victor Lyapunov wrote:
> Thanks for your answer, olli.
>
> As i send mail not from my freebsd server, but rather from the clients
> on the local network, so here's what i did:
>
> my pf.conf:
> set loginterface pflog0
> set block-policy drop
> set skip on lo0
> block drop log on em0 all
> pass log inet proto tcp from 192.168.0.0/24 to any port {smtp, pop3,
> imap, smtps, pop3s} flags S/SA keep state
> pass log proto udp from any to any port = domain keep state
>
>
> # tcpdump -net -i pflog0
> Now i went to a windows computer and tried to send an email with
> attachment to gmail.com (sending failed at 2%)
>
> here's what i got in my pflog:
>
> rule 4/0(match): pass in on em0: (tos 0x0, ttl 128, id 19860, offset
> 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
> 209.85.129.111.465: [|tcp]
> rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
> 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
> 209.85.129.111.465: tcp 28 [bad hdr length 0 - too short, < 20]
why is the [DF] bit set?
can you try with the following pf option
scrub all no-df
------------------------------
Message: 7
Date: Sat, 21 Nov 2009 20:23:47 -0600
From: "David DeSimone" <f...@verio.net>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: <freeb...@freebsd.org>
Message-ID: <2009112202...@verio.net>
Content-Type: text/plain; charset="Windows-1252"
Michael Proto <mi...@jellydonut.org> wrote:
>
> > rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
> > 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
> > 209.85.129.111.465: tcp 28 [bad hdr length 0 - too short, < 20]
>
> This looks to be your problem-- bad hdr length 0.
This is caused when tcpdump has too small a snaplen; it is not seeing
enough of the packet from the pflog interface, so it reports incorrect
information at the end.
Try adding "-s 128" to collect a larger packet and you should see the
full description from tcpdump.
That said, the original problem seems like it could easily be caused by
a PF state mismatch resulting from assymetric routing. If packets come
in a different interface than they go out, or worse, if the return path
doesn't even go through the firewall, PF cannot see the reply traffic
allowing it to update its TCP window tracking.
As a result, short TCP sessions, such as those that fit within the
default TCP window, can work okay, but longer sessions that go beyond
that window will stall out and fail.
--
David DeSimone == Network Admin == f...@verio.net
"I don't like spinach, and I'm glad I don't, because if I
liked it I'd eat it, and I just hate it." -- Clarence Darrow
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you.
------------------------------
Message: 8
Date: Sun, 22 Nov 2009 14:36:59 +0600
From: Victor Lyapunov <fullbla...@gmail.com>
Subject: Re: sending mail with attachments always fails (FreeBSD/pf)
To: freeb...@freebsd.org
Message-ID:
<6c51dbb10911220036x55b...@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Thank you guys for your attention to my problem.
This time i increased the tcpdump capture buffer to 128 bytes and i got this:
# tcpdump -s 128 -net -i pflog0
(I tried to send mail with an attachment(700kb) to gmail.com(REQUIRES
SSL) using outlook, which again timeout- failed)
rule 0/0(match): block in on em0: 192.168.0.1.2078 > 192.168.0.3.445:
P 794764624:794764677(53) ack 146734048 win 65535
rule 0/0(match): block in on em0: 192.168.0.1.2078 > 192.168.0.3.445:
P 0:53(53) ack 1 win 65535
rule 0/0(match): block in on em0: 192.168.0.1.2078 > 192.168.0.3.445:
P 0:53(53) ack 1 win 65535
rule 0/0(match): block in on em0: 192.168.0.1.2078 > 192.168.0.3.445:
P 0:53(53) ack 1 win 65535
rule 1/0(match): pass in on em0: 192.168.0.5.1025 > 192.168.0.3.53:
1016+ A? smtp.gmail.com. (32)
rule 1/0(match): pass out on em0: 192.168.0.3.61974 >
208.67.222.222.53: 44197+% [1au] A? smtp.gmail.com. (43)
rule 1/0(match): pass out on em0: 192.168.0.3.53758 >
208.67.222.222.53: 57704+% [1au][|domain]
rule 1/0(match): pass in on em0: 192.168.0.5.2029 > 74.125.39.109.465:
S 207714378:207714378(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass out on em0: 192.168.0.5.2029 >
74.125.39.109.465: S 207714378:207714378(0) win 65535 <mss
1460,nop,nop,sackOK>
rule 1/0(match): pass out on em0: 192.168.0.3.55398 >
208.67.222.222.53: 26150+% [1au][|domain]
rule 0/0(match): block in on em0: 192.168.0.1.2078 > 192.168.0.3.445:
P 0:53(53) ack 1 win 65535
rule 1/0(match): pass in on em0: 192.168.0.1.2437 > 192.168.0.3.445: S
3245362396:3245362396(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass in on em0: 192.168.0.1.2442 > 192.168.0.3.445: S
3154965483:3154965483(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass in on em0: 192.168.0.1.2444 > 192.168.0.3.445: S
3857149154:3857149154(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass in on em0: 169.254.113.220.2447 >
192.168.0.3.139: S 4208647498:4208647498(0) win 65535 <mss
1460,nop,nop,sackOK>
rule 1/0(match): pass in on em0: 192.168.0.1.2448 > 192.168.0.3.139: S
3459916613:3459916613(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass in on em0: 169.254.113.220.2449 >
192.168.0.3.139: S 2672892612:2672892612(0) win 65535 <mss
1460,nop,nop,sackOK>
17 packets captured
17 packets received by filter
0 packets dropped by kernel
After that i tried to send mail to a server that does not require ssl
and i got this:
rule 1/0(match): pass in on em0: 192.168.0.5.2035 > 94.100.177.1.25: S
237079791:237079791(0) win 65535 <mss 1460,nop,nop,sackOK>
rule 1/0(match): pass out on em0: 192.168.0.5.2035 > 94.100.177.1.25:
S 237079791:237079791(0) win 65535 <mss 1460,nop,nop,sackOK>
2 packets captured
2 packets received by filter
0 packets dropped by kernel
The sending process fails regardless of whether i use SSL or not.
192.168.0.1 -- Router
192.168.0.3 -- The FreeBSD box
192.168.0.5 -- Windows machine with default gateway set to 192.168.0.3
The ruleset is:
block drop log on em0 all
pass log on em0 all flags S/SA keep state
I can't figure out what might be the cause of the problem... Is it
possible that the router causes this?
2009/11/22 David DeSimone <f...@verio.net>:
> Michael Proto <mi...@jellydonut.org> wrote:
>>
>> > rule 4/0(match): pass out on em0: (tos 0x0, ttl 127, id 19860, offset
>> > 0, flags [DF], proto TCP (6), length 48) 192.168.0.5.1822 >
>> > 209.85.129.111.465: �tcp 28 [bad hdr length 0 - too short, < 20]
>>
>> This looks to be your problem-- bad hdr length 0.
>
> This is caused when tcpdump has too small a snaplen; it is not seeing
> enough of the packet from the pflog interface, so it reports incorrect
> information at the end.
>
> Try adding "-s 128" to collect a larger packet and you should see the
> full description from tcpdump.
>
>
> That said, the original problem seems like it could easily be caused by
> a PF state mismatch resulting from assymetric routing. �If packets come
> in a different interface than they go out, or worse, if the return path
> doesn't even go through the firewall, PF cannot see the reply traffic
> allowing it to update its TCP window tracking.
>
> As a result, short TCP sessions, such as those that fit within the
> default TCP window, can work okay, but longer sessions that go beyond
> that window will stall out and fail.
>
> --
> David DeSimone == Network Admin == f...@verio.net
> �"I don't like spinach, and I'm glad I don't, because if I
> � liked it I'd eat it, and I just hate it." -- Clarence Darrow
>
>
> This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. �Thank you.
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-...@freebsd.org"
>
------------------------------
End of freebsd-pf Digest, Vol 269, Issue 5
******************************************