Re: TCP options order changed in FreeBSD 7, incompatible with some
s3raphi
I upgraded many web servers to FreeBSD 7.0-Release several weeks ago. These
servers serve hundreds of thousands of users. Since then, we have had many
users complain that they cannot connect to these servers any more. This was
a very tricky problem to diagnose, but using packet captures on both the
servers and the clients who have the problem I ended up with the same
results as the original poster. The user can ping the server with ICMP. The
user cannot complete a TCP connection.
Client sends SYN to server
Server responds SYN/ACK
Client packet capture does not show the SYN/ACK arrive.
Connection fails.
The windows client was running wireshark.
This problem is specific to windows, but also the network it is on or
devices it goes through. The same user experiencing the problem tried to
connect using a mac, and the problem does not manifest itself. Both the mac
and the windows pc were on the same network, behind the same SOHO router,
same ISP, and talking to the same FreeBSD7.0-RELEASE server.
Baffled by what the problem could have been, I stood up one of the old
FreeBSD 6.1 servers which had not yet been replaced with FreeBSD7. The user
has no trouble at all accessing the FreeBSD 6.1 server.
More interesting info:
-This makes it look like windows:
Fails: WindowsXPpro PC -> SOHO -> ISP -> Internet -> MyDataCenter ->
FreeBSD7
Works: MacBook -> SOHO -> ISP -> Internet -> MyDataCenter -> FreeBSD7
-This makes it look like the network(router/firewall/etc..):
If the WindowsPC connects to our office VPN, the connection to the FreeBSD7
server will work without issue.
The problem is specific to some combination of Windows and networks or
network devices. I have seen users on many different ISPs, and with many
different flavors of routers/firewalls.
-The problem only effects a small percentage of our users. Most of our
Windows users have no issue.
This is a very serious problem for anyone using FreeBSD7 in production as an
internet facing server as a huge percentage of clients will be windows, and
a percentage of those users will no longer be able to use your web services.
Can the patch be made available to freebsd-update?
-Seraphi
Matt Reimer wrote:
>
> On Thu, Mar 20, 2008 at 7:09 PM, d.s. al coda <coda.t...@gmail.com>
> wrote:
>> On 3/12/08, Andre Oppermann <an...@freebsd.org> wrote:
>>
>> >
>>
>> > I'd be very interesting to know the exactly models and their firmware
>> > version
>> > of the affected routers. If available locally I'd like to obtain a
>> > similar
>> > model myself for future regression tests.
>>
>>
>> Here are the models we managed to hear about via email:
>> D-Link WBR-1310
>> Linksys WCG200 (with firewall enabled)
>> Encore Broadband Router
>> Linksys WAG354G
>> Ambit U10C019
>> Netgear CG814GCMR
>
> I've seen this on a Netgear CG814WG.
>
> Matt
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
>
>
--
View this message in context: http://www.nabble.com/TCP-options-order-changed-in-FreeBSD-7%2C-incompatible-with-some-routers-tp15996110p16497816.html
Sent from the freebsd-net mailing list archive at Nabble.com.
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
Bjoern A. Zeeb
Hi,
> I believe this has already been covered in quite some depth
> and iirc its regards the ordering of the new tcp flags
> introduced in 7. Best to have a look in the list archives for
> the specifics.
so as more users come and see this I am still trying to identify that
it's really the ordering of tcp options and not the bad padding.
History:
* the original problem showed up the first time
* people thought it was different option ordering
* tcp option ordering was changed back
* it was disocvered that there was bad padding after the options
* unfortunately changing back tcp options ordering hides the padding
problem as the bad padding does not occur
* padding fix was comitted
* both changes have been MFCed to 7-STABLE.
What you can do:
I am talking to someone else but the more people with resources could
verify this the next days/weeks the more confident we can be.
If you are running 7.0-RELEASE please apply this patch:
1.141.2.4 +10 -2 src/sys/netinet/tcp_output.c << that's the padding fix
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_output.c.diff?r1=1.141.2.3;r2=1.141.2.4
rebuild your kernel, test with your users if things work ok now. If
they do (for all of them but the usual noise;) let us know.
If it does not help, apply the following patch as well and everything should be
fine (do not use TCP-MD5 as changing the order introduced another bug).
1.157.2.2 +5 -2 src/sys/netinet/tcp_var.h << that's the ordering fix
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_var.h.diff?r1=1.157.2.1;r2=1.157.2.2
In case you do not have the resources either directly apply both
patches or go to 7-STABLE (and do not use TCP-MD5 either).
There are no binary updates for this yet.
> ----- Original Message ----- From: "s3raphi" <seraph...@gmail.com>
>>
>> I upgraded many web servers to FreeBSD 7.0-Release several weeks ago. These
>> servers serve hundreds of thousands of users. Since then, we have had many
>> users complain that they cannot connect to these servers any more. This was
>> a very tricky problem to diagnose, but using packet captures on both the
>> servers and the clients who have the problem I ended up with the same
>> results as the original poster. The user can ping the server with ICMP. The
>> user cannot complete a TCP connection.
>> Client sends SYN to server
>> Server responds SYN/ACK
>> Client packet capture does not show the SYN/ACK arrive.
>> Connection fails.
> ....
what would also be interesting to know is:
- Did the (SOHO) router drop it or the windows (XP) or another OS
like MAC OSX, linux, or ...
- Which (SOHO) router - if you think this is the culprit - vendor/model
- If it was the windows, and it was XP, was the XP firewall on?
- If it was the windows, are there any other 3rd party firewalls
running on it.
Someone may not understand why all these questions are important if
there is a patch available already:
a) finding out which change broke things is good for documentation
purposes. Especially if it would be the tcp options ordering
b) if it wasn't options ordering we might want to go back to the
more effective version
Bjoern
PS: you will also find tcpdump patches to show the tcp options and the
bad padding in this thread:
http://lists.freebsd.org/pipermail/freebsd-net/2008-March/017267.html
--
Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT
Software is harder than hardware so better get it right the first time.
Anders Nordby
I had the same problem, and temporarily worked around it by disabling
SACK:
sysctl net.inet.tcp.sack.enable=0
Which solved my problems. It would be interesting to see if this helps
you also?
If so, it seems this issue is related to SACK and TCP order maybe? Hmm.
On Fri, Apr 04, 2008 at 01:14:19PM -0700, s3raphi wrote:
> I upgraded many web servers to FreeBSD 7.0-Release several weeks ago. These
> servers serve hundreds of thousands of users. Since then, we have had many
> users complain that they cannot connect to these servers any more. This was
> a very tricky problem to diagnose, but using packet captures on both the
> servers and the clients who have the problem I ended up with the same
> results as the original poster. The user can ping the server with ICMP. The
> user cannot complete a TCP connection.
> Client sends SYN to server
> Server responds SYN/ACK
> Client packet capture does not show the SYN/ACK arrive.
> Connection fails.
>
> > _______________________________________________
> > freeb...@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
> >
> >
>
> --
> View this message in context: http://www.nabble.com/TCP-options-order-changed-in-FreeBSD-7%2C-incompatible-with-some-routers-tp15996110p16497816.html
> Sent from the freebsd-net mailing list archive at Nabble.com.
>
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"
--
Anders.
Bjoern A. Zeeb
Hi,
> I had the same problem, and temporarily worked around it by disabling
> SACK:
>
> sysctl net.inet.tcp.sack.enable=0
>
> Which solved my problems. It would be interesting to see if this helps
> you also?
This will hide any of the real problems in most cases, not fix them.
> If so, it seems this issue is related to SACK and TCP order maybe? Hmm.
So you hadn't been following the threads here the last 6 weeks?
7-STABLE already has patches. See my other mails in here....
--
Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT
Software is harder than hardware so better get it right the first time.
Anders Nordby
On Wed, Apr 09, 2008 at 08:23:13AM +0000, Bjoern A. Zeeb wrote:
>> I had the same problem, and temporarily worked around it by disabling
>> SACK:
> ok, here we go...
>
> as I had explained on freebsd-net@ lately there had been 2 changes to
> possibly fix this issue.
>
> I am currently in the process to find out which of the two is actually
> needed and which of the two prior versions caused the problems.
>
> I have a testbed setup and will have it running for another day or so.
>
> So could you try to reproduce your problems with the following three
> servers (if you could ask your affected customers to test and
> report back to you that would be highly appreciated as well).
I just got access to a remote PC which has the problem, finally. It runs
Windows 2000, service pack 4.
> Simply open the following URLs in a browser is enough. If you can
> see the pages, fine. If you cannot, tell me which worked, which
> didn't.
>
> http://tcptest1.sbone.de/
This URL does not work, I get "siden kan ikke vises" (page can not be
displayed in Norwegian).
This URL works fine, I get the FreeBSD 7.0 TCP options test page.
This URL does not work, I get "siden kan ikke vises" (page can not be
displayed in Norwegian).
> I have a tcpdump running on my side so all connections will be fully
> logged.
Good.
Your patch on
http://sources.zabbadoz.net/freebsd/patchset/20080309-01-tcp-options-padding.diff
fixes the problem for me, with SACK still being on as it is by default.
I would be much happy if this fix could find it's way to RELENG_7_0 and
errata.
Thanks a lot!
Cheers,
--
Anders.
Bjoern A. Zeeb
Hi,
in case you did not receive a private mail from me please do not go to
any of the mentioned urls or you will make a one week data collection
worthless.
Thank you.
PS: the webservers will go away within a few hours anyway...
>> Simply open the following URLs in a browser is enough. If you can
>> see the pages, fine. If you cannot, tell me which worked, which
>> didn't.
>>
>> http://tcptest1.sbone.../
>> http://tcptest2.sbone.../
>> http://tcptest3.sbone.../
--
Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT
Software is harder than hardware so better get it right the first time.