tcp_max_syn_backlog problems

[Bryan]

I have a high traffic mail server that in the past we have adjusted the
TCP kernel setting tcp_max_syn_backlog increasing to several thousand
to handle the load. However, recently with some new intel woodcrest
xeons on CentOS 4.3 and kernels RHEL 2.6.9-22smp, 2.6.17.6 I have had
some problems with this setting.

Leaving the tcp_max_syn_backlog to default of 1024 results in the
common "drop open requests" log in dmesg. By increasing this # to say
8192 the dmesg logs stop. However, by doing a netstat -n|fgrep -ci
SYN_RECV I cannot get this number to go above 1024. When it does reach
1024 incoming tcp sockets hang and no SYN/ACK is sent to client, I
verified this by sniffing w/ tcpdump, the client will transmit and
re-transmit the SYN w/ no kernel response. As soon as SYN_RECV drops
below 1024, the kernel does respond to the client.

What's more strange is my older dual xeon dempsey hardware on same RHEL
4.2 (2.6.9-22SMP) works fine, but when using this new kernel on my
woodcrest xeons it does not.

Is there another sysctl variable out there I should be looking at?

[Bryan]

Update on this case, setting

FROM
fs.file-max = 84063

TO
fs.file-max = 1012500

seems to allow SYN_RECV to go above 1024.

I am not sure I understand the relationship here, I don't think we are
maxing out the file handles.

Currently:
cat file-nr
576 0 1012500

netstat -n | grep -ci SYN_RE
27151