I can isolate the problem to being with IPSEC as follows.
1. Turn off IPSEC. Browser loads the same page with
www.nytimes.com and
199.239.236.200.
2. Turn on IPSEC which has a general rule blocking all TCP traffic to a
subnet with a low weighting and a more specific rule permitting TCP
traffic to and from 199.239.236.200 and the same subnet with a higher
weighting. You can't allow traffic in IPSEC to a domain name, it just
makes up a list of filters to the IP addresses recorded in DNS for that
name. You can permit DNS queries though in IPSEC. When IPSEC is turned
on
www.nytimes.com and 199.239.236.200 both don't connect. Therefore
IPSEC is blocking access to this web site.
3. In IPSEC delete the general rule blocking all TCP traffic to the
subnet. Then
www.nytimes.com and 199.239.236.200 both connect again.
Therefore the general rule in IPSEC even though it has a lower
weighting (I checked this already in IP Security Monitor) is still
overriding the permit rule for 199.239.236.200. I tried varying the
specific rule to different IP addresses, and nothing helped. I tried
making the specific rule more general, so any traffic to or from
199.239.236.200 to anywhere is permitted and it was still blocked. I
tried any traffic to or from 199.239.236.200 on port 80 to a specific
subnet, but that didn't work.
As far as I can tell I am not doing anything wrong. There are only a
few settings so it is hard to make a mistake.
Here's what I think is happening. If I block all traffic with a
general rule and then permit some with a specific rule on the internal
LAN then that works. If the specific rule is for connecting to the
Internet then it doesn't work. So it seems for some reason IPSEC won't
do this for the Internet but it will on an internal network. BTW this
is with 2003 server standard with all patches up to date. I first
noticed this problem a few months ago and put it aside thinking I was
making a mistake somewhere.
The simplest way is to try it yourself. Use say a local security
policy on the gateway server, this is what I do. Block all TCP even all
traffic to a particular subnet from the gateway server. Then try and
make a specific rule to connect to the internet from that subnet.
Thanks for all the help, but I am thinking that for some reason IPSEC
just can't do this.