20999ms 21 second delays in some HTTP CONNECT (Tunnel To) sessions
1,301 views
Skip to first unread message
Zephan Schroeder
Feb 7, 2012, 8:01:45 PM2/7/12
to Fiddler, zephan.s...@philips.com
I have developed a debilitating condition on a couple machines that
has me baffled. Basically TCP/IP Connect tunnels consistently takes
exactly 20999ms (21 seconds) to establish... but only for the internal
sites I need to test (and occasionally a couple public external
sites). I think this is a magic value related to some IPv6->IPv4
fallback or similar renegotiation process, but I need some help
identifying the root cause or narrowing down where to look for the
problem (client config, corporate IT network, datacenter network, or
server config).
MORE INFORMATION
================
Classic example of an impacted HTTP CONNECT trace:
--------------------------------------------------------------
Request Count: 1
Bytes Sent: 436 (headers:436; body:0)
Bytes Received: 107 (headers:107; body:0)
ACTUAL PERFORMANCE
--------------
ClientConnected: 18:39:25.986
ClientBeginRequest: 18:39:25.987
GotRequestHeaders: 18:39:25.987
ClientDoneRequest: 18:39:25.987
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 21000ms <======= Here's the symptom
HTTPS Handshake: 1ms
ServerConnected: 18:39:46.990
FiddlerBeginRequest: 18:39:46.990
ServerGotRequest: 18:39:46.990
ServerBeginResponse: 00:00:00.000
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 00:00:00.000
ClientBeginResponse: 18:39:46.990
ClientDoneResponse: 18:39:46.990
Overall Elapsed: 00:00:21.0031001
--------------------------------------------------------------
The kicker is during login we use ADFS for authentication which routes
to two other internal servers which means I've got 3-4 tunnels on
first run - for a whopping 63 to 84 seconds of wasted time. That's
definitely enough time to trigger timeouts and mess with behavior.
When did this start? This problem is new as of roughly a month or two
ago. I've been using Fiddler for well over a year in this environment
with no hint of this problem.
Client Configuration: Win7 x64, IE8 | FireFox | Chrome 16.0.912.77m,
Just one client PC? I've seen this impact 3 separate computers on
different subnets.
Fiddler settings:
* General: Check for updates, Enable IPv6 (but disable doesn't help),
Map socket to original app, Auto stream aud&vid.
* HTTPS: Capture connects, Decrypt HTTPS... from browsers only, Ignore
server certs, skip decryption for *mail*;*microsoft*;...
* Extensions: Auto-reload script, Extensions: BuiltWith,
HtmlInspector, FiddlerWatcher, neXpert..., Differ, RequestToCode,
RulesTab2, SimpleFilter, EventLog, HostsFile, Timeline.
* Connections: port 8888, Reuse client and server connections
(disabling both didn't help), chain to upstream gateway, Act as proxy
on startup, monitor all, bypass <-loopback>;*mail*;*microsoft*;...
Any other odd network behavior?
* DNS: I've seen suspicious DNS issues (can't resolve www.google.com
and/or www.microsoft.com sometimes)
* Proxy PAC file provided by IT (and related proxy server(s)) have
some history of misbehaving at times.
* IPv6 weirdness: I have found if I try "ping <servername>" from a
command prompt to my target web server I'll get an IPv6 address
resolve and the ping will always time out. However if I try "ping -4
<servername>" ping will respond very quickly and reliably. This is
very odd and disturbing. (It used to be inconsistent but now is pretty
reliably defaulting to IPv6).
* WebResource.axd requests to this same web server now return HTTP
404 (resource not found). This might be an archaic IE8 bug… Nope,
Chrome repros too.
SUSPECTS:
1. IPv6 weirdness
2. Proxy server chaining with corporate IT proxy server to specific
internal datacenter firewall proxy before finally getting to target
server. (and/or firewall configuration)
3. Newer Fiddler version installed and/or settings (or installation?)
hopelessly messed up in some very subtle way (unlikely but possible)
Help me Obi Wan, you're my only hope. I'm quickly running out of ideas
and way past time to get other opinions. Even tools or reference links
to help understand where TCP/IP CONNECT delays can occur would be
helpful.
Many thanks in advance! -Zephan
has me baffled. Basically TCP/IP Connect tunnels consistently takes
exactly 20999ms (21 seconds) to establish... but only for the internal
sites I need to test (and occasionally a couple public external
sites). I think this is a magic value related to some IPv6->IPv4
fallback or similar renegotiation process, but I need some help
identifying the root cause or narrowing down where to look for the
problem (client config, corporate IT network, datacenter network, or
server config).
MORE INFORMATION
================
Classic example of an impacted HTTP CONNECT trace:
--------------------------------------------------------------
Request Count: 1
Bytes Sent: 436 (headers:436; body:0)
Bytes Received: 107 (headers:107; body:0)
ACTUAL PERFORMANCE
--------------
ClientConnected: 18:39:25.986
ClientBeginRequest: 18:39:25.987
GotRequestHeaders: 18:39:25.987
ClientDoneRequest: 18:39:25.987
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 21000ms <======= Here's the symptom
HTTPS Handshake: 1ms
ServerConnected: 18:39:46.990
FiddlerBeginRequest: 18:39:46.990
ServerGotRequest: 18:39:46.990
ServerBeginResponse: 00:00:00.000
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 00:00:00.000
ClientBeginResponse: 18:39:46.990
ClientDoneResponse: 18:39:46.990
Overall Elapsed: 00:00:21.0031001
--------------------------------------------------------------
The kicker is during login we use ADFS for authentication which routes
to two other internal servers which means I've got 3-4 tunnels on
first run - for a whopping 63 to 84 seconds of wasted time. That's
definitely enough time to trigger timeouts and mess with behavior.
When did this start? This problem is new as of roughly a month or two
ago. I've been using Fiddler for well over a year in this environment
with no hint of this problem.
Client Configuration: Win7 x64, IE8 | FireFox | Chrome 16.0.912.77m,
Just one client PC? I've seen this impact 3 separate computers on
different subnets.
Fiddler settings:
* General: Check for updates, Enable IPv6 (but disable doesn't help),
Map socket to original app, Auto stream aud&vid.
* HTTPS: Capture connects, Decrypt HTTPS... from browsers only, Ignore
server certs, skip decryption for *mail*;*microsoft*;...
* Extensions: Auto-reload script, Extensions: BuiltWith,
HtmlInspector, FiddlerWatcher, neXpert..., Differ, RequestToCode,
RulesTab2, SimpleFilter, EventLog, HostsFile, Timeline.
* Connections: port 8888, Reuse client and server connections
(disabling both didn't help), chain to upstream gateway, Act as proxy
on startup, monitor all, bypass <-loopback>;*mail*;*microsoft*;...
Any other odd network behavior?
* DNS: I've seen suspicious DNS issues (can't resolve www.google.com
and/or www.microsoft.com sometimes)
* Proxy PAC file provided by IT (and related proxy server(s)) have
some history of misbehaving at times.
* IPv6 weirdness: I have found if I try "ping <servername>" from a
command prompt to my target web server I'll get an IPv6 address
resolve and the ping will always time out. However if I try "ping -4
<servername>" ping will respond very quickly and reliably. This is
very odd and disturbing. (It used to be inconsistent but now is pretty
reliably defaulting to IPv6).
* WebResource.axd requests to this same web server now return HTTP
404 (resource not found). This might be an archaic IE8 bug… Nope,
Chrome repros too.
SUSPECTS:
1. IPv6 weirdness
2. Proxy server chaining with corporate IT proxy server to specific
internal datacenter firewall proxy before finally getting to target
server. (and/or firewall configuration)
3. Newer Fiddler version installed and/or settings (or installation?)
hopelessly messed up in some very subtle way (unlikely but possible)
Help me Obi Wan, you're my only hope. I'm quickly running out of ideas
and way past time to get other opinions. Even tools or reference links
to help understand where TCP/IP CONNECT delays can occur would be
helpful.
Many thanks in advance! -Zephan
EricLaw
Feb 8, 2012, 8:29:42 AM2/8/12
to Fiddler
21 seconds is basically the max that any given HTTP connect can take.
Yes, IPv6 Weirdness seems like the most likely culprit, although
disabling IPv6 and restarting should fix that. Looking at a NetMon or
Wireshark capture would likely reveal the problem.
You might want to mail me directly using the link on the Help menu to
tell me whose corporate network you're hitting this on.
Yes, IPv6 Weirdness seems like the most likely culprit, although
disabling IPv6 and restarting should fix that. Looking at a NetMon or
Wireshark capture would likely reveal the problem.
You might want to mail me directly using the link on the Help menu to
tell me whose corporate network you're hitting this on.
EricLaw
Feb 8, 2012, 8:32:27 AM2/8/12
to Fiddler
Also one other thing to do is send the text of the Session Properties
dialog from the session's context menu.
dialog from the session's context menu.
Zephan Schroeder
Feb 14, 2012, 8:29:34 PM2/14/12
to Fiddler
1. Corpnet context sent privately :-).
2. Properties for an impacted tunnel connect session is below. Only
two things look a bit odd to me:
a) X-CLIENTIP 127.0.0.1 which could just mean that I'm using
fiddler on the target client rather than remotely tracing.
b) "This URL is not present in the WinINET cache. [Code: 2]" - but
I'm requesting content from this server several times each minute.
(Perhaps I took this session while I had "Reuse Connections" disabled,
but even when enabled I still get this 21 second CONNECT delay.
SESSION STATE: Done.
Request Entity Size: 822 bytes.
Response Entity Size: 685 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler, IsDecryptingTunnel] 0x2100
X-RESPONSEBODYTRANSFERLENGTH: 0
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 59676
HTTPS-CLIENT-SESSIONID: 66 11 00 00 88 85 98 E9 92 B3 88 4A 56 EE AA
95 10 50 01 C9 0A 71 D0 2B DB 83 01 9B 9E 14 8D 8A
X-PROCESSINFO: iexplore:5812
NEXPERT.STEP:
NEXPERT.RTT: Unknown
X-EGRESSPORT: 50499
X-HOSTIP: 14#.5#.#.81 (Altered for posting)
X-DNS-FAILOVER: +1
== TIMING INFO ============
ClientConnected: 17:25:32.688
ClientBeginRequest: 17:25:32.688
GotRequestHeaders: 17:25:32.688
ClientDoneRequest: 17:25:32.688
Determine Gateway: 16ms
DNS Lookup: 1846ms
TCP/IP Connect: 21301ms
HTTPS Handshake: 215ms
ServerConnected: 17:25:55.849
FiddlerBeginRequest: 17:25:55.849
ServerGotRequest: 17:25:55.849
ClientDoneResponse: 17:25:55.850
Overall Elapsed: 00:00:23.1623160
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state
at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected
Mode) cache only.
2. Properties for an impacted tunnel connect session is below. Only
two things look a bit odd to me:
a) X-CLIENTIP 127.0.0.1 which could just mean that I'm using
fiddler on the target client rather than remotely tracing.
b) "This URL is not present in the WinINET cache. [Code: 2]" - but
I'm requesting content from this server several times each minute.
(Perhaps I took this session while I had "Reuse Connections" disabled,
but even when enabled I still get this 21 second CONNECT delay.
SESSION STATE: Done.
Request Entity Size: 822 bytes.
Response Entity Size: 685 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler, IsDecryptingTunnel] 0x2100
X-RESPONSEBODYTRANSFERLENGTH: 0
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 59676
HTTPS-CLIENT-SESSIONID: 66 11 00 00 88 85 98 E9 92 B3 88 4A 56 EE AA
95 10 50 01 C9 0A 71 D0 2B DB 83 01 9B 9E 14 8D 8A
X-PROCESSINFO: iexplore:5812
NEXPERT.STEP:
NEXPERT.RTT: Unknown
X-EGRESSPORT: 50499
X-HOSTIP: 14#.5#.#.81 (Altered for posting)
X-DNS-FAILOVER: +1
== TIMING INFO ============
ClientConnected: 17:25:32.688
ClientBeginRequest: 17:25:32.688
GotRequestHeaders: 17:25:32.688
ClientDoneRequest: 17:25:32.688
Determine Gateway: 16ms
DNS Lookup: 1846ms
TCP/IP Connect: 21301ms
HTTPS Handshake: 215ms
ServerConnected: 17:25:55.849
FiddlerBeginRequest: 17:25:55.849
ServerGotRequest: 17:25:55.849
ServerBeginResponse: 00:00:00.000
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 00:00:00.000
ClientBeginResponse: 17:25:55.849
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 00:00:00.000
ClientDoneResponse: 17:25:55.850
Overall Elapsed: 00:00:23.1623160
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state
at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected
Mode) cache only.
Zephan Schroeder
Apr 19, 2012, 6:10:55 PM4/19/12
to httpf...@googlegroups.com
Small update: My 21 second delay disappeared when I disabled Tools>Fiddler Options>General tab>Enable IPv6 (if available).
I still have issues with DNS and connectivity that I don't understand. Sometimes closing and reopening Fiddler will fix the problem (especially if Fiddler has been running >24h) but sometimes the problems are independent of whether Fiddler is running or not. I'll post undate or more specific questions in this forum when I get something reproducible enough to explain.
Reply all
Reply to author
Forward
0 new messages