If HttpRequest is chunked, does fiddler also send it to the Server chunked?

[Bojin Li]

I ran a test where a request body is POSTed using TransferEncoding:  chunked. 

 

Netmon shows that the request is chunked into multiple HTTP:HTTP Payloads. 

 

However if I use Fiddler, it appears that the request is no longer chunked.

 

Don't understand why...

 

Netmon screenshot(with and without Fiddler) is attached.

[Bojin Li]

Forgot to mention that for the Fiddler case, I used FiddlerCore.

[EricLaw]

Fiddler does not alter the encoding of the request or response unless you've configured it to do so. For instance, if you use the Decode option on Fiddler's toolbar, Fiddler will unchunk and decompress all request and response bodies.

 

Now, having said that, your NetMon screenshot isn't actionable because it fails to show the request headers, so it's not possible to tell whether you've configured Fiddler to decode the request body.

[Bojin Li]

Hey Eric, for both Fiddler and FiddlerCore, I don't recall explicitly setting any Decode option.  I am assuming Decode is disabled by default?

 

I have attached the relevant frames in the Netmon capture. 

 

SelectedFramesNoFiddler - contains the capture of two Http Post requests.  The request body is chunked over many frames.

SelectedFramesFiddlerCore - another run of the two Http Post requests using FiddlerCore as proxy.  The request body no longer seems chunked.

SelectedFramesFiddler - I then ran the test again using Fiddler.exe as system proxy. The capture seems to match the FiddlerCore results.

 

Here are the Fiddler Timers for the one of the Fiddler run:

ClientConnected: 11:13:32.922
ClientBeginRequest: 11:13:32.954
GotRequestHeaders: 11:13:32.954
ClientDoneRequest: 11:13:38.079
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 2ms
HTTPS Handshake: 0ms
ServerConnected: 11:13:38.082
FiddlerBeginRequest: 11:13:38.082
ServerGotRequest: 11:13:38.083
ServerBeginResponse: 11:13:40.325
GotResponseHeaders: 11:13:40.326
ServerDoneResponse: 11:13:40.326
ClientBeginResponse: 11:13:40.326
ClientDoneResponse: 11:13:40.326

Overall Elapsed: 00:00:07.3727372

 

If you compare these timers with the No Fiddler Netmon capture, it almost seems like the client is chunking the request body to Fiddler(taking ~5 seconds), and then Fiddler is sending the request to the server non chunked(taking ~2 seconds).  Whereas the No Fiddler Netmon capture indicates that the Request body is chunked in ~5 seconds, and the server response occurs less than a second after that.

 

I am very confused and any light you can shed on this would be greatly appreciated.

 

[EricLaw]

In your SelectedFramesFiddler.cap file, you can see that the POST request explicitly specifies a Content-Length header and does not have a Transfer-Encoding: chunked header. This indicates that chunking is not in use, perhaps because the Decode option was set.

 

In the SelectedFramesFiddlerCore.cap file, you can see that the Content-Length header is not present and the Transfer-Encoding: chunked header is present, indicating that Chunked Transfer encoding is in use.

 

In the SelectedFramesNoFiddler.cap file, you can see that the Content-Length header is not present and the Transfer-Encoding: chunked header is present, indicating that Chunked Transfer encoding is in use.

 

One thing that I think you might be confused by: You say "The request body is chunked over many frames." HTTP Chunked Encoding bears no direct relationship to TCP/IP packets. Chunked Encoding simply allows the client or server to begin transmitting a response before the Content-Length is known. Use of Chunked Encoding does not mean that the "chunks" will necessarily go out in different TCP/IP packets, nor does the lack of Chunked Encoding prevent the use of multiple TCP/IP packets.

[Bojin Li]

Let's not pay too much attention to SelectedFramesFiddler.cap for now.(I included it in case it might shed light on things. I am not actually using the standalone fiddler application in what I'm building.)

 

So between SelectedFramesFiddlerCore.cap and SelectedFramesNoFiddler.cap, you verified that Chunked Transfer encoding is in use.  I understand what you are saying about the TCP/IP packets, but bear with me for a moment:

 

For SelectedFramesNoFiddler.cap, if I drill all the way down in the Network Conversions Tree view and filter only by HTTP protocol, I see this:

 

4 11:11:43 AM 6/5/2013 11.1753195 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP: {HTTP:3, TCP:2, IPv4:1}
6 11:11:44 AM 6/5/2013 11.2563155 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Continue., URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
7 11:11:44 AM 6/5/2013 11.2973307 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
9 11:11:44 AM 6/5/2013 11.6621983 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
11 11:11:44 AM 6/5/2013 12.0271864 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
13 11:11:45 AM 6/5/2013 12.3922338 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
15 11:11:45 AM 6/5/2013 12.7571569 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
17 11:11:45 AM 6/5/2013 13.1221992 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
19 11:11:46 AM 6/5/2013 13.4872242 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
21 11:11:46 AM 6/5/2013 13.8522364 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
23 11:11:46 AM 6/5/2013 14.2172258 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
25 11:11:47 AM 6/5/2013 14.5822563 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
27 11:11:47 AM 6/5/2013 14.9472912 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
29 11:11:48 AM 6/5/2013 15.3123098 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
31 11:11:48 AM 6/5/2013 15.6772954 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
33 11:11:48 AM 6/5/2013 16.0423335 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
35 11:11:48 AM 6/5/2013 16.0434200 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
37 11:11:48 AM 6/5/2013 16.0468752 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
39 11:11:49 AM 6/5/2013 16.8206507 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
41 11:11:50 AM 6/5/2013 17.5137284 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP: {HTTP:3, TCP:2, IPv4:1}
43 11:11:50 AM 6/5/2013 17.5712704 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Continue., URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
44 11:11:50 AM 6/5/2013 17.5713666 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
46 11:11:50 AM 6/5/2013 17.9364266 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
48 11:11:51 AM 6/5/2013 18.3013842 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
50 11:11:51 AM 6/5/2013 18.6664455 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
52 11:11:51 AM 6/5/2013 19.0314275 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
54 11:11:52 AM 6/5/2013 19.3965122 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
56 11:11:52 AM 6/5/2013 19.7614824 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
58 11:11:52 AM 6/5/2013 20.1271160 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
60 11:11:53 AM 6/5/2013 20.4920897 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
62 11:11:53 AM 6/5/2013 20.8570939 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
64 11:11:53 AM 6/5/2013 21.2221369 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
66 11:11:54 AM 6/5/2013 21.5871414 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
68 11:11:54 AM 6/5/2013 21.9521420 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
70 11:11:54 AM 6/5/2013 21.9529404 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
72 11:11:55 AM 6/5/2013 22.2835550 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}

 

Now, for SelectedFramesFiddlerCore.cap,  if I drill all the way down in the Network Conversions Tree view and filter only by HTTP protocol, I see this:

 

4 11:16:53 AM 6/5/2013 55.4242505 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP: {HTTP:3, TCP:2, IPv4:1}
5 11:16:53 AM 6/5/2013 55.4242951 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
26 11:16:57 AM 6/5/2013 59.3540088 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
28 11:17:03 AM 6/5/2013 65.5836170 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP: {HTTP:3, TCP:2, IPv4:1}
29 11:17:03 AM 6/5/2013 65.5836513 ReplayTool.exe 10.123.3.6 131.253.12.52 HTTP HTTP:HTTP Payload, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}
48 11:17:05 AM 6/5/2013 67.5919264 ReplayTool.exe 131.253.12.52 10.123.3.6 HTTP HTTP:Response, HTTP/1.1, Status: Ok, URL: /speech/query {HTTP:3, TCP:2, IPv4:1}

When I said "The request body is chunked over many frames.", this was the difference I tried to highlight.  For SelectedFramesFiddlerCore.cap, if you go up one level in the Network Conversions Tree View to include the TCP traffic, it seems like all the request body is transferred in TCP Protocol.  However for SelectedFramesNoFiddler.cap, it seems like the request body is transferred to the Server in these HTTP Payload calls on HTTP Protocol, and the server is giving some sort of acknowledgement in TCP Protocol after each HTTP Payload.

 

I was expecting to see the Netmon capture, at the HTTP Protocol filter, to match for both the FiddlerCore and none Fiddler case.  Is this an incorrect assumption?

[EricLaw]

> I was expecting to see the Netmon capture to match for both the FiddlerCore and non-Fiddler case.  Is this an incorrect assumption?

 

Absolutely. Fiddler & FiddlerCore are proxy servers; they faithfully capture and resend traffic at the HTTP-protocol level, not at the TCP/IP level.

 

For example, in TCP/IP, a silly client might send a HTTP Request as follows:

 

Packet #1: Size 1 bytes: Content: G

Packet #2: Size 1 bytes: Content: E

Packet #3: Size 1 bytes: Content: T

Packet #4: Size 1 bytes: Content:

Packet #5: Size 8 bytes: Content: / HTTP/1

Packet #6: Size 1 bytes: Content: .

Packet #7: Size 3 bytes: Content: 1\r\n

Packet #8: Size 15 bytes: Host: example.com\r\n\r\n

 

At any point along the way, an intermediary might convert this into:

 

Packet #1: Size 43 bytes: Content: GET / HTTP/1.1\r\nHost: example.com\r\n\r\n

 

Fiddler, for example, always reads a complete request before sending it along to the upstream gateway or server. So you may find that Fiddler sends fewer, larger, packets to a server.

 

-Eric

[Bojin Li]

But it does not seem like Fiddler is faithfully resending traffic at the HTTP-protocol level in the Netmon capture I pasted above... 

 

Netmon logs a dozen entries at the HTTP-protocol level when not using FiddlerCore. 

 

But when I redirect ReplayTool.exe's WebRequest Class calls to use FiddlerCore's Proxy, I see three entries per request at the HTTP-protocol level.

[Bojin Li]

Also I found the attached difference interesting in how Fiddler using a different ChunkPayload value.

[EricLaw]

I have no idea what you mean when you say "ChunkPayload" value.

[EricLaw]

I think you're confused about what NetMon shows you. HTTP has requests and responses, headers and bodies. It doesn't have any lower-level concepts than that. 

 

In contrast, NetMon shows traffic captures at a TCP/IP level. Perhaps confusingly, it's classifying packets by saying "Hey, this is a part of a HTTP payload" but that's simply based on an analysis of the traffic stream.