Is it possible to capture HTTP requests in CONNECT tunnel?

[Tomas]

This is not about HTTPS Connects and Decrypting, i have some programs which itself doesn't support proxy, they still talk to the server with standard HTTP protocol.
So i setup some system wide proxy like ProxyCap, to force the program to proxy through Fiddler, the proxying works fine, but the HTTP data won't be shown in Fiddler,
it will be shown as:

200 HTTP Tunnel To www.somesite.com:80

Statistics tab says:
This is a Tunnel. Status: CLOSED, Raw Bytes Out: XXX; In: XXX
The selected session is a HTTP CONNECT Tunnel. This tunnel enables a client to send raw traffic (e.g. HTTPS-encrypted streams or WebSocket messages) through a HTTP Proxy Server (like Fiddler).
This tunnel was exempt from HTTPS-decryption.

And Inspector tab says:
A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 0.0 (Unknown)
Random: null
SessionID: null
Extensions:
    none
Ciphers:
null
Compression:
(not specified)

The response raw says:
HTTP/1.0 200 Connection Established
FiddlerGateway: Direct
StartTime: 22:54:56.733
Connection: close
EndTime: 22:54:57.102
ClientToServerBytes: XXX
ServerToClientBytes: XXX

It seems fiddler will try to think all CONNECT requests as HTTPS tunnel, and if there is no SSL handshake found in the tunnel, then it won't capture the data and
will just passthrough  them. i can use wireshark to see the actual request to Fiddler is something like: (e.g: client requesting http://www.google.com/)
CONNECT www.google.com:80 HTTP/1.0
Host: www.google.com:80

GET / HTTP/1.1
Host: www.google.com
Connection: close

As you can see, the data is still standard HTTP, and you can check this from the first few bytes of request/response to see if it is HTTP request in tunnel and then
try to decode them.

Is it possible to use Fildder to capture such HTTP CONNECT requests? it will be very helpful if such feature is supported, since there are quite some programs don't
have support proxy itself, or some of them can configure proxy but are even using CONNECT instead of GET/POST for all HTTP requests.

[EricLaw]

Fiddler evaluates traffic inside HTTP CONNECT Tunnels to find WebSocket traffic and HTTPS traffic. If the traffic is of another format, then Fiddler simply shuttles it back and forth without trying to parse it.

 

While it would be possible to change the logic to parse plain HTTP traffic that's tunneled through a CONNECT, this is a pretty obscure scenario and there have only been a handful of requests for such capabilities over the decade that Fiddler has been available.

 

I'd love to hear more about what "programs which itself doesn't support proxy" you're referring to. I've seen very very few programs that support HTTP but not proxies.

[Tomas]

One example:

http://android.stackexchange.com/questions/27224/how-to-configure-bluestack-appplayer-to-use-proxy-settings-when-connecting-to-ne

Bluestacks App Player, this is android emulator which let you run android apps in windows system. which don't support proxy config yet.
And most android apps don't support proxy config itself, they use the system proxy config, if the system (like bluestacks) don't support it.
then itself won't support proxy. tools like ProxyDroid also not working in bluestacks so the only way i found to make the proxy working is
setup a system proxy in windows like i said, then force the entire bluestacks traffics to redirect through the proxy. and most of the traffic
are plain http using CONNECT.

I know i can still sniffer those traffic with wireshark or other tools, but i love to use Fiddler for HTTP, not only you can monitor the traffic, decode
and show them in various format, but also you can modify, auto response and tweak with ease. this is a excellent tool for web debugging.

There are also much more examples for this, lots of shell scripts (unix or windows) don't support proxy config directly, it is possible that you
may modify the script to make it support the proxy, but the other way is to use system wide proxy and redirect all traffic for thise program in
HTTP CONNECT.

[EricLaw]

In Windows, every HTTP stack that you might conceivably use in a "shell script" (to wit: WinINET, WinHTTP/BITS, System.NET, Java) supports proxy settings.

 

BlueStacks is an interesting and rare case of a (poorly conceived) platform that doesn't support proxy settings. It seems very likely that they'll support such settings in a future version, considering how many users have complained about this.

 

Which tool specifically are you using with BlueStacks-- proxycap? Got a link? I'd like to play with it.

 

thanks,

Eric

[Tomas]

As you can see the stackexchange question have been posted for more than one year, yet the bluestacks don't support proxy still. so i have to find own solutions instead of waiting.

I am using proxycap from http://www.proxycap.com, it is a useful tool when you want to use proxy for any program (especially socks proxy since lots of programs don't support that directly). For bluestacks, i added the rule in proxycap for HD-Network.exe to filddler proxy with HTTPS type. i understand this is a system level proxy, it has to work at TCP level instead of HTTP protocol. it don't care which high level protocol is used above TCP,  so every request it proxified will be turned into HTTP CONNECT to the fiddler.

Also, i later discovered in this way, that some of the HTTP traffic to fiddler using CONNECT are sometimes actually decoded shown as a http request correctly, just like decoded HTTPS, one line Tunnel to,  the next line is the actual HTTP request, but just some of them, there are still lots are not decoded and shown correctly (only a tunnel to line is shown). it seems fiddler already have the feature built-in, just not compatible with some case? the fiddler i am using is v2.4.5.0.

A few more tested manually are done in this way:
telnet 127.0.0.1 fiddler_port
then paste the request data (netcat not working correctly so i used this instead)
CONNECT www.google.com:80 HTTP/1.0
Content-Length: 0

Host: www.google.com:80

GET / HTTP/1.1
Host: www.google.com

or:
CONNECT www.google.com:80 HTTP/1.1

Host: www.google.com:80

GET / HTTP/1.1
Host: www.google.com

will be decoded and shown correctly as HTTP requests sometimes (not always)

but

CONNECT www.google.com:80 HTTP/1.0
Host: www.google.com:80

GET / HTTP/1.1
Host: www.google.com

won't be decoded always.

Also, excluding unix systems, even in windows, not every program using HTTP stack can support proxy setting easily,  especially for those using own HTTP stack instead of windows/standard ones.
e.g: node.js is mainly used to build server applications, but it is also good for client developing, the builtin http module in node js don't support http proxy directly, you have to use some third party modules
or do some tweaking with the code. which is not always a good choice for programs with complex code.