HOWTO use a browser's HTTP engine from outside

[David Fifield]

https://www.bamsoftware.com/hacks/fetch-rpc.zip

This is source code for a browser extension (works with Firefox and
Chromium) that allows other programs to use the browser's HTTP engine.
Specifically, it exposes a JSON interface to the fetch function, which
is like XMLHttpRequest with a better API.

With the extension installed, you can send fetch parameters to a local
TCP port (9901 by default), and see the result of the browser's
executing it for you.

$ echo '{"input": "https://example.com/"}' | nc 127.0.0.1 9901
{"response":{"body":"PCFk...Cg==","headers":{},"ok":true,"redirected":false,"status":200,"statusText":"OK","type":"basic","url":"https://example.com/"}}
$ echo '{"input": "http://scanme.nmap.org:999/"}' | nc 127.0.0.1 9901
{"error":{"message":"NetworkError when attempting to fetch resource.","name":"TypeError"}}

Request and response bodies are base64-encoded. Other than that, the
JSON objects are isomorphic to the parameters of the Request constructor
and the properties of the Response object.
https://developer.mozilla.org/en-US/docs/Web/API/Request/Request#Syntax
https://developer.mozilla.org/en-US/docs/Web/API/Response
You can override headers and set other parameters:

$ echo '{"input": "https://example.com/", "init": {"method": "POST", "body": "ABCD", "headers": {"Foo": "bar"}}}' | nc 127.0.0.1 9901
{"response":{"body":"PCFk...Cg==","headers":{},"ok":true,"redirected":false,"status":200,"statusText":"OK","type":"basic","url":"https://example.com/"}}

Overriding the Host header works in Firefox but not in Chromium:

$ echo '{"input": "https://ajax.aspnetcdn.com/", "init": {"cache": "no-cache", "headers": {"Host": "meek.azureedge.net"}}}' | nc 127.0.0.1 9901 | jq -r .response.body | base64 -d
I’m just a happy little web server.

You may know that meek has been doing this kind of thing for a long
time, using Firefox for TLS camouflage. Firefox is dropping support for
the older kind of browser extension that meek had been using, so I
needed to port it to the newer WebExtension standard (https://bugs.torproject.org/29347).
While I was at it, I thought I should also demonstrate how to use the
same technique outside of meek. The meek implementation additionally has
support for setting an upstream proxy, which isn't portable across
browsers, so I left it out of this demo; but you can see it here:
https://gitweb.torproject.org/pluggable-transports/meek.git/tree/webextension?h=webextension&id=5b96265e01cf5e82b9e82f9e9e05441ac5311245

The obvious application of this is ESNI: other programs can get TLS
camouflage and ESNI support by using Firefox as a network interface. And
that's actually pretty easy to set up. There's a README.ESNI in the
source code, but the short summary is you need to set these about:config
prefs, or equivalents (the extension cannot set them for you):
network.trr.mode=3
network.trr.uri=https://1.1.1.1/dns-query
network.security.esni.enabled=true
security.OCSP.enabled=0
(The OCSP thing is something I noticed only recently. TLS 1.3, DoH, and
ESNI all have your back, but OCSP can still reveal the server name:
https://bugzilla.mozilla.org/1535235. Disabling OCSP is suboptimal, not
least because never sending OCSP is probably a mild distinguisher. This
wasn't a consideration with domain fronting, because the OCSP requests
would have been for the name on the certificate; i.e., the front domain.)

I have a prototype of meek working with ESNI in place of domain fronting
(https://bugs.torproject.org/28168), using a configuration as described.
Currently it requires running a second, newer copy of Firefox alongside
Tor Browser, because the Firefox 60 ESR that Tor Browser currently uses
doesn't support ESNI. That will change in the next major release of
Firefox, and the same close integration will become possible with ESNI
as with domain fronting in the past. However, my personal feeling is
that it's not yet the right time to deploy something based on ESNI—I'm
curious what others think.

[Sergey Frolov]

Thanks, David, great post and work as usual.

However, my personal feeling is that it's not yet the right time to deploy something based on ESNI—I'm curious what others think. 

I have the same feeling. We've had some reports that South Korea targets ESNI, and although that report may or may not have been a misrepresentation, I wouldn't be surprised if South Korea did in fact briefly block ESNI.

Both client and server-side support for ESNI are lacking, and don't seem to provide enough collateral damage to prevent censorship.

You can see usage of encrypted SNI extention on Colorado University campus by visiting this link:

https://tlsfingerprint.io/find/extension/ffce

As of late March, there's basically no usage.

[David Fifield]

On Sun, Mar 31, 2019 at 02:57:20PM -0700, Sergey Frolov wrote:
> I have the same feeling. We've had some[1] reports that South Korea targets

> ESNI, and although that report may or may not have been a misrepresentation, I
> wouldn't be surprised if South Korea did in fact briefly block ESNI.
> Both client and server-side support for ESNI are lacking, and don't seem to
> provide enough collateral damage to prevent censorship.
>
> You can see usage of encrypted SNI extention on Colorado University campus by
> visiting this link:
> https://tlsfingerprint.io/find/extension/ffce
> As of late March, there's basically no usage.

That's a feature of tlsfingerprint.io I didn't know about. It's a great
resource.

In my testing, Firefox with ESNI produces two different TLS
fingerprints: one to the DoH server and one to the web server.
Firefox without ESNI:
https://tlsfingerprint.io/id/6bfedc5d5c740d58 rank 7
Firefox with ESNI:
https://tlsfingerprint.io/id/8300bf0e26f2a109 rank 3649
https://tlsfingerprint.io/id/2dcbeba533890640 rank 3794

Comparing them, we see the fingerprint to the DoH server is the same
except that it lacks the server_name extension, which is expected
because the server (in this case) is identified by IP address.
https://tlsfingerprint.io/compare/6bfedc5d5c740d58/8300bf0e26f2a109
The fingerprint to the web server is similar, except that it removes a
padding extension and adds an encrypted_server_name extension.
https://tlsfingerprint.io/compare/6bfedc5d5c740d58/2dcbeba533890640

I posted a short summary with pcaps and a testing protocol here:
https://www.bamsoftware.com/sec/meek-esni-tls-report/