AWS ends domain-fronting

[Dan Blah]

In case anyone missed it: https://aws.amazon.com/blogs/security/enhanced-domain-protections-for-amazon-cloudfront-requests/

CloudFront will also be soon be implementing enhanced protections against so-called “Domain Fronting” [...] Although these cases are also already handled as a breach of our AWS Terms of Service, in the coming weeks we will be checking that the account that owns the certificate we serve for a particular connection always matches the account that owns the request we handle on that connection. 

 

[David Fifield]

I don't think there's been any technical change deployed yet, so it's
hard to guess what will work or not in the near future. But currently,
SNI-less requests work, just like they still do for Google. I wonder if
Amazon would object to that, because there's no domain name involved?
(Again hard to guess, as I'll bet their actual motivation is different
than what they say in the blog post.)

Just in case, I'm working on an SNI-less mode for meek-client.
https://bugs.torproject.org/12208

[David Fifield]

On Mon, Apr 30, 2018 at 01:16:43PM -0700, David Fifield wrote:
> Just in case, I'm working on an SNI-less mode for meek-client.
> https://bugs.torproject.org/12208

I've been experimenting with ways to omit SNI while still doing proper
certificate verification (or more generally, to verify using a name that
differs from the SNI name) in Go code. It is not that straightforward. I
wrote a prototype, and I'd appreciate any comment by people who have
worked through a similar problem. One of the APIs it uses,
VerifyPeerCertificate, was only added in Go 1.8, about one year ago.

https://bugs.torproject.org/12208#comment:14
https://trac.torproject.org/projects/tor/attachment/ticket/12208/names.go


I'll paste my design rationales from the ticket here.

----

The past couple of days I've been experimenting with ways to set or omit the SNI. There's a demo program in [https://trac.torproject.org/projects/tor/attachment/ticket/12208/names.go names.go] and I'd appreciate any review or questions about design decisions. I took inspiration from Ox's [https://gist.github.com/oxtoacart/5e78d25a7f9a9cda10cd domainfront.go demo] from 2014, but changes in the standard library (especially since Go 1.8) provide better ways to do some things now. If this looks good, I'm going to use it as the basis of new code in meek-client.

The basic situation is that we are managing four domain names:
* `urlName`: this is the name we resolve and actually establish a TCP connection with.
* `hostName`: this is the name that goes in the HTTP Host header.
* `sniName`: this is the name that goes in the SNI extension.
* `verifyName`: this is the name that the client verifies the server certificate against.
In normal everyday HTTPS, all four of these names are the same. Domain fronting allows `hostName` to differ, but the other three names are the same. [https://trac.torproject.org/projects/tor/attachment/ticket/12208/names.go names.go] shows how to make all four names independent, so you can, for example, send no SNI but still verify the server certificate against a hostname, while still fronting a different domain in the Host header.

An explanation of some decisions:
1. The way to set the SNI is to modify `TLSClientConfig.ServerName`; the way to set the verification name is to modify the `TLSClientConfig.VerifyPeerCertificate` callback. Both of these are properties of [https://golang.org/pkg/net/http/#Transport http.Transport], which is a long-lived data structure that manages multiple HTTP roundtrips. We therefore need a separate `http.Transport` for each unique (`sniName`, `verifyName`) pair. The only exception is when `urlName` = `sniName` = `verifyName`: that's the built-in behavior of the net/http package, so we can share an `http.Transport` in that case. The function `getHTTPTransportForNames` creates and caches new `http.Transport`s as needed. (And note that `hostName` does not enter this logic at all: whether or not you are domain fronting is orthogonal to certificate verification.)
2. The way to omit the SNI extension is to set `TLSClientConfig.ServerName` to an IP address. The IP address is not actually used for anything else, as long as `TLSClientConfig.InsecureSkipVerify` is set. The trick of setting `TLSClientConfig.TLSDial`, and then pulling the certificates from the `tls.Conn` using `ConnectionState`, [https://github.com/golang/go/issues/9126 doesn't work] when a proxy is set, because the proxy bypasses the dialer (which is the reason why [https://github.com/golang/go/issues/16363 VerifyPeerCertificate was introduced]).
3. We need to use multiple simultaneous `http.Transport`s, so we can't just modify the global [https://golang.org/pkg/net/http/#RoundTripper http.DefaultTransport] like we do currently. But `http.DefaultTransport` has some nice default settings for things like timeouts. Go doesn't provide any method for creating a new `http.Transport` that has the default settings, and you can't just copy the struct literal because it contains mutexes. So I used a reflection trick I found on a mailing list to copy just the public struct members. Also if we naively set `TLSClientConfig`, it [https://github.com/golang/go/issues/17051 disables HTTP/2 support]; because we plan to modify `TLSClientConfig`, we have to first call `http2.ConfigureTransport`.

Here are examples use cases.

all names equal (ordinary HTTPS)::
{{{
$ ./names -host example.com -sni example.com -verify example.com https://example.com/
urlName: "example.com"
hostName: "example.com"
sniName: "example.com"
verifyName: "example.com"
HTTP/2.0 200 OK
}}}

omit SNI, but still verify::
{{{
$ ./names -host example.com -sni "" -verify example.com https://example.com/
urlName: "example.com"
hostName: "example.com"
sniName: ""
verifyName: "example.com"
HTTP/2.0 200 OK
}}}

omit DNS request and SNI, but still verify::
{{{
$ dig +short example.com
93.184.216.34
$ ./names -host example.com -sni "" -verify example.com https://93.184.216.34/
urlName: "93.184.216.34"
hostName: "example.com"
sniName: ""
verifyName: "example.com"
HTTP/2.0 200 OK
}}}

ask for one name in the SNI but verify against another (still valid) name::
{{{
$ ./names -host example.com -sni example.com -verify www.example.net https://example.com/
urlName: "example.com"
hostName: "example.com"
sniName: "example.com"
verifyName: "www.example.net"
HTTP/2.0 200 OK
}}}

try verifying against a name that's not valid for the certificate::
{{{
$ ./names -host example.com -sni example.com -verify microsoft.com https://example.com/
urlName: "example.com"
hostName: "example.com"
sniName: "example.com"
verifyName: "microsoft.com"
error: x509: certificate is valid for www.example.org, example.com, example.edu, example.net, example.org, www.example.com, www.example.edu, www.example.net, not microsoft.com
}}}

domain fronting::
{{{
$ ./names -host maps.google.com -sni www.google.com -verify www.google.com https://www.google.com/
urlName: "www.google.com"
hostName: "maps.google.com"
sniName: "www.google.com"
verifyName: "www.google.com"
HTTP/2.0 302 Found
}}}

domain fronting [https://bugs.torproject.org/25804 thwarted]::
{{{
$ ./names -host html5-demos.appspot.com -sni www.google.com -verify www.google.com https://www.google.com/
urlName: "www.google.com"
hostName: "html5-demos.appspot.com"
sniName: "www.google.com"
verifyName: "www.google.com"
HTTP/2.0 502 Bad Gateway
}}}

but works if you leave off the SNI::
{{{
$ ./names -host html5-demos.appspot.com -sni "" -verify www.google.com https://www.google.com/
urlName: "www.google.com"
hostName: "html5-demos.appspot.com"
sniName: ""
verifyName: "www.google.com"
HTTP/2.0 200 OK
}}}

[Adam Fisk]

We basically just prescan working fronting domains and associated IPs in our case and distribute those mappings out of band. So there’s nothing special to do on the cert verification side - just have to set the correct "ServerName" in the tls.Config.

See:

https://github.com/getlantern/fronted/blob/master/direct.go#L404

--
You received this message because you are subscribed to the Google Groups "Network Traffic Obfuscation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to traffic-obf+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Fisk]

Oh nope just kidding -- that'll trigger SNI as you alluded to. We're
bad and skip verification in that case, so essentially we have the
same question about the best way to go about this =):

https://github.com/getlantern/tlsdialer/blob/master/tlsdialer.go#L139

>> email to traffic-obf...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.

--
--
President
Brave New Software Project, Inc.
https://www.getlantern.org
A998 2B6E EF1C 373E 723F A813 045D A255 901A FD89

[Adam Fisk]

Just kidding again. I seem to be trigger happy with the emails this
morning. Here's where we verify:

https://github.com/getlantern/tlsdialer/blob/master/tlsdialer.go#L189

[David Fifield]

On Tue, May 01, 2018 at 10:33:44AM -0700, Adam Fisk wrote:
> We basically just prescan working fronting domains and associated IPs in our
> case and distribute those mappings out of band. So there’s nothing special to
> do on the cert verification side - just have to set the correct "ServerName" in
> the tls.Config.

Right; but I think in this case it works only because of also using a
custom SNI-less tls Dialer: https://github.com/getlantern/tlsdialer
"package tlsdialer contains a customized version of crypto/tls.Dial that
allows control over whether or not to send the ServerName extension in
the client handshake." Otherwise, setting ServerName for verification
purposes would also cause that name to be sent in the SNI.

Using a custom Dialer is a good solution, and I think it was the only
way before Go 1.8. The only disadvantage is that the custom Dialer is
bypassed if you configure a proxy (unless I'm mistaken).

[David Fifield]

On Tue, May 01, 2018 at 10:42:32AM -0700, Adam Fisk wrote:
> Just kidding again. I seem to be trigger happy with the emails this
> morning. Here's where we verify:
>
> https://github.com/getlantern/tlsdialer/blob/master/tlsdialer.go#L189

Okay, yeah, that makes sense. It's basically the same as the function I
called verifyCertificateForName, which makes sense as they they are both
based on code from crypto/tls. The main difference in
verifyCertificateForName is that you have to parse the certificates
first, because the VerifyPeerCertificate callback just gives you a
sequence of byte slices, not pre-parsed certificates. (I suspect the
motivation was certificate pinning, where you actually want the
serialized byte representation.)
https://trac.torproject.org/projects/tor/attachment/ticket/12208/names.go#L61

[Sergey Frolov]

There is a PR in utls, that adds an option to omit SNI from the parrots. Certificate will be verified, unless utls is in "just do default Golang things" mode or InsecureSkipVerify is set.

https://github.com/refraction-networking/utls/pull/3

I was originally thinking it could be useful to evade simple SNI-based censorship, but didn't merge it, since I was concerned that "certificate is still easily parsed from server response". It's possible to skip the cert via fake session ticket, and then have a custom but robust auth mechanism using sessionTicket/clientRandom + serverRandom as channels for digital signature.

Anyway, if omitSNI option is helpful as is for SNI-less domain fronting, we can just merge it now.

[Sergey Frolov]

Pardon me, "just do default Golang things" mode does verify certs, what it won't do is omit SNI.

[Ox Cart]

> certificate is still easily parsed from server response

The nice thing about SNI-less connections is that a lot of CDNs will use multi-SAN certificates for such requests, so there's not a specific single domain that correlates with these.

Cheers,

Ox

PGP Key - https://pgp.mit.edu/pks/lookup?op=get&search=o...@getlantern.org

-----------------------------------------------------------------------------------------

"I love people who harness themselves, an ox to a heavy cart,

who pull like water buffalo, with massive patience,

who strain in the mud and the muck to move things forward,

who do what has to be done, again and again."

- Marge Piercy

[Jon Camfield]

This may be crazytalk, but back in the 2012-ish era, Internews ran a project called Remedy which basically helped manage the auto-creation of randomized URLs for websites which were being blocked at the DNS level, and provided an interface for the website manager to push those out through social media.

Could something similar work in AWS-like infrastructure for a PT? Given Amazon's harsh wording, we'd probably want  to talk to them to get a vague thumbs-up, but I am thinking (again, rough concept) about a PT where the bridge server provides to each PT client that connects individual "next step" bridge DNS/IP entries which it will spin up if the client disconnects. This could be very costly if we can't also shut down bridges which have been widely blocked, and there's a ton more design to hack through, but could provide a very similar level of collateral freedom protection, as effective blocking would require blocking the whole provider's cloud?

[David Fifield]

On Wed, May 02, 2018 at 11:17:43AM -0700, Jon Camfield wrote:
> This may be crazytalk, but back in the 2012-ish era, Internews ran a project
> called Remedy which basically helped manage the auto-creation of randomized
> URLs for websites which were being blocked at the DNS level, and provided an
> interface for the website manager to push those out through social media.
>
> Could something similar work in AWS-like infrastructure for a PT? Given
> Amazon's harsh wording, we'd probably want  to talk to them to get a vague
> thumbs-up, but I am thinking (again, rough concept) about a PT where the bridge
> server provides to each PT client that connects individual "next step" bridge
> DNS/IP entries which it will spin up if the client disconnects. This could be
> very costly if we can't also shut down bridges which have been widely blocked,
> and there's a ton more design to hack through, but could provide a very similar
> level of collateral freedom protection, as effective blocking would require
> blocking the whole provider's cloud?

It could be practical, but it's hard to say. On CloudFront, for example,
when you set up a CDN endpoint, you get an (apparently) randomly
generated subdomain of cloudfront.net. You could rapidly switch between
*.cloudfront.net names, more easily than you could rapidly spin up IP
addresses. But the censor could also decide to block *.cloudfront.net.
The GFW does, I don't know if any others do.

I recall Greatfire was doing something similar, constanly generating
fresh names and indexing the current ones on a GitHub wiki page.

I don't think costs for unused bridges would be a problem, if I
understand you right. All the CDN plans I've seen, they charge per
request and per gigabyte, not per unit time. An unused CDN endpoint
doesn't cost anything.

[David Fifield]

On Tue, May 01, 2018 at 11:21:30AM -0700, Sergey Frolov wrote:
> There is a PR in utls, that adds an option to omit SNI from the parrots.
> Certificate will be verified, unless utls is in "just do default Golang things"
> mode or InsecureSkipVerify is set.
>
> https://github.com/refraction-networking/utls/pull/3
>
> I was originally thinking it could be useful to evade simple SNI-based
> censorship, but didn't merge it, since I was concerned that "certificate is
> still easily parsed from server response". It's possible to skip the cert via
> fake session ticket, and then have a custom but robust auth mechanism using
> sessionTicket/clientRandom + serverRandom as channels for digital signature.
>
> Anyway, if omitSNI option is helpful as is for SNI-less domain fronting, we can
> just merge it now.

What I personally would like is these use cases:

Do a DNS lookup but omit SNI
urlName: "www.example.com"
hostName: "www.example.com"
sniName: ""
verifyName: "www.example.com"

Use a pre-resolved IP address and omit SNI
urlName: "93.184.216.34"
hostName: "www.example.com"
sniName: ""
verifyName: "www.example.com"

Also hostName should be controllable independently, but that shouldn't
be a problem.

If omitSNI can do both of those use cases, it seems good to me. Because
I'm guessing that tls.Config.ServerName maps to what I call verifyName,
so I can still verify even if I pass a pre-resolved IP address in the
URL to http.Transport.RoundTrip or whatever.

[Sergey Frolov]

The way one would use utls is manually do tcp net.Dial(), get a `net.Conn` and pass it into utls.UClient(), which works like standard tls.Client(), but also accepts ClientHelloID. This supports both scenarios you listed. To use utls with things like http.Transport, one would probably need a wrapper for http.Transport.Dial that would do the above.

> I'm guessing that tls.Config.ServerName maps to what I call verifyName

That is correct.

[Ben Schwartz]

I've been pushing for a standards-based method of defeating SNI-based connection identification, with security properties that might make it more attractive to some of these big server operators:  https://lists.w3.org/Archives/Public/ietf-http-wg/2018AprJun/0099.html.  To a nontechnical audience we could call it "domain fronting 2.0", or maybe "server name privacy".

At the last IETF meeting, this proposal got some positive support but the chairs want it to get more discussion on the mailing list before we proceed.  It would really help if anyone here could (a) read the proposal and (b) reply on the ietf-http-wg mailing list (hopefully in support!) to generate discussion.

Thanks,

Ben

[Tom Ritter]

https://datatracker.ietf.org/doc/draft-ietf-tls-sni-encryption/ is an
WG track document, that seems like it would have more promise?
Although I suppose it would require someone big to deploy it otherwise
one could just block all SNI-less requests....

But maybe that is more palpable to cloud providers than standardizing
actual masquerading?

-tom

On 8 May 2018 at 15:45, 'Ben Schwartz' via Network Traffic Obfuscation

>> email to traffic-obf...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Network Traffic Obfuscation" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to traffic-obf...@googlegroups.com.

[Joseph Lorenzo Hall]

Ben, could you summarize the state of encrypted SNI for us at IETF.
There's the alt-svc work of you and Mike Bishop, but how does that
jive with what Christian Huitema has been doing?

Joseph Lorenzo Hall
Chief Technologist, Center for Democracy & Technology [https://www.cdt.org]
1401 K ST NW STE 200, Washington DC 20005-3497
e: j...@cdt.org, p: 202.407.8825, pgp: https://josephhall.org/gpg-key
Fingerprint: 3CA2 8D7B 9F6D DBD3 4B10 1607 5F86 6987 40A9 A871

[Ben Schwartz]

My quick summary would be that "encrypted SNI" using just TLS (without HTTP) is a much harder problem than "encrypted SNI" in HTTPS.  Christian's draft is essentially a collection of different ideas that might be part of a TLS-only "encrypted SNI" solution, but it doesn't represent a complete system, because building a complete solution into TLS would be a big change.

The Alt-Svc drafts only apply to HTTPS, which makes the problem much easier.  HTTPS gives us very powerful components like multiplexing, Alt-Svc, and Secondary Certificate Authentication, so only a little bit of glue is required to build SNI protection.

Christian and I have been talking about this problem space for over a year, so there is some amount of coordination between these proposals.  My hope is that we can solve the HTTPS case first, and worry about the general case for TLS later.

>>> email to traffic-obf+unsubscribe@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Network Traffic Obfuscation" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to traffic-obf+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "Network Traffic Obfuscation" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to traffic-obf+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

[Joseph Lorenzo Hall]

Fantastic! Thank you

>>> email to traffic-obf...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Network Traffic Obfuscation" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to traffic-obf...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "Network Traffic Obfuscation" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to traffic-obf...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

Joseph Lorenzo Hall
Chief Technologist, Center for Democracy & Technology [https://www.cdt.org]
1401 K ST NW STE 200, Washington DC 20005-3497
e: j...@cdt.org, p: 202.407.8825, pgp: https://josephhall.org/gpg-key
Fingerprint: 3CA2 8D7B 9F6D DBD3 4B10  1607 5F86 6987 40A9 A871