Hi All, FYI:

Soon we'll be launching a nightly based pref-flip shield study to confirm
the feasibility of doing DNS over HTTPs (DoH). If all goes well the study
will launch Monday (and if not, probably the following Monday). It will run
<= 1 week. If you're running nightly and you want to see if you're in the
study check about:studies

Access to global DNS data is commonly manipulated and can easily be blocked
and/or collected. DNS services are also sometimes poorly provisioned
creating performance problems. We posit that integrity and confidentiality
protected access to well provisioned larger caches will help our users. In
a nutshell, that's what DoH does.

This work relies on a IETF specification that I hope will go into Last Call
this coming week: https://datatracker.ietf.org/doc/draft-ietf-doh-
dns-over-https/

This initial test is focused on performance feasibility assessment and we
won't actually be using the DNS data returned from the DoH server (i.e. the
traditional DNS service is used in parallel and only those answers are used
- the code calls this shadow mode.) This is obviously not the optimal
arrangement of things - the anticipated end state will involve running in
"first mode" where DoH is normally used and soft fails (either based on DNS
or TCP errors) to traditional DNS. There are also modes where DoH is used
and hard fails (known as "only mode" - it requires some bootstrap info),
and a mode where DoH and traditional race against each other using
whichever is faster. Their are acomodations in place to deal with
split-horizon DNS issues.

DoH is an open standard and for this test we'll be using the DoH server
implementation at Cloudflare. As is typical for Mozilla, when we
default-interact with a third party service we have a legal agreement in
place to look out for the data retention/use/redistribution/etc interests
of both our users and Mozilla itself.

The study launch bug is https://bugzilla.mozilla.org/show_bug.cgi?id=1446404

Daniel Stenberg has written much of the code for this - he, I, and Valentin
Gosu are the team that will chase down any issues. Feel free to reach out
to us (or #necko on slack). There is currently one open issue related to
captive portals and "only mode" but that should not be triggered by the
study as "only mode" is not used.

-Patrick

On Sat, Mar 17, 2018 at 3:51 AM Patrick McManus <pmcm...@mozilla.com>
wrote:

> DoH is an open standard and for this test we'll be using the DoH server
> implementation at Cloudflare. As is typical for Mozilla, when we
> default-interact with a third party service we have a legal agreement in
> place to look out for the data retention/use/redistribution/etc interests
> of both our users and Mozilla itself.
>

So my understanding of the study is that for those in the study branch (50%
of Nightly users) we'll be sending every hostname they visit to Cloudflare.
That sounds problematic to me. Can you give more details about the legal
agreement?

Obviously, using a central resolver is the downside to this approach - but
its being explored because we believe that using the right resolver can be
a net win compared to the disastrous state of unsecured local DNS and
privacy and hijacking problems that go on there. Its just a swamp out there
(you can of course disable this from about:studies or just by setting your
local trr.mode pref to 0 - but this discussion is meaningfully about
defaults.)

And in this case the operating agreement with the dns provider is part of
making that right choice. For this test that means the operator will not
retain for themselves or sell/license/transfer to a third party any PII
(including ip addresses and other user identifiers) and will not combine
the data it gets from this project with any other data it might have. A
small amount of data necessary for troubleshooting the service  can be kept
at most 24 hrs but that data is limited to name, dns type, a timestamp, a
response code, and the CDN node that served it.



On Sun, Mar 18, 2018 at 11:51 PM, Dave Townsend <dtow...@mozilla.com>
wrote:

On Sun, Mar 18, 2018 at 5:27 PM Patrick McManus <pmcm...@mozilla.com>
wrote:

> Obviously, using a central resolver is the downside to this approach - but
> its being explored because we believe that using the right resolver can be
> a net win compared to the disastrous state of unsecured local DNS and
> privacy and hijacking problems that go on there. Its just a swamp out there
> (you can of course disable this from about:studies or just by setting your
> local trr.mode pref to 0 - but this discussion is meaningfully about
> defaults.)
>

I believe that a good resolver makes all the difference. I'm just concerned
about the privacy aspects of this, particularly since we're not really
messaging this to users. Is there a reason we need a full 50% of Nightly
population to get the data we need here?

On that topic I'm interested in what data we expect to get, is it just
comparing how the resolver performs from a variety of locations and for a
variety of lookups?
Is there some mechanism in place for users who's normal DNS resolver
intentionally returns different results to global DNS (e.g. for region
spoofing etc.)?

> And in this case the operating agreement with the dns provider is part of
> making that right choice. For this test that means the operator will not
> retain for themselves or sell/license/transfer to a third party any PII
> (including ip addresses and other user identifiers) and will not combine
> the data it gets from this project with any other data it might have. A
> small amount of data necessary for troubleshooting the service  can be kept
> at most 24 hrs but that data is limited to name, dns type, a timestamp, a
> response code, and the CDN node that served it.
>

Not retaining IP addresses is good. Can they perform aggregate tracking of
hostname requests, or tie common hostname requests from an origin together
somehow? What is our recourse if they break this agreement (the recent
Facebook debacle seems likely to make folks jumpy).

I definitely see some easy ways this could be problematic from a public
relations perspective given things going on in the industry these days and
some of our own mistakes the in the past. It's definitely worth taking a
little while to consider the implications before throwing the switch.

On Sun, Mar 18, 2018 at 8:39 PM, Dave Townsend <dtow...@mozilla.com>
wrote:
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>



--

Eric Shepherd
Senior Technical Writer
Mozilla
Blog: http://www.bitstampede.com/
Twitter: http://twitter.com/sheppy
Check my Availability <https://freebusy.io/eshepherd@mozilla.com>

On Sun, 18 Mar 2018, Eric Shepherd (Sheppy) wrote:

I don't have such a far-reaching agreement with my ISP and its DNS. I don't
have such an agreement at all with 8.8.8.8 or other publicly provided DNS
operators.

What other precautions or actions can we do to reduce the risk of this being
perceived as problematic? Would reducing the test population really make it
much different?
  / daniel.haxx.se

On Mon, Mar 19, 2018 at 3:18 AM, Eric Shepherd (Sheppy)
<eshe...@mozilla.com> wrote:
> I definitely see some easy ways this could be problematic from a public
> relations perspective given things going on in the industry these days and
> some of our own mistakes the in the past. It's definitely worth taking a
> little while to consider the implications before throwing the switch.

Indeed.

Sending the hostnames the browser accesses to a third party that would
not normally be part of the network activity (regardless of what
policy agreements Mozilla has with them) should be opt-in even if it
makes the study data less representative, even if it's Nightly only
and even if Cloudflare is better than some people's ISPs.

--
Henri Sivonen
hsiv...@hsivonen.fi
https://hsivonen.fi/

On 19/03/2018 09:07, Daniel Stenberg wrote:
> On Sun, 18 Mar 2018, Eric Shepherd (Sheppy) wrote:
>
> I don't have such a far-reaching agreement with my ISP and its DNS. I
> don't have such an agreement at all with 8.8.8.8 or other publicly
> provided DNS operators.

Yes, you're perfectly right, but you had a chance to choose it (or at
least, you feel like you've got the option).

>
> What other precautions or actions can we do to reduce the risk of this
> being perceived as problematic? Would reducing the test population
> really make it much different?

Reducing the population won't make any difference, unless that
population feels they had a choice.
If possible, I'd suggest to give maximum publicity to this experiment
beforehand, exposing all the good arguments above (and not having it
"discovered" after the fact on Reddit or Slashdot, which ensures only
the "shady" and possibly baseless edges get told in outrage) and
proposing the change with a splash page or something like that, even as
the default choice, but not silently pre-enabled.

Just my 2 cents.
-- G

>
>> I definitely see some easy ways this could be problematic from a public
>> relations perspective given things going on in the industry these
>> days and
>> some of our own mistakes the in the past. It's definitely worth taking a
>> little while to consider the implications before throwing the switch.
>>

Giorgio Maone
https://maone.net

On Mon, Mar 19, 2018 at 10:07 AM, Daniel Stenberg <dste...@mozilla.com> wrote:
> On Sun, 18 Mar 2018, Eric Shepherd (Sheppy) wrote:
>
> I don't have such a far-reaching agreement with my ISP and its DNS.

1) Mozilla doesn't choose the ISP on users' behalf. (This is the key reason.)
2) The ISP sees the Host header in unencrypted traffic and SNI in
encrypted traffic anyway. (This is a secondary reason.)

> I don't
> have such an agreement at all with 8.8.8.8 or other publicly provided DNS
> operators.

Using such resolvers is a decision that the user makes and not a
decision that Mozilla *silently* makes on their behalf.

> What other precautions or actions can we do to reduce the risk of this being
> perceived as problematic?

I suggested two ways on the bug.

> Would reducing the test population really make it
> much different?

No.

On Mon, Mar 19, 2018 at 8:10 AM, Henri Sivonen <hsiv...@hsivonen.fi> wrote:
> On Mon, Mar 19, 2018 at 3:18 AM, Eric Shepherd (Sheppy)
> <eshe...@mozilla.com> wrote:

>> I definitely see some easy ways this could be problematic from a public
>> relations perspective given things going on in the industry these days and
>> some of our own mistakes the in the past. It's definitely worth taking a
>> little while to consider the implications before throwing the switch.
>

> Indeed.
>
> Sending the hostnames the browser accesses to a third party that would
> not normally be part of the network activity (regardless of what
> policy agreements Mozilla has with them) should be opt-in even if it
> makes the study data less representative, even if it's Nightly only
> and even if Cloudflare is better than some people's ISPs.

Agreed, especially since the experiment as announced is Cloudflare in
addition to your ISP. So even if we could say they're better than your
ISP, which seems tough on a world-wide scale, that defense won't work
here.


--
https://annevankesteren.nl/

The objective here is a net improvement for privacy and integrity. It is
indeed a point of view with Nightly acting as an opinionated User Agent on
behalf of its users. IMO we can't be afraid of pursuing experiments that
help develop those ideas even when they move past traditional modes.
Traditional DNS is a swamp - ignoring that isn't doing our users any
favors. This is obviously not an engineering only driven effort.

Nightly is an explicitly experimental channel which is part of the reason
it is the choice for the first validation.

A question came up about geo based DNS and I've got a couple technical
comments about risk mitigation there:
 1] geo dns use is on the wane as TCP anycast approaches work much better
in practice
 2] the granularity of the CDN being used is much finer than the
granularity of most geoDNS resolution which tends to choose between very
big regions (O(~ 1/2 a continent)) so that should continue to work the same.

I initiated this thread on dev-platform because imo it is a reasonable
scope for nightly changes, especially ephemeral flip pref changes, and
that's why the FYI goes here. Its definitely not a secret. Messaging to a
larger user base than is impacted invites confusion. Future possible
changes impacting larger populations or putting things on trains would use
other, more broadly read communications channels.

-Patrick

Yes, it pays to remember that this is Nightly.

The precautions Henri suggests are good, but more appropriate to
experiments we would do on Release.  For TLS 1.3, we did that sort of
thing so that we could get measurements from Release; we just flipped
the switch to "on" for Nightly.

I don't know if it is possible to know if you have a
manually-configured DNS server, but disabling this experiment there if
we can determine that would be good - that might not be something to
worry about with Nightly, but it seems like it might be needed for
this to hit the trains.

How do we otherwise determine that a DNS server is not safe to
replace?  Split horizon DNS is going to cause unexpected failures when
users - particularly enterprise users - try to reach names that aren't
public.  That's not just an enterprise thing; this will break my home
router in some ways as well, though I'm actually totally OK with that
in this case :)

On Mon, Mar 19, 2018 at 11:48 AM, Martin Thomson <m...@mozilla.com> wrote:
> Yes, it pays to remember that this is Nightly.

Even on Nightly we place pretty severe restrictions on ourselves when
it comes to user data, e.g., for telemetry. This definitely goes
beyond the kind of data I would expect Mozilla, let alone a
third-party, to collect from Nightly users.


--
https://annevankesteren.nl/

Daniel

Le 19 mars 2018 à 17:07, Daniel Stenberg <dste...@mozilla.com> a écrit :
> What other precautions or actions can we do to reduce the risk of this being perceived as problematic?

opt-in only. That's the only way.

What seems innocuous for someone deep into the topic is not necessary the same for others. We all have different moral compass on what is acceptable and not acceptable. So Mozilla should just propose in this case and let the person decides.


--
Karl Dubost, mozilla 💡 Webcompat
http://www.la-grange.net/karl/moz

On Mon, Mar 19, 2018 at 1:25 PM, Patrick McManus <pmcm...@mozilla.com> wrote:
> The objective here is a net improvement for privacy and integrity.

I understand that the goal is better privacy. But it's likely that
people get outraged if a browser sends information about what is
browser to an off-path destination without explicit consent regardless
of intention, nightliness or promises the destination has made.

Opt-in is the way to go to avoid damaging trust.

Like I said on the bug: "the way people are known to react this kind
of thing isn't in our power to negotiate". Hence, the intention being
more privacy doesn't mean that if we do this without explicit consent
people won't be outraged.

> Nightly is an explicitly experimental channel which is part of the reason
> it is the choice for the first validation.

It's totally reasonable from a user perspective to expect Nightly to
run the latest and potentially buggy code, but it doesn't follow that
it's OK to give Nightly users less control of their privacy.

FWIW, from the point of view of my expectations as a Nightly user,
this goes against the old "No surprises" privacy language we had. (It
seems that the "No surprises" privacy language has been removed. It's
not good that the new language doesn't make it obvious at a glance
whether Mozilla permits itself to do what's proposed here without
explicit opt in. It think it would be better for Mozilla to
unambiguously promise not to do the kind of thing that's being
proposed here without explicit opt in.)

> I initiated this thread on dev-platform because imo it is a reasonable
> scope for nightly changes, especially ephemeral flip pref changes, and
> that's why the FYI goes here. Its definitely not a secret. Messaging to a
> larger user base than is impacted invites confusion. Future possible
> changes impacting larger populations or putting things on trains would use
> other, more broadly read communications channels.

It seems to me that the appropriate messaging would be in-Nightly
messaging asking if the user wants to participate in an experiment
that uses Cloudflare as the DNS provider in place of whatever DNS
provider their system would otherwise use.

Is running the service ourselves out of the question? If so, how come?
I mean I know we're not really in the business of running massive
scale DNS; but running it for a month, and ramping up the people
included in the study so we can monitor load seems feasible.

The goal of the study is described as "performance feasibility" - but
won't the data we get from it assume a performance conclusion based on
Cloudflare? (Which we might consider 'best-case'?) And that any other
DoH provider would be worse performance, by a factor we don't know?

If we ran the backend ourselves, we would know the geo distribution of
clients sending us traffic and it seems like we could even measure
their latency passively[1]. So we'd have more data than if we used
Cloudflare.

-tom

[0] Not necessary keeping people running the study for a month; but
over a month ramping up until we have encompassed 100% of the
population.
[1] It seems possible to do this since the client's going to be
sending us multiple packets, but yea I don't know any tools that would
actually do this.

On Mon, 19 Mar 2018, Martin Thomson wrote:

> I don't know if it is possible to know if you have a manually-configured DNS
> server, but disabling this experiment there if we can determine that would
> be good - that might not be something to worry about with Nightly, but it
> seems like it might be needed for this to hit the trains.
>
> How do we otherwise determine that a DNS server is not safe to replace?
> Split horizon DNS is going to cause unexpected failures when users -
> particularly enterprise users - try to reach names that aren't public.
> That's not just an enterprise thing; this will break my home router in some
> ways as well, though I'm actually totally OK with that in this case :)

I don't think it is possible - with any particularly high degree of certainty
- to know if a DNS server has been manually configured (or even if the term
itself is easy to define). The system APIs for name lookups typically don't
even expose which DNS server they use, they just resolve host names to
addresses for us.

For TRR, we've instead focused pretty hard on providing a "retry-algorithm" so
that Firefox can (if asked), retry a failed name resolve or TCP connect
without TRR and then "blacklist" that host for further TRR use for a period
into the future.

For hosts that are TRR-blacklisted this way, we also check the next-level
domain of it in the background to see if we should also blacklist the whole
domain from TRR use. Ie if "www.example.com" fails with TRR, it gets
blacklisted, retried with the native resolver and "example.com" is tested to
see if the entire domain should be blacklisted.

--

  / daniel.haxx.se

Hi!

Thanks for all the thoughtful comments about this experiment. The intent of
this work is to provide users privacy-respecting DNS. Status quo for DNS
does not offer many users reasonable, informed choice about log retention,
and doesn't offer encrypted DNS. Users also cannot be reasonably expected
to negotiate on their own with their ISPs/VPN providers for things like
24-hour retention for logs that can be used to create profiles. Today's
default environment (speaking technically wrt lack of encryption and log
storage, and also in terms of the regulatory environment in the US) allows
*all* of this data to be collected indefinitely and sold to third parties.

There's a lot of thinking that went into the agreement we have with
Cloudflare to enable this experiment in a way that respects user privacy.
We also want to explain the impact we think this kind work will have on the
privacy of the Internet. I'd like the team to share this in a blog post
about the experiment, and so have started work with them on it. More on
this shortly!

-selena



On Mon, Mar 19, 2018 at 8:16 AM Daniel Stenberg <dste...@mozilla.com>
wrote:

On Mon, Mar 19, 2018 at 7:08 PM, Selena Deckelmann <sel...@mozilla.com> wrote:
> and also in terms of the regulatory environment in the US) allows *all* of
> this data to be collected indefinitely and sold to third parties.

Some users are in countries where it's illegal for the ISP to sell
this information to third parties, so they might rightly be upset
about diverting this information elsewhere without opt-in.

> There's a lot of thinking that went into the agreement we have with
> Cloudflare to enable this experiment in a way that respects user privacy.

This seems like a great feature when the user is in control of whether
to use it.

I think it's a problem when we determine that Mozilla has negotiated
privacy terms, therefore users are protected by policy and there's no
problem. Our review processes show things are fine but some people
still get upset. Legal review doesn't cover the spectrum of attitudes
that users have.

Partly it's that people get upset before they stop to read what policy
Mozilla negotiated. Partly it's that people take the position that
they want privacy by design where the third party isn't contacted in
the first place so whatever the third party has promised is moot. For
yet other people, it's just a matter of feeling that they are in
control if they opt in but feel they are not in control when Mozilla
does these things without prior consent.

Why risk upsetting users in this case instead of obtaining consent first?

It's fine to embed this experiment in the product, and blog about it, but it's definitely not fine to have it enabled by default and send every DNS request to a third-party.

I can understand that the intent must be good, and for better privacy, but the approach of doing so is not acceptable. Users would think Firefox is going to just send data to arbitrary third-party without agreement from them.

As you can see from the replies, almost all people outside the network team has expressed concerns about this, which should be considered a signal already that how other technical users may feel about this experiment, and how technical news would create a title for this.

I strongly suggest we make this experiment opt-in rather than opt-out. You can use various channels to ask people to opt-in this experiment themselves, but making them on by default is not the right thing to do I fear.

- Xidorn

On Tue, Mar 20, 2018, at 4:08 AM, Selena Deckelmann wrote:
> Hi!
>
> Thanks for all the thoughtful comments about this experiment. The intent of
> this work is to provide users privacy-respecting DNS. Status quo for DNS
> does not offer many users reasonable, informed choice about log retention,
> and doesn't offer encrypted DNS. Users also cannot be reasonably expected
> to negotiate on their own with their ISPs/VPN providers for things like
> 24-hour retention for logs that can be used to create profiles. Today's
> default environment (speaking technically wrt lack of encryption and log

> storage, and also in terms of the regulatory environment in the US) allows

> *all* of this data to be collected indefinitely and sold to third parties.
>

> There's a lot of thinking that went into the agreement we have with
> Cloudflare to enable this experiment in a way that respects user privacy.

Hi all,

On Mon, Mar 19, 2018 at 3:56 PM, Xidorn Quan <m...@upsuper.org> wrote:

> It's fine to embed this experiment in the product, and blog about it, but
> it's definitely not fine to have it enabled by default and send every DNS
> request to a third-party.
>
> I can understand that the intent must be good, and for better privacy, but
> the approach of doing so is not acceptable. Users would think Firefox is
> going to just send data to arbitrary third-party without agreement from
> them.
>
> As you can see from the replies, almost all people outside the network
> team has expressed concerns about this, which should be considered a signal
> already that how other technical users may feel about this experiment, and
> how technical news would create a title for this.
>

Let me add my voice as a person outside the network team who can understand
the concerns and _still thinks we should be doing this_.

In particular, I'd like to argue against Henri Sivonen's rhetorical
question, "Why risk upsetting users in this case instead of obtaining
consent first?"

In today's age of impenetrable licensing agreements, the defaults matter.
It's not reasonable for users of the Web to assume the totality of the
risks of using the Web, and I think it's critical that Mozilla assume some
risk for its users.  That's why we should be bold, try things, and figure
out if we can move the default to be better for the mass market.  (This was
one of the points that Mikko Hyppönen emphasized for the security industry
in his recent talk to Mozilla.)

With regard to this experiment: we have a default right now that has
evolved over the last two decades to privilege forces close to the user
(ISP, DNS provider).  This experiment privileges forces farther away from
the user (the DoH provider).  The hope, as I see it, is that there will be
more robust competition in the market when the DoH provider can be
unbundled from the last mile connectivity provider.  (We've seen that last
mile connectivity providers don't have a lot of competition in many parts
of the world.)  I am interpreting this as something parallel to VPN
providers, where there's a robust market with diversified offerings.  Right
now, users have two functional choices: ISP-provided DNS or Google's DNS,
and both have serious downsides.  I think it's 100% Mozilla's role to
negotiate privacy-respecting agreements and service contracts -- things
that no individual user can arrange at this time.

I'm willing to upset some users in order to shift the defaults at scale.

Nick

On Mon, Mar 19, 2018 at 07:27:39PM -0700, Nicholas Alexander wrote:
>Hi all,
>
>On Mon, Mar 19, 2018 at 3:56 PM, Xidorn Quan <m...@upsuper.org> wrote:
>
>> It's fine to embed this experiment in the product, and blog about it, but
>> it's definitely not fine to have it enabled by default and send every DNS
>> request to a third-party.
>>
>> I can understand that the intent must be good, and for better privacy, but
>> the approach of doing so is not acceptable. Users would think Firefox is
>> going to just send data to arbitrary third-party without agreement from
>> them.
>>
>> As you can see from the replies, almost all people outside the network
>> team has expressed concerns about this, which should be considered a signal
>> already that how other technical users may feel about this experiment, and
>> how technical news would create a title for this.
>>
>
>Let me add my voice as a person outside the network team who can understand
>the concerns and _still thinks we should be doing this_.

I'll second this.

I think it's reasonable to be concerned about the public reaction to
this, but I also think it's worth doing. The end result of this will
almost certainly be improved privacy and security for users who have
this enabled by default, and we can't get to that point without doing a
study like this.

I think it's worth the risk of a backlash. But I also think we should do
what we can to minimize that backlash.

On 3/19/18 1:08 PM, Selena Deckelmann wrote:
> There's a lot of thinking that went into the agreement we have with
> Cloudflare to enable this experiment in a way that respects user privacy.

I would like us to be very clear that there are two separate things here:

1)  Is this behavior good for users?
2)  Will people think this behavior is good for users and for them?
(Maybe this should itself be two separate things.)

Here's how I see this:

* There are some concerns being raised about item 1 (e.g. it may be good
for users in the US but less good for users in jurisdictions where the
legal obligations of ISPs are qutie different).  Have we considered
doing the experiment only in some geographies, ones where we can make a
particularly strong case for the status quo being user-hostile?

* For item 2, fundamentally, we want to avoid people feeling like they
are being betrayed when they discover they are part of this experiment.
To me that seems like it requires clear messaging that they _are_ part
of the experiment.  If we tell a nightly user "you are part of a DNS
experiment, here are the details, here is how you opt out", that leaves
a _very_ different impression from (hypothetically; I haven't checked
whether this would be a failure mode of the proposed setup) the nightly
user being unable to access some intranet site that they set up
/etc/hosts entries for, spending a bunch of time figuring out why, and
then discovering that we silently changed how their browser does DNS.
In the latter situation people will be predisposed to believe the worst
and not listen to any explanations.

* Assuming we go forward with this, we should very seriously think about
the messaging, both in-product and out-of-product.  For example, I would
think that we would want this to appear on tech news sites _before_ we
start doing the experiment, not after.  That gives us a chance to
present our case in a non-crisis atmosphere, gives people a heads-up
about what they should expect, and is a lot less likely to be perceived
as us trying to sneak things in.

* A lot of this is about trust; both building trust and destroying it.
Fundamentally, for most people (I'd guess nearly all) trust is not a
logical decision; it's based on gut reactions.  Trying to logically
convince people that they should trust us is just not going to work if
their instincts are screaming at them not to trust us.  That means that
even if we're 100% sure something is better for users and even if we
have super-convincing arguments for it, we need to seriously think about
the way it's messaged (or not) and the resulting impact on trust.  It
doesn't help that what comes across as reassuring to one person comes
across as weaselly information-free double-speak to another....

> I'd like the team to share this in a blog post
> about the experiment

This seems like a good start, and we may want to then make sure whatever
information we are trying to put out there actually gets picked up by
widely-enough-read news bits so people aren't blindsided.

-Boris

On 3/19/18 8:59 PM, Boris Zbarsky wrote:
> On 3/19/18 1:08 PM, Selena Deckelmann wrote:
>> There's a lot of thinking that went into the agreement we have with
>> Cloudflare to enable this experiment in a way that respects user privacy.
>
> I would like us to be very clear that there are two separate things here:
>
> 1)  Is this behavior good for users?
> 2)  Will people think this behavior is good for users and for them?
> (Maybe this should itself be two separate things.)
>
> Here's how I see this:

[snipping all the stuff Boris has saved me from writing]

>> I'd like the team to share this in a blog post
>> about the experiment
>
> This seems like a good start, and we may want to then make sure whatever
> information we are trying to put out there actually gets picked up by
> widely-enough-read news bits so people aren't blindsided.

It'd be great if we could write about it in such a compelling way that
people get excited about participating in the experiment (BTW I think
opt-in is best). This work is important and, if successful, will help us
protect people's privacy more effectively. Let's do everything we can to
make sure it's a success.

Peter

As one of the folks who brought up the initial concern let me be clear that
at this point my only real concern here is one of optics. The DoH service
we're using is likely more private than anything the user is currently
using. I just don't want to see random folks on the web "discover" these
DoH requests and not be able to find details about them and so cause a
press cycle.

Having a good blog post up, particularly if it jumps out when you search
for the address that we're querying for DoH and gives good instructions on
how to opt-out does a lot to mitigate that. Reducing the population would
likely also help with that if that is an option.

On 20.03.2018 04:33, Dave Townsend wrote:
> The DoH service
> we're using is likely more private than anything the user is currently
> using.

This is only true for some parts of the world.
I'd like us not to regress for our user base globally here.

On 2018-03-19 11:33 PM, Dave Townsend wrote:
> As one of the folks who brought up the initial concern let me be clear that
> at this point my only real concern here is one of optics. The DoH service

> we're using is likely more private than anything the user is currently
> using.

It isn't explicit right now that using nightly means opting in to
participating in studies like this, and I think the text of the download
page antedates our ability to do those studies. The text of the Firefox
privacy page says that prerelease products "may contain different
privacy characteristics" than release, but doesn't enumerate them. I
also can't find a public-facing description of how we handle, secure and
audit PII data in experiments involving partner organizations.

In both cases I'm confident we have solid policies and protocols there,
I just don't see a way to point a concerned user to that information.

I'm working on that now.

- mhoye

On Tue, Mar 20, 2018 at 11:43:13AM +0100, Frederik Braun wrote:
>
>
>On 20.03.2018 04:33, Dave Townsend wrote:

>> The DoH service
>> we're using is likely more private than anything the user is currently
>> using.
>

>This is only true for some parts of the world.
>I'd like us not to regress for our user base globally here.

That's only true to a certain extent. Some parts of the world may have
better regulations for ISPs than others, but the situation for users on
shared networks and unencrypted WiFi is more or less the same. DNS is
still the most easily snoopable and spoofable security/privacy weak
point for those users.

In any case, the point of this experiment, as described, is to figure
out how useful/workable DoH would be for users in the wild. If we limit
the study to certain regions, it's hard to say with any certainty that
users in other regions wouldn't benefit from it.

On Tue, Mar 20, 2018 at 3:02 AM, Henri Sivonen <hsiv...@hsivonen.fi> wrote:

> I understand that the goal is better privacy. But it's likely that
> people get outraged if a browser sends information about what is
> browser to an off-path destination without explicit consent regardless
> of intention, nightliness or promises the destination has made.
>

Today the Mozilla blog says:

> At Mozilla, our approach to data is simple: no surprises, and user choice
is critical.
https://blog.mozilla.org/blog/2018/03/20/mozilla-statement-petition-facebook-cambridge-analytica/

It would be unfortunate to undermine that message in the current
environment.

Rob
--
Su ot deraeppa sah dna Rehtaf eht htiw saw hcihw, efil lanrete eht uoy ot
mialcorp ew dna, ti ot yfitset dna ti nees evah ew; deraeppa efil eht. Efil
fo Drow eht gninrecnoc mialcorp ew siht - dehcuot evah sdnah ruo dna ta
dekool evah ew hcihw, seye ruo htiw nees evah ew hcihw, draeh evah ew
hcihw, gninnigeb eht morf saw hcihw taht.

Note, this effort is already being reported in the tech press based on this
thread.  For example:

https://www.theregister.co.uk/AMP/2018/03/20/mozilla_firefox_test_of_privacy_mechanism_prompts_privacy_worries/

A blog post does sound like a good idea.

On Mon, Mar 19, 2018, 11:33 PM Dave Townsend <dtow...@mozilla.com> wrote:

> As one of the folks who brought up the initial concern let me be clear that

> at this point my only real concern here is one of optics. The DoH service

> we're using is likely more private than anything the user is currently

> using. I just don't want to see random folks on the web "discover" these
> DoH requests and not be able to find details about them and so cause a
> press cycle.
>
> Having a good blog post up, particularly if it jumps out when you search
> for the address that we're querying for DoH and gives good instructions on
> how to opt-out does a lot to mitigate that. Reducing the population would
> likely also help with that if that is an option.
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

+1 (as a Moz fan and privacy expert)

On Tue, Mar 20, 2018 at 2:35 AM, Kris Maglione <kmag...@mozilla.com> wrote:
> On Mon, Mar 19, 2018 at 07:27:39PM -0700, Nicholas Alexander wrote:
>>
>> Hi all,
>>
>> On Mon, Mar 19, 2018 at 3:56 PM, Xidorn Quan <m...@upsuper.org> wrote:
>>
>>> It's fine to embed this experiment in the product, and blog about it, but
>>> it's definitely not fine to have it enabled by default and send every DNS
>>> request to a third-party.
>>>
>>> I can understand that the intent must be good, and for better privacy,
>>> but
>>> the approach of doing so is not acceptable. Users would think Firefox is
>>> going to just send data to arbitrary third-party without agreement from
>>> them.
>>>
>>> As you can see from the replies, almost all people outside the network
>>> team has expressed concerns about this, which should be considered a
>>> signal
>>> already that how other technical users may feel about this experiment,
>>> and
>>> how technical news would create a title for this.
>>>
>>
>> Let me add my voice as a person outside the network team who can
>> understand
>> the concerns and _still thinks we should be doing this_.
>
>
> I'll second this.
>
> I think it's reasonable to be concerned about the public reaction to this,
> but I also think it's worth doing. The end result of this will almost
> certainly be improved privacy and security for users who have this enabled
> by default, and we can't get to that point without doing a study like this.
>
> I think it's worth the risk of a backlash. But I also think we should do
> what we can to minimize that backlash.

>
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

--
Joseph Lorenzo Hall
Chief Technologist, Center for Democracy & Technology [https://www.cdt.org]
1401 K ST NW STE 200, Washington DC 20005-3497
e: j...@cdt.org, p: 202.407.8825, pgp: https://josephhall.org/gpg-key
Fingerprint: 3CA2 8D7B 9F6D DBD3 4B10  1607 5F86 6987 40A9 A871

CDT's Annual Dinner, Tech Prom, is March 29, 2018. Don't miss the tech
event of the year!
Reserve a table today.: https://cdt.org/annual-dinner/

On Tuesday, March 20, 2018 at 3:35:48 AM UTC+1, Kris Maglione wrote:
> >Let me add my voice as a person outside the network team who can understand
> >the concerns and _still thinks we should be doing this_.
>
> I'll second this.
>
> I think it's reasonable to be concerned about the public reaction to
> this, but I also think it's worth doing.

I don't see anyone in this thread arguing against doing this.  I also don't see any arguments why this *needs* to be opt-out?  Or even why we can't open a tab with details upon enabling it (if it's on by default).

Otherwise, someone should start drafting the next apology.

On 3/21/18 10:02 AM, tom...@gmail.com wrote:
> I also don't see any arguments why this *needs* to be opt-out?

The point is to gather data on how this behaves in the wild.  If the
study is opt-in, then you have to try to figure out what part of the
effect you're seeing (if any) is just selection effects.

Controlling for selection effects is possible if you have enough
information about who is selecting into the study vs not selecting into
it to see whether there are biases in the study participants.  But doing
that requires having _way_ more information about the study participants
than Mozilla has or wants to have.

So this doesn't _need_ to be opt-out, as long as you're willing to not
believe any data it produces.  But then what's the point?

-Boris

On Wednesday, March 21, 2018 at 3:30:48 PM UTC+1, Boris Zbarsky wrote:
> The point is to gather data on how this behaves in the wild.  If the
> study is opt-in, then you have to try to figure out what part of the
> effect you're seeing (if any) is just selection effects.

From my understanding of Patrick's original message, the control in this study isn't "the 50% of Nightly users who don't have DoH enabled", but instead the existing DNS calls on each and every resolve:

On Saturday, March 17, 2018 at 11:51:02 AM UTC+1, Patrick McManus wrote:
> This initial test is focused on performance feasibility assessment and we
> won't actually be using the DNS data returned from the DoH server (i.e. the
> traditional DNS service is used in parallel and only those answers are used
> - the code calls this shadow mode.)

While yes, self-selection bias could still influence the results, it isn't obvous to me at least that it would be significant.

> So this doesn't _need_ to be opt-out, as long as you're willing to not
> believe any data it produces.  But then what's the point?

Thats seems like an exageration.


-Tom

I have a couple of further questions:

One is about the legal impact on users. DNS mangling is part of law
enforcement strategies in many parts of the world (incl Germany). We
should restrict this experiment to regions where Mozilla knows that
there's no legal trouble of using DoH and cloudflare. Circumventing law
enforcement can get pretty hairy in some regions, I suspect.

The other is a bit more detail on the scope of Mozilla's agreement with
cloudflare extending the experiment. Does our agreement extend to people
not using Firefox? What happens to folks that in some weird way are
stuck with the experiment DoH setup once the experiment ends? It'd be a
great pitch if the agreement was that cloudflare offers this service
with said terms. If they stopped liking the terms, they'd have to shut
the service down.

These questions are really only about the scope, not so much about if we
should do it.

Axel

Am 19.03.18 um 18:08 schrieb Selena Deckelmann:
> Hi!
>
> Thanks for all the thoughtful comments about this experiment. The intent of
> this work is to provide users privacy-respecting DNS. Status quo for DNS
> does not offer many users reasonable, informed choice about log retention,
> and doesn't offer encrypted DNS. Users also cannot be reasonably expected
> to negotiate on their own with their ISPs/VPN providers for things like
> 24-hour retention for logs that can be used to create profiles. Today's
> default environment (speaking technically wrt lack of encryption and log
> storage, and also in terms of the regulatory environment in the US) allows
> *all* of this data to be collected indefinitely and sold to third parties.
>
> There's a lot of thinking that went into the agreement we have with
> Cloudflare to enable this experiment in a way that respects user privacy.
> We also want to explain the impact we think this kind work will have on the
> privacy of the Internet. I'd like the team to share this in a blog post
> about the experiment, and so have started work with them on it. More on
> this shortly!
>
> -selena
>
>
>
> On Mon, Mar 19, 2018 at 8:16 AM Daniel Stenberg <dste...@mozilla.com>
> wrote:
>
>> On Mon, 19 Mar 2018, Martin Thomson wrote:
>>
>>> I don't know if it is possible to know if you have a manually-configured
>> DNS
>>> server, but disabling this experiment there if we can determine that
>> would
>>> be good - that might not be something to worry about with Nightly, but it
>>> seems like it might be needed for this to hit the trains.
>>>
>>> How do we otherwise determine that a DNS server is not safe to replace?
>>> Split horizon DNS is going to cause unexpected failures when users -
>>> particularly enterprise users - try to reach names that aren't public.
>>> That's not just an enterprise thing; this will break my home router in
>> some
>>> ways as well, though I'm actually totally OK with that in this case :)
>>
>> I don't think it is possible - with any particularly high degree of
>> certainty
>> - to know if a DNS server has been manually configured (or even if the term
>> itself is easy to define). The system APIs for name lookups typically don't
>> even expose which DNS server they use, they just resolve host names to
>> addresses for us.
>>
>> For TRR, we've instead focused pretty hard on providing a
>> "retry-algorithm" so
>> that Firefox can (if asked), retry a failed name resolve or TCP connect
>> without TRR and then "blacklist" that host for further TRR use for a period
>> into the future.
>>
>> For hosts that are TRR-blacklisted this way, we also check the next-level
>> domain of it in the background to see if we should also blacklist the whole
>> domain from TRR use. Ie if "www.example.com" fails with TRR, it gets
>> blacklisted, retried with the native resolver and "example.com" is tested
>> to
>> see if the entire domain should be blacklisted.
>>
>> --
>>
>>    / daniel.haxx.se

On 3/21/18 10:53 AM, tom...@gmail.com wrote:
> On Wednesday, March 21, 2018 at 3:30:48 PM UTC+1, Boris Zbarsky wrote:
>> The point is to gather data on how this behaves in the wild.  If the
>> study is opt-in, then you have to try to figure out what part of the
>> effect you're seeing (if any) is just selection effects.
>
>  From my understanding of Patrick's original message, the control in this study isn't "the 50% of Nightly users who don't have DoH enabled", but instead the existing DNS calls on each and every resolve

Sure.  But the point is that you have to assume that the relationship
between DoH and the existing DNS calls among the study sample is somehow
representative of that relationship in the overall userbase, right?

> While yes, self-selection bias could still influence the results, it isn't obvous to me at least that it would be significant.

My point is that to evaluate whether it's significant one needs to know
about who is self-selecting into the study and how they differ from the
people who are not self-selecting.

How would you propose evaluating that?

>> So this doesn't _need_ to be opt-out, as long as you're willing to not
>> believe any data it produces.  But then what's the point?
>
> Thats seems like an exageration.

OK, fair.  If we have a large-enough group self-select in and BoH turns
out to not work well enough for a large enough percentage of it, that
can be used to place a lower bound on the fraction of users it won't
work for in general.

But conclusions other than that seem a bit hard to come by...

-Boris

P.S.  There is, of course, the problem that the nightly audience is
_already_ subject to serious selection effects.

P.P.S. Note that I am not explicitly advocating for this study; just
presenting an argument for why making sense of opt-in data might be ...
complicated at best.

On 3/21/18 9:04 AM, Axel Hecht wrote:
> I have a couple of further questions:
>
> One is about the legal impact on users. DNS mangling is part of law
> enforcement strategies in many parts of the world (incl Germany).

Would you mind describing this in more detail? What kind of DNS mangling
do you have in mind? How is the transport method used for DNS resolution
(HTTPS vs. unsecured TCP or UDP) relevant? Why would the end user take
on legal responsibility?

> We
> should restrict this experiment to regions where Mozilla knows that
> there's no legal trouble of using DoH and cloudflare. Circumventing law
> enforcement can get pretty hairy in some regions, I suspect.

Pending your answers to the questions above, I don't see how this is a
matter of circumventing law enforcement (HTTPS is used for just about
everything these days, so resolving DNS queries over HTTPS is simply
another use case).

> The other is a bit more detail on the scope of Mozilla's agreement with
> cloudflare extending the experiment. Does our agreement extend to people
> not using Firefox?

Are you thinking about other Mozilla applications (say, Lockbox), or
non-Mozilla applications? The scope of the proposed shield study is
Firefox Nightly, so perhaps the agreement is limited to that, but I
don't know.

> What happens to folks that in some weird way are
> stuck with the experiment DoH setup once the experiment ends?

How would that happen?

> It'd be a
> great pitch if the agreement was that cloudflare offers this service
> with said terms. If they stopped liking the terms, they'd have to shut
> the service down.

Typically, agreements of this kind have clauses that enable either party
to disengage under certain conditions. I'd think that if Mozilla stops
liking the terms, we can stop using their service. Forcing Cloudfare to
stop offering a service seems a bit orthogonal to the proposed shield
study because we can always turn it off in Nightly. What happens after
the study is done is another matter (I don't know if the agreement
extends beyond the scope of this study).

> These questions are really only about the scope, not so much about if we
> should do it.

With appropriate safeguards, we should do what is good for Mozilla users
(including their privacy and security) and the health of the Internet in
general:

https://www.mozilla.org/en-US/about/manifesto/

We think that DNS over HTTPS improves the privacy and security of
Mozilla users (which is why we've been working to implement and deploy
it, and why the proposed shield study is important), so all things being
equal we should do it (IMHO). Of course we should do it in the most
transparent and user-respecting manner possible, but we also need to
keep the broader goal in mind because the existing state of DNS
resolution (and the user information is throws off) is not good. We can
and should do better by our users.

Peter

Am 21.03.18 um 19:03 schrieb Peter Saint-Andre:

> On 3/21/18 9:04 AM, Axel Hecht wrote:
>> I have a couple of further questions:
>>
>> One is about the legal impact on users. DNS mangling is part of law
>> enforcement strategies in many parts of the world (incl Germany).
>
> Would you mind describing this in more detail? What kind of DNS mangling
> do you have in mind? How is the transport method used for DNS resolution
> (HTTPS vs. unsecured TCP or UDP) relevant? Why would the end user take
> on legal responsibility?

Starting with the last question. The actual question goes the other way
around. Can someone with keys to a jail get away with blaming a Firefox
user? Totally. Google for Mehmet Altan for an example of somebody that,
according to the local high court shouldn't be in jail, is in jail.

We also have public record now that Mozilla employees state that
choosing Nightly is choosing Mozilla's choices.

As for concrete examples of using DNS mangling, I found quite a varied
set by searching for "dns censorship" on google. Germany, China, Turkey,
and so forth. Each with their own local agenda. Child porn, Hitler fan
sites, and so forth.

Going back up one question. Secure transport intercepts a couple of MITM
routes for local institutions. Be that ISPs or Great Walls or so forth.

Going back up to a topic you didn't enumerate. Cloudfront may or may not
have a point of presence (POP) in your local legislation. They may or
may not adhere to your local legislation. They may or may not do that
timely.

If they do, and if they do faster than everybody else, we're OK. If they
don't, and it matters if it's a crime or not, and it's not a crime,
we're OK.

Otherwise, we're not OK.

>
>> We
>> should restrict this experiment to regions where Mozilla knows that
>> there's no legal trouble of using DoH and cloudflare. Circumventing law
>> enforcement can get pretty hairy in some regions, I suspect.
>
> Pending your answers to the questions above, I don't see how this is a
> matter of circumventing law enforcement (HTTPS is used for just about
> everything these days, so resolving DNS queries over HTTPS is simply
> another use case).

The search results for "dns censorship" give a good overview here.

>> The other is a bit more detail on the scope of Mozilla's agreement with
>> cloudflare extending the experiment. Does our agreement extend to people
>> not using Firefox?
>
> Are you thinking about other Mozilla applications (say, Lockbox), or
> non-Mozilla applications? The scope of the proposed shield study is
> Firefox Nightly, so perhaps the agreement is limited to that, but I
> don't know.

I suspect that someone savvy could inspect our experiment to find out
the DoH service we use from cloudfront. Maybe they can set it up in
Chromium or Edge (don't know the technical details). If so, would they
be covered?

>> What happens to folks that in some weird way are
>> stuck with the experiment DoH setup once the experiment ends?
>
> How would that happen?

Stuck on an old build would be one example. Trying to play with your
local preferences back and forth, and not resetting your pref but
setting your local prefs to the cloudfront setup.

Or just boogz.

Axel

How would this be expected to impact those users who are using local or remote DNS services which filter results bases on security (malware) or content (pornography, gambling)?

What is the expected reply to individuals which use local (intranet) DNS services and did not wish to leak information about local infrastructure?


Thank you

- Tom

One relevant point: Google's DNS service has a very good privacy policy:

https://developers.google.com/speed/public-dns/privacy

I can't find comparable retention policies and details for Cloudflare; are they public anywhere?

On Sunday, March 18, 2018 at 6:18:32 PM UTC-7, Eric Shepherd (Sheppy) wrote:
> I definitely see some easy ways this could be problematic from a public
> relations perspective given things going on in the industry these days and
> some of our own mistakes the in the past. It's definitely worth taking a
> little while to consider the implications before throwing the switch.
>
> On Sun, Mar 18, 2018 at 8:39 PM, Dave Townsend <dtow...@mozilla.com>
> wrote:
>
> > On Sun, Mar 18, 2018 at 5:27 PM Patrick McManus <pmcm...@mozilla.com>
> > wrote:
> >
> > > Obviously, using a central resolver is the downside to this approach -
> > but
> > > its being explored because we believe that using the right resolver can
> > be
> > > a net win compared to the disastrous state of unsecured local DNS and
> > > privacy and hijacking problems that go on there. Its just a swamp out
> > there
> > > (you can of course disable this from about:studies or just by setting
> > your
> > > local trr.mode pref to 0 - but this discussion is meaningfully about
> > > defaults.)
> > >
> >
> > I believe that a good resolver makes all the difference. I'm just concerned
> > about the privacy aspects of this, particularly since we're not really
> > messaging this to users. Is there a reason we need a full 50% of Nightly
> > population to get the data we need here?
> >
> > On that topic I'm interested in what data we expect to get, is it just
> > comparing how the resolver performs from a variety of locations and for a
> > variety of lookups?
> > Is there some mechanism in place for users who's normal DNS resolver
> > intentionally returns different results to global DNS (e.g. for region
> > spoofing etc.)?
> >
> >
> > > And in this case the operating agreement with the dns provider is part of
> > > making that right choice. For this test that means the operator will not
> > > retain for themselves or sell/license/transfer to a third party any PII
> > > (including ip addresses and other user identifiers) and will not combine
> > > the data it gets from this project with any other data it might have. A
> > > small amount of data necessary for troubleshooting the service  can be
> > kept
> > > at most 24 hrs but that data is limited to name, dns type, a timestamp, a
> > > response code, and the CDN node that served it.
> > >
> >
> > Not retaining IP addresses is good. Can they perform aggregate tracking of
> > hostname requests, or tie common hostname requests from an origin together
> > somehow? What is our recourse if they break this agreement (the recent
> > Facebook debacle seems likely to make folks jumpy).

> > _______________________________________________
> > dev-platform mailing list
> > dev-pl...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
>
>
>

> --
>
> Eric Shepherd
> Senior Technical Writer
> Mozilla
> Blog: http://www.bitstampede.com/
> Twitter: http://twitter.com/sheppy
> Check my Availability <https://freebusy.io/eshepherd@mozilla.com>

On Monday, March 19, 2018 at 10:59:27 PM UTC-4, Boris Zbarsky wrote:
> * A lot of this is about trust; both building trust and destroying it.

And Mozilla is on *incredibly* thin ice right now, in terms of community trust. Long-time Firefox user here, very invested in Mozilla's mission, and I've been aghast at some recent decisions.

Now is not the time to play with trust. Opt-in is the only way Mozilla can support user self-determination, differentiate itself from other browser vendors, and—frankly—survive.

 - Tim McCormack