Credentials and connection pools

[Anne van Kesteren]

As I understand things we pick connections to reuse based on an origin and a credentials flag (set/unset). This got a little bit more complicated with HTTP/2 as it's not just an origin A, but also any other "origin" entries in A's certificate, but that's not what I'm after.

What I'd like to understand is the history behind using credentials as a key and what we can do to possibly change it. We now have some features that don't send credentials by default (even same-origin), such as <script type=module> and fetch(), and as a result you get two same-origin HTTP/2 connections. This plays poorly with HTTP/2 push.

A standards-related discussion on this is hosted here: https://github.com/whatwg/fetch/issues/341. (Fetch also defines when connections get reused (though it doesn't define the HTTP/2 certificate bits yet).)

(As an aside, having someone from the networking team watch whatwg/fetch and give feedback would really help that work progress faster. Alternatively, if you give me some GitHub IDs to ping when I get stuck that could work too. Much appreciated.)

[Patrick McManus]

The history predates me, I presume it was a well intentioned privacy rule -
but partitioning according to anon/non-anon is rather pointless- the peer
can correlate by address or dns-cookies just as effectively if it wishes
to.. and as you point out this partition is really painful - it has both
performance implications and often leads to hard to explain outcomes.
(fonts interacting with preconnect were a good pain poiint to highlight).
I'd be happy to get rid of the separation and doing so in gecko would be
trivial. (the anon flag is part of the hash key, it would just need to be
removed.)

> _______________________________________________
> dev-tech-network mailing list
> dev-tech...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-network
>
>

[Christian Biesinger]

Well some authentication mechanisms are per-connection, not per-request
(such as NTLM). Just make sure that this does not get co-mingled with
requests that are supposed to be anonymous.

Christian

On Sun, May 7, 2017 at 2:29 PM Patrick McManus <mcm...@ducksong.com> wrote:

> The history predates me, I presume it was a well intentioned privacy rule -
> but partitioning according to anon/non-anon is rather pointless- the peer
> can correlate by address or dns-cookies just as effectively if it wishes
> to.. and as you point out this partition is really painful - it has both
> performance implications and often leads to hard to explain outcomes.
> (fonts interacting with preconnect were a good pain poiint to highlight).
> I'd be happy to get rid of the separation and doing so in gecko would be
> trivial. (the anon flag is part of the hash key, it would just need to be
> removed.)
>
>
>
> On Sun, May 7, 2017 at 2:07 AM, Anne van Kesteren <ann...@annevk.nl>
> wrote:
>

[Patrick McManus]

hi :biesi!

Its a good point, but the hash also has some credential info in it for the
case of ntlm cause you also don't want to mix user a and user b when you
are doing conn based auth. Hopefully that wouldn't need to surface up at
whatwg/w3c level.

-P


On Sun, May 7, 2017 at 2:36 PM, Christian Biesinger <cbies...@gmail.com>
wrote:

> Well some authentication mechanisms are per-connection, not per-request
> (such as NTLM). Just make sure that this does not get co-mingled with
> requests that are supposed to be anonymous.
>
> Christian
>
> On Sun, May 7, 2017 at 2:29 PM Patrick McManus <mcm...@ducksong.com>
> wrote:
>
>> The history predates me, I presume it was a well intentioned privacy rule
>> -
>> but partitioning according to anon/non-anon is rather pointless- the peer
>> can correlate by address or dns-cookies just as effectively if it wishes
>> to.. and as you point out this partition is really painful - it has both
>> performance implications and often leads to hard to explain outcomes.
>> (fonts interacting with preconnect were a good pain poiint to highlight).
>> I'd be happy to get rid of the separation and doing so in gecko would be
>> trivial. (the anon flag is part of the hash key, it would just need to be
>> removed.)
>>
>>
>>
>> On Sun, May 7, 2017 at 2:07 AM, Anne van Kesteren <ann...@annevk.nl>
>> wrote:
>>

[Anne van Kesteren]

On Sunday, May 7, 2017 at 8:54:54 PM UTC+2, Patrick McManus wrote:
> Its a good point, but the hash also has some credential info in it for the
> case of ntlm cause you also don't want to mix user a and user b when you
> are doing conn based auth. Hopefully that wouldn't need to surface up at
> whatwg/w3c level.

So I don't know how these connection-level authentication mechanisms work in detail, but they came up in the issue as well, and might well have been the motivator for the credentials flag. If we don't want those to be used on connections that also carry requests without credentials, how do we go about that?

Can these authentication mechanisms be negotiated after the connection is opened? Can the client easily refuse? Can the server assert it won't ever start such a negotiation?

(Having statistics on how often these mechanisms are used would also be interesting, though I suspect we can't do much about it given intranet deployments and such.)

FWIW, I suspect Fetch will need to keep some handle on how to allocate connections, just to deal with WebSocket, error handling, the upcoming token binding integration, and features like preconnect.

[Patrick McManus]

an authenticated connection is the moral equivalent of having credentials
on every request on that conn.. so its non-sensical to have an anonymous
request on a authenticated connection. (so yes, if its anonymous it would
need to be on a non authenticated connection.. I'm not sure fetch needs to
say that - its sufficient to define an anonymous request as lacking
credentials I would think)

but that doesn't mean we need to ban the mingling of authenticated and non
authenticated on the same connection when we aren't using connection based
auth (which is approximately all the time outside some enterprise cases).

On Sun, May 7, 2017 at 3:33 PM, Anne van Kesteren <ann...@annevk.nl> wrote:

> On Sunday, May 7, 2017 at 8:54:54 PM UTC+2, Patrick McManus wrote:
> > Its a good point, but the hash also has some credential info in it for
> the
> > case of ntlm cause you also don't want to mix user a and user b when you
> > are doing conn based auth. Hopefully that wouldn't need to surface up at
> > whatwg/w3c level.
>
> So I don't know how these connection-level authentication mechanisms work
> in detail, but they came up in the issue as well, and might well have been
> the motivator for the credentials flag. If we don't want those to be used
> on connections that also carry requests without credentials, how do we go
> about that?
>
> Can these authentication mechanisms be negotiated after the connection is
> opened?

yes.. typically triggered by a subset of resources on the server

> Can the client easily refuse?

same process as other http based auth. I'm not sure I'd call that 'easy' -
but there ya go

> Can the server assert it won't ever start such a negotiation?
>

no.. (at least not that I know of)

> (Having statistics on how often these mechanisms are used would also be
> interesting, though I suspect we can't do much about it given intranet
> deployments and such.)
>
>

very enterprisey. Most of it is actually used to authenticate to proxies,
which isn't a consideration here, but there are a few cases of endpoints
doing windows auth.

> FWIW, I suspect Fetch will need to keep some handle on how to allocate
> connections, just to deal with WebSocket, error handling, the upcoming
> token binding integration, and features like preconnect.

[Eric Rescorla]

On Sun, May 7, 2017 at 1:12 PM, Patrick McManus <mcm...@ducksong.com>
wrote:

> an authenticated connection is the moral equivalent of having credentials
> on every request on that conn.. so its non-sensical to have an anonymous
> request on a authenticated connection. (so yes, if its anonymous it would
> need to be on a non authenticated connection.. I'm not sure fetch needs to
> say that - its sufficient to define an anonymous request as lacking
> credentials I would think)
>
> but that doesn't mean we need to ban the mingling of authenticated and non
> authenticated on the same connection when we aren't using connection based
> auth (which is approximately all the time outside some enterprise cases).

Hmm... What about when you have post-handshake auth that retroactively
blesses requests that should have been anonymous?

-Ekr

>
>
>
> On Sun, May 7, 2017 at 3:33 PM, Anne van Kesteren <ann...@annevk.nl>
> wrote:
>
> > On Sunday, May 7, 2017 at 8:54:54 PM UTC+2, Patrick McManus wrote:
> > > Its a good point, but the hash also has some credential info in it for
> > the
> > > case of ntlm cause you also don't want to mix user a and user b when
> you
> > > are doing conn based auth. Hopefully that wouldn't need to surface up
> at
> > > whatwg/w3c level.
> >
> > So I don't know how these connection-level authentication mechanisms work
> > in detail, but they came up in the issue as well, and might well have
> been
> > the motivator for the credentials flag. If we don't want those to be used
> > on connections that also carry requests without credentials, how do we go
> > about that?
> >
> > Can these authentication mechanisms be negotiated after the connection is
> > opened?
>
>

> yes.. typically triggered by a subset of resources on the server
>
>

> > Can the client easily refuse?
>
>

> same process as other http based auth. I'm not sure I'd call that 'easy' -
> but there ya go
>
>

> > Can the server assert it won't ever start such a negotiation?
> >
>

> no.. (at least not that I know of)
>
>

> > (Having statistics on how often these mechanisms are used would also be
> > interesting, though I suspect we can't do much about it given intranet
> > deployments and such.)
> >
> >

> very enterprisey. Most of it is actually used to authenticate to proxies,
> which isn't a consideration here, but there are a few cases of endpoints
> doing windows auth.
>
>
>

> > FWIW, I suspect Fetch will need to keep some handle on how to allocate
> > connections, just to deal with WebSocket, error handling, the upcoming
> > token binding integration, and features like preconnect.

[Patrick McManus]

On Sun, May 7, 2017 at 4:44 PM, Eric Rescorla <e...@rtfm.com> wrote:

>
> Hmm... What about when you have post-handshake auth that retroactively
> blesses requests that should have been anonymous?
>

TLS client auth doesn't retroactively apply. so yeah, when I said every
request on that conn I should have said every request on that conn while it
is authenticated (and it can change and whatnot, not trying to write a
taxonomy here.). windows auth has a similar property that it starts with an
unauthenticated connection and a http response at any point could choose to
start the authentication dance - but it doesn't apply backwards. You can
see why the mulitplexing of h2 booted all this stuff off the island.

[Eric Rescorla]

On Sun, May 7, 2017 at 2:06 PM, Patrick McManus <mcm...@ducksong.com>
wrote:

>

> On Sun, May 7, 2017 at 4:44 PM, Eric Rescorla <e...@rtfm.com> wrote:
>
>>
>> Hmm... What about when you have post-handshake auth that retroactively
>> blesses requests that should have been anonymous?
>>
>
>
> TLS client auth doesn't retroactively apply.
>

Not sure I agree here. One of the standard idioms is that the server
receivesn
a a sensitive request, then sends HelloRequest, and then when the handshake
completes, delivers the response.I would call that retroactive

-Ekr

[Patrick McManus]

This is probably particular to TLS, as the windows auth stuff does apply
more like http basic auth (it returns 401 to bootstrap things, rather than
making a challenge at the session layer with a request in flight)

wrt tls client auth I would think if the outstanding requests is marked
anonymous we should not allow authentication to proceed if the server sends
a helloRequest (and I have no idea whether we apply that check now or not -
that code hasn't changed in forever) just because that's a nonsenica
combinationl, right? Again, seems like we can handle that as an
implementation issue without drawing a bright line for 99% of the world
that doesn't do connection based client auth and creating a burden on
webdevs to get this junk right in markup.

[Eric Rescorla]

On Sun, May 7, 2017 at 2:46 PM, Patrick McManus <mcm...@ducksong.com>
wrote:

> This is probably particular to TLS, as the windows auth stuff does apply
> more like http basic auth (it returns 401 to bootstrap things, rather than
> making a challenge at the session layer with a request in flight)
>
> wrt tls client auth I would think if the outstanding requests is marked
> anonymous we should not allow authentication to proceed if the server sends
> a helloRequest (and I have no idea whether we apply that check now or not -
> that code hasn't changed in forever) just because that's a nonsenica
> combinationl, right? Again, seems like we can handle that as an
> implementation issue without drawing a bright line for 99% of the world
> that doesn't do connection based client auth and creating a burden on
> webdevs to get this junk right in markup.
>

I think you'd have to handle this by actually tearing down the connection
and re-initiating on another
one.

That said, I'm also worried that there are misguided servers which bind
HTTP-level authentication
to the TLS connection. Those servers are going to have a bad day if we
coalesce... Do we know
that there aren't any?

-Ekr

[Patrick McManus]

On Sun, May 7, 2017 at 5:51 PM, Eric Rescorla <e...@rtfm.com> wrote:

>
>
> On Sun, May 7, 2017 at 2:46 PM, Patrick McManus <mcm...@ducksong.com>
> wrote:
>
>> This is probably particular to TLS, as the windows auth stuff does apply
>> more like http basic auth (it returns 401 to bootstrap things, rather than
>> making a challenge at the session layer with a request in flight)
>>
>> wrt tls client auth I would think if the outstanding requests is marked
>> anonymous we should not allow authentication to proceed if the server sends
>> a helloRequest (and I have no idea whether we apply that check now or not -
>> that code hasn't changed in forever) just because that's a nonsenica
>> combinationl, right? Again, seems like we can handle that as an
>> implementation issue without drawing a bright line for 99% of the world
>> that doesn't do connection based client auth and creating a burden on
>> webdevs to get this junk right in markup.
>>
>
> I think you'd have to handle this by actually tearing down the connection
> and re-initiating on another
> one.
>

by handle it I figured that meant fail the one in flight.. why would the
server do something different on a new connection? The server demands auth,
the request is marked anonymous - that's not going to work out.

> That said, I'm also worried that there are misguided servers which bind
> HTTP-level authentication
> to the TLS connection. Those servers are going to have a bad day if we
> coalesce... Do we know
> that there aren't any?
>
>

'any' is not a standard the web can ever meet :). You're concerned that
cookies on transaction 1 will be implicitly applied to transaction 2 even
though 2 doesn't have cookies? That would be a pretty big bug I think we
would have already seen - but that's basically what the change would boil
down to.

[Eric Rescorla]

On Sun, May 7, 2017 at 3:01 PM, Patrick McManus <mcm...@ducksong.com>
wrote:

>
>

> On Sun, May 7, 2017 at 5:51 PM, Eric Rescorla <e...@rtfm.com> wrote:
>
>>
>>
>> On Sun, May 7, 2017 at 2:46 PM, Patrick McManus <mcm...@ducksong.com>
>> wrote:
>>
>>> This is probably particular to TLS, as the windows auth stuff does
>>> apply more like http basic auth (it returns 401 to bootstrap things, rather
>>> than making a challenge at the session layer with a request in flight)
>>>
>>> wrt tls client auth I would think if the outstanding requests is marked
>>> anonymous we should not allow authentication to proceed if the server sends
>>> a helloRequest (and I have no idea whether we apply that check now or not -
>>> that code hasn't changed in forever) just because that's a nonsenica
>>> combinationl, right? Again, seems like we can handle that as an
>>> implementation issue without drawing a bright line for 99% of the world
>>> that doesn't do connection based client auth and creating a burden on
>>> webdevs to get this junk right in markup.
>>>
>>
>> I think you'd have to handle this by actually tearing down the connection
>> and re-initiating on another
>> one.
>>
>
> by handle it I figured that meant fail the one in flight.. why would the
> server do something different on a new connection? The server demands auth,
> the request is marked anonymous - that's not going to work out.
>

Sorry, I should have expressed my concern differently. Say that we decide
to allow coalescence
and then we have two requests:

A (anonymous)
N (non-anonymous)

So, what happens if from the client's perspective we have N, A, HR. At this
point we
need to re-issue N on a new connection so it doesn't contaminate A. We may
actually
need to tear down the connection and start two new ones, one with N and one
with
A.

That said, I'm also worried that there are misguided servers which bind
>> HTTP-level authentication
>> to the TLS connection. Those servers are going to have a bad day if we
>> coalesce... Do we know
>> that there aren't any?
>>
>>
> 'any' is not a standard the web can ever meet :). You're concerned that
> cookies on transaction 1 will be implicitly applied to transaction 2 even
> though 2 doesn't have cookies? That would be a pretty big bug I think we
> would have already seen - but that's basically what the change would boil
> down to.
>

Not just cookies. Say, for instance, usernames and passwords. Maybe it
doesn't happen.

-Ekr

[Patrick McManus]

On Sun, May 7, 2017 at 6:13 PM, Eric Rescorla <e...@rtfm.com> wrote:

>
> So, what happens if from the client's perspective we have N, A, HR. At
> this point we
> need to re-issue N on a new connection so it doesn't contaminate A. We may
> actually
> need to tear down the connection and start two new ones, one with N and
> one with
> A.
>

req N ->
<- resp (200) N
:: N is done and gone - in caches and spread far and wide ::
req A ->
<- HR
close conn and fail A I suppose

or are you thinking of some scenario where N and A are both outstanding
with no replies? We don't have such a scenario right now (pipelines are
dead, and you can't do this in h2, how else would it happen?), but if we
did I would agree you would need to treat it as a reset and replay them on
different conns until you figured out what the HR applied to. So whatever
the h2 extension is that makes something like this happen should take that
into account - but we've gotten a bit afield from the original question,

>
> That said, I'm also worried that there are misguided servers which bind
>>> HTTP-level authentication
>>> to the TLS connection. Those servers are going to have a bad day if we
>>> coalesce... Do we know
>>> that there aren't any?
>>>
>>>
>> 'any' is not a standard the web can ever meet :). You're concerned that
>> cookies on transaction 1 will be implicitly applied to transaction 2 even
>> though 2 doesn't have cookies? That would be a pretty big bug I think we
>> would have already seen - but that's basically what the change would boil
>> down to.
>>
>
> Not just cookies. Say, for instance, usernames and passwords. Maybe it
> doesn't happen.
>

I trolling just a tad with the cookie line - cause its the 99.9% use case
here. If we had reason to think that traditional auth was broken I guess we
could design around that - but I haven't seen a case of that.

[Eric Rescorla]

On Sun, May 7, 2017 at 3:31 PM, Patrick McManus <mcm...@ducksong.com>
wrote:

>

> On Sun, May 7, 2017 at 6:13 PM, Eric Rescorla <e...@rtfm.com> wrote:
>
>>
>> So, what happens if from the client's perspective we have N, A, HR. At
>> this point we
>> need to re-issue N on a new connection so it doesn't contaminate A. We
>> may actually
>> need to tear down the connection and start two new ones, one with N and
>> one with
>> A.
>>
>
> req N ->
> <- resp (200) N
> :: N is done and gone - in caches and spread far and wide ::
> req A ->
> <- HR
> close conn and fail A I suppose
>
> or are you thinking of some scenario where N and A are both outstanding
> with no replies? We don't have such a scenario right now (pipelines are
> dead, and you can't do this in h2, how else would it happen?),
>

I was thinking of pipelining, but why would you fail A, rather than
treating it as connection
termination and restarting?

-Ekr

[Patrick McManus]

On Sun, May 7, 2017 at 6:42 PM, Eric Rescorla <e...@rtfm.com> wrote:

> I was thinking of pipelining, but why would you fail A, rather than
> treating it as connection
> termination and restarting?
>

pipelines are dead for all practical purposes (removed from firefox, not
available in chrome or msft iirc)..They never really worked, were not
widely deployed, and h2 is the right answer to the same question. Now that
h2 is widely available, there is no reason to entertain the pipeline
fantasy. So I really can't see where >1 outstanding transaction can
happen.. and my 'fail' comment was in that context - what good does
replaying the same transaction on a new connection get you.. what's
different (the conflicting property is part of the request)? if you had an
ambiguity about the HR, then retry makes sense..

Tommy Pauly from Apple did a interesting TAPS presentation at IETF 98 where
he had a slide lableed multi-streaming and listed H1 pipelines under it. It
was an interesting characterization... and for the same reason h2 doesn't
cope with post handshake auth pipelines really should not either - though I
cannot recall it ever being discussed.. that might be because pipelines
weren't much of a thing in practice.

-P

[Anne van Kesteren]

On Sunday, May 7, 2017 at 10:12:20 PM UTC+2, Patrick McManus wrote:
> an authenticated connection is the moral equivalent of having credentials
> on every request on that conn.. so its non-sensical to have an anonymous
> request on a authenticated connection. (so yes, if its anonymous it would
> need to be on a non authenticated connection.. I'm not sure fetch needs to
> say that - its sufficient to define an anonymous request as lacking
> credentials I would think)

How is that sufficient? How does Servo derive an implementation strategy from that that's not different from that of other browsers in ways that can be observed by web sites?

> but that doesn't mean we need to ban the mingling of authenticated and non
> authenticated on the same connection when we aren't using connection based
> auth (which is approximately all the time outside some enterprise cases).

Sure, then the question becomes what the new policy would be.


Let me try a description of the new model to see if we're on the same page. The connection pool which holds all connections is global for the user agent. A connection is keyed in part on a set of (domain, port) tuples, one of which needs to match.

A connection can become authenticated. This can happen at any point for the duration of the connection, except when the connection is anonymous-request-tainted.

A connection can become anonymous-request-tainted when you send an anonymous request over it. If a connection is authenticated the client won't send an anonymous request over it and instead open a new connection.

When a connection is anonymous-request-tainted and the server tries to authenticate the connection, the client fails that process. This would be a potentially breaking change.

So basically, authenticated and anonymous-request-tainted would be mutually exclusive and also part of the connection key.

(With such a setup, there's probably not much advantage to TLS extensions surrounding connection authentication. (I was thinking that if the server could assert it would never use it, we could only coalesce then (and fail later on if the server lied). Alternatively the client could assert it would never need it.)

[Eric Rescorla]

On Mon, May 8, 2017 at 12:36 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> On Sunday, May 7, 2017 at 10:12:20 PM UTC+2, Patrick McManus wrote:
> > an authenticated connection is the moral equivalent of having credentials
> > on every request on that conn.. so its non-sensical to have an anonymous
> > request on a authenticated connection. (so yes, if its anonymous it would
> > need to be on a non authenticated connection.. I'm not sure fetch needs
> to
> > say that - its sufficient to define an anonymous request as lacking
> > credentials I would think)
>
> How is that sufficient? How does Servo derive an implementation strategy
> from that that's not different from that of other browsers in ways that can
> be observed by web sites?
>

Way is this an imperative? I should think the requirement was merely that
it not break Web sites.

> > but that doesn't mean we need to ban the mingling of authenticated and
> non
> > authenticated on the same connection when we aren't using connection
> based
> > auth (which is approximately all the time outside some enterprise cases).
>
> Sure, then the question becomes what the new policy would be.
>
>
> Let me try a description of the new model to see if we're on the same
> page. The connection pool which holds all connections is global for the
> user agent. A connection is keyed in part on a set of (domain, port)
> tuples, one of which needs to match.
>

Nit: scheme/host/port

A connection can become authenticated. This can happen at any point for the
> duration of the connection, except when the connection is
> anonymous-request-tainted.
>
> A connection can become anonymous-request-tainted when you send an
> anonymous request over it. If a connection is authenticated the client
> won't send an anonymous request over it and instead open a new connection.
>
> When a connection is anonymous-request-tainted and the server tries to
> authenticate the connection, the client fails that process. This would be a
> potentially breaking change.
>

I know that this is what McManus was suggesting, but if you mean that fail
causes the fetch to fail, that seems unnecessary. We could just simulate
network error and retry on a different, non-tainted, connection I should
think.

So basically, authenticated and anonymous-request-tainted would be mutually
> exclusive and also part of the connection key.
>
> (With such a setup, there's probably not much advantage to TLS extensions
> surrounding connection authentication. (I was thinking that if the server
> could assert it would never use it, we could only coalesce then (and fail
> later on if the server lied). Alternatively the client could assert it
> would never need it.)
>

This seems like something that would have an incredibly long deployment
curve.

-Ekr

[Anne van Kesteren]

On Mon, May 8, 2017 at 1:52 PM, Eric Rescorla <e...@rtfm.com> wrote:

> On Mon, May 8, 2017 at 12:36 AM, Anne van Kesteren <ann...@annevk.nl> wrote:
>> How is that sufficient? How does Servo derive an implementation strategy
>> from that that's not different from that of other browsers in ways that can
>> be observed by web sites?
>

> Way is this an imperative? I should think the requirement was merely that it
> not break Web sites.

And if that requirement is to remain true, we should document what
sites can come to depend upon. And likely will depend on given how
HTTP/2 push works. Part of the reason I'm raising this is because
folks are running into issues with the current model. If we change the
model but don't make sure that all browsers adopt it, developers will
still end up running into issues.


> Nit: scheme/host/port

Oops, bit optimistic there.

>> When a connection is anonymous-request-tainted and the server tries to
>> authenticate the connection, the client fails that process. This would be a
>> potentially breaking change.
>

> I know that this is what McManus was suggesting, but if you mean that fail
> causes the fetch to fail, that seems unnecessary. We could just simulate
> network error and retry on a different, non-tainted, connection I should
> think.

We cannot always retry. E.g., with fetch() using streams the user
agent cannot reproduce the body (by design, to use less memory;
therefore also fails on most redirects and HTTP authentication).

>> (With such a setup, there's probably not much advantage to TLS extensions
>> surrounding connection authentication. (I was thinking that if the server
>> could assert it would never use it, we could only coalesce then (and fail
>> later on if the server lied). Alternatively the client could assert it would
>> never need it.)
>

> This seems like something that would have an incredibly long deployment
> curve.

Yeah, I suspected as much. Happy that we have alternatives to consider.


--
https://annevankesteren.nl/

[Patrick McManus]

On Mon, May 8, 2017 at 3:36 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> A connection can become anonymous-request-tainted when you send an
> anonymous request over it. If a connection is authenticated the client
> won't send an anonymous request over it and instead open a new connection.
>

In at least the common case, cookies, there is just no need for this rule
and it leads to most of the confusion we have. The connection can easily
mix cookies and no-cookies.

[Anne van Kesteren]

On Mon, May 8, 2017 at 2:28 PM, Patrick McManus <mcm...@ducksong.com> wrote:

> On Mon, May 8, 2017 at 3:36 AM, Anne van Kesteren <ann...@annevk.nl> wrote:
>> A connection can become anonymous-request-tainted when you send an
>> anonymous request over it. If a connection is authenticated the client won't
>> send an anonymous request over it and instead open a new connection.
>

> In at least the common case, cookies, there is just no need for this rule
> and it leads to most of the confusion we have. The connection can easily mix
> cookies and no-cookies.

Agreed. I don't know why you think what I wrote touches that? I'm only
talking about authenticated connections (not requests with
credentials) and requests without credentials (anonymous requests) as
it was my understanding that we did not want to mix those.


--
https://annevankesteren.nl/

[Eric Rescorla]

On Mon, May 8, 2017 at 5:17 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> On Mon, May 8, 2017 at 1:52 PM, Eric Rescorla <e...@rtfm.com> wrote:

> > On Mon, May 8, 2017 at 12:36 AM, Anne van Kesteren <ann...@annevk.nl>
> wrote:
> >> How is that sufficient? How does Servo derive an implementation strategy
> >> from that that's not different from that of other browsers in ways that
> can
> >> be observed by web sites?
> >

> > Way is this an imperative? I should think the requirement was merely
> that it
> > not break Web sites.
>
> And if that requirement is to remain true, we should document what
> sites can come to depend upon.

Sure, but that's a much weaker invariant than cannot be observed.

> And likely will depend on given how
> HTTP/2 push works. Part of the reason I'm raising this is because
> folks are running into issues with the current model. If we change the
> model but don't make sure that all browsers adopt it, developers will
> still end up running into issues.
>

Hmm... Can you describe more about what issues developers of Web sites
are running into?

> >> When a connection is anonymous-request-tainted and the server tries to
> >> authenticate the connection, the client fails that process. This would
> be a
> >> potentially breaking change.
> >

> > I know that this is what McManus was suggesting, but if you mean that
> fail
> > causes the fetch to fail, that seems unnecessary. We could just simulate
> > network error and retry on a different, non-tainted, connection I should
> > think.
>
> We cannot always retry. E.g., with fetch() using streams the user
> agent cannot reproduce the body (by design, to use less memory;
> therefore also fails on most redirects and HTTP authentication).
>

Hmm.... This seems like a separate topic, but as you describe it, this has
the
potential to interact very badly with TLS 1.3 0-RTT (and QUIC too) which
both
depend on the assumption that 0-RTT data can fail and have to be replayed
semantically.

-Ekr

[Patrick McManus]

On Mon, May 8, 2017 at 8:17 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> >> When a connection is anonymous-request-tainted and the server tries to
> >> authenticate the connection, the client fails that process. This would
> be a
> >> potentially breaking change.
> >

> > I know that this is what McManus was suggesting, but if you mean that
> fail
> > causes the fetch to fail, that seems unnecessary. We could just simulate
> > network error and retry on a different, non-tainted, connection I should
> > think.
>
> We cannot always retry. E.g., with fetch() using streams the user
> agent cannot reproduce the body (by design, to use less memory;
> therefore also fails on most redirects and HTTP authentication).

My mental model is inverted. A connection would be come ineligible for
anonymous requests only when some kind of connection based auth was done to
it (e.g. client certs, or ntlm). The presumption is that this does not
happen often. In its absence there is only one kind of connection and it
can carry anonymous and non anonymous requests.

When some kind of connection based auth is triggered there are 2 classes of
requests that have to deal with this
a] a single request in flight (this is single because this is an
h1-only-no-pipeline issue)
b] other requests routed to the same host but not yet written out

I suggest that if [a] is an anonymous request it needs to fail because [a]
triggered the authentication. We can retry it on a clean connection, but
because it itself is the trigger I don't see why that would make any
progress. Maybe I'm misundertanding ekr here.

members of [b] that are anonymous simply need to be late-bound to a
different connection. They haven't been tried yet and aren't failed
(perhaps that was part of the confusion?)

[Anne van Kesteren]

On Mon, May 8, 2017 at 2:33 PM, Eric Rescorla <e...@rtfm.com> wrote:

> On Mon, May 8, 2017 at 5:17 AM, Anne van Kesteren <ann...@annevk.nl> wrote:
>> And likely will depend on given how
>> HTTP/2 push works. Part of the reason I'm raising this is because
>> folks are running into issues with the current model. If we change the
>> model but don't make sure that all browsers adopt it, developers will
>> still end up running into issues.
>
> Hmm... Can you describe more about what issues developers of Web sites
> are running into?

One thing that is reported to happen is that the user navigates to /,
then that pushes /x and /y as well. The browser then requests /x
without credentials and doesn't end up using the pushed resource. But
you're probably right that this in essence doesn't depend on the
connection as pushes are request/response bound.

There's also a more general concern that's often aired about using
multiple connections to begin with in an HTTP/2 world, but I'm not
really sure how to qualify that concern. It seems to me that as long
as HTTP/2 goes over TCP, having multiple connections might actually be
beneficial at times.

Nevertheless, connections can be observed and recommending a model for
them doesn't seem like a bad idea to me. Having some kind of contract
as to how they work helps folks who define things like preconnect.
Since if that doesn't know about the full key used for establishing a
connection, you can end up just opening connections that go without
use and neither party really knows why.

>> We cannot always retry. E.g., with fetch() using streams the user
>> agent cannot reproduce the body (by design, to use less memory;
>> therefore also fails on most redirects and HTTP authentication).
>

> Hmm.... This seems like a separate topic, but as you describe it, this has
> the potential to interact very badly with TLS 1.3 0-RTT (and QUIC too) which
> both depend on the assumption that 0-RTT data can fail and have to be replayed
> semantically.

Ouch. Filed https://github.com/whatwg/fetch/issues/538.


--
https://annevankesteren.nl/

[Patrick McManus]

On Mon, May 8, 2017 at 8:33 AM, Eric Rescorla <e...@rtfm.com> wrote:

>
>
> > And likely will depend on given how
> > HTTP/2 push works. Part of the reason I'm raising this is because
> > folks are running into issues with the current model. If we change the
> > model but don't make sure that all browsers adopt it, developers will
> > still end up running into issues.
> >
>
> Hmm... Can you describe more about what issues developers of Web sites
> are running into?

I would encourage us not to take this thread in that direction. h2 push
impls (ours and chrome's) currently scope push to be connection based
rather than writing directly into the cache - and you can see why
connection management would play a role with that little fact of life. But
that algorithm is really just a conservative holding pattern to figure out
the right scope of pushed info. And frankly push is huge amounts of
machinery and corner cases for rather marginal benefits, so I would much
rather fit it into whatever anonymous model we want, rather than working
the other way around.

sleevi has a thread open somewhere about his concerns about not just
sticking pushed data in the cache.. and I have some concerns about what to
do with dynamic data (all pushed data can be consumed once, even if
uncachable.. which makes a lot of sense but then if the http cache is your
backing store how to you make sure more than one copy ends up in the right
place, etc..)

anyhow, that's probably not the right direction for this thread..

[Anne van Kesteren]

On Mon, May 8, 2017 at 2:39 PM, Patrick McManus <mcm...@ducksong.com> wrote:

> On Mon, May 8, 2017 at 8:17 AM, Anne van Kesteren <ann...@annevk.nl> wrote:
>> We cannot always retry. E.g., with fetch() using streams the user
>> agent cannot reproduce the body (by design, to use less memory;
>> therefore also fails on most redirects and HTTP authentication).
>

> My mental model is inverted. A connection would be come ineligible for
> anonymous requests only when some kind of connection based auth was done to
> it (e.g. client certs, or ntlm). The presumption is that this does not
> happen often. In its absence there is only one kind of connection and it can
> carry anonymous and non anonymous requests.

Yes, that's exactly what I wrote down. I still don't understand why
you keep saying it's different.

> When some kind of connection based auth is triggered there are 2 classes of
> requests that have to deal with this
> a] a single request in flight (this is single because this is an
> h1-only-no-pipeline issue)
> b] other requests routed to the same host but not yet written out
>
> I suggest that if [a] is an anonymous request it needs to fail because [a]
> triggered the authentication. We can retry it on a clean connection, but
> because it itself is the trigger I don't see why that would make any
> progress. Maybe I'm misundertanding ekr here.
>
> members of [b] that are anonymous simply need to be late-bound to a
> different connection. They haven't been tried yet and aren't failed (perhaps
> that was part of the confusion?)

Okay, so instead of failing the connection you fail just the request.
Are you also saying that only HTTP/1 can have authenticated
connections at this point?


--
https://annevankesteren.nl/

[Patrick McManus]

On Mon, May 8, 2017 at 8:55 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> Okay, so instead of failing the connection you fail just the request.
> Are you also saying that only HTTP/1 can have authenticated
> connections at this point?
>

I am saying fail the request. The disposition of the connection is a
protocol detail depending on the auth details.. It seems for
TLS-client-auth you would need to fail the connection because the http bits
are stalled mid flight, but for something like NTLM you have a clean
resolution to the auth trigger (it came back with a 401 that we're not
going to act on) and the connection could still be used for other requests.

wrt h1 - yes, I believe right now the only client-authenticated connections
are in h1.

[Patrick McManus]

On Mon, May 8, 2017 at 8:33 AM, Eric Rescorla <e...@rtfm.com> wrote:

> We cannot always retry. E.g., with fetch() using streams the user
> > agent cannot reproduce the body (by design, to use less memory;
> > therefore also fails on most redirects and HTTP authentication).
> >
>

> Hmm.... This seems like a separate topic, but as you describe it, this has
> the
> potential to interact very badly with TLS 1.3 0-RTT (and QUIC too) which
> both
> depend on the assumption that 0-RTT data can fail and have to be replayed
> semantically.

I don't think this is a fetch() level problem - those are both basically
transport issues with unacknowledged data and the transport is always going
to need a copy of that information to accommodate packet loss.. 0-rtt fail
is really just a twist on loss from a replay pov. Our definition of
transport has moved up a bit in the stack from socket buffers, but it
hasn't made it to fetch() yet :)

[Eric Rescorla]

On Mon, May 8, 2017 at 6:57 AM, Patrick McManus <mcm...@ducksong.com>
wrote:

Right. I think in this case it's the semantic layer of HTTP (i.e., the one
that spans h1/h2/h2s/h2q)

-Ekr

[Eric Rescorla]

McManus and I talked offline, and I think we agree:

1. It is not currently possible to have two outstanding on the wire
requests with
different auth semantics:

- People don't do H1 pipelining so the outstanding requests can be
late-bound to
the right connection.

- H2 prohibits renegotiation (though we might in future have to worry about
TLS 1.3 post-handshake auth or Nick Sullivan's proposed extensions).


2. With both H1 and H2, you *do* need to store unacknowledged requests so
you can replay them in the face of a variety of transport issues.

-Ekr

[Patrick McManus]

On Mon, May 8, 2017 at 11:02 AM, Eric Rescorla <e...@rtfm.com> wrote:

> 2. With both H1 and H2, you *do* need to store unacknowledged requests so
> you can replay them in the face of a variety of transport issues.
>

to further clarify - the "you" in that sentence refers to the http
implementation, not the content using xhr/fetch.. and there is some flow
control in fetch streams, so it doesn't have an unbounded commitment.

[Eric Rescorla]

Agreed.

-Ekr


On Mon, May 8, 2017 at 8:11 AM, Patrick McManus <mcm...@ducksong.com>
wrote:

>

[Anne van Kesteren]

So a simple change would be to just start reusing HTTP/2 connections
and leave HTTP/1 alone, but more aggressive seems acceptable too, if
everyone can agree on failing the request for NTLM and failing the
connection for TLS-client-auth. Would the appropriate next step be a
bug against Gecko?


The request retry issue is probably best further discussed here:
https://github.com/whatwg/fetch/issues/538 (thanks Eric!).


As for the HTTP/2 push concerns. There's quite a bit of debate here:
https://github.com/whatwg/fetch/issues/354. Including feedback from
sleevi, mt, and mnot. If you could add your perspective there that
would help I think.


Anything else?


--
https://annevankesteren.nl/

[Patrick McManus]

On Mon, May 8, 2017 at 1:30 PM, Anne van Kesteren <ann...@annevk.nl> wrote:

> So a simple change would be to just start reusing HTTP/2 connections
> and leave HTTP/1 alone, but more aggressive seems acceptable too, if
> everyone can agree on failing the request for NTLM and failing the
> connection for TLS-client-auth. Would the appropriate next step be a
> bug against Gecko?
>

I think what you describe (as more aggressive) would be a fine gecko
implementation. Hopefully fetch would say something much more generic?

[Anne van Kesteren]

On Mon, May 8, 2017 at 7:52 PM, Patrick McManus <mcm...@ducksong.com> wrote:
> I think what you describe (as more aggressive) would be a fine gecko
> implementation.

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1363284.

> Hopefully fetch would say something much more generic?

Could you explain why it's not desirable to describe this in more
detail? It sounds like you don't agree with my preconnect example and
giving developers a guaranteed contract to write code against, but you
didn't really address it.


--
https://annevankesteren.nl/

[Patrick McManus]

I think saying that anonymous and non-anonymous requests can share a
connection in most cases is fine - but that when connection auth is present
those connections cannot carry anonymous requests. I believe that satisfies
the need without talking about various bits of protocol versioning and
reuse strategies.

In general I think the previous mistake made (in various documents) was
overspecifying things about the transport and this is unwinding it. I
wouldn't unwind it by making it even more specific.

[blartp...@gmail.com]

On Sunday, May 7, 2017 at 2:07:39 AM UTC-4, Anne van Kesteren wrote:
> As I understand things we pick connections to reuse based on an origin and a credentials flag (set/unset). This got a little bit more complicated with HTTP/2 as it's not just an origin A, but also any other "origin" entries in A's certificate, but that's not what I'm after.
>
> What I'd like to understand is the history behind using credentials as a key and what we can do to possibly change it. We now have some features that don't send credentials by default (even same-origin), such as <script type=module> and fetch(), and as a result you get two same-origin HTTP/2 connections. This plays poorly with HTTP/2 push.
>
> A standards-related discussion on this is hosted here: https://github.com/whatwg/fetch/issues/341. (Fetch also defines when connections get reused (though it doesn't define the HTTP/2 certificate bits yet).)
>
> (As an aside, having someone from the networking team watch whatwg/fetch and give feedback would really help that work progress faster. Alternatively, if you give me some GitHub IDs to ping when I get stuck that could work too. Much appreciated.)

,m