Speculation: QUIC as a transport layer for RPC

[ondr...@seznam.cz]

While QUIC is being developed to replace TCP under HTTP 2, i think it's also a good fit for RPC.

Independent streams:
Self explanatory, multiple streams with connection flow control and congestion avoidance, without packet loss blocking unrelated streams. To satisfy e-order, messages to a capability would have to be transmitted over one stream.

0-RTT connections (with encryption):
With the cost of some per connection persistent state, 0-RTT connections are possible. Especially useful in 3+ party scenarios. The only catch is that forward secrecy kicks in only after 1-RTT, so critical data should be held off for that period.

Forward error correction:
Again, by sending more data up-front, some amount of lost packets can be reconstructed, avoiding extra round trips.

I'm not suggesting we should immediately go and implement it, it's still in development, and adoption is only in chromium/google services. But the package of independent streams and reduced RTTs seems really enticing over building something specially for Cap'n Proto.

[Kenton Varda]

Last time I looked at QUIC (long ago), the implementation was too tied with Chrome and HTTP to be useful, but maybe it has changed now that it's getting closer to standardization.

For 3-party handoff scenarios, we want 0-RTT that does not require having ever connected to that party before, but can instead be initialized with a pre-shared key. The introducer would generate a key and send it to both of the parties being introduced. In this case you get something like forward secrecy even with 0-RTT, since the PSK is in fact an ephemeral key for just that one session (and is never stored to disk).

Maybe QUIC can do that, though it's a little different from what it was designed for.

Also tricky is the fact that Cap'n Proto isn't really a stream protocol. It's a bunch of independent requests which may or may not have ordering constraints with respect to each other. It would be useful if the Cap'n Proto RPC system could receive the unordered requests and then sort them out for itself, enforcing e-order as necessary, rather than have the underlying transport always provide perfect in-order delivery.

But certainly, I'm not excited about writing a new UDP protocol from scratch... so it would be nice to reuse if it fits.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[i...@ucantrip.com]

Hi,

    Let me mention ENet as an alternative pretty designed lightweight UDP library.  It is simple and robust network communication layer on top of UDP.The primary feature it provides is optional reliable, in-order delivery of packets.

    ENet omits certain higher level networking features such as authentication, lobbying, server discovery, encryption, or other similar tasks that are particularly application specific so that the library remains flexible, portable, and easily embeddable.

    ENet is event based asynchronous library that may be better match with cap'n proto design.

    Please look at ENet Features specially channels and adaptability. we can set bandwidth limits for every channel. When Cap'n proto Implements Fibers, It would allow us to prioritize different channels more precisely (I think in separating RPC commands and high priority messages  from normal messages (that may be so big) in different channels).

QUIC looks like a good all in one alternative to TCP. But I afraid it may bloat the code with huge dependencies (like boringssl library).

Side note: I think ENet on github is more active than QUIC.

Finally I encourage you to put more effort to support UDP based transports.

[ondr...@seznam.cz]

I don't think intruducer generated PSK is a good idea. If the PSK is a symmetric key, then a connection made on behalf of a compromised introducer will be insecure, and so will be all the connections introduced over this one, potentially posining the whole network. Having the PSK be two asymmetric keys leads to slightly better outcome: second order connections would only be readable in one direction, but it doesn't solve the whole poison thing.

Instead, each node should generate iteself DH pair, the public value of which it advertises to it's neighbors. The introducers job would then be to send each side the DH of their peer (maybe mixing in salt).

About streams vs messages. The semantics of sending a stream vs message is that stream packets should arrive in order (to reduce buffering), while message packets just need to arrive. Which makes the retransmission mechanism simpler and more efficient. Ideally you would have both, streams for stream-like data, and messeges for message-like data. But you can always adapt a stream for messages by framing, while adapting messages for a stream isn't as easy.

Finally, a thing to consider about UPD protocols is path MTU discovery, which cannot use ICMP over raw sockets because they need admin privileges.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

[Kenton Varda]

On Sat, Jun 16, 2018 at 8:14 AM, <ondr...@seznam.cz> wrote:

I don't think intruducer generated PSK is a good idea. If the PSK is a symmetric key, then a connection made on behalf of a compromised introducer will be insecure, and so will be all the connections introduced over this one, potentially posining the whole network.

This is already fundamentally true regardless of what crypto you use at the transport level. If the introducer is evil (or compromised), they may introduce you to an evil middleman instead of the party that was intended. You cannot trust an introduction any more than you trust the introducer. This is a fundamental aspect of the capability model (and, incidentally, of real life, totally independent of computers).

In order to trust the introduction more than the introducer, one of the following must occur:

1. Two different introducers introduce you to the same third party. If you can prove they introduced you to the *same* party, then you can combine the trust from both introducers.

2. The party authenticates themselves independently of any introducer, e.g. by presenting a certificate.

In either of these scenarios, some sort of additional cryptographic handshake certainly has to occur, so that the original PSK is no longer sufficient to decode the session. But most introductions won't lead to either of these scenarios, in which case there is no need for an additional handshake, because there is no additional security to be gained.

Having the PSK be two asymmetric keys leads to slightly better outcome: second order connections would only be readable in one direction, but it doesn't solve the whole poison thing.

Instead, each node should generate iteself DH pair, the public value of which it advertises to it's neighbors. The introducers job would then be to send each side the DH of their peer (maybe mixing in salt).

This creates a few new (admittedly minor) problems, though:

- Each actor's private key must be relatively long-lived, so that an introducer can use it at any later date to initiate an introduction. If that private key leaks, then all sessions can now be decrypted, including ones in the past. (Alternatively, some key rotation strategy can be devised, but this is cumbersome.)

- DH operations are relatively expensive in CPU time, compared to already having a PSK.

- The protocol complexity is generally higher.

If this approach provided a meaningful security benefit, then it could be worthwhile... but it's not clear to me that it does.

 

About streams vs messages. The semantics of sending a stream vs message is that stream packets should arrive in order (to reduce buffering), while message packets just need to arrive. Which makes the retransmission mechanism simpler and more efficient. Ideally you would have both, streams for stream-like data, and messeges for message-like data. But you can always adapt a stream for messages by framing, while adapting messages for a stream isn't as easy.

Adapting a stream for messages is indeed easy, but of course comes at a performance cost if the application doesn't actually require ordering, or especially for real-time use cases where a delayed message is useless anyway and should be dropped rather than retransmitted. To me part of the purpose of using a UDP protocol would be for Cap'n Proto to be able to expose such functionality to applications (in a controlled, opt-in way), so if we can't achieve that with QUIC, then I'm much less excited about it.

Finally, a thing to consider about UPD protocols is path MTU discovery, which cannot use ICMP over raw sockets because they need admin privileges.

The kernel still does its own MTU discovery and you can query it, though.

But yes, there is certainly lots of complexity to implementing UDP-based protocols. Congestion control is also quite complicated.

-Kenton

 

On Tuesday, June 12, 2018 at 3:05:01 AM UTC+2, Kenton Varda wrote:

Last time I looked at QUIC (long ago), the implementation was too tied with Chrome and HTTP to be useful, but maybe it has changed now that it's getting closer to standardization.

For 3-party handoff scenarios, we want 0-RTT that does not require having ever connected to that party before, but can instead be initialized with a pre-shared key. The introducer would generate a key and send it to both of the parties being introduced. In this case you get something like forward secrecy even with 0-RTT, since the PSK is in fact an ephemeral key for just that one session (and is never stored to disk).

Maybe QUIC can do that, though it's a little different from what it was designed for.

Also tricky is the fact that Cap'n Proto isn't really a stream protocol. It's a bunch of independent requests which may or may not have ordering constraints with respect to each other. It would be useful if the Cap'n Proto RPC system could receive the unordered requests and then sort them out for itself, enforcing e-order as necessary, rather than have the underlying transport always provide perfect in-order delivery.

But certainly, I'm not excited about writing a new UDP protocol from scratch... so it would be nice to reuse if it fits.

-Kenton

On Mon, Jun 11, 2018 at 3:11 AM, <ondr...@seznam.cz> wrote:

While QUIC is being developed to replace TCP under HTTP 2, i think it's also a good fit for RPC.

Independent streams:
Self explanatory, multiple streams with connection flow control and congestion avoidance, without packet loss blocking unrelated streams. To satisfy e-order, messages to a capability would have to be transmitted over one stream.

0-RTT connections (with encryption):
With the cost of some per connection persistent state, 0-RTT connections are possible. Especially useful in 3+ party scenarios. The only catch is that forward secrecy kicks in only after 1-RTT, so critical data should be held off for that period.

Forward error correction:
Again, by sending more data up-front, some amount of lost packets can be reconstructed, avoiding extra round trips.

I'm not suggesting we should immediately go and implement it, it's still in development, and adoption is only in chromium/google services. But the package of independent streams and reduced RTTs seems really enticing over building something specially for Cap'n Proto.

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.

[M. Taha]

Hi Kenton & G_glop,

As Kenton finalized his last post in this topic:

>But yes, there is certainly lots of complexity to implementing UDP-based protocols. Congestion control is also quite complicated.

Your comment on ENet related post is appreciated. I need to build a Cannp based application on UDP protocol and it would be better if it is more supprted in Capn"n proto level. ENet seems to be a powerful small library, Any advice on how to use it with current Cap"n proto UDP implementation.

[Kenton Varda]

Sorry, I don't know anything about enet.

-Kenton

--
***Web:

*
*** www.must.edu.eg <http://www.must.edu.eg/>

*
***Facebook:

*

*** facebook.com/mustuni <http://facebook.com/mustuni>


****Twitter:

*

*** 

*twitter.com/must_university <http://twitter.com/must_university>

***Instagram: instagram.com/mustuni <http://instagram.com/mustuni>

*
**

***Pinterest: pinterest.com/mustuni <http://pinterest.com/mustuni>

*
**


*Think Green – *Please do not print this email unless you really need to.*


*
******-*

**   *This email message is intended for the use of the person
to whom it has been sent, and may contain information that is confidential
or legally protected. If you are not the intended recipient or have
received this message in error, you are not authorized to copy, distribute,
or otherwise use this message or its attachments. Please notify the sender
immediately by return e-mail and permanently delete this message and any
attachments. Misr University for Science & Technology makes no warranty
that this email is error or virus free.** Thank you.*

*


*
***

[ondr...@seznam.cz]

Now thinking about it, DH keys don't have a point. Either you don't care what the introducer sees (or who you're introduced to), or you immediatly upgrade the connection using certificates/joins, which would be pipelined, so nothing to be gained from completing part of it ahead of time.

More practical question of introductions is authorization.

Normally you worry about ip spoofing as a DDoS amplification tool, but here you don't even have to spoof anything. You can just instruct the server to send something someone, as a normal part of the protocol. Other protocols either eat an extra round trip and/or pad the initial packets to validate ip ownership. I'm wondering if Cap'n proto can do something better, Maybe include a capability that points to a geologically closer node, which you thrust to decrement a counter on each validation and reject if it reaches zero. (You would use your personal capability to increment it) Thus limiting the amount the number of connections that could be made without your intervention.

Although with that the chance of an existing protocol being suitable decreases. But it's always possible to just use parts of a protocol, for example QUICs ip migration, which is essential you if want long lived connections on mobile devices.

[Luka Perkov]

Hello there,

It's been a while since this post was active and it might be a good timing now to get some input on this from the community. As you all well know RFC9000 has been recently published and I'm curious if there has been any discussions on how to make QUIC as a transport layer within capnproto?

In general we think QUIC is the way to go for a number of use-cases (including RPC). For example, performance wise one can combine (a) for the RX side AF_XDP socket so that the QUIC "payload" can be consumed directly in user space application (in-kernel fast path) and (b) for the TX side leverage the new features within io_uring framework such as zerocopy send [1].

To best of my knowledge, today there is no (de-facto) standard RPC mechanism built on top of QUIC and we anticipate that this will change.

We (Sartura) would be more then willing to take part in defining and implementing QUIC support within capnproto.

Looking forward to any inputs & suggestions,

Thanks,

Luka

[1] https://git.kernel.dk/cgit/linux-block/log/?h=for-5.20/io_uring-zerocopy-send

[Ian Denhardt]

I guess my question is, what even needs defining? It's tempting to try
to use the independent streams for flow control, but:

- By now capnp-c++ already has some built-in support for this that will
need to stay there (and probably be extended eventually) for other
transports.
- The go implementation also already has some rudimentary flow control
support, and probably in the next month or two I'll end up extending
this to be fairly complete.
- Getting capnp rpc to use an underlying multiple-streams transport
smells to me like a *major* engineering/design effort, resulting in
what may effectively be a different protocol.

...So I'm not sure it's even worth trying to leverage QUIC streams.

Once that's out the window, is there anything to do beyond just use it
like it's TCP? It's quite possible I'm missing something, as I have a
very cursory understanding of QUIC.

-Ian

Quoting Luka Perkov (2022-07-31 12:24:27)

> Visit this group at [1]https://groups.google.com/group/capnproto.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to capnproto+...@googlegroups.com.

> Visit this group at [2]https://groups.google.com/group/capnproto.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [3]capnproto+...@googlegroups.com.
> To view this discussion on the web visit
> [4]https://groups.google.com/d/msgid/capnproto/540d1935-fbd0-40cb-bfbf-
> 1e55030ca10an%40googlegroups.com.
>
> Verweise
>
> 1. https://groups.google.com/group/capnproto
> 2. https://groups.google.com/group/capnproto
> 3. mailto:capnproto+...@googlegroups.com
> 4. https://groups.google.com/d/msgid/capnproto/540d1935-fbd0-40cb-bfbf-1e55030ca10an%40googlegroups.com?utm_medium=email&utm_source=footer