On Fri, 29 Sep 2023 11:26:05 -0500
Kenton Varda <
ken...@cloudflare.com> wrote:
> On Wed, Sep 27, 2023 at 4:37 PM 'Alex' via Cap'n Proto <
>
capn...@googlegroups.com> wrote:
>
> > 1. I would like to add a new RPC Message in rpc.capnp:
> >
> > goodbye @14 :Void;
> >
> > This message indicates to the recipient that the sender has nothing
> > more to say, and that it should stop read()ing the socket. In other
> > words, upon receipt of rpc::Message::GOODBYE, messageLoop() ends
> > gracefully (no exceptions are thrown). The sender then shuts down
> > its write() side of the connection, causing a TCP FIN to be
> > delivered to the recipient. Because there is no read() in progress,
> > an exception shall not be thrown.
> >
> > The recipient performs whatever cleanup is necessary and sends a
> > reciprocal GOODBYE, causing the same logic described above to be
> > invoked on the other end.
> >
> > Do you think this is a good solution?
> >
>
> Hmm, what's the benefit of this, vs. simply sending EOF?
>
Currently when an EOF occurs, there is no way to discern between an
exceptional circumstance and a normal/expected circumstance.
For example, consider a program which performs file transfer between
two machines/vats:
filectl get capnproto://
192.168.1.2/tmp/movie.mp4
If my machine establishes a Cap'n Proto session with 192.168.1.2 and
successfully downloads the file, there is nothing left to do. If the
hypothetical filectl program simply exits, then the remote daemon is
going to raise an exception (e.g. "Peer disconnected"), and that
exception is going to be written to the now-dead connection in the form
of an RPC Abort message. Since the program has terminated, the socket
is closed and my machine will send a TCP RST packet in response.
If I am collecting metrics across a fleet of machines, and one of those
metrics is the number of exceptions thrown or the number of connection
resets, my charts will show a constant flow of exceptions, leaving me
unable to determine whether or not an outage is occurring.
> To extend the protocol in this way we would have to think about
> backwards compatibility. If a peer running an older version of capnp
> receives the "goodbye" message, it will respond with an
> "unimplemented" message, which seems like it could make things worse?
>
It's unclear to me how it would make things worse, since the connection
is in the process of being shut down anyway. I am not saying it
can't/wouldn't make things worse, I am only saying that it is not clear
to me how that could be so.
>
> > 2. In your opinion, what is the best way to expose this graceful
> > disconnect functionality to applications?
>
>
> This is a bit tricky.
>
> For rpc-twoparty.h I think it's straightforward, it could simply be a
> method on `TwoPartyClient` and `TwoPartyServer` to signal graceful
> disconnect. (This would have to be a method returning a promise which
> resolves when all buffers are flushed and such, so I don't think it
> can just be destructor behavior.)
>
I will take a look there.
> But in the full many-party vision of Cap'n Proto, the application is
> not really intended to know what connections exist. The application
> could receive two capabilities from two different parties which both
> happened to point to the same third party, and those two capabilities
> end up sharing a connection, even though they came from different
> places. So it seems like the application has no reasonable way to
> express that it wants a connection to shut down, if it doesn't even
> know a connection exists.
>
As I understand it, the RPC system has no notion of an underlying
network structure (a wonderful feature!). Capabilities may reside on
the same machine or on different machines, but it shouldn't matter to
the application. The application is only concerned about VatIds. In the
two-party case, there are only two possible VatIds, "client" and
"server". In the multi-party case, VatIds would likely take the form of
a public key. This leads to my next question: In the CapTP/E/Vat
paradigm, is it valid for a single RPC system to form multiple
independent connections to the same VatId? In other words, if I call:
connA = VatNetwork::connect(vatIdA);
followed by:
connB = VatNetwork::connect(vatIdB);
where vatIdA == vatIdB, should connA and connB refer to the same object
in memory -- thus only ever creating a single RpcConnectionState? Or,
should connA and connB instead be two independent objects in memory,
each with their own independent underlying connection and thus,
independently evolving RpcConnectionState?
> I think, then, it has to be up to the RPC system to shut down
> connections that are idle. Probably RpcSystem could signal to the
> underlying VatNetwork whenever a connection has reached an idle
> state, meaning it has no outstanding RPCs nor capabilities. The
> VatNetwork could choose to close such a connection if it feels like
> it -- some transports may want to do this on a timeout, others may
> decide it's better to keep the connection open.
>
I have no strong opinion on this.
> But I'd suggest not worrying about that for now and focusing just on
> rpc-twoparty, since that's what most people are using today.
>
Indeed, a PR is forthcoming.
> -Kenton
>
Alex