Implementing RPC over SSL

[Remy Blank]

I'm trying to implement Cap'n Proto RPC over an SSL channel. I gave up
on OpenSSL because its interface is just exceedingly complicated, but it
looks like GnuTLS should work nicely.

I know I have to implement an AsyncIoStream subclass and manage the
connection setup myself, but I'm hitting a few roadblocks.

- There's a fair amount of code currently hidden in async-io.c++ that I
could re-use for SSL. In particular, the following could be re-used as-is:

OwnedFileDescriptor
SocketAddress
FdConnectionReceiver
NetworkAddressImpl
SocketNetwork

If I understood the non-blocking behavior of GnuTLS correctly, I could
probably even re-use AsyncIoProviderImpl, and just implement a different
LowLevelAsyncIoProvider for it, as well as a custom AsyncIoStream.

All these symbols are currently hidden, so I would have to copy-paste
close to 1000 lines of code, which is far from ideal. Any chance of
exposing these symbols?

- LowLevelAsyncIoProviderImpl currently embeds a UnixEventPort, an
EventLoop and a WaitScope. Since there can only be one EventLoop per
thread, this means that a thread can only manage connections from a
single LowLevelAsyncIoProvider. Or in other words, connections managed
by a thread must either all be SSL, or all be "raw". Shouldn't the
EventPort and EventLoop rather be provided to the
LowLevelAsyncIoProvider instead? This would lift the restriction. The
WaitScope isn't even used internally, so it could also simply be
instantiated wherever it's needed, instead of being retrieved from the
provider.

- newOneWayPipe is problematic for an SSL IO provider, because an SSL
connection is inherently bi-directional (due to the crypto handshake).
But since it isn't used anywhere in the RPC protocol, I could just throw
a "not supported" exception there.

- A newDualPipe method on the provider that makes the connection run
through a pair of pipes (instead of a single socketpair) would allow
running RPC through stdin/stdout, and would make my life easier for
testing the SSL code, since I could pipe the client through "openssl
s_client" and test the server SSL code only. It could also allow running
a server behind xinetd, so together with "openssl s_server", you would
get a poor-man's SSL channel.

- EzRpc* classes would be re-usable for custom IO providers if it was
possible to provide an AsyncIoContext factory to EzRpcContext, instead
of hard-coding the call to setupAsyncIo. Considering how EzRpcContext is
instantiated, the factory would probably have to be a global variable,
though, so it wouldn't be very elegant.

Thoughts? I'd be happy to provide patches for any of the items above, if
desired.

-- Remy

[Kenton Varda]

Hi Remy,

Sorry, I've done a bad job documenting my high-level design here.

The purpose of the EzRpc* classes is to provide a very simple API covering the "common case", so that people who don't want to do anything "custom" don't need to muck around in the details of hooking up all the different objects.  These classes are not meant to be flexible as flexibility would bring with it complexity that would defeat the purpose.  If these classes ever support encryption, it ought to be via a boolean "enable encryption" option which sets everything up in one specific recommended way, but at this point we don't have that.  If you want to do something custom, then you're actually supposed to skip the EZ RPC interfaces entirely and deal directly with RpcSystem / TwoPartyVatNetwork.

With that sorted out, the rest of your questions turn out to be mostly moot.  TwoPartyVatNetwork just wants an AsyncIoStream.  There's actually no need to implement AsyncIoProvider nor LowLevelAsyncIoProvider.  You just want to implement an AsyncIoStream which supports encryption.  Your stream implementation might wrap another, raw AsyncIoStream (i.e. from AsyncIoProvider) and add encryption on top of that, or it might skip AsyncIoProvider altogether and just wrap UnixEventPort directly, or maybe you'll event want to use some other event loop library and write a custom EventPort.  These options represent different trade-offs between ease of integration with other KJ code vs. ease of integration with some other event loop framework.

FWIW, you can construct an AsyncIoProviderImpl on top of an arbitrary LowLevelAsyncIoProvider using capnp::newAsyncIoProvider(), but you only really need to do that if you are implementing a custom event loop framework and you need to call some code that wants an AsyncIoProvider implementation -- at present, there is no such code to call.

-Kenton

[Remy Blank]

Kenton Varda wrote:
> The purpose of the EzRpc* classes is to provide a very simple API
> covering the "common case", so that people who don't want to do anything
> "custom" don't need to muck around in the details of hooking up all the
> different objects. These classes are not meant to be flexible as
> flexibility would bring with it complexity that would defeat the
> purpose. If these classes ever support encryption, it ought to be via a
> boolean "enable encryption" option which sets everything up in one
> specific recommended way, but at this point we don't have that. If you
> want to do something custom, then you're actually supposed to skip the
> EZ RPC interfaces entirely and deal directly with RpcSystem /
> TwoPartyVatNetwork.

I'm not actually using the EzRpc* classes, but I did copy out the bits
that I needed to set up the network. This includes the parts with
RpcSystem and TwoPartyVatNetwork.

> With that sorted out, the rest of your questions turn out to be mostly
> moot. TwoPartyVatNetwork just wants an AsyncIoStream. There's actually
> no need to implement AsyncIoProvider nor LowLevelAsyncIoProvider. You
> just want to implement an AsyncIoStream which supports encryption. Your
> stream implementation might wrap another, raw AsyncIoStream (i.e. from
> AsyncIoProvider) and add encryption on top of that, or it might skip
> AsyncIoProvider altogether and just wrap UnixEventPort directly, or
> maybe you'll event want to use some other event loop library and write a
> custom EventPort. These options represent different trade-offs between
> ease of integration with other KJ code vs. ease of integration with some
> other event loop framework.

TBH, what I currently want is to write as little code as possible to get
the SSL part into the connection. I certainly don't want to implement
another event loop or anything like that.

Digging some more, it indeed looks like the providers aren't actually
useful to me. But from what I understand so far, I would still have to
re-implement (meaning copy) most of what's in SocketNerwork,
SocketAddress, NetworkAddressImpl and FdConnectionReceiver, in addition
to implementing my own AsyncIoStream.

In more detail, looking at EzRpcServer:

- EzRpcServer::Impl calls SocketNetwork.parseAddress(), which calls
SocketAddress::parse() to get an array of SocketAddress instances, which
end up in a NetworkAddressImpl returned from the promise.

- EzRpcServer calls NetworkAddress::listen(), which returns a
FdConnectionReceiver.

- The accept loop calls ConnectionReceiver::accept() and gets an
AsyncStreamFd back. I wrap that one, because GnuTLS does its own
receiving and sending over the socket descriptor, so I need access to
the latter.

And there's a similar situation on the client side, although I haven't
looked at it closely yet.

Sure, I could re-implement all the address parsing, listening and
accepting, but it's all done already in async-io.c++, and I could re-use
it as-is. The only trouble is that it's all hard-wired in its private
parts, and I have no way of plugging in anything in-between that would
allow me to have ConnectionReceiver::accept() return the file descriptor
instead of an AsyncStreamFd, so that I could wrap it in my own
AsyncIoStream.

Or is there? NetworkAddressImpl uses
LowLevelIoProvider::wrapListenSocketFd() to get the ConnectionReceiver,
so if I implement my own LowLevelIoProvider and pass it to
newAsyncIoProvider(), I only have to copy the functionality that's in
FdConnectionReceiver (and therefore OwnedFileDescriptor, setNonBlocking
and setCloseOnExec). Is this how you intended this to be done?

-- Remy

[Kenton Varda]

Hi Remy,

From what I can tell, GNUTLS is well-designed in that it lets you implement an arbitrary asynchronous transport layer underneath.  You don't need to provide it with a file descriptor and you can use any event mechanism you want.  See:

http://www.gnutls.org/manual/gnutls.html#Setting-up-the-transport-layer

In that case, what you want to do is implement an AsyncIoStream which wraps some other AsyncIoStream.  The inner AsyncIoStream represents the underlying socket and can be constructed using the standard AsyncIoProvider using the existing public interface.  Then there is no need to copy any code at all.

I'm definitely open to making async-io.c++ more reusable if necessary, but I want to make sure we are on the same page about what parts actually need to be reused.

-Kenton

[Remy Blank]

Remy Blank wrote:
> Or is there? NetworkAddressImpl uses
> LowLevelIoProvider::wrapListenSocketFd() to get the ConnectionReceiver,
> so if I implement my own LowLevelIoProvider and pass it to
> newAsyncIoProvider(), I only have to copy the functionality that's in
> FdConnectionReceiver (and therefore OwnedFileDescriptor, setNonBlocking
> and setCloseOnExec). Is this how you intended this to be done?

OwnedFileDescriptor would be useful to be exposed, so that my
AsyncIoStream and ConnectionReceiver could inherit from it.

I could avoid copying FdConnectionReceiver if it called
LowLevelAsyncIoProvider::wrapSocketFd() instead of instantiating
AsyncStreamFd directly. I would also need to be able to instantiate it
in my LowLevelAsyncIoProvider::wrapListenSocketFd(). But that doesn't
fit too well, as this would expose the fact that it takes a
UnixEventPort, so it's probably better to re-implement it.

FdConnectionReceiver.getPort() uses SocketAddress, which is private. So
if I implement my own, I need to copy SocketAddress::getLocalAddress()
and SocketAddress::getPort(). Not much, but still.

So how does the following sound:

- Make OwnedFileDescriptor public. This cuts about 60 lines that I have
to copy-paste.

- (Optional) Make SocketAddress public. In addition to being useful for
ConnectionReceiver.getPort(), it neatly encapsulates the socket
operations and DNS lookup, so I can see it being useful elsewhere. This
cuts another 15 lines.

Then I can implement AsyncIoStream, ConnectionReceiver and
LowLevelAsyncIoProvider specifically for SSL connections.

-- Remy

[Remy Blank]

Kenton Varda wrote:
> From what I can tell, GNUTLS is well-designed in that it lets you
> implement an arbitrary asynchronous transport layer underneath. You
> don't need to provide it with a file descriptor and you can use any
> event mechanism you want. See:
>
> http://www.gnutls.org/manual/gnutls.html#Setting-up-the-transport-layer

That's true, but it looks like it's more complicated than letting GnuTLS
manage the sockets itself. If the descriptors are set non-blocking
(which is the case here), GnuTLS operations become non-blocking as well,
and the AsyncIoStream can be implemented almost the same as
AsyncIoStreamFd except using GnuTLS operations instead of syscalls for
reading and writing. See:

http://www.gnutls.org/manual/gnutls.html#Asynchronous-operation

> In that case, what you want to do is implement an AsyncIoStream which
> wraps some other AsyncIoStream. The inner AsyncIoStream represents the
> underlying socket and can be constructed using the standard
> AsyncIoProvider using the existing public interface. Then there is no
> need to copy any code at all.

This would mean I have to implement an AsyncIoStream and additionally
the various callbacks for GnuTLS to push and pull data. Also, I'm not
sure how I would implement the pull_timeout function, since the event
loop doesn't support timers. But maybe this function isn't needed in
non-blocking mode.

> I'm definitely open to making async-io.c++ more reusable if necessary,
> but I want to make sure we are on the same page about what parts
> actually need to be reused.

OwnedFileDescriptor and maybe (selected parts of) SocketAddress would be
most useful.

BTW, if it looks like my ideas are jumping all over the place, it's
because they are :) I'm still learning the basics of Cap'n Proto and how
to use it properly.

-- Remy

[Kenton Varda]

On Thu, Feb 20, 2014 at 3:55 PM, Remy Blank <remy....@pobox.com> wrote:

That's true, but it looks like it's more complicated than letting GnuTLS

manage the sockets itself.

I don't know, it doesn't look very complicated to me.  Certainly it looks a lot less complicated than refactoring async-io.c++ and maintaining more public interfaces long-term.

More importantly, though, it's the Right Solution, because it is composable:  someone could trivially throw TLS on top of an arbitrary stream that isn't a network socket.  This could actually be useful e.g. for Sandstorm where the platform API might offer the ability to open a socket to the outside world, but returns a Cap'n Proto capability representing the socket, rather than an OS-level socket.  You could easily implement an AsyncIoStream wrapping an RPC interface, then throw your TLS wrapper on it.  If the TLS library only supports sockets, though, then you'd have to create a socket pair and redundantly push all the data through the kernel.

Meanwhile, I'm hesitant to make parts of async-io.c++ public without careful thought about the interfaces.  When I write a private interface I tend not to think very hard about it and take ugly shortcuts -- e.g. OwnedFileDescriptor's fd field being public, and the use of implementation inheritance.  It will take some thought to decide the best way to expose these interfaces to the world.  Also, Andy wants me to make some other aspects of this file more reusable, for the purpose of making it easier to integrate with external event loops, so if we're going to overhaul this file then we should probably take those requirements into account as well.

I'm willing to do this work if it's really necessary, but it seems like implementing a custom GNUTLS transport is a better design and probably easier anyway.

 

Also, I'm not
sure how I would implement the pull_timeout function, since the event
loop doesn't support timers. But maybe this function isn't needed in
non-blocking mode.

Since the documentation describes this function as blocking, I'd assume it either isn't used in nonblocking mode or should simply fail with EWOULDBLOCK if there is no data -- never wait for the timeout.

We do need to get timeouts implemented, though...

 

BTW, if it looks like my ideas are jumping all over the place, it's

because they are :) I'm still learning the basics of Cap'n Proto and how
to use it properly.

I'm curious to know what you're using it for.  I see your G+ profile says you're at Google, but I would think Google projects would stick to protobufs.  :)

-Kenton