RPC Level 2 questions - SturdyRefs, restorers, authentication and encryption

[Thomas Leonard]

Hi,

I'm currently trying to implement RPC level 2 (for the OCaml RPC implementation - see https://github.com/mirage/capnp-rpc#encryption-and-authentication for the current status).

I have some questions...

https://capnproto.org/cxxrpc.html says:

Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is a Level 1 implementation. Persistent capabilities, three-way introductions, and distributed equality are not yet implemented.

But I imagine this is out of date.

The RPC spec says:

How exactly a SturdyRef is restored to a live object is specified along with the SturdyRef definition (i.e. not by rpc.capnp).

and

However, in practice, the ability to restore SturdyRefs is itself a capability that may require going through an authentication process to obtain. Thus, it makes more sense to define a "restorer service" as a full Cap'n Proto interface. If this restorer interface is offered as the vat's bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if every SturdyRef has its own restorer API then an RPC implementation won't know how to authenticate to it when the user does something like:

liveRef := sturdyRef.getRcvr()

Is this restorer API specified somewhere? The Python docs mention an ez_restore method and say "Refer to the Cap’n Proto docs if you don’t know what this means", but it's not clear to me what I should be looking at.

For now, my SturdyRefs can only refer to bootstrap services.

Also, I couldn't find much about authentication. https://capnproto.org/rpc.html#encryption says:

At this time, Cap’n Proto does not specify an encryption scheme, but as it is a simple byte stream protocol, it can easily be layered on top of SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just as a container for the public key) and a validator that checks a hash of the server key fingerprint. My SturdyRefs convert to URIs that currently look like this:

  capnp://sha-256:s16WV4JeGusAL_nTjvICi...@127.0.0.1:7000

(based on the format at http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind)

However, https://capnproto.org/roadmap.html says:

Cap’n Proto RPC should support an encrypted transport which uses capability-based authorization (not PKI), can accomplish zero-round-trip three-party introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is not designed for this, but we don’t want to invent new crypto; we intend to build on libsodium and the Noise Protocol Framework as much as possible.

For interoperability, I allowed "insecure@" to disable encryption. e.g. to connect my OCaml client to the C++ calculator service I can use:

  capnp://inse...@127.0.0.1:7000

I treat an empty host:port section as requesting a Unix-domain socket, e.g.

  capnp://insecure@/tmp/calc

It's a bit of a hack, but I'm not sure what would be better.

Things I'd like to know:

- Do any other implementations support authentication or encryption?

- Has anyone else defined a URI format I can use for sturdy refs?

- Where in the URI should I put the service ID for the restorer server?

- Is there a spec defining what a StudyRef and restorer service should look like for the default public Internet?
  If not, is there anything obvious I should change in my current implementation to match what this will look like?

Thanks,

[Thomas Leonard]

On Wednesday, August 23, 2017 at 3:48:45 PM UTC+1, Thomas Leonard wrote:

Hi,

I'm currently trying to implement RPC level 2 (for the OCaml RPC implementation - see https://github.com/mirage/capnp-rpc#encryption-and-authentication for the current status).

I have some questions...

https://capnproto.org/cxxrpc.html says:

Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is a Level 1 implementation. Persistent capabilities, three-way introductions, and distributed equality are not yet implemented.

But I imagine this is out of date.

The RPC spec says:

How exactly a SturdyRef is restored to a live object is specified along with the SturdyRef definition (i.e. not by rpc.capnp).

and

However, in practice, the ability to restore SturdyRefs is itself a capability that may require going through an authentication process to obtain. Thus, it makes more sense to define a "restorer service" as a full Cap'n Proto interface. If this restorer interface is offered as the vat's bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if every SturdyRef has its own restorer API then an RPC implementation won't know how to authenticate to it when the user does something like:

liveRef := sturdyRef.getRcvr()

Is this restorer API specified somewhere? The Python docs mention an ez_restore method and say "Refer to the Cap’n Proto docs if you don’t know what this means", but it's not clear to me what I should be looking at.

For now, my SturdyRefs can only refer to bootstrap services.

It seems that this uses the "deprecatedObjectId". The spec says it's deprecated because:

1. "in practice, the ability to restore SturdyRefs is itself a capability that may require going through an authentication process to obtain."

2. "the earliest version of the EZ RPC interfaces set a precedent of exporting multiple main interfaces by allowing them to be exported under string names"

3. "Often, "main" or "well-known" capabilities exported by a vat are in fact not public: they are intended to be accessed only by clients who are capable of forming a connection to the vat." and "the supervisor has to be careful not to honor an external request addressing the application's default capability".

However:

1. If a sturdy ref is a capability reference then holding it should be sufficient to use it, without needing additional capabilities. It's not clear why the spec thinks this isn't enough.

2. It's not clear why one implementation using string names means the protocol needs to change.

3. This seems like a sandstorm-specific problem, and could be fixed in one place (the supervisor). As a safer alternative, sandstorm apps could instead use a pre-exported "export 0" for their admin API. The protocol should already prevent external clients from getting access to unrelated exports.

Given the lack of an alternative, I've implemented this "deprecated" scheme in the OCaml version now. It seems to work fine, and works with the other implementations.

 

Also, I couldn't find much about authentication. https://capnproto.org/rpc.html#encryption says:

At this time, Cap’n Proto does not specify an encryption scheme, but as it is a simple byte stream protocol, it can easily be layered on top of SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just as a container for the public key) and a validator that checks a hash of the server key fingerprint. My SturdyRefs convert to URIs that currently look like this:

  capnp://sha-256:s16WV4JeGusAL_nTjvICiQOFqm3LqYrDj3K-HXdMi8s@127.0.0.1:7000

(based on the format at http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind)

However, https://capnproto.org/roadmap.html says:

Cap’n Proto RPC should support an encrypted transport which uses capability-based authorization (not PKI), can accomplish zero-round-trip three-party introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is not designed for this, but we don’t want to invent new crypto; we intend to build on libsodium and the Noise Protocol Framework as much as possible.

For interoperability, I allowed "insecure@" to disable encryption. e.g. to connect my OCaml client to the C++ calculator service I can use:

  capnp://inse...@127.0.0.1:7000

I treat an empty host:port section as requesting a Unix-domain socket, e.g.

  capnp://insecure@/tmp/calc

It's a bit of a hack, but I'm not sure what would be better.

Things I'd like to know:

- Do any other implementations support authentication or encryption?

- Has anyone else defined a URI format I can use for sturdy refs?

- Where in the URI should I put the service ID for the restorer server?

I'm now storing the service ID, base64-encoded, as the final path component in the URI.

 

- Is there a spec defining what a StudyRef and restorer service should look like for the default public Internet?
  If not, is there anything obvious I should change in my current implementation to match what this will look like?

I also have some additional questions now:

- The schema language doesn't define a SturdyRef type. Since it seems that interfaces shouldn't contain network-specific types, this means that SturdyRefs currently have to be defined as AnyPointer. It would be useful if the spec could pre-define this abstract (and generic) type. Then schema compilers could output type-safe code for building and reading them.

- It seems that sturdy refs are intended to be stored as plain data. However, this is usually readable by the application code. Perhaps they should instead go in the CapDescriptor table?

The spec says:

"we specify several "parameter" types.  Each type is defined here as an alias for `AnyPointer`, but a specific network will want to define a specific set of types to use.  All vats in a vat network must agree on these parameters in order to be able to communicate.  Inter-network communication can be accomplished through "gateways" that perform translation between the primitives used on each network; these gateways may need to be deeply stateful, depending on the translations they perform."

Does "deeply stateful" mean that gateways need to understand the schema to know which fields are sturdy refs?

[Kenton Varda]

Hi Thomas,

Sorry I missed this earlier!

So, "level 2" of the RPC protocol / SturdyRefs turn out to be something that does not make sense to specify as part of the Cap'n Proto implementation itself. Probably level 2 should be ripped out of the spec entirely. This is what we learned as we build Sandstorm: SturdyRefs and how to restore them turned out to be intrinsically tied to the Sandstorm environment, and attempts to define an "abstract" SturdyRef format not dependent on Sandstorm did not seem to fit what Sandstorm needed.

To understand this, consider a few different cases:

- A Sandstorm grain (app instance).

- A node in the Blackrock (clustered Sandstorm) infrastructure.

- An application on the general internet that has nothing to do with Sandstorm.

Now try to answer the question: What does a SturdyRef express, and how does one restore it?

The answer is totally different depending on the context:

- For a Sandstorm grain, a SturdyRef can be an opaque byte string, which refers to an object in another grain. The client grain passes the SturdyRef to the Sandstorm API to restore it. The Sandstorm infrastructure then looks up the token in its database, verifies that the token belongs to the requesting grain, finds out to what grain the token points, starts up that grain, asks that grain for a live ref of the desired capability, and then returns that to the requesting grain.

- For a Blackrock node, a SturdyRef typically refers to another component of the Blackrock infrastructure: maybe an object in Blackrock storage (a graph store of Cap'n Proto objects), or a running container on one of the worker nodes. Or, it could also refer to something hosted in a grain, or a totally external capability. See the definition here: https://github.com/sandstorm-io/blackrock/blob/master/src/blackrock/cluster-rpc.capnp#L67 Notice how the namespace of SturdyRefs as seen by the infrastructure itself is completely different from the namespace of SturdyRefs seen by apps -- although some kinds of objects can be represented by both. Also notice that depending on the type of SturdyRef, the process for restoring is different: for "transient" objects located on a specific machine, the restorer connects directly to the target, but for stored object, the restorer connects to "the storage service" which it is introduced to independently at startup, and for external caps, it connects to "the gateways", etc.

- For the public internet, you probably want a SturdyRef to encode a hostname and perhaps a pinned certificate list. Maybe it even encodes an HTTP URL, to which a Cap'n Proto session can be created over WebSocket or streaming HTTP/2. Additionally, it would encode some sort of object ID, probably as an AnyPointer. The target host would provide a bootstrap interface with a restore() method that takes this AnyPointer.

As you can see, in each case the format of a SturdyRef and the procedure for restoring it is completely different, so much so that it doesn't appear that any "standard" definition makes sense.

At some point I do want to spec out the "public internet" SturdyRef format and protocol. But, for now, I think implementations should leave SturdyRefs up to the application to define.

On a side note, it seems like you were confused a bit by EZ RPC's mechanism for exporting capabilities by name. We deprecated this in favor of a singleton bootstrap interface because you can trivially implement the same thing by defining a bootstrap interface with a "restore(name)" method.

-Kenton

On Wed, Aug 23, 2017 at 7:48 AM, Thomas Leonard <tal...@gmail.com> wrote:

Hi,

I'm currently trying to implement RPC level 2 (for the OCaml RPC implementation - see https://github.com/mirage/capnp-rpc#encryption-and-authentication for the current status).

I have some questions...

https://capnproto.org/cxxrpc.html says:

Current Status: As of version 0.4, Cap’n Proto’s C++ RPC implementation is a Level 1 implementation. Persistent capabilities, three-way introductions, and distributed equality are not yet implemented.

But I imagine this is out of date.

The RPC spec says:

How exactly a SturdyRef is restored to a live object is specified along with the SturdyRef definition (i.e. not by rpc.capnp).

and

However, in practice, the ability to restore SturdyRefs is itself a capability that may require going through an authentication process to obtain. Thus, it makes more sense to define a "restorer service" as a full Cap'n Proto interface. If this restorer interface is offered as the vat's bootstrap interface, then this is equivalent to the old arrangement.

I imagine this must be some network-realm-wide restorer API, because if every SturdyRef has its own restorer API then an RPC implementation won't know how to authenticate to it when the user does something like:

liveRef := sturdyRef.getRcvr()

Is this restorer API specified somewhere? The Python docs mention an ez_restore method and say "Refer to the Cap’n Proto docs if you don’t know what this means", but it's not clear to me what I should be looking at.

For now, my SturdyRefs can only refer to bootstrap services.

Also, I couldn't find much about authentication. https://capnproto.org/rpc.html#encryption says:

At this time, Cap’n Proto does not specify an encryption scheme, but as it is a simple byte stream protocol, it can easily be layered on top of SSL/TLS or other such protocols.

For now, I have used TLS with self-signed non-expiring certificates (just as a container for the public key) and a validator that checks a hash of the server key fingerprint. My SturdyRefs convert to URIs that currently look like this:

  capnp://sha-256:s16WV4JeGusAL_nTjvICiQOFqm3LqYrDj3K-HXdMi8s@127.0.0.1:7000

(based on the format at http://iiw.idcommons.net/HTTPSY_%E2%80%93_Leave_the_Certificate_Authority_Behind)

However, https://capnproto.org/roadmap.html says:

Cap’n Proto RPC should support an encrypted transport which uses capability-based authorization (not PKI), can accomplish zero-round-trip three-party introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is not designed for this, but we don’t want to invent new crypto; we intend to build on libsodium and the Noise Protocol Framework as much as possible.

For interoperability, I allowed "insecure@" to disable encryption. e.g. to connect my OCaml client to the C++ calculator service I can use:

  capnp://inse...@127.0.0.1:7000

I treat an empty host:port section as requesting a Unix-domain socket, e.g.

  capnp://insecure@/tmp/calc

It's a bit of a hack, but I'm not sure what would be better.

Things I'd like to know:

- Do any other implementations support authentication or encryption?

- Has anyone else defined a URI format I can use for sturdy refs?

- Where in the URI should I put the service ID for the restorer server?

- Is there a spec defining what a StudyRef and restorer service should look like for the default public Internet?
  If not, is there anything obvious I should change in my current implementation to match what this will look like?

Thanks,

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[Thomas Leonard]

On 6 September 2017 at 18:16, Kenton Varda <ken...@cloudflare.com> wrote:
> Hi Thomas,
>
> Sorry I missed this earlier!
>
> So, "level 2" of the RPC protocol / SturdyRefs turn out to be something that
> does not make sense to specify as part of the Cap'n Proto implementation
> itself. Probably level 2 should be ripped out of the spec entirely. This is
> what we learned as we build Sandstorm: SturdyRefs and how to restore them
> turned out to be intrinsically tied to the Sandstorm environment, and
> attempts to define an "abstract" SturdyRef format not dependent on Sandstorm
> did not seem to fit what Sandstorm needed.
>
> To understand this, consider a few different cases:
> - A Sandstorm grain (app instance).
> - A node in the Blackrock (clustered Sandstorm) infrastructure.
> - An application on the general internet that has nothing to do with
> Sandstorm.
>
> Now try to answer the question: What does a SturdyRef express, and how does
> one restore it?

At the application level:

It represents the ability to get a live capability reference to a
service. You restore it by calling its "connect" method (which is
possibly the only method it has).

At the vat/network level: whatever the network defines it as.

(using "SturdyRef" to mean both of these things is perhaps confusing though)

> The answer is totally different depending on the context:
>
> - For a Sandstorm grain, a SturdyRef can be an opaque byte string, which
> refers to an object in another grain. The client grain passes the SturdyRef
> to the Sandstorm API to restore it. The Sandstorm infrastructure then looks
> up the token in its database, verifies that the token belongs to the
> requesting grain, finds out to what grain the token points, starts up that
> grain, asks that grain for a live ref of the desired capability, and then
> returns that to the requesting grain.
>
> - For a Blackrock node, a SturdyRef typically refers to another component of
> the Blackrock infrastructure: maybe an object in Blackrock storage (a graph
> store of Cap'n Proto objects), or a running container on one of the worker
> nodes. Or, it could also refer to something hosted in a grain, or a totally
> external capability. See the definition here:
> https://github.com/sandstorm-io/blackrock/blob/master/src/blackrock/cluster-rpc.capnp#L67

That's a useful link, thanks!

> If the sender's public key
> is less than the receiver's, this number must be even, otherwise it must be odd, so that
> connection IDs in opposite directions between the same vats never collide. Any existing
> connections with lower connection IDs must be invalidated when a new connection starts.

That's useful to have specified. For my implementation, the rule I
used was "the connection to keep is the one initiated by the peer with
the highest vat ID." But your scheme looks better, as it copes with
one vat losing a connection and retrying while the other thinks the
old connection is still OK. Can I use this header format with e.g. the
Python client? My current code does have the advantage of being just
plain TLS with client and server certificates.

> Notice how the namespace of SturdyRefs as seen by the infrastructure itself
> is completely different from the namespace of SturdyRefs seen by apps --
> although some kinds of objects can be represented by both. Also notice that
> depending on the type of SturdyRef, the process for restoring is different:
> for "transient" objects located on a specific machine, the restorer connects
> directly to the target, but for stored object, the restorer connects to "the
> storage service" which it is introduced to independently at startup, and for
> external caps, it connects to "the gateways", etc.
>
> - For the public internet, you probably want a SturdyRef to encode a
> hostname and perhaps a pinned certificate list. Maybe it even encodes an
> HTTP URL, to which a Cap'n Proto session can be created over WebSocket or
> streaming HTTP/2. Additionally, it would encode some sort of object ID,
> probably as an AnyPointer. The target host would provide a bootstrap
> interface with a restore() method that takes this AnyPointer.

Yes, this is roughly what I have (though I encoded it as a simple
string URL, since it supports that form anyway).
It would be very useful to specify this so that different
implementations can talk to each other.

For example, I'd like to be able to cut-and-paste a URL from an OCaml
service and connect to it using the Python client.

> As you can see, in each case the format of a SturdyRef and the procedure for
> restoring it is completely different, so much so that it doesn't appear that
> any "standard" definition makes sense.

At the network level, yes. But at the application level it still makes
sense to have an abstract SturdyRef type in the schema language and in
the API I think. And it should be possible for applications to
exchange sturdy refs in messages without knowing what kind of network
their vats use.

> At some point I do want to spec out the "public internet" SturdyRef format
> and protocol. But, for now, I think implementations should leave SturdyRefs
> up to the application to define.

I don't see how this can work. For the public internet, I need to know
what to connect to, what protocol to speak (TCP, TLS, HTTP, etc) and
how to authenticate the peer. The object ID can be opaque, but the
rest must be known.

> On a side note, it seems like you were confused a bit by EZ RPC's mechanism
> for exporting capabilities by name. We deprecated this in favor of a
> singleton bootstrap interface because you can trivially implement the same
> thing by defining a bootstrap interface with a "restore(name)" method.

I could. But then I'm just replacing a somewhat standard interface
with a non-standard one that the other clients don't have built-in
support for. It doesn't seem like an improvement.

>> capnp://sha-256:s16WV4JeGusAL_nTjvICi...@127.0.0.1:7000

>> email to capnproto+...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/capnproto.
>
>

--
talex5 (GitHub/Twitter) http://roscidus.com/blog/
GPG: 5DD5 8D70 899C 454A 966D 6A51 7513 3C8F 94F6 E0CC