guidance

[Newbugreport]

I'm writing to ask for some guidance with capnp. Many of the concepts used to explain capnp are explained in terms of other technologies which I'm not familiar with, or just assumed. So please bear with my questions and comments.

1. capnproto.org has some high level documentation, but header comments are the main documentation. Is there other documentation or samples I should read? Perhaps a suggested reading order?

2. The documentation describes capnp as a capability based system. This is a new concept to me, but Wikipedia says capabilities are object references that can't be forged. The implementation looks more like interface pointers. From my naive perspective this looks like a simple concept made difficult to understand by describing it in unfamiliar terms.

2a. I assume in capnp this reference is just the interfaceId in the rpc protocol, which would refer to an address or table entry on the server. Is capnp doing anything to make the ids difficult to guess/forge/eavesdrop?

2b. The ez-rpc header states it can only support public, singleton capabilities without state. Yet the Calculator sample returns a Value RPC object. This leaves me confused about what the limitations are. Am I failing to understand something about capabilities here?

2c. Is there a sample demonstrating capabilities, assuming the Calculator sample doesn't?

4. The install page suggests linux users download a tarball, however the tarball doesn't have samples from git. As a user struggling to learn the system, the samples would have helped.

5. Could you add a capnp parameter to output .cpp files? I respect that .c++ is a better name but there are a surprising number of places I have to change to handle .c++ files (build scripts, editor settings).

7. A link on the RPC page to Google Groups is not accessible without an account. Could you move or open that document?

8. The numbers in my email didn't survive so many edits. I have the same problem with .capnp files, and it's a surprising time sink. It occurs to me the capnp tool could modify capnp files to fix indexes, the only piece of missing information is which indexes already shipped and can't be changed.

Even though I'm a little frustrated right now, I do appreciate what you've done here. If nothing else, I hope my new user perspective helps highlight the new user pain points.

Regards,

NBR

[Ian Denhardt]

I'll weigh in on a few of these, though others will have to cover the
C++ specific details; I've mostly used the Go and Haskell
implementations.

Quoting 'Newbugreport' via Cap'n Proto (2019-10-22 06:01:02)

> 2. The documentation describes capnp as a capability based system. This
> is a new concept to me, but Wikipedia says capabilities are object
> references that can't be forged. The implementation looks more like
> interface pointers. From my naive perspective this looks like a simple
> concept made difficult to understand by describing it in unfamiliar
> terms.

More or less, though the thing that seems to be harder to convey to
people is how these can be used as robust, flexible security mechanism,
and the implications of that. I recommend reading this if you're
interested:

http://habitatchronicles.com/2017/05/what-are-capabilities/

..which maybe the page should actually link to? I think it's a better
reference than either of the things referred to now (I believe the above
post is more recent than the rpc docs page).

> 2a. I assume in capnp this reference is just the interfaceId in the

> [1]rpc protocol, which would refer to an address or table entry on the
> server.

At the protocol level, the reference would actually map to the
import/answer IDs, where as interfaceId is just used in identifying
which method to call on a given object. If you're interested in the gory
details, see the discussion in the section "The Four Tables", and the
definition and use of the MessageTarget struct.

> Is capnp doing anything to make the ids difficult to
> guess/forge/eavesdrop?

Because the ids are indices into a table that is per-connection, they
don't have to be secret to be unforgeable. Think of them like file
descriptors: If I have a file open under fd #5, then knowing that
doesn't help you gain access to the file yourself -- it's an index into
*my* process's fd table, and your fd #5 is different (if it exists). So
all that remains is to secure the connection itself, which you can do
with TLS.

> 7. A [3]link on the [4]RPC page to Google Groups is not accessible

> without an account. Could you move or open that document?

Not sure I follow? Isn't that just a link to the mailing list?

[Kenton Varda]

On Tue, Oct 22, 2019 at 3:01 AM 'Newbugreport' via Cap'n Proto <capn...@googlegroups.com> wrote:

1. capnproto.org has some high level documentation, but header comments are the main documentation. Is there other documentation or samples I should read? Perhaps a suggested reading order?

Sorry, not really at the moment. I have it on my todo list for this quarter to write real documentation for KJ...

 

2. The documentation describes capnp as a capability based system. This is a new concept to me, but Wikipedia says capabilities are object references that can't be forged. The implementation looks more like interface pointers. From my naive perspective this looks like a simple concept made difficult to understand by describing it in unfamiliar terms.

2a. I assume in capnp this reference is just the interfaceId in the rpc protocol, which would refer to an address or table entry on the server. Is capnp doing anything to make the ids difficult to guess/forge/eavesdrop?

Ian did a good job answering these.

Just to be extra clear: The interface ID is simply the type ID of the interface whose method is being called. This is public information derived from the schema file. An (interface id, method number) pair identifies a specific method schema. The RPC protocol does not rely on any unguessable strings; instead, each side keeps track of a table of capabilities that have been passed over the connection, much like the operating system tracks a file descriptor table for each process.

 

2b. The ez-rpc header states it can only support public, singleton capabilities without state. Yet the Calculator sample returns a Value RPC object. This leaves me confused about what the limitations are. Am I failing to understand something about capabilities here?

Hmm, that header comment is misleading. It's been a long time since I wrote it, but I think what I was trying to say is that it doesn't support "persistence", aka "level 2" of the RPC protocol. Experience since then has shown that it doesn't make sense to think of persistence as an RPC feature at all; instead it needs to be managed by the orchestration layer / service mesh. Cap'n Proto should be agnostic to those.

In any case, ez-rpc *does* support passing capabilities.

FWIW, I no longer recommend the EZ interfaces. They were probably a mistake to create and I ought to deprecate them. Instead, use TwoPartyClient and TwoPartyServer, after using kj::setupAsyncIo() to initialize the event loop.

2c. Is there a sample demonstrating capabilities, assuming the Calculator sample doesn't?

The calculator sample does in fact demonstrate capabilities, although it's somewhat contrived.

4. The install page suggests linux users download a tarball, however the tarball doesn't have samples from git. As a user struggling to learn the system, the samples would have helped.

Can you file an issue on GitHub suggesting they be included?

5. Could you add a capnp parameter to output .cpp files? I respect that .c++ is a better name but there are a surprising number of places I have to change to handle .c++ files (build scripts, editor settings).

Happy to accept a PR implementing this as an option.

(You could also perhaps wrap the tool in a little script that renames the files after output.)

7. A link on the RPC page to Google Groups is not accessible without an account. Could you move or open that document?

Oops, that link goes to a Google+ post I wrote about concurrency mechanisms. But Google+ has shut down, so the post is now gone. :(

 

8. The numbers in my email didn't survive so many edits. I have the same problem with .capnp files, and it's a surprising time sink. It occurs to me the capnp tool could modify capnp files to fix indexes, the only piece of missing information is which indexes already shipped and can't be changed.

This could be dangerous because renumbering a schema breaks binary compatibility (that's why the numbers exist in the first place -- to maintain compatibility across changes). I would recommend this technique:

- When adding a new field, always assign it the number 1 + the previous highest number. Don't worry if the numbers end up out-of-order, that is fine.

- Don't even delete fields. Rename them to add the prefix "obsolete". You can't delete a field because doing so breaks compatibility. (You can, however, reclaim the space in the message by retroactively unionizing the field with some new future field.)

-Kenton

[Newbugreport]

Thank you for your replies, Kenton and Ian.

Kenton, I'd like to contribute a new sample to help new users. I don't use GitHub but I can email you the work. The sample would be tutorial-oriented and not use ez-rpc. I'll look through the ez-rpc source to see how it uses the other classes, though any tips or insight would be appreciated.

I'm working on my own project, and I have capnp working and my five services are all talking to each other. Two of the services in particular do a lot of relaying, in one case three services provide the same interface and one query would get relayed from one to another to another. Is there a way to efficiently forward these queries and responses? I'm trying to see how to forward efficiently in terms of single messages (copying, processing time) and also in terms of many messages (not blocking one message while waiting for a response from another).

Thank you,

NBR

[Kenton Varda]

On Wed, Oct 30, 2019 at 1:29 PM Newbugreport <newbug...@protonmail.com> wrote:

Thank you for your replies, Kenton and Ian.

Kenton, I'd like to contribute a new sample to help new users. I don't use GitHub but I can email you the work. The sample would be tutorial-oriented and not use ez-rpc. I'll look through the ez-rpc source to see how it uses the other classes, though any tips or insight would be appreciated.

Cool! But it would be really helpful if you could create a GitHub account and submit as a pull request. That way the git history will show you as the author, we'll have a public record of the conversation around the submission, and we can use the GitHub pull request interface to do a code review.

I'm working on my own project, and I have capnp working and my five services are all talking to each other. Two of the services in particular do a lot of relaying, in one case three services provide the same interface and one query would get relayed from one to another to another. Is there a way to efficiently forward these queries and responses? I'm trying to see how to forward efficiently in terms of single messages (copying, processing time) and also in terms of many messages (not blocking one message while waiting for a response from another).

With Cap'n Proto, you can freely pass a capability across multiple RPC hops and it will automatically set up proxying that is as fast as possible. In the future, I intend to extend the protocol so that when you forward a capability, the eventual client will automatically form a direct connection to the server, to avoid proxying. The RPC protocol actually already contains a spec for this, it just hasn't been implemented yet.

-Kenton