Standardized schema interchange

[Ryan Patterson]

I'm in the process of writing an interactive debugger for capnp. It's easy enough to manually provide a schema when connecting to another peer, but I think it would be good long-term to support dynamic introspection as a feature of the RPC system.

I'm quite new to capnp in general, but after reading over the documentation, I think the best way to do this is for capnp to dictate a standardized optional interface which returns the original CodeGeneratorRequest that was used to implement the peer. Ideally, the code generator could automatically implement this interface for every generated interface, or perhaps the capnp library could automatically implement for on Bootstrap and Accept capabilities. When interacting with services which don't support introspection (possibly because of space, security, or other concerns), the method call simply fails because it refers to an invalid interface ID. The important thing is that such an interface has to have a fixed ID to be useful, which is why I think it should be established as a core part of capnp.

Here is my straw man version of this interface. I've selected AnyPointer so that implementing this interface doesn't require pulling the entire capnp schema into your own schema. Since the capnp schema is known to both parties (even if it's a different version), this is fine.

interface Introspectable {

  codeGeneratorRequest @0 () -> (cgr :AnyPointer);

}

[Ian Denhardt]

I was actually thinking about reflection like this this morning, though
my use case is a bit different: I'm building a data store on top of
capnproto, and would like to be able to introspect the stored objects
without a-priori knowing their schema. This probably means needing to
store the schema, so the client needs to provide the server with the cgr
when the data is stored. But it seems wasteful to have to attach the cgr
to each object, so some protocol is needed for the server to say to the
client, "hey, I don't know about this type yet?"

Quoting Ryan Patterson (2020-09-15 07:52:11)

> Here is my straw man version of this interface. I've selected
> AnyPointer so that implementing this interface doesn't require pulling
> the entire capnp schema into your own schema. Since the capnp schema is
> known to both parties (even if it's a different version), this is fine.
> interface Introspectable {
> codeGeneratorRequest @0 () -> (cgr :AnyPointer);
> }

A couple thoughts:

* I'm not 100% sure I understand the problem you're trying to solve by
using AnyPointer. Is it just avoiding code bloat due to having to pull
in the declarations for CodeGeneratorRequest? Maybe Kenton can shed
some more light, but my understanding is that (for the C++
implementation anyway) the generated code is mostly inline accessors,
so I'm not sure how important that is? But perhaps it would more
adversely affect other implementations.
* I can think of a few things you probably would want to do that this
interface can't handle:
* Asking what interfaces the object supports; the above is enough to
introspect messages that cross the wire since they include the
interface ID, but it seems natural to also want to enumerate the
available interfaces.
* Fetching dependencies of a CodeGeneratorRequest; In general, the cgr
will only include information for the schema for which code was
generated; if that schema imported other (separately compiled)
schema, there may be dangling references to types not defined in the
cgr.

Here's another stab:

```
interface SchemaRegistry {
cgrForType(typeId :UInt64) -> (cgr :CodeGeneratorRequest);
# Returns the cgr containing the given type Id.
}

interface Introspectable extends SchemaRegistry {
supportedInterfaces @1 () -> (interfaces :List(UInt64);
# Returns a list of the interface ids that this object supports.
# superclasses can be omitted as they are implied.
}
```

I believe that supports the use cases outlined so far. I split
SchemaRegistry out as a separate type so it's possible to query
information about types that aren't interfaces, as in my data store
use case.

Note that for the C++ implementation, there'd be no reason for
supportedInterfaces() to return a list, since you need to have a common
subtype at export time anyway, but this isn't true of all
implementations; at least the Go implementation lets you put method
sets together at runtime.

The above could use some improvement in that it might take several round
trips to get what you want, but my brain is moving slowly so I'm to go
get some caffeine instead of trying to make it perfect on this iteration.

-Ian

[Ryan Patterson]

You raise a great point about the CodeGeneratorRequest being "incomplete" and also not knowing the type ID of the held capability, which does complicate things. That said, I'm not convinced about your data store use case. It doesn't seem like something that needs to be integral to the capnp protocol, since there's already a shared schema between the peers. For example, it seems to me that your server interface would probably just take a Struct's type ID, and fail to set it if the type ID was unknown. Then provide a separate method to register a schema, which the client can use. A similar process would apply for StrudyRefs/Provided capabilities in levels 2/3.

```

enum ErrorCode {

  unknownSchema @0;

}

interface DataStore {

  setStruct @0 (obj :AnyPointer, type :UInt64) -> (err :ErrorCode);

  registerSchemas @1 (schemas :List(CodeGeneratorRequest)) -> ();

}

```

When you are coming at the problem with no shared schema (only the capnp rpc protocol), there isn't (currently) any way to get the schema at all, since you can't send a meaningful request to the peer to get it.

```

interface Introspectable {

  supportedInterfaces @0 () -> (interfaces :List(UInt64);

  schemaForTypes @1 (types :List(UInt64)) -> (schemas :List(CodeGeneratorRequest));

}

```

I've dropped the second interface since, as far as I can tell, the only place this needs to be implemented is again on the root object from Bootstrap/Accept. I've kept the List(UInt64), but switched to a multi-get for the schema fetcher, which has the nice advantage of allowing pipelining if you believe ahead of time you won't have the schema for the returned interfaces.

While it does feel a little weird to attach a global "schemaForTypes" method to any arbitrary capability, it ends up being necessary in the case of capability proxies; and the only alternative would be to implement it as part of the RPC protocol. That would be a root-level message to request schemas, which means that capability proxies now also need to be aware of the schema of the capability they hold.

[Ian Denhardt]

Quoting Ryan Patterson (2020-09-16 00:44:05)

> That said, I'm not convinced about your store use case. It doesn't

> seem like something that needs to be integral to the capnp protocol,
> since there's already a shared schema between the peers.

Having slept on it I think you're right; the data store can probably be
dealt with in a way that is orthogonal to Introspectable.

> I've dropped the second interface since, as far as I can tell, the
> only place this needs to be implemented is again on the root object
> from Bootstrap/Accept. I've kept the List(UInt64), but switched to
> a multi-get for the schema fetcher,


> which has the nice advantage of allowing pipelining if you believe
> ahead of time you won't have the schema for the returned interfaces.

I don't think this currently works; afaik there's no way to feed a
pending non-capability pointer directly as the argument to another call;
it only works on capabilities.

A possible alternative would be to have a method that just explicitly
requests the schema for all of the interfaces the receiver supports.

-Ian

[Kenton Varda]

This is something I've wanted to do for a while (built into the `capnp` tool).

I agree that RPC types should be introspectable. I'd design the interface like this:

    getSchema @0 () -> (typeId :UInt64, nodes :List(Schema.Node))

`nodes` would be a list of schema nodes describing the transitive closure of dependencies. (Each `Node` describes one type, like an interface type, a struct type, etc. See schema.capnp for details.)

CodeGeneratorRequest isn't quite the right thing to use here, because:

- It's specifically designed to provide information needed to generate source code, which is a superset of what you need at runtime. Schema.Node is intended to contain specifically the info that's useful at runtime.

- The CodeGeneratorRequest may include a lot of types that aren't actually used by the specific interface.

- We don't actually embed the full CodeGeneratorReequest inside the generated code. We embed each Node separately. I wouldn't want to store a second copy of all that data, and trying to rebuild the CGR from the components seems a little weird.

Next thing: I don't think the generated code would implement the getSchema() RPC directly for each type. Instead I think I'd add a new method to the `Capability::Server` and `ClientHook` like:

    virtual kj::Promise<capnp::InterfaceSchema> getSchema() = 0;

The generated code for each Server type would implement this virtual method, which would be a one-liner (`return capnp::Schema::from<ThisType>();`).

It would be the RPC system's responsibility to actually turn these into RPC calls, and to maintain a capnp::SchemaLoader used to translate 

I think the RPC system would have to be responsible for maintaining a capnp::SchemaLoader containing all schemas that had been loaded from the remote server, in order to implement the local version of `getSchema()`.

I'm a bit worried about redundantly transmitting the same schema nodes over the wire many times, especially when dealing with many different interfaces that implement the same underlying types. I wonder if we need a more complicated mechanism to deal with this. Maybe the RPC interface would let the client specify which schemas they already know about. Or maybe this would be tracked within the RPC system itself on a per-connection basis? There's also the problem that two different servers could have conflicting versions of a schema, but could both end up proxying through the same third server, meaning you might get different versions of the same schema for different capabilities on the same connection. Maybe it's not actually feasible to share schemas between capabilities at all? Ugh.

So... yeah, I really want to see this happen, but there's kind of a lot of design questions to work through. Big project.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/131b69c1-8782-4935-97d0-6fd11fa31e06n%40googlegroups.com.

[Ryan Patterson]

Hi Kenton, thanks for weighing in!

> I don't think the generated code would implement the getSchema() RPC directly for each type. Instead I think I'd add a new method to the `Capability::Server` and `ClientHook` like:

I think we are thinking the same thing here, but to confirm: The capability would respond to a standardized method, but the implementation of this standard method is provided by the library and not by the generated code. Are we in agreement?

> I'm a bit worried about redundantly transmitting the same schema nodes over the wire many times, especially when dealing with many different interfaces that implement the same underlying types.

Presumably, a peer would call it on the Bootstrap/Accepted capability, which would necessarily pull in every reachable Schema.Node, except for through AnyPointer boundaries. So, while the first call might be redundant if I did already have the schema, it wouldn't be "many times" in general. The worst case is a highly-dynamic interface where every method returns an untyped Capability. In such a case, 2 methods (my proposal in the last email) makes more sense than a combined method. And via promise pipelining, the 2 method solution shouldn't materially be slower than the combined method in the usual case.

> There's also the problem that two different servers could have conflicting versions of a schema, but could both end up proxying through the same third server, meaning you might get different versions of the same schema for different capabilities on the same connection. Maybe it's not actually feasible to share schemas between capabilities at all?

Let's split this into 2 problems: known (:MyInterface) and unknown (:Capability) capabilities. In the first case, a message broker is proxying MyInterface between peers which have different underlying versions of the same schema. This means that the message broker has a particular schema which it supports, and the introspecting peer should treat everything coming from the broker as though it were that schema.

In the second case, the message broker doesn't have a schema for those types, so the introspecting code automatically knows it needs to request the schema for the received capability. Because of the case where these unknown capabilities might be from different schemas (or different versions of one), the introspecting peer shouldn't share schemas between these capabilities. The net cost of this is  transferring a schema object whenever introspecting an untyped Capability (although that transfer may not actually be redundant--we won't know ahead of time).

In conclusion, it feels like attempting to cache/reuse schemas makes the project big and possibly intractable, but an introspection API which doesn't support that has none of the problems and a marginally increased wire cost.

[Kenton Varda]

On Wed, Sep 23, 2020 at 11:52 PM Ryan Patterson <cgame...@cgamesplay.com> wrote:

Hi Kenton, thanks for weighing in!

> I don't think the generated code would implement the getSchema() RPC directly for each type. Instead I think I'd add a new method to the `Capability::Server` and `ClientHook` like:

I think we are thinking the same thing here, but to confirm: The capability would respond to a standardized method, but the implementation of this standard method is provided by the library and not by the generated code. Are we in agreement?

Right. But I thought you were suggesting the implementation lives in generated code, whereas I'm suggesting it mostly lives in the RPC system.

 

> I'm a bit worried about redundantly transmitting the same schema nodes over the wire many times, especially when dealing with many different interfaces that implement the same underlying types.

Presumably, a peer would call it on the Bootstrap/Accepted capability, which would necessarily pull in every reachable Schema.Node, except for through AnyPointer boundaries. So, while the first call might be redundant if I did already have the schema, it wouldn't be "many times" in general. The worst case is a highly-dynamic interface where every method returns an untyped Capability. In such a case, 2 methods (my proposal in the last email) makes more sense than a combined method.

Technically any capability could implement arbitrary interfaces -- it could implement a derived class of what it is statically declared to implement, and that derived class could inherit any other interface as well.

I suppose you could have the calling code explicitly say when it wishes to query the capability for additional interfaces, rather than assuming the statically-derived type.

 

And via promise pipelining, the 2 method solution shouldn't materially be slower than the combined method in the usual case.

Not sure if promise pipelining helps here. You'd need to call supportedInterfaces() first and wait for it to complete in order to find out which type IDs are not in the set you already know about, and only then could you call schemaForTypes(). So it's still two network round trips.

To do it in one round trip the client would have to tell the server about types it already knows about. I suppose if the caller already has an expectation of the capability's type via static typing, it could send that, so that it only gets a non-empty response if the capability turns out to implement more than that.

 

> There's also the problem that two different servers could have conflicting versions of a schema, but could both end up proxying through the same third server, meaning you might get different versions of the same schema for different capabilities on the same connection. Maybe it's not actually feasible to share schemas between capabilities at all?

Let's split this into 2 problems: known (:MyInterface) and unknown (:Capability) capabilities. In the first case, a message broker is proxying MyInterface between peers which have different underlying versions of the same schema. This means that the message broker has a particular schema which it supports, and the introspecting peer should treat everything coming from the broker as though it were that schema.

Cap'n Proto supports passing through messages without knowing their full schema. This is very important for message brokers/proxies in particular -- you don't want to have to recompile the broker every time you add a new field to the message. Only the initial producer and final consumer need to know about a field.

 

In the second case, the message broker doesn't have a schema for those types, so the introspecting code automatically knows it needs to request the schema for the received capability. Because of the case where these unknown capabilities might be from different schemas (or different versions of one), the introspecting peer shouldn't share schemas between these capabilities. The net cost of this is  transferring a schema object whenever introspecting an untyped Capability (although that transfer may not actually be redundant--we won't know ahead of time).

But if you assume the types are disjoint, does that mean that if you discover the new capability implements interface Foo, you can't actually use it in a call to the original capability that expects a Foo?

Well, I suppose we could allow it to be used as long as the schemas have the same type ID, independent of whether the schemas are actually compatible.

So sure, in that case I buy that when people explicitly query a capability's dynamic type, we could build a whole new schema graph separate from the previously-known static type. In most cases it won't be necessary to query the dynamic type anyway since the static type will have everything you need.

-Kenton

 

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/9be29b6b-11db-4aa5-9124-b4fd12008f59n%40googlegroups.com.