Schemas on the wire Dynamic API

[Dane]

Hi 

I am trying to implement a dynamic API. I have followed the suggestions in https://groups.google.com/g/capnproto/c/Ivl3u4JJoEQ/m/MIH2a2nPBQAJ and
https://groups.google.com/g/capnproto/c/fNnr3Om9haM/m/1TdBqwsNBwAJ

My implementation attempt lies at https://github.com/DaneSlattery/capnp_generic_poc .

The desired data flow is as follows:

1. A client connects to a server, and receives a capability that allows the client to load a schema (the InterfaceLoader). When connecting, the client also passes a ClientInterface to the server.

2. The client registers a schema using the InterfaceLoader. The client must send a Schema.Node parameter to the server. The server must load the schema.

3. The server uses the given loaded schema to communicate with the client interface, which takes a parameter of AnyStruct.

I have a few problems:

Firstly, when the client attempts to set the schema using  capnp::Schema::from<Message>().getProto() , line 683 of 
capnproto-c++-0.10.3/src/capnp/dynamic.c++ blows up. I have changed this line to, instead assessing the ID of the schema for equality :
KJ_REQUIRE(structValue.getSchema().getProto().getId() == structType.getProto().getId(), "Value type mismatch.")
{
return;
}

I think this change should perhaps go in a PR, unless there is another, better way.

Secondly, I can't figure out how to send messages of the loaded schema back from the server to the client. Can anyone see if the POC linked is viable, and how to get to the final step?

- Dane

[Dane]

A follow up after simplifying the API to the core issue:

The data flow is now:

1. A client connects to a server, passing a Schema.Node as a parameter. 

2. A server loads the Schema.Node and creates a new message of the given schema, which is returned as an AnyStruct

Problem 1 above still exists.

We have figured out how to handle point 2, but not without a malloc. How can I init the result as a DynamicStruct, instead of copying as in the below snippet:
auto message = context.getParams();
auto schema_ = message.getSchema();
auto schema_loader = capnp::SchemaLoader();
auto loaded_schema = schema_loader.load(schema_);

// how to do below without malloc?

// cannot init result as a DynamicStruct
auto mb = capnp::MallocMessageBuilder();
auto msg = mb.initRoot<capnp::DynamicStruct>(loaded_schema.asStruct());
msg.set("text", "abc");

auto results = context.initResults();
results.setValue(msg.asReader());

Would this work as above if we had nested structs? 

Aside: I am also trying to replicate this in pycapnp, which is proving deeply problematic since it skips out on a lot of ported functionality from capnp.

- Dane