Middleware & Attaching metadata to calls/returns?

[Erin Shepherd]

Many protocols provide some way to attach metadata to requests & responses. HTTP and protocols derved from it provide this in the form of headers (and occasionally trailers), for example.

There are various reasons why this can be very useful, even in a capability oriented system:

• Propagating a request correlation ID from node to node as a call passes through multiple parts of a system for logging correlation purposes
  
• Even better, propagating an OpenTracing SpanContext or similar to enable correlating multiple parts of an action in a distributed system with tools like Jaeger
  
• Attaching Lamport timestamps to every request and reply, to propagate an inter-system causal ordering
  

These sorts of things are very useful to convey implicitly. When every node in your system deals in lamport timestamps, then manually annotating every call with a timestamp parameter & manually handling that parameter and adding it to the response will quickly get tedious. If you want to do distributed tracing, then you want that context everywhere.

These kinds of things represent the kind of state/context which is implicit from the callstack in traditional single-threaded applications

There are some other places where I think metadata could be useful, although they probably are also more matters of taste, like identity/authentication (one of my thoughts here is that sometimes it might be advantageous to use metadata to pass this kind of information because this ensures it is always in a uniform location where intermediate proxies/membranes can find it without needing to be aware of involved schemas. You could potentially see a system which implements authentication/authorisation through collaborating membranes where those membranes need to communicate identifying information with each other, for example.)

There are some questions that would need resolving:

• How should metadata be represented? (I'd lean towards something like a Map<UInt64 TypeID, AnyPointer assumed to be of TypeID>)
  
• What messages can it apply to? Call/Return are obvious, but there are other possibilities that might be necessary (Embargo/Disembargo? Provide/Accept? Join?). Maybe metadata should just be a paramete of the top-level message?
  
• How to access/manipulate it? Presumably in C++ this could be exposed via CallContext on the server side / Request on the client side?
  
• How to support "middleware" and/or pass such ambient context around? I feel the potentially "easy" option is for middleware to simply be a membrane, but ambient context will depend upon language. In Go you might use context.Context for example; not sure what you would do in e.g. KJ
  

I realise this probably opens several cans of worns, and that metadata tends to always be somewhat inelegant, but it seems worthwhile to talk about becuase it's important in lots of distributed systems scenarios.

- Erin

[Kenton Varda]

Hi Erin,

In Cloudflare Workers, we've had some success attaching certain kinds of metadata by having a wrapper object which is called first, like:

    interface Endpoint {
      startCall @0 (metadata :Metadata) -> (dispatcher :CallDispatcher);
    }

    interface CallDispatcher {
      foo @0 (...) -> (...);
      bar @1 (...) -> (...);
      ...
    }

Here, the client always first calls startCall() to pass the metadata, then makes a pipelined call to the specific method. By convention, a `CallDispatcher` is expected to be used for exactly one call.

Of course, there's a limit to how far this pattern can go. It has worked well for us so far but it's definitely a bit clunky.

I'm really hesitant to attach a totally untyped int -> any map to every call. Having built several systems using such a pattern in the past, I know it can be very useful, but also tends to turn into a mess in certain ways.

For the specific use case of tracing, I can see a strong argument that this should be built into the system. You really do want trace IDs to pass down the call stack implicitly, without the business logic having to think about it (except in cases where the business logic wishes to explicitly declare a span). To make this work well in Cap'n Proto C++, we'd want to extend KJ's promise framework to implicitly propagate a trace context along the promise chain, so that the RPC system can pick it up.

I think authentication is at the opposite end of the spectrum. Implicitly propagating authentication credentials goes against the capability philosophy and gives rise to confused deputy problems. Authentication should be achieved by having the client exchange their credentials for an authenticated capability upfront, and then use that capability for further calls. A membrane should revoke the capability if the credentials become invalid. So I would not want to create a system that might be abused for authentication.

Lamport timers are interesting. I don't have a lot of intuition here as I haven't worked with them much. Naively it seems like you want to maintain a time counter in each vat, send the vector with each call/return, and merge the vectors received... however, this implies that the timer is ambient mutable state, which introduces lots of problems. Does it become a side channel through which it's possible to observe what else is going on within the vat? This makes me a bit uncomfortable.

Maybe you could implement Lamport clocks with membranes. You could wrap each group of objects that share a time counter in a single membrane. When a call exits the membrane, the membrane could first generate a preceding call to the same target object to some common propagateLamportTime() method. On the receiving end, the membrane intercepts this call and updates the receiver's clock. E-order guarantees that the propagateLamportTime() call is received immediately before the call that caused it. Alternatively, you could actually wrap outgoing calls in an envelope that include the timestamp, and have the receiving membrane unwrap the envelope.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/9ccbf8e1-b013-4744-be5f-d7b789bfa555%40www.fastmail.com.