"Standardizing" CapTP?

[Ian Denhardt]

I know Chris has talked about wanting to see if they can achieve some
interoperability between their CapTP implementation and Agoric's stuff.
I'm reading about the work Chris is doing while I work with Cap'n Proto,
and it similarly feels like a missed opportunity to have three different
protocols that are all kinda the same thing but can't talk to each
other. Especially given Cap'n proto has working rpc implementations in
roughly half a dozen languages, plus a few more that have bindings to
the C++ implementation. So I wanted to get a conversation going on what
this might look like.

This came up briefly on one of the Sandstorm project's "office hours"
calls, and I remember Kenton saying something to the effect that Mark
says he's a big fan of Cap'n Proto, so if Agoric isn't using it there's
probably a good reason. Mark, do you want to weigh in on this?

Curious to everyone else's thoughts, too.

-Ian

[William ML Leslie]

On Wed, 2 Sep 2020 at 06:27, Ian Denhardt <i...@zenhack.net> wrote:

I know Chris has talked about wanting to see if they can achieve some
interoperability between their CapTP implementation and Agoric's stuff.
I'm reading about the work Chris is doing while I work with Cap'n Proto,
and it similarly feels like a missed opportunity to have three different
protocols that are all kinda the same thing but can't talk to each
other.  Especially given Cap'n proto has working rpc implementations in
roughly half a dozen languages, plus a few more that have bindings to
the C++ implementation. So I wanted to get a conversation going on what
this might look like.

Part of the challenge there is that while there are many implementations of CapTP, they often use different transports underneath. E, including E-on-CL, uses the pluribus protocol over TCP, which absolutely made sense at the time.  The javascript implementation of CapTP which iirc lives in Caja is a unibus system, but iiuc is binary compatible if you can get a matching transport serverside.  Similarly, when I started the python implementation, I wanted to speak CapTP over websockets so that I could build web applications; I'd been doing a custom pseudo-captp per app for years at that point and wanted something more fully-featured and reusable.

I had gotten a bit of an impression from Kevin that it's not a mammoth task to just port E to your platform and use that version.  E-on-CL is impressive for how minimal it is, but given that people are often funny about the size of dependencies I get why just doing CapTP might make sense.

 

This came up briefly on one of the Sandstorm project's "office hours"
calls, and I remember Kenton saying something to the effect that Mark
says he's a big fan of Cap'n Proto, so if Agoric isn't using it there's
probably a good reason. Mark, do you want to weigh in on this?

Curious to everyone else's thoughts, too.

The key difference between the two, which I think speaks to the domain in which they are optimised for, is that Cap'nProto is a data-first protocol, where DataE seems more purpose-built for distributing object graphs.  In Cap'nProto, the result of a method call is a data structure containing a number of capabilities within, but in CapTP, an answer can be resolved with an arbitrary expression that could supply an existing local ref, construct a unum presence, build a data value, or it could not be resolved at all.  Either way, in E you can start making eventual sends to the promise right away.

Also, when Agoric were starting out, I don't think there was a pure javascript Cap'nProto implementation (is there now?).  There was a node.js version that wrapped the C++ library.  I don't even think the TypedArray was widely available, which would be required for an efficient pure javascript Cap'nProto.

--

William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered under copyright law.  You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement.

[Ian Denhardt]

Quoting William ML Leslie (2020-09-01 19:39:04)

> Part of the challenge there is that while there are many
> implementations of CapTP, they often use different transports
> underneath. E, including E-on-CL, uses the pluribus protocol over TCP,
> which absolutely made sense at the time.� The javascript
> implementation of CapTP which iirc lives in Caja is a unibus system,
> but iiuc is binary compatible if you can get a matching transport
> serverside.� Similarly, when I started the python implementation, I
> wanted to speak CapTP over websockets so that I could build web
> applications;

Fwiw, capnproto can run over any reliable transport; just last week I
had use for speaking capnproto over a websocket and wrote the necessary
glue code in ~5 min or so. So far nobody's written an implementation
that supports third party handoff, but the C++ library is designed to
make that pluggable too (some of the others have this less carefully
factored out, but i probably wouldn't be major surgery to add it. And
once you have one implementation that supports this you can write
gateway proxies...)

> Either way, in E you can start making eventual sends to the promise
> right away.

You can do this in capnproto too, as the protocol supports projecting on
fields of not-yet-arrived results. But also the fact that method
results are supposed to be structs is itself an arbitrary restriction,
and it would be trivial to relax it in a backwards-compatible way (to
the point where an implementation taking a postel's law approach might
already support this, by accident).

> Also, when Agoric were starting out, I don't think there was a pure
> javascript Cap'nProto implementation (is there now?).

There's one that's serialization only, but RPC hasn't been implemented.
I'm definitely interested in making this happen though, as it would be
great to have an implementation that runs in the browser, and also the
node implementation adds some complications to Sandstorm's build system
that I would like to be rid of.

-Ian

[Christopher Lemmer Webber]

Took a while but an issue that's basically relevant to Ian's comments here:
https://github.com/Agoric/agoric-sdk/issues/1827

[mostawe...@gmail.com]

The Monte stance on all of this is to do what works. Our current framing is the venerable AMP (https://amp-protocol.net/), which can be implemented relatively quickly, and our encoding is JSON plus a little "dollar dollar" trick to encode opaque refs as strings. But also, we want to support Capn Proto for framing and encoding because we think that it will be faster and lighter.

The bigger issue is that the calling conventions are not fully interoperable between various systems. I think that Monte's calling convention subsumes Goblins' and E's and even Agoric's, but the objects at each end of the connection are different, and possibly so different that a Monte client might well want a non-Monte vat to first "lay out the welcome mat" and set up a Monte-compatible scope by booting a small kernel.

There are also structural speed concerns. My current test work involves raytracing, and I distribute the work among AMP workers with a promise chain that roughly looks like:

    def color := drawable<-drawAt(x, y, => aspectRatio, => pixelRadius)

    def srgb := color<-sRGB()

    when (srgb) ->

        def [r, g, b, a] := srgb

Where `drawable` is a magic ref which does load-balancing; see https://github.com/monte-language/typhon/blob/master/mast/lib/promises.mt#L48-L80

This is too slow. Vanilla single-core Monte can do hundreds, sometimes thousands of pixels/second, but the encoding and round-trip overhead mean that I only get about a dozen pixels/second/worker. I can imagine a couple ways to batch pixel requests, but I'd love for a generic solution to exist; I don't want Goblins+Monte interoperability, at real-time speeds suitable for video-game production, to be dependent on extremely fiddly details of how developers use our CapTP libraries.

[Christopher Lemmer Webber]

Do you do full CapTP serialization/deserialization between vats in the
same OS process? We don't in Goblins... I'd think that for video game
cases that helps with the problem, maybe... but as I type this, I
realize maybe not, the same OS process in Goblins is merely
green-threaded, and we'll probably have to suffer
serialization/deserialization between threads or processes anyway...

It seems realistic for game systems that we're not talking about sending
stuff over a wire, just between threads. Maybe things will be different
in the Guile version, which won't be green-threaded, using actual OS
threads.

- Chris

[mostawe...@gmail.com]

Sadly, Typhon doesn't multi-thread in a useful way; it's built on RPython, and has a GIL just like PyPy. Since we don't run I/O during mutation, the GIL's basically always held by a single mutating vat who just wants to take its turn. So we don't get to have the problem that you're outlining.

On a post-Typhon system, then yeah, we would hope that we can do something lighter-weight. In Typhon, there's a prototype "local" ref which behaves like a far ref but transparently migrates DeepFrozen objects between vats. The idea is to simulate a multitude of isolated process spaces within a single Linux-style multi-threaded process. Each thread associates itself to a vat, using thread locals. DeepFrozen objects can be invoked from any thread at a time without reentrancy problems, in theory, so it doesn't matter which vat we assign them to.

On a *real* post-Typhon system (but still on GNU/Linux) which wants to go fast, MPS (https://github.com/Ravenbrook/mps) is recently relicensed under BSD 2-clause! This library supports a radical number of arena-and-pool allocator configurations, including a desired scheme where each thread has a pool and each vat has a temporary arena for each turn, which is marked and swept up into the thread's pool, in a way which doesn't permanently tie vats to particular threads.

FWIW Allen and I have been conceptualizing game logic as largely in one process, cooperatively threaded between several small tasklet objects who agree to yield quickly after enqueuing I/O. It's what we're used to from Twisted, and it's how the pioneers used to make their video games back in the single-core days. I think that SMP multi-core seems like the only time when the Linux-style threads and the single big address space can work together for speed, and I don't know how long it's going to last.

[Raoul Duke]

hmm afaik things like haskell and erlang were doing preemptive concurrency even before getting smp? i know, they aren't python.