Wikipedia Draft Article

[Zach Lym]

I have a draft entry for Cap'n Proto awaiting review on Wikipedia.  While Wikipedia's conflict-of-interest policy appears to prevent contributors to Cap'n Proto from making edits directly, just put it in the talk page and tag it with `{{Request edit}}` : D

https://en.wikipedia.org/wiki/Draft:Cap%E2%80%99n_Proto

-Zach Lym

[Kenton Varda]

Hi Zach,

Thanks for doing this!

A few nitpicks:

> the former maintainer of Google's popular Protocol Buffers RPC framework

Technically Protocol Buffers is a serialization framework, not RPC. gRPC and Stubby are RPC frameworks built on Protobuf, but I never worked on either of them.

> #@n marks order of values in the message.

Technically it marks the order in which fields were added to the message definition, not literally the order in which they appear in the serialization. The order in which fields were added to the definition is important in order to ensure that fields don't change position as new fields are added. However, newer fields can be placed before older fields, if there happened to be space available from padding.

> as apposed text encoding used

Typo: opposed

> Cap'n Proto instead tries to represent values in wire-protocol as they would be when written to disk or even in-memory as much as possible

This sentence implies that on-disk formats and in-memory formats are typically similar, with network formats being different. In practice, though, on-disk formats are typically closer to network protocols. Like protocols, storage formats often need to be position-independent, forwards- and backwards-compatible, language-independent, architecture-independent, secure, etc.

I think what this sentence was trying to get at is that Cap'n Proto tries to make the storage/network protocol appropriate as an in-memory format, so that no translation step is needed when reading data into memory or writing data out of memory.

> Unlike other binary serialization protocols such as XMI or Protocol Buffers, Cap'n Proto considers fine-grained data validation at the RPC level an anti-feature that limits a protocols ability to evolve. This was informed by experiences at Google where simply changing a field from mandatory to optional would cause complex operational failures

For the most part, Protobuf followed the same philosophy from the start. The only exception was required vs. optional, which is a fairly minor bit of validation. As you note, "required" was removed from the language in proto3, making Protobuf and Cap'n Proto basically equivalent in this regard. But, even before "required" was removed, I think they were pretty similar.

I don't know anything about XMI.

> As of October 2020, the reference implementation only supports level 2

The level numbers were defined by Cap'n Proto itself, so it's a little weird to reference "level 2" without defining what that means. I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto."

> After Sandstorm.io transitioned from a commercial to community focus

You can just say: "After Sandstorm failed commercially"

> Cloudflare, which subsequently adopted Cap'n Proto as their primary internal RPC system

This isn't quite accurate.

Cloudflare used Cap'n Proto serialization long before I joined, but it also uses many other technologies, including Protobuf/gRPC, JSON, etc., in other places.

I joined and started the Cloudflare Workers project. Workers specifically makes use of Cap'n Proto in its implementation. However, the rest of Cloudflare still uses a variety of technologies.

> The primary speedup over Protocol Buffers comes from using arena-style memory allocation.

I would argue the primary speedup comes from zero-copy parsing/serialization, not from arena-style allocation. Protobuf actually supports arena-style allocation, at least in the C++ implementation.

That said, in reality this all depends on the use case. Sometimes arena allocation might be what makes the difference. Sometimes Protobuf is actually faster. But often Cap'n Proto is faster because it doesn't need a translation step.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/2fac5f42-8d56-4e06-af47-7e55c6aba890n%40googlegroups.com.

[Mark S. Miller]

Rather than linking "promise pipelining" to a wikipedia page on "pipelining", which is a reach, just link to https://en.wikipedia.org/wiki/Futures_and_promises#Promise_pipelining

--

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/2fac5f42-8d56-4e06-af47-7e55c6aba890n%40googlegroups.com.

--

  Cheers,
  --MarkM

[Zach Lym]

On Mon, Mar 15, 2021 at 8:43 PM Kenton Varda <ken...@cloudflare.com> wrote:

Hi Zach,

Thanks for doing this!

Thanks for trying to advance capability oriented security! 

 

> Cap'n Proto instead tries to represent values in wire-protocol as they would be when written to disk or even in-memory as much as possible

This sentence implies that on-disk formats and in-memory formats are typically similar, with network formats being different. In practice, though, on-disk formats are typically closer to network protocols. Like protocols, storage formats often need to be position-independent, forwards- and backwards-compatible, language-independent, architecture-independent, secure, etc.

I think what this sentence was trying to get at is that Cap'n Proto tries to make the storage/network protocol appropriate as an in-memory format, so that no translation step is needed when reading data into memory or writing data out of memory.

Is that last sentence CC0 licensed?

> As of October 2020, the reference implementation only supports level 2

The level numbers were defined by Cap'n Proto itself, so it's a little weird to reference "level 2" without defining what that means.

My eyes kinda glazed over at that point and I don't have time right now to write my own summary.  I guess the website is MIT licensed, so I could copy and paste stuff, but I'm kinda unhappy with the website's explanation.  I think it would be better to wait for myself or someone else to show up and enhance the article.

 

I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto."

That sounds a bit too massaged.  I think it should flatly state that the reference implementation doesn't implement the full standard.

> Cloudflare, which subsequently adopted Cap'n Proto as their primary internal RPC system

This isn't quite accurate.

Cloudflare used Cap'n Proto serialization long before I joined, but it also uses many other technologies, including Protobuf/gRPC, JSON, etc., in other places.

I joined and started the Cloudflare Workers project. Workers specifically makes use of Cap'n Proto in its implementation. However, the rest of Cloudflare still uses a variety of technologies.

Google also uses a variety of other technologies, but I think it's accurate to say that Protobufs/gRPC is their "primary" RPC system.  I know you don't want to oversell your role in the company, but I don't think we should split hairs here.  What I'm trying to get across is that there is a massive technology company which adopted it internally and is actively contributing financial support to the project.  Does Cloudflare pay developers to hack on protobufs or Thrift?

- Zach

[Kenton Varda]

On Tue, Mar 16, 2021 at 12:13 AM Zach Lym <indol...@gmail.com> wrote:

Is that last sentence CC0 licensed?

Sure, feel free to reuse anything I wrote verbatim. CC0 license, or public domain, whichever you prefer.

 

I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto."

That sounds a bit too massaged.  I think it should flatly state that the reference implementation doesn't implement the full standard.

Fair enough.

Maybe: "As of March 2021, Cap'n Proto's reference implementation does not support all of the features that have been specified in its own protocol. In particular, three-party handoff and capability equality -- identified in the spec as level 3 and level 4, respectively -- are not yet supported."

Google also uses a variety of other technologies, but I think it's accurate to say that Protobufs/gRPC is their "primary" RPC system.  I know you don't want to oversell your role in the company, but I don't think we should split hairs here.  What I'm trying to get across is that there is a massive technology company which adopted it internally and is actively contributing financial support to the project.  Does Cloudflare pay developers to hack on protobufs or Thrift?

I don't think you can consider these equivalent. Nearly every project and engineer at Google uses Protobuf. If you try to choose something else you'll get pushback. At Cloudflare, each team makes their own decisions about what to use and doesn't receive any pressure to choose one tech over the other. I think only three projects use Cap'n Proto, out of dozens. There are probably more projects using Protobuf than Cap'n Proto (but I haven't done a survey).

Cloudflare doesn't really pay anyone to hack on Cap'n Proto specifically. They pay us to build Cloudflare Workers, and in doing that we add what we need to Cap'n Proto (or, more commonly, KJ). If someone on a Protobuf-using project decided they needed a change in Protobuf, they could go make that change. Of course, employing the owner of the project makes it a lot easier to get changes accepted, so there's that.

In any case, it's accurate to say Cloudflare is using and driving development of Cap'n Proto, but it's not accurate to say that Cap'n Proto is Cloudflare's "primary" internal RPC system.

-Kenton

[Zach Lym]

I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto."

That sounds a bit too massaged.  I think it should flatly state that the reference implementation doesn't implement the full standard.

Fair enough.

Maybe: "As of March 2021, Cap'n Proto's reference implementation does not support all of the features that have been specified in its own protocol. In particular, three-party handoff and capability equality -- identified in the spec as level 3 and level 4, respectively -- are not yet supported."

I agree with you that an explanation of the protocols should precede this statement.  But if you are self conscious about Cap'n Proto not supporting levels 3 & 4 ... it is weird and it makes Cap'n Proto feel half baked 😟.

I'm not saying this because I want to be mean, I love Cap'n Proto and want to see it succeed!  But the reference implementation being in C++ means it's not suitable for medium-to-high assurance projects, it lacks features that really set it apart from other frameworks (zero-copy IPC, advanced OCap functionality, and built in encryption), and it's not available in as many languages as competitors.  I really don't like giving this type of feedback, but hopefully it's constructive.

 

Google also uses a variety of other technologies, but I think it's accurate to say that Protobufs/gRPC is their "primary" RPC system.  I know you don't want to oversell your role in the company, but I don't think we should split hairs here.  What I'm trying to get across is that there is a massive technology company which adopted it internally and is actively contributing financial support to the project.  Does Cloudflare pay developers to hack on protobufs or Thrift?

I don't think you can consider these equivalent. Nearly every project and engineer at Google uses Protobuf. If you try to choose something else you'll get pushback. At Cloudflare, each team makes their own decisions about what to use and doesn't receive any pressure to choose one tech over the other. I think only three projects use Cap'n Proto, out of dozens. There are probably more projects using Protobuf than Cap'n Proto (but I haven't done a survey).

Cloudflare doesn't really pay anyone to hack on Cap'n Proto specifically. They pay us to build Cloudflare Workers, and in doing that we add what we need to Cap'n Proto (or, more commonly, KJ). If someone on a Protobuf-using project decided they needed a change in Protobuf, they could go make that change. Of course, employing the owner of the project makes it a lot easier to get changes accepted, so there's that.

In any case, it's accurate to say Cloudflare is using and driving development of Cap'n Proto, but it's not accurate to say that Cap'n Proto is Cloudflare's "primary" internal RPC system.

That's fair.  I've switched it to "which uses Cap'n Proto internally".

 

-Kenton

[Kenton Varda]

On Tue, Mar 16, 2021 at 2:26 PM Zach Lym <indol...@gmail.com> wrote:

I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto."

That sounds a bit too massaged.  I think it should flatly state that the reference implementation doesn't implement the full standard.

Fair enough.

Maybe: "As of March 2021, Cap'n Proto's reference implementation does not support all of the features that have been specified in its own protocol. In particular, three-party handoff and capability equality -- identified in the spec as level 3 and level 4, respectively -- are not yet supported."

I agree with you that an explanation of the protocols should precede this statement.  But if you are self conscious about Cap'n Proto not supporting levels 3 & 4 ... it is weird and it makes Cap'n Proto feel half baked 😟.

Eh? No, none of this bothers me in that way. I just thought it was weird to refer to "level 2" without any explanation of what that means, so I was trying to come up with ways to explain. If you want to just say "Eight years after introduction, Cap'n Proto's reference implementation still doesn't implement all of the protocol as specified." that's fine with me.

I'm not saying this because I want to be mean, I love Cap'n Proto and want to see it succeed!  But the reference implementation being in C++ means it's not suitable for medium-to-high assurance projects, it lacks features that really set it apart from other frameworks (zero-copy IPC, advanced OCap functionality, and built in encryption), and it's not available in as many languages as competitors.  I really don't like giving this type of feedback, but hopefully it's constructive.

Hmm, it seems like you're worried I'd find something hurtful here, but I don't, and I'm not even sure what it is that would be hurtful.

I built Cap'n Proto to suit my needs. When there's something missing that I need, I add it. If there are things missing that other people need, I invite them to add it. But if they decide to use something else instead, that doesn't bother me; I want them to use whatever works for them. I have no particular need or desire for Cap'n Proto to be widely adopted, I only need it to get the job done in my other projects. I'm way more interested in adoption of Cloudflare Workers than Cap'n Proto -- the former puts money in my pocket, the latter does not.

If you think it'd be useful to quote me on that to explain the state of the project, feel free. :)

-Kenton