Implementing Level-3 Protocol (in Go)

[lthibault]

Hello everyone,

I'm currently using the Go implementation of Cap'n Proto, which features Level 1 support for the RPC protocol, and I'm discovering that my use case would benefit greatly from Level 3 support.  Since Ross is no longer actively maintaining the project, I'm investigating the possibility of implementing Level 3 support myself, and possibly volunteering to maintain the project.  

I've read through the Cap'n Proto spec (i.e. the .capnp files in the github repository, especially the network interface sketch), perused conversations in this Google Group, and even tried to scavenge from the E language website/wiki, but I'm still struggling to get a "big picture" idea of how the 3-way introduction protocol works.

As such, I'm you kind folks might be able to point me in the right direction, starting with two very general questions:

1. How does the 3-way introduction protocol work?  Is it specified anywhere (e.g. in an RFC?)
2. What does the roadmap look like from Leve 1 to Level 3?
3. My understanding from reading the source is that Level 2 can largely be ignored.  Does Level 3 depend on level 2 in any way?  The numbering suggests so...

Many thanks in advance,
Louis

[Kenton Varda]

Hi Louis,

This is somewhat embarrassing, but there is no level 3 implementation of Cap'n Proto today, in any language. It's likely that when I get around to implementing it in C++, the protocol spec will change to solve unanticipated problems. As such, trying to implement the spec as-written in Go today might be frustrating, as it may be hard to tell the difference between the spec having a bug vs. the spec doing something weird intentionally, and there's no reference implementation to test against. :/

As for level 2, yes, we eventually realized that level 2 doesn't really belong in the core spec at all, so you can pretty much ignore the level 2 interfaces. Maybe I should actually delete them...

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/e9ad3f6d-a78c-4991-a3de-9a0fc071b906n%40googlegroups.com.

[lthibault]

Hi Kenton,

Wow -- thanks for the prompt reply! :)  This makes sense, thanks for clarifying.

I'd be more than happy to get involved in making the Level 3 spec and implementation a reality, and I have the bandwidth to dedicate to such a project.  Any chance we might be able to work together on this?

Again, I'm quite eager to push this forward as 3-way introductions are a killer feature for a project I'm working on.

Let me know!

Louis

[Ian Denhardt]

If you're keen on working on this I can probably be roped in; I've used
the go implementation a fair bit and contributed some patches, and it
would be neat to have a level 3 implementation out there.

Maybe you're aware of this, but you probably want to start from the v3
branch of the Go implementation, as it fixes some problems in the RPC
layer.

-Ian

Quoting lthibault (2020-09-23 12:21:41)

> On Wed, Sep 23, 2020 at 11:02 AM lthibault <[1]loui...@gmail.com>
> wrote:
>
> Hello everyone,
>
> I'm currently using the [2]Go implementation of Cap'n Proto, which

> features Level 1 support for the RPC protocol, and I'm discovering that
> my use case would benefit greatly from Level 3 support.� Since Ross is
> no longer actively maintaining the project, I'm investigating the
> possibility of implementing Level 3 support myself, and possibly
> volunteering to maintain the project.� �
>
> I've read through the Cap'n Proto spec (i.e. the .capnp files in the

> github repository, especially the [3]network interface sketch), perused

> conversations in this Google Group, and even tried to scavenge from the
> E language website/wiki, but I'm still struggling to get a "big
> picture" idea of how the 3-way introduction protocol works.
>
> As such, I'm you kind folks might be able to point me in the right
> direction, starting with two very general questions:
>

> 1. How does the 3-way introduction protocol work?� Is it specified

> anywhere (e.g. in an RFC?)

> 2. What does the roadmap look like from Leve 1 to Level 3?
> 3. My understanding from reading the source is that Level 2 can

> largely be ignored.� Does Level 3 depend on level 2 in any way?�
> The numbering suggests so...
>
> Many thanks in advance,
> Louis
>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to [4]capnproto+...@googlegroups.com.

> To view this discussion on the web visit

> [5]https://groups.google.com/d/msgid/capnproto/e9ad3f6d-a78c-4991-a3
> de-9a0fc071b906n%40googlegroups.com.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [6]capnproto+...@googlegroups.com.

> To view this discussion on the web visit

> [7]https://groups.google.com/d/msgid/capnproto/1eaff9b1-c74f-47de-9f8f-
> 35eb4edc73c2n%40googlegroups.com.
>
> Verweise
>
> 1. file:///tmp/m1urjosi.html
> 2. https://github.com/capnproto/go-capnproto2
> 3. https://github.com/capnproto/capnproto/blob/cf34b937f08af618b4dc057a0f5dd47acbf4e615/c%2B%2B/src/capnp/rpc.capnp#L1375-L1476
> 4. file:///tmp/m1urjosi.html
> 5. https://groups.google.com/d/msgid/capnproto/e9ad3f6d-a78c-4991-a3de-9a0fc071b906n%40googlegroups.com?utm_medium=email&utm_source=footer
> 6. mailto:capnproto+...@googlegroups.com
> 7. https://groups.google.com/d/msgid/capnproto/1eaff9b1-c74f-47de-9f8f-35eb4edc73c2n%40googlegroups.com?utm_medium=email&utm_source=footer

[lthibault]

Ian,

Great!  I've used the Go library quite heavily and have poked around the code a little bit, but I have yet to contribute anything, so I'm grateful for the collaboration.

>Maybe you're aware of this, but you probably want to start from the v3 branch of the Go implementation, as it fixes some problems in the RPC layer.

Good call.  I was aware that v3 was in the works, but haven't looked at the branch.

How do you suggest we proceed?  I'm thinking it might be useful to get synchronized over a Zoom call since (a) I could benefit from a guided tour of v3 and (b) my initial high-level questions about the Level 3 protocol remain open.  Would be cool if Kenton could join us too. :)

Thoughts?

[Ian Denhardt]

Quoting lthibault (2020-09-23 14:41:13)

> How do you suggest we proceed? I'm thinking it might be useful to get
> synchronized over a Zoom call since (a) I could benefit from a guided
> tour of v3 and (b) my initial high-level questions about the Level 3
> protocol remain open. Would be cool if Kenton could join us too. :)

I'd be up for a video call. Are you amenable to Jitsi Meet[1] instead of
Zoom? We've been using it for the weekly Sandstorm office hours calls.

We should also perhaps reach out to Ross; I know he doesn't have time to
hack on it, but might still be interested in weighing in on design
questions/offering advice. Want to open an issue re: level 3 on the Go
implementation's issue tracker? Maybe we can coordinate the rest of
this there. I know Ross is also on this mailing list, so he may see this
anyway, but he'll definitely see it there.

-Ian

[1]: https://meet.jit.si

[lthibault]

>Are you amenable to Jitsi Meet[1] instead of Zoom? 

That's totally fine.  I've been meaning to ditch Zoom, anyway.

>We should also perhaps reach out to Ross

Absolutely.  Would you be able to reach out to him?

I think you and I are in the same time zone (I'm in New Haven, CT), and I believe Kenton and Ross are on the west coast, so maybe early early afternoon?

Any particular preference for the day?

Cheers,

Louis

[Ian Denhardt]

I just opened an issue:

https://github.com/capnproto/go-capnproto2/issues/160

I believe Kenton is in Texas these days. For my part I could manage most
afternoons/evenings (eastern time) though Monday & Thursday afternoons
are bad. But if Ross and/or Kenton are interested in joining us I
suspect their schedules are much more constrained than mine.

-Ian

Quoting lthibault (2020-09-23 18:31:05)

> [1]: [1]https://meet.jit.si

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [2]capnproto+...@googlegroups.com.

> To view this discussion on the web visit

> [3]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-
> e105626e24b3n%40googlegroups.com.
>
> Verweise
>
> 1. https://meet.jit.si/
> 2. mailto:capnproto+...@googlegroups.com
> 3. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-e105626e24b3n%40googlegroups.com?utm_medium=email&utm_source=footer

[Louis Thibault]

Sounds good to me — looking forward to it.

- Louis

[Ross Light]

I'm available next Friday (Oct 2) from 1p PT onward.

[lthibault]

Friday October 2 at 13:00 PST (16:00 EST for Ian and myself) works for me.

Ian, is that ok with you?

Kenton, any chance we might count you in as well?

- Louis

[Ian Denhardt]

Works for me.

Quoting lthibault (2020-09-25 11:04:56)

> Friday October 2 at 13:00 PST (16:00 EST for Ian and myself) works for
> me.
>
> Ian, is that ok with you?
>
> Kenton, any chance we might count you in as well?
>
> - Louis
>
> On Friday, September 25, 2020 at 10:52:25 AM UTC-4 rli...@gmail.com
> wrote:
>
> I'm available next Friday (Oct 2) from 1p PT onward.
>
> On Friday, September 25, 2020 at 7:36:07 AM UTC-7 lthibault wrote:
>
> Sounds good to me � looking forward to it.
> - Louis
> > On Sep 24, 2020, at 17:17, Ian Denhardt <i...@zenhack.net> wrote:
> >
> > I just opened an issue:
> >

> > [1]https://github.com/capnproto/go-capnproto2/issues/160

> >> [1]: [1][2]https://meet.jit.si

> >>
> >> --
> >> You received this message because you are subscribed to the
> Google
> >> Groups "Cap'n Proto" group.
> >> To unsubscribe from this group and stop receiving emails from it,
> send
> >> an email to [2]capnproto+...@googlegroups.com.
> >> To view this discussion on the web visit
> >>

> [3][3]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52
> -afca-
> >> e105626e24b3n%[4]40googlegroups.com.
> >>
> >> Verweise
> >>
> >> 1. [5]https://meet.jit.si/
> >> 2. mailto:capnproto+...@googlegroups.com
> >> 3.
> [6]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-af
> ca-e105626e24b3n%40googlegroups.com?utm_medium=email&utm_source=foot
> er

>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [7]capnproto+...@googlegroups.com.

> To view this discussion on the web visit

> [8]https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-4041-adaa-
> e482f6b3d402n%40googlegroups.com.
>
> Verweise
>
> 1. https://github.com/capnproto/go-capnproto2/issues/160
> 2. https://meet.jit.si/
> 3. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-
> 4. http://40googlegroups.com/
> 5. https://meet.jit.si/
> 6. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-e105626e24b3n%40googlegroups.com?utm_medium=email&utm_source=footer
> 7. mailto:capnproto+...@googlegroups.com
> 8. https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-4041-adaa-e482f6b3d402n%40googlegroups.com?utm_medium=email&utm_source=footer

[Kenton Varda]

Sorry for the very slow response this time. We had a big launch on Monday and I haven't had much time to reply to e-mail.

Incidentally that launch is vaguely relevant here.

https://blog.cloudflare.com/introducing-workers-durable-objects/

We built the infrastructure for this on Cap'n Proto. When you make a request to a Durable Object, you're actually making Cap'n Proto RPC requests all they way from the call site to the destination object. Since Durable Objects will soon be able to migrate around the world, 3-party handoff is likely to become important as old locations will need to forward requests to new locations.

The Workers Runtime is written in C++, but we do have a fair amount of Go code at Cloudflare, so it'd be neat if 3PH were supported there too.

In any case, I'd like to start working on this soon, but I also have a big pile of other things I need to work on so I'm not sure how much time I'll have to dedicate.

Based on my experience working on the RPC system in the past, I think it's safe to expect that as we implement 3PH, we're likely to run into a bunch of little spec issues. It'll probably take the least time overall if I can manage to take an initial crack at it first in C++ and just power through things, rather than have other people try to implement the spec as written and then have to ask a lot of questions when there's confusion. I'm hoping to get at least a few days to work on it maybe the week after next.

As it happens I am available at 13:00 PDT (15:00 CDT) Friday if you still want to chat then.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/160104957747.876.5192754664800499465%40localhost.localdomain.

[Louis Thibault]

Hi Kenton,

Thanks for your email and congratulations on the launch!  Looks really cool!

It'll probably take the least time overall if I can manage to take an initial crack at it first in C++ and just power through things, rather than have other people try to implement the spec as written and then have to ask a lot of questions when there's confusion.

Sure, that makes perfect sense.  I’m thinking it’s probably best if we push back the meeting until after you’ve laid down the foundations for 3PH, at which point we can kill two birds with one stone:  changes to the spec based on your C++ implementation, and considerations for the Go implementation.

Kenton, I don’t want to pressure you into meeting any sort of deadline, but I also don’t want this project to fall through the cracks.  Is it ok if we tentatively reschedule for e.g. Friday October 16th at 13:00 PDT and make any adjustments as-needed?  (Ross, Ian, does that date also work for you?)

Cheers,

Louis

[Ian Denhardt]

I'm okay re-scheduling for the 16th if others are. I think we should
just plan to move forward one way or the other from there though;
there's some utility maybe in having Kenton work out the kinks first,
but ultimately I feel like I understand things well enough to move ahead
of the C++ implementation if we need to, and I'd rather not block on it
indefinitely.

-Ian

Quoting Louis Thibault (2020-09-30 15:45:21)

> Hi Kenton,
>
> Thanks for your email and congratulations on the launch! Looks really
> cool!
>
> It'll probably take the least time overall if I can manage to take an
> initial crack at it first in C++ and just power through things, rather
> than have other people try to implement the spec as written and then
> have to ask a lot of questions when there's confusion.
>
> Sure, that makes perfect sense. I'm thinking it's probably best if we
> push back the meeting until after you've laid down the foundations for
> 3PH, at which point we can kill two birds with one stone: changes to
> the spec based on your C++ implementation, and considerations for the
> Go implementation.
>
> Kenton, I don't want to pressure you into meeting any sort of deadline,
> but I also don't want this project to fall through the cracks. Is it
> ok if we tentatively reschedule for e.g. Friday October 16th at 13:00
> PDT and make any adjustments as-needed? (Ross, Ian, does that date
> also work for you?)
>
> Cheers,
>
> Louis
>

> On Sep 30, 2020, at 15:29, Kenton Varda <[1]ken...@cloudflare.com>

> wrote:
>
> Sorry for the very slow response this time. We had a big launch on
> Monday and I haven't had much time to reply to e-mail.
> Incidentally that launch is vaguely relevant here.

> [2]https://blog.cloudflare.com/introducing-workers-durable-objects/

> We built the infrastructure for this on Cap'n Proto. When you make a
> request to a Durable Object, you're actually making Cap'n Proto RPC
> requests all they way from the call site to the destination object.
> Since Durable Objects will soon be able to migrate around the world,
> 3-party handoff is likely to become important as old locations will
> need to forward requests to new locations.
> The Workers Runtime is written in C++, but we do have a fair amount of
> Go code at Cloudflare, so it'd be neat if 3PH were supported there too.
> In any case, I'd like to start working on this soon, but I also have a
> big pile of other things I need to work on so I'm not sure how much
> time I'll have to dedicate.
> Based on my experience working on the RPC system in the past, I think
> it's safe to expect that as we implement 3PH, we're likely to run into
> a bunch of little spec issues. It'll probably take the least time
> overall if I can manage to take an initial crack at it first in C++ and
> just power through things, rather than have other people try to
> implement the spec as written and then have to ask a lot of questions
> when there's confusion. I'm hoping to get at least a few days to work
> on it maybe the week after next.
> As it happens I am available at 13:00 PDT (15:00 CDT) Friday if you
> still want to chat then.
> -Kenton

> On Fri, Sep 25, 2020 at 10:59 AM Ian Denhardt <[3]i...@zenhack.net>

> wrote:
>
> Works for me.
> Quoting lthibault (2020-09-25 11:04:56)
> > Friday October 2 at 13:00 PST (16:00 EST for Ian and myself)
> works for
> > me.
> >
> > Ian, is that ok with you?
> >
> > Kenton, any chance we might count you in as well?
> >
> > - Louis
> >
> > On Friday, September 25, 2020 at 10:52:25 AM UTC-4

> [4]rli...@gmail.com

> > wrote:
> >
> > I'm available next Friday (Oct 2) from 1p PT onward.
> >
> > On Friday, September 25, 2020 at 7:36:07 AM UTC-7 lthibault
> wrote:
> >
> > Sounds good to me � looking forward to it.
> > - Louis
> > > On Sep 24, 2020, at 17:17, Ian Denhardt

> <[5]i...@zenhack.net> wrote:
> > >
> > > I just opened an issue:
> > >

> > > [1][6]https://github.com/capnproto/go-capnproto2/issues/160

> > [7]i...@zenhack.net

> > >> [1]: [1][2][8]https://meet.jit.si

> > >>
> > >> --
> > >> You received this message because you are subscribed to
> the
> > Google
> > >> Groups "Cap'n Proto" group.
> > >> To unsubscribe from this group and stop receiving emails
> from it,
> > send

> > >> an email to [2][9]capnproto+...@googlegroups.com.

> > >> To view this discussion on the web visit
> > >>
> >

> [3][3][10]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-
> 4e52
> > -afca-
> > >> e105626e24b3n%[4][11]40googlegroups.com.
> > >>
> > >> Verweise
> > >>
> > >> 1. [5][12]https://meet.jit.si/
> > >> 2. mailto:[13]capnproto+...@googlegroups.com
> > >> 3.
> >
> [6][14]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e5
> 2-af
> >
> ca-e105626e24b3n%[15]40googlegroups.com?utm_medium=email&utm_source=
> foot

> > er
> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "Cap'n Proto" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send

> > an email to [7][16]capnproto+...@googlegroups.com.

> > To view this discussion on the web visit
> >

> [8][17]https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-404
> 1-adaa-
> > e482f6b3d402n%[18]40googlegroups.com.
> >
> > Verweise
> >
> > 1. [19]https://github.com/capnproto/go-capnproto2/issues/160
> > 2. [20]https://meet.jit.si/
> > 3.
> [21]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-a
> fca-
> > 4. [22]http://40googlegroups.com/
> > 5. [23]https://meet.jit.si/
> > 6.
> [24]https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-a
> fca-e105626e24b3n%40googlegroups.com?utm_medium=email&utm_source=foo
> ter
> > 7. mailto:[25]capnproto+...@googlegroups.com
> > 8.
> [26]https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-4041-a
> daa-e482f6b3d402n%40googlegroups.com?utm_medium=email&utm_source=foo
> ter

> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to [27]capnproto+...@googlegroups.com.

> To view this discussion on the web visit

> [28]https://groups.google.com/d/msgid/capnproto/160104957747.876.519
> 2754664800499465%40localhost.localdomain.
>
> Verweise
>
> 1. mailto:ken...@cloudflare.com
> 2. https://blog.cloudflare.com/introducing-workers-durable-objects/
> 3. mailto:i...@zenhack.net
> 4. mailto:rli...@gmail.com
> 5. mailto:i...@zenhack.net
> 6. https://github.com/capnproto/go-capnproto2/issues/160
> 7. mailto:i...@zenhack.net
> 8. https://meet.jit.si/
> 9. mailto:capnproto%2B...@googlegroups.com
> 10. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52
> 11. http://40googlegroups.com/
> 12. https://meet.jit.si/
> 13. mailto:capnproto%2B...@googlegroups.com
> 14. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-af
> 15. http://40googlegroups.com/?utm_medium=email&utm_source=foot
> 16. mailto:capnproto%2Bunsu...@googlegroups.com
> 17. https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-4041-adaa-
> 18. http://40googlegroups.com/
> 19. https://github.com/capnproto/go-capnproto2/issues/160
> 20. https://meet.jit.si/
> 21. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-
> 22. http://40googlegroups.com/
> 23. https://meet.jit.si/
> 24. https://groups.google.com/d/msgid/capnproto/7bdd072b-9496-4e52-afca-e105626e24b3n%40googlegroups.com?utm_medium=email&utm_source=footer
> 25. mailto:capnproto%2Bunsu...@googlegroups.com
> 26. https://groups.google.com/d/msgid/capnproto/ed1fa9e0-362e-4041-adaa-e482f6b3d402n%40googlegroups.com?utm_medium=email&utm_source=footer
> 27. mailto:capnproto%2Bunsu...@googlegroups.com
> 28. https://groups.google.com/d/msgid/capnproto/160104957747.876.5192754664800499465%40localhost.localdomain

[Louis Thibault]

100% agree — let’s not lose momentum.  I’ll send out another invite when we hear back from Ross.

Cheers,

Louis

[Kenton Varda]

I'm still happy to chat this Friday.

-Kenton

[Louis Thibault]

Alright, it seems like everyone is eager to meet this Friday, so let’s do that. :)

See you all on Friday, as initially planned.

- Louis

[Kenton Varda]

Was someone going to send out a meeting invite or something?