Road map

[Kenton Varda]

This will go up on the web page along with the release on Wednesday, but you can see it in the repo now:

https://github.com/kentonv/capnproto/blob/master/doc/roadmap.md

Any thoughts?

-Kenton

[Andreas Stenius]

2013/9/2 Kenton Varda <temp...@gmail.com>

This will go up on the web page along with the release on Wednesday, but you can see it in the repo now:

https://github.com/kentonv/capnproto/blob/master/doc/roadmap.md

Any thoughts?

I've been pondering a bit on the RPC stuff (not at a protocol level, but at a transport level); and reading about your roadmap for RPC transports, I was just wondering if you're planning on doing this from scratch?, or borrow/use stuff from libraries that were developed specifically for this kind of stuff (thinking of zeromq/nanomsg and similar).

[Motiejus Jakštys]

On 2013.09.02 11:17, capn...@googlegroups.com wrote:
> Kenton Varda <temp...@gmail.com> Sep 01 05:16PM -0700

>
> This will go up on the web page along with the release on Wednesday, but
> you can see it in the repo now:
>
> https://github.com/kentonv/capnproto/blob/master/doc/roadmap.md
>

Hi Kenton,

not sure if there is a "+1" for wishlist; however, I want to point these
out:

JSON codec: API for transcoding to JSON format, useful for
interacting with legacy infrastructure.
JSON-HTTP proxy: Develop a web server which can expose a Cap'n
Proto RPC backend as a JSON-over-HTTP protocol.

At Spil Games we are exposing all our services via PB and JSON using
piqi[1] and piqi-rpc[2]. That way we have a RESTful interface for all
http+json (external, browser-based APIs), http+pb (internal APIs) and
native erlang+pb (internal Erlang APIs) using the same definition
format. As far as I know, piqi is the only tool that allows that kind of
stuff. I want to try another approach, so I am looking at capnp.

So once JSON codec is ready, I will start poking around with Erlang side
of things quite seriously.

Motiejus

[1]: http://piqi.org/
[2]: http://piqi.org/doc/piqi-rpc/

[Kenton Varda]

On Mon, Sep 2, 2013 at 1:01 AM, Andreas Stenius <g...@astekk.se> wrote:

I've been pondering a bit on the RPC stuff (not at a protocol level, but at a transport level); and reading about your roadmap for RPC transports, I was just wondering if you're planning on doing this from scratch?, or borrow/use stuff from libraries that were developed specifically for this kind of stuff (thinking of zeromq/nanomsg and similar).

I've been looking at these, and at first I was excited about using nanomsg, but the more I look at it, the more I realize it's not a good fit.

The thing is, where zeromq/nanomsg seek to provide various "socket patterns" like publish-subscribe, Cap'n Proto's capability-based model provides building blocks which you can use to build all those patterns.  If you want pub-sub, for instance, you'd write an interface like:

    interface Stream {

      subscribe(subscriber :Subscriber): Void;

      # Call `subscriber.receive()` whenever there is a new item.

    }

    interface Subscriber {

      receive(item :Item): Void;

    }

There's simply no need for the system to have any built-in notion of pub-sub, because it's so easy to define on top of it.

So Cap'n Proto would not use zeromq/nanomsg's pub-sub.  What would it use?  Well, the only pattern that seems remotely useful is the request-response pattern.  But I'm actually not sure we can even leverage that.  In a Cap'n Proto connection, there is no defined requester or responder.  Rather, both ends of the connection can hold any number of capabilities pointing at the other end and can make requests to those as needed.  I can't quite tell if this is possible at all with zeromq/nanomsg.

It's also unclear to me whether the zeromq/nanomsg APIs allow true zero-copy communication through shared memory.  This requires that the transport itself be responsible for allocating the buffer in which the message is built.

In any case, it seems like most (all?) of what these systems provide, Cap'n Proto by its nature wouldn't be able to use.  Luckily this doesn't mean we have to reinvent everything they do, because a lot of what they do is stuff that would be done above Cap'n Proto in application code (or in higher-level libraries).

These are preliminary thoughts, though.  I'm going to keep investigating and see if I missed something.

-Kenton

[Andreas Stenius]

2013/9/2 Motiejus Jakštys <desir...@gmail.com>

[...]

So once JSON codec is ready, I will start poking around with Erlang side of things quite seriously.

Hi Motiejus,

Just a heads up that I've got some basic capnp for erlang done at http://github.com/kaos/ecapnp in case you don't want to start from scratch.. ;)

[Andreas Stenius]

2013/9/2 Kenton Varda <temp...@gmail.com>

On Mon, Sep 2, 2013 at 1:01 AM, Andreas Stenius <g...@astekk.se> wrote:

I've been pondering a bit on the RPC stuff (not at a protocol level, but at a transport level); and reading about your roadmap for RPC transports, I was just wondering if you're planning on doing this from scratch?, or borrow/use stuff from libraries that were developed specifically for this kind of stuff (thinking of zeromq/nanomsg and similar).

[...]

It's also unclear to me whether the zeromq/nanomsg APIs allow true zero-copy communication through shared memory.  This requires that the transport itself be responsible for allocating the buffer in which the message is built.

Yeah, I started wondering about this myself too.. in particular if the transport protocol they use would incur extra overhead that we would like to avoid.

 

...

These are preliminary thoughts, though.  I'm going to keep investigating and see if I missed something.

I'll look forward to what you come up with, and I'll toss in any ideas I might get during the time as well :)

[Motiejus Jakštys]

On 2013.09.02 15:45, Andreas Stenius wrote:
>
> Hi Motiejus,
>
> Just a heads up that I've got some basic capnp for erlang done at
> http://github.com/kaos/ecapnp in case you don't want to start from
> scratch.. ;)

Hi Andreas,

sure, I know about your implementation and already had a (very quick)
look. However, cannot do the JSON part without the JSON codec API. :-)

Since we are talking already, why did you choose to implement message
generation/parsing in pure Erlang, rather than using libcapnp?

Motiejus

[Jan Huwald]

On 09/02/2013 02:16 AM, Kenton Varda wrote:
> https://github.com/kentonv/capnproto/blob/master/doc/roadmap.md
>
> Any thoughts?

Things I am looking forward (in order of urgency):
* Inline lists
* Schema compatibility checker
* mmap-friendly mutable storage format
Concurrent access or not? With relative pointers and arena
allocation you are already pretty close. Should there be a single
root per mmap-arena?
* Implement parameterized types
Will the related code leak deeply into API or can it be constrained
to the compiler?
* Type aliases
Nice too have, especially with parametrized types. I am peering at
partial specialization.

As a general note: several features on the roadmap (RPC, ORM, Snappy)
add transport/endpoint specification. As I do not intent to use any of
them, I hope that future code does not rely on them being used.

Regards,
Jan

[Kenton Varda]

On Mon, Sep 2, 2013 at 1:49 PM, Jan Huwald <j...@sotun.de> wrote:

* mmap-friendly mutable storage format
    Concurrent access or not? With relative pointers and arena
    allocation you are already pretty close. Should there be a single
    root per mmap-arena?

The major problem right now is that the standard serialization format places the segment table up front.  With mmap, you'd need to be able to allocate new segments over time without having to shift all existing data forward to make room in the segment table, so the table would have to be a slightly more complicated data structure.

Also, there's the problem that arena allocation makes it hard to reuse space from discarded objects.  I think to really claim that our mmap storage format is "mutable", we need a better story here.

Whether or not to have a single root is a really good question.  It might make a lot of sense to design the format to store a lot of small messages rather than one big message.  The idea would be that you write each small message as a single unit, and later on you can only replace it, not modify it.  Or maybe you can modify it, but given the above caveats about arena allocation, so you are usually encouraged to replace it entirely.

In any case, there's a lot to think about here.

 

* Implement parameterized types
    Will the related code leak deeply into API or can it be constrained
    to the compiler?

I'm not completely sure yet, but I think it will only affect the schema and dynamic APIs, and in a way that only affects you if you actually use parameterized types.  And of course the generated code will be different, but again only if you use parameterized types.

 

* Type aliases
    Nice too have, especially with parametrized types. I am peering at
    partial specialization.

Heh, I'll have to remember to make that work...

 

As a general note: several features on the roadmap (RPC, ORM, Snappy)
add transport/endpoint specification. As I do not intent to use any of
them, I hope that future code does not rely on them being used.

Yes, I believe in loose coupling.  :)  The serialization API will always exist roughly as it exists now, and using the other stuff on top of that will be optional.  The RPC system will probably be a separate library, even.

-Kenton

[Kenton Varda]

Hi Motiejus,

Thanks for the feedback.  JSON transcoding and the JSON-over-HTTP proxy will be necessary for my own future plans, but not right away.  So I can more or less guarantee that they'll happen eventually, but it may be many months away.  Unless someone decides to take this up and contribute an implementation, of course.  :)

Out of curiosity, do you see it as a strict requirement for your purposes that this translation layer be usable in-process in an Erlang program, or are you amenable to the idea of a stand-alone proxy written in pure C++ which accepts JSON-HTTP traffic and forwards it to your Erlang process as Cap'n Proto?

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[j.e....@gmail.com]

For RPC design inspiration, the thoughts of Martin Sústrik (of ZeroMQ fame) on his re-design in nanomsg might be worth considering: https://github.com/250bpm/nanomsg   and  http://nanomsg.org/documentation-zeromq.html

[Motiejus Jakštys]

On 2013.09.03 02:04, Kenton Varda wrote:
> Out of curiosity, do you see it as a strict requirement for your
> purposes that this translation layer be usable in-process in an Erlang
> program, or are you amenable to the idea of a stand-alone proxy written
> in pure C++ which accepts JSON-HTTP traffic and forwards it to your
> Erlang process as Cap'n Proto?

Hi Kenton,

For this purpose we are not doing anything more than request/response;
logic in web server would be quite simple: decode, pass to the
application, encode, give result back.

Decode and encode steps are straightforward, but talking to the
application layer and giving the result back to the client is subtle:
routing logic, service registration, error handling, partial failure,
return codes, custom headers are all part of the application/library,
not Cap'n Proto's responsibility. I would like if Cap'n Proto sticked
being black box with well defined encoding/decoding boundaries.

What is more, we are really sensitive about adding a new web server to
our application stack, but not so much a new library. I think there
should be a single implementation of schema parser, encoder and decoder,
but many (competing) implementations of web server, RPC and messaging
libraries built on top of Cap'n Proto. But nothing of these should be in
core. What is your motivation for current design of the interfaces? What
is the use case? And why RPC?

Besides, did you create/register the MIME type for Cap'n Proto?

Motiejus

[Kenton Varda]

On Tue, Sep 3, 2013 at 2:26 AM, <j.e....@gmail.com> wrote:

For RPC design inspiration, the thoughts of Martin Sústrik (of ZeroMQ fame) on his re-design in nanomsg might be worth considering: https://github.com/250bpm/nanomsg   and  http://nanomsg.org/documentation-zeromq.html

Indeed, I will be looking closely at these.

[Kenton Varda]

Mi Motiejus,

If I understand you correctly, it sounds like you're only interested in the JSON transcoder, not the proposed JSON-HTTP RPC proxy, because you want to keep control over how you handle HTTP traffic.  Is that correct?

That's fine with me.  Cap'n Proto will always allow you to use the serialization layer independently of any RPC system or transport.  Though my hope is that eventually the Cap'n Proto RPC system will be so convenient that you'd rather use it.  :)

-Kenton

[Nicholas Palko]

Out of sheer curiosity - are you leaning towards implementing the JSON-HTTP proxy as a FastCGI plugin, or were you thinking of going the route of mongrel2 and putting some HTTP scaffolding on top of your RPC platform?

[Kenton Varda]

I haven't thought deeply about it, but I was imagining implementing it as a module on top of some existing C/C++ HTTP server, perhaps nginx.  I am imagining this being something that a Cap'n-Proto-based shop would use to expose APIs for use by third parties (or perhaps by your own in-browser Javascript clients), therefore you'd only run the server as a frontend of sorts, while all of your backends would speak pure Cap'n Proto.  Since you probably need something like nginx as a reverse proxy anyway, might as well plug in the translation layer there.

-Kenton

--

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

[Andrew Lutomirski]

On Wed, Sep 11, 2013 at 4:44 PM, Kenton Varda <temp...@gmail.com> wrote:
> I haven't thought deeply about it, but I was imagining implementing it as a
> module on top of some existing C/C++ HTTP server, perhaps nginx. I am
> imagining this being something that a Cap'n-Proto-based shop would use to
> expose APIs for use by third parties (or perhaps by your own in-browser
> Javascript clients), therefore you'd only run the server as a frontend of
> sorts, while all of your backends would speak pure Cap'n Proto. Since you
> probably need something like nginx as a reverse proxy anyway, might as well
> plug in the translation layer there.

Yuck. nginx barely supports websockets, and I suspect that many users
are likely to want websockets. I'm about to deploy a production web
service using Twisted as the frontend because holy cow everything in
the space sucks. Memo to people who write HTTP servers: do not have
stupid default policies like advertising your version or listing
directories (or even serving files at all).

In general, there seems to be a lack of decent (or even non-awful)
interfaces for web backends to talk to web frontends. SCGI seems like
one of the less bad choices.

--Andy

[Kenton Varda]

Yeah, as I said, I have not thought deeply about this.  And I'm sure everyone has their favorite / least favorite web server and few people agree.  :)  I imagine writing the code in such a way that you could plug it into other HTTP servers without too much difficulty.

[Andreas Stenius]

2013/9/12 Kenton Varda <temp...@gmail.com>

Yeah, as I said, I have not thought deeply about this.  And I'm sure everyone has their favorite / least favorite web server and few people agree.  :)  I imagine writing the code in such a way that you could plug it into other HTTP servers without too much difficulty.

+1. indeed there's a lot of alternatives out there, all with there own set of dedicated followers. I think it wise to make it as easy as possible to support any and all servers with as little effort as possible.