Reviving the JavaScript implementation

[Julián Díaz]

For better or worse my curiosity has led me to try reviving the JavaScript implementation of Cap'n Proto; at least all the way to level 1 RPC in native JavaScript.

Looking at the current two implementations they don't really feel viable to hack on - one is using AMD for the browser (we've moved on!) and the other is clearly a mess as-is.

After scratching the surface a good bit, my initial goals for the library are:

- Full level-1 RPC support

- Extensive unit test coverage

- Full browser and nodejs compatibility

- Pure ES5 implementation (for speed)

- Low-to-zero library dependencies (lodash might become a must-have)

- Loading schema files directly

Stretch goals:

- WebSocket transport support

- Precompiled schema files

- Web worker support

- Level 2+ using WebRTC (can it be done!?)

- First-class support for bluebird promises

Would love some feedback from this list to hear if there's appetite for this, and any features that would make this super useful. I will initially aim to provide a similar API as the reference C++ implementation but I may take extreme measures to cater more to the target language and typical audience. I personally tend to prefer stateless, immutable APIs and will try to support that here as well.

- Julián

[Kenton Varda]

Hi Julián,

This is exciting! In Sandstorm we have a lot of use cases for speaking Cap'n Proto all the way from the browser, but we have not been able to find time to work on JS bindings. We would also like to ditch the V8-specific implementation we use server-side in favor of a pure-JS implementation.

Some thoughts:

- You list WebSocket as a stretch goal, but how do you imagine implementing RPC without WebSocket? It seems to me that this (or a WebSocket emulation layer like sock.js) is a necessity given the symmetric nature of Cap'n Proto RPC.

- For Sandstorm, being able to use postMessage() as a transport would be extremely useful. Of course, combining WebSocket, postMessage, WebRTC, and server<->server TCP, all with 3-party handoff (level 3), would be the holy grail... but tricky. :)

- Hmm, what are the "speed" reasons to use pure ES5? I'm not familiar with this.

- Have you considered using TypeScript or Flow? Cap'n Proto follows a strongly-typed philosophy, and although Sandstorm does not use strongly-typed JS today, everyone on the team wants to move towards it in the future.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[Nathan Hourt]

As far as the appetite front goes, I've rather wanted a pure javascript implementation of capnp on several occasions now, so I'd be quite happy to see this happen.

I think websockets support is a must, but ideally the library would be sufficiently portable that I would be able to use the library, unmodified, on any particular transport layer I want, with nothing more than a little glue code between interfaces. The library could then provide some built-in adaptors for common transport layers (like Node sockets, websockets, etc) which would allow those transports to be used with no customization, and also documents how to write adaptors so it's easy to see how to use other transports instead (exactly like what we see with the C++ implementation with AsyncIoStream: I can use any transport I want simply by implementing the interface).

[julia...@onefinancialholdings.com]

After playing around a little and realizing how limited ES5 really is, I'm open to writing this thing in 100% TypeScript. There's some caveats, though - if the schema file is loaded dynamically it'll be hard (impossible?) to add type safety without runtime type checks, and I can see that getting expensive. TypeScript almost gets in the way in this case. However, a schema compiler can be set up to emit either typescript or vanilla ES5 code; that could work quite well! ArrayBuffers will contain the raw data and property access/method calls will just reference indices in the array - the only super annoying thing about it is there's no native support for 64-bit words in JavaScript so everything has to be broken up into hi/lo 32-bit words in a Uint32Array (yuck!).

There are concerns about TypeScript being generally slower if you lean heavily on the ES6 features (https://kpdecker.github.io/six-speed/), but with proper profiling it shouldn't be hard to identify hotspots and turn them into tightly optimized code that won't be modified much by the transpiler. The speed problems come in when you heavily depend on things like arrow functions and the transpiler ends up adding extra assignments and variables you didn't expect. For what it's worth, I expect most performance concerns won't have anything to do with using ES6.

I agree that the transport layer needs to be pluggable and that's the design I'm going to go for right from the start. Realistically I'm going to have a hard time designing the whole thing without a sample app to reference so some form of transport needs to be supported "out of the box". WebSockets feels like an easy initial target for this. Supporting window.postMessage should also be easy, and since you can pass ArrayBuffers with zero-copy (I think??) it'll be crazy fast!

Speaking of sample app; I plan on writing a TodoMVC type thing in separate repos as a reference implementation (a node server and a browser client w/ React). Any thoughts on this? Should I instead try to go for something that better showcases the power of capability-based access control? A todo list where you can securely share edit/view access with others could be a fun demo. I want to keep the UI complexity low and really focus on the RPC stuff, of course.

Could also use tips on which order to implement things since this is a bit of a daunting spec to implement. I'm currently having fun with the unpacking algorithm. :)

[Kenton Varda]

For Javascript, I actually don't think runtime schema loading should be a goal. These days Javascript is essentially a compiled language anyway, with all the transpiling and minifying. And to load schemas at runtime in a browser, you'd presumably need to rewrite the parser in pure-Javascript. I'd like to avoid having multiple schema parser implementations as they'd be likely to have subtle incompatibilities that would be a pain for developers.

So, yes, capnp -> TypeScript compile-time codegen sounds like a pretty good plan. Then we can get auto-complete in our IDEs and all that cool stuff. :)

-Kenton

[Julián Díaz]

Generated TypeScript it'll be, then! It'll be pretty trivial to subsequently invoke the TypeScript compiler on the TypeScript generated code to get "free" ES5 code for everyone else.

I'm really digging how much of a great fit TypeScript's type system is for this stuff (so far, at least). Will report back once I have something that resembles a code generator.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

[Wink Saville]

Julian, What's the status of this?

[julia...@onefinancialholdings.com]

It is stalled, I'm afraid. Not quite in a condition where someone else could pick it up, either. This is mostly due to lack of time with work obligations, though that's on track to change soon.

Truthfully, I may need to nuke-and-pave the current repository and start over with ES5 JavaScript. Translating the C++ code more or less verbatim has proven extraordinarily difficult for three reasons: pointer arithmetic (I managed to create nice helpers to deal with this), C++11's robust templating and type system which is used heavily in the capnp source (TypeScript cannot come close), and pass-by-reference function arguments which are mutated by the method itself (JavaScript cannot do this). I totally understand now why the other two JS libs have been basically abandoned.

I'll use ES5 directly for the second pass rather than ES6/2016/etc since I'll have much tighter control over the performance characteristics that way. I will also take more of a top-down approach and loosely follow the C++ API, only taking bits of inspiration from the original source when appropriate. This is easier now that I have a deeper understanding of the internals.

Switching the library to ES5 isn't a death sentence for TypeScript support. I can manually generate type definitions for the library and even potentially have the code generator output full typescript or type definition files as well.

In any case, this project hasn't been forgotten!

[Wink Saville]

:(

I'm new to this Javascript/Typescript/Web programming, but I find comfort in the quasi strict types of Typescript. I'm surprised Typescript is getting in the way, especially since Typescript is a "proper" superset of Javascript. I can certainly see why translating the C++ to Typescript would be alluring, but I can imagine its neigh on impossible given what you've said.

I wonder if there might be another approach, WebAssembly/wasm. I find this extremely intriguing and maybe there is a way to use the C++ code compiled to wasm, no idea if that is possible or practical, but just a thought. Of course its very early days for wasm, but I'm following it and just beginning to play around with it. The WebAssembly Minimally Viable Product (MVP) is available on the latest Chrome and Firefox browsers. And Node v8 will support it when its released around the end of the month, so you can actually play around with "released" code.

[Ian Denhardt]

Quoting Wink Saville (2017-04-07 23:31:43)

> Of course its very early days for wasm, but I'm following it and just
> beginning to play around with it.

Worth pointing out, there's also Emscripten, which has been around a
while. So that could be used while wasm support filters into things.

Has anyone tried running capnproto through Emscripten? Might be nice to
try plumbing the rust implementation through it as well.

Along similar lines, there's a mature Go -> Javascript compiler
(gopherjs), and a pure-go implementation of capnproto. This might "just
work," and if not I strongly suspect the fixes will be easy.

[Wink Saville]

> Of course its very early days for wasm, but I'm following it and just
> beginning to play around with it.

Worth pointing out, there's also Emscripten, which has been around a
while. So that could be used while wasm support filters into things. 

Yep, Emscripten is the path I was thinking, it now only output asmjs but

also wasm.

[Ross Light]

I have had somebody use GopherJS to read data client side. It works, but I can't vouch for how well.

What parts are you having difficulty with representing in JS? Are you still on serialization or the RPC part? I found that for Go I would look at the C++ implementation to get the spirit of a feature, and then write it very differently to fit in with Go.

--

[Julián Díaz]

The serialization part is where I fell on my face; I was definitely following the C++ implementation a bit too closely.

There's a huge amount of type polymorphism needed and that was tripping me up pretty hard as well; I'm going to instead actually take advantage of the fact that JS is a dynamic language and that should help cut down on the amount of code needed.

WASM is definitely an appealing thought and probably the ideal way to use capnp in the browser, but the way I see it a pure JS implementation will continue to be appealing for as long as we have other non-desktop devices with JS but not WASM support (I'm looking at Android 4.x, especially - that's going to be around for a while).

[m...@bradstewart.co]

Anything ever happen with this? 

[Kenton Varda]

On Wed, Apr 5, 2017 at 9:01 PM, <m...@bradstewart.co> wrote:

Anything ever happen with this? 

(Moderator's note: This message was actually sent four days ago, before the recent activity on this thread, but got caught in the moderation queue.)

[Kenton Varda]

I don't think WASM is likely to be a good fit here, for two reasons:

1) libcapnp and libkj together add up to some 730k of code (text segment) these days. Unless emscripten builds are significantly smaller, that's probably too big. (If you stick to lite mode, it's still 401k, which is still probably too big.)

2) The interface between JS and WASM would probably be very slow if every accessor has to go through it. You'd probably end up wanting to translate the whole capnp to a JSON object upfront, which of course defeats a lot of the purpose.

So I definitely think a pure-JS implementation is still desirable.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.

[Wink Saville]

On Sunday, April 9, 2017 at 3:36:11 PM UTC-7, Kenton Varda wrote:

I don't think WASM is likely to be a good fit here, for two reasons:

1) libcapnp and libkj together add up to some 730k of code (text segment) these days. Unless emscripten builds are significantly smaller, that's probably too big. (If you stick to lite mode, it's still 401k, which is still probably too big.)

For some use cases it might not matter.

 

2) The interface between JS and WASM would probably be very slow if every accessor has to go through it. You'd probably end up wanting to translate the whole capnp to a JSON object upfront, which of course defeats a lot of the purpose.

I'm hoping shared memory might be used, we could have serdes in wasm and access in JS, take with huge grain of salt :)

 

So I definitely think a pure-JS implementation is still desirable.

Yea, definitely wouldn't hurt.

[Kenton Varda]

On Sun, Apr 9, 2017 at 7:04 PM, Wink Saville <wi...@saville.com> wrote:

2) The interface between JS and WASM would probably be very slow if every accessor has to go through it. You'd probably end up wanting to translate the whole capnp to a JSON object upfront, which of course defeats a lot of the purpose.

I'm hoping shared memory might be used, we could have serdes in wasm and access in JS, take with huge grain of salt :)

But in Cap'n Proto, there is no serdes. All of the logic happens at access time.

-Kenton

[Mark Miller]

"serdes"?

--

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

--

  Cheers,
  --MarkM

[Wink Saville]

serial/deserializer

On Sunday, April 9, 2017 at 8:30:51 PM UTC-7, Mark Miller wrote:

"serdes"?

On Sun, Apr 9, 2017 at 10:22 PM, Kenton Varda <ken...@sandstorm.io> wrote:

On Sun, Apr 9, 2017 at 7:04 PM, Wink Saville <wi...@saville.com> wrote:

2) The interface between JS and WASM would probably be very slow if every accessor has to go through it. You'd probably end up wanting to translate the whole capnp to a JSON object upfront, which of course defeats a lot of the purpose.

I'm hoping shared memory might be used, we could have serdes in wasm and access in JS, take with huge grain of salt :)

But in Cap'n Proto, there is no serdes. All of the logic happens at access time.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

Visit this group at https://groups.google.com/group/capnproto.

--

  Cheers,
  --MarkM

[Wink Saville]

Hmmm, I forgot, thinking of my old protobuf days. Anyway, one of the use cases is using is using messaging to communicate between wasm and js so if we can't share any implementation so be it.

 

-Kenton

[Ian Denhardt]

Quoting Kenton Varda (2017-04-09 18:35:48)

> 1) libcapnp and libkj together add up to some 730k of code (text
> segment) these days. Unless emscripten builds are significantly
> smaller, that's probably too big.

Hard to know without trying it, but it may well be the case that wasm
builds will be smaller. The VM seems to be designed for small code size
(sensibly, given its target use case). This obviously doesn't apply to
the asm.js output.

That said, I agree having a pure JS implementation is preferable.

[Julián Díaz]

I'm happy to report some real progress!

https://github.com/jdiaz5513/capnp-ts

Right now it's not very useful at all (I just barely have serialization working) but it's a solid starting point to wrap up the serialization API. The peanut gallery can start poking around to see how I organized things – it does depart slightly from the reference implementation but I'm still aiming to make an external API that's very similar to the C++ one.

Once the Struct/List classes are complete I'll move on to the schema compiler, which looks like it'll be a cinch. Hoping I can keep up the steady progress from here.

[Harris Hancock]

Just wanted to say this is awesome, and thanks for sharing it! I took it for a test drive (all tests pass), and look forward to banging on it more soon.

--

[Kenton Varda]

Sweet!

Totally random comment from totally randomly opening a file and looking at it:

I see lists are accessed via a method .get(n). Have you considered using proxies to allow array subscript [] syntax? I guess some 10-20% of browsers still don't support proxies but that number will only go down.

-Kenton

--

[Mark Miller]

https://kangax.github.io/compat-table/es6/

It looks like proxies are supported everywhere.

--

  Cheers,
  --MarkM

[Kenton Varda]

On Wed, May 10, 2017 at 11:17 PM, Mark Miller <eri...@gmail.com> wrote:

https://kangax.github.io/compat-table/es6/

It looks like proxies are supported everywhere.

Unfortunately a lot of people still use old browsers.

http://caniuse.com/#feat=proxy -- click on the "usage relative" box.

-Kenton

[Julián Díaz]

Can't use proxies at all in TypeScript unless I set the target to ES6 - right now I want to keep it compiling to ES5 so it's immediately useful for a wider range of people.

It also seems like it's going to perform like crap: http://thecodebarbarian.com/thoughts-on-es6-proxies-performance.html

I'll add it to the TODO, regardless; a separate ES6 build would be useful for many people. I could document it with the caveat that .get() will always be faster.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

Visit this group at https://groups.google.com/group/capnproto.

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

Visit this group at https://groups.google.com/group/capnproto.

--

  Cheers,
  --MarkM

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

[Julián Díaz]

For those itching to get on the bleeding edge, I've got a working schema compiler now!

Serialization seems to be working okay with some unimplemented edges here and there. Perhaps not surprisingly, I'm already seeing places where this can outperform JSON.parse, so that's a major win!

I almost had compile-to-JS support working as well, but the TypeScript compiler is refusing to play nice with me right now. (See: https://github.com/jdiaz5513/capnp-ts/issues/5)

If anyone is really interested in using this stuff today please reach out to me so I can better understand what you need and perhaps rearrange how I implement things. Otherwise, there's still lots to do before 1.0.0!

PS: For the compiler nerds: the schema compiler actually uses the TypeScript compiler API directly to build an AST before printing it to a file.

[Kenton Varda]

Sweet!

On Thu, Jun 8, 2017 at 7:01 PM, Julián Díaz <jdia...@gmail.com> wrote:

If anyone is really interested in using this stuff today please reach out to me so I can better understand what you need and perhaps rearrange how I implement things. Otherwise, there's still lots to do before 1.0.0!

I can wait until its more ready, but once it is I'll be eager to try it out in Sandstorm.

It would be amazing if we could retire node-capnp which Sandstorm uses currently. It leaks a lot of memory due to inability to GC through C++ objects and the V8 C++ API being hard to use correctly in general. Of course, we'll need RPC before this can happen.

PS: For the compiler nerds: the schema compiler actually uses the TypeScript compiler API directly to build an AST before printing it to a file.

Nice!

-Kenton

 

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.

[Tatsuyuki Ishi]

I wonder if there's something I can help out. I'm interested in building (web) applications with cpnp and would like to do some contribution if possible.

I would also appreciate tracking the work more precisely with GitHub issues.

2016年10月1日土曜日 9時08分01秒 UTC+9 Julián Díaz:

[Julián Díaz]

@Tatsuyuki-san I hear you; now that I'm recently (f)unemployed I've got time to really dedicate to this project now.

I'm going to clean up the roadmap today and start making granular pushes toward finishing the test coverage. Here's a brain dump of where it stands now:

1. There are bugs lurking in the serialization implementation, so top priority is 100% test coverage for serialization and the compiler before working on anything else.
2. Implement toJSON() and fromJSON().
3. Publish to NPM as 0.1.0.
4. Flesh out the RPC classes.
5. Implement a proof-of-concept websocket RPC transport (with simple demo application).

As I start seeing small tasks that can be worked on independently I'll pile on some issues in the github tracker for others to pick off.

PS: If anyone reading this needs an A+ full-stack engineer I'm available for contract/salary work. ;)

- Julián

[Kenton Varda]

So I looked at this today and I'm pretty impressed! Code looks clean and seems to be following best practices. I made some notes on the issue tracker, as you probably saw, but generally looks pretty good. I'm pretty excited to start using this -- and I really want to migrate Sandstorm to TypeScript.

One question: Have you written tests using the test data in the capnp repo?

https://github.com/capnproto/capnproto/tree/master/c++/src/capnp/testdata

This would help check for any misreads of the spec.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.

[Julián Díaz]

Appreciate the endorsement!

I did in fact borrow some of that test data directly, though there's an interesting divergence in the TypeScript version of the packing algorithm so I wound up editing segmented-packed by hand to match: https://github.com/jdiaz5513/capnp-ts/pull/10.

Still making slow but steady progress writing tests and finding broken stuff. I'll publish to npm once the serialization part isn't so... broken.

- Julián

[Kenton Varda]

On Tue, Jul 18, 2017 at 9:00 AM, Julián Díaz <jdia...@gmail.com> wrote:

Appreciate the endorsement!

I did in fact borrow some of that test data directly, though there's an interesting divergence in the TypeScript version of the packing algorithm so I wound up editing segmented-packed by hand to match: https://github.com/jdiaz5513/capnp-ts/pull/10.

Hmm. I suspect this is because the C++ implementation packs each segment separately, and so the two runs of zeros came before and after a segment boundary. Unfortunately, assuming my suspicion is correct, then I suspect the C++ implementation will not be able to unpack your version. This is because it unpacks each segment into a separate array, but for speed reasons the core unpacking loop targets a single array at a time and has no way to pass along its state to the next call. I suppose this needs to be documented.

You can check if this is the case by using `capnp decode -p` on your version of the data. I suspect it will throw an error.

If it doesn't throw an error, then I'm confused.

-Kenton

 

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.

[Julián Díaz]

And this is exactly why I brought that up; had a feeling I might have done something fishy. :)

It does throw; shouldn't be too hard to modify mine to pack segment by segment. Thanks!