Incomplete compile-time Zig implementation

[Yorhel]

Hi list,

I've been working on a Cap'n Proto implementation for the Zig language.
This implementation is unusual in that it leverages the language's
compile-time code execution capabilities to read a schema file - written
in Cap'n Proto's schema language, no separate compilation step required
- and then make the interfaces defined in the schema available to
library users.

I wasn't 100% sure if this was going to work, but I now have a parser,
struct layouting algorithm and some rudimentary experiments with a
type-safe struct read/write API that compiles down to code that *should*
be just as efficient as if a code generation step was involved.

Sadly, now that I've convinced myself that this might actually work out,
I've lost the motivation to do the remaining 80% of the work and turn it
into a usable library. So instead I'm publishing the unfinished product
with the hope that it might inspire or prove useful to someone. And, who
knows, maybe I'll continue to work on it some time in the future.

The code is at https://code.blicky.net/yorhel/capnzig

Documentation's kind of lacking - the code is commented here and there,
but making sense of everything may take effort. I'll be around to answer
questions if there are any.

Here's hoping we'll have a mature Cap'n Proto implementation for Zig at
some point, regardless of which code generation strategy is taken.

Yorhel.

[Kenton Varda]

Hi Yorhel,

Cool stuff. It's neat that Zig can run code at compile time.

Question, though: are you calling the C++ parser library to parse Cap'n Proto schema language, or did you write a whole new parser for it? If the latter, I'm worried as the algorithm to decide the layout of structs is quite complex, especially when unions of groups are involved. If you are re-implementing that, it would be pretty easy to get wrong leading to subtle incompatibilities. Is it possible to invoke the parser from the C++ implementation as a library, to convert from capnp IDL into schema.capnp format, which could then be interpreted by your implementation?

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021.

[Kenton Varda]

(Sorry for the very slow reply, my inbox is a mess.)

-Kenton

[Yorhel]

Hi Kenton,

The project comes with a whole new parser and layouting implementation.
I have some tests[1] for complicated struct/union layouts, but you're
absolutely right that this needs much more rigorous testing and an
enormous disclaimer - both of which I had planned had I continued.

It's not possible to invoke external programs or call into other
languages from Zig comptime[2], that would require a separate build
step. One idea I considered is to have the parser recognize the
comments generated by `capnp -o capnp` and use them for validation, so
that it's at least easier to get that extra assurance. Loading a file in
schema.capnp format could also be done, of course, but then one might as
well introduce a separate build step.

1) https://code.blicky.net/yorhel/capnzig/src/commit/5618819e01c8dda38962b986e4107d7d20e863f5/src/schema.zig#L812

2) Okay, Zig's comptime being turing complete one could potentially
write a C++ interpreter and run capnp's layouting code during Zig
compilation, but let's not go there. :)

Yorhel.

> To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com.

[Kenton Varda]

Hmm, it's interesting that you're allowed to read non-zig source files from disk in "comptime" but not allowed to execute external binaries or call into non-zig libraries. If the capnp file contains an import statement, are you allowed to open whatever file it references? If so that sounds like you basically have arbitrary (read-only?) filesystem access?

-Kenton

[Yorhel]

Nah, compile-time filesystem access is very limited. The Zig compiler
only has a single builtin function for it: @embedFile(), and it's
limited to only reading files inside the project directory.
Automatically resolving capnp import statements might be possible in
some cases, but path resolution will be tricky to get right since
@embedFile() resolves relative to the source file calling it. My idea
for a workaround was to have the library user do all the @embedFile()
calls and pass a list of paths and contents to the parser.

> > https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com
> > .

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQnM5Q2Bd-mPdwmsAijNYx2if4AsV2SwL8W4xU2rzUxQew%40mail.gmail.com.