Thinking about zig implementation

[qu...@daurnimator.com]

As https://capnproto.org/otherlang.html says:

> If you’d like to own the implementation of Cap’n Proto in some particular language, let us know!

I'm looking into playing with capnp and zig (https://ziglang.org/).

Zig is a relatively new language that aims to one day replace C.

Principles include:
  - No hidden control flow (which means that there can be e.g. no hidden allocations)

  - Unified compiletime and runtime syntax and semantics: no separate language for compile like you have with the C preprocessor or C++ templates

  - Extensive self-reflection capabilities

  - Easy cross-compilation for non-native targets

As an ahead-of-time compiled languages, it seems like the recommended path forward would be to implement a `capnp` plugin.

Daurn

[Ian Denhardt]

Zig is neat! Yes, you'll want to write a schema compiler plugin, and
will also need to write some code to handle non-schema-specific details
of the format; I would start by reading the encoding docs and build some
support for reading "untyped" structs/lists/etc.

You'll want to decide what language to write the schema compiler plugin
in. Using something other than Zig might be easier since that way you
don't have to deal with the bootstrapping issues; If you choose zig,
you'll have to write wrappers for the types in schema.capnp by hand in
order to bootstrap.

If you have more specific questions, I'll gladly field them. Keep us
posted.

-Ian

Quoting qu...@daurnimator.com (2019-11-29 05:44:58)

> As https://capnproto.org/otherlang.html says:
> > If you�d like to own the implementation of Cap�n Proto in some

> particular language, [1]let us know!

> I'm looking into playing with capnp and zig (https://ziglang.org/).
> Zig is a relatively new language that aims to one day replace C.
> Principles include:
> � - No hidden control flow (which means that there can be e.g. no
> hidden allocations)
> � - Unified compiletime and runtime syntax and semantics: no separate
> language for compile like you have with the C preprocessor or C++
> templates
> � - Extensive self-reflection capabilities
> � - Easy cross-compilation for non-native targets
> As an ahead-of-time compiled languages, it seems like the recommended
> path forward would be to implement a `capnp` plugin.
> Daurn
>

> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [2]capnproto+...@googlegroups.com.
> To view this discussion on the web visit
> [3]https://groups.google.com/d/msgid/capnproto/f6c9d668-d1a0-4415-8366-
> b49cd1cbe36b%40googlegroups.com.
>
> Verweise
>
> 1. https://groups.google.com/group/capnproto
> 2. mailto:capnproto+...@googlegroups.com
> 3. https://groups.google.com/d/msgid/capnproto/f6c9d668-d1a0-4415-8366-b49cd1cbe36b%40googlegroups.com?utm_medium=email&utm_source=footer

[Siva Mahadevan]

Is there any progress on a Zig implementation? I'd like to start contributing to one if there is already one that exists.

[Daurnimator]

On Mon, 15 Nov 2021 at 05:47, Siva Mahadevan <svm...@gmail.com> wrote:
> Is there any progress on a Zig implementation? I'd like to start contributing to one if there is already one that exists.

No, I didn't get any further than playing with it for a day or two.

Some misc notes:

- I recall issues with using zig structs, as there is no way to e.g.
little-endian integer type
- Zig doesn't have anonymous fields
- Things would be nicer with https://github.com/ziglang/zig/issues/6478

I hit a bit of a dead end, and a future implementation should probably
have getters/setters for fields.
Where the getter/setter key is an enum.

[Ian Denhardt]

Quoting Daurnimator (2021-11-14 19:00:47)

> Some misc notes:
>
> - I recall issues with using zig structs, as there is no way to e.g.
> little-endian integer type
> - Zig doesn't have anonymous fields
> - Things would be nicer with https://github.com/ziglang/zig/issues/6478

Yeah, I would not try to map capnproto structs directly to
language-native structs, except as part of an up-front parsing API for
performance-insensitive use. The normal thing is to just define wrapper
types with accessors. Note that even with control over layout, a
struct's data section can be shorter or longer than what the compiled
schema suggests, so you still end up needing to do a bounds check.

-Ian

[Hugo Rens]

Hello everyone, I've been interested in a zig plugin for a few weeks now, too : I gave it a try and the prototype is available here [1].

It is clearly missing a lot of features, but the general structure is there and I think that by now it is interesting (at least for the purpose of discussion !).

One can find :

- The plugin, of course, implemented in (ugly but easily modifiable) C++ in `src/plugin/`, that generates zig files from generation requests.

- A capnp zig module in `src/` and `src/capnp/ that contains the runtime code. Again, it is partial, but a lot of things are there already :

    - Writing of structures, groups, enums, primitive types and lists
    - Reading of structures, primitive types and lists.

    - Compressor and Decompressor, implemented as Zig's reader and writer (there are tests for those, in tests.zig).

- Use-cases, in `examples/`, which I find interesting as they demonstrate how a user can use a StreamSource (for reading) and an Allocator (for writing).

- I think the best place to start exploring the code, if need be, is `examples/emitter.zig`. It writes a file that respects the primitive.capnp schema. (The unionmess.capnp one is still WIP).

Again, I don't know if any other implementation exist, but I'll continue to work on this version (not much in the next few months, though).
Still I'm totally interested in the discussion that might follow :)
For instance :

- How can one manage groups correctly ?

- Is the reader/writer approach for decompression a good idea or a bad idea ? I like the fact that it is quite composable with other kind of readers/writers.

- What do you think of the View/Stencil notation instead of Reader/Builder ? (which I find clearly nicer personally, but the downside of changing names is an obvious limitation).

About the remarks on a direct mapping to zig structs : I didn't try this approach ; my prototype is much more conventional in this regard, and you will simply find getters and setters (without keys as enums).

Hugo

[1] https://gitlab.com/Zarzwick/capnproto-zig-alternative

PS : Hopefully I'm not necrobumping anything by replying here (otherwise I'm sorry). I saw this thread before going into my own implementation but still wanted to try things out before diving in the discussion.
PS : The ROADMAP.txt is not exactly up-to-date.

[Hugo Rens]

Hello everyone (nearly one year later :D), this mail to say I've been able to work more than usual on my implementation of a zig plugin, and it's taking shape. It is not usable yet, but I think the architecture of the project is now viable w.r.t the missing features, and what exists is generally mirrored in tests (which are now much more clean). It has moved to [1] which is a mirror of some fossil repository of mine (which I prefer as long as I'm the only contributor).

Regarding points that I would discuss (even though I realize they require some level of comprehension of the code) :

• Right now the serialization and de-serialization parts are completely separated, and objects in the serialization part have both a View (a.k.a Reader) and a Stencil (a.k.a Builder). There will inevitably be redundancy when writing the reading part (which I removed in the course of the year). The separation is motivated by very different underlying memory objects when reading and writing. It is inelegant, but in some way I'm not sure the extra complexity that would be implied is worth it. A feature that would make this necessary would be the ability to open one message A in read mode and one B in write mode, and do b.setSomeNonPrimitiveThing(a.getSomeNonPrimitiveThing()). I haven't checked yet if the C++ api is capable of this, though, but regardless of this it would fairly nice.
  
• I'm also looking forward to comments on zig details, regarding my currently massive use of @This(), and @sizeOf, which is apparently runtime, in the generated code.
• I feel comfortable with the plugin (or compiler) in C++, but it is sometimes weird because of the two layers of the capnp library (the generated layer and the "convenience" API which I use only rarely). If someone on this mailing list is familiar with the library and wants to add criticism I'm equally all ears :)

Hugo

[1] https://gitlab.com/Zarzwick/capnproto-zig