[ANN] github.com/jba/codec, a fast encoder for Go

[Jonathan Amsterdam]

Uses code generation for fast encoding and decoding of Go values to bytes. Handles sharing and cycles too.

https://pkg.go.dev/github.com/jba/codec

[roger peppe]

This is interesting, thanks! Is there a full description of the encoding somewhere? (e.g. how are structs represented? what about interface values, etc? is the schema implicit or sent on the wire?)

  cheers,

    rog.

On Tue, 19 Jan 2021 at 14:59, Jonathan Amsterdam <jbams...@gmail.com> wrote:

Uses code generation for fast encoding and decoding of Go values to bytes. Handles sharing and cycles too.

https://pkg.go.dev/github.com/jba/codec

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2cc71201-ddbf-4524-88ba-7d0875072d80n%40googlegroups.com.

[Jonathan Amsterdam]

The encoding scheme is described briefly in the README[0] and the code[1].

To answer your two specific questions, interfaces are represented as a pair (typeNumber, value) where typeNumber maps to a registered type. (Like gob, types must be registered.) Structs are represented as: startCode (fieldNumber value)* endCode. The field numbers are assigned by the generator.

The schema is implicit. The model is protobufs, where there is just enough information in the wire protocol to skip values, but you need something external to interpret types and struct field numbers.

[0] https://github.com/jba/codec#encoding-scheme

[1] https://github.com/jba/codec/blob/v0.3.1/codecapi/codecapi.go#L200

[roger peppe]

On Wed, 20 Jan 2021 at 13:31, Jonathan Amsterdam <jbams...@gmail.com> wrote:

The encoding scheme is described briefly in the README[0] and the code[1].

To answer your two specific questions, interfaces are represented as a pair (typeNumber, value) where typeNumber maps to a registered type. (Like gob, types must be registered.) Structs are represented as: startCode (fieldNumber value)* endCode. The field numbers are assigned by the generator.

It might be good to be more explicit about how the field numbers are assigned. From a brief experiment, it seems like there's not a deterministic

relationship between a struct and its wire representation, and instead the generated field numbers are taken from the generated code file

when it's present. So ISTM that any user of this must be very careful to preserve that file, and realise that it's not OK to generate

the codec code for a type independently.

I'd also suggest that it would be good to fully document the syntax and explain the trade-offs of this format and when

it might or might not be appropriate to use.

One other question: how are the type numbers maintained as stable entities over time?

  cheers,

    rog.

[Jonathan Amsterdam]

Thanks for the suggestions. I created https://github.com/jba/codec/pull/1 to address them. You can comment in more detail there if you'd like.

[Jonathan Amsterdam]

I changed the encoding so that the previous generated code is no longer necessary. Encoded data now carries all the information needed to decode struct fields correctly, even if fields have been added and reordered.