New Macaroons Binary Format

[Robert Escriva]

Based on previous discussions on and off this list, I've implemented a
new binary format for libmacaroons[1]. The new format is significantly
more compact than the previous format, but is not ascii-safe.

The current implementation in libmacaroons's git repository supports
serializing and deserializing both the old and new formats, and I fully
intend to maintain compatibility with the old format and the finalized
form of the new format.

To help facilitate cross-language implementations of macaroons, I've
also started to assemble some test cases for both the verification and
serialization aspects of a macaroons implementation[2]. My goal here
was to write simple-to-parse text files that make it easy to test the
verifier code path with an established context and a set of macaroons.
There are also some tests for verifying the various serialized forms of
macaroons that show the same macaroon in multiple formats.

I'm writing this post to solicit feedback on the format and
implementation, and will finalize it in a new release of libmacaroons
one week after discussion in this thread goes cold. The release after
that will include support for the JSON format described in [1].

-Robert

[1] https://github.com/rescrv/libmacaroons/blob/master/doc/format.txt
[2] https://github.com/rescrv/libmacaroons/blob/master/doc/tests.txt

[Tony Arcieri]

What's the best way to refer to the old and new formats? v1 and v2? (I notice the version field is set to 2 in the new format)

--
You received this message because you are subscribed to the Google Groups "Macaroons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to macaroons+...@googlegroups.com.
To post to this group, send email to maca...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/macaroons/20160727170523.GA21295%40rave.rescrv.net.
For more options, visit https://groups.google.com/d/optout.

--

Tony Arcieri

[Robert Escriva]

Yes. The packet based format in the original release is designated v1.
The new format is designated as v2. The JSON format will be designated
v2j. In the code, you can freely select between them[1].

-Robert

[1] https://github.com/rescrv/libmacaroons/blob/be76d41d9e4d1e93af34b550a1f41074061efcf1/macaroons.h#L180-L185

> CAHOTMVKuYudrR9tX44_1cZo7jgGLp6b25Q3WXLmPH%2BU7YHiENg%40mail.gmail.com.

[Jonathan Rudenberg]

> On Jul 27, 2016, at 13:05, Robert Escriva <rob...@rescrv.net> wrote:
>
> I'm writing this post to solicit feedback on the format and
> implementation, and will finalize it in a new release of libmacaroons
> one week after discussion in this thread goes cold. The release after
> that will include support for the JSON format described in [1].

The binary format looks great!

Is there a reason why the JSON format supports values encoded in UTF-8, hex, regular base64, and URL-safe base64?

Those are all standard encodings, but I’m not convinced that they are all necessary. It seems like UTF-8 and one form of base64 should be sufficient. From a decoding perspective, every additional encoding is more work for the implementor, and I’m not sure what the reason for that complexity is.

Jonathan

[Martin W. Kirst]

Hi,

I don't want to be the devils advocate,

but out of curiosity – may I ask, what's the motivation/purpose behind a new binary format? It adds more complexity in terms of (de-)serializing and reduces general compatibility among multiple libraries/implementations.

Or in other words, why other macaroons implementation author's should support the new format?

Regards

Martin

--
You received this message because you are subscribed to the Google Groups "Macaroons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to macaroons+...@googlegroups.com.
To post to this group, send email to maca...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/macaroons/40CC4212-A0CA-4190-A7E2-518687589181%40titanous.com.

[Robert Escriva]

I think the motivation was to cover our bases, but you're right that
only one is necessary. I would probably go for both regular base64 and
URL-safe base64 because their availability differs, and it's easy to
convert a string from one to the other should one implementation be
unavailable. Hex is unnecessary we can drop it easily as JSON is not
yet implemented.

-Robert

[Robert Escriva]

On Wed, Jul 27, 2016 at 08:33:09PM +0200, Martin W. Kirst wrote:
> Hi,
>
> I don't want to be the devils advocate,
> but out of curiosity – may I ask, what's the motivation/purpose behind a new
> binary format? It adds more complexity in terms of (de-)serializing and reduces
> general compatibility among multiple libraries/implementations.
> Or in other words, why other macaroons implementation author's should support
> the new format?
>
> Regards
> Martin

Having implemented both the old and new formats, I think the newer
format is easier to implement once you have a correct varint
implementation. This is probably a language-specific observation.

The motivation for the new format was that the v1 format is quite
verbose and inflates the size of macaroons. In testing, the new format
can easily be 50% smaller than the old, which makes it much easier to
fit macaroons into constrained spaces, such as a 4093 bytes available
for cookies in an HTTP request.

The new format was a collaboration between myself and maintainers of
other macaroons libraries. I believe it was inspired by a binary format
that's used in production (but don't quote me on it; I would like for
someone to chime in and confirm that my memory on this is accurate).

-Robert

[roger peppe]

I'd agree with dropping the hex format. I have implemented the new binary
and JSON formats (see https://godoc.org/gopkg.in/macaroon.v2-unstable),
and although my code accepts the hex variant, it
never generates it.

[roger peppe]

The new binary format wasn't inspired by anything that I've seen in production.
It was just as simple, regular and as compact a format as I could think of
that fitted in with the existing macaroon fields and structure.
Scribbled down half way through a transatlantic flight to San Francisco.

I guess the nearest inspiration was protobufs, just for the
varint format.

cheers,
rog.

[Robert Escriva]

I've updated the code and test cases and am once again looking for
testers. The binary format was changed to remove a spurious separator.
I also removed the "hex" encoding from the specification as base64 is
sufficient for saving binary data.

I've also implemented preliminary JSON support for serializing and
deserializing. It has known shortcomings when it comes to binary data
and unicode conversions, so it is disabled by default.

You can test the current code on your platform with:

wget http://rescrv.net/libmacaroons-0.4.dev.tar.gz
tar xzf libmacaroons-0.4.dev.tar.gz
cd libmacaroons-0.4.dev
./configure --enable-json-support-that-is-not-production-ready
make
make check

If there's no negative feedback by Thursday, I'll release this code[1]
as libmacaroons 0.4.0. It is fully backwards compatible with previous
versions, but introduces the new formats. The binary format should be
preferred for most environments, while the JSON format (once implemented
fully) will be recommended for browser-based macaroons applications.

-Robert

[1] commit cc3b0f9f605aa7b6b118323d12431a4f35fa7bd0
tarball sha256:
0e8a1f888ea4570be8afcd0a5687e811d047daf97b2186dcb231abc4af1fcd60

[Robert Escriva]

On Sat, Aug 06, 2016 at 10:33:48AM -0400, Robert Escriva wrote:
> If there's no negative feedback by Thursday, I'll release this code[1]
> as libmacaroons 0.4.0. It is fully backwards compatible with previous
> versions, but introduces the new formats. The binary format should be
> preferred for most environments, while the JSON format (once implemented
> fully) will be recommended for browser-based macaroons applications.

I should clarify that this is *format* compatibility. The library has
some slight API changes and will thus have an ABI bump.

-Robert

[m...@bitnami.com]

On Thursday, July 28, 2016 at 1:16:02 PM UTC+2, roger peppe wrote:

> I guess the nearest inspiration was protobufs, just for the 

varint format. 

Hi!

quick question: why not just use protobuf then?

[Robert Escriva]

Being compact by virtue of varint is the biggest similarity to protobuf.
Protobuf is much more flexible and has a different goal (general
serializability/deserializability) than libmacaroons.

The libmacaroons binary format is simple and easy to parse. The
implementation is 1057 lines of C including comments, whitespace, and
bracket-only lines. And that includes the (incomplete) JSON library.

-Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Macaroons" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to macaroons+...@googlegroups.com.
> To post to this group, send email to maca...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> macaroons/a9221b3d-c3b5-451e-80f9-47ae7acc7c2e%40googlegroups.com.

[Evan Cordell]

Also fwiw, I did a quick implementation of protobuf serialization a while ago:

https://github.com/ecordell/pymacaroons/blob/protobuf/pymacaroons/serializers/protobuf_serializer.py

I was interested to see if serialization performance would improve, but some very unscientific microbenchmarks seemed to show they were no faster (there are probably cases where they are).

[m...@bitnami.com]

> Protobuf is much more flexible and has a different goal (general 
serializability/deserializability) than libmacaroons. 

well, the same can be said about JSON :-)

is up to you to pick a between a full-fledged library that can speak a protocol/format, generate mapping to/from your language types

or use a minimal implementation of a codec that can only deal with your specific format (e.g. for the sake of reducing the size of your dependencies).

The nice thing about protobuf is that there is an IDL you can use to document your format. E.g.:

syntax = "proto3";

message Macaroon {

    int32 version = 1;

    bytes location = 2;

    bytes id = 3;

    repeated Caveat caveats = 4;

    bytes signature = 5;

}

message Caveat {

    bytes location = 1;

    bytes id = 2;

    bytes vid = 3;

}

This schema defines a precise and unambiguous binary encoding (It also allows you to pass that schema to a code generator and supporting library,

but that's another story).

It also offers a compact and practical way to perform backwards compatible extensions of the protocol.

Your encoding use a EOS separator between caveats, protobuf would prefix each caveat with a varint size. Apart from that and some minor

differences with which type tags are encoded there is no big difference.

Just in case you wonder: the protobuf encoding rules are not just an implementation details of the protobuf library, that is subject to arbitrary changes between releases.

This means it shouldn't be too hard to actually implement an encoder and a decoder that will follow the same rules and be devoid of any unwanted dependency

to big and complex projects (which is a concern I do understand). It's definitely way less code that what I see you have to encode JSON!

The reason I would personally prefer this approach is that if you describe the encoding with a protobuf IDL and match its encoding, anybody

can easily write a macaroon library in any language (ruby, js, $your_favourite_unpopular_language_here). (I'm talking about native impls, where just binding with the C library is impossible or undesirable).

Perhaps, I'm biased in favour of protobuf out of familiarity, perhaps there is a better widely used and understood binary encoding, with support for a wide range of languages out there and yet simple to implement (e.g. not, ehm ehm ASN.1).

Cheers,

Marko

PS: For a compact JSON library you can take a look at a library I co-authored: https://docs.cesanta.com/frozen/master/; it also deals with string escaping. Note thee %V formatter is undocumented: it encodes the argument with base64. We designed it for embedded (as in <2K ram) use cases so it's very small and uses as little memory as possible.