piqic pure erlang

[Dave Cottlehuber]

> However, once piqic-erlang is rewritten in Erlang, adding support for

Hi Anton,

How far away do you think this might be? Do you anticipate much
changes to the generated APIs from the current approach?

I want to integrate
https://github.com/aphyr/riemann-java-client/blob/master/src/main/proto/riemann/proto.proto
into some Erlang code, and I'd like to use piqi as my understanding is
that I'll be able to provide a pure erlang implementation, without
needing to install protocol buffer tools on e.g. Windows boxes.

A+
Dave

[Anton Lavrik]

On Fri, Feb 8, 2013 at 4:15 AM, Dave Cottlehuber <d...@jsonified.com> wrote:
>> However, once piqic-erlang is rewritten in Erlang, adding support for
>
> Hi Anton,
>
> How far away do you think this might be? Do you anticipate much
> changes to the generated APIs from the current approach?

Hi Dave,

It will probably take a month or so given my current schedule.

> I want to integrate
> https://github.com/aphyr/riemann-java-client/blob/master/src/main/proto/riemann/proto.proto
> into some Erlang code, and I'd like to use piqi as my understanding is
> that I'll be able to provide a pure erlang implementation, without
> needing to install protocol buffer tools on e.g. Windows boxes.

Working with .proto files and Protocol Buffers from Erlang using Piqi
involves exactly 3 steps:

1. convert proto.proto to proto.proto.piqi by running "piqi to-proto
proto.proto"
2. generate Erlang serializer and deserializer by running
"piqic-erlang proto.proto.piqi"
3. use the generated Erlang modules in your application to work with
Protocol Buffers

Step 1 uses "protoc" and "piqi" executables. It is not going to change.

Step 2 relies on "piqic" executable. After "piqic-erlang" is rewritten
in Erlang, there will not be any visible changes in the interface. The
only difference is that "piqic" will go away and "piqi" executable
will be used instead.

Step 3 doesn't require any of "protoc" and piqi executables even today.

This means that you can have a pure Erlang implementation on Windows
today if you either build your application on Linux or include
precompiled "proto_piqi.erl" and "proto_piqi.hrl" in your application.
Moreover, the rewrite of "piqic-erlang" backend in Erlang doesn't seem
to make any effect for your use case.

Anton

[Paul Oliver]

Hey Anton,

What would be required to have the _ext json/xml stuff work in pure Erlang as well?

Cheers!

Paul. 

[Anton Lavrik]

On Fri, Feb 8, 2013 at 9:07 AM, Anton Lavrik <ala...@piqi.org> wrote:
> On Fri, Feb 8, 2013 at 4:15 AM, Dave Cottlehuber <d...@jsonified.com> wrote:
>>> However, once piqic-erlang is rewritten in Erlang, adding support for
>

>> I want to integrate
>> https://github.com/aphyr/riemann-java-client/blob/master/src/main/proto/riemann/proto.proto
>> into some Erlang code, and I'd like to use piqi as my understanding is
>> that I'll be able to provide a pure erlang implementation, without
>> needing to install protocol buffer tools on e.g. Windows boxes.
>

> Step 3 doesn't require any of "protoc" and piqi executables even today.
>
> This means that you can have a pure Erlang implementation on Windows
> today if you either build your application on Linux or include
> precompiled "proto_piqi.erl" and "proto_piqi.hrl" in your application.

Correction: if you build your app and generate rebar releases on
Linux, there's no point of including precompiled _piqi.erl and
_piqi.hrl, because they will be included in the release automatically.

One thing I would like to add. Today, piqi-erlang doesn't build on
Windows, partly because of the complicated build and dependency
handling that relies on shell scripts and symlinks. This will be
definitely improved as a part of the piqic-erlang rewrite and dropping
piqi_src dependency.

However, rebar builds are still not going to work on Windows out of
the box. This is because the standard way to run "piqic-erlang" is to
call it from a Makefile that is, in turn, called as a pre-compile hook
defined in rebar.config. The only way to make it work on Windows
without running buidls under Cygwin is to write a rebar plugin that
would remove the need for Makefiles.

Anton

[Anton Lavrik]

On Fri, Feb 8, 2013 at 9:13 AM, Paul Oliver <puzz...@gmail.com> wrote:
>
> Hey Anton,
>
> What would be required to have the _ext json/xml stuff work in pure Erlang
> as well?

Hi Paul,

Can you tell more about your use case for this?

I can think of 3 practical cases.

1. Optimizing raw performance, i.e. how fast one can convert JSON and
XML bits into Erlang representation.
2. Minimizing IPC latency or eliminating IPC overhead entirely. This
can be important for latency-sensitive applications. Also, it improves
throughput of single-threaded applications.
3. Better portability of Erlang applications that use Piqi for JSON
and XML serialization. We need to be very specific here about
potential deployment platforms. I would say this is more of a
usability problem. In other words, it comes down to the question of
how much effort it takes for a user to setup piqi-erlang on platform
X. Ideally, the effort should be 0 (it should already be this way on
Linux and Mac).

Let's take a closer look at potential optimization gains. I ran a
series of erlang serialization tests with Piqi and mochijson2 on my
dual-core mac and the micro-benchmarks showed that:

- If implemented using mochijson2:decode, JSON deserialization
performance would be comparable with Piqi.
- JSON serialization based on mochijson2:encode would be roughly 2.5
times faster. Not sure where such a big difference come from.
- IPC latency overhead, when serializing small data structures with
Piqi, stays within 40-50 microseconds.
- Native Erlang implementation would easily utilize all available
multi-core CPU resource. IPC, at least the current implementation,
doesn't allow Piqi-based serialization achieve 100% CPU resource
utilization, but it is getting close starting from moderate
concurrency levels (in my case, it was 8 threads on 2 cores).

I haven't tested xmerl, but I expect XML serialization to be slower in
case of a pure Erlang implementation.

At the same time, there are still a lot of optimization opportunities
in the existing implementation based on using the external piqi
program. There are certain limits of course. For example, IPC latency
won't go away completely. Another fundamental limit of this approach
is that it assumes one extra serialization cycle for sending Erlang
data structures to Piqi over pipe and reading them into internal
representation on the other end.

Now, to answer your question of what it would take to add support for
native Erlang JSON serialization. It is totally doable:

- Implementation effort -- varies depending on who will do it.
Comparable with piqic-erlang backend for protocol buffers
serialization + small runtime support library. All can be done in
Erlang. I've written 6 different backends for Piqi -- it is fairly
mechanical work for the most part but it needs to be done accurately.
This assumes using a solid third-party JSON library such as
mochijson2.
- Giving up on some features such as JSON pretty-printing and location
information in parsing errors.
- Risk of introducing subtle differences in behavior between native
Erlang and Piqi-based serialization (alternatively, developing
conformance tests and testing the new implementation against the
existing one).
- Risk of increased maintenance and support overhead.
- To add native XML support on top of that, multiply everything by 2.

Anton

[Paul Oliver]

Hey Anton,

Sorry, I should have given my use case.  It's for platforms where ocaml isn't well supported, such as MIPS/Linux.

Cheers!

Paul.

[Anton Lavrik]

On Sat, Feb 9, 2013 at 10:09 AM, Paul Oliver <puzz...@gmail.com> wrote:
>
>
> On Sat, Feb 9, 2013 at 6:04 AM, Anton Lavrik <ala...@piqi.org> wrote:
>>
>> On Fri, Feb 8, 2013 at 9:13 AM, Paul Oliver <puzz...@gmail.com> wrote:
>> >
>> > Hey Anton,
>> >
>> > What would be required to have the _ext json/xml stuff work in pure
>> > Erlang
>> > as well?
>>
>

> Hey Anton,
>
> Sorry, I should have given my use case. It's for platforms where ocaml
> isn't well supported, such as MIPS/Linux.

Speaking of OCaml for MIPS/Linux, it looks like somebody made it work:
https://github.com/retired-camels/ocaml#mips They even claim that it
is well tested.

Generally, for less supported platforms, you can always switch to
bytecode. It is very easy -- just change build target from "nc" to
"bc" in piqi-tools/Makefile and piqic/Makefile. You may loose some
performance and will need to install OCaml bytecode interpreter along
with piqi, but it should work just fine.

Anton

[Paul Oliver]

Thanks, Anton.  That's very helpful.  I'll check it out.

Paul.