Piqi-0.6.0 is released!

[Anton Lavrik]

I am proud to announce that Piqi version 0.6.0 is released!

This is the second major Piqi release and the second biggest
milestone in the history of the project.

Release highlights:

1. With this release, the Piqi language self-definition [1]
became stable. Why is it a big deal? There are several
implications.

- Any .piqi module (including self-definition itself!) can be
automatically and reliably converted to and from any
supported portable data formats. It means JSON, XML and
Protocol Buffers supported today, but also any other format
that Piqi may support in the future.

- In practical terms, it means that programs written in any
language can consume and manipulate type definitions from
.piqi modules. Also, because external representation of
Piqi modules is not going to change in a backward
incompatible way, applications built around Piqi type
definitions won't be forced to change often (if at all).

[1] http://piqi.org/self-definition/#piqi_piqi_source

2. Support for untyped JSON and XML and other changes in
encoding handling.

- As a new experimental feature, it is possible to represent
untyped JSON and XML as values of `piqi-any` type.
- generalized and improved encodings schemes for top-level
JSON and XML values
- improved documentation

3. Build and distribution improvements

- conventional ./configure script
- for OCaml users, Piqi can be now installed via
OPAM package [2] (OPAM is a new package manager for OCaml)
- greatly simplified installation from source code on Mac
(thanks to OPAM)
- Piqi documentation is now a part of the main Piqi GitHub
repository [3]

[2] (pending merge) https://github.com/OCamlPro/opam-repository/pull/363
[3] https://github.com/alavrik/piqi/tree/dev/doc

Backward incompatible changes:

- (piqi) Significant incompatibilities in external Piqi modules
representation which means that older applications that rely
on it won't work with the new version (e.g. older versions of
"piqi call" command-line piqi-rpc client)
- (piqi) Remove support for "piq-word" and "piq-text" built-in
types and provide an alternative way to control Piq
representation of string values using a new "piq-format"
property
- (piqi) Rename "piq-any" built-in type to "piqi-any"
- (xml) always use <value> as a top-level XML element instead
of tag derived from specific typename
- (json) top-level values of primitive types and enum constants
encoded in JSON as {"value": ...} instead of {"_": ...}
- (tools) use $PIQI_PATH environment variable for Piqi module
search paths instead of $PIQI_DIR


Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES
Updated roadmap: http://piqi.org/roadmap/
Downloads: http://piqi.org/downloads/


What's next:

Now that we've got stable external representation of Piqi
modules, the next goal is to rewrite "piqic-erlang" (piqi
compiler backend for Erlang) in Erlang.

Once this is done, the "piqi-erlang" and "piqi-rpc" sub-projects
will be moved into separate repositories. Not having
"piqic-erlang" fully written in Erlang and separated from the
main Piqi repository has been the major blocker from faster
feature development and accepting contributions from Erlang
users.

Some other features like adding default field values to generated
Erlang record definitions will follow shortly.


Anton

[Paul Oliver]

Great stuff, Anton!  I'm really looking forward to the piqic-erlang rewrite!

Anton

--
You received this message because you are subscribed to the Google Groups "piqi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to piqi+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Motiejus Jakštys]

On Mon, Feb 04, 2013 at 02:46:19AM -0600, Anton Lavrik wrote:
> I am proud to announce that Piqi version 0.6.0 is released!
>
> This is the second major Piqi release and the second biggest
> milestone in the history of the project.

Congratulations!

> Release highlights:
>
> 1. With this release, the Piqi language self-definition [1]
> became stable. Why is it a big deal? There are several
> implications.
>
> - Any .piqi module (including self-definition itself!) can be
> automatically and reliably converted to and from any
> supported portable data formats. It means JSON, XML and
> Protocol Buffers supported today, but also any other format
> that Piqi may support in the future.
>
> - In practical terms, it means that programs written in any
> language can consume and manipulate type definitions from
> .piqi modules. Also, because external representation of
> Piqi modules is not going to change in a backward
> incompatible way, applications built around Piqi type
> definitions won't be forced to change often (if at all).
>
> [1] http://piqi.org/self-definition/#piqi_piqi_source

Please tell me if I understand correctly the consequence: if I make a
library which understands piqi in <insert language here>, then I can be
confident that it will be backwards compatible forever?

I still think I don't get the point (my conclusion sounds wrong). Can
you tell more why it can be useful, maybe with a few examples? Seems
like a major change/improvement, but I can't understand its importance.

> 2. Support for untyped JSON and XML and other changes in
> encoding handling.
>
> - As a new experimental feature, it is possible to represent
> untyped JSON and XML as values of `piqi-any` type.

When I saw this, I thought that you meant type inference (and woo-hood
in my desk). For example, currently we have:

.variant [
.name filter-parm
.option [ .type float ]
.option [ .type int ]
.option [ .type string ]
]

with instances like

{float, 3.15}
{int, 5}
{string, "yadda"}

Will be able to be replaced by untagged values, like '3.15', '5' and
"yadda". Which is too good to be true. And examples show:

For primitive values in JSON:
{ "piqi_type": "piqi-any", "value": null }
and XML:
<?xml version="1.0" encoding="UTF-8"?>
<value/>

Why are raw untyped real values, not-nulls, converted to voids? Isn't
counter-intuitive?

However, if you specify the type at run-time (and in json), you get the
right value...

{ "piqi_type": "float", "value": 1.2345 }

Why is this feature useful? We already have a variant, which does the
same thing? Please elaborate. :-)

> - generalized and improved encodings schemes for top-level
> JSON and XML values

Please tell more about this or link to the documentation. Does it mean
that strings now are encoding aware? How is it represented in JSON?

How it used to be, and how is it now?

> - improved documentation

This is always great. ;)

> Backward incompatible changes:

>
> Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES

- Allow interchangeable use of '_' and '-' in imported module paths
and names

So what is the resulting name in (e|h)rl files if I use "_" now?
What about "-"? I can use "_" in module paths and names only for
filenames and this doesn't concern names of the variables/variants?

Documentation still states:
NOTE: use of underscore (_) characters in Piqi identifiers is not
allowed. Hyphens (-) should be used instead.

I quite liked the way it was done before: always "-" in piqi, and always
"_" in Erlang types.

- New experimental .deprecated flag for fields and options

Does the compiler give a warning now? Or at runtime when converting? It
is definitely useful, but how can we exploit it? Documentation is quite
scarce on it.

> What's next:
>
> Now that we've got stable external representation of Piqi
> modules, the next goal is to rewrite "piqic-erlang" (piqi
> compiler backend for Erlang) in Erlang.

So it will be just the compiler change, right (piqi files to .hrl and
.erl)? We will still need piqi binary in the runtime in order to use it,
correct?

> Once this is done, the "piqi-erlang" and "piqi-rpc" sub-projects
> will be moved into separate repositories. Not having
> "piqic-erlang" fully written in Erlang and separated from the
> main Piqi repository has been the major blocker from faster
> feature development and accepting contributions from Erlang
> users.

This has been a major headache in maintaining piqi-rpc builds. Sorry for
being impolite, but piqi-rpc is the second most horrible project after
exmpp regarding build complexity. Many man-hours have been spent
"fixing" it. I hope this change will allow us having a much cleaner
build system in Erlang. Maybe you are going to exploit "raw"
dependencies, which you introduced to rebar (we are very thankful for it
to you already)? I am ready to invest a few hours to revamp piqi build
system for Erlang, but am hestitating, because releases and major
changes are being rolled out, and I have a feeling that this will
eventually be much easier or just be solved when the time comes.

Can we have
{deps, [
{piqi, "0.6.1",
{git, "bb.org/alavrik/piqi-rpc.git", {tag, "0.6.1"}}
}
]
}.

Which would fetch piqi-erlang (and maybe piqi), and all these projects
would have like 1 post-compile hook and 1 post-clean hook in total? And
no "fake_src" and other kind of hacks?

Raw dependencies seem like a good start, but it still needs some
attention.

Regards,
Motiejus

[Motiejus Jakštys]

On Mon, Feb 04, 2013 at 06:02:49PM +0000, Motiejus Jakštys wrote:
> On Mon, Feb 04, 2013 at 02:46:19AM -0600, Anton Lavrik wrote:
> > Backward incompatible changes:
> >
> > Full list of changes: https://github.com/alavrik/piqi/blob/master/CHANGES
>
> - Allow interchangeable use of '_' and '-' in imported module paths
> and names
>
> So what is the resulting name in (e|h)rl files if I use "_" now?
> What about "-"? I can use "_" in module paths and names only for
> filenames and this doesn't concern names of the variables/variants?
>
> Documentation still states:
> NOTE: use of underscore (_) characters in Piqi identifiers is not
> allowed. Hyphens (-) should be used instead.
>
> I quite liked the way it was done before: always "-" in piqi, and always
> "_" in Erlang types.
>

Actually, is this conversion from "-" to "_" official? Will this hold
for the far-future that "_" are reserved and cannot be used for
names/stuff, and "-" are translated to "_"?

I couldn't find evidence in documentation, but (unfortunately?) we are
exploiting this wildly all over the place.

Regards,
Motiejus

[Thijs Terlouw]

Congratulations Anton! 

I'm now going over the changes and look forward to trying it as well.

[Anton Lavrik]

Hi Motiejus,

Thank you! I will try to answer some of your questions inline and I'll
start separate email threads to answer the rest of them.

On Mon, Feb 4, 2013 at 12:02 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Mon, Feb 04, 2013 at 02:46:19AM -0600, Anton Lavrik wrote:
>>
>> 1. With this release, the Piqi language self-definition [1]
>> became stable. Why is it a big deal? There are several
>> implications.
>>
>> - Any .piqi module (including self-definition itself!) can be
>> automatically and reliably converted to and from any
>> supported portable data formats. It means JSON, XML and
>> Protocol Buffers supported today, but also any other format
>> that Piqi may support in the future.
>>
>> - In practical terms, it means that programs written in any
>> language can consume and manipulate type definitions from
>> .piqi modules. Also, because external representation of
>> Piqi modules is not going to change in a backward
>> incompatible way, applications built around Piqi type
>> definitions won't be forced to change often (if at all).
>>
>> [1] http://piqi.org/self-definition/#piqi_piqi_source
>
> Please tell me if I understand correctly the consequence: if I make a
> library which understands piqi in <insert language here>, then I can be
> confident that it will be backwards compatible forever?

Yes. At least this is the goal. We are still at version 0.6 but I
really tried to stabilize it as best as I could. In fact, this is
where I spent 90% of effort in this release.

> I still think I don't get the point (my conclusion sounds wrong). Can
> you tell more why it can be useful, maybe with a few examples? Seems
> like a major change/improvement, but I can't understand its importance.

Think language or protocol standards. Like C89 or SQL92 or various
RFCs. One part of standardization is fixing the interface various
parties can rely on.

Another example would be operating system ABIs. I don't believe they
are standardized, but there are well known examples of very stable
ABIs that have been backward compatible for over a decade now.

There is one more example that stands out: JSON. It is a "forever"
spec meaning that the author didn't want it to ever evolve and didn't
put any versioning in place.

Speaking of Piqi, it is likely to evolve over time and build up more
features. It is quite natural, because it is self-defined and,
therefore, easily extensible in a backward compatible way. But I
expect the core spec to remain stable for a long time. This, I guess,
makes Piqi conceptually similar to ABIs and its core spec, hopefully,
to JSON. Depending on how it goes, I may or may not introduce
versioning of the core spec.

What makes Piqi unique is that everybody is free to extend Piqi the
spec and the language in any way he needs without writing a single
line of code and interfering with the core features. I'll show how it
works in practice in the new piqic-erlang.

Keep in mind that Piqi the language is slightly different from Piqi
the portable spec. While the spec is stable and shouldn't change, the
Piqi language that we use in .piqi modules _can_, in fact, change in
backward incompatible ways. But it will never happen overnight. For
example, although there are some syntax changes in this release, the
older syntax is still supported. If you use older syntax with the new
Piqi version, you will notice warnings that prompt you to use the new
syntax. At some later point, I may put together some tooling that will
automatically migrate the specs to the new syntax (this would work
similar to "gofix" tool in the Go language). Overall, this makes the
Piqi language similar to other computer languages that evolve through
versioning.

>> - generalized and improved encodings schemes for top-level
>> JSON and XML values
>
> Please tell more about this or link to the documentation. Does it mean
> that strings now are encoding aware? How is it represented in JSON?

I used "encodings" as a general term that refers to how data is
encoded into (or represented in) various formats. There was no change
in character encodings handling in strings -- it is strictly UTF-8 as
it was before.

There are 2 important backward incompatible changes:

1. Top-level values of primitive types, when serialized to JSON, are
now wrapped in {"value": ...} object instead of {"_": ...}.
2. Top-level values, when serialized to XML, are now wrapped into
<value>...</value> top-level element. In the previous version, the
name of the top-level element was derived from the name of the type,
e.g. <int>1</int> for values of type int.

The old format is still accepted during deserialization, but never
produced during serialization.

Another change related to JSON is that JSON fields like {...
"field_name": []} are omitted from serialized output by default.
Previously, it was the case only for fields like {... "field_name":
null}. The "json-omit-null-fields" setting can explicitly control this
behavior in "piqi convert", Erlang and OCaml serialization.

There were some other minor changes and improvements here and there,
but they shouldn't have any visible effect

> - New experimental .deprecated flag for fields and options
>
> Does the compiler give a warning now? Or at runtime when converting? It
> is definitely useful, but how can we exploit it? Documentation is quite
> scarce on it.

At the moment, this flag is not handled in any way by the
implementation. But it is recognized as a valid property one can
define for fields and options. Later, I'll think of adding an
automatically generated deprecation warning on attempt to use a field
or an option that was marked as .deprecated.


Anton

[Anton Lavrik]

Hi Motiejus,

Let me clarify the idea behind use of '-' (hyphens) and '_'
(underscores) in .piqi file names and identifiers inside .piqi inside
.piqi modules.

First of all, I need to point out that the 0.6 behavior should be 100%
backward compatible with earlier Piqi versions when it comes to naming
files and identifiers. In particular, there were no changes in
generated Erlang code. This means that all your current Erlang
application and build scripts should compile and work with Piqi 0.6.0
out of the box.

To be more specific -- and exactly as you say -- Piqi identifiers use
hyphens and generated Erlang identifiers always use underscores.

The change in behavior I referred to ("Allow interchangeable use of
'_' and '-' in imported module paths") is related to how the Piqi
implementation locates .piqi files when they are imported or included
from other Piqi modules. It actually became more flexible in 0.6.0.

The short story is that you can keep naming your .piqi files and
directory paths using whatever notation you are comfortable with.
There are some restrictions in naming .piqi files, because we need to
be able to derive a valid module name from the file name. This is not
changing.

What changed in the new version, is that one can now import module
foo_bar.piqi as .import [ .module foo-bar ] (note the '-' in the name
of the imported module). That was the main idea.

Here are some relevant documentation sections if you are interested in details:

http://piqi.org/doc/piqi/#modulenames
http://piqi.org/doc/piqi/#namingofcodepiqicodefiles
http://piqi.org/doc/piqi/#anoteaboutdirectorypaths

Anton

[Anton Lavrik]

On Mon, Feb 4, 2013 at 12:02 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Mon, Feb 04, 2013 at 02:46:19AM -0600, Anton Lavrik wrote:
>> What's next:
>>
>> Now that we've got stable external representation of Piqi
>> modules, the next goal is to rewrite "piqic-erlang" (piqi
>> compiler backend for Erlang) in Erlang.
>
> So it will be just the compiler change, right (piqi files to .hrl and
> .erl)? We will still need piqi binary in the runtime in order to use it,
> correct?

Correct. You will still need piqi binary if you want to use JSON or
XML serialization from your Erlang program. This is unlikely to change
in foreseeable future.

>> Once this is done, the "piqi-erlang" and "piqi-rpc" sub-projects
>> will be moved into separate repositories. Not having
>> "piqic-erlang" fully written in Erlang and separated from the
>> main Piqi repository has been the major blocker from faster
>> feature development and accepting contributions from Erlang
>> users.
>
> This has been a major headache in maintaining piqi-rpc builds. Sorry for
> being impolite, but piqi-rpc is the second most horrible project after
> exmpp regarding build complexity. Many man-hours have been spent
> "fixing" it. I hope this change will allow us having a much cleaner
> build system in Erlang.

I'm a little surprised to hear that. I've been using piqi-rpc in a
number of projects and haven't had any serious issues with it. I've
also seen some other people using it just fine.

Granted, this dependency stuff looks scary inside, build examples may
not be that great, and I would totally understand if it takes some
time to understand how to make it work.

Still, in order to make it run smoothly for a simple project, even
today, you need only 3 things:

1. add piqi_rpc dependency into your rebar.config: {piqi_rpc, "",
{git, "https://github.com/alavrik/piqi-rpc.git", {tag, "v0.6.0"}}}

2. add 2 hooks like these to your rebar.config:
https://github.com/alavrik/piqi-erlang-examples/blob/master/piqi_rpc/rebar.config

3. put together (copy-paste) a makefile like this one:
https://github.com/alavrik/piqi-erlang-examples/blob/master/piqi_rpc/src/Makefile.piqi

From my experience, things can get more complicated if your .piqi
modules live in a separate repository. But this problem wouldn't be
unique to piqi-rpc.

> Maybe you are going to exploit "raw"
> dependencies, which you introduced to rebar (we are very thankful for it
> to you already)?

This would definitely help to make the existing dependency chain
cleaner, but I didn't want to spend time on it knowing that it is all
going to change and become much more simpler after piqic-erlang
rewrite anyway. At the same time, the existing system should just
work.

It could be that your environment is different from mine or you use it
differently. I may be able to help you If you describe specific
problems you are having with piqi-rpc builds.

Anton

[Motiejus Jakštys]

Correct. I realized that I was just bashing the build system without
giving any evidence. Bad.

If you are curious, we are using rebar shared dependencies[1]. It makes
life easier for us, but more difficult with non-standard projects. It
wasn't me who ran into those issues, but I remember some of my
colleagues saying that he had to do some changes to the piqi build
system in order for shared deps to work properly.

Now it works, and, after the rewrite of piqic-erlang, I will look at it
again, and, if upstream doesn't work, I will report and/or propose
something.

That being said, I never had problems using piqi in projects that do not
use rebar shared dependencies. So for a normal user it is great.

[1]: https://github.com/basho/rebar/pull/262

Regards,
Motiejus

[Dave Parfitt]

Congrats, Anton - great work!

Dave

On Mon, Feb 4, 2013 at 3:46 AM, Anton Lavrik <ala...@piqi.org> wrote:

Anton

[Anton Lavrik]

On Mon, Feb 4, 2013 at 12:02 PM, Motiejus Jakštys <desir...@gmail.com> wrote:

> On Mon, Feb 04, 2013 at 02:46:19AM -0600, Anton Lavrik wrote:
>
>> 2. Support for untyped JSON and XML and other changes in
>> encoding handling.
>>
>> - As a new experimental feature, it is possible to represent
>> untyped JSON and XML as values of `piqi-any` type.

This new feature deserved a more specific example. Here it is.

Suppose you have a Piqi record defined as:

$ cat person.piqi
.record [
.name person
.field [
.name name
.type string
]
.field [
.name phone
.type string
.repeated
]

% field containing untyped JSON or XML
.field [
.name custom
.type piqi-any
.optional
]
]

And, let's say, you have a server written in Erlang that implements an
API for storing and getting values of type "person". The client
application is written in JavaScript and it communicates to the server
to store and, later, retrieve objects of this type in JSON format.
Roughly speaking, the server is acting as a sort of DB for your person
records and all it cares about is what was defined in the schema. The
browser-based JavaScript client, on the other hand, deals with
presentation and although presentation aspects are not directly
related to the data itself, as it turns out, it is convenient to store
some details about object presentation next to the object itself.
Moreover, these presentation details are not very well defined and can
change frequently and radically due to experimentation and rapid
iteration on UI features. As the development progresses, it becomes
obvious that reflecting all those details in the Piqi schema would be
completely unreasonable.

Keeping all that in mind, what we can do is define one field with type
"piqi-any" which purpose is to hold all that messy presentation
details as a part of the data object.

For example, this is what it might look in JSON:

{
"name": "J. Random Hacker",
"phone": [ "(111) 123 45 67" ],
"custom": {
"created_at": "2013-01-02T12:07Z",
"modified_at": "2013-02-02T12:07Z",
"modified_by": "Joe User"
}
}

Now, if you run

person_piqi_ext:parse_person(Json, 'json').

or receive it as a piqi-rpc parameter, you'll get this Erlang record
(note the untyped JSON binary as a #piqi_any field):

#person_person{
name = <<"J. Random Hacker">>,
phone = [<<"(111) 123 45 67">>],
custom =
#piqi_any{
type = undefined,protobuf = undefined,
json =
<<"{\n \"created_at\": \"2013-01-02T12:07Z\",\n
\"modified_at\": \"2013-02-02T12:07Z\",\n \"modi"...>>,
xml = undefined}}

Obviously, if you run

person_piqi_ext:gen_person(Person, 'json_pretty')

you'll get your original JSON back that will include the value of the
"custom" field.


I pushed a complete example here:
https://github.com/alavrik/piqi/tree/dev/examples/erlang-piqi-any

The original purpose of this feature has nothing to do with the above
story, but this story is probably closer to potential practical
applications.

Anton