Piqi-erlang and Piqi-RPC v0.7.0 released

[Anton Lavrik]

I'm pleased to announce the release of Piqi-erlang and Piqi-RPC v0.7.0!

There were two major groups of changes: rewriting the Piqi-erlang code generator

in Erlang and moving Piqi-RPC and Piqi-Erlang into their own repositories.

Both changes are extremely important and represent a major milestone. The

particular benefits include:

- From now on, Piqi-erlang doesn't depend on the core Piqi project development

  and release cycles. Before that, it was pretty much impossible to release the

  three projects (Piqi, Piqi-erlang and Piqi-RPC) independently.

- Packaging and using Piqi-RPC and Piqi-erlang has become a lot simpler. Now

  they look and act almost like any other Erlang library.

- The fact that Piqi-erlang code generation is written in Erlang opens up a

  clear path for contributions from Erlang users. No OCaml knowledge or even

  OCaml installation is needed.

To simplify things even further, separate release git branches are gone and

everything is now in "master".

There was only one backward-incompatible change. The "piqic-erlang-ext" compiler

and therefore *_piqi_ext.erl go away. "piqic-erlang" now generates serializers

and deserializers for multi-format converters as gen_<typename>/2, /3 and

parse_<typename>/2, /3 functions) directly in *_piqi.erl. Migrating to the new

version should be as easy as changing all _piqi_ext:* function calls to call

_piqi:* instead.

Some other functional changes:

- "piqic-erlang --gen-defaults" option is deprecated. "piqic-erlang" now

  generates _piqi:default_X/1 without asking

- generate default values for Erlang record fields if they are defined in the

  original Piqi spec

- add a new "erlang-default" alias property; when specified, it overrides the

  Piqi-default value for the type in the generated "*_piqi:default_<type>/0"

  functions

Along with the new Piqi-Erlang and Piqi-RPC versions, we are also releasing Piqi

v0.6.5. This is mostly a maintenance release -- same as the four previous Piqi

releases (v0.6.1 -- v0.6.4). Among other things it includes:

- a fix for incorrect parsing of large (> 2^63 - 1) unsigned decimal literals in

  Piq, JSON and XML

- a new "json-omit-missing" field property that controls, on a per-field basis,

  whether to exclude "null" and "[]" fields from JSON output

- better piqic tooling support, namely inclusion of the original .piqi file name

  in the compiled Piqi spec

We have also made a significant progress in non-functional areas such as:

- The original documentation for Piqi, Piqi-RPC and Piqi-erlang is now a part of

  source code tree. You can find it under the doc/ sub-subdirectory of

  correspondent repos and format it any way you like.

- Continuous builds and tests(!) through Travis-CI. Some new tests were added

  and interfaces to run the tests were automated.

- Initial Debian and RPM packaging

Overall, this was a major effort and it wouldn't have been possible without

support of many Piqi users. Thank you everyone for your enormous support,

contributions and patience.

Special thanks goes to Motiejus Jakštys for bug fixes and numerous other

contributions in the areas of OS packaging, documentation building scripts and

Travis-CI integration.

I would also like to thank Dennis Docter who troubleshooted an Erlang name

normalization problem in the new "piqic-erlang" implementation and contributed a

solution.

Links to the changelogs and updated roadmap:

https://github.com/alavrik/piqi/blob/master/CHANGES

https://github.com/alavrik/piqi-erlang/blob/master/CHANGES

https://github.com/alavrik/piqi-rpc/blob/master/CHANGES

http://piqi.org/roadmap/

Anton

[Ignas Vyšniauskas]

Hi Anton,

Pretty excited to hear about this and thanks again for the great work
you put into piqi. All the changes seem to be nice improvements.

Looking forward to bumping our projects at Spilgames to this version
too. I think we'll be doing this pretty soon, expect feedback ,-)

--
Ignas

[Anton Lavrik]

Hi Ignas,

Thank you!  Looking forward to your feedback. I don't expect a lot of issues as things have been fairly stable in master and there weren't many changes lately. It was important to (finally!) release it though. Good thing is that the changes in this release allow for simpler and more granular future releases.

Speaking of the way you are using Piqi at Spilgames. This questions is also for Motiejus. Would you guys be interested in updating the rpm and deb packaging you've created back in April?

I've been also thinking about a central location for various repositories related to the project and just created the piqi account on github (https://github.com/piqi). My long-term goal is to move everything under to the common community-maintained account. We could start with packaging. So, how about moving piqi-rpm and piqi-deb there?

Anton

--
You received this message because you are subscribed to the Google Groups "piqi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to piqi+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Motiejus Jakštys]

On Wed, Nov 6, 2013 at 10:32 AM, Anton Lavrik <ala...@piqi.org> wrote:
> Hi Ignas,
>
> Thank you! Looking forward to your feedback. I don't expect a lot of issues
> as things have been fairly stable in master and there weren't many changes
> lately. It was important to (finally!) release it though. Good thing is that
> the changes in this release allow for simpler and more granular future
> releases.
>
> Speaking of the way you are using Piqi at Spilgames. This questions is also
> for Motiejus. Would you guys be interested in updating the rpm and deb
> packaging you've created back in April?

Hi Anton,

I am excited about the changes and am very happy you released 0.7.
Like we said, Ignas now is doing some work on way we compose
interfaces. Will give some feedback then.

Right after your announcement, I put "update rpms and debs" task to my
todo queue. However, have been too busy to pick that up yet. Of course
we have interest to update the packages (we need them, we work with
them!), and surely I will update them. I think it will be able to put
some time in the next couple of days.

> I've been also thinking about a central location for various repositories
> related to the project and just created the piqi account on github
> (https://github.com/piqi). My long-term goal is to move everything under to
> the common community-maintained account. We could start with packaging. So,
> how about moving piqi-rpm and piqi-deb there?

I agree it's a very good idea to centralize the repos. Feel free to
fork from me, I will send pull requests once I do the builds for 0.7.

Now some thoughts on the future of building piqi. openSUSE Build
Service (OBS) also supports Debian and a few other distributions.
However, I couldn't get around it. Anyone interested in playing with
OBS to support Debian and potentially other distros? Now there are
builds for Ubuntu 10.04-13.04 in ppa[1] and Fedora 17-18 in OBS[2].

Anton, are you using the packaged version yourself, or putting the
piqi ocaml sources as dependencies? What do you think about putting
piqi to debian/contrib? How about production?

Just for you to know.. We are handling tenths of thousands requests
per second with piqi-rpc, more and more APIs at Spil are behind it.
Works great.

[1]: https://launchpad.net/~motiejus/+ppa-packages
[2]: http://software.opensuse.org/download.html?project=home:motiejusj&package=piqi

Motiejus

[Motiejus Jakštys]

On Wed, Nov 6, 2013 at 1:42 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Wed, Nov 6, 2013 at 10:32 AM, Anton Lavrik <ala...@piqi.org> wrote:
> Right after your announcement, I put "update rpms and debs" task to my
> todo queue. However, have been too busy to pick that up yet. Of course
> we have interest to update the packages (we need them, we work with
> them!), and surely I will update them. I think it will be able to put
> some time in the next couple of days.

I have stolen some time from myself and uploaded piqi to 0.6.5
packages for lucid, precise, quantal, raring, saucy and trusty[1].
Launchpad says they should be ready for download in a couple of hours
(provided they build without errors).

> I agree it's a very good idea to centralize the repos. Feel free to
> fork from me, I will send pull requests once I do the builds for 0.7.

I see you created the team repository and added me to the team. Do you
want me to push directly there?

> Now some thoughts on the future of building piqi. openSUSE Build
> Service (OBS) also supports Debian and a few other distributions.
> However, I couldn't get around it. Anyone interested in playing with
> OBS to support Debian and potentially other distros? Now there are
> builds for Ubuntu 10.04-13.04 in ppa[1] and Fedora 17-18 in OBS[2].

I have reached the decision to NOT maintain OBS. I *will*, however,
maintain piqi.spec. I am looking for someone willing to take up piqi
maintenance in OBS or a similar service which provides RPMs for
popular rpm-based distributions.

I am also thinking about a FreeBSD port (started playing with it, I
quite like the system). No promises though.

[1]: https://launchpad.net/~motiejus/+archive/piqi

--
Motiejus Jakštys

[Anton Lavrik]

On Wed, Nov 6, 2013 at 4:42 AM, Motiejus Jakštys <desir...@gmail.com> wrote:

On Wed, Nov 6, 2013 at 10:32 AM, Anton Lavrik <ala...@piqi.org> wrote:

> I've been also thinking about a central location for various repositories
> related to the project and just created the piqi account on github
> (https://github.com/piqi). My long-term goal is to move everything under to
> the common community-maintained account. We could start with packaging. So,
> how about moving piqi-rpm and piqi-deb there?

I agree it's a very good idea to centralize the repos. Feel free to
fork from me, I will send pull requests once I do the builds for 0.7.

Great! I have a better idea actually. I created two new repos piqi-rpm and piqi-deb under the organization account. You should have push access. It would be great if you could push the updated versions there. I believe it is better to do to keep the central version as a primary source and fork it if necessary than the other way around. This will help to keep things clean and consistent as the project evolves.

 

Now some thoughts on the future of building piqi. openSUSE Build
Service (OBS) also supports Debian and a few other distributions.
However, I couldn't get around it. Anyone interested in playing with
OBS to support Debian and potentially other distros? Now there are
builds for Ubuntu 10.04-13.04 in ppa[1] and Fedora 17-18 in OBS[2].

Anton, are you using the packaged version yourself, or putting the
piqi ocaml sources as dependencies? What do you think about putting
piqi to debian/contrib? How about production?

I'm using it mostly on Mac and FreeBSD these days. On Mac, I'm just using piqi-binary as a dependency and on FreeBSD I build it separately and set up the $PIQI variable to point to the build. This is not ideal as we've discussed before, so I definitely interested in packaging, picking the binaries from $PATH and so on.

What do you mean by debian/contrib and production?

 

Just for you to know.. We are handling tenths of thousands requests
per second with piqi-rpc, more and more APIs at Spil are behind it.
Works great.

This is wonderful! Thank you for mentioning it.

Anton

[Anton Lavrik]

On Thu, Nov 7, 2013 at 12:55 AM, Motiejus Jakštys <desir...@gmail.com> wrote:

On Wed, Nov 6, 2013 at 1:42 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Wed, Nov 6, 2013 at 10:32 AM, Anton Lavrik <ala...@piqi.org> wrote:

> Right after your announcement, I put "update rpms and debs" task to my
> todo queue. However, have been too busy to pick that up yet. Of course
> we have interest to update the packages (we need them, we work with
> them!), and surely I will update them. I think it will be able to put
> some time in the next couple of days.

I have stolen some time from myself and uploaded piqi to 0.6.5
packages for lucid, precise, quantal, raring, saucy and trusty[1].
Launchpad says they should be ready for download in a couple of hours
(provided they build without errors).

Awesome! Thank you. I will be updating the downloads section on the website then.

 

> I agree it's a very good idea to centralize the repos. Feel free to
> fork from me, I will send pull requests once I do the builds for 0.7.

I see you created the team repository and added me to the team. Do you
want me to push directly there?

Yes, please. Only thing: I noticed that for the deb packaging you essentially cloned the whole upstream source tree. Is that how it is supposed to work or this was just the easiest way to do the packaging?

> Now some thoughts on the future of building piqi. openSUSE Build
> Service (OBS) also supports Debian and a few other distributions.
> However, I couldn't get around it. Anyone interested in playing with
> OBS to support Debian and potentially other distros? Now there are
> builds for Ubuntu 10.04-13.04 in ppa[1] and Fedora 17-18 in OBS[2].

I have reached the decision to NOT maintain OBS. I *will*, however,
maintain piqi.spec. I am looking for someone willing to take up piqi
maintenance in OBS or a similar service which provides RPMs for
popular rpm-based distributions.

I have never dealt OBS before so it is hard for me to comment. I'll take a look at it.

 

I am also thinking about a FreeBSD port (started playing with it, I
quite like the system). No promises though.

Ah, that would be really cool if you get to it.

Thanks again you for all your help.

Anton

[Motiejus Jakštys]

On Thu, Nov 7, 2013 at 10:00 AM, Anton Lavrik <ala...@piqi.org> wrote:
>
> Great! I have a better idea actually. I created two new repos piqi-rpm and
> piqi-deb under the organization account. You should have push access. It
> would be great if you could push the updated versions there. I believe it is
> better to do to keep the central version as a primary source and fork it if
> necessary than the other way around. This will help to keep things clean and
> consistent as the project evolves.

I pushed piqi-rpm there. Some comments about piqi-deb in another email.

> What do you mean by debian/contrib and production?

1.Put the package to debian repositories (most likely section 'contrib').
2. I don't know what I meant by 'production'. Forget it.

Motiejus

[Motiejus Jakštys]

On Thu, Nov 7, 2013 at 10:07 AM, Anton Lavrik <ala...@piqi.org> wrote:
>
> On Thu, Nov 7, 2013 at 12:55 AM, Motiejus Jakštys <desir...@gmail.com>
> wrote:
> Yes, please. Only thing: I noticed that for the deb packaging you
> essentially cloned the whole upstream source tree. Is that how it is
> supposed to work or this was just the easiest way to do the packaging?

There are very many ways to do packaging in debian. I looked *really
hard* for a way that doesn't involve having the source in the
repository. Couldn't find it. Most modern and common approach I found
is like how I do: have an 'upstream' branch, and always merge from
there to distribution-specific branches:

https://github.com/Motiejus/piqi-deb/tree/upstream (identical to alavrik/master)
and merge `upstream` to these branches, which have 'debian' directory:
https://github.com/Motiejus/piqi-deb/tree/debian/snapshot
https://github.com/Motiejus/piqi-deb/tree/lucid/snapshot
https://github.com/Motiejus/piqi-deb/tree/precise/snapshot
... etc ...

If there is a way to avoid having piqi source tree in the repository,
I would love to hear it. For exactly this reason I found RPM building
mechanism much cleaner and more straightforward.

So now I have a bunch of branches for every distribution, which sounds
unclean. I am also doing mistakes and sometimes rewriting history of
topic branches. Do you still want me to send the messy approach to
main piqi-deb repo?

Motiejus

[Anton Lavrik]

Agree -- this does't look very clean. Let me do some research on the topic first. In any case, it looks like most of these actions can be automated.

Anton

[Anton Lavrik]

I read through Debian developers docs and it appears that Debian packaging is not much more complicated than RPM (at least at its core). For instance, although people use git for maintaining packages, it is not required. Especially for a fairly simple package like piqi. Can't speak for Ubuntu though -- haven't checked what's going on there.

With that, I decided to give it a try and make things as simple as possible: https://github.com/piqi/piqi-deb

What do you think? Essentially, all I did was adding a couple of scripts on top of your original packaging structure. There were a couple of small changes in debian/* -- see the latest commits. If that works as a general approach, we can improve it further any way we like.

As I see it, the immediate goal of this exercise is for Piqi users to be able to go to a well-defined place, follow simple instructions and get a nice .deb or .rpm package for their use. It looks like we are almost there. Making the packages available through public repositories and getting Piqi into distros is definitely useful as well. However, there's some specialized knowledge on how do it and it won't be possible to cover everything at once. We'll get there eventually.

Anton

[Motiejus Jakštys]

2013.11.12 09:50, Anton Lavrik rašė:
>
>
> On Thu, Nov 7, 2013 at 1:42 AM, Anton Lavrik <ala...@piqi.org
> <mailto:ala...@piqi.org>> wrote:
>
> I read through Debian developers docs and it appears that Debian
> packaging is not much more complicated than RPM (at least at its core).
> For instance, although people use git for maintaining packages, it is
> not required. Especially for a fairly simple package like piqi. Can't
> speak for Ubuntu though -- haven't checked what's going on there.
>
> With that, I decided to give it a try and make things as simple as

Hi Anton,

I like the approach, especially because there is no piqi source in the
repository. I've overused various 'dpkg utilities', which just made life
more difficult.

Ubuntu is the same as debian, just replace 'unstable' to distribution
name like 'lucid', 'precise', ... in debian/changelog, and you have an
ubuntu source package ready for upload/use.

> possible: https://github.com/piqi/piqi-deb
>
> What do you think? Essentially, all I did was adding a couple of scripts
> on top of your original packaging structure. There were a couple of
> small changes in debian/* -- see the latest commits. If that works as a
> general approach, we can improve it further any way we like.

Like I said, I like it. A few comments:
* one changelog entry has my name and your email
* to be consistent with RPM (and be safer in general), I would include
SHA256 of the git tree of the particular version, not only the tag, in
download-upstream-tarball. If I remember correctly, you can specify the
SHA256 as well as tag in github API.
* Makefile needs a switch to create a source package for sending to
debian/ubuntu build system. I will create one when I will update the PPA
again.

In general, I think it's very good to use. Good job.

> As I see it, the immediate goal of this exercise is for Piqi users to be
> able to go to a well-defined place, follow simple instructions and get a
> nice .deb or .rpm package for their use. It looks like we are almost
> there. Making the packages available through public repositories and
> getting Piqi into distros is definitely useful as well. However, there's
> some specialized knowledge on how do it and it won't be possible to
> cover everything at once. We'll get there eventually.

I will see what I can do to push piqi to debian repositories. Ubuntu
will follow once it's in the debian queue.

Motiejus

[Anton Lavrik]

On Tue, Nov 12, 2013 at 2:23 AM, Motiejus Jakštys <desir...@gmail.com> wrote:

Like I said, I like it. A few comments:
* one changelog entry has my name and your email

Fixed.

 

* to be consistent with RPM (and be safer in general), I would include SHA256 of the git tree of the particular version, not only the tag, in download-upstream-tarball. If I remember correctly, you can specify the SHA256 as well as tag in github API.

Done.

I also made some similar changes in the RPM packaging and added links to the packaging repos from piqi.org/downloads/

At this point, both Debian and RPM look pretty good. Let's use regular GitHub pull requests/issues for any future work on packaging.

I will see what I can do to push piqi to debian repositories. Ubuntu will follow once it's in the debian queue.

That would be really cool!

Anton