separate piqi-rpc, piqi-erlang from main piqi repository

[Motiejus Jakštys]

Hi, Anton,

We are thinking to package piqi to an RPM*, and have only "piqi-rpc"
as Erlang dependencies in our applications. Because of a few reasons:

1. We want to use company-wide piqi version and have a central place
to update it (our package repositories).
2. For smaller projects compiling piqi takes 90% of the total project
build time.
3. Be more explicit about the project dependencies. I.e. instead of
saying: for your tiny-project these packages must be installed
system-wide: "erlang ocaml ocaml-findlib ocaml-caml4p-devel", it will
be "erlang piqi".

Same applies if piqi is downloaded via opam. And old behaviour will be
preserved (if system has no piqi, it must download and compile it at
build time).

For that to work we need to clearly separate piqi-erlang code from
piqi-ml code. I propose moving piqi-rpc, piqi-erlang code to their
repositories. That will allow us not to download the whole piqi-ml
shebang if we already have it (now it is needed for piqi-erlang and
piqi-rpc .erl files).

New build scenario: assume you want piqi-rpc. Piqi-rpc depends on
piqi-erlang. Piqi-erlang has rebar.config.script, which checks if
"which piqi" returns a path. If "which piqi" returns error, we add
"piqi" as a dependency of piqi-erlang, and the compilation stuff will
be done like now.

I will do this only if it makes sense for you and there is a remote
chance of it being merged. Otherwise I do not want to maintain this
fork forever. :-)
If you agree with this, another question: do you want to preserve
history of piqi-erlang and piqi-rpc? I.e. I hack with git a bit and
put history of piqi/piqi-erlang to piqi-erlang. Same with piqi-rpc. Do
you want that? My opinion: I am against it, because you will still
have the history in main piqi repository, and that new history will be
synthetic and will not reflect reality.

[*]: a free bonus: DEB packages as well. We will contribute DEB/RPM
build scripts for community.

--
Motiejus Jakštys

[Anton Lavrik]

On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> Hi, Anton,
>
> We are thinking to package piqi to an RPM*, and have only "piqi-rpc"
> as Erlang dependencies in our applications. Because of a few reasons:
>
> 1. We want to use company-wide piqi version and have a central place
> to update it (our package repositories).
> 2. For smaller projects compiling piqi takes 90% of the total project
> build time.
> 3. Be more explicit about the project dependencies. I.e. instead of
> saying: for your tiny-project these packages must be installed
> system-wide: "erlang ocaml ocaml-findlib ocaml-caml4p-devel", it will
> be "erlang piqi".
>
> Same applies if piqi is downloaded via opam. And old behaviour will be
> preserved (if system has no piqi, it must download and compile it at
> build time).

That was roughly my plan as well. The difference is that I never have
to compile piqi executables from sources, because I always use one of
the released stable versions that includes the prebuilt piqi and piqic
executables. However, I've got some really good news for you -- read
on!

> For that to work we need to clearly separate piqi-erlang code from
> piqi-ml code. I propose moving piqi-rpc, piqi-erlang code to their
> repositories. That will allow us not to download the whole piqi-ml
> shebang if we already have it (now it is needed for piqi-erlang and
> piqi-rpc .erl files).

In fact, I've just finished it. And that includes the rewrite of
piqic-erlang backend in Erlang.

What it means in practice?

1. Complete piqi-erlang codebase, including the new "piqic-erlang"
compiler, is now located and maintained here:
https://github.com/alavrik/piqi-erlang (in dev branch, which is the
new default). There's no piqi_src dependency any more. In addition,
the repository includes all the tests, examples and documentation.

2. piqi-erlang has a new dependency called piqi-binary
(https://github.com/alavrik/piqi-binary) that includes precompiled
"piqi" executable for various platforms. This means that there's no
need to build piqi from OCaml sources any more. By the way,
"piqic-erlang" no longer requires "piqic". Instead it relies on "piqi
compile".

3. Similarly to piqi-erlang, piqi-rpc codebase is now fully contained
at https://github.com/alavrik/piqi-rpc Naturally, it depends on
piqi-erlang but that's pretty much it. Oh, I also bumped Webmachine
version to 1.9.3

4. "piqic-erlang-ext" and therefore *_piqi_ext.erl go away.
"piqic-erlang" now includes serializers and deserializers for
multi-format converters (i.e. gen_X*/2, /3 and parse_X*/2, /3
functions) in *_piqi.erl.

5. "--gen-defaults" compiler option is deprecated. "piqic-erlang" now
generates default_X/1 without asking.

I've been testing this new stuff for the last couple of weeks and I'm
pretty confident about it. It should be almost a drop-in replacement
for piqi-0.6.0. Only some minor makefile and code tweaking may be
necessary, e.g. renaming to _piqi_ext:gen_X calls to _piqi:gen_X.

My plan is to release this version and officially announce it in the
next week or two. In the meantime it is fully ready for user testing.

It would be great if you guys could try it out and send your comments,
bug reports and fixes :) With the new repository layout it should be
so much easier to contribute.

> New build scenario: assume you want piqi-rpc. Piqi-rpc depends on
> piqi-erlang. Piqi-erlang has rebar.config.script, which checks if
> "which piqi" returns a path. If "which piqi" returns error, we add
> "piqi" as a dependency of piqi-erlang, and the compilation stuff will
> be done like now.

I think this would be a very useful addition. Especially for platforms
for which we don't have pre-built piqi binaries. Now that repositories
are separated, I'd rather have users build and install "piqi"
executable separately and not include piqi_src as a dependency and
have it compiled during "rebar compile".

> I will do this only if it makes sense for you and there is a remote
> chance of it being merged. Otherwise I do not want to maintain this
> fork forever. :-)

I can merge it with no problem. My only suggestion would be to
parameterize it using some environment variable instead of just
relying on "piqi" executable accessibility.

> If you agree with this, another question: do you want to preserve
> history of piqi-erlang and piqi-rpc? I.e. I hack with git a bit and
> put history of piqi/piqi-erlang to piqi-erlang. Same with piqi-rpc. Do
> you want that? My opinion: I am against it, because you will still
> have the history in main piqi repository, and that new history will be
> synthetic and will not reflect reality.

That's already done. The history was preserved.

> [*]: a free bonus: DEB packages as well. We will contribute DEB/RPM
> build scripts for community.

That would be awesome and very much appreciated!

Anton

[Motiejus Jakštys]

On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:
> On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> > Hi, Anton,
> >
> > We are thinking to package piqi to an RPM*, and have only "piqi-rpc"
> > as Erlang dependencies in our applications. Because of a few reasons:
> >
> > 1. We want to use company-wide piqi version and have a central place
> > to update it (our package repositories).
> > 2. For smaller projects compiling piqi takes 90% of the total project
> > build time.
> > 3. Be more explicit about the project dependencies. I.e. instead of
> > saying: for your tiny-project these packages must be installed
> > system-wide: "erlang ocaml ocaml-findlib ocaml-caml4p-devel", it will
> > be "erlang piqi".
> >
> > Same applies if piqi is downloaded via opam. And old behaviour will be
> > preserved (if system has no piqi, it must download and compile it at
> > build time).
>
> That was roughly my plan as well. The difference is that I never have
> to compile piqi executables from sources, because I always use one of
> the released stable versions that includes the prebuilt piqi and piqic
> executables.

In the past these did not work with some of our CI machines due to older
glibc version. Current piqi-binary works, but I am unsure why: either
because of system upgrade (unlikely that it happened), or because you
changed something in the binary build process, and it Just started working.
Unlikely as well, I guess.

Besides, maybe it is possible to fully statically compile piqi (i.e.
include libc)? That would be awesome. I wouldn't be surprised if it
worked with some other libc and could fully statically link.

Building piqi fails for me since 0.6.0 (just noticed), so I can't really
try. I will send you a proper bug report sometime soon.

> What it means in practice?
>
> 1. Complete piqi-erlang codebase, including the new "piqic-erlang"
> compiler, is now located and maintained here:
> https://github.com/alavrik/piqi-erlang (in dev branch, which is the
> new default). There's no piqi_src dependency any more. In addition,
> the repository includes all the tests, examples and documentation.
>
> 2. piqi-erlang has a new dependency called piqi-binary
> (https://github.com/alavrik/piqi-binary) that includes precompiled
> "piqi" executable for various platforms. This means that there's no
> need to build piqi from OCaml sources any more. By the way,
> "piqic-erlang" no longer requires "piqic". Instead it relies on "piqi
> compile".

I had a quick look, now it really looks cleaner. Good job!

As the artefact build system changed, Makefiles will need revamping on
our side. It is a good chance to redo the thing, though will take a
while.

> 3. Similarly to piqi-erlang, piqi-rpc codebase is now fully contained
> at https://github.com/alavrik/piqi-rpc Naturally, it depends on
> piqi-erlang but that's pretty much it. Oh, I also bumped Webmachine
> version to 1.9.3
>
> 4. "piqic-erlang-ext" and therefore *_piqi_ext.erl go away.
> "piqic-erlang" now includes serializers and deserializers for
> multi-format converters (i.e. gen_X*/2, /3 and parse_X*/2, /3
> functions) in *_piqi.erl.
>
> 5. "--gen-defaults" compiler option is deprecated. "piqic-erlang" now
> generates default_X/1 without asking.
>
> I've been testing this new stuff for the last couple of weeks and I'm
> pretty confident about it. It should be almost a drop-in replacement
> for piqi-0.6.0. Only some minor makefile and code tweaking may be
> necessary, e.g. renaming to _piqi_ext:gen_X calls to _piqi:gen_X.

We are still on 0.5.7 and were waiting for default-field-values (not a
big deal, because I can cherry-pick) and piqic-erlang rewrite (big
deal). So now we will start migrating gradually I guess.

Is default-field-values in the job pipeline? We are looking forward. :-)

FYI, today I cherry-picked our "main" internal piqi fork with
experimental-erlang-defaults-0.5.7-dev-backport. So all our new projects
will use it, and the ones that will update piqi. Everyone in the company
are happy so far.

I also made some changes today which *depend* on that feature. So we are
dependent. Thanks again, it really allowed us to improve our data model.

> My plan is to release this version and officially announce it in the
> next week or two. In the meantime it is fully ready for user testing.
>
> It would be great if you guys could try it out and send your comments,
> bug reports and fixes :) With the new repository layout it should be
> so much easier to contribute.

Once we migrate from 0.5.7 to 0.6.2 to "tests pass" stage, I will send
feedback how it went.

How do you build piqi-binary repo? Do you have a script that
cross-compiles everything, or you do it by hand and collect binaries? I
am asking, because nobody will allow us to put a pre-compiled binary to
production which is not signed with an official CentOS or SL repository
key or was not built internally. And I am looking for ways for me to
make this process as close to upstream, and easy of course.

I know it might sound stupid, but I would appreciate if you could sign
the tags you create in piqi-binary repository:

git tag --sign v0.6.2

Not a big deal to do, but will avoid some unnecessary questions and will
give a bit higher degree of confidence to the binaries.

> > New build scenario: assume you want piqi-rpc. Piqi-rpc depends on
> > piqi-erlang. Piqi-erlang has rebar.config.script, which checks if
> > "which piqi" returns a path. If "which piqi" returns error, we add
> > "piqi" as a dependency of piqi-erlang, and the compilation stuff will
> > be done like now.
>
> I think this would be a very useful addition. Especially for platforms
> for which we don't have pre-built piqi binaries. Now that repositories
> are separated, I'd rather have users build and install "piqi"
> executable separately and not include piqi_src as a dependency and
> have it compiled during "rebar compile".
>

> I can merge it with no problem. My only suggestion would be to
> parameterize it using some environment variable instead of just
> relying on "piqi" executable accessibility.

This will have to wait until I find a machine in our environment on
which pre-built binaries do not work out of the box. Then I will have a
reason to pick up this task.

Regards,
Motiejus

[1]:
https://help.ubuntu.com/community/GnuPrivacyGuardHowto#Uploading_the_key_to_Ubuntu_keyserver

[Anton Lavrik]

On Sun, Mar 17, 2013 at 7:53 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:
>> On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
>> >
>> That was roughly my plan as well. The difference is that I never have
>> to compile piqi executables from sources, because I always use one of
>> the released stable versions that includes the prebuilt piqi and piqic
>> executables.
>
> In the past these did not work with some of our CI machines due to older
> glibc version. Current piqi-binary works, but I am unsure why: either
> because of system upgrade (unlikely that it happened), or because you
> changed something in the binary build process, and it Just started working.
> Unlikely as well, I guess.

Of the top of my head, I don't remember any changes in my environment
that could contribute to it. I build 64-bit Linux version on Debian
Squeeze, and 32-bit Debian Lenny. I'm also using a fairly old 10.6.8
version of Mac OS X for Mac builds.

> Besides, maybe it is possible to fully statically compile piqi (i.e.
> include libc)? That would be awesome. I wouldn't be surprised if it
> worked with some other libc and could fully statically link.

I totally agree -- it would be fantastic to have. Unfortunately, as
far as I know, libc6 doesn't like static linking. I don't remember
specific details, but it is something related to it still trying to
dlopen some other libraries under the hood. There is a chance static
linking is still possible for such as simple program (and it should be
possible for sure if not with glibc but with other libc
implementations), but it is likely to turn into a massive project that
would involve digging into OCaml and things like that. I'm really not
looking forward to doing that any time soon. I'm getting enough
cross-platform headaches already without it (e.g. with windows
builds).

>> 3. Similarly to piqi-erlang, piqi-rpc codebase is now fully contained
>> at https://github.com/alavrik/piqi-rpc Naturally, it depends on
>> piqi-erlang but that's pretty much it. Oh, I also bumped Webmachine
>> version to 1.9.3
>>
>> 4. "piqic-erlang-ext" and therefore *_piqi_ext.erl go away.
>> "piqic-erlang" now includes serializers and deserializers for
>> multi-format converters (i.e. gen_X*/2, /3 and parse_X*/2, /3
>> functions) in *_piqi.erl.
>>
>> 5. "--gen-defaults" compiler option is deprecated. "piqic-erlang" now
>> generates default_X/1 without asking.
>>
>> I've been testing this new stuff for the last couple of weeks and I'm
>> pretty confident about it. It should be almost a drop-in replacement
>> for piqi-0.6.0. Only some minor makefile and code tweaking may be
>> necessary, e.g. renaming to _piqi_ext:gen_X calls to _piqi:gen_X.
>
> We are still on 0.5.7 and were waiting for default-field-values (not a
> big deal, because I can cherry-pick) and piqic-erlang rewrite (big
> deal). So now we will start migrating gradually I guess.
>
> Is default-field-values in the job pipeline? We are looking forward. :-)

Yes, it is. Implementing it efficiently in Erlang is fairly
straightforward. I'll try to include it into the upcoming release. You
are welcome :)

> How do you build piqi-binary repo? Do you have a script that
> cross-compiles everything, or you do it by hand and collect binaries? I
> am asking, because nobody will allow us to put a pre-compiled binary to
> production which is not signed with an official CentOS or SL repository
> key or was not built internally. And I am looking for ways for me to
> make this process as close to upstream, and easy of course.

I build it manually and it is not cross-compilation. I can build all 4
binaries on my laptop, but it takes about 30 minutes. I will be
definitely looking at automating this in the future.

It looks like if you build it separately and rpm-package it, it should
solve the problem of trusted builds for you. Automating unix builds is
easy. And you probably have only one or two Linux version that you
need to support. (Unfortunately, working with OCaml on Windows is
still very painful.)

> I know it might sound stupid, but I would appreciate if you could sign
> the tags you create in piqi-binary repository:
>
> git tag --sign v0.6.2
>
> Not a big deal to do, but will avoid some unnecessary questions and will
> give a bit higher degree of confidence to the binaries.

Hmm. I've never actually gotten to maintaining a GPG key. I can look
at this if you end deciding to use piqi-binary in your environment.

By the way, at Alert Logic, we do verification of every dependency
based on its commit sha1 (we have some very simple but very useful
scripts for doing that). It gives us integrity guarantees. But we
never verify tags for trust. As far as I can tell, most of open-source
repos do not use signed tags anyway.

Anton

[Motiejus Jakštys]

On Mon, Mar 18, 2013 at 12:53:46AM +0000, Motiejus Jakštys wrote:
> On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:
> > On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> Building piqi fails for me since 0.6.0 (just noticed), so I can't really
> try. I will send you a proper bug report sometime soon.

Posted[1].

Is github the right place to register issues, or is it better to post
them here?

> > I can merge it with no problem. My only suggestion would be to
> > parameterize it using some environment variable instead of just
> > relying on "piqi" executable accessibility.
>
> This will have to wait until I find a machine in our environment on
> which pre-built binaries do not work out of the box. Then I will have a
> reason to pick up this task.

Not true anymore. We decided to take this seriously and make the
packages. But first I must be able to compile it[1] and run tests on my
machine.

Also, /tests/README asks to source setenv.sh, but /INSTALL doesn't. Is
this correct? Looks inconsistent, it is/isn't?

Motiejus

[1]: https://github.com/alavrik/piqi/issues/25

[Anton Lavrik]

On Wed, Mar 20, 2013 at 7:58 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Mon, Mar 18, 2013 at 12:53:46AM +0000, Motiejus Jakštys wrote:
>> On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:
>> > On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
>> Building piqi fails for me since 0.6.0 (just noticed), so I can't really
>> try. I will send you a proper bug report sometime soon.
>
> Posted[1].
>
> Is github the right place to register issues, or is it better to post
> them here?

Commented on the problem there.

GitHub issues is the right place. I missed the fact that you opened
the issue, because GitHub for some reason changed my feed's
authentication token.

>> > I can merge it with no problem. My only suggestion would be to
>> > parameterize it using some environment variable instead of just
>> > relying on "piqi" executable accessibility.
>>
>> This will have to wait until I find a machine in our environment on
>> which pre-built binaries do not work out of the box. Then I will have a
>> reason to pick up this task.
>
> Not true anymore. We decided to take this seriously and make the
> packages.

Looking forward to it!

> Also, /tests/README asks to source setenv.sh, but /INSTALL doesn't. Is
> this correct? Looks inconsistent, it is/isn't?

I agree, it is inconsistent. I don't think it is a big deal, just
didn't have time to update all the tests. I will fix it eventually.


Anton

[Motiejus Jakštys]

On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:

> On Sun, Mar 17, 2013 at 6:41 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> > I will do this only if it makes sense for you and there is a remote
> > chance of it being merged. Otherwise I do not want to maintain this
> > fork forever. :-)
>
> I can merge it with no problem. My only suggestion would be to
> parameterize it using some environment variable instead of just
> relying on "piqi" executable accessibility.

Hi,

one of the main goals to package `piqi` is to legally put it to under
/usr/bin/piqi. That way absolutely no knowledge of deps directory path
is required.

I propose two things:
1. Do not download piqi-binary if `piqi` can be found in path.
2. Put `which piqi` in piqic_erlang:find_piqi_executable().

I think we should gradually make the package installation from
repositories a `standard practice`. Thoughts?

Motiejus

[1]: https://launchpad.net/~motiejus/+archive/piqi

[Anton Lavrik]

This is already implemented. The runtime doesn't require piqi-binary
if the "PIQI" environment variable is defined or the "piqi" executable
can be found in path. A couple of days ago, I added the last missing
piece that skips the "piqi-binary" rebar dependency if "PIQI" is
defined: https://github.com/alavrik/piqi-erlang#advanced-installation

As you can see, during installation, it is a little bit more explicit
than what you described. But I don't expect it to be a problem. In
this case, requiring explicit choice is a good thing.

> [1]: https://launchpad.net/~motiejus/+archive/piqi

Nice! I'll try to look at the DEBs and the rpm packaging over the weekend.

Anton

[Motiejus Jakštys]

On Sat, Apr 13, 2013 at 01:59:14AM -0500, Anton Lavrik wrote:
> On Fri, Apr 12, 2013 at 12:06 PM, Motiejus Jakštys
> <desir...@gmail.com> wrote:
> > On Sun, Mar 17, 2013 at 06:38:04PM -0500, Anton Lavrik wrote:
> >
> > I think we should gradually make the package installation from
> > repositories a `standard practice`. Thoughts?
>

> As you can see, during installation, it is a little bit more explicit
> than what you described. But I don't expect it to be a problem. In
> this case, requiring explicit choice is a good thing.

Ah, I see. It is there. I missed the part `os:find_executable(piqi)`
somehow. Thanks. Then it all seems to be OK.

Motiejus