extending built-in piqi types

[Motiejus Jakštys]

Hi, Anton,

I am trying to understand piqi extension language. I want to a top-level type "endpoint", a list of strings.

piqi.endpoint
-------------
.include [ .module piqi ]
.extend [
.typedef piqi
.with.field [
.name endpoint
.type string
.repeated
]
]

demo.piqi
---------
.endpoint [
"foo"
"bar"
]

$ piqi cc piqi.endpoint -o piqi.piqi
$ piqi check demo.piqi
Warning: demo.piqi:1:1: unknown field: endpoint

What am I doing wrong? Do I misunderstand the extension mechanism or the 'how to include files' mechanism?

Is this the correct approach to extend the piqi language (i.e. create piqi.piqi *locally* and import that)? If it is completely wrong, can you point me to an example where a *top level* type is added outside of piqi repository?

I am experimenting with a new way to define our interfaces.. Might ask more (hopefully lesser stupid) questions like that.

Thanks,
Motiejus

[Anton Lavrik]

Hi Motiejus,

You were on the right track. The interfaces for what you are trying to do are very explicit. There is no magic.

Here is the correct sequence of commands:

$ piqi cc  > piqi.piqi

$ piqi cc -t pb -o endpoint.piqi.pb piqi.endpoint.piqi

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi 

Also, you demo.piqi doesn't match the definition of the endpoint field in piqi.endpoint.piqi. This is the correct one:

demo.piqi

--------------

.endpoint "foo"

.endpoint "bar"

% alternatively, you can use a shorter syntax: (.endpoint "foo" "bar")

.custom-field endpoint  % suppressing the unknown field warning

--------------

With this, you should be able to get the desired output:

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi 

:piqi [

    .module demo

    .custom-field endpoint

    .endpoint "foo"

    .endpoint "bar"

    .file "demo.piqi"

]

A couple of comments.

Top-level extensions have always been supported. erlang-module from piqi.erlang.piqi is a good example.

"piqi cc" and "piqi compile" are meant to be used together and their only purpose is to work with language extensions. They are useless otherwise.

To understand what is going on under the hood, especially with regard to imports/includes you can run commands with --trace flag (well, to some extent...).

If you want portable output, call "piqi compile" with -o json|xml|pb ...

I don't mind questions at all. Besides, this is an advanced use case and interfaces are not really documented.

Anton

[Motiejus Jakštys]

2014.08.12 11:26, Anton Lavrik rašė:
> Here is the correct sequence of commands:
>
> $ piqi cc > piqi.piqi

$ piqi cc -o piqi.endpoint.piqi piqi.endpoint # missed this

> $ piqi cc -t pb -o endpoint.piqi.pb piqi.endpoint.piqi
> $ piqi compile --self-spec endpoint.piqi.pb demo.piqi

$ piqi cc > piqi.piqi

Put built-in piqi specification (NOT language) to a file. This has default extensions (protobuf, json).

$ piqi cc -o piqi.endpoint.piqi piqi.endpoint

Concatenate piqi.endpoint and piqi.piqi. Make an extended piqi definition here.

$ piqi cc -t pb -o endpoint.piqi.pb piqi.endpoint.piqi

Make protobuf version of extended piqi definition.

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi -o demo.pb

So this is the xml|json|pb|piqi version of my demo.piqi. To prove a point, I could load it from C++:
* $ piqi to-proto piqi.endpoint.piqi
* piqi.endpoint.piqi.proto is the definition of the language (compile C++ files from it)
* demo.piqi is the protobuf payload.

Now the question is: if I want to extend piqic-erlang to generate a constant with a list in a header file, I have to modify piqic-erlang, in particular src/piqic_erlang_types.erl.

Is my understanding sort of correct?

> "piqi cc" and "piqi compile" are meant to be used together and their only purpose is to work with language extensions. They are useless otherwise.

`piqi cc` manipulates the (extended) language.
`piqi compile` manipulates the payload in a given extended language.

Is my understanding correct?

> To understand what is going on under the hood, especially with regard to imports/includes you can run commands with --trace flag (well, to some extent...).

'--trace' only prints when it's searching for included modules and trying to resolve symbols.. Useful, but doesn't help understanding the flow.

If my assumptions about piqic-erlang are correct, it will keep me busy for a while (or at least evaluate that we are not going this route). Thanks!

Motiejus

[Anton Lavrik]

On Tue, Aug 12, 2014 at 4:28 AM, Motiejus Jakštys <desir...@gmail.com> wrote:

2014.08.12 11:26, Anton Lavrik rašė:

> Here is the correct sequence of commands:
>
> $ piqi cc  > piqi.piqi

$ piqi cc -o piqi.endpoint.piqi piqi.endpoint  # missed this

> $ piqi cc -t pb -o endpoint.piqi.pb piqi.endpoint.piqi
> $ piqi compile --self-spec endpoint.piqi.pb demo.piqi

$ piqi cc  > piqi.piqi

Put built-in piqi specification (NOT language) to a file. This has default extensions (protobuf, json).

$ piqi cc -o piqi.endpoint.piqi piqi.endpoint

Concatenate piqi.endpoint and piqi.piqi. Make an extended piqi definition here.

$ piqi cc -t pb -o endpoint.piqi.pb piqi.endpoint.piqi

Make protobuf version of extended piqi definition.

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi

$ piqi compile --self-spec endpoint.piqi.pb demo.piqi -o demo.pb

So this is the xml|json|pb|piqi version of my demo.piqi. To prove a point, I could load it from C++:
* $ piqi to-proto piqi.endpoint.piqi
* piqi.endpoint.piqi.proto is the definition of the language (compile C++ files from it)
* demo.piqi is the protobuf payload.

Correct and correct.

 

Now the question is: if I want to extend piqic-erlang to generate a constant with a list in a header file, I have to modify piqic-erlang, in particular src/piqic_erlang_types.erl.

Is my understanding sort of correct?

Well, after you've reached this step, it is totally up to you what to do with the extensions.

 

> "piqi cc" and "piqi compile" are meant to be used together and their only purpose is to work with language extensions. They are useless otherwise.

`piqi cc` manipulates the (extended) language.
`piqi compile` manipulates the payload in a given extended language.

Is my understanding correct?

Yes. 'piqi cc' stands for compiler compiler. It compiles (extended) piqi self-spec which is, in turn, used to compile other piqi specs (by 'piqi compile').

Anton