finding the module of non-qualified types

[Motiejus Jakštys]

Hi!

I've been playing around with generating documentation from piqi files.
Whenever I encounter a type while parsing the piqi structure, I want to create
a hyperlink to where the type is defined.

Problem: when I get a type without its module (eg. '.type time'[1]), I:

1. The type is a built-in to piqi lang: link to 'piqi' module, type 'time'.
2. The type is in local module; link to 'current module', type 'time'.

My workaround would be to extract the list of built-in types in piqi-lang and,
whenever I find an unqualified type, do a simple lookup: is it in my module or
piqi-lang.

Is this the right approach, or I should be working at another layer of
abstraction? Here[2] you can also see which files I`m parsing.

Question 2.

I want to convert my type definitions to XML. I can do it like this:

piqi compile --self-spec doc.piqi.pb demo.piqi -t xml -o -

But, if I already have demo.piqi.pb[2], why doesn't this work?

% piqi convert -f pb -t xml --type piqi demo.piqi.pb -o -
unknown:0:0: piqi module name can not be derived from the file name

Thanks,
Motiejus

[1]: https://github.com/Motiejus/piqigames/blob/f3f33ce56ab571cb515d149c2ec648d84afaa0f7/definition/demo.piqi#L19
[2]: https://github.com/Motiejus/piqigames/blob/f3f33ce56ab571cb515d149c2ec648d84afaa0f7/definition/Makefile#L17

[Motiejus Jakštys]

On 2014.09.28 22:42, Motiejus Jakštys wrote:
> My workaround would be to extract the list of built-in types in piqi-lang and,
> whenever I find an unqualified type, do a simple lookup: is it in my module or
> piqi-lang.

I realized I didn't state my problem clearly enough.

Ideally, while parsing the protobuf structure (demo.piqi.pb), I would like to know which module every type comes from (in the example, I don't know where is demo/echo/output/zeit defined). The only way to know now where is 'zeit' defined is to do the above workaround. Any alternative suggestions?

Not saying it's very hard or I can't do it; just want to make sure I am doing it right.

I hope it's clear now.

Best,
Motiejus

[Anton Lavrik]

On Sun, Sep 28, 2014 at 1:49 PM, Motiejus Jakštys <desir...@gmail.com> wrote:

On 2014.09.28 22:42, Motiejus Jakštys wrote:
> My workaround would be to extract the list of built-in types in piqi-lang and,
> whenever I find an unqualified type, do a simple lookup: is it in my module or
> piqi-lang.

Hi Motiejus, 

 

Yes, it is the right approach overall. All piqi implementations do it this way. Only thing, it is better to build the map of built-in types first, as not all types defined in piqi.piqi (i.e. piqi spec) are considered built-in types. Here is the logic implemented in Erlang: https://github.com/alavrik/piqi-erlang/blob/master/src/piqic.erl#L114

Question 2.

I want to convert my type definitions to XML. I can do it like this:

    piqi compile --self-spec doc.piqi.pb demo.piqi -t xml -o -

But, if I already have demo.piqi.pb[2], why doesn't this work?

     % piqi convert -f pb -t xml --type piqi demo.piqi.pb -o -
    unknown:0:0: piqi module name can not be derived from the file name

"piqi compile" produces what I call "piqi bundle" instead of standalone piqi.  Bundle is simply a list of piqi modules containing the module being compiled and all of its dependencies, i.e. imported modules. Conversion will start working if you use piqi/piqi-list as a type:

piqi convert -f pb -t xml --type piqi/piqi-list -o - demo.piqi.pb

Take a look at what piqic:init_context() function does in the above link. You'll probably want to do something similar in your documentation generator. Or maybe you can take a shortcut for such a simple use case.

Cheers,

Anton

[Motiejus Jakštys]

On 2014.09.29 10:09, Anton Lavrik wrote:
> Hi Motiejus,
>
> Yes, it is the right approach overall. All piqi implementations do it
> this way. Only thing, it is better to build the map of built-in types
> first, as not all types defined in piqi.piqi (i.e. piqi spec) are
> considered built-in types. Here is the logic implemented in Erlang:
> https://github.com/alavrik/piqi-erlang/blob/master/src/piqic.erl#L114
>

Hi, Anton,

thanks, that clarifies a bit.

In piqic:load_self_spec/0 I see you are getting the self-spec from
piqi_piqi.erl:

PiqiBin = hd(piqi_piqi:piqi()),

In my case, self-spec is in the pb file iff I '.import [ .module piqi ]' in the
definition file[2]. Otherwise there is no self-spec (which I believe is the pb
of piqi?) and therefore am unable to get information you are suggesting.

Is there a way in the compile chain to force-include self-spec to demo.piqi.pb[1]
without needing to include `piqi` module in the definition file[2]?

Reason: I want to take care of the compilation chain, but let the user write
the definitions (and extended language maybe) without the need to worry about
compilation.

Or can I get the piqi self-spec from somewhere else, maybe from piqi binary
directly?

> Take a look at what piqic:init_context() function does in the above
> link. You'll probably want to do something similar in your
> documentation generator. Or maybe you can take a shortcut for such a
> simple use case.

I want to understand piqi better, so taking shortcuts goes against the purpose.
:-) This is a personal side project.

Regards,
Motiejus

[1]: https://github.com/Motiejus/piqigames/blob/8ac7cd11b510c88435f762eaec25023352d36c79/definition/Makefile#L17
[2]: https://github.com/Motiejus/piqigames/blob/8ac7cd11b510c88435f762eaec25023352d36c79/definition/demo.piqi#L2

[Anton Lavrik]

On Tue, Sep 30, 2014 at 12:59 PM, Motiejus Jakštys <desir...@gmail.com> wrote:

On 2014.09.29 10:09, Anton Lavrik wrote:
> Hi Motiejus,
>
> Yes, it is the right approach overall. All piqi implementations do it
> this way. Only thing, it is better to build the map of built-in types
> first, as not all types defined in piqi.piqi (i.e. piqi spec) are
> considered built-in types. Here is the logic implemented in Erlang:
> https://github.com/alavrik/piqi-erlang/blob/master/src/piqic.erl#L114
>

Hi, Anton,

thanks, that clarifies a bit.

In piqic:load_self_spec/0 I see you are getting the self-spec from
piqi_piqi.erl:

    PiqiBin = hd(piqi_piqi:piqi()),

In my case, self-spec is in the pb file iff I '.import [ .module piqi ]' in the
definition file[2]. Otherwise there is no self-spec (which I believe is the pb
of piqi?) and therefore am unable to get information you are suggesting.

Is there a way in the compile chain to force-include self-spec to demo.piqi.pb[1]
without needing to include `piqi` module in the definition file[2]?

 

Why would you want to do something like that? It feels very unnatural to include language definition in your application module -- regardless of whether it is automated or manual. Wouldn't be more reasonable to keep and handle language syntax definition separately from whatever you want to define in this language?

Reason: I want to take care of the compilation chain, but let the user write
the definitions (and extended language maybe) without the need to worry about
compilation.

Or can I get the piqi self-spec from somewhere else, maybe from piqi binary
directly?

You can just carry the compiled self-spec file around (which is, as you correctly assumed, just a pb of itself). Or you can have it embedded in your code and given to "piqi compile" when needed, like piqic-erlang does. You can also pass it to stdin if you want by specifying "piqi compile --self-spec -". 

Perhaps I am missing something. Can you give a specific example? You still have to call "piqi compile" from your compilation chain. And after that, you load the result into a program, like piqic-erlang that is supposed to understand the extended syntax of what you are loading. Am I right or you are doing something different here?

I want to understand piqi better, so taking shortcuts goes against the purpose.
:-) This is a personal side project.

Let's dig in then :)

Anton 

[Motiejus Jakštys]

On 2014.10.01 12:15, Anton Lavrik wrote:
>
> On Tue, Sep 30, 2014 at 12:59 PM, Motiejus Jakštys

> <desir...@gmail.com <mailto:desir...@gmail.com>> wrote: On

> 2014.09.29 10 <tel:2014.09.29%2010>:09, Anton Lavrik wrote:
>
> Why would you want to do something like that? It feels very
> unnatural to include language definition in your application module
> -- regardless of whether it is automated or manual. Wouldn't be more
> reasonable to keep and handle language syntax definition separately
> from whatever you want to define in this language?

I agree. But I am looking for a way *not* to keep self-spec in the repository in any way; be it in piq, piqi or pb. piqi (the compiled binary) is already a dependency. My question was: maybe it has a self-spec embedded, so I can get it like

$ piqi self-spec -t pb | piqi compile --self-spec - <...>

> You can just carry the compiled self-spec file around (which is, as
> you correctly assumed, just a pb of itself). Or you can have it
> embedded in your code and given to "piqi compile" when needed, like
> piqic-erlang does. You can also pass it to stdin if you want by
> specifying "piqi compile --self-spec -".

If piqi (the compiled binary) does not have self-spec built in, I`ll carry it around. I guess I need more time to understand the rationale why it doesn't expose and/or have it.

> Perhaps I am missing something. Can you give a specific example? You
> still have to call "piqi compile" from your compilation chain. And
> after that, you load the result into a program, like piqic-erlang
> that is supposed to understand the extended syntax of what you are
> loading. Am I right or you are doing something different here?

You see it correctly. The problem I am trying to solve is just "can I hand over maintenance of self-spec to something else, most conveniently piqi binary which is already a dependency?"

You answered the questions for now, unless piqi (binary) does indeed have a self-spec accessible. :-)

Regards,
Motiejus

[Motiejus Jakštys]

On Wed, Oct 1, 2014 at 10:49 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On 2014.10.01 12:15, Anton Lavrik wrote:
>>
>> On Tue, Sep 30, 2014 at 12:59 PM, Motiejus Jakštys
>> <desir...@gmail.com <mailto:desir...@gmail.com>> wrote: On
>> 2014.09.29 10 <tel:2014.09.29%2010>:09, Anton Lavrik wrote:
>>
>> Why would you want to do something like that? It feels very
>> unnatural to include language definition in your application module
>> -- regardless of whether it is automated or manual. Wouldn't be more
>> reasonable to keep and handle language syntax definition separately
>> from whatever you want to define in this language?
>
> I agree. But I am looking for a way *not* to keep self-spec in the repository in any way; be it in piq, piqi or pb. piqi (the compiled binary) is already a dependency. My question was: maybe it has a self-spec embedded, so I can get it like

[...]

> You answered the questions for now, unless piqi (binary) does indeed have a self-spec accessible. :-)

I`m sorry, pardon my ignorance. `piqi cc` is exactly what I was
looking for. Let's close this thread now. :-)

--
Motiejus Jakštys

[Anton Lavrik]

I was about to suggest it but wasn't sure if this is what you were looking for, because you have already used it in your Makefile :)

This maybe somewhat relevant. I already keep generated piqi stubs for self-specs in the repositories (piqi, piqi-ocaml, piqi-erlang). There is no reasonable way around it.

I also keep the binary self-specs (e.g. piqi_piqi:piqi()). It may not look very nice, but there is no harm otherwise because the self-spec format is stable.

However, at least for OCaml and Erlang it is possible to store them as actual code. For instance, for Erlang, you can print self-spec as a term: io:format("~p", [piqi_piqi:parse_piqi(piqi_piqi:piqi())] and write it to piqi_boot.erl Later, you can take this term and serialize it back to pb if needed. I do it with OCaml self-spec here: https://github.com/alavrik/piqi/blob/master/piqilib/piqi_boot.ml

Anton