xml -> piq ":piqi" wrapper

[Motiejus Jakštys]

Hi,

I am trying to exploit piqi self-definition and define a
datatype/interface in XML. Having problems. Assume time.xml:
<<<<<<<<<
<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>time</module>
<typedef>
<record>
<name>date</name>
<field><name>year</name><type>int</type><mode>required</mode></field>
<field><name>month</name><type>int</type><mode>required</mode></field>
<field><name>day</name><type>int</type><mode>required</mode></field>
</record>
</typedef>
</value>
>>>>>>>>>

Conveting to time.piqi:
<<<<<<<<<<<
$ piqi convert --type piqi -f xml -t piq time.xml -o time.piqi
$ cat time.piqi
:piqi [
.module time
.record [
.name date
.field [
.name year
.type int
]
.field [
.name month
.type int
]
.field [
.name day
.type int
]
]
]
>>>>>>>>>

The bad thing is that I can't use it with piqic-erlang:
<<<<<<<<<<
$ piqic-erlang time.piqi
Warning: time.piqi:1:1: unknown field: :piqi [ ...
Warning: unknown:0:0: unknown field: file
Warning: time.piqi:1:1: unknown field: :piqi [ ...
$

Does what I am trying to do make sense? If yes, which place is best to
"make this work"?

Thanks,
Motiejus

[Motiejus Jakštys]

On Tue, Nov 26, 2013 at 8:18 PM, Motiejus Jakštys <desir...@gmail.com> wrote:
> Hi,

> The bad thing is that I can't use it with piqic-erlang:
> <<<<<<<<<<
> $ piqic-erlang time.piqi
> Warning: time.piqi:1:1: unknown field: :piqi [ ...
> Warning: unknown:0:0: unknown field: file
> Warning: time.piqi:1:1: unknown field: :piqi [ ...
> $
>
> Does what I am trying to do make sense? If yes, which place is best to
> "make this work"?
>

Note: if I unwrap everything inside :piqi [ ... ] (leave only the
dots), piqic-erlang stuff generates the way I want.

I would also like to make writing-piqi-in-xml workable with extensions
(not sure what it means yet).

--
Motiejus Jakštys

[Anton Lavrik]

This is a great question.

The reason why you couldn't achieve this easily is that, in general, there could be several piqi modules embedded in xml or any other supported data format (except for Protocol Buffers that doesn't have a notion of framing and can't hold more than one object).

Eventually, I was planning to write packing and unpacking tools for embedded Piqi modules. For instance, the unpacking tool would extract all embedded modules and create individual .piqi files according to their module names. Keep in mind that it is typical for embedded modules to have their dependencies (i.e. imported modules) embedded as well.

In the meantime, I've added a couple of experimental "piqi convert" parameters to the "frameless" branch: https://github.com/alavrik/piqi/tree/frameless

For instance, if you add "--piq-frameless-output true" to your "piqi convert" command it will skip the frame and produce the desired output.

Anton

--
You received this message because you are subscribed to the Google Groups "piqi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to piqi+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Motiejus Jakštys]

On Wed, Nov 27, 2013 at 11:31 AM, Anton Lavrik <ala...@piqi.org> wrote:
> In the meantime, I've added a couple of experimental "piqi convert"
> parameters to the "frameless" branch:
> https://github.com/alavrik/piqi/tree/frameless
>
> For instance, if you add "--piq-frameless-output true" to your "piqi
> convert" command it will skip the frame and produce the desired output.

Thanks! I am not 100% sure how this hack (is it?) fits piqi long-term
goals, but the current hack works perfectly for my purposes.

Motiejus

[Anton Lavrik]

This is definitely useful on its own. In fact, I've been using frameless piq (sort of --piq-frameless-input) for a DSL and it works amazingly well. Especially when coupled with --piq-relaxed-input. Can't think of any other abstract syntax representation that would be as concise and expressive as piq. Another application for frameless piq would be using it as a config file format.

I'm going to expose these options through "piqi server" and, after that, merge the whole thing into master.

Anton

[Ignas Vyšniauskas]

Hi Anton,

On 11/28/2013 11:16 AM, Anton Lavrik wrote:
> This is definitely useful on its own. In fact, I've been using
> frameless piq (sort of --piq-frameless-input) for a DSL and it works
> amazingly well. Especially when coupled with --piq-relaxed-input.

Could you elaborate on what piq-relaxed-input does? Couldn't figure it
out from a quick glance at the diff.

A short example would be great.

Thanks,
Ignas

[Anton Lavrik]

Hi Ignas, 

With --piq-relaxed-input you can omit quotes around single-word string literals. For example, if you have a string field like thins one:

.field [

    .name s

    .type string

]

in piq input, you'd normally need to write

.s "foo"

however, with --piq-relaxed-input, you can just write

.s foo

Without --piq-relaxed-input the same input would trigger a "quoted string literal expected instead of word" error.

I may later add a couple of other relaxed parsing rules such as treating ".foo" as ".foo true" for boolean fields.

Anton

[Anton Lavrik]

On Wed, Nov 27, 2013 at 2:31 AM, Anton Lavrik <ala...@piqi.org> wrote:

In the meantime, I've added a couple of experimental "piqi convert" parameters to the "frameless" branch: https://github.com/alavrik/piqi/tree/frameless

For instance, if you add "--piq-frameless-output true" to your "piqi convert" command it will skip the frame and produce the desired output.

This is in master now. Works in command-line mode and also from OCaml and Erlang.

Anton

[Ignas Vyšniauskas]

Hi Anton,

I would like to exploit this functionality to create a "sugared" XML
version of piqi definitions which I would then desugar to the raw piqi
definition language in XML and then proceeded pretty much as Motiejus
described to generate actual .piqi files.

However I'm bumping into some problems. E.g. how do I define top-level
properties such as `.erlang-type-prefix` or `.custom-field` in the XML
version? I tried reverse-engineering by converting a .piqi file to XML,
but they seem to be simply stripped from the XML, example here:
https://gist.github.com/yfyf/7785654

Obviously I could just desugar directly to .piqi instead of XML, but
XML->XML transformations are much easier to implement.

Thanks,
Ignas

[Ignas Vyšniauskas]

Btw, one obvious hack would be to add an element like:

<piqi-verbatim>add_any_piqi_stuff_here</piqi-verbatim>

Not sure what your philosophy of unstructured extensions like this is.

--
Ignas

[Anton Lavrik]

Hi Ignas,

This is a great question. There's an established way to do what you've described going from Piqi to XML. However I've never imagined people wanting it to do the way around -- by generating Piqi from XML, so a couple of pieces are missing here and there. In other words you can't do exactly that right now however the general framework is there and it can be supported with some work.

Let me show how to convert Piqi to XML while preserving erlang-specific properties based on your example. This is essentially the mechanism used by piqic-erlang.

1. First, we'll need to have the original piqi self-spec ready. This is easy:

    piqi cc -o piqi.piqi

2. Define the extensions you want the self-spec to have. In this case, we are using piqi.erlang.piqi taken from piqi-erlang/src/ It defines Erlang specific extensions for the self-spec, e.g. erlang-name, erlang-module, erlang-type-prefix and so on.

    cp piqi-erlang/src/piqi.erlang.piqi .

3. From the self-spec and the set of defined extensions, we need to generate an extended self-spec. This is achieved by running

    piqi cc -e erlang -t pb -o piqi.piqi.pb piqi.piqi

Note, we need the resulting extended spec to be in protobuf format for the next step, this is why -t pb -o piqi.piqi.pb

4. Finally, converting example.piqi to XML by using the extended self-spec in order to preserve our custom properties:

$ piqi compile --self-spec piqi.piqi.pb -t xml example.piqi 

<?xml version="1.0" encoding="UTF-8"?>

<value>

  <piqi>

    <module>example</module>

    <typedef>

      <alias>

        <name>itemid</name>

        <type>uint32</type>

      </alias>

    </typedef>

    <typedef>

      <record>

        <name>foobar</name>

        <field>

          <type>itemid</type>

          <mode>required</mode>

          <code>1</code>

        </field>

      </record>

    </typedef>

    <file>example.piqi</file>

    <erlang-type-prefix></erlang-type-prefix>

    <erlang-string-type>binary</erlang-string-type>

  </piqi>

</value>

As you can see "erlang-type-prefix" was generated in the output.

Calling "piqi compile" in step 4 is exactly what piqic-erlang does. "piqi compile" is similar to "piqi convert". The difference is that it works only on Piqi specs (equivalent to  "piqi convert --type piqi") and also accepts an extra --self-spec parameter for the extended self-spec.

If you are wondering about steps 1-3, they are taken care of automatically (after piqic-erlang is bootstrapped) as piqic-erlang puts the extended self-spec in .pb format in the generated Erlang code. It is returned by piqi_piqi:piqi().

Just one more thing to keep in mind. Portable Piqi spec (i.e. represented in one of XML/PB/JSON) is actually a subset of the Piqi language. What it means is that some Piqi language features such as 'include', 'extend' and the ability to define function parameter types in-place are not available in the portable specs (all of these features are syntax based and expanded by piqi automatically when converted to portable specs).

Anton

[Ignas Vyšniauskas]

Hi Anton,

Thanks for the extensive reply!

It's early in the morning, so I hope I understood this right: you are
simply saying that piqi<->portable-specs transformations are not
invertible because you never intended them to be such, correct?

Currently this is not a blocker for me, but in the future I would
definitely be interested in such functionality. What kind of changes
would need to be made to enable this? I've been meaning to hack on some
ML for some time now, so this might be a nice opportunity to do so. But
you seem to be also hinting that this is a rather complicated change, so
perhaps it's not viable.

On 12/08/2013 12:20 AM, Anton Lavrik wrote:
> Just one more thing to keep in mind. Portable Piqi spec (i.e.
> represented in one of XML/PB/JSON) is actually a subset of the Piqi
> language. What it means is that some Piqi language features such as
> 'include', 'extend' and the ability to define function parameter
> types in-place are not available in the portable specs (all of these
> features are syntax based and expanded by piqi automatically when
> converted to portable specs).

Hmm, so would it not be possible to define purely syntactic
transformations (i.e. grammar based) between the portable specs and
piqi, ensuring they are invertible?

Alternatively, you can at least ensure they are bidirectional (meaning
that the transformations will not be exact inverses, but you are able to
at least use all piqi features in whatever representation language), by
doing something like this:

Input some piqi defs in XML -> syntax-based transformation to .piqi ->
perform whatever the custom stuff needs to be done (syntax expansions,
etc) -> syntax-based transformation to output XML.

Obviously some target languages are not very expressive, so you might
need some smart embedding, but in theory this should still work, no?

Best,
Ignas

[Anton Lavrik]

On Tue, Dec 10, 2013 at 1:43 AM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

Hi Anton,

Thanks for the extensive reply!

It's early in the morning, so I hope I understood this right: you are
simply saying that piqi<->portable-specs transformations are not
invertible because you never intended them to be such, correct?

Hi Ignas, 

Correct. Note the lack of reversibility of piqi -> spec -> piqi' conversion affects only a subset of syntax elements. The semantics is fully reversible. For instance, "piqic-erlang" will generate exactly the same output for piqi and piqi'. In fact, it performs piqi -> spec (piqi.pb) conversion under the hood. Non-reversible syntax elements are kind of like macros in a sense that their expansion is one-way. There is also "piqi expand" that can perform the expansions selectively.

Currently this is not a blocker for me, but in the future I would
definitely be interested in such functionality. What kind of changes
would need to be made to enable this? I've been meaning to hack on some
ML for some time now, so this might be a nice opportunity to do so. But
you seem to be also hinting that this is a rather complicated change, so
perhaps it's not viable.

It is similar to the existing code for handling self-specs. In fact, it could be even simpler. I'll be happy help if you decide to hack on it.

 

On 12/08/2013 12:20 AM, Anton Lavrik wrote:
> Just one more thing to keep in mind. Portable Piqi spec (i.e.
> represented in one of XML/PB/JSON) is actually a subset of the Piqi
> language. What it means is that some Piqi language features such as
> 'include', 'extend' and the ability to define function parameter
> types in-place are not available in the portable specs (all of these
>  features are syntax based and expanded by piqi automatically when
> converted to portable specs).

Hmm, so would it not be possible to define purely syntactic
transformations (i.e. grammar based) between the portable specs and
piqi, ensuring they are invertible?

Alternatively, you can at least ensure they are bidirectional (meaning
that the transformations will not be exact inverses, but you are able to
at least use all piqi features in whatever representation language), by
doing something like this:

Input some piqi defs in XML -> syntax-based transformation to .piqi ->
perform whatever the custom stuff needs to be done (syntax expansions,
etc) -> syntax-based transformation to output XML.

Obviously some target languages are not very expressive, so you might
need some smart embedding, but in theory this should still work, no?

Hmm... you are right!  Pure syntax-based conversion of it is possible even today. It won't be as portable as piqi spec, because the Piqi language may change (unlike piqi spec), and the output .piqi may not look very pretty but I can't see any reason why it wouldn't work.

All you need to do is to take piqi-lang.piqi [1], put piqi.piqi and all your custom extensions next to it like in the example from my previous reply, and then call "piqi convert" as follows:

$ piqi convert -e erlang --type piqi-lang/piqi --piq-frameless-output true example.xml

.module example

.typedef.alias [

    .name itemid

    .type uint32

]

.typedef.record [

    .name foobar

    .field [

        .type itemid

        .mode.required

        .code 1

    ]

]

.file "example.piqi"

.erlang-type-prefix ""

.erlang-string-type.binary

$ cat example.xml 

<?xml version="1.0" encoding="UTF-8"?>

<value>

  <module>example</module>

  <typedef>

    <alias>

      <name>itemid</name>

      <type>uint32</type>

    </alias>

  </typedef>

  <typedef>

    <record>

      <name>foobar</name>

      <field>

        <type>itemid</type>

        <mode>required</mode>

      </field>

    </record>

  </typedef>

  <erlang-type-prefix></erlang-type-prefix>

  <erlang-string-type>binary</erlang-string-type>

</value>

Anton

[1] https://github.com/alavrik/piqi/blob/master/piqi/piqi-lang.piqi

[Anton Lavrik]

On Thu, Dec 12, 2013 at 12:54 PM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

Hi Anton,

Thanks. This looks great and exactly what I need, I'm kind of impressed
with the flexibility of extensions. I will try this out tomorrow.

By the way, today I realised that one can consider piqi, among other
things, as a language-agnostic module system, which IMHO is not
implemented well in any other project and is a very useful feature.

Thanks. Yes, the module system is a very important piece and I agree that it is broken in many other languages. To be honest, considering how minimalistic Piqi is it was rather straightforward to come up with a reasonable base module system for it (by base I mean imports). For instance, programming languages have other use cases for their module systems and associated challenges that Piqi doesn't have to worry about.

I'd like to expand a little bit on that. The hardest part was handling includes with extensions, i.e. when foo.<extension>.piqi are involved. In such cases, foo.<extension>.piqi and foo.piqi become mutually recursive, but a good thing is that Piqi handles this under the hood so you won't even notice it. If curious, you can run something like "piqi check --trace" on piqi.piqi and piqi.protobuf.piqi in the "piqi" subdirectory to see what is actually going on. I'm not going to even try documenting this any time soon :)

Another thing is that the module system is not complete yet. I'm planning to add dynamic extensions similar to what Protocol Buffers offer. Such extensions will enable some really important use-cases.

The overall goal for the piqi module system is to be able to handle a global public repository of piqi modules. By this point Piqi should be powerful enough to define structured layout for most practical L7 protocols which will hopefully put most of modern-day API (e.g. REST) and protocol definition practices (e.g. RFCs) out of existence. Oh well... one step at a time :) It is just nice to contemplate about it.

P.S. Did you mean to exclude the mailing list? Your reply is very useful
information that should be out there!

Just realized that and re-sent it to the list. Thanks!

Anton

[Ignas Vyšniauskas]

Hi Anton,

On 12/12/2013 10:13 PM, Anton Lavrik wrote:
> Hmm... you are right! Pure syntax-based conversion of it is possible
> even today. It won't be as portable as piqi spec, because the Piqi
> language may change (unlike piqi spec), and the output .piqi may not
> look very pretty but I can't see any reason why it wouldn't work.
>
> All you need to do is to take piqi-lang.piqi [1], put piqi.piqi and
> all your custom extensions next to it like in the example from my

> previous reply, and then call "piqi convert" as follows: <..>

Thanks once again -- this works. However the caveat is that now I also
have to further desugar the XML encoding of piqi specs into an XML
representation of piqi lang (and then apply my own transformations),
because extensions like .function are invalid in piqi-lang (which you
already mentioned) and hence also can't be used in the XML representation.

This makes things quite cumbersome in my case, because I need to do:
1. Sugared XML -> Desugar to XML encoding piqi specs
2. Convert the XML encoding piqi specs to XML encoding piqi lang
3. Perform further XSLT transformations on the XML
4. Convert back to piqi lang

In fact, I'm not even sure how to do step 2 (I guess it's possible to do
a round-trip XML -> piqi -> piqi-lang ->XML?).

By the way, feel free to just say "sorry, you're on your own from here",
you've been great help already!

--
Ignas

[Anton Lavrik]

On Fri, Dec 13, 2013 at 2:48 AM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

Hi Anton,

On 12/12/2013 10:13 PM, Anton Lavrik wrote:
> Hmm... you are right!  Pure syntax-based conversion of it is possible
> even today. It won't be as portable as piqi spec, because the Piqi
> language may change (unlike piqi spec), and the output .piqi may not
> look very pretty but I can't see any reason why it wouldn't work.
>
> All you need to do is to take piqi-lang.piqi [1], put piqi.piqi and
> all your custom extensions next to it like in the example from my

> previous reply, and then call "piqi convert" as follows: <..>

Thanks once again -- this works. However the caveat is that now I also
have to further desugar the XML encoding of piqi specs into an XML
representation of piqi lang (and then apply my own transformations),
because extensions like .function are invalid in piqi-lang (which you
already mentioned) and hence also can't be used in the XML representation.

I don't believe this is true. Functions are surely supported in piqi lang. What you write in .piqi *is* the Piqi language (piqi lang). Perhaps I should have clarified the use of terms of piqi lang and piqi spec. Piqi spec is a portable stable subset of Piqi lang. To be specific, Piqi lang is Piqi spec + 'extend', 'include', and the ability to define types of function parameters in-place (e.g. in .piqi this becomes valid: .function [ .input.record [ ... ] ] ... ] whereas in piqi spec you can only refer types by names, e.g. .function [ .input <some-record-name>  ... ])

Essentially, if my understanding of your use case is correct, you need only two steps to achieve what you want:

1. Sugared XML -> Desugar to XML-encoded piqi-lang

2. convert the XML-encoded piqi-lang to .piqi as shown by the example in my previous reply

Can you think of an example where it would't work?

Anton

[Ignas Vyšniauskas]

On 12/16/2013 03:26 AM, Anton Lavrik wrote:
> Essentially, if my understanding of your use case is correct, you
> need only two steps to achieve what you want:
>
> 1. Sugared XML -> Desugar to XML-encoded piqi-lang 2. convert the
> XML-encoded piqi-lang to .piqi as shown by the example in my
> previous reply

OK, after two hours of messing around with this, I *think* I figured it out:

Consider foobar.piqi:
### foobar.piqi ###
.module foobar

.record [
.name delete-output-type
.field [
.name id
.type int32
]
]

.function [
.name delete
.output delete-output-type
]

.erlang-type-prefix ""
#############

First of all, the the output of

piqi compile --self-spec piqi.piqi.pb -t xml -o foobar.xml foobar.piqi

is framed (i.e. wrapped in <piqi>) and piqi convert does not seem to
accept that with any of --type piqi, --type piqi/piqi or --type
piq-lang/piqi, it outputs:

(Warning: foobar.xml:3:8: unknown field: "piqi")

and produces no output.

But this is not so relevant for me (as I generate this and can just not
generate the <piqi> element). What I was confused about is that when
trying to do (assuming no <piqi> framing in foobar.xml):

piqi convert -e erlang --type piqi -f xml foobar.xml

this works, but you get:

Warning: foobar.xml:26:24: unknown field: "erlang-type-prefix"
Warning: foobar.xml:27:24: unknown field: "erlang-string-type"

and the unknown fields are dropped.

Alternatively, if you try:

piqi convert -e erlang --type piqi-lang/piqi -f xml foobar.xml

this fails in the presence of e.g. <function> definitions with

foobar.xml:23:14: exactly one XML element expected as a variant value

Line 23 is <output>delete-output-type</output>

It turns out I had to use `piqi convert` with `--type piqi/piqi`!
Perhaps this is something obvious, but I am personally a bit confused
about the whole extension mechanism. Maybe some time should be invested
in creating a few diagrams regarding the relationship between the three
types (piqi, piqi/piqi, piqi-lang/piqi) in the docs.

Also now when using piqic-erlang I get warnings (which are harmless, but
not nice to see):

Warning: blergh.piqi:21:1: unknown field: erlang-type-prefix
Warning: blergh.piqi:22:1: unknown field: erlang-string-type

though the output is correct (i.e. no prefixes) so they *are* recognised.

Anyway, I'm mostly posting this for documentation purposes, as it seems
I can do everything I want now.

I do think however that:
* the framing problem seems like a bug and should be fixed
* "more bidirectional transformations should be possible"
* the warnings in piqic-erlang should not be there

Thanks again,
Ignas

P.S. I owe you 10 beers if we meet at some conference or so!

[Anton Lavrik]

Hey Ignas,

I'm completely confused whether there is a problem with reproducing the steps or we are talking about new topics including finding bugs in unrelated parts.

If I remember correctly, I posted 2 examples: one with "piqi compile" working on specs and another one with "piqi convert" working on a full lang.

My understanding was that piqi specs were somewhat limiting for your use case. Also keep in mind that the instant you use "piqi compile" it automatically means working with specs in their external representations. As you already noticed, framing rules are a little bit different in case of specs (they are still trivial by the way -- you just need to convert any piqi spec to XML to see what's going on).

So, let's focus on the piqi lang example and "piqi convert". In this case, both input and output is just data and we are talking about converting piqi lang represented _as data_ from once format to another. To be specific, from XML to Piq an back.

As everything Piqi does with data relies on schema, we'll need schema in our case as well. It comes in three parts: piqi.piqi, piqi.erlang.piqi and piqi-lang.piqi. As the first step, we need to get these three files and copy them to our current working directory. These are 99% regular piqi specs. I strongly encourage you to look through them -- it may answer most if not all of your questions.

Second, copy-and-paste example.xml:

$ cat example.xml 

<?xml version="1.0" encoding="UTF-8"?>

<value>

  <module>example</module>

  <typedef>

    <alias>

      <name>itemid</name>

      <type>uint32</type>

    </alias>

  </typedef>

  <typedef>

    <record>

      <name>foobar</name>

      <field>

        <type>itemid</type>

        <mode>required</mode>

      </field>

    </record>

  </typedef>

  <erlang-type-prefix></erlang-type-prefix>

  <erlang-string-type>binary</erlang-string-type>

  <function>

    <name>f</name>

    <input>

      <name>int</name>

    </input>

    <output>

      <record>

        <field>

          <type>foobar</type>

        </field>

      </record>

    </output>

  </function>

</value>

After that, you should have these files:

$ ls

example.xml  piqi.erlang.piqi  piqi-lang.piqi  piqi.piqi

Finally, let's run the conversion itself:

$ piqi convert -e erlang --type piqi-lang/piqi --piq-frameless-output true example.xml | tee example.piqi 

.module example

.typedef.alias [

    .name itemid

    .type uint32

]

.typedef.record [

    .name foobar

    .field [

        .type itemid

        .mode.required

    ]

]

.function [

    .name f

    .input.name int

    .output.record [

        .field [ .type foobar ]

    ]

]

.erlang-type-prefix ""

.erlang-string-type.binary

Now, converting it the other way around (it is easy to see why it works once you start treating the input as data):

$ piqi convert -e erlang --type piqi-lang/piqi --piq-frameless-input true -f piq -t xml -o - example.piqi | tee example.piqi.xml

...

Comparing the result of the XML -> Piq -> XML conversion:

$ diff -u example.xml example.piqi.xml

... (you should see some lines being moved but functionally they should be identical)

Are you getting the same output? There shouldn't be any warnings.

Anton

[Ignas Vyšniauskas]

Hi Anton,

Thanks for clarifying the situation, I think I'm starting to see the
overall picture, yet I'm still slightly confused.

On 12/17/2013 08:56 AM, Anton Lavrik wrote:
> I'm completely confused whether there is a problem with reproducing
> the steps or we are talking about new topics including finding bugs
> in unrelated parts.

My main confusion is that I am expecting:

piqi compile --self-spec piqi.piqi.pb -t xml -o foobar.xml foobar.piqi

to produce an XML which is valid input for

piqi convert -e erlang --type piqi-lang/piqi -f xml foobar.xml

## or --type piqi

Which it is not, because... in encodings other than piq you can only use
piqi specs (i.e. no extensions)? I'm guessing here, but the guess does
not seem to align with the availability of the `--type` parameter, so
I'm still confused. The explanation that this only converts data does
not seem to resolve this case too.

(Also, what is the difference between --type piqi-lang/piqi and --type
piqi? Or alternatively, between --piqi and --piqi/piqi?)

In fact, what I *really* want to do is to take something defined in piqi
lang and desugar it to piqi spec (regardless of the encoding format).
For example, `.input int` should be expended to `.input.name int` and
inline function type definitions changed into references. Is this
possible at all?

> Are you getting the same output? There shouldn't be any warnings.

Yes and no warnings. The only difference between your example and mine
is that you use a more verbose sytnax for the piqi definitions: e.g.
`.input.name int` instead of `.input int`. What part of piqi lang is
this actually using? You seem to be using extensions since you are able
to define types in place for function output.

Regards,
Ignas

[Anton Lavrik]

On Tue, Dec 17, 2013 at 9:48 AM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

> Are you getting the same output? There shouldn't be any warnings.

Yes and no warnings. The only difference between your example and mine
is that you use a more verbose sytnax for the piqi definitions: e.g.
`.input.name int` instead of `.input int`. What part of piqi lang is
this actually using? You seem to be using extensions since you are able
to define types in place for function output.

Sorry for the confusion... 

This is how piqi lang looks like without pretty printing and what you call extensions is a part of the language. Normally, you deal with piqi definitions in a shorter (pretty) syntax. What you see here is the full (let's call it abstract) syntax.

I should have really covered some basic concepts earlier in this thread. So, here we go. 

1) The full (abstract) syntax of the Piqi language is defined by the record type "piqi" inside the "piqi-lang.piqi" module. In other words it *is* a data structure defined using the Piqi language itself. This is the thing I referred to as "self-definition".

When you use piqi command-line tools, the --type parameter can specify one of the built-in types (such as int or string) or a user-defined type that have the following format: <module-name>/<type-name>. So, in our case, --type piqi-lang/piqi literally means: look for type "piqi" inside module "piqi-lang", which is, as you may guess, defined by the "piqi-lang.piqi" file.

When you call "piqi convert --type piqi-lang/piqi", it loads "piqi-lang.piqi", looks up the "piqi" type and uses that for performing schema-based conversion of some input data in some format to output in some output format. The output format can be specified explicitly as -t <format> or the "piq" output format is assumed by default.

Let's put all this knowledge together and look at one of the earlier examples:

    piqi convert -e erlang --type piqi-lang/piqi --piq-frameless-output true example.xml 

This command produces as its output the result of XML->Piq conversion of the data structure that corresponds to the "piqi-lang/piqi" data type.

At this point, if you could spend 10 minutes and read "piqi-lang.piqi" and "piqi.piqi", "piqi.erlang.piqi" (the latter gets loaded and included automatically when you specify "-e erlang"). This could save us hours of explanations and exploring things by trial and error. This is the Piqi language syntax reference.

2) The regular, shorter (pretty) Piqi syntax is just a simple and straightforward syntax-based transformation that the piqi command-line tools apply under the hood to make the syntax look prettier and more lightweight. When it comes to reading the Piqi language, the pretty syntax and abstract syntax are interchangeable. This is possible due to the way the Piq data representation format was designed.

In theory, "piqi pp" should be able to take the full abstract Piqi syntax and prettify it. But it doesn't do it quite yet. However, there's a way to achieve that by calling this command:

    piqi convert --piq-frameless-output true example.piqi

To eliminate the warnings, add the following two lines to your original example.xml:

    <custom-field>erlang-string-type</custom-field>

    <custom-field>erlang-type-prefix</custom-field>

3) The piqi spec is defined as a (portable) subset of the piqi lang. You can read about functional differences between these two in my previous replies. As for the syntax differences, you should be able to see the differences in their syntax by comparing "piqi.piqi" and "piqi-piqi.lang". The piqi spec syntax is defined by the "piqi" type in the "piqi.piqi" module. So, when you see "piqi convert --piqi/piqi ...", similarly to the above examples, it means: go load the "piqi.piqi" module, find the "piqi" type and convert a data structure of this type from one format to another.

4) "piqi convert --type piqi". In this case, "piqi" refers to the built-in type named "piqi". This type corresponds to a valid Piqi module (not only syntactically correct, but also its contents are semantically valid). As you already noticed, Piqi modules are treated specially during conversion. I mention a couple of key things.

First of all, in *portable* data formats such as XML, JSON and Protobuf, the modules are represented as (*portable*) Piqi specs meaning if there were any Piqi-lang extensions in the input module (such as includes), during conversion, they are expanded to functionally equivalent Piqi-spec constructs. Naturally, when you convert --type piqi form one of the portable formats, it is expected to be in the (portable) Piqi-spec format, no Piqi-lang extensions are possible there.

Second of all, if there are imported modules, they can be included in the output as dependencies making the output self-containing. However, we can't do it in XML because of its limitations (it is really a separate story).

5) "piqi compile" is very similar to "piqi convert --type piqi". But it has two key differences.

First, unlike convert, it allows to use a custom self-spec (which is normally pre-loaded with some custom extensions) via --self-spec parameter. I could explain that concept in more detail, but there was a step-by-step example earlier in this thread that shows how to use "piqi compile". Hopefully, it is more or less clear.

Second, its output always includes dependencies (i.e. other piqi modules that the module being compiled imports). The output type is actually defined as piqi/piqi-list -- naturally, you can find its definition in "piqi.piqi".

Now, to answer your first question, in XML, this extra framing makes it impossible for "piqi convert" to directly consume output of "piqi compile". In general, "piqi compile" output is intended to be consumed by tools such as piqic-erlang -- it is not meant to be immediately reversible.

However, if you run "piqi convert --type piqi/piqi-list" on "piqi compile" output, it will successfully convert a list of piqi specs. Add "-e erlang" to avoid the warnings about unknown Erlang-specific fields.

As a rule of thumb, it is good to remember that both "piqi convert --type piqi" and "piqi compile" generate piqi specs and expect piqi specs when reading from portable formats. So, if you want convert full-featured piqi-lang back and forth, you'd have to rely on the syntax-based conversion of piqi-lang as I showed in the example.

Anton

[Ignas Vyšniauskas]

Hi Anton,

Sorry for the late response, was away on holidays.
Thank you again for the long explanation -- I had already figured out
most of this myself, though your write-up is surely helpful to make the
situation clear. You should link to this thread / email from the doc page!

One final point:

On 12/18/2013 11:53 AM, Anton Lavrik wrote:
> In theory, "piqi pp" should be able to take the full abstract Piqi
> syntax and prettify it. But it doesn't do it quite yet. However,
> there's a way to achieve that by calling this command:
>
> piqi convert --piq-frameless-output true example.piqi

But I want to do the opposite: I want to take take the pretty syntax and
uglify it. I need it in particular because the "beautiful" syntax can't
be used
with `piqi convert` as we saw earlier (you e.g. used the ugly syntax in
all of the XML <-> Piq conversion examples)

Is there a way to do that?

Thanks,
Ignas

[Anton Lavrik]

Hi Ignas,

No worries about the delay. I hope you had great holidays!

To answer your question, there's no way to "uglify" the piqi syntax. But the point I was trying to make is that the "ugly" syntax is never needed. On the one hand, we want output to be pretty so I can't see a point of generating "ugly" output in the first place. On the other hand, when it comes to taking "ugly" as input, the pretty input should work equally well by design.

I skimmed through the previous discussion thread and couldn't find a case in which we would specifically need "ugly" piqi syntax and couldn't use the "pretty" one instead. Maybe I missed something... A specific example would be helpful here.

Anyway, I'll be happy to clarify the remaining questions. Only thing -- for the sake of efficiency, I suggest that we take it private and then, perhaps, we could post a summary here.

Thank you for pushing it, by the way. There's a ton of useful content that came out of this thread. I'll be definitely borrowing some of it for "piqi compile" documentation.

Anton

[Motiejus Jakštys]

On Wed, Jan 8, 2014 at 10:20 AM, Anton Lavrik <ala...@piqi.org> wrote:

Nononono please. I`m following carefully. And future readers might
strongly benefit IMO. Having these discussions open is the reason why
we use open mailing list in the first place. :-)

> Thank you for pushing it, by the way. There's a ton of useful content that
> came out of this thread. I'll be definitely borrowing some of it for "piqi
> compile" documentation.
>

+1.

Motiejus

[Anton Lavrik]

On Wed, Jan 8, 2014 at 1:26 AM, Motiejus Jakštys <desir...@gmail.com> wrote:

On Wed, Jan 8, 2014 at 10:20 AM, Anton Lavrik <ala...@piqi.org> wrote:

Nononono please. I`m following carefully. And future readers might
strongly benefit IMO. Having these discussions open is the reason why
we use open mailing list in the first place. :-)

Wow! Let's continue here, then :)

Look forward to the day when we split the list into piqi[-user] and piqi-dev. With you guys it looks like this moment is approaching faster than I thought.

[Ignas Vyšniauskas]

On 01/08/2014 10:20 AM, Anton Lavrik wrote:
> I skimmed through the previous discussion thread and couldn't find a
> case in which we would specifically need "ugly" piqi syntax and
> couldn't use the "pretty" one instead. Maybe I missed something... A
> specific example would be helpful here.

Hmm, I thought I couldn't convert my sugared `foobar.piqi` example[1] to
and from XML the way you did with your desugared `example.xml`[2], but
now it seems to work OK. Sorry, perhaps I made some stupid mistake along
the way.

> Thank you for pushing it, by the way. There's a ton of useful
> content that came out of this thread. I'll be definitely borrowing
> some of it for "piqi compile" documentation.

My pleasure! Thanks to you for your patience and answers, hopefully this
discussion is over now!

--
Ignas

[1]: https://groups.google.com/d/msg/piqi/V9AUEVl5Ti0/801_js2baGIJ
[2]: https://groups.google.com/d/msg/piqi/V9AUEVl5Ti0/eZi7-PTmstcJ