piqi convert, piqi-lang and default values

[Ignas Vyšniauskas]

Hi Anton,

I'm continuing my journey of using piqi-lang/piqi hacks and now bumped
into a more serious problem, I think.

If I try the following:

------------------

$ cat foo.xml
<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>foo</module>
<typedef>
<record>
<name>bar</name>
<field>
<type>int</type>
<name>qux</name>
<mode>optional</mode>
<default>3</default>
<code>1</code>
</field>
</record>
</typedef>
</value>

$ piqi convert -f xml --type piqi-lang/piqi foo.xml
:piqi-lang/piqi [
.module foo
.typedef.record [
.name bar
.field [
.name qux
.type int
.mode.optional
.default (xml
# <value>3</value>
)
.code 1
]
]
]

------------------

I obviously don't want the xml literal to be embedded there. The same
happens with `--type piqi/piqi`.

`--type piqi` works as expected:

------------------

$ piqi convert -f xml --type piqi foo.xml
:piqi [
.module foo
.record [
.name bar
.field [
.name qux
.type int
.optional
.default 3
.code 1
]
]
]
------------------

Is there any way I can make `piqi convert` treat .default fields in the
case of `--type piqi-lang/piqi` the same as it does in `--type piqi`?

I'm forced to use piqi-lang/piqi due to reasons discussed in the
previous thread.

Thanks,
Ignas

[Anton Lavrik]

Hi Ignas,

Can't think of such a way of the top of my head... The "default" property is defined as "any" which means it is holds a dynamically typed value. The actual semantics of the value is subject to a specific interpretation. In this case, it is the Piqi language implementation that handles it when you call it with --type piqi. Pure syntax-based conversion, obviously, wouldn't know how to interpret it.

I'm forced to use piqi-lang/piqi due to reasons discussed in the
previous thread.

Can you remind me what was the reason? I tried to find it but there's just too much stuff and I don't want to misinterpret it.

Anton

[Ignas Vyšniauskas]

On 02/22/2014 10:25 AM, Anton Lavrik wrote:
> Can't think of such a way of the top of my head... The "default"
> property is defined as "any" which means it is holds a dynamically
> typed value. The actual semantics of the value is subject to a
> specific interpretation. In this case, it is the Piqi language
> implementation that handles it when you call it with --type piqi.
> Pure syntax-based conversion, obviously, wouldn't know how to
> interpret it.

Yeah, I understand that. Though wouldn't it make sense to remove
.default from piqi lang completely and just have it as a piqi spec only
extension?

>> I'm forced to use piqi-lang/piqi due to reasons discussed in the
>> previous thread.
>
> Can you remind me what was the reason? I tried to find it but there's
> just too much stuff and I don't want to misinterpret it.

The general problem is, vaguely, "you can't do everything in the XML
representation of piqi spec".

The first reason was that I couldn't define a piqi spec in XML with an
<erlang-type-prefix/>, i.e. this example fails:

$ cat foo.piqi.xml

<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>foo</module>

<custom-field>erlang-type-prefix</custom-field>
<erlang-type-prefix></erlang-type-prefix>
<typedef>
<alias>
<name>foo</name>
<type>int32</type>
</alias>
</typedef>
</value>

$ piqi convert --type piqi -e erlang --piq-frameless-output true
foo.piqi.xml
Warning: foo.piqi.xml:4:16: unknown field: "custom-field"
Warning: foo.piqi.xml:5:22: unknown field: "erlang-type-prefix"
.module foo
.alias [
.name foo
.type int32
]

Notice: erlang-type-prefix declarations are gone.

To fix this, you suggested to do a purely syntax based conversion, i.e.
using `--type piqi-lang/piqi`. This works correctly, but now I'm facing
the problem described in this thread.

Moreover, I am now also using <erlang-name> and explicit piqi-lang
representations, to do the function I/O typename change we discussed in
a previous thread, which is not possible using piqi spec even in Piq.
Though I would rather drop this and somehow work around it, than give up
the availability of defaults.

I understand I'm doing unorthodox things here, however this mismatch
between possible functionality in different representations is a bit
confusing.

One possible hack for me would be to generate piqi spec in XML without
using the extra features, then desugar the XML into piqi-lang (so expand
defaults, function definitions, etc) and then do another transformation
on this XML to add things like <erlang-type-prefix> and <erlang-name>.
But earlier you mentioned there's no way to do the desugaring either, so
not sure.

I could also ditch generating XML completely, but since my initial
source of transformation is XML, it's much more elegant to at first
generate piqi in XML and then let piqi convert to the rest of the work.

Cheers,
Ignas

P.S. As an aside, I think bidirectional transformations between the
sugared and desugared forms would benefit piqi in the long term, just
like in any language! If only for being able to inspect "how things work".

[Anton Lavrik]

Hi Ignas,

Let's try another approach and see if it works.

I've added the ability to read xml input, and generate .piqi output to piqi compile. You can find these changes in the experimental branch: https://github.com/alavrik/piqi/tree/experimental-piqi-compile

Now you can do this:

$ cat foo.piqi.xml 

<?xml version="1.0" encoding="UTF-8"?>

<value>

  <module>foo</module>

  <custom-field>erlang-type-prefix</custom-field>

  <erlang-type-prefix></erlang-type-prefix>

  <typedef>

    <alias>

      <name>foo</name>

      <type>int32</type>

    </alias>

  </typedef>

</value>

$ cat piqi.custom.piqi 

.include [ .module piqi ]

.extend [

    .typedef piqi

    .with.field [

        .name custom-field

        .type word

        .repeated

    ]

]

You'll also need the standard piqi.piqi and piqi.erlang.piqi.

1. compile the extended self-spec that has all the erlang stuff and, in addition, defines the custom-field property

$ piqi cc -e erlang -e custom -t pb -o piqi.piqi.pb piqi.piqi

2. use the extended self-spec to convert foo.piqi.xml back to piqi:

$ piqi compile --self-spec piqi.piqi.pb -f xml -t piqi foo.piqi.xml 

.module foo

.alias [

    .name foo

    .type int32

]

.file ""

.erlang-type-prefix ""

.custom-field erlang-type-prefix

If the XML module you are converting (or should I say compiling?) has imports, make sure you convert them beforehand.

This should also properly handle defaults.

How does this look?

Anton

--
You received this message because you are subscribed to the Google Groups "piqi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to piqi+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ignas Vyšniauskas]

Hi Anton,

Your response times are amazing!

This works great, including defaults. However, I still have a problem
with being able to define function input/output/error aliases.

Any way I could get it to work with this?

$ cat foo.piqi.xml
<?xml version="1.0" encoding="UTF-8"?>
<value>
<module>foo</module>

<custom-field>erlang-name</custom-field>
<typedef>
<record>
<name>fake-name</name>
</record>
</typedef>
<function>
<name>get</name>
<input>
<alias>
<name>fake-name</name>
<type>get-input-type</type>
<erlang-name>fake_name</erlang-name>
</alias>
</input>
</function>
</value>

This (obviously) fails due to piqi compile expecting piqi spec input:

$ piqi compile --self-spec piqi.piqi.pb -f xml -t piqi foo.piqi.xml

foo.piqi.xml:12:13: string constant expected

Otherwise, once again, any way I could get the extended piqi-lang
representation from the piqi spec one? If I had something like "piqi
expand", I could then do this hack as a post-processing stage, via a
hilarious roundtrip:

XML --[XSLT]--> piqi spec in XML
--[piqi compile]--> piqi spec in Piq
--[piqi "expand"] --> piqi lang in Piq
--[piqi convert] --> piqi lang in XML
--[XSLT post-processs]--> piqi lang in XML
--[piqi convert]--> piqi lang in Piq


Alternatively, if you think attempting to access piqi-lang features is
the totally-wrong-path-to-take in general, then please say so. I will
try to figure out alternative workarounds.

As usual, very grateful for your help.

--
Ignas

[Anton Lavrik]

On Wed, Feb 26, 2014 at 4:37 AM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

Hi Anton,

Your response times are amazing!

No wonder. You ask fun questions :)

It looks like this equivalent spec works the way you need (requires the latest piqi-erlang -- had to fix a small problem there):

$ cat foo.piqi.xml 

<?xml version="1.0" encoding="UTF-8"?>

<value>

  <module>foo</module>

  <custom-field>erlang-name</custom-field>

  <function>

      <name>f</name>

      <input>f-input</input>

  </function>

  <typedef>

    <alias>

        <name>f-input</name>

        <type>int</type>

        <erlang-name>e_f_input</erlang-name>

    </alias>

  </typedef>

</value>

Otherwise, once again, any way I could get the extended piqi-lang
representation from the piqi spec one? If I had something like "piqi
expand", I could then do this hack as a post-processing stage, via a
hilarious roundtrip:

    XML  --[XSLT]--> piqi spec in XML
            --[piqi compile]--> piqi spec in Piq
            --[piqi "expand"] --> piqi lang in Piq
            --[piqi convert] --> piqi lang in XML
            --[XSLT post-processs]--> piqi lang in XML
            --[piqi convert]--> piqi lang in Piq

This is the part I'm struggling to understand: piqi spec in Piq  --[piqi "expand"] --> piqi lang in Piq

What does this actually mean? Given that piqi-spec is a subset of piqi-lang, how one could expand a subset into a superset?

Alternatively, if you think attempting to access piqi-lang features is
the totally-wrong-path-to-take in general, then please say so. I will
try to figure out alternative workarounds.

If does kind of feel wrong. I'd rather find a direct solution that doesn't involve hacking stuff that's never meant to be portable. Let's see how far we could get. I think we are getting pretty close or should I say we are already there?

If positive, I think we should make "custom-field" a part of the piqi-spec and merge the experimental branch into master.

Anton

[Ignas Vyšniauskas]

Hi again!

>
> On 02/27/2014 09:11 AM, Anton Lavrik wrote:
>
>
> It looks like this equivalent spec works the way you need (requires
> the latest piqi-erlang -- had to fix a small problem there):
>

Awesome, this seems to work for me too.

>
> Otherwise, once again, any way I could get the extended piqi-lang
> representation from the piqi spec one? If I had something like "piqi
> expand", I could then do this hack as a post-processing stage, via a
> hilarious roundtrip:
>
> XML --[XSLT]--> piqi spec in XML
> --[piqi compile]--> piqi spec in Piq
> --[piqi "expand"] --> piqi lang in Piq
> --[piqi convert] --> piqi lang in XML
> --[XSLT post-processs]--> piqi lang in XML
> --[piqi convert]--> piqi lang in Piq
>
>
> This is the part I'm struggling to understand: piqi spec in
> Piq --[piqi "expand"] --> piqi lang in Piq
>
> What does this actually mean? Given that piqi-spec is a subset of
> piqi-lang, how one could expand a subset into a superset?

By "expand" I mean "desugar". Since piqi-spec is defined in terms of
piqi-lang, we can always desugar it to the equivalent piqi-lang version,
i.e. explicit .typedef's and so on.

>
> Alternatively, if you think attempting to access piqi-lang features is
> the totally-wrong-path-to-take in general, then please say so. I will
> try to figure out alternative workarounds.
>
>
> If does kind of feel wrong. I'd rather find a direct solution that
> doesn't involve hacking stuff that's never meant to be portable. Let's
> see how far we could get. I think we are getting pretty close or
> should I say we are already there?
>
> If positive, I think we should make "custom-field" a part of the
> piqi-spec and merge the experimental branch into master.
>

It seems to work, so I guess we are there indeed. The only caveat is
that piqic-erlang now seems to complain about the usage of
"custom-field" everywhere:

----
piqic-erlang foo.piqi
Warning: the following property is not recognized by the custom
self-spec and will be ignored:
Warning: unknown:0:0: unknown field: custom-field
----

but I guess this would be fixed by making custom-field a part of
piqi-spec, which sounds like a sensible thing to do in general.

Thanks again for your help!

--
Ignas

P.S. I seem to have discovered an unrelated bug while testing this:
https://github.com/alavrik/piqi/issues/39#issuecomment-36350135

[Ignas Vyšniauskas]

On 02/28/2014 03:18 PM, Ignas Vyšniauskas wrote:
>> Otherwise, once again, any way I could get the extended piqi-lang
>> representation from the piqi spec one? If I had something like "piqi
>> expand", I could then do this hack as a post-processing stage, via a
>> hilarious roundtrip:
>>
>> XML --[XSLT]--> piqi spec in XML
>> --[piqi compile]--> piqi spec in Piq
>> --[piqi "expand"] --> piqi lang in Piq
>> --[piqi convert] --> piqi lang in XML
>> --[XSLT post-processs]--> piqi lang in XML
>> --[piqi convert]--> piqi lang in Piq
>>
>>
>> This is the part I'm struggling to understand: piqi spec in
>> Piq --[piqi "expand"] --> piqi lang in Piq
>>
>> What does this actually mean? Given that piqi-spec is a subset of
>> piqi-lang, how one could expand a subset into a superset?
>
> By "expand" I mean "desugar". Since piqi-spec is defined in terms of
> piqi-lang, we can always desugar it to the equivalent piqi-lang version,
> i.e. explicit .typedef's and so on.

Err, I obviously mixed these two up and hence confused us both. In fact,
I have been probably using piqi-spec and piqi-lang in mixed ways for a
long time and confusing you horribly, sorry!

I guess the correct statement would be "expand piqi definitions (written
down in piqi spec) to the elaborated piqi-lang definitions, again,
*written down in piqi-spec*".

So for example, function I/O fields would be expanded from just
typenames to 'function-param' variants like in piqi-lang?
https://github.com/alavrik/piqi/blob/master/piqi/piqi-lang.piqi#L44

In other words, what I mean is "since piqi-lang is defined in terms of
piqi-spec and piqi definitions are elaborated to piqi-lang definitions,
it should be possible to represent the elaborated versions in piqi-spec
too."

At least, I hope I understand this correctly now and what I'm saying
makes sense.

As a side note, I was going through the self-bootstrapping code of piqi
yesterday -- I have to admit I was confused as hell :-)

Not saying that it is in any way your fault, but the cyclic dependency
of piqi-ocaml <-> piqi and the mixture of generated code +
self-referential definitions makes it pretty hard to follow when you
don't know where to start from, especially if you know almost no Ocaml, hah!

It would be great to have a high-level overview of the full
bootstrapping / build process and what happens when the piqi spec
changes, mentioning the implicit dependencies and so on. I want to start
some hacking on piqi itself, but I'm afraid I still don't get the full
picture and will do things in the "wrong way".

--
Ignas

[Anton Lavrik]

On Fri, Feb 28, 2014 at 6:18 AM, Ignas Vyšniauskas <bali...@gmail.com> wrote:

Hi again!

It seems to work, so I guess we are there indeed. The only caveat is
that piqic-erlang now seems to complain about the usage of
"custom-field" everywhere:

----
piqic-erlang foo.piqi
Warning: the following property is not recognized by the custom
self-spec and will be ignored:
Warning: unknown:0:0: unknown field: custom-field
----

but I guess this would be fixed by making custom-field a part of
piqi-spec, which sounds like a sensible thing to do in general.

You guessed correctly. I made custom-field a part of the piqi-spec, which removed the warning, and merged the whole thing into master.

 

P.S. I seem to have discovered an unrelated bug while testing this:
https://github.com/alavrik/piqi/issues/39#issuecomment-36350135

Fixed as well.

Anton

[Anton Lavrik]

I think you got it fight this time. Still not sure what would be a practical purpose for turning spec back to lang. Nested function parameters is the only thing that can be recovered (un-expanded) after expanding them as a part of lang -> spec conversion. The problem is that you can't be 100% sure if they were nested or defined as top-level types in the original (unexpanded) spec.

Think of nested function parameters as a syntax shorthand. Kind of a macro. They exist just for convenience. The portable spec exists for consumption by other programs. if you need to share .piqi with other people just give them the original one.

 

As a side note, I was going through the self-bootstrapping code of piqi
yesterday -- I have to admit I was confused as hell :-)

If you had seen it 6 or even 2 months ago you'd have appreciated the way it works today :)

 

Not saying that it is in any way your fault, but the cyclic dependency
of piqi-ocaml <-> piqi and the mixture of generated code +
self-referential definitions makes it pretty hard to follow when you
don't know where to start from, especially if you know almost no Ocaml, hah!

piqi-ocaml is needed only for bootstrapping, i.e. if you want to hack on internals... There are probably a lot more things that can be improved, but to be fair, some things Piqi does are just difficult. First of all, mapping of a self-describing extensible language to statically typed OCaml is not trivial. Aside of that and the overall language design, there are lot of purely technical challenges, such as .piqi source code location tracking, handling extensions and implementing module extensions as mutually recursive includes.

It would be great to have a high-level overview of the full
bootstrapping / build process and what happens when the piqi spec
changes, mentioning the implicit dependencies and so on. I want to start
some hacking on piqi itself, but I'm afraid I still don't get the full
picture and will do things in the "wrong way".

The build is rather straightforward. If you want to add properties to piqi-lang, you need to make modifications in piqi/*.piqi and then run "make piqi" in the piqilib directly.

In order to be able to do this, you need to download piqi-ocaml (https://github.com/alavrik/piqi-ocaml) and put it next to your piqi repo. Here's a full script:

git clone https://github.com/alavrik/piqi

cd piqi

./configure

make deps

make

make ocaml-install

cd ..

git clone https://github.com/alavrik/piqi-ocaml

cd piqi-ocaml

make

cd ..

cd piqi

# make changes in piqi/*.piqi

make piqi  (or make -C piqilib piqi)  # bootstrap from the updated self-specs

make  # rebuild

What happens under the hood when you run "make piqi" is tricker to explain. You can follow the build steps and let me know if anything is unclear. Just keep in mind that you don't have to fully understand what's going on to add new properties to piqi self-specs and use them in your code.

Anton