"default" values in field specifications

[Motiejus Jakštys]

Hi,

when default value of a field is specified, sometimes leaving no space
between the value "default" is required, and sometimes not. This is
inconvenient, because our piqi definition files are generated (from an
XML) and I have two silly corner cases.

Say we have the following:

.enum [
.name gender
.option [ .name m ]
.option [ .name f ]
]

.record [
.name profiles
.field [
.name gender
.type gender
.optional
.default.u
]
.field [
.name age
.type int
.optional
.default 0
]
]

If I make a space between "default" and ".u", I get this error:
profiles-defs.piqi:44:3: value must be specified for field "default"

Would it be possible to update the parser to work with spaces in these
special cases?

Thanks,
Motiejus

[Anton Lavrik]

On Thu, Mar 28, 2013 at 4:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> Hi,
>
> when default value of a field is specified, sometimes leaving no space
> between the value "default" is required, and sometimes not. This is
> inconvenient, because our piqi definition files are generated (from an
> XML)

Just curious -- why not use piqi in the first place? Piq is a much
friendlier notation than XML for authoring pretty much anything.
Starting from Piqi v0.6.0 you can extend the Piqi language with
arbitrary properties, have them validated and converted back to XML if
necessary. One case where it won't work better than XML is when you
rely on IDEs or XML editors.

> and I have two silly corner cases.
>
> Say we have the following:
>
> .enum [
> .name gender
> .option [ .name m ]
> .option [ .name f ]
> ]
>
> .record [
> .name profiles
> .field [
> .name gender
> .type gender
> .optional
> .default.u
> ]
> .field [
> .name age
> .type int
> .optional
> .default 0
> ]
> ]
>
> If I make a space between "default" and ".u", I get this error:
> profiles-defs.piqi:44:3: value must be specified for field "default"
>
> Would it be possible to update the parser to work with spaces in these
> special cases?

Unfortunately, it is not that simple. This is one of those subtle
cases where space or absence of it matters, because these really are
different syntactic constructs.

By design, the Piq language doesn't have element separators or
delimiters (',' or ';' common in other languages). Instead, we rely on
labels (i.e. words starting with '.') for delimiting previous labeled
values or labels.

For example,

".foo 1 .bar 2" with explicit control of associativity would be
represented as "(.foo 1) (.bar 2)"

Similarly,

".foo .bar" = "(.foo) (.bar)" -- two independent labels

However,

".foo.bar" = " .foo (.bar) " that is label "foo" applied to the value ".bar"

The last example, ".foo.bar" is actually a syntax shorthand for ".foo
(.bar)". It is also documented here:
http://piqi.org/doc/piq/#builtinsyntaxabbreviation There is also the
"piqi pp --expand-abbr" command that will turn the latter into the
former.

I agree that this space sensitivity doesn't look very nice, but I
think it is better than extra parenthesis as in ".foo (.bar)". So far,
I have failed to come up with a more elegant solution. Also, I didn't
want to introduce new lexical elements. Any ideas on how to improve
that would be very welcome.

Anyway, since your .piqi modules are auto-generated, there are two
easy approaches that may help. First, you can check if the value
starts with '.', and wrap such value in parenthesis or glue it with
the label. Second, you can wrap all field values in parenthesis. In
your example, it would become .default (0) and .default (.u).
Actually, the .default (0) is broken right now. I'll fix it over the
weekend. And there is other bug that I discovered while answering your
question.

Thanks for asking!

Anton

[Motiejus Jakštys]

On Fri, Mar 29, 2013 at 01:06:34AM -0500, Anton Lavrik wrote:
> On Thu, Mar 28, 2013 at 4:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> > Hi,
> >
> > when default value of a field is specified, sometimes leaving no space
> > between the value "default" is required, and sometimes not. This is
> > inconvenient, because our piqi definition files are generated (from an
> > XML)
>
> Just curious -- why not use piqi in the first place? Piq is a much
> friendlier notation than XML for authoring pretty much anything.

I kind of agree with this point, but piqi is not the only artefact that
gets generated from the service definition XML. Some examples of what is
more generated: piqi-rpc piqi files, some custom application-level
constraints (which kind of extend piqi constraint language
business-wise, a bit better version of XSD) and some more stuff like SQL
table definitions, some documentation, etc. It is really handy to have a
single XML and a bunch of XSLTs to generate artefacts. While this might
sound horrible (XML and XSD in 2013!?!??), but the end result is
actually quite neat (if you keep the LOC in Makefiles low ....)

I think you would be interested about this constraint mechanism.. Maybe
making it a side-by-side library of piqi. It integrates with piqi really
well, and does the 'constraints' that piqi is not designed to do. If you
are interested, maybe you will be in EUG Stockholm this year, so we can
have a chat about it? Or somewhere else in Europe? Come to Netherlands.
:-)

> Starting from Piqi v0.6.0 you can extend the Piqi language with
> arbitrary properties, have them validated and converted back to XML if
> necessary. One case where it won't work better than XML is when you
> rely on IDEs or XML editors.

No, we do not rely on IDEs. But, like I wrote above, piqi record
definitions is not the only thing that gets generated from it. Unless
you can convince us that there is a better approach to transform piqi
files to anything better than XSLT has for XML. :-)

> The last example, ".foo.bar" is actually a syntax shorthand for ".foo
> (.bar)". It is also documented here:
> http://piqi.org/doc/piq/#builtinsyntaxabbreviation There is also the
> "piqi pp --expand-abbr" command that will turn the latter into the
> former.
>
> I agree that this space sensitivity doesn't look very nice, but I
> think it is better than extra parenthesis as in ".foo (.bar)". So far,
> I have failed to come up with a more elegant solution. Also, I didn't
> want to introduce new lexical elements. Any ideas on how to improve
> that would be very welcome.
>
> Anyway, since your .piqi modules are auto-generated, there are two
> easy approaches that may help. First, you can check if the value
> starts with '.', and wrap such value in parenthesis or glue it with
> the label. Second, you can wrap all field values in parenthesis. In
> your example, it would become .default (0) and .default (.u).

Another gentle reminder for me to RTFM more carefully. Parenthesis are
there, and they are exactly what I am looking for.

> Actually, the .default (0) is broken right now. I'll fix it over the
> weekend. And there is other bug that I discovered while answering your
> question.

Haha, nice. I will change our XSLTs to use parenthesis everywhere in
default values once you fix this + we migrate to 0.6.2. :-)

Thanks a lot for great support.

Motiejus

[Anton Lavrik]

On Fri, Mar 29, 2013 at 7:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> On Fri, Mar 29, 2013 at 01:06:34AM -0500, Anton Lavrik wrote:
>> On Thu, Mar 28, 2013 at 4:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
>> > Hi,
>> >
>> > when default value of a field is specified, sometimes leaving no space
>> > between the value "default" is required, and sometimes not. This is
>> > inconvenient, because our piqi definition files are generated (from an
>> > XML)
>>
>> Just curious -- why not use piqi in the first place? Piq is a much
>> friendlier notation than XML for authoring pretty much anything.
>
> I kind of agree with this point, but piqi is not the only artefact that
> gets generated from the service definition XML. Some examples of what is
> more generated: piqi-rpc piqi files, some custom application-level
> constraints (which kind of extend piqi constraint language
> business-wise, a bit better version of XSD) and some more stuff like SQL
> table definitions, some documentation, etc. It is really handy to have a
> single XML and a bunch of XSLTs to generate artefacts. While this might
> sound horrible (XML and XSD in 2013!?!??), but the end result is
> actually quite neat (if you keep the LOC in Makefiles low ....)

I see. I would still claim that all that can be represented in .piqi.
You can extend the Piqi language with your custom attributes and have
the specs validated and converted to XML. Not sure if it would make
things more practical for you, but syntactically it would arguably
make specs look a lot nicer. If you are interested I can show you how
to achieve this using a combination of "piqi cc" and "piqi compile"
commands. In fact, this mechanism is already used in the new
piqic_erang.erl -- see piqi-erlang/src/piqi.erlang.piqi as an example
of the Piqi language extension.

> I think you would be interested about this constraint mechanism.. Maybe
> making it a side-by-side library of piqi. It integrates with piqi really
> well, and does the 'constraints' that piqi is not designed to do.

This sounds interesting. Can you give an example of how you define
such constraints and how they are represented in the generated Erlang
code?

> If you
> are interested, maybe you will be in EUG Stockholm this year, so we can
> have a chat about it? Or somewhere else in Europe? Come to Netherlands.
> :-)

I would love to and definitely will some day. Thank you for the invitation :)

>> Starting from Piqi v0.6.0 you can extend the Piqi language with
>> arbitrary properties, have them validated and converted back to XML if
>> necessary. One case where it won't work better than XML is when you
>> rely on IDEs or XML editors.
>
> No, we do not rely on IDEs. But, like I wrote above, piqi record
> definitions is not the only thing that gets generated from it. Unless
> you can convince us that there is a better approach to transform piqi
> files to anything better than XSLT has for XML. :-)

I don't have to convince you, because you can convert .piqi to XML and
keep using XSLT.

>> Actually, the .default (0) is broken right now. I'll fix it over the
>> weekend. And there is other bug that I discovered while answering your
>> question.
>
> Haha, nice. I will change our XSLTs to use parenthesis everywhere in
> default values once you fix this + we migrate to 0.6.2. :-)

I fixed it in dev branch and will be releasing a new version this
week. (By the way, I'm also going to drop the dev branches and move
development of piqi, piqi-erlang and piqi-rpc to the master branches).

Anton

[Motiejus Jakštys]

On Mon, Apr 01, 2013 at 02:35:35AM -0500, Anton Lavrik wrote:
> On Fri, Mar 29, 2013 at 7:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
> > On Fri, Mar 29, 2013 at 01:06:34AM -0500, Anton Lavrik wrote:
> >> On Thu, Mar 28, 2013 at 4:38 AM, Motiejus Jakštys <desir...@gmail.com> wrote:
>
> I see. I would still claim that all that can be represented in .piqi.
> You can extend the Piqi language with your custom attributes and have
> the specs validated and converted to XML. Not sure if it would make
> things more practical for you, but syntactically it would arguably
> make specs look a lot nicer. If you are interested I can show you how
> to achieve this using a combination of "piqi cc" and "piqi compile"
> commands. In fact, this mechanism is already used in the new
> piqic_erang.erl -- see piqi-erlang/src/piqi.erlang.piqi as an example
> of the Piqi language extension.

You certainly have a point, but I don't know yet how complex it might
be.

> > I think you would be interested about this constraint mechanism.. Maybe
> > making it a side-by-side library of piqi. It integrates with piqi really
> > well, and does the 'constraints' that piqi is not designed to do.
>
> This sounds interesting. Can you give an example of how you define
> such constraints and how they are represented in the generated Erlang
> code?

An extract from our simplified example bucket definition:

<bucket>
<fieldlist>
<field tag="1" name="gid" type="types/gid" />
<field tag="2" name="given-name" type="string" >
<constraint name="length" max="255" />
</field>
<field tag="3" name="birthdate" type="types/date" null="true"
default="[.year 1990 .month 1 .day 1]">
<!-- Allow only 18 year 1 day+ persons -->
<constraint name="age_over" delta_months="5">
<param name="delta_years">18</param>
<param name="delta_days">1</param>
</constraint>
</field>
</fieldlist>
<gidconstraints>
<constraint name="unique"><field>email</field></constraint>
</gidconstraints>
<sql version="1"><![CDATA[
CREATE TABLE `demobucket` (
`gid` bigint(20) unsigned NOT NULL,
`name` varchar(255) NOT NULL,
`birthdate` varchar(10) NULL,
KEY `GID` (`GID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
]]>
</sql>
<sql version="2"><![CDATA[
CREATE TABLE `demobucket` (
`GID` bigint(20) unsigned NOT NULL,
`givenname` enum('m','f','u') NOT NULL,
`birthdate` date NOT NULL,
KEY `GID` (`GID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
]]>
</sql>
</bucket>

This gets converted to

.record [
.name demobucket
.field [ .name gid .type types/gid .code 1 ]
.field [ .name given-name .type string .code 2 ]
.field [ .name birthdate .type types/date .code 6 .optional
.default [.year 1990 .month 1 .day 1] ]
]

And

-define(TYPE_CONSTRAINTS, [
{'gid', 'types/gid', []},
{'given_name', 'string', []},
{'birthdate', 'types/date', [
{'null', <<"true">>},
{'default', <<"[.year 1990 .month 1 .day 1]">>}
]}
]
).
-define(FIELD_CONSTRAINTS, [
{'given_name', [ {'length', [ {'max', <<"255">>} ]} ]},
{'birthdate', [
{'age_over', [
{'delta_months', <<"5">>},
{'delta_years', <<"18">>},
{'delta_days', <<"1">>}
]}
]}
]
).
-define(GID_CONSTRAINTS, [ {'unique', ['email'], [] } ]).

Then there is some code to verify the constraints just behind our API
(in demobucket_piqi_impl.erl). Every constraint has its own function,
which does the validation. For example,
`bucket_type_constraints:length(Bin, Length)` returns `true` if `Bin`
has less than `Length` utf8 codepoints.

We also create a few SQL files (one per `version`), another .piqi file
suitable for piqi-rpc (get/update/delete and some more functions). As
you can imagine, it involves a fair number of Makefiles has a few moving
parts. If we see a way to reduce it in some non-complex way, we would
be happy with a cleaner second iteration.

I believe these type definitions could be extended, but I do not see an
immediate way to do that after reading piqi-erlang/doc/piqi-erlang.md.
Is it possible to extend the piqi language in the record definition
file?

Motiejus