sexp_default

Showing 1-18 of 18 messages
sexp_default Markus 3/30/12 9:19 AM
Hi,

I have a patch for sexplib, which adds a new record field qualifier
"sexp_default".  It allows users to explicitly add the default
expression to be used when a record field is missing in an
S-expression.  E.g.:

  type t = { foo : int sexp_default(42) } with sexp

In this definition "foo" will default to "42" if the S-expression does
not specify "foo".  Any valid OCaml-expression is allowed within the
parentheses, including variables and even function calls.  When such a
record is converted to an S-expression, the normal behavior is to
always emit the field, no matter whether it is equivalent to the
default or not.  The reason is that unlike "sexp_list", "sexp_array",
etc., it may not be obvious to the reader of an S-expression what the
default may be.

If, however, default fields should be excluded from the conversion,
e.g. to generate a minimal default configuration file for later
refinement, the following is supported (note the exclamation mark at
the end):

  type t = { foo : int sexp_default(42)! } with sexp

Using manifest types, developers can easily generate equivalent
instantiations of a record type with different conversion behavior.
E.g.:

    type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp

This way different "default views" could be emitted for a given record.

Using manifest types together with module functors and maybe
first-class modules to do this dynamically for large numbers of
instantiations, one can also easily generate numerous default
converters for different contexts.  E.g.:

  module Make (Defaults : sig val default : t end) = struct
    type s = t = { foo : int sexp_default(Defaults.default.foo) } with sexp
  end

Some wizardry with modules and manifest types provides for almost
unlimited power for providing defaults and configuring how they are
emitted.

Let me know if this patch would likely be accepted or if there are any
suggestions for improvement, and I'll push it to my fork of OCaml
Core.

Regards,
Markus

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Yaron Minsky 3/30/12 12:45 PM
This seems like a cool hack.  I'd like to wait for some of the other
folk who think about the syntax extensions to weigh in, but at first
glance, the proposal seems good.

I am a little worried about the semantics of !, though.  If I create a type:

    type t = { foo : int sexp_default(7)! } with sexp

then the round-tripping could be pretty weird.  In particular,
consider this sequence:

   { foo = 3 }
   |! sexp_of_t  (* this is () *)
   |! t_of_sexp  (* this is { foo = 7 } *)

This behavior seems pretty unfortunate.  Is that really what it does?

As a side note, Markus, if you're working on more patches: please
don't do merges in the stuff your proposing.  Yury's conversion code
doesn't work well on merges.  (merges purely within your code are
actually OK, as long as you're not merging in from different points in
the history that was exported from Jane Street.)

y

Re: sexp_default Markus 3/30/12 1:25 PM
On Fri, Mar 30, 2012 at 15:45, Yaron Minsky <ymi...@janestreet.com> wrote:
> I am a little worried about the semantics of !, though.  If I create a type:
>
>    type t = { foo : int sexp_default(7)! } with sexp
>
> then the round-tripping could be pretty weird.  In particular,
> consider this sequence:
>
>   { foo = 3 }
>   |! sexp_of_t  (* this is () *)
>   |! t_of_sexp  (* this is { foo = 7 } *)
>
> This behavior seems pretty unfortunate.  Is that really what it does?

No, it would consider equivalence to the default expression first to
determine whether to drop the field in the S-expression.  In your case
it would convert the record to ((foo 3)).  If, however, you had used {
foo = 7 }, then it would convert it to () as expected.  Without the !,
it would always emit the field no matter the content.

Due to that equivalence check it is probably not a good idea to use
expensive function calls or gargantuan values as default expressions.
But that doesn't seem necessary in practice anyway.

> As a side note, Markus, if you're working on more patches: please
> don't do merges in the stuff your proposing.  Yury's conversion code
> doesn't work well on merges.  (merges purely within your code are
> actually OK, as long as you're not merging in from different points in
> the history that was exported from Jane Street.)

Hm, well, that surely makes it hard to work on the same files, which
would require merges.  As long as we don't tread on each other's feet
that should be possible though.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Yaron Minsky 3/30/12 1:44 PM
On Fri, Mar 30, 2012 at 4:25 PM, Markus Mottl <markus...@gmail.com> wrote:
> On Fri, Mar 30, 2012 at 15:45, Yaron Minsky <ymi...@janestreet.com> wrote:
>> I am a little worried about the semantics of !, though.  If I create a type:
>>
>>    type t = { foo : int sexp_default(7)! } with sexp
>>
>> then the round-tripping could be pretty weird.  In particular,
>> consider this sequence:
>>
>>   { foo = 3 }
>>   |! sexp_of_t  (* this is () *)
>>   |! t_of_sexp  (* this is { foo = 7 } *)
>>
>> This behavior seems pretty unfortunate.  Is that really what it does?
>
> No, it would consider equivalence to the default expression first to
> determine whether to drop the field in the S-expression.  In your case
> it would convert the record to ((foo 3)).  If, however, you had used {
> foo = 7 }, then it would convert it to () as expected.  Without the !,
> it would always emit the field no matter the content.
>
> Due to that equivalence check it is probably not a good idea to use
> expensive function calls or gargantuan values as default expressions.
> But that doesn't seem necessary in practice anyway.

Got it.  Is the equality check based on polymorphic compare, or
equality of the s-expression representation?

>> As a side note, Markus, if you're working on more patches: please
>> don't do merges in the stuff your proposing.  Yury's conversion code
>> doesn't work well on merges.  (merges purely within your code are
>> actually OK, as long as you're not merging in from different points in
>> the history that was exported from Jane Street.)
>
> Hm, well, that surely makes it hard to work on the same files, which
> would require merges.  As long as we don't tread on each other's feet
> that should be possible though.

Yeah, we'll have to figure it out over time.  I think right now one
can do all the merging one wants, subject to the constraint that all
the work is anchored to the same spot in our HG repo.  But the
simplest use-case is just linear histories.  Right now Yury is
manually dealing with the conversion of the revisions we've pulled in
so far, which is kind of painful.

> --
> Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Markus 3/30/12 1:57 PM
On Fri, Mar 30, 2012 at 16:44, Yaron Minsky <ymi...@janestreet.com> wrote:
> On Fri, Mar 30, 2012 at 4:25 PM, Markus Mottl <markus...@gmail.com> wrote:
>> Due to that equivalence check it is probably not a good idea to use
>> expensive function calls or gargantuan values as default expressions.
>> But that doesn't seem necessary in practice anyway.
>
> Got it.  Is the equality check based on polymorphic compare, or
> equality of the s-expression representation?

Polymorphic equality, which seems preferable for efficiency.  Neither
it nor equality of S-expressions is guaranteed to work on abstract
datatypes anyway.

> Yeah, we'll have to figure it out over time.  I think right now one
> can do all the merging one wants, subject to the constraint that all
> the work is anchored to the same spot in our HG repo.  But the
> simplest use-case is just linear histories.  Right now Yury is
> manually dealing with the conversion of the revisions we've pulled in
> so far, which is kind of painful.

Ok, I'll not merge your tree into the one from which I push in the future.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Yaron Minsky 3/30/12 2:28 PM
On Fri, Mar 30, 2012 at 4:57 PM, Markus Mottl <markus...@gmail.com> wrote:
> On Fri, Mar 30, 2012 at 16:44, Yaron Minsky <ymi...@janestreet.com> wrote:
>> On Fri, Mar 30, 2012 at 4:25 PM, Markus Mottl <markus...@gmail.com> wrote:
>>> Due to that equivalence check it is probably not a good idea to use
>>> expensive function calls or gargantuan values as default expressions.
>>> But that doesn't seem necessary in practice anyway.
>>
>> Got it.  Is the equality check based on polymorphic compare, or
>> equality of the s-expression representation?
>
> Polymorphic equality, which seems preferable for efficiency.  Neither
> it nor equality of S-expressions is guaranteed to work on abstract
> datatypes anyway.

Hrm.  The polymorphic equality thing makes me a little sick  --- we're
actually trying to avoid polymorphic compare wherever we can these
days, for both performance and semantic reasons.  Is there a reason it
can't use comparison of s-expressions?  I don't think efficiency is a
primary concern here, since this is already fairly slow
pretty-printing code.

>> Yeah, we'll have to figure it out over time.  I think right now one
>> can do all the merging one wants, subject to the constraint that all
>> the work is anchored to the same spot in our HG repo.  But the
>> simplest use-case is just linear histories.  Right now Yury is
>> manually dealing with the conversion of the revisions we've pulled in
>> so far, which is kind of painful.
>
> Ok, I'll not merge your tree into the one from which I push in the future.
>
> --
> Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Markus 3/30/12 3:09 PM
On Fri, Mar 30, 2012 at 17:28, Yaron Minsky <ymi...@janestreet.com> wrote:
> Hrm.  The polymorphic equality thing makes me a little sick  --- we're
> actually trying to avoid polymorphic compare wherever we can these
> days, for both performance and semantic reasons.  Is there a reason it
> can't use comparison of s-expressions?  I don't think efficiency is a
> primary concern here, since this is already fairly slow
> pretty-printing code.

I guess nobody is fully satisfied with polymorphic equality in
general.  I'm not sure there is any true gain in comparing
S-expressions.  One could imagine that some S-expression converters
normalize the representation so that e.g. sets, maps, etc., would
compare consistently, but this is not a given (e.g. hashtables).
Furthermore, in the vast majority of cases we would needlessly convert
a value to an S-expression just for equality checking even though it
need not be emitted in the first place - a rather expensive default
behavior.

The only consistent solution I can see is if the user can optionally
specify an equivalence relation, e.g. by having another optional
expression that contains such a function.  But this could turn out to
be extremely expensive: think of comparing hashtables.

Default field values are probably hardly ever changed back and forth
at runtime, especially in ways that would destroy pointer equality or
worse to a structurally distinct but semantically equivalent
representation.  Polymorphic equality checks would therefore almost
always just compare two identical pointers anyway, which is both
consistent and maximally efficient.  Even if a field were emitted that
need not be, this should never cause any issues other than diminished
output readability.

I'd therefore suggest to use polymorphic equality for the while being.
 If it really turns out to cause issues in practice, you can always
add the mentioned optional equality function later.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Jim 3/31/12 2:31 AM
Hi, Markus. Your proposal is interesting, but I'm a little concerned
about this part:

> Using manifest types, developers can easily generate equivalent
> instantiations of a record type with different conversion behavior.
> E.g.:
>
>    type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp
>
> This way different "default views" could be emitted for a given record.

It is not clear to me whether this is best construed as a feature or a
bug. It seems to encourage people to make type-equivalences that have
sexp representations that can be generated by one type and parsed by
an "equivalent" type that has silently incompatible semantics. This,
combined with the fact that sexp representations do not embed the name
of the types or modules that generated them, looks to me to like a
recipe for bugs.

Do you consider this facilitation of different "default views" to be
an important aspect of the proposal? Or is there a milder version of
the proposal that does not have this feature?

- Jim

Re: sexp_default Yaron Minsky 3/31/12 4:50 AM
On Fri, Mar 30, 2012 at 6:09 PM, Markus Mottl <markus...@gmail.com> wrote:
> On Fri, Mar 30, 2012 at 17:28, Yaron Minsky <ymi...@janestreet.com> wrote:
>> Hrm.  The polymorphic equality thing makes me a little sick  --- we're
>> actually trying to avoid polymorphic compare wherever we can these
>> days, for both performance and semantic reasons.  Is there a reason it
>> can't use comparison of s-expressions?  I don't think efficiency is a
>> primary concern here, since this is already fairly slow
>> pretty-printing code.
>
> I guess nobody is fully satisfied with polymorphic equality in
> general.  I'm not sure there is any true gain in comparing
> S-expressions.  One could imagine that some S-expression converters
> normalize the representation so that e.g. sets, maps, etc., would
> compare consistently, but this is not a given (e.g. hashtables).
> Furthermore, in the vast majority of cases we would needlessly convert
> a value to an S-expression just for equality checking even though it
> need not be emitted in the first place - a rather expensive default
> behavior.

Fair.

> The only consistent solution I can see is if the user can optionally
> specify an equivalence relation, e.g. by having another optional
> expression that contains such a function.  But this could turn out to
> be extremely expensive: think of comparing hashtables.
>
> Default field values are probably hardly ever changed back and forth
> at runtime, especially in ways that would destroy pointer equality or
> worse to a structurally distinct but semantically equivalent
> representation.  Polymorphic equality checks would therefore almost
> always just compare two identical pointers anyway, which is both
> consistent and maximally efficient.  Even if a field were emitted that
> need not be, this should never cause any issues other than diminished
> output readability.
>
> I'd therefore suggest to use polymorphic equality for the while being.
>  If it really turns out to cause issues in practice, you can always
> add the mentioned optional equality function later.

That sounds pretty reasonable.  Another option would be to use
physical equality, which would have the property that if you left the
default unmodified, it would work, but otherwise would re-output the
value.  That's got its own set of surprising issues as well, though.

y

Re: sexp_default Yaron Minsky 3/31/12 4:53 AM
I think of this kind of multiple-view trick as pretty standard.  We
use this internally quite a bit when we want to generate an alternate
view without creating a new type.  We just create a new type-alias,
and attach the specialized converters to that type.

I guess I'm just saying I'm in the "feature, not bug" camp.  That
said, I'm not sure what you mean when you talk about "silently
incompatible semantics".  Can you create a toy example of the kind of
bug you're concerned about?

y

Re: sexp_default Jim 3/31/12 12:16 PM
Perhaps I shouldn't have used the word "semantics". Here is an example
of what I mean:

module A = struct
  type t = {
    guest : string;
    food : string sexp_default("steak")!;
  } with sexp
end

module B = struct
  type t = A.t = {
    guest : string;
    food : string sexp_default("fish");
  } with sexp
end

  let () =
    let original = { A.
      guest = "Bob";
      food = "steak";
    } in
    let x = B.t_of_sexp (A.sexp_of_t original) in
    printf "%s ordered %s\n" x.A.guest x.A.food

If I understand the proposal correctly, this will say that Bob ordered
fish even though he originally ordered steak. Obviously, one can write
custom sexp converters to get this behavior already. I'm just
questioning the desirability of having this be easy.

- Jim

Re: sexp_default Markus 3/31/12 12:25 PM
On Sat, Mar 31, 2012 at 05:31, Jim Clune <jim....@gmail.com> wrote:
> Hi, Markus. Your proposal is interesting, but I'm a little concerned
> about this part:
>
>> Using manifest types, developers can easily generate equivalent
>> instantiations of a record type with different conversion behavior.
>> E.g.:
>>
>>    type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp
>>
>> This way different "default views" could be emitted for a given record.
>
> It is not clear to me whether this is best construed as a feature or a
> bug. It seems to encourage people to make type-equivalences that have
> sexp representations that can be generated by one type and parsed by
> an "equivalent" type that has silently incompatible semantics. This,
> combined with the fact that sexp representations do not embed the name
> of the types or modules that generated them, looks to me to like a
> recipe for bugs.

S-expressions are generally just structural.  There is no type
associated with any of them.  Nothing will prevent you from converting
an S-expression to a different type than the one it originated from as
long as the value is compatible in structure.  The same is true of the
binary protocol and the OCaml marshaling module btw.  If you need
guarantees about the originating type of such a value, you as a
developer need to take care of that.  E.g. by "nominal typing", i.e.
you annotate the value with an unambiguous identifier of its
originating type.  Or by more advanced structural matching that
requires that the originating type can be unified with the destination
type.  This would require storing the type-specification along with
the value.

The above discussion of "views" is somewhat unrelated to this.  All
views should really be equivalent.  It's more about what kind of
information should be revealed.  E.g. if a program raises an exception
and needs to show its user-supplied configuration for debugging
purposes, you may want to emit all record fields, because sometimes
the bug may be that the user had incorrect assumptions of what the
default is.  In other cases you might want to ask a process to
generate a minimal configuration file that reflects its current state.
 Leaving out default values would then make these configurations much
shorter and readable.  That's what I mean by "views".

> Do you consider this facilitation of different "default views" to be
> an important aspect of the proposal? Or is there a milder version of
> the proposal that does not have this feature?

There is no facilitation of "default views" in my proposal.  The macro
merely allows you to define defaults for record fields.  There is
nothing I can do to prevent users from using (or abusing) it by
combining it with features of the type and module system.  The
examples (e.g. views) I mentioned merely show potential use cases
where this feature can be creatively employed.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Markus 3/31/12 12:35 PM

There is no reasonable way for me to prevent users from shooting
themselves in the foot.  Manifest types need to be supported,
otherwise you wouldn't be able to e.g. automatically generate
S-expression converters for concrete types in external libraries that
don't use sexplib.  I cannot easily test whether anybody uses this
feature to generate incompatible defaults in equivalent types.  In
fact, one can imagine scenarios where different defaults are even
intentional.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Markus 3/31/12 7:00 PM
On Sat, Mar 31, 2012 at 07:50, Yaron Minsky <ymi...@janestreet.com> wrote:
> On Fri, Mar 30, 2012 at 6:09 PM, Markus Mottl <markus...@gmail.com> wrote:
>> I'd therefore suggest to use polymorphic equality for the while being.
>>  If it really turns out to cause issues in practice, you can always
>> add the mentioned optional equality function later.
>
> That sounds pretty reasonable.  Another option would be to use
> physical equality, which would have the property that if you left the
> default unmodified, it would work, but otherwise would re-output the
> value.  That's got its own set of surprising issues as well, though.

I've given it some more thought and may have an even better idea: the
user can optionally specify a function which, when evaluated on the
current field value, returns a boolean to indicate whether the field
should be included.  This is more general and useful than a mere
equivalence check against the default value.  This way a program could
even choose at runtime which fields should be included.  I'll
implement this once I have some more time.  I've pushed my preliminary
patch for the current approach, which will likely change
significantly.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Thomas Gazagnaire 4/2/12 1:33 AM
>  type t = { foo : int sexp_default(42) } with sexp

Why it this extension related to S-Exp and not type-conv directly? I mean, it would be nice if any type-conv plugin could use it (that's something that atdgen is doing for instance).

--
Thomas

Re: sexp_default Markus 4/2/12 5:56 AM

In fact, this is exactly what I'm intending to do.  My current idea is
the following:

  type t = { foo : int with sexp_default(42) } with sexp

The "with" extension would be part of type_conv again and would allow
you to record arbitrary "tags" in a hashtable for a specific location
(record field).  Such tags could then be optionally associated with
expressions.  I'm not sure other syntactic constructs (modules, etc.)
should also be allowed.  I guess for the while being expressions would
be sufficient.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com

Re: sexp_default Thomas Gazagnaire 4/2/12 6:12 AM
>>>  type t = { foo : int sexp_default(42) } with sexp
>>
>> Why it this extension related to S-Exp and not type-conv directly? I mean, it would be nice if any type-conv plugin could use it (that's something that atdgen is doing for instance).
>
> In fact, this is exactly what I'm intending to do.  My current idea is
> the following:
>
>  type t = { foo : int with sexp_default(42) } with sexp
>
> The "with" extension would be part of type_conv again and would allow
> you to record arbitrary "tags" in a hashtable for a specific location
> (record field).  Such tags could then be optionally associated with
> expressions.

so why not simply using:

 type t = { foo : int with default(42) } with sexp, binprot, ...

ie, using the same "default" field mean that the same default value can also be used by binprot (and others type converters).

>  I'm not sure other syntactic constructs (modules, etc.)
> should also be allowed.  I guess for the while being expressions would
> be sufficient.

Indeed, expressions would be sufficient.

--
Thomas

Re: sexp_default Markus 4/3/12 7:30 PM
On Mon, Apr 2, 2012 at 09:12, Thomas Gazagnaire
<thomas.g...@gmail.com> wrote:
> so why not simply using:
>
>  type t = { foo : int with default(42) } with sexp, binprot, ...
>
> ie, using the same "default" field mean that the same default value can also be used by binprot (and others type converters).

Though defaults would be useless with bin_prot due to the nature of
the protocol, I agree that this would be more generic.  I have
therefore made the "default" tag part of type_conv.  It is now trivial
for extensions using type_conv to add new handlers for record fields
or access the default expression for a given field.

>>  I'm not sure other syntactic constructs (modules, etc.)
>> should also be allowed.  I guess for the while being expressions would
>> be sufficient.
>
> Indeed, expressions would be sufficient.

The solution is fully generic now, too.  This means one could pass
arbitrary OCaml code (e.g. module names, etc.) as arguments, assuming
the registered tag can handle it.  In fact, part of the solution
already existed in type_conv for the usual "with" extension.

Sexplib will now behave as expected with e.g.:

  type t = { foo : int with default(42) } with sexp

Two more tags are available, too.  The first one is "sexp_drop_default", e.g.:

  type t = { foo : int with default(42), sexp_drop_default } with sexp

This will drop record fields that are (polymorphically) equivalent to
their default.  The second tag is "sexp_drop_if", e.g.:

  type t = { foo : int with sexp_drop_if((=) 3) } with sexp

"sexp_drop_if" specifies a function, which takes the current record
field value as argument and must return "true" if the field should be
dropped.  It can be combined with "default", but not with
"sexp_drop_default".  This solution makes it possible to configure
emission of record fields at runtime.

I have pushed the changes to my fork.  Please let me know if you have
any further suggestions.  Otherwise we can probably already merge
this.

--
Markus Mottl        http://www.ocaml.info        markus...@gmail.com