sexp_default
Markus Mottl
I have a patch for sexplib, which adds a new record field qualifier
"sexp_default". It allows users to explicitly add the default
expression to be used when a record field is missing in an
S-expression. E.g.:
type t = { foo : int sexp_default(42) } with sexp
In this definition "foo" will default to "42" if the S-expression does
not specify "foo". Any valid OCaml-expression is allowed within the
parentheses, including variables and even function calls. When such a
record is converted to an S-expression, the normal behavior is to
always emit the field, no matter whether it is equivalent to the
default or not. The reason is that unlike "sexp_list", "sexp_array",
etc., it may not be obvious to the reader of an S-expression what the
default may be.
If, however, default fields should be excluded from the conversion,
e.g. to generate a minimal default configuration file for later
refinement, the following is supported (note the exclamation mark at
the end):
type t = { foo : int sexp_default(42)! } with sexp
Using manifest types, developers can easily generate equivalent
instantiations of a record type with different conversion behavior.
E.g.:
type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp
This way different "default views" could be emitted for a given record.
Using manifest types together with module functors and maybe
first-class modules to do this dynamically for large numbers of
instantiations, one can also easily generate numerous default
converters for different contexts. E.g.:
module Make (Defaults : sig val default : t end) = struct
type s = t = { foo : int sexp_default(Defaults.default.foo) } with sexp
end
Some wizardry with modules and manifest types provides for almost
unlimited power for providing defaults and configuring how they are
emitted.
Let me know if this patch would likely be accepted or if there are any
suggestions for improvement, and I'll push it to my fork of OCaml
Core.
Regards,
Markus
--
Markus Mottl http://www.ocaml.info markus...@gmail.com
Yaron Minsky
folk who think about the syntax extensions to weigh in, but at first
glance, the proposal seems good.
I am a little worried about the semantics of !, though. If I create a type:
type t = { foo : int sexp_default(7)! } with sexp
then the round-tripping could be pretty weird. In particular,
consider this sequence:
{ foo = 3 }
|! sexp_of_t (* this is () *)
|! t_of_sexp (* this is { foo = 7 } *)
This behavior seems pretty unfortunate. Is that really what it does?
As a side note, Markus, if you're working on more patches: please
don't do merges in the stuff your proposing. Yury's conversion code
doesn't work well on merges. (merges purely within your code are
actually OK, as long as you're not merging in from different points in
the history that was exported from Jane Street.)
y
Markus Mottl
> I am a little worried about the semantics of !, though. If I create a type:
>
> type t = { foo : int sexp_default(7)! } with sexp
>
> then the round-tripping could be pretty weird. In particular,
> consider this sequence:
>
> { foo = 3 }
> |! sexp_of_t (* this is () *)
> |! t_of_sexp (* this is { foo = 7 } *)
>
> This behavior seems pretty unfortunate. Is that really what it does?
No, it would consider equivalence to the default expression first to
determine whether to drop the field in the S-expression. In your case
it would convert the record to ((foo 3)). If, however, you had used {
foo = 7 }, then it would convert it to () as expected. Without the !,
it would always emit the field no matter the content.
Due to that equivalence check it is probably not a good idea to use
expensive function calls or gargantuan values as default expressions.
But that doesn't seem necessary in practice anyway.
> As a side note, Markus, if you're working on more patches: please
> don't do merges in the stuff your proposing. Yury's conversion code
> doesn't work well on merges. (merges purely within your code are
> actually OK, as long as you're not merging in from different points in
> the history that was exported from Jane Street.)
Hm, well, that surely makes it hard to work on the same files, which
would require merges. As long as we don't tread on each other's feet
that should be possible though.
Yaron Minsky
> On Fri, Mar 30, 2012 at 15:45, Yaron Minsky <ymi...@janestreet.com> wrote:
>> I am a little worried about the semantics of !, though. If I create a type:
>>
>> type t = { foo : int sexp_default(7)! } with sexp
>>
>> then the round-tripping could be pretty weird. In particular,
>> consider this sequence:
>>
>> { foo = 3 }
>> |! sexp_of_t (* this is () *)
>> |! t_of_sexp (* this is { foo = 7 } *)
>>
>> This behavior seems pretty unfortunate. Is that really what it does?
>
> No, it would consider equivalence to the default expression first to
> determine whether to drop the field in the S-expression. In your case
> it would convert the record to ((foo 3)). If, however, you had used {
> foo = 7 }, then it would convert it to () as expected. Without the !,
> it would always emit the field no matter the content.
>
> Due to that equivalence check it is probably not a good idea to use
> expensive function calls or gargantuan values as default expressions.
> But that doesn't seem necessary in practice anyway.
Got it. Is the equality check based on polymorphic compare, or
equality of the s-expression representation?
>> As a side note, Markus, if you're working on more patches: please
>> don't do merges in the stuff your proposing. Yury's conversion code
>> doesn't work well on merges. (merges purely within your code are
>> actually OK, as long as you're not merging in from different points in
>> the history that was exported from Jane Street.)
>
> Hm, well, that surely makes it hard to work on the same files, which
> would require merges. As long as we don't tread on each other's feet
> that should be possible though.
Yeah, we'll have to figure it out over time. I think right now one
can do all the merging one wants, subject to the constraint that all
the work is anchored to the same spot in our HG repo. But the
simplest use-case is just linear histories. Right now Yury is
manually dealing with the conversion of the revisions we've pulled in
so far, which is kind of painful.
Markus Mottl
> On Fri, Mar 30, 2012 at 4:25 PM, Markus Mottl <markus...@gmail.com> wrote:
>> Due to that equivalence check it is probably not a good idea to use
>> expensive function calls or gargantuan values as default expressions.
>> But that doesn't seem necessary in practice anyway.
>
> Got it. Is the equality check based on polymorphic compare, or
> equality of the s-expression representation?
Polymorphic equality, which seems preferable for efficiency. Neither
it nor equality of S-expressions is guaranteed to work on abstract
datatypes anyway.
> Yeah, we'll have to figure it out over time. I think right now one
> can do all the merging one wants, subject to the constraint that all
> the work is anchored to the same spot in our HG repo. But the
> simplest use-case is just linear histories. Right now Yury is
> manually dealing with the conversion of the revisions we've pulled in
> so far, which is kind of painful.
Ok, I'll not merge your tree into the one from which I push in the future.
Yaron Minsky
> On Fri, Mar 30, 2012 at 16:44, Yaron Minsky <ymi...@janestreet.com> wrote:
>> On Fri, Mar 30, 2012 at 4:25 PM, Markus Mottl <markus...@gmail.com> wrote:
>>> Due to that equivalence check it is probably not a good idea to use
>>> expensive function calls or gargantuan values as default expressions.
>>> But that doesn't seem necessary in practice anyway.
>>
>> Got it. Is the equality check based on polymorphic compare, or
>> equality of the s-expression representation?
>
> Polymorphic equality, which seems preferable for efficiency. Neither
> it nor equality of S-expressions is guaranteed to work on abstract
> datatypes anyway.
Hrm. The polymorphic equality thing makes me a little sick --- we're
actually trying to avoid polymorphic compare wherever we can these
days, for both performance and semantic reasons. Is there a reason it
can't use comparison of s-expressions? I don't think efficiency is a
primary concern here, since this is already fairly slow
pretty-printing code.
Markus Mottl
> Hrm. The polymorphic equality thing makes me a little sick --- we're
> actually trying to avoid polymorphic compare wherever we can these
> days, for both performance and semantic reasons. Is there a reason it
> can't use comparison of s-expressions? I don't think efficiency is a
> primary concern here, since this is already fairly slow
> pretty-printing code.
I guess nobody is fully satisfied with polymorphic equality in
general. I'm not sure there is any true gain in comparing
S-expressions. One could imagine that some S-expression converters
normalize the representation so that e.g. sets, maps, etc., would
compare consistently, but this is not a given (e.g. hashtables).
Furthermore, in the vast majority of cases we would needlessly convert
a value to an S-expression just for equality checking even though it
need not be emitted in the first place - a rather expensive default
behavior.
The only consistent solution I can see is if the user can optionally
specify an equivalence relation, e.g. by having another optional
expression that contains such a function. But this could turn out to
be extremely expensive: think of comparing hashtables.
Default field values are probably hardly ever changed back and forth
at runtime, especially in ways that would destroy pointer equality or
worse to a structurally distinct but semantically equivalent
representation. Polymorphic equality checks would therefore almost
always just compare two identical pointers anyway, which is both
consistent and maximally efficient. Even if a field were emitted that
need not be, this should never cause any issues other than diminished
output readability.
I'd therefore suggest to use polymorphic equality for the while being.
If it really turns out to cause issues in practice, you can always
add the mentioned optional equality function later.
Jim Clune
about this part:
> Using manifest types, developers can easily generate equivalent
> instantiations of a record type with different conversion behavior.
> E.g.:
>
> type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp
>
> This way different "default views" could be emitted for a given record.
It is not clear to me whether this is best construed as a feature or a
bug. It seems to encourage people to make type-equivalences that have
sexp representations that can be generated by one type and parsed by
an "equivalent" type that has silently incompatible semantics. This,
combined with the fact that sexp representations do not embed the name
of the types or modules that generated them, looks to me to like a
recipe for bugs.
Do you consider this facilitation of different "default views" to be
an important aspect of the proposal? Or is there a milder version of
the proposal that does not have this feature?
- Jim
Yaron Minsky
> On Fri, Mar 30, 2012 at 17:28, Yaron Minsky <ymi...@janestreet.com> wrote:
>> Hrm. The polymorphic equality thing makes me a little sick --- we're
>> actually trying to avoid polymorphic compare wherever we can these
>> days, for both performance and semantic reasons. Is there a reason it
>> can't use comparison of s-expressions? I don't think efficiency is a
>> primary concern here, since this is already fairly slow
>> pretty-printing code.
>
> I guess nobody is fully satisfied with polymorphic equality in
> general. I'm not sure there is any true gain in comparing
> S-expressions. One could imagine that some S-expression converters
> normalize the representation so that e.g. sets, maps, etc., would
> compare consistently, but this is not a given (e.g. hashtables).
> Furthermore, in the vast majority of cases we would needlessly convert
> a value to an S-expression just for equality checking even though it
> need not be emitted in the first place - a rather expensive default
> behavior.
Fair.
> The only consistent solution I can see is if the user can optionally
> specify an equivalence relation, e.g. by having another optional
> expression that contains such a function. But this could turn out to
> be extremely expensive: think of comparing hashtables.
>
> Default field values are probably hardly ever changed back and forth
> at runtime, especially in ways that would destroy pointer equality or
> worse to a structurally distinct but semantically equivalent
> representation. Polymorphic equality checks would therefore almost
> always just compare two identical pointers anyway, which is both
> consistent and maximally efficient. Even if a field were emitted that
> need not be, this should never cause any issues other than diminished
> output readability.
>
> I'd therefore suggest to use polymorphic equality for the while being.
> If it really turns out to cause issues in practice, you can always
> add the mentioned optional equality function later.
That sounds pretty reasonable. Another option would be to use
physical equality, which would have the property that if you left the
default unmodified, it would work, but otherwise would re-output the
value. That's got its own set of surprising issues as well, though.
y
Yaron Minsky
use this internally quite a bit when we want to generate an alternate
view without creating a new type. We just create a new type-alias,
and attach the specialized converters to that type.
I guess I'm just saying I'm in the "feature, not bug" camp. That
said, I'm not sure what you mean when you talk about "silently
incompatible semantics". Can you create a toy example of the kind of
bug you're concerned about?
y
Jim Clune
of what I mean:
module A = struct
type t = {
guest : string;
food : string sexp_default("steak")!;
} with sexp
end
module B = struct
type t = A.t = {
guest : string;
food : string sexp_default("fish");
} with sexp
end
let () =
let original = { A.
guest = "Bob";
food = "steak";
} in
let x = B.t_of_sexp (A.sexp_of_t original) in
printf "%s ordered %s\n" x.A.guest x.A.food
If I understand the proposal correctly, this will say that Bob ordered
fish even though he originally ordered steak. Obviously, one can write
custom sexp converters to get this behavior already. I'm just
questioning the desirability of having this be easy.
- Jim
Markus Mottl
> Hi, Markus. Your proposal is interesting, but I'm a little concerned
> about this part:
>
>> Using manifest types, developers can easily generate equivalent
>> instantiations of a record type with different conversion behavior.
>> E.g.:
>>
>> type drop_foo_default = t = { foo : int sexp_default(foo)! } with sexp
>>
>> This way different "default views" could be emitted for a given record.
>
> It is not clear to me whether this is best construed as a feature or a
> bug. It seems to encourage people to make type-equivalences that have
> sexp representations that can be generated by one type and parsed by
> an "equivalent" type that has silently incompatible semantics. This,
> combined with the fact that sexp representations do not embed the name
> of the types or modules that generated them, looks to me to like a
> recipe for bugs.
S-expressions are generally just structural. There is no type
associated with any of them. Nothing will prevent you from converting
an S-expression to a different type than the one it originated from as
long as the value is compatible in structure. The same is true of the
binary protocol and the OCaml marshaling module btw. If you need
guarantees about the originating type of such a value, you as a
developer need to take care of that. E.g. by "nominal typing", i.e.
you annotate the value with an unambiguous identifier of its
originating type. Or by more advanced structural matching that
requires that the originating type can be unified with the destination
type. This would require storing the type-specification along with
the value.
The above discussion of "views" is somewhat unrelated to this. All
views should really be equivalent. It's more about what kind of
information should be revealed. E.g. if a program raises an exception
and needs to show its user-supplied configuration for debugging
purposes, you may want to emit all record fields, because sometimes
the bug may be that the user had incorrect assumptions of what the
default is. In other cases you might want to ask a process to
generate a minimal configuration file that reflects its current state.
Leaving out default values would then make these configurations much
shorter and readable. That's what I mean by "views".
> Do you consider this facilitation of different "default views" to be
> an important aspect of the proposal? Or is there a milder version of
> the proposal that does not have this feature?
There is no facilitation of "default views" in my proposal. The macro
merely allows you to define defaults for record fields. There is
nothing I can do to prevent users from using (or abusing) it by
combining it with features of the type and module system. The
examples (e.g. views) I mentioned merely show potential use cases
where this feature can be creatively employed.
Markus Mottl
There is no reasonable way for me to prevent users from shooting
themselves in the foot. Manifest types need to be supported,
otherwise you wouldn't be able to e.g. automatically generate
S-expression converters for concrete types in external libraries that
don't use sexplib. I cannot easily test whether anybody uses this
feature to generate incompatible defaults in equivalent types. In
fact, one can imagine scenarios where different defaults are even
intentional.
Markus Mottl
I've given it some more thought and may have an even better idea: the
user can optionally specify a function which, when evaluated on the
current field value, returns a boolean to indicate whether the field
should be included. This is more general and useful than a mere
equivalence check against the default value. This way a program could
even choose at runtime which fields should be included. I'll
implement this once I have some more time. I've pushed my preliminary
patch for the current approach, which will likely change
significantly.
Thomas Gazagnaire
Why it this extension related to S-Exp and not type-conv directly? I mean, it would be nice if any type-conv plugin could use it (that's something that atdgen is doing for instance).
--
Thomas
Markus Mottl
In fact, this is exactly what I'm intending to do. My current idea is
the following:
type t = { foo : int with sexp_default(42) } with sexp
The "with" extension would be part of type_conv again and would allow
you to record arbitrary "tags" in a hashtable for a specific location
(record field). Such tags could then be optionally associated with
expressions. I'm not sure other syntactic constructs (modules, etc.)
should also be allowed. I guess for the while being expressions would
be sufficient.
Thomas Gazagnaire
>>
>> Why it this extension related to S-Exp and not type-conv directly? I mean, it would be nice if any type-conv plugin could use it (that's something that atdgen is doing for instance).
>
> In fact, this is exactly what I'm intending to do. My current idea is
> the following:
>
> type t = { foo : int with sexp_default(42) } with sexp
>
> The "with" extension would be part of type_conv again and would allow
> you to record arbitrary "tags" in a hashtable for a specific location
> (record field). Such tags could then be optionally associated with
> expressions.
so why not simply using:
type t = { foo : int with default(42) } with sexp, binprot, ...
ie, using the same "default" field mean that the same default value can also be used by binprot (and others type converters).
> I'm not sure other syntactic constructs (modules, etc.)
> should also be allowed. I guess for the while being expressions would
> be sufficient.
Indeed, expressions would be sufficient.
--
Thomas
Markus Mottl
<thomas.g...@gmail.com> wrote:
> so why not simply using:
>
> type t = { foo : int with default(42) } with sexp, binprot, ...
>
> ie, using the same "default" field mean that the same default value can also be used by binprot (and others type converters).
Though defaults would be useless with bin_prot due to the nature of
the protocol, I agree that this would be more generic. I have
therefore made the "default" tag part of type_conv. It is now trivial
for extensions using type_conv to add new handlers for record fields
or access the default expression for a given field.
>> I'm not sure other syntactic constructs (modules, etc.)
>> should also be allowed. I guess for the while being expressions would
>> be sufficient.
>
> Indeed, expressions would be sufficient.
The solution is fully generic now, too. This means one could pass
arbitrary OCaml code (e.g. module names, etc.) as arguments, assuming
the registered tag can handle it. In fact, part of the solution
already existed in type_conv for the usual "with" extension.
Sexplib will now behave as expected with e.g.:
type t = { foo : int with default(42) } with sexp
Two more tags are available, too. The first one is "sexp_drop_default", e.g.:
type t = { foo : int with default(42), sexp_drop_default } with sexp
This will drop record fields that are (polymorphically) equivalent to
their default. The second tag is "sexp_drop_if", e.g.:
type t = { foo : int with sexp_drop_if((=) 3) } with sexp
"sexp_drop_if" specifies a function, which takes the current record
field value as argument and must return "true" if the field should be
dropped. It can be combined with "default", but not with
"sexp_drop_default". This solution makes it possible to configure
emission of record fields at runtime.
I have pushed the changes to my fork. Please let me know if you have
any further suggestions. Otherwise we can probably already merge
this.