Whither sharp variables, uneval, toSource
Brendan Eich
successful and standardized in ES3, others will be standardized in
ES5. Still others are nominally "Harmonious", so likely to show up in
a future standard:
http://wiki.ecmascript.org/doku.php?id=harmony:harmony
But not everything is likely to be standardized, and no sunk costs are
cost-free goods in our evolving, security-bug-conscious, make-
everything-faster world.
Sharp variables (after Common Lisp), uneval, and the object-based
toSource method uneval calls for non-primitive value arguments, are
contentious. See
https://bugzilla.mozilla.org/show_bug.cgi?id=514981
These are used (you readers of this group, please post about your
uses). Firefox code used them at one point. The odds of uses in Weave
or a popular add-on are all too high. So we won't remove sharp
variables, etc., in a hurry or lightly. If we do that with everything
not yet standardized or on track to be standardized, we'll be a lot
more like our competition, and less positioned to lead.
But leadership on the sharp variable front really means putting the
right tools in library authors' hands so they can self-host
serialization routines like uneval, themselves. There's nothing
particularly privileged, or shoud not be anyway, about uneval/
toSource, that requires nativei implementation.
So my position is that we should self-host uneval/toSource/sharp-
variables (at least all but the parser code, which may remain native
for a while yet ;-). Problem is, you can't serialize a general object
graph in JS without some way of memoizing objects. JS objects as
hashtables annoyingly convert keys to strings (or act as if).
The solution is in sight for ECMAScript Harmony:
http://wiki.ecmascript.org/doku.php?id=strawman:weak_references
If we implement something like this (the draft needs a bit more work
and agreement), then we can self-host or defer to library authors the
in-language hosting of serializers and deserializers of arbitrary
object graphs. That would be a good day to wake up to, wouldn't it?
Anyone want to take on shepherding/prototyping WeakRefs in
SpiderMonkey? It goes along with our GC work:
https://wiki.mozilla.org/JavaScript:SpiderMonkey:GC_Futures
/be
Brendan Eich
js> var o = {p:1, q:2, r:3}
js> o.cycle = o
[object Object]
js> uneval(o)
#1={p:1, q:2, r:3, cycle:#1#}
js> o.q = ["foo"]
foo
js> o.r = o.q
foo
js> uneval(o)
#2={p:1, q:#1=["foo"], r:#1#, cycle:#2#}
js> a = [4,5,6]
4,5,6
js> a[2] = o
[object Object]
js> uneval(a)
[4, 5, #2={p:1, q:#1=["foo"], r:#1#, cycle:#2#}]
js> o.p = a
4,5,[object Object]
js> uneval(a)
#1=[4, 5, #3={p:#1#, q:#2=["foo"], r:#2#, cycle:#3#}]
/be
Jeff Walden
> Sharp variables (after Common Lisp), uneval, and the object-based
> toSource method uneval calls for non-primitive value arguments, are
> contentious.
Please don't conflate sharp variables with uneval and toSource as being contentious, at least with respect to me. If I've ever given the impression that I was doing so, I apologize; I didn't mean to do so.
uneval and toSource are nominally useful debugging aids; I wouldn't add them myself if we didn't already have them (mostly due to the capturing problem for embedded functions), but I see little reason to remove them. Both, I believe, can be made cost-free (save for codesize and for constant runtime overhead to define 'uneval' and 'toSource' properties on the global and on Object.prototype, plus a small constant amount during GC to trace them, but we pay as much for many other useful extensions -- the array extras for one), if they aren't already. Your point about self-hosting is a good one which we should indeed explore (although I regrettably remain focused on ES5 work at the moment and don't have the time to follow up on it).
On the other hand, sharp variables impose meaningful costs. They complicate decompilation of array and object literals (decompilation in general being an already-tricky process fraught in the past with bugs of which I'm sure we've not seen the end, in old or new code). They add extra code to the array-literal parsing methods -- well-predicted code, to be sure, but code and (more importantly) complexity nevertheless. They require special-casing of the opcodes emitted for array and object literals that use them, which further requires special-casing of the interpreting and tracing code that handles those opcodes. They have further complexified eval. For a long time they have fattened SpiderMonkey stack frames, a cost only just now being removed. All this imposes costs on hackers modifying SpiderMonkey internals, sometimes in areas far from literal-processing.
The only uses of sharp variables in the Mozilla tree are in the SpiderMonkey test suite, as best as I can tell. Past uses have been removed because the syntax is arcane. I discovered the sharp variable syntax when stumbling over one such use, and it took some effort to find any documentation at all of the syntax (it doesn't help that the syntax has no keywords which might make web searches more likely to succeed). Until about two years ago, they were entirely undocumented except in an old ECMAScript proto-draft promulgated by Netscape (maybe Mozilla too depending on its date of publication). Of the SpiderMonkey extensions, they are perhaps the least-documented.
Sharp variables are and have been ripe for removal. We have removed features such as import/export with more documentation (they were explicitly documented in Netscape's JS1.5 docs) in the past. We have removed features such as assigning to method return values (element.childNodes.item(0) = elt) with more use (or so I expect, given general awareness of the syntax in standards communities in which I've participated). We have taken pains to support non-standard features like an optional second argument to eval, only to discover that when we've forgotten to preserve that support no one has complained.
https://bugzilla.mozilla.org/show_bug.cgi?id=446026
I do not expect to see more than one or two uses (at most) among commenters in this newsgroup. I would be similarly surprised if more than one or two extensions use them, given that there are so few examples of their use. (It's true we can't know the numbers for either case -- but that can be said of other instances where we have nevertheless taken leaps.) Existing uses can be rewritten without much difficulty or loss of understandability (indeed an increase in it, for those not familiar with our extensions). Of the areas where we might or should show leadership, I don't think the market has demonstrated or will demonstrate that this is an area where our efforts are appreciated. I do not think we should spare this city, even for ten righteous uses, not when there are simple workarounds and not while we pay a continuing cost for a feature seeing almost no use.
Maybe (probably) I won't change your mind, but I feel compelled to state my case (I know others agree with it), and I hope in doing so others will rally to it. Where I fail alone, maybe I can succeed with assistance.
Jeff
Brendan Eich
but about the subject matter: removing an extension vs. maintaining it
and adding value by providing self-hosting (and generally useful/
powerful) infrastructure.
The upshot of my reply was that we don't (we meaning neither Jeff nor
I) get to *assume* sharp variables are unused, while making irrelevant
excuses for uneval/toSource, or inconsistent comparisons to the
removal of broken-import/export or un-JITtable-lvalue-return from
SpiderMonkey as an embeddable engine.
We'll get rid of sharp variables when the time is ripe to spend cycles
removing them, where "ripe" means exactly what I proposed: that we
support WeakRefs and EphemeronTables so that anyone can write object
graph serializers using whatever syntax they prefer. In such a future,
sharp variable support becomes a library, which we may or may not
bundle.
My explicit call for leadership on specifying and prototyping WeakRefs
is not to be confused (or conflated :-P) with saying sharp variables
should be supported forever, or standardized as-is. I think this is
crystal clear from the original post.
In point of fact, serializing arbitrarily connected object graphs via
uneval, using sharp variable syntax, is the reason for uneval as well
as sharp vars. And notable SpiderMonkey embedders do use sharp
variables along with uneval in significant, non-web or otherwise
firewalled embeddings. The ones I'm thinking of can identify
themselves if they read this. Anyway, no one gets to shoot from the
hip on yanking sharps while these embedders have no backup or ability
to self-host.
At one point Igor wanted to replace sharp variables with lambdas
enclosing object initialisers, binding lets to cycling/joined object
subgraphs, returning values to outer initializers/lambda-calls. This
could be done (getting arguments through eval would be tricky) but it
looks like a lot of work, with high opportunity cost given the GC work
before us.
This all may seem overdone, too "inside baseball", to you readers if
you don't know what uneval is, but it is important to proceed in a
principled fashion when dropping SpiderMonkey extensions or (better)
boiling them down to more powerful infrastructure that can be used for
self-hosting the built-in feature as library code. Breaking too much
without replacement parts or a credible reason why the removals
couldn't have replacements is a fast way to lose embedders.
/be
Brendan Eich
> In point of fact, serializing arbitrarily connected object graphs via
> uneval, using sharp variable syntax, is the reason for uneval as well
> as sharp vars. And notable SpiderMonkey embedders do use sharp
> variables along with uneval in significant, non-web or otherwise
> firewalled embeddings. The ones I'm thinking of can identify
> themselves if they read this. Anyway, no one gets to shoot from the
> hip on yanking sharps while these embedders have no backup or ability
> to self-host.
Apart from embedders, there's at least this much usage:
http://www.google.com/codesearch?q=lang%3Ajavascript+%22%231%3D%22&hl=en&btnG=Search+Code
Some still in hg.mozilla.org/mozilla-central, it seems.
/be
Jason Orendorff
On 09/09/2009 06:56 PM, Brendan Eich wrote:
> In point of fact, serializing arbitrarily connected object graphs via
> uneval, using sharp variable syntax, is the reason for uneval as well
> as sharp vars.
I use uneval frequently, but in debugging, tests, and error messages.
Never for this kind of serialization.
I wish the shell would auto-uneval in interactive mode, because what I'm
doing in the shell is asking questions, and I prefer precise answers.
js> x
1,,,,,
js> uneval(x)
[1, ["", ""], [, ,], (void 0)]
Python and Ruby do it right.
In tests, I can say
assertEq(uneval(actual), uneval(expected));
to ask for, roughly, Lisp (equal) as opposed to (eq). We have
several tests that do this. Offhand:
js/src/trace-test/tests/basic/testNEWINIT.js
js/tests/js1_7/extensions/iterator-ctor.js
assertEq uses JS_ValueToSource to produce useful error messages. So does
the jsapi-tests harness. Several tests use uneval() that way.
I think these uses are a big deal and uneval is worth keeping.
-j
Brendan Eich
> Let me try and make a relevant excuse for uneval.
No need, I like it too. For some reason jsversion.h separates uneval
and toSource but they are really the same feature, call it "uneval".
Embedders using sharps along with uneval for serialization that JSON
can't handle (join-points,cycles) do care about sharps, but let's
separate that concern.
Even without sharps, you have an issue with toSource/uneval. Should we
just suppress runaway recursion and produce the empty string? Or some
kind of #cycle# ugly but unparseable sentinel? Here's what you get
configuring JS_HAS_SHARP_VAR 0 but leaving JS_HAS_UNEVAL and
JS_HAS_TOSOURCE:
$ ./Darwin_DBG.OBJ/js
js> a = [1,2,3]
1,2,3
js> a[0] = a
,2,3
js> uneval(a)
typein:3: InternalError: too much recursion
And here's a join point:
js> b = [4,5,6]
4,5,6
js> b[0] = {p:42}
[object Object]
js> b[2] = b[0]
[object Object]
js> uneval(b)
[#1={p:42}, 5, {}]
Not so nice, kinda misleading. But we keep sharps for now, so no
worries. Getting all this self-hosted and memory-safe with high
control-flow-integrity is the prize.
But good point about the shell, a closer prize worth winning. Please
do file a bug asking for the shell to uneval instead of toString the
result of each complete input logical-line.
/be
Donny Viszneki
worthy segue I could find for the question I'd like to raise...
On Wed, Sep 9, 2009 at 6:37 PM, Jeff Walden <jwald...@mit.edu> wrote:
> Sharp variables are and have been ripe for removal.
The feature that sharp variables provide could be very expensive to
provide in pure Javascript. How is it implemented right now in C++? I
imagine that you must walk the entire object graph just to take
inventory of every object you'll be serializing. In Javascript, we
don't have meaningful comparison operators beyond equality, so it's
not like we could do a binary search on that inventory, could we? Just
to take inventory on an object graph with 'n' nodes costs like
O(n^2/2).
On the other hand, maybe serialization is never a bottleneck, anyway,
since if you're serializing something you probably want to send it
somewhere...
Brendan Eich
> The feature that sharp variables provide could be very expensive to
> provide in pure Javascript. How is it implemented right now in C++? I
> imagine that you must walk the entire object graph just to take
> inventory of every object you'll be serializing. In Javascript, we
> don't have meaningful comparison operators beyond equality, so it's
> not like we could do a binary search on that inventory, could we? Just
> to take inventory on an object graph with 'n' nodes costs like
> O(n^2/2).
This is a bug in JS, not a fault of sharp variables. Its fix is in
sight for Harmony, as noted:
http://wiki.ecmascript.org/doku.php?id=strawman:weak_references
> On the other hand, maybe serialization is never a bottleneck, anyway,
> since if you're serializing something you probably want to send it
> somewhere...
Right.
/be
Mike Shaver
> Sharp variables in action via uneval:
>
> js> var o = {p:1, q:2, r:3}
> js> o.cycle = o
> [object Object]
> js> uneval(o)
> #1={p:1, q:2, r:3, cycle:#1#}
An alternate production here, which would not require sharp variables
and could therefore be consumed by any ECMA engine, would be:
js> uneval(o)
((function() { var o_ = {p:1, q:2, r:3}; o_.cycle = o_; return o_;})())
This would let us isolate the complexity cost more completely to the
implementation and uses of uneval, I think, or at least let us
separate the discussions of sharp variables from those of uneval or
toSource.
There are a handful of sharp-variable uses in mozilla-central, some of
which would not be easily expressible using an imperative sort of
form, but excising them is probably a matter of good style independent
of any decision we take about the future of their support in
SpiderMonkey proper.
Mike
Brendan Eich
> On Wed, Sep 9, 2009 at 3:36 PM, Brendan Eich <bren...@mozilla.org> wrote:
> > Sharp variables in action via uneval:
>
> > js> var o = {p:1, q:2, r:3}
> > js> o.cycle = o
> > [object Object]
> > js> uneval(o)
> > #1={p:1, q:2, r:3, cycle:#1#}
>
> An alternate production here, which would not require sharp variables
> and could therefore be consumed by any ECMA engine, would be:
>
> js> uneval(o)
> ((function() { var o_ = {p:1, q:2, r:3}; o_.cycle = o_; return o_;})())
I mentioned this in a later post in this thread:
"At one point Igor wanted to replace sharp variables with lambdas
enclosing object initialisers, binding lets to cycling/joined object
subgraphs, returning values to outer initializers/lambda-calls. This
could be done (getting arguments through eval would be tricky) but it
looks like a lot of work, with high opportunity cost given the GC work
before us."
> This would let us isolate the complexity cost more completely to the
> implementation and uses of uneval, I think, or at least let us
> separate the discussions of sharp variables from those of uneval or
> toSource.
This would be a ton of work rearranging the deck chairs on the
Titanic :-P.
Prioritizing this ahead of weak ref support that allows either sharps
or lambdas to be used for serializing would be a mistake.
/be
Jason Orendorff
> Getting all this self-hosted and memory-safe with high
> control-flow-integrity is the prize.
OK. We should get "pure js implementation of uneval" filed with the
student-project tag.
However, currently JS doesn't have the right building blocks to
implement this efficiently. Lars' proposed-for-ES4 identity hashmap
would do it, I think. Or just a function
Object.identityHashCode(obj) -> number
> But good point about the shell, a closer prize worth winning. Please
> do file a bug asking for the shell to uneval instead of toString the
> result of each complete input logical-line.
Bug 516715, with patch, r?you.
-j
Donny Viszneki
<joren...@mozilla.com> wrote:
> However, currently JS doesn't have the right building blocks to implement
> this efficiently.
It just occurred to me that if the Array.indexOf() instance method is
efficient, then you can implement this in Javascript with reasonable
efficiency.