Source maps in ES7?

[Fitzgerald, Nick]

There has been a little bit of talk on es-discuss about properly
standardizing source maps for inclusion in ES7. My reply outgrew an
email response, and so I collected my thoughts here:
http://fitzgeraldnick.com/weblog/55/

It is probably of interest to this group as well, so I'm linking it here.

Nick

[Conrad Irwin]

Thanks for writing this down Nick. Do you have a link to the ES7 discussion?

I love the idea of Sourcemaps.next, though would love to see the
initial source-maps standardized too. They're a tool that, despite the
warts, fixes a large number of real-world problems.

Thanks again,
Conrad

> _______________________________________________
> dev-js-sourcemap mailing list
> dev-js-s...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-js-sourcemap

[Fitzgerald, Nick]

The link to es-discuss is here:
http://esdiscuss.org/topic/multiline-strings#content-7

And here: http://esdiscuss.org/topic/source-maps-was-multiline-strings

A lot of people have expressed interest in properly standardizing source
maps, and this was just the most recent (and being es-discuss) somewhat
formal request.

[Conrad Irwin]

Thanks. Yes, I'm also very interested :).

Conrad

On Wed, Mar 12, 2014 at 2:35 PM, Fitzgerald, Nick

[Brian Slesinsky]

Interesting. I think the sticky part will be providing the getLocals(),
getDisplayValue(), and eval() functions. I think it's right that they
should be language-specific JavaScript code provided by the compiler, since
this allows a lot of flexibility. But which JavaScript environment should
they run in and what do they have access to? I think it would have to be a
sandbox of some sort and there may be security implications for curious
developers who run untrusted code with the debugger window open.

Another issue is discovering the language-specific runtime type associated
with an arbitrary JavaScript object. For complicated data structures,
knowing the AST node for a local variable may not provide much help. For
example, in Java, you may have a variable with a static type of Object and
the runtime type could be anything. You also might want to look at globals
from the console, and language-specific variable inspectors would be useful
there too.

How to discover the runtime type will be language-specific, and
furthermore, the value stored in the variable may come from a different
language. For example, GWT allows passing both Java and JavaScript objects
back and forth. So I think we will need a way to ask each language if it
wishes to "claim" an object and provide the variable inspector for that
object. Probably the language associated with the AST node goes first, but
if it doesn't know what it is, the other languages should be queried.

- Brian

[Fitzgerald, Nick]

On 3/12/14 2:40 PM, Brian Slesinsky wrote:
> Interesting. I think the sticky part will be providing the
> getLocals(), getDisplayValue(), and eval() functions. I think it's
> right that they should be language-specific JavaScript code provided
> by the compiler, since this allows a lot of flexibility. But which
> JavaScript environment should they run in and what do they have access
> to? I think it would have to be a sandbox of some sort and there may
> be security implications for curious developers who run untrusted code
> with the debugger window open.

I was imagining it would run in the frame the debugger was paused in.
That way it would have access to whatever JavaScript locals exist.

Perhaps I am missing something, but if you are already running a site's
code, why would they need to send the bad stuff via the debugging
information?

It should still respect CORS, though.

Furthermore, using the debugging information should always be optional.

>
> Another issue is discovering the language-specific runtime type
> associated with an arbitrary JavaScript object. For complicated data
> structures, knowing the AST node for a local variable may not provide
> much help. For example, in Java, you may have a variable with a static
> type of Object and the runtime type could be anything. You also might
> want to look at globals from the console, and language-specific
> variable inspectors would be useful there too.
>
> How to discover the runtime type will be language-specific, and
> furthermore, the value stored in the variable may come from a
> different language. For example, GWT allows passing both Java and
> JavaScript objects back and forth. So I think we will need a way to
> ask each language if it wishes to "claim" an object and provide the
> variable inspector for that object. Probably the language associated
> with the AST node goes first, but if it doesn't know what it is, the
> other languages should be queried.
>

Sure. This is a perfect example of why I focused on an extensible format
-- so we can add annotations that we initially overlooked, and fix
mistakes post facto.

Nick

[Brian Slesinsky]

On Wed, Mar 12, 2014 at 5:04 PM, Fitzgerald, Nick
<nfitz...@mozilla.com>wrote:

> On 3/12/14 2:40 PM, Brian Slesinsky wrote:
> > Interesting. I think the sticky part will be providing the
> > getLocals(), getDisplayValue(), and eval() functions. I think it's
> > right that they should be language-specific JavaScript code provided
> > by the compiler, since this allows a lot of flexibility. But which
> > JavaScript environment should they run in and what do they have access
> > to? I think it would have to be a sandbox of some sort and there may
> > be security implications for curious developers who run untrusted code
> > with the debugger window open.
>
> I was imagining it would run in the frame the debugger was paused in.
> That way it would have access to whatever JavaScript locals exist.
>

However that means the debugger's view of things could possibly be broken
by someone monkey-patching a prototype in the web page itself (or possibly
by an extension changing something). A debugger or a debugger plugin should
work even when things have gone very wrong in the program being debugged.
But it's hard to write code sufficiently paranoid to work in any
environment.

> Perhaps I am missing something, but if you are already running a site's
> code, why would they need to send the bad stuff via the debugging
> information?
>

We would be creating a place to hide JavaScript code where nearly all web
developers wouldn't suspect it, which sounds like an interesting target for
a hacker.

It would be a problem if it somehow ran with higher privileges than the
code being debugged. For example, suppose the debugger called the plugin
code when it's looking at the local variables of a function that's running
in an iframe that came from a different origin? Now we have an interesting
way to get around same-origin restrictions.

So I think running each compiler's custom code as a separate low-privilege
extension makes the most sense, both to protect it from outside
interference and to make sure it can't do anything malicious.

- Brian

[John Lenz]

Two things:

I'm very interested in prior art in this area. This isn't a new problem we
are trying to solve.

The original source map deliberately ignore these features to all multiple
levels of translation and at a mid level there might not be a JS AST
representation to hook on to.

[Joey Schorr]

As John alluded to, the original idea (way back in the old days of V1) was
to do the more-detailed mapping out of band, given the variety and
complexity of the possible translations. For example, having either a
server API that understood how to do additional mapping of error messages
after the fact or having the running debugger provide real-time mapping via
its own API.

I'm new to the recent discussion (so I apologize if this has been suggested
before), but perhaps a solution is to allow the source map to call a
predefined external API that can perform these additional calculations;
that way, there isn't any untrusted code running on the user's machine,
there is no need to ship the full AST/compiler information and it allows
for flexible mappings. If the API were to run on top of a websocket, it
could even be done in real-time. Local debuggers could then translate
locally, while production code (that's being debugged) could provide their
own pre-loaded state via extra endpoint(s).

[Fitzgerald, Nick]

On 3/12/14 5:30 PM, Brian Slesinsky wrote:
> It would be a problem if it somehow ran with higher privileges than
> the code being debugged.

Yeah, definitely shouldn't get any special privileges.

[Fitzgerald, Nick]

On 3/13/14 9:49 PM, John Lenz wrote:
>
> Two things:
>
> I'm very interested in prior art in this area. This isn't a new
> problem we are trying to solve.
>

Totally agreed.

One of the reasons I tried to instill the importance of an extensible
architecture is from what I've learned about DWARF and talking with Jim
Blandy (who was on the DWARF committee). DWARF is built upon Debugging
Information Entries which are basically arbitrary debugging annotations
that can be skipped over by debuggers that are unfamiliar with the
specific annotation. Furthermore, this allows compilers to progressively
add more debugging information and debuggers can start taking advantage
of new annotations as they start appearing.

http://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf

> The original source map deliberately ignore these features to all
> multiple levels of translation and at a mid level there might not be a
> JS AST representation to hook on to.
>

This is a very good point, and is something I overlooked in my proposal.

I'm not married to my proposal and the AST format. I do think that we
*must* solve the requirements I outlined in the blog post for any debug
format of the web that was to be properly standardized.

I also feel very strongly that every compile-to-js language shouldn't
have to write its own browser devtools extension for each browser that
they want to debug.

[Fitzgerald, Nick]

On 3/13/14 10:21 PM, Joey Schorr wrote:
> As John alluded to, the original idea (way back in the old days of V1)
> was to do the more-detailed mapping out of band, given the variety and
> complexity of the possible translations. For example, having either a
> server API that understood how to do additional mapping of error
> messages after the fact or having the running debugger provide
> real-time mapping via its own API.
>
> I'm new to the recent discussion (so I apologize if this has been
> suggested before), but perhaps a solution is to allow the source map
> to call a predefined external API that can perform these additional
> calculations; that way, there isn't any untrusted code running on the
> user's machine, there is no need to ship the full AST/compiler
> information and it allows for flexible mappings. If the API were to
> run on top of a websocket, it could even be done in real-time. Local
> debuggers could then translate locally, while production code (that's
> being debugged) could provide their own pre-loaded state via extra
> endpoint(s).

To me, this seems to have many more moving parts for the user who is
debugging source that has been compiled to JS. They have to set up this
external API.

I'd much prefer that compilers were the ones that dealt with this
complexity (they would have to supply the external API, anyways; might
as well make it easy for the end user).

[Brian Slesinsky]

On Fri, Mar 14, 2014 at 1:33 PM, Fitzgerald, Nick
<nfitz...@mozilla.com>wrote:

>

> One of the reasons I tried to instill the importance of an extensible
> architecture is from what I've learned about DWARF and talking with Jim
> Blandy (who was on the DWARF committee). DWARF is built upon Debugging
> Information Entries which are basically arbitrary debugging annotations
> that can be skipped over by debuggers that are unfamiliar with the
> specific annotation. Furthermore, this allows compilers to progressively
> add more debugging information and debuggers can start taking advantage
> of new annotations as they start appearing.
>

We already have extensibility at the top level - you can always add more
fields to the JSON map. I guess the question is whether we can hang
annotations somewhere else such as on the character ranges.

I suppose you could abuse the "names" index (field 5) to create a list
parallel to "names" that has whatever JSON type you want. I believe the
"names" index currently means "what function name should be returned here
in a stack trace?" So it's only good for a parallel annotation that applies
to an entire function in the source language. To annotate smaller AST nodes
we'd need additional indexes.

- Brian