Source Maps, Debuggers, and Generated Code

[Ron Buckton]

Many transpilers today (TypeScript, Traceur, 6to5, etc.) often need to emit generated code, that sometimes does not have a one-to-one mapping with the original source. One of the most notorious examples is the down-level transpile for ES6 generators, which often introduces multiple statements that correspond to a single statement or expression in the original source. TypeScript either elides mapping information for helper functions such as the __extends helper in some cases and in others maps the multiple steps to a single source span, such as how it handles a rest parameter. Traceur adds the generated code to the "sourcesContent" array in the source map.

In addition, various debuggers with source-map support handle generated content differently. In the Chrome Dev Tools and in Visual Studio, any step in the debugger that is not mapped to a source span results in stepping through the generated javascript file until a mapping is encountered, while in the IE F12 tools the debugger makes every effort to keep the source content visible and highlights the last source span for each step in the generated output.

Is there a standard approach to how this content should be mapped, or how stepping should be handled? I'm currently investigating whether it makes sense to propose an extension to the source map specification to classify a range in the generated output as "debugger hidden", so that a debugger will always step through any statements within this range. Alternatively, I'm interested in whether it would make sense for debuggers to treat generated lines with no mappings as "debugger hidden", as well as step through multiple operations if they all map to the same source location.

Best regards,
Ron

[Brian Slesinsky]

I don't think this exists yet. Hiding helper functions and skipping them
while stepping is something GWT and Dart could use as well.

But as far as I know, nobody is working on defining an improved version of
sourcemaps and getting browsers on board.

- Brian


On Mon, Feb 2, 2015 at 11:57 AM, Ron Buckton <Ron.B...@microsoft.com>
wrote:

> _______________________________________________
> dev-js-sourcemap mailing list
> dev-js-s...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-js-sourcemap
>

[Nick Fitzgerald]

Hi Ron,

Replies inline below.

On Mon, Feb 2, 2015 at 11:57 AM, Ron Buckton <Ron.B...@microsoft.com>
wrote:

> Many transpilers today (TypeScript, Traceur, 6to5, etc.) often need to
> emit generated code, that sometimes does not have a one-to-one mapping with
> the original source. One of the most notorious examples is the down-level
> transpile for ES6 generators, which often introduces multiple statements
> that correspond to a single statement or expression in the original source.
> TypeScript either elides mapping information for helper functions such as
> the __extends helper in some cases and in others maps the multiple steps to
> a single source span, such as how it handles a rest parameter. Traceur adds
> the generated code to the "sourcesContent" array in the source map.
>

​I think there are two cases here:

1. When the compiler needs to embed a runtime with various helper functions
in the generated JS code. In this case, I think it makes sense to add an
entry to the sources list and map the appropriate code to this runtime
source.

2. When a single statement in the source language results in many
statements in the JS code. In this case, I believe that all the generated
statements in the JS code should map back to the same location in the
source language. When implementing source level single stepping, debuggers
should continue JS object code level single stepping until the source map
reports a new source location for the current JS code. This is conceptually
the same as when single stepping C-source level statements in gdb/lldb/etc
that map to multiple asm instructions: you don't want to pause after
executing each asm instruction, but once all the asm instructions for that
C statement have been executed.

>
> In addition, various debuggers with source-map support handle generated
> content differently. In the Chrome Dev Tools and in Visual Studio, any step
> in the debugger that is not mapped to a source span results in stepping
> through the generated javascript file until a mapping is encountered, while
> in the IE F12 tools the debugger makes every effort to keep the source
> content visible and highlights the last source span for each step in the
> generated output.
>
>

​For the record, Firefox will continue through null mappings as well.​

> Is there a standard approach to how this content should be mapped, or how
> stepping should be handled? I'm currently investigating whether it makes
> sense to propose an extension to the source map specification to classify a
> range in the generated output as "debugger hidden", so that a debugger will
> always step through any statements within this range. Alternatively, I'm
> interested in whether it would make sense for debuggers to treat generated
> lines with no mappings as "debugger hidden", as well as step through
> multiple operations if they all map to the same source location.
>

​I don't think this needs an extension: I believe continuing through null
mappings is the correct behavior.

[Ron Buckton]

> 1. When the compiler needs to embed a runtime with various helper
> functions in the generated JS code. In this case, I think it makes sense to add
> an entry to the sources list and map the appropriate code to this runtime
> source.

Is there a general best-practice for this? Traceur adds an entry to the 'sources' list that is prefixed with "@traceur/generated", as well as adding an entry to 'sourcesContent' containing the same text written to the generated output file. Adding a string to sourceContent for this purpose seems like unnecessary overhead. TypeScript emits null mappings for each line of generated content.

> 2. When a single statement in the source language results in many
> statements in the JS code. In this case, I believe that all the generated
> statements in the JS code should map back to the same location in the source
> language. When implementing source level single stepping, debuggers
> should continue JS object code level single stepping until the source map
> reports a new source location for the current JS code. This is conceptually the
> same as when single stepping C-source level statements in gdb/lldb/etc that
> map to multiple asm instructions: you don't want to pause after executing
> each asm instruction, but once all the asm instructions for that C statement
> have been executed.

While this makes sense for statements that generate a sequential set of steps, it breaks down slightly when you are generating something like the downlevel emit for a generator or async function, where you might end up with something like this:

```TypeScript
async function asyncFunc(p: Promise<number>): Promise<number> {
console.log("before");
var i = await p;
console.log("after");
return i;
}
```

Which generates (with the current Async Functions prototype, plus additional comments):
```js
var __awaiter = ...; // runtime helper
var __generator = ...; // runtime helper
function asyncFunc(p) {
// (a)
return new Promise(function(_resolve) {
_resolve(__awaiter(__generator(function(_state) {
// (b)
switch (_state) {
// (c)
case 0:
// (d)
console.log("before");
return [4 /*yield*/, p];
// (e)
case 1:
// (f)
i = _state.sent;
console.log("after");
return [2 /*return*/, i + 1];
}
});
});
var i;
}
```

When the debugger steps through 'asyncFunc', it will execute [a], [b], [c], and [d] immediately before yielding, and when the state machine is resumed it will execute [b], [c], [e], and [f]. As a result, [b] and [c] are executed twice (and for a larger function body with multiple 'case' clauses, it would likely step through each case clause up to the current "label"). Based on my understanding of your response below, we should emit "null" mappings for [a]-[c] and for [e]. When the debugger steps into the function for the first time it should step through until it hits [d], and after it resumes it should step through until it hits [f]. Is this an accurate assumption?

> ​I don't think this needs an extension: I believe continuing through null
> mappings is the correct behavior.

I agree that this makes sense. I just want to verify the expected behavior.

Thanks,
Ron

[Brian Slesinsky]

On Mon, Feb 2, 2015 at 1:14 PM, Nick Fitzgerald <nfitz...@mozilla.com>
wrote:

>
> ​I think there are two cases here:
>

> 1. When the compiler needs to embed a runtime with various helper functions
> in the generated JS code. In this case, I think it makes sense to add an
> entry to the sources list and map the appropriate code to this runtime
> source.
>

> 2. When a single statement in the source language results in many
> statements in the JS code. In this case, I believe that all the generated
> statements in the JS code should map back to the same location in the
> source language.

One problem is that JavaScript code may be reused (for example, it could be
a generic helper function called from many places). But a sourcemap can
only map it to one source location. You could get around this by generating
the code N times for N locations, but this creates code bloat similar to
inlining.

- Brian

[Nick Fitzgerald]

​It sounds​ to me like this is case (1) then, rather than case (2). If you
want the debugger to stop at this generic helper function, create a
"source" for your runtime. If you want it to be stepped through, use null
mappings.

I guess you could even have a debug mode that generates the code bloat, if
you want, but I'd think either embedding a source for the runtime in the
sourcemap or using null mappings should handle whatever behavior you want.

[Nick Fitzgerald]

On Mon, Feb 2, 2015 at 2:06 PM, Ron Buckton <Ron.B...@microsoft.com>
wrote:

> > 1. When the compiler needs to embed a runtime with various helper
> > functions in the generated JS code. In this case, I think it makes sense
> to add
> > an entry to the sources list and map the appropriate code to this runtime
> > source.
>

> Is there a general best-practice for this? Traceur adds an entry to the
> 'sources' list that is prefixed with "@traceur/generated", as well as
> adding an entry to 'sourcesContent' containing the same text written to the
> generated output file. Adding a string to sourceContent for this purpose
> seems like unnecessary overhead. TypeScript emits null mappings for each
> line of generated content.
>

​It depends on whether you want to step through the runtime source in
debuggers. If you do, then embed the runtime as a source in the sourcemap
like Traceur does. If you don't want to step through your runtime's source,
then use null mappings and debuggers should step through it.​

If you're concerned about sending the runtime source over the wire twice
(once in the generated JS code and once embedded in the source map) then
you can load the runtime as a separate script in debug builds. I don't have
any other great answers.

>
> > 2. When a single statement in the source language results in many
> > statements in the JS code. In this case, I believe that all the generated
> > statements in the JS code should map back to the same location in the
> source

​This sounds correct to me.​