Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

I would appreciate some feedback on the Source Map developer tool (planning stage)

27 views
Skip to first unread message

Nick Fitzgerald

unread,
Jun 30, 2011, 5:57:55 PM6/30/11
to
Hello everyone!

I would like to open up discussion and ask for some feedback on the
Source Map feature. Basically, the goal is to provide a way to map line
and column information from errors or logged messages in JS back to the
un-minified, original source code, or the pre-"transpiled" source (for
example, some CoffeeScript file).

You can learn more here:
https://wiki.mozilla.org/DevTools/Features/SourceMap

I thought that the people on this list would probably have an opinion on
the following open-ended issues (these are also in the above link,
possibly more readable):

* How is the browser notified that a source map exists for a given script?
** Should it be a comment at the top of the source? Something like
//@sourceMap=http://example.com/path/to/source/map
*** This could work with "eval" and script tag injection.
** Should it be an attribute on the script's tag? Something like <script
src='/path/to/foo.js' data-source-map='path/to/foo.smap'>
*** This would not work with eval, but would work with script tag injection.
** Should it just be implicit that the source map file is a replacement
of the suffix of the script's file? For example, if a script's URL is
"foo.js", the source map is implied to be at "foo.smap"?
*** This would not work with eval, but would work with script tag injection.
*** This would result in unnecessary HTTP requests when the source map
does not exist.
** What is th URL pointing to the source map relative to?
*** The script itself, or the page loading the script?

* What requirements must be satisfied for the browser to fetch a source map?
** Should the JavaScript engine be in debug-mode?
** Should a flag or preference be set?
** Should the console/firebug/debugger/etc have to flip a switch
explicitly when they are started and flip it back off again when they exit?

* When should the browser fetch the source map?
** As soon as it knows of the existence of the source map?
** By the time a message is logged, or an exception is thrown, it is
already too late to fetch the source map because we do not want to block
while we go fetch the source map.

* When (if at all) should the browser fetch the original source file[s]
referenced from a source map?
** As soon as it has retrieved the source map?

* From an API standpoint, how should the browser tell the JS engine that
a source map has been fetched and is available?

* Where are the original source files referenced from the source map?
** Presumably, they are URLs; what are they relative to?
*** The origin of the script?
*** The origin of the page?

* Where can source maps be located? Do we want to let script's say they
come from any arbitrary place?
** Should they be restricted to the same origin as their script?
** Should they be restricted to the same origin as the webpage?
** From either of the above?

Thanks for all your help,

Nick

John J Barton

unread,
Jun 30, 2011, 6:34:10 PM6/30/11
to
Nick Fitzgerald wrote:
> Hello everyone!
>
> I would like to open up discussion and ask for some feedback on the
> Source Map feature. Basically, the goal is to provide a way to map line
> and column information from errors or logged messages in JS back to the
> un-minified, original source code, or the pre-"transpiled" source (for
> example, some CoffeeScript file).
>
> You can learn more here:
> https://wiki.mozilla.org/DevTools/Features/SourceMap
>
> I thought that the people on this list would probably have an opinion on
> the following open-ended issues (these are also in the above link,
> possibly more readable):


>
> * How is the browser notified that a source map exists for a given script?
> ** Should it be a comment at the top of the source? Something like
> //@sourceMap=http://example.com/path/to/source/map

You'll want to put comments at the bottom so dynamic addition of the
comment does not alter the line numbering.

> *** This could work with "eval" and script tag injection.
> ** Should it be an attribute on the script's tag? Something like <script
> src='/path/to/foo.js' data-source-map='path/to/foo.smap'>
> *** This would not work with eval, but would work with script tag
> injection.
> ** Should it just be implicit that the source map file is a replacement
> of the suffix of the script's file? For example, if a script's URL is
> "foo.js", the source map is implied to be at "foo.smap"?
> *** This would not work with eval, but would work with script tag
> injection.
> *** This would result in unnecessary HTTP requests when the source map
> does not exist.
> ** What is th URL pointing to the source map relative to?
> *** The script itself, or the page loading the script?
>
> * What requirements must be satisfied for the browser to fetch a source
> map?

The browser should not fetch the source map. It has no use for it. It
should be fetched by the tool that needs the map.

> ** Should the JavaScript engine be in debug-mode?
> ** Should a flag or preference be set?
> ** Should the console/firebug/debugger/etc have to flip a switch
> explicitly when they are started and flip it back off again when they exit?
>
> * When should the browser fetch the source map?

As above, not at all.

> ** As soon as it knows of the existence of the source map?
> ** By the time a message is logged, or an exception is thrown, it is
> already too late to fetch the source map because we do not want to block
> while we go fetch the source map.

Why not? Just in time is the right time.

>
> * When (if at all) should the browser fetch the original source file[s]
> referenced from a source map?
> ** As soon as it has retrieved the source map?

Only when the user needs it. The user will be looking at 40 line of
code, you don't want to load the whole map for 70kloc.

>
> * From an API standpoint, how should the browser tell the JS engine that
> a source map has been fetched and is available?

Why? The JS engine has no use for it.

>
> * Where are the original source files referenced from the source map?
> ** Presumably, they are URLs; what are they relative to?
> *** The origin of the script?
> *** The origin of the page?
>
> * Where can source maps be located? Do we want to let script's say they
> come from any arbitrary place?
> ** Should they be restricted to the same origin as their script?
> ** Should they be restricted to the same origin as the webpage?
> ** From either of the above?

The important missing question is the format of the file. The Closure
compiler format exists and has demonstrated successfully by integration
in Firebug, so I'd say it should be the starting point.

jjb

Gervase Markham

unread,
Jul 1, 2011, 5:41:48 AM7/1/11
to
On 30/06/11 22:57, Nick Fitzgerald wrote:
> ** Should it just be implicit that the source map file is a replacement
> of the suffix of the script's file? For example, if a script's URL is
> "foo.js", the source map is implied to be at "foo.smap"?
> *** This would not work with eval, but would work with script tag
> injection.
> *** This would result in unnecessary HTTP requests when the source map
> does not exist.

No comment on the rest, but a plea: please don't do this :-)

Gerv

Nick Fitzgerald

unread,
Jul 1, 2011, 12:45:00 PM7/1/11
to
>> * How is the browser notified that a source map exists for a given
>> script?
>> ** Should it be a comment at the top of the source? Something like
>> //@sourceMap=http://example.com/path/to/source/map
>
> You'll want to put comments at the bottom so dynamic addition of the
> comment does not alter the line numbering.

Noted in the wiki, thank you.

>> * What requirements must be satisfied for the browser to fetch a
>> source map?
>
> The browser should not fetch the source map. It has no use for it. It
> should be fetched by the tool that needs the map.
>
>> ** Should the JavaScript engine be in debug-mode?
>> ** Should a flag or preference be set?
>> ** Should the console/firebug/debugger/etc have to flip a switch
>> explicitly when they are started and flip it back off again when they
>> exit?
>>
>> * When should the browser fetch the source map?
>
> As above, not at all.

Ah yes, such as the debugger. Then if the debugger (or whatever tool) is
open, it can fetch the source map itself, and otherwise we don't need to
worry about it. Thanks, this side-steps the issues above elegantly.

>> ** As soon as it knows of the existence of the source map?
>> ** By the time a message is logged, or an exception is thrown, it is
>> already too late to fetch the source map because we do not want to
>> block while we go fetch the source map.
>
> Why not? Just in time is the right time.

The issue is that we don't want to delay displaying a user a message
while we go round trip over the network, nor do we want to delay
execution of js until we have the source map.

One work-around is to just immediately display the message (in terms of
the minified/transpiled js) if there is no source map yet, and only
displaying the message with the lines/cols mapped back to the original
source once the map has been loaded and parsed. This hints to me that
there is an advantage to loading the source map early, so that it is
more likely to be ready and parsed by the time the first messages are
logged or errors thrown. Thoughts?

>> * When (if at all) should the browser fetch the original source
>> file[s] referenced from a source map?
>> ** As soon as it has retrieved the source map?
>
> Only when the user needs it. The user will be looking at 40 line of
> code, you don't want to load the whole map for 70kloc.

This makes sense; noted on the wiki.

> The important missing question is the format of the file. The Closure
> compiler format exists and has demonstrated successfully by integration
> in Firebug, so I'd say it should be the starting point.

Yes, this is an important question. I have been looking in to the
Closure compiler's format and reading their source code for consuming
and generating the files, but I don't yet feel I have a complete grasp
of the ins and outs. I didn't want to let that get in the way of gaining
valuable feedback in these other issues I have run in to so far. If you
have any insights regarding potential formats, I'd love to hear those, too.

Thanks for the feedback (it is very appreciated), and for bearing with
me while I develop a better mental picture of the problem domain.

Nick

Mike Shaver

unread,
Jul 1, 2011, 12:49:36 PM7/1/11
to Nick Fitzgerald, dev-pl...@lists.mozilla.org
On Fri, Jul 1, 2011 at 12:45 PM, Nick Fitzgerald
<nfitz...@mozilla.com> wrote:
> The issue is that we don't want to delay displaying a user a message while
> we go round trip over the network, nor do we want to delay execution of js
> until we have the source map.

Debuggers that use symbol servers or even lazy loading of coupled
symbols usually delay. As long as the UI is responsive and the user
can continue to work if they don't care about the details of the frame
in question, I think that's probably the right approach.

Mike

johnjbarton

unread,
Jul 1, 2011, 1:34:31 PM7/1/11
to
On 7/1/2011 9:45 AM, Nick Fitzgerald wrote:
> The issue is that we don't want to delay displaying a user a message
> while we go round trip over the network, nor do we want to delay
> execution of js until we have the source map.
>
> One work-around is to just immediately display the message (in terms of
> the minified/transpiled js) if there is no source map yet, and only
> displaying the message with the lines/cols mapped back to the original
> source once the map has been loaded and parsed. This hints to me that
> there is an advantage to loading the source map early, so that it is
> more likely to be ready and parsed by the time the first messages are
> logged or errors thrown. Thoughts?

Round trip times can be reduced lots of ways orthogonal to your problem.
Thus you can focus on the scaling issues that you face. You should be
thinking about apps with hundreds or thousands of files, 100kloc, and
2-10 transcodings (JS.es3 <- JS.es7 <- CoffeScript <- compressed
CoffeeScript). So preloading needs to be like cache optimization not
like page loading.

Think about the timing in reverse: the dev needs the info when they are
going to take action, not when the error message is logged or thrown.
Typical Web pages log hundreds of messages; typically 100% of them are
ignored. Don't optimize for the case were messages are few and important.
...


>> The important missing question is the format of the file. The Closure
>> compiler format exists and has demonstrated successfully by integration
>> in Firebug, so I'd say it should be the starting point.
>
> Yes, this is an important question. I have been looking in to the
> Closure compiler's format and reading their source code for consuming
> and generating the files, but I don't yet feel I have a complete grasp
> of the ins and outs.

Yeah, I couldn't make head or tails of the format from that source
either. I bet an example file would make things a lot clearer.

>I didn't want to let that get in the way of gaining
> valuable feedback in these other issues I have run in to so far. If you
> have any insights regarding potential formats, I'd love to hear those, too.

I think a productive model might be SourceRegions <-> SourceRegions. By
SourceRegions I mean array of (start, end) coordinates, coordinates as
(line, col). This constrains the two sides to be text, a limitation that
adds a lot of simplicity. (line,col) allows fall back to lines for older
tool integration and allows many edits to be local, an advantage for
dynamic tools. Arrays of regions allows for optimizers and refactoring.
A common API and a set of BSD licensed libraries for JS, Java and C++
would be awesome.

jjb

Mike Shaver

unread,
Jul 1, 2011, 2:34:28 PM7/1/11
to johnjbarton, dev-pl...@lists.mozilla.org
On Fri, Jul 1, 2011 at 1:34 PM, johnjbarton <johnj...@johnjbarton.com> wrote:
> Round trip times can be reduced lots of ways orthogonal to your problem.
> Thus you can focus on the scaling issues that you face. You should be
> thinking about apps with hundreds or thousands of files, 100kloc, and 2-10
> transcodings (JS.es3 <- JS.es7 <- CoffeScript <- compressed CoffeeScript).
>  So preloading needs to be like cache optimization not like page loading.
>
> Think about the timing in reverse: the dev needs the info when they are
> going to take action, not when the error message is logged or thrown.
> Typical Web pages log hundreds of messages; typically 100% of them are
> ignored. Don't optimize for the case were messages are few and important.

Thanks for raising these, John, they're critical points.

A frame that's from eval could well be displayed as "eval at file:no
[+]" and then we load/process the sourcemap when the user wants the
extra detail. We could let people autoload as we see frames too, but
I wouldn't make that the default.

Mike

Nick Fitzgerald

unread,
Jul 6, 2011, 1:07:28 PM7/6/11
to
On 7/1/11 10:34 AM, johnjbarton wrote:
> Round trip times can be reduced lots of ways orthogonal to your problem.
> Thus you can focus on the scaling issues that you face. You should be
> thinking about apps with hundreds or thousands of files, 100kloc, and
> 2-10 transcodings (JS.es3 <- JS.es7 <- CoffeScript <- compressed
> CoffeeScript). So preloading needs to be like cache optimization not
> like page loading.
>
> Think about the timing in reverse: the dev needs the info when they are
> going to take action, not when the error message is logged or thrown.
> Typical Web pages log hundreds of messages; typically 100% of them are
> ignored. Don't optimize for the case were messages are few and important.
> ...

Ok, you have convinced me. Will note this in the wiki page.

Regarding multiple translations, there are two options:

1. Have the user (or tools the user is using to generate the source
maps) compose the source maps in to one source map before-hand.

2. Traverse the mapping between all the source maps repeatedly "by hand"
in the debugger until the last one is found.

It seems to me that the first one makes it easier on the debugger but
harder on the user, the second one is the opposite. If we did go the
second route, the file format would probably want to have some way of
telling the debugger about other source maps down the chain so that they
could all be downloaded in parallel and we wouldn't have to fetch them
one by one as we discover them.

I've made note of this in the wiki page as well, and am interested in
your thoughts on this problem.

> Yeah, I couldn't make head or tails of the format from that source
> either. I bet an example file would make things a lot clearer.

I am currently in the process of giving it a deep read through. After
they parse the file format, it gives them enough to run queries which
look like this:

(sourceLine X sourceColumn) -> (originalFile X originalLine X
originalColumn X originalIdentifier)

That looks pretty solid to me; you definitely need to know which file a
particular line/col came from. The one thing I am iffy on is the
identifier. They went to great pains to make sure that their file format
was small, and yet they are including these non-minified identifiers in
the source map. It doesn't even seem like an identifier is that helpful
because it doesn't give you enough to really debug things and you will
need to look at the whole source to understand that part of the code and
what exception has occurred. There has to be some reason.

The other thing I need to understand better is the difference between
the "lineMaps" attribute and the "mappings" attribute of their JSON
source map file format. Need to read the code again.

> I think a productive model might be SourceRegions <-> SourceRegions. By
> SourceRegions I mean array of (start, end) coordinates, coordinates as
> (line, col). This constrains the two sides to be text, a limitation that
> adds a lot of simplicity. (line,col) allows fall back to lines for older
> tool integration and allows many edits to be local, an advantage for
> dynamic tools. Arrays of regions allows for optimizers and refactoring.

Yeah, the one thing I would say that needs to be included that you
haven't mentioned is which original file the given region comes from.
Also, you can just have the mappings as an array of the starts of the
regions and that way you can rebuild the regions implicitly by comparing
one mapping to its neighbors in the array. This keeps the file format
small (and is what Closure does).

> A common API and a set of BSD licensed libraries for JS, Java and C++
> would be awesome.

The plan right now is to create and integrate reference source map
generators for UglifyJS and CoffeeScript, as well as integrate a
reference source map consumer in to the console and debugger. I want
these to be as general as possible (hopefully with the integration and
core library separated), and I also want to be able to point to working
examples by the end of my internship at Mozilla.

Interested in hearing your replies,

Nick

johnjbarton

unread,
Jul 6, 2011, 2:46:17 PM7/6/11
to

You only need to demonstrate that the API implied by your source map
supports composition. You're off to a great start by knowing to include
a link to the next map file in the chain is part of that demonstration.
The composition itself can be a library function used by either end.

Yes.

> Also, you can just have the mappings as an array of the starts of the
> regions and that way you can rebuild the regions implicitly by comparing
> one mapping to its neighbors in the array. This keeps the file format
> small (and is what Closure does).

This does imply that the language has a way to unambiguously segment the
source into regions. Probably ok. Perhaps a language expert can help
with this question. I guess the difference between start/end and just
start will be the whitespace (or other non-language characters?)
following a token that starts a region. (I believe the source maps
should support all of the browser's source languages and having a common
file format would be great, though multiple formats with a common API
would be enough).

I wonder about the last source region in embedded code:
<script>
var a = 1;
</script>
or <a href="javascript: alert('hi');">
or <div onclick="alert('bye');">
Probably there is an independent way of knowing the eof so its not an issue.

>
>> A common API and a set of BSD licensed libraries for JS, Java and C++
>> would be awesome.
>
> The plan right now is to create and integrate reference source map
> generators for UglifyJS and CoffeeScript, as well as integrate a
> reference source map consumer in to the console and debugger. I want
> these to be as general as possible (hopefully with the integration and
> core library separated), and I also want to be able to point to working
> examples by the end of my internship at Mozilla.

Sounds great, please let me know what bug you will post your progress to.
See also
https://bugs.webkit.org/show_bug.cgi?id=63940

jjb

Nick Fitzgerald

unread,
Jul 6, 2011, 11:41:45 PM7/6/11
to
>> Also, you can just have the mappings as an array of the starts of the
>> regions and that way you can rebuild the regions implicitly by comparing
>> one mapping to its neighbors in the array. This keeps the file format
>> small (and is what Closure does).
>
> This does imply that the language has a way to unambiguously segment the
> source into regions. Probably ok. Perhaps a language expert can help
> with this question. I guess the difference between start/end and just
> start will be the whitespace (or other non-language characters?)
> following a token that starts a region. (I believe the source maps
> should support all of the browser's source languages and having a common
> file format would be great, though multiple formats with a common API
> would be enough).

Well technically, it would map a region of the generated source to a
specific line/col in the original source. I think it is safe to say that
if you have a set of entries like this:

[ {line: 0, col: 0},
{line: 0, col: 10 },
{line: 1, col: 0 },
{line: 1, col: 13 },
...
]

and you get an error which points to line 1 column 5, that you should
use the {line: 1, col: 0} entry. This is what Closure compiler does, and
I think it is reasonable. Imagine that the entry {line: 1, col: 0}
represents some identifier or function name which is 12 characters long;
if the error points to the middle of the identifier, and you have an
entry pointing to the start of the identifier, it makes sense to use it.

> I wonder about the last source region in embedded code:
> <script>
> var a = 1;
> </script>
> or <a href="javascript: alert('hi');">
> or <div onclick="alert('bye');">
> Probably there is an independent way of knowing the eof so its not an
> issue.

Tricky case; not sure what to think. I am kind of doubtful that almost
anyone uses DOM level 0 stuff with minified/tranlated code. I have been
surprised before, though.

>> The plan right now is to create and integrate reference source map
>> generators for UglifyJS and CoffeeScript, as well as integrate a
>> reference source map consumer in to the console and debugger. I want
>> these to be as general as possible (hopefully with the integration and
>> core library separated), and I also want to be able to point to working
>> examples by the end of my internship at Mozilla.
>
> Sounds great, please let me know what bug you will post your progress to.
> See also
> https://bugs.webkit.org/show_bug.cgi?id=63940

Right now, I am tracking all of the spec/requirements/planning stuff on
the wiki page: https://wiki.mozilla.org/DevTools/Features/SourceMap

I have that webkit bug linked in under the "Other Stuff" section.

I'm planning on making a first draft of a proposed file format by
Monday, if everything goes smoothly. It will likely be heavily
influenced by Closure's format. I will make a bug to house discussion
once I have this ready, and as I finish the spec/planning work and move
in to actual coding and features I will definitely create some bugs and
post them back here and on the wiki.

I spent all of today and last night reading Closure's source map code
trying to understand everything going on in versions 2 and 3 of the
format. I think I pretty much get it all now, and I have to say I'm very
impressed. Parts seem convoluted -- like how the mappings are
represented by variable length base64 values which are relative to the
previous value -- but in the end it seems it was all for good reason and
keeping the files as small as possible. Just waiting on a reply from the
Closure compiler team to a few last questions I had, then I should be
able to bust out a format proposal.

Take it easy,

Nick

johnjbarton

unread,
Jul 7, 2011, 12:05:02 AM7/7/11
to

I was only worried about the case where the identifier is eg 2
characters long. Then line 1 col 5 is ambiguous. As long as we can be
comfortable that any hit between start and (next - 1) belongs with start
its ok.

>
>> I wonder about the last source region in embedded code:
>> <script>
>> var a = 1;
>> </script>
>> or <a href="javascript: alert('hi');">
>> or <div onclick="alert('bye');">
>> Probably there is an independent way of knowing the eof so its not an
>> issue.
>
> Tricky case; not sure what to think. I am kind of doubtful that almost
> anyone uses DOM level 0 stuff with minified/tranlated code. I have been
> surprised before, though.

I hope you will be surprised again. In addition to
<a href="coffescript: alert('hi');">
we also want to use source maps broadly, for things like "which JS code
or HTML file inserted that href attribute?" minify/translate are only
one kind of dynamic mapping between developer source and an entry in the
browser data structure. We need to unravel all of them.

jjb

Nick Fitzgerald

unread,
Jul 7, 2011, 1:33:13 PM7/7/11
to
So the Google guys have opened up their documentation for their source
maps. Currently, V3 is still in the works, and they are proposing we
collaborate with them and use this as the starting point for open
standard which all browsers/debuggers/etc can support.

I have edit permissions, and it should be publicly view-able. If anyone
has comments or concerns, I can make sure they are raised.

http://code.google.com/p/closure-compiler/wiki/SourceMaps

I think this is great news!

Nick

Mike Shaver

unread,
Jul 7, 2011, 1:38:21 PM7/7/11
to Nick Fitzgerald, dev-pl...@lists.mozilla.org
On Thu, Jul 7, 2011 at 1:33 PM, Nick Fitzgerald <nfitz...@mozilla.com> wrote:
> I think this is great news!

You are correct!

Mike

0 new messages