Summary of Web Rebels 2012 party discussions

[August Lilleaas]

Chris and myself had a mini Buster.JS meetup with Stein Magnus Jodal (jodal) and Jesper Karsrud (jkarsrud) at Web Rebels today! I thought I'd make a short writeup of what we discussed there.

1) A resource in a resource set should be able to store a "compiled" version. The resource itself may be for a .coffee file. An extension can process that file and store the compiled JS as a separate part of that resource (perhaps .compiledResource?), and also store the source map along with it so that we in the future are able to implement stack traces that work great for coffeescript and other langs like Dart and ClojureScript.

2) Autotest should not manually grep for (test|spec)/**/*.js. Instead it should use the resource set from the config file as the base for where and what to watch. This means automatic autotest support for .coffee files and any other format that is added in the future, as long as a resource is text/javascript or has a compiledResource, it will be autotested.

3) Buster.JS should be released soon ffs.

It's getting ridiculously late, signing off!

[Stein Magnus Jodal]

On Fri, May 25, 2012 at 1:57 AM, August Lilleaas <aug...@augustl.com> wrote:
> Chris and myself had a mini Buster.JS meetup with Stein Magnus Jodal (jodal)
> and Jesper Karsrud (jkarsrud) at Web Rebels today! I thought I'd make a
> short writeup of what we discussed there.
>
> 1) A resource in a resource set should be able to store a "compiled"
> version. The resource itself may be for a .coffee file. An extension can
> process that file and store the compiled JS as a separate part of that
> resource (perhaps .compiledResource?), and also store the source map along
> with it so that we in the future are able to implement stack traces that
> work great for coffeescript and other langs like Dart and ClojureScript.

Or, instead of having a "compiled resource", a resource can have
multiple "representations", each with a specified MIME type. Then
things could work approximately like this:

1) When the .coffee file is read from disk, the resource is created
with the .coffee path and the CoffeeScript content with a CoffeeScript
MIME type (text/x-coffeescript, or text/coffeescript if one ignores
that that MIME type isn't registed with IANA).

2) When buster-coffee (or the future buster-resource-compiler) is run,
it loops over all the resources, and for each resource with a
CoffeeScript MIME type, adds a new representation with a JavaScript
MIME type and a content function that will compile the CoffeeScript
content.

3) When e.g. buster-html-docs is run, it loops over all resources, and
extracts docs for each resource with JavaScript contents available,
which includes the .coffee resources, which have been previously
"enriched" with a JavaScript version by buster-coffee.

At the moment I can't come up with an example where more than two
representations will be needed, but if we allow multiple MIME type
identified representations, we have a bit more flexibility. Who knows
what representations future extensions will find useful to piggy-back
on resources.

--
Stein Magnus Jodal

[August Lilleaas]

That makes sense. Instead of having just one "content" and just one mime type, one resource could basically have an array of resources with content + mime type, or an object where the key is the mime type and the value is the content. That feels more like a proper implementation, and less tacked onto what we already have.

[Christian Johansen]

Hey,

Or, instead of having a "compiled resource", a resource can have
multiple "representations", each with a specified MIME type. Then
things could work approximately like this:

1) When the .coffee file is read from disk, the resource is created
with the .coffee path and the CoffeeScript content with a CoffeeScript
MIME type (text/x-coffeescript, or text/coffeescript if one ignores
that that MIME type isn't registed with IANA).

2) When buster-coffee (or the future buster-resource-compiler) is run,
it loops over all the resources, and for each resource with a
CoffeeScript MIME type, adds a new representation with a JavaScript
MIME type and a content function that will compile the CoffeeScript
content.

3) When e.g. buster-html-docs is run, it loops over all resources, and
extracts docs for each resource with JavaScript contents available,
which includes the .coffee resources, which have been previously
"enriched" with a JavaScript version by buster-coffee.

At the moment I can't come up with an example where more than two
representations will be needed, but if we allow multiple MIME type
identified representations, we have a bit more flexibility. Who knows
what representations future extensions will find useful to piggy-back
on resources.

I'm just about to start implementing this. One question: For this to work
a resource should be able to tell me if it has content for a given MIME
type right? E.g.:

if (resource.respondsTo("text/javascript")) {
    resource.content("text/javascript", function (content) { ... });
} else if (resource.respondsTo("text/css")) {
    // ...
}

Or should a resource have a "preferred" content type? E.g.:

if (resource.contentType() == "text/javascript") {
    resource.content("text/javascript", function (content) { ... });
}

In the first approach, we have a generally more flexible system, but where
whoever is using the resource is responsible for knowing what the
resource should be used for. The second example moves the control
into the resource, but is somewhat less flexible.

--
MVH
Christian

[August Lilleaas]

I can't think of any reason for why you would want to have more than two "content types" for each resource. For now, the only use case is to store "something" that isn't JS, a source map, and the resulting JS from compiling the "something".

Perhaps we should just explicitly support this, instead of having a generic content type system?

Specific support for non-JS + source map + JS (alternatively, only JS) would probably make it easier to serve the JS with X-Sourcemap, etc.

[Christian Johansen]

I can't think of any reason for why you would want to have more than two "content types" for each resource. For now, the only use case is to store "something" that isn't JS, a source map, and the resulting JS from compiling the "something".

From a web server viewpoint, the sourcemap is just a .json file/resource. So a
resource could respond to the following MIME-types: text/x-coffeescript
(source), text/javascript (build target), application/json (sourcemap). I've not
read the entire sourcemap specification, but as far as I gather at this point we
don't need to do anything special with them.
 

Perhaps we should just explicitly support this, instead of having a generic content type system?

Specific support for non-JS + source map + JS (alternatively, only JS) would probably make it easier to serve the JS with X-Sourcemap, etc.

Well, in general this will not work for a few reasons:

• A resource != A JavaScript file
• We support resources that have no, or even varying,
  content-types (i.e. proxies)
  
• As Stein Magnus said, a generic resource system allows for
  creative extension authors to piggy-back the caching system in
  various ways
  
• Other resource types may have multiple representations too:
  Images (e.g. png/base64 data uris), CSS (e.g. less/CSS)
  

I'm hard-pressed to come up with examples of more than two MIME
types that are not input/output/sourcemap, but I don't see what
we gain from making this restriction (it's likely to make the
implementation more specific/complicated), while the "n MIME
types" approach solves all the problems we've identified so far.
Please correct me if I'm mistaken.

Christian

[August Lilleaas]

I've not read the entire sourcemap specification, but as far as I gather at this point we don't need to do anything special with them.

We need to set X-Sourcemap on the .js files if their source is compiled code, i.e. they have a source map. Setting this header causes the browsers to automatically fetch the source map etc.

I'm not sure I understand your critisism, perhaps you misunderstood me. In essence, what I'm saying is that since we today support different types of resources - proxies and raw content - it makes sense to add support for a 3rd type of resource - a compiled script that has a source map. I don't see how we can add support for source maps with a generic content-type based setup alone.

[Christian Johansen]

I've not read the entire sourcemap specification, but as far as I gather at this point we don't need to do anything special with them.

We need to set X-Sourcemap on the .js files if their source is compiled code, i.e. they have a source map. Setting this header causes the browsers to automatically fetch the source map etc.

I'm not sure I understand your critisism, perhaps you misunderstood me. In essence, what I'm saying is that since we today support different types of resources - proxies and raw content - it makes sense to add support for a 3rd type of resource - a compiled script that has a source map. I don't see how we can add support for source maps with a generic content-type based setup alone.

I did misunderstand, so disregard the unrelated parts of my previous email.

Let's take the buster-coffee as an example on how we may support
in/out/sourcemap using MIME types:

We start with a resource:

    src/my-module.coffee

It is added with MIME-type "text/x-coffee", the path and the file's contents
(lazy loaded). The buster-coffee extension adds another representation for
"text/javascript". The content of this representation is a function that will
compile the "text/x-coffee" content to JS. Additionally, it will add the
X-Sourcemap header and set its value to
"src/my-module.coffee?mime=application/json".

There are a few unanswered questions here:

• What if the multiple representations (mainly the sourcemap, JSON, and the JS)
  require different headers?
• The resource does not know the URL it is served over, so the resource middleware
  somehow needs to be able to "fix" the X-Sourcemap header so it includes the full
  URL.

If a resource has multiple representations and those have their own headers,
then we're approaching a situation where a resource may be a collection of
resources, which may be the solution we're looking for?

Christian

[August Lilleaas]

The path in X-Sourcemap is a bitch.. Perhaps we should  have a special "path" type:

    {headers: {"X-Sourcemap": new ramp.resources.Path(sourcemapPath)}}

[Christian Johansen]

The path in X-Sourcemap is a bitch.. Perhaps we should  have a special "path" type:

    {headers: {"X-Sourcemap": new ramp.resources.Path(sourcemapPath)}}

That looks good. So, I'm thinking:

var resource = resourceSet.addFileResource("my/module.coffee");

resource.addAlternative({
    contentType: "text/javascript",
    content: function () {
        // Compile
    }
});

resource.addAlternative({
    contentType: "text/json",
    content: function () {
        // Generate Sourcemap
    }
});

resource.setHeader("X-Sourcemap", rampResources.path(resource.pathFor("text/json")));

[August Lilleaas]

Compilation and source map generation is generally the same process so it would be nice if the two could somehow go together. With two separate callbacks you would essentially need to compile twice.

[Christian Johansen]

On Wed, Aug 22, 2012 at 9:12 PM, August Lilleaas <aug...@augustl.com> wrote:

Compilation and source map generation is generally the same process so it would be nice if the two could somehow go together. With two separate callbacks you would essentially need to compile twice.

The suggested API is usable enough to allow you to do that if you wish:

var resource = resourceSet.addFileResource("my/module.coffee");

(function () {
    var compiled, sourceMap;

    function compile() {
        var result = CoffeeScript.compileWithSourceMap(this.content("text/coffeescript"));
        compiled = result.javascript();
        sourceMap = result.sourceMap();

    }

    resource.addAlternative({
        contentType: "text/javascript",
        content: function () {

            if (!compiled) { compile(); }
            return compiled;

        }
    });

    resource.addAlternative({
        contentType: "text/json",
        content: function () {

                if (!sourceMap) { compile(); }
                return sourceMap;

        }
    });

    resource.setHeader("X-Sourcemap", rampResources.path(resource.pathFor("text/json")));

}());

 

[August Lilleaas]

Seems like this covers all the use cases, in addition to being a generally sensible an flexible approach.

[Christian Johansen]

Yay, sweet.