Standardise CDNs as library sources

[The Configurator]

Imagine this, if you will:

A JavaScript + CSS library is published. It is given a uri, for example lib://mylib.com/libname. Thereafter, you can load the library with the following tag:

<library uri="lib://mylib.com/libname" version="1.2.3">

  <script src="/libraries/libname.1.2.3.js"></script>

  <link rel="stylesheet" src="/libraries/libname.1.2.3.css" />

</library>

What this does is the following:

• Older browsers will ignore the library tag, and load the script and stylesheet as usual.
  
  
• Newer browsers will recognise it as a published library, and will check availability with their installed CDNs.

1. Make a DNS request to mylib.com. Check a TXT record (or a new kind of record) that will contain "libname=ABCD". This is mandatory and gives a public key.
2. For each library CDN it knows about, it will make a request to download a library from lib://mylib.com/libname. The library can have JavaScript and stylesheets, somehow compressed into one file. If the CDN doesn't have that library, the browser will try a different CDN. If it does, the file must be signed, with the public key given in the DNS record.
3. If the DNS record is missing, or no CDN has the library, the browser will revert to downloading the files locally.

This allows advanced caching - since all CDNs will use the same library URI for the same library, libraries can be cached per uri per version, possibly precompiled by the browser. Popular libraries like jQuery will only be downloaded once - and we developers don't have to rely on Google or Microsoft's CDN because the browser will use it automatically and revert to our own files on failure.

A couple of usage examples:

<library uri="lib://jquery.com/jquery" version="1.6.4"><script src="/scripts/jquery.1.6.4.min.js"></script></library>

<library uri="lib://jquery.com/jquery-ui" version="1.8.16"><script src="/scripts/jquery-ui.1.8.16.min.js"></script><link rel="stylesheet" src="/css/jquery-ui.1.8.16.css" /></library>

What are your thoughts on this? Is this feasible? Are there any problems I may have missed?

I think this has the potential to make the web a slightly better place.

- The Configurator

[Ryan Grove]

On Wed, Oct 26, 2011 at 9:12 AM, The Configurator <config...@gmail.com> wrote:

A couple of usage examples:

<library uri="lib://jquery.com/jquery" version="1.6.4"><script src="/scripts/jquery.1.6.4.min.js"></script></library>

<library uri="lib://jquery.com/jquery-ui" version="1.8.16"><script src="/scripts/jquery-ui.1.8.16.min.js"></script><link rel="stylesheet" src="/css/jquery-ui.1.8.16.css" /></library>

What are your thoughts on this? Is this feasible? Are there any problems I may have missed?

I think this has the potential to make the web a slightly better place.

I'm not sure the new syntax is justified by the functionality it provides. Existing syntax already serves this use case fairly well, and building CDN-specific knowledge into browsers seems like a recipe for headaches.

As a sidenote, it may be premature to assume that a large percentage of users would benefit from cached CDN resources. For example, most Yahoo! properties load YUI from the same yui.yahooapis.com CDN, but the actual number of users who arrive at a given web page with a primed cache tends to be fairly low (in the area of 30%). If anyone knows of a larger study on this topic, I'd be interested to see the results.

This is getting better as browsers increase their default cache sizes, but a non-trivial number of users still habitually clear their caches or use third-party "privacy-enhancing" software that does this.

- Ryan

[The Configurator]

As far as I know, the existing syntax doesn't allow falling back to access the scripts locally. If you use yui.yahooapis.com and that is down, or blocked for a specific user, how do you specify to the browser that it should use a local server version instead?

This syntax also provides something else which I should have mentioned in the original email but slipped my mind - it doesn't actually specify the CDN. It just specifies which library you want. So if there are Google's, Microsoft's, Yahoo's CDN etc. installed on a user's machine, it would check them all, or be smarter about it and have a list of current items in each CDN so that it can just request it from the one that is fastest on that user's connection. Since libraries are signed they are guaranteed to be legit, and thus the using site would work with any of them quite happily.

About the benefits of caching: you make a valid point. However, using CDNs more is exactly what we should strive for - and having a special syntax for it is one thing that can make it more mainstream.

- The Configurator

[Abraham Williams]

You can see an example of falling back to locally hosted if a CDN is down in the HTML5 Boilerplate source: https://github.com/h5bp/html5-boilerplate/blob/master/index.html#L48

Abraham
-------------
Abraham Williams | InboxQ | abrah.am | abraham+

@abraham | github.com/abraham | blog.abrah.am
This email is: [ ] shareable [x] ask first [ ] private.

[Ryan Grove]

On Wed, Oct 26, 2011 at 4:34 PM, The Configurator <config...@gmail.com> wrote:

As far as I know, the existing syntax doesn't allow falling back to access the scripts locally. If you use yui.yahooapis.com and that is down, or blocked for a specific user, how do you specify to the browser that it should use a local server version instead?

Existing syntax allows for error handling on script nodes, which makes it possible to provide failure fallbacks. But I agree that this is something that could be made easier. I think that could be a separate proposal from the <library> proposal, though.

This syntax also provides something else which I should have mentioned in the original email but slipped my mind - it doesn't actually specify the CDN. It just specifies which library you want. So if there are Google's, Microsoft's, Yahoo's CDN etc. installed on a user's machine, it would check them all, or be smarter about it and have a list of current items in each CDN so that it can just request it from the one that is fastest on that user's connection. Since libraries are signed they are guaranteed to be legit, and thus the using site would work with any of them quite happily.

There are a couple of things that worry me about this.

As a library user, I want to have full control over which CDN gets used for my site. If some users get directed to one CDN while others get directed to another depending on which browser they're using, which version of that browser, or potentially their geographical location, I have no way of reliably pinpointing or reproducing issues a user reports that may be related to a malfunctioning CDN, stale CDN cache, etc.

As a library developer, I know that not all CDNs are created equal. A library like jQuery, for instance, has different CDN needs than a library like YUI. Loading YUI from a CDN that doesn't support combo-handled requests can result in poor performance because YUI is highly modular, whereas jQuery is delivered as a single file. Would browsers be expected to know that, when loading YUI, they should always prefer yui.yahooapis.com over ajax.googleapis.com?

 

It seems like <library> would replicate much of the behavior of <script>, but with some extra baked-in sugar to make loading libraries easier. I like the idea of making failure fallbacks easier to implement, but I think that should be decoupled from the idea of baking predefined CDN knowledge into browsers.

 

About the benefits of caching: you make a valid point. However, using CDNs more is exactly what we should strive for - and having a special syntax for it is one thing that can make it more mainstream.

I'm not sure I agree that a special syntax would make CDN usage more mainstream, but I definitely agree that CDN usage (and particularly the usage of common shared CDNs rather than site-specific CDNs) should be encouraged. Google's doing a pretty fantastic job of this, as is cdnjs.com.

- Ryan

[Eli Perelman]

Interesting idea. Not to cause a lot of scope creep, but you could add attributes to script and link tags for environments. I am just thinking of this as I go, but I suppose you could have a special environment tag to help facilitate selection of assets and even use it as an application-wide variable:

<environment name="{local|dev|test|prod}" />

<!-- src attribute for backwards compatibility: -->

<script

    src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js" 

    local="/scripts/jquery.js"

    dev="https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.js"

    test="https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js"

    prod="https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js"

    fallback="/scripts/jquery.js"></script>

Obviously there is a lot of crap there, and I'm not saying all these should be included. Just throwing out some ideas off the top of my head.

Thanks,

Eli Perelman

 

 

Thanks,

 

Eli

[The Configurator]

On Thu, Oct 27, 2011 at 00:52, Ryan Grove <ry...@wonko.com> wrote:

On Wed, Oct 26, 2011 at 4:34 PM, The Configurator <config...@gmail.com> wrote:

As far as I know, the existing syntax doesn't allow falling back to access the scripts locally. If you use yui.yahooapis.com and that is down, or blocked for a specific user, how do you specify to the browser that it should use a local server version instead?

Existing syntax allows for error handling on script nodes, which makes it possible to provide failure fallbacks. But I agree that this is something that could be made easier. I think that could be a separate proposal from the <library> proposal, though.

I stand corrected - apparently it is possible, though inelegant. There should be a way to load scripts that's baked into the language rather than document.write('<script>'), but I digress. 

This syntax also provides something else which I should have mentioned in the original email but slipped my mind - it doesn't actually specify the CDN. It just specifies which library you want. So if there are Google's, Microsoft's, Yahoo's CDN etc. installed on a user's machine, it would check them all, or be smarter about it and have a list of current items in each CDN so that it can just request it from the one that is fastest on that user's connection. Since libraries are signed they are guaranteed to be legit, and thus the using site would work with any of them quite happily.

There are a couple of things that worry me about this.

As a library user, I want to have full control over which CDN gets used for my site. If some users get directed to one CDN while others get directed to another depending on which browser they're using, which version of that browser, or potentially their geographical location, I have no way of reliably pinpointing or reproducing issues a user reports that may be related to a malfunctioning CDN, stale CDN cache, etc.

Libraries are meant to have one and only one representation per version, so that it actually wouldn't matter - any library will be served the same by any CDN that has it. Maybe signing is not enough to guarantee that, but I like to think that it is. So the user should actually not care where the library comes from - only that it's there, reliably, and that it's the right contents. A library, once published, has to be immutable.

As a library user, I definitely don't want to know which CDN gets used for my site. That's my main motivation for posting this suggestion.

As a comparison, look at an app store or market, or at apt-get, or at ruby gems. All of these have one thing in common: even though I've used all of them, I have no idea where the data came from. And they all work great. I want the same thing - but for client-side libraries.

 

As a library developer, I know that not all CDNs are created equal. A library like jQuery, for instance, has different CDN needs than a library like YUI. Loading YUI from a CDN that doesn't support combo-handled requests can result in poor performance because YUI is highly modular, whereas jQuery is delivered as a single file. Would browsers be expected to know that, when loading YUI, they should always prefer yui.yahooapis.com over ajax.googleapis.com?

The way I envisaged it, the CDN would have to support whichever protocols are required by the library to host it. I would also assume requesting multiple files at one time has to be possible for most libraries even though it isn't for jQuery.

I also think browsers should keep track of which CDNs are quicker in general so they can make intelligent choices. And when CDNs publish lists of library they could say "I provide yui but only partial support for required features", so a browser could know to prefer one with full support if it exists.

It seems like <library> would replicate much of the behavior of <script>, but with some extra baked-in sugar to make loading libraries easier. I like the idea of making failure fallbacks easier to implement, but I think that should be decoupled from the idea of baking predefined CDN knowledge into browsers. 

True, we need better ways to fallback. I don't think this detracts from the idea of standardising CDNs.

About the benefits of caching: you make a valid point. However, using CDNs more is exactly what we should strive for - and having a special syntax for it is one thing that can make it more mainstream.

I'm not sure I agree that a special syntax would make CDN usage more mainstream, but I definitely agree that CDN usage (and particularly the usage of common shared CDNs rather than site-specific CDNs) should be encouraged. Google's doing a pretty fantastic job of this, as is cdnjs.com.

- Ryan

In short, this idea combines the following:

• CDNs which 'just work',
• bundling some script files and some css files as a single library,
• a simple fallback mechanism.

I'm not sure it's a perfect idea, but it still sounds good to me.

- Dor

[Thomas Davis]

This could be easily built as a browser extension right?

I work on http://cdnjs.com and heavily use Require.js for development so can offer some insight to anyone who embarks on the journey.

[The Configurator]

On Thu, Oct 27, 2011 at 02:36, Thomas Davis <thomasal...@gmail.com> wrote:

This could be easily built as a browser extension right?

It could, but then the end users would have to install it. Why would an end user install a browser extension if it only benefits the library user / site developer?

I suspect as an extension it wouldn't be useful except as a proof of concept.

- Dor