Avoiding conflicting/repeat imports

[Michael Bleigh]

Something that I've been thinking a lot about lately is the idea of companies offering importable elements instead of their existing JavaScript APIs. So Mixpanel, or Google Analytics, or even GitHub would, instead of saying "load this JS" just say:

<link rel="import" href="https://api.some-company.com/api.html">

Which I think is a fantastic and exciting premise! What isn't as clear or exciting is how to manage the potentially conflicting dependencies. Let's say that the imported API element uses core-ajax but so does my application. How is that conflict resolved?

Of course Bower solves this by simply only having one of a given dependency, but we're talking about the Web Platform here and Bower, cool as it is, isn't really core Web Platform tech. Not to mention that there are big potential wins to being able to import an always-up-to-date API directly from the source.

Right now it *seems* as though first-to-register wins. I see errors in my console if an element tries to be registered twice. But I'm not certain whether the Polymer behaviors are likewise ignored, are re-applied to the existing definition, or what.

Basically I just think this is something that merits some thought and discussion, and I'd love to hear what you all think.

[Ian MacLeod]

Here's some previous threads that might help spur some discussion:

• Canonical CDNs? (not really)
• Centralized components host?
• AMD/requirejs + imports?
• ... and some others that I'm totally failing to find :(
• Also, there's core-shared-lib which sorta tackles this

It feels like it's best to tackle that at import time (as opposed to registration time) - purely for reducing bytes over the wire. Though, it's not clear how to do that w/o adding a ton of complexity to imports.

It'd be interesting to have some sort of scheme where you can identify assets via some means other than a URL. Probably some sort of hash, cert, or signature. Doing it in a secure way would be challenging, though

Follow Polymer on Google+: plus.google.com/107187849809354688692
---
You received this message because you are subscribed to the Google Groups "Polymer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to polymer-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/polymer-dev/93ab4fd9-adba-49a3-9368-03e6f5166509%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Bleigh]

Thanks for the links! That's super-helpful. Completely agree that import-time is the right time to tackle this. Also agree that it's possible that a new spec would need to be added to really make this work.

Unless I'm misunderstanding, it seems like the security concerns wouldn't be that big of a deal. You should never, ever be importing content from a source you don't already trust, so some form of resource hashing seems like it would only be an efficiency/deduplication issue, not a security issue. I can imagine something like a "unique" or "once" attribute on <link> that would basically say "don't load this resource if it's already been loaded." However this would require fetching and hashing the resource (or having an agreed-upon header to respond with in HEAD to verify the hash), which means you're still sending the data over the wire.

One way to manage this in a way that leaves most of the solution up to userland (which is imo a good idea in most cases) would be to have an attribute on the link tag called e.g. imports that would be a space-delimited token attribute. The idea here would be that if the resources specified by imports are already loaded (or already being loaded from an uncomplete import) they are simply skipped. Of course a <link> tag without this attribute would load regardless. To illustrate what I mean, imagine something like this:

<link rel="import" href="https://api.github.com/wc/repo.html" imports="core-ajax github-repo github-user">

You would want to have some kind of canonical API to be able to manipulate imports via JS as well, something like document.imports. OK, so this would solve some of our problems but leave others:

1. What about conflicting versions of an import? We could do versioning e.g. core...@0.3.3 but we can't really expect to put semantic versioning into an HTML spec.
2. Depending on the import, that "imports" list could become very long indeed.

2. is a mostly aesthetic concern and I suppose the length of your imports list would be determined by how much you care about dependency deduplication.

1. is a much trickier issue, and to solve it I would propose that you be able to actually reach in and define the import resolver yourself. For example, document.imports.resolver = function(import){ }

If you did that, you'd be able to slot in something for whatever system you like that maybe *would* understand semantic versioning and throw errors when versions are too mismatched but allow minor version incompatibilities, etc. You could even theoretically feed it configuration a la bower.json and let it run from there.

Eric's HTML5 Rocks article is titled "HTML Imports: #include for the web". I think that's a fantastic ideal but one that isn't quite being lived up to yet. Until we're able to properly resolve dependency conflicts and multiple imports, we're not going to be able to simply import and forget it. Obviously the Polymer team is running head-on into this issue by the universal use of Bower, a third-party package manager.

I know there has to be a way to do this natively in a way that still allows for proper userland expansion. Anyways, thanks for taking the time to listen to my rambling, would love to discuss further.

To unsubscribe from this group and stop receiving emails from it, send an email to polymer-dev...@googlegroups.com.

[Ian MacLeod]

On Fri Jul 11 2014 at 11:01:05 AM, Michael Bleigh <mbl...@gmail.com> wrote:

Thanks for the links! That's super-helpful. Completely agree that import-time is the right time to tackle this. Also agree that it's possible that a new spec would need to be added to really make this work.

Unless I'm misunderstanding, it seems like the security concerns wouldn't be that big of a deal. You should never, ever be importing content from a source you don't already trust, so some form of resource hashing seems like it would only be an efficiency/deduplication issue, not a security issue. I can imagine something like a "unique" or "once" attribute on <link> that would basically say "don't load this resource if it's already been loaded." However this would require fetching and hashing the resource (or having an agreed-upon header to respond with in HEAD to verify the hash), which means you're still sending the data over the wire.

Yeah, you're right here - the hashing approach really only tackles identical resources, and your suggested approach below nicely circumvents that :)

 

One way to manage this in a way that leaves most of the solution up to userland (which is imo a good idea in most cases) would be to have an attribute on the link tag called e.g. imports that would be a space-delimited token attribute. The idea here would be that if the resources specified by imports are already loaded (or already being loaded from an uncomplete import) they are simply skipped. Of course a <link> tag without this attribute would load regardless. To illustrate what I mean, imagine something like this:

<link rel="import" href="https://api.github.com/wc/repo.html" imports="core-ajax github-repo github-user">

I like this! Best of all it's simple and obvious to the user.

 

You would want to have some kind of canonical API to be able to manipulate imports via JS as well, something like document.imports. OK, so this would solve some of our problems but leave others:

1. What about conflicting versions of an import? We could do versioning e.g. core...@0.3.3 but we can't really expect to put semantic versioning into an HTML spec.

I tend to agree, after watching various communities try to adhere to semantic versioning to varying degrees of success, it seems too heavy-handed.

Why not just treat them as strings? You could even cheat a little:

<link rel="import" href="https://api.foo.com/bar.html" imports="bar-awesome bar-awesome@0 bar-awesome@0.1 bar-a...@0.1.2">

This example is probably overkill, but is certainly flexible.

1. Depending on the import, that "imports" list could become very long indeed.

 

2. is a mostly aesthetic concern and I suppose the length of your imports list would be determined by how much you care about dependency deduplication.

1. is a much trickier issue, and to solve it I would propose that you be able to actually reach in and define the import resolver yourself. For example, document.imports.resolver = function(import){ }

Starts to sound pretty similar, mechanically, to ES6 modules 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/polymer-dev/4b09301e-053a-4402-8d2d-1afb1e5490dd%40googlegroups.com.

[Scott Miles]

Please everybody remember that Bower is only a means to an end and actually has no relevance to your run-time environment. Once files exist on a server somewhere, it does not matter whether they were obtained via Bower, Zip, npm, Git, etc.

The actual rule is this: from the _client's_ perspective, all components come from a single folder. 

Now, It's absolutely true that rule is limiting, and we are working on improving the situation in various ways, but mostly this involves changing HTMLImports spec which is a laborious standards process.

We also acknowledge the problem with de-duplicating concatenated resources. If `imports.html` includes `core-ajax.html` there is no way for HTMLImports to understand that. This is something we are also working on at the spec level, and something we are trying to address in Vulcanizer.

Re: versioning, the strategy today is that one must handle this at the project level. Whatever `<components>` folder a page is using must be populated with the correct component versions for this project. If the project has component A that requires C.v1 and component B that requires C.v2, this is an fundamental incompatibility that one must resolve by updating A or removing one of A or B.

I expect to get approximately a zillion replies about how this is not acceptable. All I can say is that we do understand the consequences, but this is the reality for now.

To view this discussion on the web visit https://groups.google.com/d/msgid/polymer-dev/CAKc-BFiSTPCi3K9r8jFRVYYCLTUGoVdgPKULOpOoMsRk7_Hwpw%40mail.gmail.com.

[Michael Bleigh]

Hey Scott,

I absolutely know that Bower is a means to an end and meant nothing disparaging about what your team is already doing. I started the discussion because I'd like to be part of the solution. Can you give any broad strokes about what you're proposing to the HTML Imports spec that would help address these issues?

The goal that I have in mind when I'm imagining the future is that you should be able to build a solid web application without any kind of package manager or build process. Polymer gets us so much further down that road than we were (Ele, for instance, right now has no build process. We will probably end up vulcanizing it for performance reasons but it works perfectly fine as a flat directory. Still needs Bower, though).

My thought is more along the lines of "in for a penny, in for a pound." If we're already proposing drastic changes to web standards to get Web Components going, we may as well try to address this issue while we're at it. Prior statement being offered with zero experience dealing with standards bodies or multiple browser vendors with their own agendas. :)

[Scott Miles]

>> I absolutely know that Bower is a means to an end and meant nothing disparaging 

No offense was taken =P. But there continues to be confusion at large that Bower is required, or is related to any of these run-time issues. I take every opportunity to try to dispel this confusion.

>> can you give any broad strokes about what you're proposing to the HTML Imports spec that would help address these issues?

There are a few possibilities:

• A (low-level) imports hook that could preview an import href, modify it, or cancel it based on user logic. This is tricky to implement, because of interactions with things like the preload scanner (that allow the user-agent to discover and begin downloading imports in advance of parsing or execution of JavaScript). There are discussions about blending this with the ES6 module loader system, since both systems need to hook into the user-agent network stack.
  
• A better de-duplication heuristic. For example, de-duplicating on the last two parts of the path (e.g., given '/server/foo/path/core-foo/core-foo.html', the key would be 'core-foo/core-foo.html'), would be backward compatible with all current Polymer projects, but would allow loading components from heterogeneous component sources. (i.e. import 'foo/core-ajax/core-ajax.html' and 'cdn/core-toolbar/core-toolbar.html', which you cannot do today).
• A method for _salting_ the imports cache, so that one could programmatically tell the import system that a resource has already been marshaled and future requests should be de-duped.

>> The goal that I have in mind when I'm imagining the future is that you should be able to build a solid web application without any kind of package manager or build process. 

Again, 'package manager' is a convenient way of acquiring packages. Afaict, only in the distant future will it *never* be advantageous to have a local copy of some resources. Package managers exist only to make your life easier, they are not a necessary evil.

Having said that, eliminating any barriers between development, testing, and production (aka build process) is one of my goals too. There are some problems we cannot overcome purely client-side today, and that's where Vulcanizer comes in (and Karma, and a few other things).

Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/polymer-dev/23bbb692-01fd-4871-86b5-aa36cd264f1c%40googlegroups.com.

[dqu...@versestudios.com]

I may be way off base, as my experience ends at having watched a few videos and played with Polymer, but couldn't this be implemented using service workers to handle the caching logic, hash comparisons, etc.?