DOM-injecting script-tag wrapped-module loader implementation notes
Wes Garland
James Burke, you mentioned that RequireJS uses a timeout-based scheme to try and detect load failure, and that load-failure is the one point where you can't run the CommonJS Modules/1.0 test suite properly.
I think I have basically solved this problem (tested only in Firefox, Safari, and IE 8). The key is that the program semantics require that load-failure be memoized during dependency resolution, so that require() can throw an exception when the unresolved module is required. Remember also that CommonJS semantics require that the module factory itself is also not invoked until the first call to require.
Detecting load failure in non-IE can be done with an onerror handler on the script element. Module identity is determined the same way it is for the onload event, and a "load failure" is memoized.
Detecting load failure in IE can be done with the onreadystatechange event handler. When
- the handler trips and this.readyState === "loaded"
- the module has not previously been memoized
- the module id of the current module (per Kris Zyp's algorithm) is the same as the module id that created the event handle (captured via closure)
..you can memoize load failure the same way as non-IE in the onerror handler
The third point in the IE algorithm is required to handle the case of cyclic events.
The module loader I'm playing with now successfully passes all the modules/1.0 tests with wrapped modules and a DOM-injecting loader.
I am currently not doing parallel loads. Are you? Will explore this at a later date. Would like to load all dependencies in parallel rather than one at a time via ~ CPS.
Wes
--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
Sander Tolsma
After observing the discussions in the same threads I also decided to build
a loader platform to learn about possible implementation pitfalls and to
understand how modules work... ;-)
My experience is almost as you describe it; except that I see IE loading
from cache and executing defines/module.decare without calling
onreadystatechange. So I implemented the algorithm of Kris Zyp and that
solved the problem! I also solved the load failures for IE in the same way
you described but didn't have time yet to implement the onerror handler for
all other browsers:
> Detecting load failure in IE can be done with the onreadystatechange event
handler. When
> - the handler trips and this.readyState === "loaded"
> - the module has not previously been memoized
> - the module id of the current module (per Kris Zyp's algorithm) is the
same as the module id that created the event handle (captured via closure)
> ..you can memoize load failure the same way as non-IE in the onerror
handler
The following is new to me, need to find out if this has implications on the
algorithm I implemented...
> Remember also that CommonJS semantics require that the module factory
itself is also not invoked until the first call to require.
> I am currently not doing parallel loads. Are you? Will explore this at a
later date. Would like to load all dependencies in parallel rather than one
at a time via ~ CPS.
Yep, my code does parallel loads. That was giving the problems with IE...
;-)
For anyone interested I put the code on github at:
https://github.com/stolsma/apprequire ,documentation will follow in the next
two weeks. To be sure: I'm not intending to bring it to a bigger audience,
just trying to use it to learn the CommonJS drafts/specs...
BTW I hope to interact in the discussions from now on (instead of just
reading, that's what I've done for the last 2 months). This is the first
time I post something to this group so sorry if something goes wrong... ;-)
Regards,
Sander
Sander Tolsma
Tolsma ICT Consultancy
www.tolsma.net
khs4473
Like the package loading stuff in there, too. : )
khs4473
Sander Tolsma
Oepss, did I put the wrong name there ?? :-( Who is the real author then ??
> Like the package loading stuff in there, too. : )
You're fast !! :-) I'm busy writing another msg with some questions about
Packages, hope someone can help me with the answers (shouldn't be a problem
with all those smart people around here I think... ;-))
Sander Tolsma
In my first post in reply to Wesley I explained that I'm trying to
understand the CommonJS drafts by writing a module loader for browser side
use. At this moment the code is passing all the Modules/1.0 tests so I did
the next step and implemented a first version of the Packages drafts
(Packages/Transport, Packages/1.0 and Packages/Mappings/C).
After trying to write some Package test I found out that I maybe didn't
understand the Package Mappings draft. The question I have is about two
Packages that need the same subpackage, ie:
Package1 needs Package3
Package2 needs Package3 too..
Package3 is for both packages at the same Mappings location...
Following Mappings/C Package1 and Package2 have their own entry point to
Package3 modules (for example Package1 is using
MyPackage3/path/to/module/in/Package3 and Package2 is using
TestPackage/path/to/module/in/Package3) but now my question is if both get
the same exports if they request the same module from Package3 ? At this
moment I implemented totally separated exports i.e : Package1 get another
exports then Package2..
Can someone explain what is the preferred implementation, or is this not
defined yet ?? Thanks a lot!!!!
Kris Kowal
> Can someone explain what is the preferred implementation, or is this not
> defined yet ?? Thanks a lot!!!!
I do not believe that this has been defined yet. It may be left to
implementors for the present.
One dependent factor is package dependency cycles. I'm generally
against allowing these, since modules with cyclic dependencies should
generally be developed in coordination; that is: in a package. This
constraint would mitigate one possible cause for necessitating shared
exports.
The only other issue I can think of is whether there should be shared
state among all occurrences of a package in a sandbox. Whether that's
desirable probably varies from case to case.
Kris Kowal
Kris Zyp
full absolute canonical URL/identification of the module. If the
mappings resolve to different URLs for the module in Package3 from
Package2 and Package1, then it should get a different exports,
otherwise, it can be the same.
--
Thanks,
Kris
Christoph Dorn
> Mappings allow the package handler or module loader to determine the
> full absolute canonical URL/identification of the module. If the
> mappings resolve to different URLs for the module in Package3 from
> Package2 and Package1, then it should get a different exports,
> otherwise, it can be the same.
On 10-11-18 1:35 PM, Kris Kowal wrote:
> On Thu, Nov 18, 2010 at 1:16 PM, Sander Tolsma<goo...@tolsma.net> wrote:
>> Can someone explain what is the preferred implementation, or is this not
>> defined yet ?? Thanks a lot!!!!
>
> I do not believe that this has been defined yet. It may be left to
> implementors for the present.
>
> The only other issue I can think of is whether there should be shared
> state among all occurrences of a package in a sandbox. Whether that's
> desirable probably varies from case to case.
I think we need to specify this ASAP now that mappings/C is gaining more
implementations (I just patched requireJS). There are real concequences
on program code and implementations must be consistent for programs to
be portable.
I have been using mappings extensively in my programs and think my POV
is aligned with Kris Zyp's. This is how I see things:
* A CommonJS environment may contain one or more sandboxes
* Every sandbox has it's own independent module namespace
thus within the context of a sandbox:
* Relative module IDs are converted to canonical IDs based on the
calling module
* The same canonical ID will always return the same exports
* Mappings convert an alias + ID to a canonical ID
It is left up to the implementation as to what the canonical ID of a
mapping target looks like within the namespace of the sandbox but the
same canonical ID must always return the same exports.
That is given a package such as:
http://foo.com/bar.zip!/package.json {
"uid": "<UID>",
}
an implementation could store module exports for this package in the
namespace in any of the following ways:
root: foo.com/bar/<moduleId>
root: foo.com/bar.zip/<moduleId>
root: foo.com/bar.zip!/<moduleId>
root: foo.com/bar!/<moduleId>
root: <UID>/<moduleId>
root: com.foo/bar/<moduleId>
root: com/foo/bar/<moduleId>
etc...
The only time a module may need to require another module by canonical
ID is when the module does not have mappings defined in it's own
package.json, when it is requiring a system module (on the environment
global namespace) or when the module is trying to 'enter' a package for
the first time by requiring a module from it (as would happen during the
bootstrapping process).
Some canonical ID formats allow the loader to parse out the package to
module ID boundary and thus load the correct package environment and
corresponding mappings:
root: foo.com/bar.zip!/<moduleId> // split by !
root: foo.com/bar!/<moduleId> // split by !
root: <UID>/<moduleId> // assume one-term packageID
Others may need to match known packageIDs against the provided ID.
The most consistent way to 'enter' a package would be to provide a
mapping that the loader can resolve. e.g.
PM.package("http://foo.com/bar.zip").require("<moduleID>");
PM.package({
"archive": "http://foo.com/bar.zip"
}).require("<moduleID>");
PM.package({
"catalog": "http://foo.com/packages-catalog.json",
"name": "bar"
}).require("<moduleID>");
It would be great if this could be specified to make programs portable.
The above outlined canonical ID policy breaks if within a program two
packages map to the same package via two different catalogs or
strategies (one may specify the archive directly while another specifies
the URL to the package root).
The only solution I have been able to come up with to address this is to
have packages provide a UID. The UID is a string identifying the package
uniquely among all existing packages. Rather than using a hash a URL
seems to make most sense. So a package hosted on github at:
https://github.com/cadorn/insight
could for example have a UID of:
package.json ~ {
"uid": github.com/cadorn/insight
}
This package can now get a canonical ID == UID no matter where it came
from and the mappings strategy used.
If the repository is forked and the fork wishes to release a patched
version the UID must be changed to indicate that it is a different package.
It is up to the user to decide which mapping strategies to use and which
catalogs to trust. It is up to the catalog authors to verify the
authenticity of the packages they include and they reserve the right to
rewrite the UIDs of included packages.
Without the UID I think it is impractical/impossible to provide the same
exports to 'equivalent' mappings. Thus I think the wording should be
something as follows.
In the context of package mappings:
* A loader *must* provide the same exports for a module that is
contained within a package specifying a UID
* A loader *may attempt* to provide the same exports for a module
that is contained within a package *not* specifying a UID
This allows a package author to declare and ensure that by default only
one instance of their modules will be loaded within the same sandbox.
Having the same exports (and module instances) available has been a huge
bonus in my work where I can have modules keep their own state no matter
how they are referenced within the same sandbox. It allows for easy
conceptualization and clean module coupling with minimal external
dependencies for a module other than pure 'helper' modules for static
functionality or mixins.
This makes packages very light as they do not layer any restrictions on
how their modules are to be used within a sandbox. A package is simply a
facility to group modules, declare dependencies and pre-processors and
allow for arbitrary composition within sandboxes. The package boundaries
effectively disappear within a sandbox other than impacting require()
resolution semantics.
Christoph
Christoph Dorn
Correction. See below.
On 10-11-18 5:27 PM, Christoph Dorn wrote:
> https://github.com/cadorn/insight
>
> could for example have a UID of:
>
> package.json ~ {
> "uid": github.com/cadorn/insight
> }
>
> This package can now get a canonical ID == UID no matter where it came
> from and the mappings strategy used.
This should be:
package.json ~ {
"uid": "github.com/cadorn/insight",
"version": "1.2.3"
}
canonicalID = <uid>/<version>/<moduleID>
Christoph
Wes Garland
> I like how you attribute FlyWrap to Jimmy Page. That's awesome!!Oepss, did I put the wrong name there ?? :-( Who is the real author then ??
Jimmy Page was the lead guitarist for the now-defunct group Led Zeppelin, which is probably the best blues/rock band of all time and a major influence on the genre as a whole. Jimmy Page himself is probably one of the top 10 guitarist-musicians alive today.
Kevin seems to like to checking in stuff as other people. He is the author of the Fly tools.
Also, if you look through his repo, you'll notice Jimi Hendrix also "contributes". Hendrix was another blues-based Rocker, and possibly the best guitarist that ever walked the face of the earth. His influence on Rock music is on the same order of magnitude to Chuck Berry and The Beatles.
Wes
> Like the package loading stuff in there, too. : )You're fast !! :-) I'm busy writing another msg with some questions about
Packages, hope someone can help me with the answers (shouldn't be a problem
with all those smart people around here I think... ;-))
--
You received this message because you are subscribed to the Google Groups "CommonJS" group.
To post to this group, send email to comm...@googlegroups.com.
To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.
khs4473
look at them : )
BTW - I just pushed a ton of changes, and also changed the name of
this doohickey to "FlyScript" (the PHP thing is now "FlyServer" and
will eventually bundle in FlyScript). I tried to make the code more
readable for anyone else working on similar projects (https://
github.com/khs4473/FlyScript).
The onerror handler is great for detecting 404's, but I haven't
figured out a way to detect them in Opera, yet.
I've got web worker support going, if anyone want to check that out.
It's all serial loading in web workers, though, as I haven't
researched yet how to get importScripts to actually download things in
parallel. I referenced James' work in requirejs while writing the web
worker support.
I've also added a "mount" function for mapping parts of the module
namespace to different URLs. Multiple module IDs can be mapped to an
arbitrary number of URLs, and eventually this will allow it to connect
to module servers and receive module bundles. Examples:
FlyScript.mount(
{
"jquery": "http://code.jquery.com",
"my-server": function(ids) { ... }
});
FlyScript.load("jquery/jquery-1.4.4.min.js", function() { ... });
I changed the signature of the "module" methods a bit:
module.declare(dep, factory) : For this one, the source code is
scanned ONLY when the dependency array is not present. This allows
for optimization where the developer doesn't want scanning to happen.
module.attach(id, dep, factory) : This is the first transport format
optimized for concatenation. Once again, the dependency array is
optional. The source code is scanned ONLY when the dependency array
is not present.
module.attachSet(ids, dep) : This is the second transport format,
optimized for bundling by module servers. Pretty straightforward - no
scanning.
Eventually I'll add some documentation...
Hope it helps!
Kevin
> > .
Sander Tolsma
When I put the reference to Jimmy Page in the code I was already thinking
that there was something wrong, but yeah some parents like to name their
children after famous persons eh. ;-) But thanks for the explanation, will
change the ref !!! :-)
Also I want to apologize to you because I somewhat hijacked your start
post.. I just replied to your post and changed the subject without knowing
it was still going in the same thread. :-( Won't happen again.
Sander
James Burke
Thanks for sharing your findings! I put up a basic browser test to
test different error case, script error (throw), a parse error and a
404. It does seem that the script load event fires in the IE case of a
404, and for Firefox, the script load does not fire, but the onerror
does. So I can see a combination of those two things giving a way to
identify a failed module in a more timely fashion than a timeout.
Neat.
http://www.requirejs.org/temp/browsertests/onerror/
GitHub: https://github.com/jrburke/requirejs/tree/master/tests/browsertests/onerror/
It seems helpful to also allow for a timeout for slow responding
scripts -- something where a connection is made to the server, or is
still open, but may never complete. The app in this case may want to
give a shorter timeout than a network timeout to give the user a more
timely message.
> I am currently not doing parallel loads. Are you? Will explore this at a
> later date. Would like to load all dependencies in parallel rather than one
> at a time via ~ CPS.
Yes, dependencies are loaded in parallel in RequireJS.
James
khs4473
> test different error case, script error (throw), a parse error and a
> 404. It does seem that the script load event fires in the IE case of a
> 404, and for Firefox, the script load does not fire, but the onerror
> does. So I can see a combination of those two things giving a way to
> identify a failed module in a more timely fashion than a timeout.
> Neat.
detect it by polling the readyState (it will change to "loaded" when
the 404 response is received), but I'm worried that the poller would
incorrectly fire before the the onload on a 200.
jbrantly
onerror, and timeout) to handle a number of cases. Failures can
usually be detected immediately, but I also observed that Opera had to
fall back to a timeout. I might have to look more into the polling
readyState idea if I can find some time.
Relevant code at https://github.com/jbrantly/yabble/blob/master/lib/yabble.js#L365-414
Not currently using the "anonymous module" stuff though (where the
module ID is not specified).
Sander Tolsma
Thanks for your replies, looks like this needs some thorough thinking..
On 10-11-18 1:38 PM, Kris Zyp wrote:
> Mappings allow the package handler or module loader to determine the
> full absolute canonical URL/identification of the module. If the
> mappings resolve to different URLs for the module in Package3 from
> Package2 and Package1, then it should get a different exports,
> otherwise, it can be the same.
That was my thinking too !!
On 10-11-18 1:35 PM, Kris Kowal wrote:
> I do not believe that this has been defined yet. It may be left to
> implementors for the present.
>
> One dependent factor is package dependency cycles. I'm generally
> against allowing these, since modules with cyclic dependencies should
> generally be developed in coordination; that is: in a package. This
> constraint would mitigate one possible cause for necessitating shared
> exports.
> The only other issue I can think of is whether there should be shared
> state among all occurrences of a package in a sandbox. Whether that's
> desirable probably varies from case to case.
Hmmm, your right about cyclic dependency between packages. I will make some
cyclic package unit tests when I changed my current package implementation.
See what happens then.. ;-)
On 10-11-18 6:27 PM, Christoph Dorn wrote:
> I think we need to specify this ASAP now that mappings/C is gaining more
> implementations (I just patched requireJS). There are real concequences
> on program code and implementations must be consistent for programs to
> be portable.
That's also the reason I wanted to check if there was already any thinking
and spec building done before I started to make my own implementation.
> This is how I see things:
>
> * A CommonJS environment may contain one or more sandboxes
> * Every sandbox has it's own independent module namespace
>
> thus within the context of a sandbox:
>
> * Relative module IDs are converted to canonical IDs based on the
> calling module
> * The same canonical ID will always return the same exports
> * Mappings convert an alias + ID to a canonical ID
>
> It is left up to the implementation as to what the canonical ID of a
> mapping target looks like within the namespace of the sandbox but the
> same canonical ID must always return the same exports.
Exactly how I see it too...
> The only time a module may need to require another module by canonical
> ID is when the module does not have mappings defined in it's own
> package.json, when it is requiring a system module (on the environment
> global namespace) or when the module is trying to 'enter' a package for
> the first time by requiring a module from it (as would happen during the
> bootstrapping process).
Do you have an example of the first point (package needs another package but
failed to refer in its package.json file) ? My opinion is that if that
happens the module needs to call a function to have a package added to the
mappings list of its own package. Then normal package mapping procedures can
be followed (like getting the package.json file and perhaps with that
preloading modules with the transport format like described in
Packages/Transport for browsers).
The second point is valid I think, how can a module in a package request a
module in the 'main'/'default'/'system' package. Maybe that can be done by
defining a standard identifier as package mapping identifier like
'main/path/to/module'? Then packages don't need to know where the system
package is..
> Some canonical ID formats allow the loader to parse out the package to
> module ID boundary and thus load the correct package environment and
> corresponding mappings:
>
> root: foo.com/bar.zip!/<moduleId> // split by !
> root: foo.com/bar!/<moduleId> // split by !
> root: <UID>/<moduleId> // assume one-term packageID
>
> Others may need to match known packageIDs against the provided ID.
This is implementation specific ? Standard modules/packages don't need to
know this behavior or do you want to standardize this? (which is already
done I think in the Mappings/C draft)
> The most consistent way to 'enter' a package would be to provide a
> mapping that the loader can resolve. e.g.
Yep, as we do already with Mappings/C
> The above outlined canonical ID policy breaks if within a program two
> packages map to the same package via two different catalogs or
> strategies (one may specify the archive directly while another specifies
> the URL to the package root).
That's what I found out too with a test implementation I build yesterday.
> The only solution I have been able to come up with to address this is to
> have packages provide a UID. The UID is a string identifying the package
> uniquely among all existing packages. Rather than using a hash a URL
> seems to make most sense.
> It is up to the user to decide which mapping strategies to use and which
> catalogs to trust. It is up to the catalog authors to verify the
> authenticity of the packages they include and they reserve the right to
> rewrite the UIDs of included packages.
I don't like this intertwining of plain package standards and Package
Manager/Catalogs, my feelings are that Package managers and catalogs need to
be another standards layer on top of a defined packages definition, just
like packages are using the standard module layer. I.e. Modules
Layer->Packages Layer->Package Managers/Catalogs Layer->Anything on top of
that..
> In the context of package mappings:
>
> * A loader *must* provide the same exports for a module that is
> contained within a package specifying a UID
> * A loader *may attempt* to provide the same exports for a module
> that is contained within a package *not* specifying a UID
>
> This allows a package author to declare and ensure that by default only
> one instance of their modules will be loaded within the same sandbox.
Ok, if you define it like this then I think I can agree with the UID stuff.
Will implement the second bullet this week and do some testing to see what
pitfalls there are..
Again thanks for all the responses !!
Christoph Dorn
> On 10-11-18 6:27 PM, Christoph Dorn wrote:
> > I think we need to specify this ASAP now that mappings/C is gaining more
> > implementations (I just patched requireJS). There are real concequences
> > on program code and implementations must be consistent for programs to
> > be portable.
>
> That's also the reason I wanted to check if there was already any thinking
> and spec building done before I started to make my own implementation.
mappings/C has been well defined based on the underlying theory of
proper program composition via packages as well as package authoring and
distribution considerations.
Some implementations have been verifying this approach to varying
degrees and I think it holds up well so far.
The catalog support and UID property has not been implemented outside of
my own work, but I believe is essential to complete the picture.
I hope to refine and publish my work in this area soon (~3 months) by
providing an open infrastructure for a distributed package registry
system based on catalogs. I have been using this internally for over a
year now and it has literally changed the way I work for the better.
You can read about my approach here [1]. Note that it uses require()
with two arguments vs using the first term of the ID to map to the
package. I will be changing this to comply with mappings/C.
> > The only time a module may need to require another module by canonical
> > ID is when the module does not have mappings defined in it's own
> > package.json, when it is requiring a system module (on the environment
> > global namespace) or when the module is trying to 'enter' a package for
> > the first time by requiring a module from it (as would happen during the
> > bootstrapping process).
>
> Do you have an example of the first point (package needs another package but
> failed to refer in its package.json file) ? My opinion is that if that
> happens the module needs to call a function to have a package added to the
> mappings list of its own package. Then normal package mapping procedures can
> be followed (like getting the package.json file and perhaps with that
> preloading modules with the transport format like described in
> Packages/Transport for browsers).
My primary use-case for this has been to temporarily require modules
during development used for debugging or other purposes.
I know that one of my packages includes the debugging modules so instead
of mapping these in all my packages I can just use the canonical ID in
any of my modules to require the same with no extra effort. (I could put
these modules on the system path but then I have to deal with naming
conflicts.) I also see this useful for debugging and development tools
but have not explored this aspect much.
I am of the opinion that the package mappings should never be changed or
updated once a program has booted. I would thus be against a function to
add a mapping at runtime. I treat the package.json file as source code
and atomically versioned with the package source. (with the exception
that a catalog or package.local.json file may make changes).
Frozen mappings also make loaders and build tools simpler as there needs
to be no consideration whether a package has been included in a build or
needs to be built and fetched separately.
To add a package to the sandbox one should supply a mapping to the
loader for it to load it as a separate package. The loader should return
the canonical ID of the package you can use to require a module from it.
I call this loading a plugin package.
Your implementation can of course provide a function to add a mapping
dynamically and we can keep the wording in the spec open for this
although I don't think it should be encouraged.
> The second point is valid I think, how can a module in a package request a
> module in the 'main'/'default'/'system' package. Maybe that can be done by
> defining a standard identifier as package mapping identifier like
> 'main/path/to/module'? Then packages don't need to know where the system
> package is..
>
> > Some canonical ID formats allow the loader to parse out the package to
> > module ID boundary and thus load the correct package environment and
> > corresponding mappings:
> >
> > root: foo.com/bar.zip!/<moduleId> // split by !
> > root: foo.com/bar!/<moduleId> // split by !
> > root:<UID>/<moduleId> // assume one-term packageID
> >
> > Others may need to match known packageIDs against the provided ID.
>
> This is implementation specific ? Standard modules/packages don't need to
> know this behavior or do you want to standardize this? (which is already
> done I think in the Mappings/C draft)
It is up to the implementation as to how canonical IDs are constructed
and yes standard modules/packages relying on mappings do not need
knowledge of this.
> > The most consistent way to 'enter' a package would be to provide a
> > mapping that the loader can resolve. e.g.
>
> Yep, as we do already with Mappings/C
Right, but this aspect has not yet been specified. I think we should add
some wording in this regard once we have identified the best point of
entry. require() cannot be used for this as it requires a string literal
as ID. Maybe something like:
require.pkg({"archive": "..."}).require("moduleID")
This would allow the loader to verify that the requiring module has
authority to require the package with minimal extra effort as require()
is already scoped for the module.
> > The above outlined canonical ID policy breaks if within a program two
> > packages map to the same package via two different catalogs or
> > strategies (one may specify the archive directly while another specifies
> > the URL to the package root).
>
> That's what I found out too with a test implementation I build yesterday.
Good to hear you are exploring the spec to this extent.
> > The only solution I have been able to come up with to address this is to
> > have packages provide a UID. The UID is a string identifying the package
> > uniquely among all existing packages. Rather than using a hash a URL
> > seems to make most sense.
>
> > It is up to the user to decide which mapping strategies to use and which
> > catalogs to trust. It is up to the catalog authors to verify the
> > authenticity of the packages they include and they reserve the right to
> > rewrite the UIDs of included packages.
>
> I don't like this intertwining of plain package standards and Package
> Manager/Catalogs, my feelings are that Package managers and catalogs need to
> be another standards layer on top of a defined packages definition, just
> like packages are using the standard module layer. I.e. Modules
> Layer->Packages Layer->Package Managers/Catalogs Layer->Anything on top of
> that..
Right. The comment about catalog authors was meant to illustrate that
additional layer. These concerns must be separate and that was an
important goal behind the current package and mappings specs. Many
traditional package formats conjoin these aspects leading to undesirable
consequences. This is one area we hope to do better than prior art.
The package, mappings and catalog specs must and will exist separately
based on the requirements put forth by their respective areas of concern.
> > In the context of package mappings:
> >
> > * A loader *must* provide the same exports for a module that is
> > contained within a package specifying a UID
> > * A loader *may attempt* to provide the same exports for a module
> > that is contained within a package *not* specifying a UID
> >
> > This allows a package author to declare and ensure that by default only
> > one instance of their modules will be loaded within the same sandbox.
>
> Ok, if you define it like this then I think I can agree with the UID stuff.
The UID property makes sense even when only considering the package
itself. It allows for example for providing a collection service that
anybody can publish bug reports to that get relayed to the original
package author. I am using the UID property in numerous ways in the
toolchain I am building.
> Will implement the second bullet this week and do some testing to see what
> pitfalls there are..
Looking forward to your feedback.
Christoph
[1] - https://github.com/cadorn/pinf/blob/master/docs/Design/Foundation.md
Sander Tolsma
On 10-11-21 11:54 AM, Christoph Dorn wrote:
> The catalog support and UID property has not been implemented outside
> of my own work, but I believe is essential to complete the picture.
>
> I hope to refine and publish my work in this area soon (~3 months) by
> providing an open infrastructure for a distributed package registry
> system based on catalogs. I have been using this internally for over a
> year now and it has literally changed the way I work for the better.
>
> You can read about my approach here [1]. Note that it uses require()
> with two arguments vs using the first term of the ID to map to the
> package. I will be changing this to comply with mappings/C.
Thanks for the info. What you describe will probably be an answer to one of
the questions of one of my customers so I will be very interested in the
endresult !!
> My primary use-case for this has been to temporarily require modules
> during development used for debugging or other purposes.
>
> I know that one of my packages includes the debugging modules so
> instead of mapping these in all my packages I can just use the canonical
ID in
> any of my modules to require the same with no extra effort. (I could
> put these modules on the system path but then I have to deal with naming
> conflicts.) I also see this useful for debugging and development tools
> but have not explored this aspect much.
Ah, ok.. I deal with that by putting those modules in the 'System' or 'main'
package when needed and then let them put a referring boolean in de system
main module so every other module can see that debugging is there and
needed. Because I handle the (root) system modules together as a root
startup package I can just have a debug system package and a normal system
package. I tested this only on the browser side so I don't know if this will
work on the server side...
> I am of the opinion that the package mappings should never be changed
> or updated once a program has booted. I would thus be against a function
> to add a mapping at runtime. I treat the package.json file as source code
> and atomically versioned with the package source. (with the exception
> that a catalog or package.local.json file may make changes).
Snip...
>
> Your implementation can of course provide a function to add a mapping
> dynamically and we can keep the wording in the spec open for this
> although I don't think it should be encouraged.
I totally agree with you therefore my initial (because I didn't understand
you) reaction... ;-) The thing I'm worried about though is that if it is
allowed to add a package mapping after the package is loaded people will use
it mainstream for other things than the intended purpose (in your case
debugging etc..)
> It is up to the implementation as to how canonical IDs are constructed
> and yes standard modules/packages relying on mappings do not need
> knowledge of this.
Jippie !! So I did understand the drafts correctly ;-)
> > > The most consistent way to 'enter' a package would be to provide a
> > > mapping that the loader can resolve. e.g.
> >
> > Yep, as we do already with Mappings/C
>
> Right, but this aspect has not yet been specified. I think we should
> add some wording in this regard once we have identified the best point of
> entry. require() cannot be used for this as it requires a string
> literal as ID. Maybe something like:
>
> require.pkg({"archive": "..."}).require("moduleID")
>
> This would allow the loader to verify that the requiring module has
> authority to require the package with minimal extra effort as require()
> is already scoped for the module.
Hmm, I do not understand this.. Mappings/C already describes this or am I
wrong ? If that's the case why do you need something like
require.pkg({"archive": "..."}).require("moduleID") ??
If a mapping is made (i.e. the package.json definition in a package called
pack1 maps 'pack2' to subpackage 'subpack2') then every module in package
pack1 can require a module in that subpackage 'pack2' by using
require('pack2/path/to/module') or did I misunderstand this? Or is this a
server thing ?
> > I don't like this intertwining of plain package standards and Package
> > Manager/Catalogs, my feelings are that Package managers and catalogs
> > need to be another standards layer on top of a defined packages
> > definition, just
> > like packages are using the standard module layer. I.e. Modules
> > Layer->Packages Layer->Package Managers/Catalogs Layer->Anything on
> > top of that..
>
> Right. The comment about catalog authors was meant to illustrate that
> additional layer. These concerns must be separate and that was an
> important goal behind the current package and mappings specs. Many
> traditional package formats conjoin these aspects leading to
> undesirable
> consequences. This is one area we hope to do better than prior art.
>
> The package, mappings and catalog specs must and will exist separately
> based on the requirements put forth by their respective areas of
> concern.
Ok, nice to hear this.. I think I need to read all of those 2000+ msgs in
the group to understand all of those past discussions and conclusions... :-(
> The UID property makes sense even when only considering the package
> itself. It allows for example for providing a collection service that
> anybody can publish bug reports to that get relayed to the original
> package author. I am using the UID property in numerous ways in the
> toolchain I am building.
Ah, I just read your referred github link and I now understand how you want
to make the UID unique !! And I like the URL approach for the UID just like
identifying identities with OpenId etc... As it looks now I will implement
this UID thingy and try it..
Regards,
Sander
Christoph Dorn
>> You can read about my approach here [1]. Note that it uses require()
>> with two arguments vs using the first term of the ID to map to the
>> package. I will be changing this to comply with mappings/C.
>
> Thanks for the info. What you describe will probably be an answer to one of
> the questions of one of my customers so I will be very interested in the
> endresult !!
I look forward to your feedback when the time comes.
>> Your implementation can of course provide a function to add a mapping
>> dynamically and we can keep the wording in the spec open for this
>> although I don't think it should be encouraged.
>
> I totally agree with you therefore my initial (because I didn't understand
> you) reaction... ;-) The thing I'm worried about though is that if it is
> allowed to add a package mapping after the package is loaded people will use
> it mainstream for other things than the intended purpose (in your case
> debugging etc..)
I don't update the mappings of my packages during runtime. I use the
canonical ID to require the module from a package which is typically
accessed via mappings. This works if you know how your loader constructs
its canonical IDs.
The mappings specify the dependency tree of a program and should be
frozen once published (other than updating the archives/catalog each
mapping points to). You would never add or remove a mapping beyond what
is published with the original source.
If you agree and we can get consensus I think this should be specified
after all.
>> Right, but this aspect has not yet been specified. I think we should
>> add some wording in this regard once we have identified the best point of
>> entry. require() cannot be used for this as it requires a string
>> literal as ID. Maybe something like:
>>
>> require.pkg({"archive": "..."}).require("moduleID")
>>
>> This would allow the loader to verify that the requiring module has
>> authority to require the package with minimal extra effort as require()
>> is already scoped for the module.
>
> Hmm, I do not understand this.. Mappings/C already describes this or am I
> wrong ? If that's the case why do you need something like
> require.pkg({"archive": "..."}).require("moduleID") ??
> If a mapping is made (i.e. the package.json definition in a package called
> pack1 maps 'pack2' to subpackage 'subpack2') then every module in package
> pack1 can require a module in that subpackage 'pack2' by using
> require('pack2/path/to/module') or did I misunderstand this? Or is this a
> server thing ?
This is just for 'entering' a package from a system/bootstrap module for
the first time. You provide the mapping vs the canonical ID of the
package to be loader agnostic. Without such a facility each loader will
likely specify it's own way to enter packages and thus impact the
portability of programs.
> Ok, nice to hear this.. I think I need to read all of those 2000+ msgs in
> the group to understand all of those past discussions and conclusions... :-(
I listed a bunch of relevant discussions at the bottom of the link I
pointed you to. There have been a few threads since relating to
mappings/C that I have yet to add.
>> The UID property makes sense even when only considering the package
>> itself. It allows for example for providing a collection service that
>> anybody can publish bug reports to that get relayed to the original
>> package author. I am using the UID property in numerous ways in the
>> toolchain I am building.
>
> Ah, I just read your referred github link and I now understand how you want
> to make the UID unique !! And I like the URL approach for the UID just like
> identifying identities with OpenId etc... As it looks now I will implement
> this UID thingy and try it..
Let me know how it works out.
Christoph