> I don't see the need to modify Rhino or add an XHR module for
> server-side use. In fact, I am not particularly a fan of the systems
> that try to simulate the client on the server. I am all for code
> sharing but simulating an XHR object so server-side code can load
> other server-side code seems like one level of abstraction too high
> for me. I'm a simpleton.
>
> I would think that there could just be two different "require"
> functions. One implemented for the server and one for the client. The
> server one uses what I have in Rhino and the client one uses XHR.
I agree completely. I think we need to have "require" and "include"
with all of the same module API, including "log", "error", "warn",
"foreignModuleBind", "moduleUrl", and "modulesUrl". I don't think
this would be hard, and if you provided a simple API for grabbing the
text of a script instead of simply loading it, I could easily
implement this in JavaScript land for you. Alternately, if you could
implement "getModule" instead of "require", the "include" or "require"
function could be written for both client and server in terms of
"getModule". Please disregard the implication that we need an exact
XHR emulation layer. The main idea is that I think we can provide
insular modularity with stronger primitives out of the box, easily and
simply.
> By the way, I would probably not use an XHR to load JavaScript into a
> client. There are at least two reasons. I'm willing to be corrected on
> any of these things.
>
> The first is caching. I'm not confident that all browsers take
> advantage of caching the response to an XHR.
All browsers do HTTP-level cache control and abstract it to the XHR
API. The only caveat is that Konqueror neglected for a time to change
the cache code 304 to 200 to provide a completely transparent cache
hit. I fix this in my code.
> The second is having a client-side "require" function encourages
> loading bits of JavaScript as needed. This encourages more
> client-server requests. More requests is something that I want to
> avoid. If the lazy loading of scripts all occur in the first, say, 30
> seconds, then Apache pipelining, for example, makes these many
> requests cheaper (but still not free.) However if these files are
> loaded as the user interacts over a long period of time then each
> little request will open a new connection which is expensive.
Lazy loading is actually beneficial for certain kinds of
optimizations; I use this to guarantee fast load of a splash page for
example, and then proceed to add particular behaviors when the user
demands them.
Lazy loading, as you mention, can be detrimental to performance if
you're performing lots of HTTP requests to serve very small modules
(made small by necessity of organization, rather than for
performance). When I'm developing for JavaScript, I just suck it up
and tolerate the performance loss (which isn't much) because it
provides fine grain line numbers and file names for thrown errors.
However, Chiron has an optional build step for taking a body of
JavaScript modules and condensing them into a single file, observing
all of that module's dependencies by tracking include and require
calls. After that, I pass the file through YUICompressor and gzip.
After passing through those three tools, I get a single, small HTTP
request with a lot of functionality. Particularly, having all of the
code in one gzip stream gets good performance out of the LZW
algorithm, and it's efficacy furthermore increases because
YUICompressor turns unrelated names into a short list of names that
are even more patternous and thus compressible.
> I have also never had a situation where one of my page loads takes
> long even though I load all my scripts in the <head> of the document.
> I write small JavaScript, use a build process to include only what is
> necessary, minify and gzip. The total size of the JavaScript never
> turns out to be even the size of a medium sized jpg. This is the
> easiest system on the server and when a user interacts with the page
> there is never the need to wait for some JavaScript to load. It is
> already there.
I think we'll find that liberating ourselves from the need for a small
client-side code base when writing server code will be a major boon.
I'm in favor of writing minimal code, but that goal is best served by
a big tight-knit framework that simplifies the interfaces to lots of
lower-level code.
> All that said, I don't think the idea of lazy loading JavaScript into
> the browser is a very bad idea. I just haven't encountered a situation
> so complex that the page load time is a problem due to the JavaScript
> loading.
>
> There is at least one other library that does use XHR to load scripts bit-by-bit
>
> http://openjsan.org/doc/c/cw/cwest/JSAN/0.10/lib/JSAN.html
>
There's also Dojo.
> I spent about half an hour poking around there last night. It does
> seem like some of the code could be shared both client-server and
> chiron-xjs.
>
> Much of your code is client-side. What I still haven't figured out is
> how to distribute (distribute to developers that is and where to put
> it on the server) code that will be shared client-server. It is a
> slightly strange problem. Security is an issue and I use a build
> process so that all has to be worked together. I haven't had a clear
> vision yet because I've never had the opportunity to share code
> client-server before.
I think that boot.js, base.js and boost.js would be the most useful in
a server side scripting environment. It's true that most everything
else is browser-bound. I think it would also be good to make the
build step optional, if possible, for native JavaScript only modules.
Perhaps we can knock a tier of organization out of the library source.
>
>> And my documentation for those two functions:
>> http://modulesjs.com/nightly/build/doc/glossary.html#require
>
> You write "returns or passes a Module object for the JavaScript file
> at a given module URL". That is confusing to me. For me, "returns" is
> what a function returns. "passes" is when I send to a function. Is the
> "or passes" part unnecessary? Would the following be more clear?
I should be more clear by breaking this doc up into two parts.
"require" supports "optional continuation passing". That is to say,
that when you call require like this:
var http = require('http.js');
It "blocks" (makes a synchronous call) and returns the HTTP module
object when it's ready. If you call require like this:
require('http.js', function (http) {
});
It's an asynchronous form where require passes what would be retuned
as an argument to a continuation. I'll have to rewrite that
particular bit of documentation to be more lucid. This is a pattern
in Chiron. http.request and its friends all support the same notion
of optional continuation passing. I even have a continuation passing
style decorator so that any function can support CPS, including error
passing.
>
> Your require is not like the Ruby, Perl or xjs require which just
> evaluate code in the global scope. I think your require could more
> descriptively be called "getModuleObject" if that name was not so
> long.
Ruby, Perl, Python, and PHP all have more complicated scope
arrangements than JavaScript or C. I agree that Chiron "require" is
not the same as these languages, but all of those languages do
distinguish global and package or module scope. However, to a casual
observer, there might appear to be the same. JavaScript does not
natively support a module scope. In C, you have a "module scope" by
virtue of the "static" keyword used in "global scope" that reduces the
scope of a term to a single file after the preprocessor has merged in
all your headers. Perl has package scope, which is just a better way
to organize global scope explicitly. Ruby and Python both have module
scope, wherein the variables of a file are part of a corresponding
module object. The "require" function (and "import" directive in
Python) operate by copying names from the other module into the
current module (or in the case of import *, adding the other module to
the current module's scope chain), not by evaluating the other module
in the same scope as your file. Again, I decided to call this
"include". I'm yet to think of a better couple of names. I did
consider "getModule" for "require", but it's just too kludgy for such
a common function call. "import" is a reserved keyword. "use", via
Perl parlance, is available. I could use "require" instead of
"include" if we can agree on a better name than "getModule" for my
current "require".
> I think exploration of the client-server code sharing is most
> important. How code is distributed to developers and where it is
> located on the server. How it is built or required bit-by-bit by the
> client. What code can be shared and what cannot.
I don't think we should separate the module name spaces for client and
server modules because this would reduce reusability in the long run.
I think it would be acceptable to use comment directives like
/*server-only*/ and /*client-only*/ to distinguish how a client and a
server build of the library would be pared.
> Peter
Kris
[snip]
>> I would think that there could just be two different "require"
>> functions. One implemented for the server and one for the client. The
>> server one uses what I have in Rhino and the client one uses XHR.
>
> I agree completely. I think we need to have "require" and "include"
> with all of the same module API, including "log", "error", "warn",
> "foreignModuleBind", "moduleUrl", and "modulesUrl".
I don't know what all these do. Links?
> I don't think
> this would be hard, and if you provided a simple API for grabbing the
> text of a script instead of simply loading it, I could easily
> implement this in JavaScript land for you. Alternately, if you could
> implement "getModule" instead of "require", the "include" or "require"
> function could be written for both client and server in terms of
> "getModule".
What does getModule do exactly? Is this your require? If that is the
case I'm not sure that my require could be written in terms of your
require. My require allows for a very loose module structure. The
module's code is not part of a single object by necessity. A single
module can have any code in any structure. Only best practice
recommendations would indicate this is a bad idea but there are no
restrictions. A module could manipulate other global objects (i.e.
wrap global functions) etc.
[snip]
> All browsers do HTTP-level cache control and abstract it to the XHR
> API.
All browsers with XHR. This gets into tricky territory. For example,
IE6 with script enabled but ActiveX disabled would not be able to load
scripts. Now, I know this is my comp.lang.javascript client-side
perfectionism bent coming into scope. Because client-side code is a
very polarizing subject, all of these sorts of client-side
integrations would need to be in non-core xjs modules so they could be
disregarded by uptight client-side programmers.
[snip]
>> I think exploration of the client-server code sharing is most
>> important. How code is distributed to developers and where it is
>> located on the server. How it is built or required bit-by-bit by the
>> client. What code can be shared and what cannot.
>
> I don't think we should separate the module name spaces for client and
> server modules because this would reduce reusability in the long run.
agreed
> I think it would be acceptable to use comment directives like
> /*server-only*/ and /*client-only*/ to distinguish how a client and a
> server build of the library would be pared.
The modules Xmakefile.js has all the module details for building the
modules specification JSON file. One of these details could be "host"
with values something like "server" or "browser".
---
I was more addressing where the files reside on disk and how a client
request can get them.
For example, suppose a JSON module with JSON.dump and JSON.load is to
be shared. The normal location of this file on the server might be
something like
/usr/local/lib/xjs/modules/JSON/lib/JSON.js
The static files of the web application will be elsewhere. Perhaps
/home/peter/webapp/www
How does webapp make the shared JSON.js file available to the client.
It can be done with symbolic links and a build process like I
described in a recent blog article.
http://peter.michaux.ca/article/7836
The problem with this is that symbolic links on the development and
production machines may not work together.
Another option is to have a different build process that takes place
on the development machine that gets the JSON.js file and caches it in
webapp/www. The webapp/www/JSON.js file would then be added to source
control for webapp. This is a bit kludgy as the shared files in
webapp/www may be older than the files in /usr/local/lib/xjs if the
lib files are updated.
Peter
>>> I would think that there could just be two different "require"
>>> functions. One implemented for the server and one for the client. The
>>> server one uses what I have in Rhino and the client one uses XHR.
>>
>> I agree completely. I think we need to have "require" and "include"
>> with all of the same module API, including "log", "error", "warn",
>> "foreignModuleBind", "moduleUrl", and "modulesUrl".
>
> I don't know what all these do. Links?
I'm cleaning up these documents for you. I've made on pass on
improving my draft comment documentation. This is a good place to
start since it has links to all the functions I put in ModuleScope:
http://modulesjs.com/nightly/build/doc/glossary.html#ModuleScope
>
>> I don't think
>> this would be hard, and if you provided a simple API for grabbing the
>> text of a script instead of simply loading it, I could easily
>> implement this in JavaScript land for you. Alternately, if you could
>> implement "getModule" instead of "require", the "include" or "require"
>> function could be written for both client and server in terms of
>> "getModule".
>
> What does getModule do exactly? Is this your require? If that is the
> case I'm not sure that my require could be written in terms of your
> require. My require allows for a very loose module structure. The
> module's code is not part of a single object by necessity. A single
> module can have any code in any structure. Only best practice
> recommendations would indicate this is a bad idea but there are no
> restrictions. A module could manipulate other global objects (i.e.
> wrap global functions) etc.
My "include" function is no more restrictive than your "require", in
these senses. It only makes using best practices easier. Take a look
at the scope chain inside a Chiron module:
1. "global" (this is window in the client)
2. "moduleScope" (contains "require", "include", "module", "global",
items included from other modules, &c)
3. "module" (also the context object, "this", where you expose
exportable methods)
4. lambda (an enclosure so that "var" variables inside the script are
actually private to the module, instead of some other scope. This is
consistent with the behavior of "var" inside a constructor function.)
This means that you still have the ability to get and set global scope
members. Steve Levithan's regexp.js module does this.
> All browsers with XHR. This gets into tricky territory. For example,
> IE6 with script enabled but ActiveX disabled would not be able to load
> scripts. Now, I know this is my comp.lang.javascript client-side
> perfectionism bent coming into scope. Because client-side code is a
> very polarizing subject, all of these sorts of client-side
> integrations would need to be in non-core xjs modules so they could be
> disregarded by uptight client-side programmers.
I don't believe that these issues are relevant. modules.js itself is
only useful for the client. The idea is to make XJS provide the same
module environment for the server side that modules.js provides for
the client. A lot of the modules in Chiron are only relevant to the
client, in fact, but some very large and powerful ones, like base.js,
are client/server agnostic. Perhaps I should restrict the scope of my
argument to a.) providing the same interface for client and server
modules and b.) integrate only modules that are client-server
agnostic.
I guess I don't understand what distinguishes a "core XJS module" from
a peripheral one and why this distinction is important. I think my
recommendation of using a single src/ directory for module scripts and
building them into separate server/ and client/ module roots would
sufficiently cover this problem. This would involve the comment
directives I mentioned, /*server*/ and /*client*/ where modules that
contain neither would appear in both the client/ and server/ builds.
This also addresses the issue that client code needs to be compressed,
but server code does not, so the build processes for each would
benefit from decoupling.
>> I don't think we should separate the module name spaces for client and
>> server modules because this would reduce reusability in the long run.
>
> agreed
>
Sweet. I think that supports the idea of keeping one src/ dir and
bulding separate server/ and client/ module roots.
>> I think it would be acceptable to use comment directives like
>> /*server-only*/ and /*client-only*/ to distinguish how a client and a
>> server build of the library would be pared.
>
> The modules Xmakefile.js has all the module details for building the
> modules specification JSON file. One of these details could be "host"
> with values something like "server" or "browser".
I really think that's untenable in the long run. We don't keep our
documentation or licenses in separate files because we don't want them
to be out of sync with the code and because we want the authors of
modules to have self determination (not have to modify files they
don't own to make their module work). We shouldn't keep the module
dependencies anywhere but in the file that depends. That's one of the
great things about having an "include" and "require" function that
imply dependency. That information can be scraped from the code
easily to build the make-file style topological sort of dependencies.
Like, really really easily. I do that in the bundle script in my
build process.
https://cixar.com/tracs/javascript/browser/trunk/bin/build.bundle
> I was more addressing where the files reside on disk and how a client
> request can get them.
>
> For example, suppose a JSON module with JSON.dump and JSON.load is to
> be shared. The normal location of this file on the server might be
> something like
>
> /usr/local/lib/xjs/modules/JSON/lib/JSON.js
>
> The static files of the web application will be elsewhere. Perhaps
>
> /home/peter/webapp/www
>
> How does webapp make the shared JSON.js file available to the client.
> It can be done with symbolic links and a build process like I
> described in a recent blog article.
>
> http://peter.michaux.ca/article/7836
>
> The problem with this is that symbolic links on the development and
> production machines may not work together.
>
> Another option is to have a different build process that takes place
> on the development machine that gets the JSON.js file and caches it in
> webapp/www. The webapp/www/JSON.js file would then be added to source
> control for webapp. This is a bit kludgy as the shared files in
> webapp/www may be older than the files in /usr/local/lib/xjs if the
> lib files are updated.
>
Oh.
I see.
I guess we're mostly on the same page, but we need to come up with a
scheme for handling (server, client) x (production, development).
I personally don't think we should tie this to a particular server
configuration. One of the nice things about the Python module install
process is that Python knows where it is and can figure out its own
installation prefix and thereby infer the location of it's peer
modules. Maybe we ought to have something like that.
/prefix/bin/xjs
/prefix/share/xjs/production/client/
/prefix/share/xjs/production/server/
/prefix/share/xjs/development/client/
/prefix/share/xjs/development/server/
Kris
On Mon, May 19, 2008 at 5:45 PM, Kris Kowal <kris....@cixar.com> wrote:
>
> On Mon, May 19, 2008 at 9:01 AM, Peter Michaux <peterm...@gmail.com> wrote:
[snip I'll reply to this snipped content separately]
agreed
> One of the nice things about the Python module install
> process is that Python knows where it is and can figure out its own
> installation prefix and thereby infer the location of it's peer
> modules. Maybe we ought to have something like that.
Perl and Ruby also know about their own installation.
I added this to xjs a few days ago when I switched to using autoconf.
There is now a "Config" module with things like
Config.XJS_MODULES_DIR = "/usr/local/lib/xjs/modules"
> /prefix/bin/xjs
> /prefix/share/xjs/production/client/
> /prefix/share/xjs/production/server/
> /prefix/share/xjs/development/client/
> /prefix/share/xjs/development/server/
I'm not sure what you mean by this list. Just in case, I think that on
a single machine all modules should be in the same directory (e.g.
"/usr/local/lib/xjs/modules").
The symbolic link idea like I described in the article works perfectly.
ln -s /usr/local/lib/xjs/modules webapp/www/src/js/modules
The symbolic link wouldn't be needed on the production server if the
build happens on the development machine. I suppose that two
developers may need different symbolic links. If a developer happens
to have his install of modules in a different place than
/usr/local/lib/xjs/modules then he could be instructed to change
whatever symbolic link is stored in webapp/www/src/js/modules in the
version control system.
Alternately, the web server could be beefed up so it has access to
/usr/local/lib/xjs/modules automatically during development but then
how to orchestrate the build and/or security?
I don't see a completely elegant solution. They symbolic link looks
the best to me so far.
Peter
On Mon, May 19, 2008 at 5:45 PM, Kris Kowal <kris....@cixar.com> wrote:
>
> On Mon, May 19, 2008 at 9:01 AM, Peter Michaux <peterm...@gmail.com> wrote:
>> On Mon, May 19, 2008 at 1:18 AM, Kris Kowal <kris....@cixar.com> wrote:
>>> On Wed, May 14, 2008 at 6:34 PM, Peter Michaux <peterm...@gmail.com> wrote:
>
>
>>>> I would think that there could just be two different "require"
>>>> functions. One implemented for the server and one for the client. The
>>>> server one uses what I have in Rhino and the client one uses XHR.
>>>
>>> I agree completely. I think we need to have "require" and "include"
>>> with all of the same module API, including "log", "error", "warn",
>>> "foreignModuleBind", "moduleUrl", and "modulesUrl".
>>
>> I don't know what all these do. Links?
>
> I'm cleaning up these documents for you. I've made on pass on
> improving my draft comment documentation. This is a good place to
> start since it has links to all the functions I put in ModuleScope:
>
> http://modulesjs.com/nightly/build/doc/glossary.html#ModuleScope
I just read the introduction documents to your module.js and scanned
quickly through the remainder of the document.
From what I understand, the idea is clever. Very clever. Being able to
get a chunk of code an inject into some non-global scope is a neat
idea.
The main problem for me with adopting an idea like this for xjs core
is I'm not averse to using the global namespace as the central anchor
point for all modules that have been required by my require function.
In my opinion, I don't think managing the global namespace is so
difficult to need the trickery of your module.js to avoid the global
namespace completely. Just understanding the nuances of your module.js
looks like a big project to learn.
Your module uses some things I don't ever use. I am completely averse
to "with" and I'm not the only one with that opinion. It could/would
be the source of major agony. One of my favorite threads on
comp.lang.javascript that I have read shows how complex "with" can be
I'm not sharing that link because I'm proud of my contribution to that
discussion. I would have achieved about a C grade at best on that
test.
Another issue I have with these sorts of techniques in module.js is
that ES4 is coming to Rhino by the end of the year according to Steve
Yegge. That means JavaScript's own "Packages" etc will be available
for use with xjs. If they seem appropriate in some places I would
employ them. However that could put client-server code sharing in
jeopardy for five to eight years as IE6 isn't going away fast enough.
So I would be very careful about this.
I suppose it comes down to how much a particular developer thinks the
global namespace needs to be protected. I think it needs to be
protected but using psuedo namespace objects (e.g. YAHOO, FORK, etc)
is conceptually simple, unlikely to cause difficult debugging,
performs relatively well compared to some techniques, and I don't
think it is so tricky to manage up to even a couple hundred global
objects which would only happen for an extremely huge server-side
project. Java essentially has a global namespace for packages.
Developers just use very specific package names "org.mozilla", for
example.
With all that said, if I can provide enough in the xjs core to enable
you to use your module.js as an extra module then that would be great.
I'm sure it would make the core more flexible.
As I said above, I think this is all a lot to expect someone to learn
on top of all of JavaScript's features like closures and
prototype-based inheritance.
> This means that you still have the ability to get and set global scope
> members. Steve Levithan's regexp.js module does this.
>
>> All browsers with XHR. This gets into tricky territory. For example,
>> IE6 with script enabled but ActiveX disabled would not be able to load
>> scripts. Now, I know this is my comp.lang.javascript client-side
>> perfectionism bent coming into scope. Because client-side code is a
>> very polarizing subject, all of these sorts of client-side
>> integrations would need to be in non-core xjs modules so they could be
>> disregarded by uptight client-side programmers.
>
> I don't believe that these issues are relevant. modules.js itself is
> only useful for the client. The idea is to make XJS provide the same
> module environment for the server side that modules.js provides for
> the client. A lot of the modules in Chiron are only relevant to the
> client, in fact, but some very large and powerful ones, like base.js,
> are client/server agnostic. Perhaps I should restrict the scope of my
> argument to a.) providing the same interface for client and server
> modules and b.) integrate only modules that are client-server
> agnostic.
That seems like the best place to start.
> I guess I don't understand what distinguishes a "core XJS module" from
> a peripheral one and why this distinction is important.
The core ones are part of the install of xjs. The shell, xmake, JSON,
the xmod utility for installing other modules (like cpan or gem).
These are the modules about which, at the very least, I need to be
completely happy. Of course I hope others like them too because they
are the common base. The idea is to keep them small so it is easier to
find them agreeable. The core modules provide just enough
functionality to make xjs extensible and to be able to install other
modules using the xmod command line utility. These core modules are
the modules that bootstrap a developer into the xjs world of modules.
The other modules, the xmod modules on the central site, are community
written and can live and die based on their popularity. I don't want
to have any control over those other than they have enough data to be
indexed on the central site. I'll offer advice if someone asks but
that is it. So very little agreement needs to be reached in these
areas. Eventually theses modules won't even need to be discussed with
me other than to have permission to upload under a particular module
name. And eventually someone else may do that job.
So as far as our discussion goes, we only really need to discuss
issues that relate to the core modules and your interests with scoping
and making sure code can be shared client/server. Other modules are
out of my control but I'd be happy to discuss them just out of
interest, of course, when there is time for such leisure. :-)
> I think my
> recommendation of using a single src/ directory for module scripts and
> building them into separate server/ and client/ module roots would
> sufficiently cover this problem. This would involve the comment
> directives I mentioned, /*server*/ and /*client*/ where modules that
> contain neither would appear in both the client/ and server/ builds.
> This also addresses the issue that client code needs to be compressed,
> but server code does not, so the build processes for each would
> benefit from decoupling.
I don't see how separate install directories solve any of the
client-server sharing problems we've been discussing. If the code in
the client build directory is to be shared that is the same problem as
sharing code in any directory.
>>> I don't think we should separate the module name spaces for client and
>>> server modules because this would reduce reusability in the long run.
>>
>> agreed
>>
>
> Sweet. I think that supports the idea of keeping one src/ dir and
> bulding separate server/ and client/ module roots.
>
>>> I think it would be acceptable to use comment directives like
>>> /*server-only*/ and /*client-only*/ to distinguish how a client and a
>>> server build of the library would be pared.
>>
>> The modules Xmakefile.js has all the module details for building the
>> modules specification JSON file. One of these details could be "host"
>> with values something like "server" or "browser".
>
> I really think that's untenable in the long run. We don't keep our
> documentation or licenses in separate files because we don't want them
> to be out of sync with the code and because we want the authors of
> modules to have self determination (not have to modify files they
> don't own to make their module work). We shouldn't keep the module
> dependencies anywhere but in the file that depends. That's one of the
> great things about having an "include" and "require" function that
> imply dependency. That information can be scraped from the code
> easily to build the make-file style topological sort of dependencies.
> Like, really really easily. I do that in the bundle script in my
> build process.
>
> https://cixar.com/tracs/javascript/browser/trunk/bin/build.bundle
>
The specs in the Xmakefile.js can now be specified in the
documentation system in the actual code files. The two are now
essentially the same. I will write more about this and the
documentation soon.
agreed
> One of the nice things about the Python module install
> process is that Python knows where it is and can figure out its own
> installation prefix and thereby infer the location of it's peer
> modules. Maybe we ought to have something like that.
Perl and Ruby also know about their own installation.
I added this to xjs a few days ago when I switched to using autoconf.
There is now a "Config" module with things like
Config.XJS_MODULES_DIR = "/usr/local/lib/xjs/modules"
> /prefix/bin/xjs
> /prefix/share/xjs/production/client/
> /prefix/share/xjs/production/server/
> /prefix/share/xjs/development/client/
> /prefix/share/xjs/development/server/
I'm not sure what you mean by this list. Just in case, I think that on