Yes, Aptana's timing is unfortunate.  While I was at Apple, a startup
pulled my arm for a whole year, I started here in January, and have
been quite happy.  I would have been happy to take the position under
different circumstances, and it sounded like your kind of project. I'm
sorry it's bad timing for you too.

I've been looking over GoogJSL and XJS for "the good bits" for the
last couple days.  I think that XJS could gain a lot from using some
of the modules in my library, Chiron, like test.js (which is like your
assert.js, but a bit more comprehensive).  Really the best way for us
to share code is for me modify Rhino (add an XHR-like system for
fetching local scripts) and modules.js so that they can play well
together (provide the same environment to module scripts).  With the
API changes I recommended, using modules.js client side would be
nearly identical to server side, except that you'd have access to the
Java integration in XJS, which is quite powerful.

In the mean time, the smallest change with the largest impact would be
for either me to rename my "include" function to "require", or you to
rename your "require" function to "include".  The trouble for me is
that I have both "require" and "include" functions.  I've had to
struggle with this nomenclature problem quite a bit and never became
truly satisfied with my choice.  My "require" function just grabs,
evaluates, and returns a module.  This has no impact on the global
scope, which for your system, is synonymous with module scope.
"include", on the other hand, requires the module, then copies the
exported values into module scope, which again, is synonymous with
global scope.  Since global scope and module scope are the same for
you, your "require" function behaves pretty much the same way as
Chiron's "include" (with a lot of caveats that I'm hoping you'll
appreciate in time).  I don't feel that the English meanings of
"include" and "require" make this distinction inherently clear, but I
definitely don't think that reversing them is any clearer.  "require"
does, after all, imply blocking pending dependency resolution, and
"include" does, after all, bring another module into the current
module.  I know this is a moot distinction for your lib, but you'll
have to tell me what it's worth to you.

Have you read any of Chiron?  It's at
http://cixar.com/tracs/javascript/browser/trunk/src

And my documentation for those two functions:
http://modulesjs.com/nightly/build/doc/glossary.html#require
http://modulesjs.com/nightly/build/doc/glossary.html#include

Let me know what steps you think we should take.

Kris

Hi Kris,

Thanks for the reply.

Yes. My modules are very spartan. I have been focused on sketching out
the way modules interact from the shell module all the way up through
utility modules, shell scripts, and a web app. Certainly things need
beefing up once the major APIs are relatively stable.

I don't see the need to modify Rhino or add an XHR module for
server-side use. In fact, I am not particularly a fan of the systems
that try to simulate the client on the server. I am all for code
sharing but simulating an XHR object so server-side code can load
other server-side code seems like one level of abstraction too high
for me. I'm a simpleton.

I would think that there could just be two different "require"
functions. One implemented for the server and one for the client. The
server one uses what I have in Rhino and the client one uses XHR.

By the way, I would probably not use an XHR to load JavaScript into a
client. There are at least two reasons. I'm willing to be corrected on
any of these things.

The first is caching. I'm not confident that all browsers take
advantage of caching the response to an XHR. That alone would not go
over well with the sysadmin where I work. I could be wrong about this
but using dynamic script element insertion does seem to take advantage
browser caching. By caching I don't mean how your require controls
caching. I mean actual browser caching between page loads.

The second is having a client-side "require" function encourages
loading bits of JavaScript as needed. This encourages more
client-server requests. More requests is something that I want to
avoid. If the lazy loading of scripts all occur in the first, say, 30
seconds, then Apache pipelining, for example, makes these many
requests cheaper (but still not free.) However if these files are
loaded as the user interacts over a long period of time then each
little request will open a new connection which is expensive.

I have also never had a situation where one of my page loads takes
long even though I load all my scripts in the <head> of the document.
I write small JavaScript, use a build process to include only what is
necessary, minify and gzip. The total size of the JavaScript never
turns out to be even the size of a medium sized jpg. This is the
easiest system on the server and when a user interacts with the page
there is never the need to wait for some JavaScript to load. It is
already there.

What I do use XHR for is sending JSON data to and from the server that
needs to persist on the database etc.

All that said, I don't think the idea of lazy loading JavaScript into
the browser is a very bad idea. I just haven't encountered a situation
so complex that the page load time is a problem due to the JavaScript
loading.

There is at least one other library that does use XHR to load scripts bit-by-bit

http://openjsan.org/doc/c/cw/cwest/JSAN/0.10/lib/JSAN.html

I agree that all of these words are not particularly distinct. The
reason I choose "load" and "require" are because they seem to be
relatively popular the way I use them. My load and require are the
same as Ruby's. My load is like Scheme's load. My require is like
Perl's require and Emacs Lisp require.

Really better names for my functions would be

load -> globalEvalFile
require -> globalEvalFileIfNotDoneBefore

I'd rather stay with the precedents set by Perl, Ruby, Scheme and
Emacs Lisp. They've been quite successful and are well known by many
that may happen come to xjs.

I spent about half an hour poking around there last night. It does
seem like some of the code could be shared both client-server and
chiron-xjs.

Much of your code is client-side. What I still haven't figured out is
how to distribute (distribute to developers that is and where to put
it on the server) code that will be shared client-server. It is a
slightly strange problem. Security is an issue and I use a build
process so that all has to be worked together. I haven't had a clear
vision yet because I've never had the opportunity to share code
client-server before.

You write "returns or passes a Module object for the JavaScript file
at a given module URL". That is confusing to me. For me, "returns" is
what a function returns. "passes" is when I send to a function. Is the
"or passes" part unnecessary? Would the following be more clear?

"returns the Module object defined in the JavaScript file at a given URL"

Your require is not like the Ruby, Perl or xjs require which just
evaluate code in the global scope. I think your require could more
descriptively be called "getModuleObject" if that name was not so
long.

It looks like your include is similar to Ruby's include which is a way
of doing a mixin.

I think exploration of the client-server code sharing is most
important. How code is distributed to developers and where it is
located on the server. How it is built or required bit-by-bit by the
client. What code can be shared and what cannot.

Peter

I agree completely.  I think we need to have "require" and "include"
with all of the same module API, including "log", "error", "warn",
"foreignModuleBind", "moduleUrl", and "modulesUrl".  I don't think
this would be hard, and if you provided a simple API for grabbing the
text of a script instead of simply loading it, I could easily
implement this in JavaScript land for you.  Alternately, if you could
implement "getModule" instead of "require", the "include" or "require"
function could be written for both client and server in terms of
"getModule".  Please disregard the implication that we need an exact
XHR emulation layer.  The main idea is that I think we can provide
insular modularity with stronger primitives out of the box, easily and
simply.

All browsers do HTTP-level cache control and abstract it to the XHR
API.  The only caveat is that Konqueror neglected for a time to change
the cache code 304 to 200 to provide a completely transparent cache
hit.  I fix this in my code.

Lazy loading is actually beneficial for certain kinds of
optimizations; I use this to guarantee fast load of a splash page for
example, and then proceed to add particular behaviors when the user
demands them.

Lazy loading, as you mention, can be detrimental to performance if
you're performing lots of HTTP requests to serve very small modules
(made small by necessity of organization, rather than for
performance).  When I'm developing for JavaScript, I just suck it up
and tolerate the performance loss (which isn't much) because it
provides fine grain line numbers and file names for thrown errors.
However, Chiron has an optional build step for taking a body of
JavaScript modules and condensing them into a single file, observing
all of that module's dependencies by tracking include and require
calls.  After that, I pass the file through YUICompressor and gzip.
After passing through those three tools, I get a single, small HTTP
request with a lot of functionality.  Particularly, having all of the
code in one gzip stream gets good performance out of the LZW
algorithm, and it's efficacy furthermore increases because
YUICompressor turns unrelated names into a short list of names that
are even more patternous and thus compressible.

I think we'll find that liberating ourselves from the need for a small
client-side code base when writing server code will be a major boon.
I'm in favor of writing minimal code, but that goal is best served by
a big tight-knit framework that simplifies the interfaces to lots of
lower-level code.

There's also Dojo.

I think that boot.js, base.js and boost.js would be the most useful in
a server side scripting environment.  It's true that most everything
else is browser-bound.  I think it would also be good to make the
build step optional, if possible, for native JavaScript only modules.
Perhaps we can knock a tier of organization out of the library source.

I should be more clear by breaking this doc up into two parts.
"require" supports "optional continuation passing".  That is to say,
that when you call require like this:

var http = require('http.js');

It "blocks" (makes a synchronous call) and returns the HTTP module
object when it's ready.  If you call require like this:

require('http.js', function (http) {

It's an asynchronous form where require passes what would be retuned
as an argument to a continuation.  I'll have to rewrite that
particular bit of documentation to be more lucid.  This is a pattern
in Chiron.  http.request and its friends all support the same notion
of optional continuation passing.  I even have a continuation passing
style decorator so that any function can support CPS, including error
passing.

Ruby, Perl, Python, and PHP all have more complicated scope
arrangements than JavaScript or C.  I agree that Chiron "require" is
not the same as these languages, but all of those languages do
distinguish global and package or module scope.  However, to a casual
observer, there might appear to be the same.  JavaScript does not
natively support a module scope.  In C, you have a "module scope" by
virtue of the "static" keyword used in "global scope" that reduces the
scope of a term to a single file after the preprocessor has merged in
all your headers.  Perl has package scope, which is just a better way
to organize global scope explicitly.  Ruby and Python both have module
scope, wherein the variables of a file are part of a corresponding
module object.  The "require" function (and "import" directive in
Python) operate by copying names from the other module into the
current module (or in the case of import *, adding the other module to
the current module's scope chain), not by evaluating the other module
in the same scope as your file.  Again, I decided to call this
"include".  I'm yet to think of a better couple of names.  I did
consider "getModule" for "require", but it's just too kludgy for such
a common function call.  "import" is a reserved keyword.  "use", via
Perl parlance, is available.  I could use "require" instead of
"include" if we can agree on a better name than "getModule" for my
current "require".

I don't think we should separate the module name spaces for client and
server modules because this would reduce reusability in the long run.
I think it would be acceptable to use comment directives like
/*server-only*/ and /*client-only*/ to distinguish how a client and a
server build of the library would be pared.

Kris

[snip]

I don't know what all these do. Links?

What does getModule do exactly? Is this your require? If that is the
case I'm not sure that my require could be written in terms of your
require. My require allows for a very loose module structure. The
module's code is not part of a single object by necessity. A single
module can have any code in any structure. Only best practice
recommendations would indicate this is a bad idea but there are no
restrictions. A module could manipulate other global objects (i.e.
wrap global functions) etc.

[snip]

All browsers with XHR. This gets into tricky territory. For example,
IE6 with script enabled but ActiveX disabled would not be able to load
scripts. Now, I know this is my comp.lang.javascript client-side
perfectionism bent coming into scope. Because client-side code is a
very polarizing subject, all of these sorts of client-side
integrations would need to be in non-core xjs modules so they could be
disregarded by uptight client-side programmers.

[snip]

agreed

The modules Xmakefile.js has all the module details for building the
modules specification JSON file. One of these details could be "host"
with values something like "server" or "browser".

---

I was more addressing where the files reside on disk and how a client
request can get them.

For example, suppose a JSON module with JSON.dump and JSON.load is to
be shared. The normal location of this file on the server might be
something like

/usr/local/lib/xjs/modules/JSON/lib/JSON.js

The static files of the web application will be elsewhere. Perhaps

/home/peter/webapp/www

How does webapp make the shared JSON.js file available to the client.
It can be done with symbolic links and a build process like I
described in a recent blog article.

http://peter.michaux.ca/article/7836

The problem with this is that symbolic links on the development and
production machines may not work together.

Another option is to have a different build process that takes place
on the development machine that gets the JSON.js file and caches it in
webapp/www. The webapp/www/JSON.js file would then be added to source
control for webapp. This is a bit kludgy as the shared files in
webapp/www may be older than the files in /usr/local/lib/xjs if the
lib files are updated.

Peter

I'm cleaning up these documents for you.  I've made on pass on
improving my draft comment documentation.  This is a good place to
start since it has links to all the functions I put in ModuleScope:

http://modulesjs.com/nightly/build/doc/glossary.html#ModuleScope

My "include" function is no more restrictive than your "require", in
these senses.  It only makes using best practices easier.  Take a look
at the scope chain inside a Chiron module:

1. "global" (this is window in the client)
2. "moduleScope" (contains "require", "include", "module", "global",
items included from other modules, &c)
3. "module" (also the context object, "this", where you expose
exportable methods)
4. lambda (an enclosure so that "var" variables inside the script are
actually private to the module, instead of some other scope.  This is
consistent with the behavior of "var" inside a constructor function.)

This means that you still have the ability to get and set global scope
members.  Steve Levithan's regexp.js module does this.

I don't believe that these issues are relevant.  modules.js itself is
only useful for the client.  The idea is to make XJS provide the same
module environment for the server side that modules.js provides for
the client.  A lot of the modules in Chiron are only relevant to the
client, in fact, but some very large and powerful ones, like base.js,
are client/server agnostic.  Perhaps I should restrict the scope of my
argument to a.) providing the same interface for client and server
modules and b.) integrate only modules that are client-server
agnostic.

I guess I don't understand what distinguishes a "core XJS module" from
a peripheral one and why this distinction is important.  I think my
recommendation of using a single src/ directory for module scripts and
building them into separate server/ and client/ module roots would
sufficiently cover this problem.  This would involve the comment
directives I mentioned, /*server*/ and /*client*/ where modules that
contain neither would appear in both the client/ and server/ builds.
This also addresses the issue that client code needs to be compressed,
but server code does not, so the build processes for each would
benefit from decoupling.

Sweet.  I think that supports the idea of keeping one src/ dir and
bulding separate server/ and client/ module roots.

I really think that's untenable in the long run.  We don't keep our
documentation or licenses in separate files because we don't want them
to be out of sync with the code and because we want the authors of
modules to have self determination (not have to modify files they
don't own to make their module work).  We shouldn't keep the module
dependencies anywhere but in the file that depends.  That's one of the
great things about having an "include" and "require" function that
imply dependency.  That information can be scraped from the code
easily to build the make-file style topological sort of dependencies.
Like, really really easily.  I do that in the bundle script in my
build process.

https://cixar.com/tracs/javascript/browser/trunk/bin/build.bundle

Oh.

I see.

I guess we're mostly on the same page, but we need to come up with a
scheme for handling (server, client) x (production, development).

I personally don't think we should tie this to a particular server
configuration.  One of the nice things about the Python module install
process is that Python knows where it is and can figure out its own
installation prefix and thereby infer the location of it's peer
modules.  Maybe we ought to have something like that.

/prefix/bin/xjs
/prefix/share/xjs/production/client/
/prefix/share/xjs/production/server/
/prefix/share/xjs/development/client/
/prefix/share/xjs/development/server/

Kris

Hi Kris,

I just read the introduction documents to your module.js and scanned
quickly through the remainder of the document.

From what I understand, the idea is clever. Very clever. Being able to
get a chunk of code an inject into some non-global scope is a neat
idea.

The main problem for me with adopting an idea like this for xjs core
is I'm not averse to using the global namespace as the central anchor
point for all modules that have been required by my require function.
In my opinion, I don't think managing the global namespace is so
difficult to need the trickery of your module.js to avoid the global
namespace completely. Just understanding the nuances of your module.js
looks like a big project to learn.

Your module uses some things I don't ever use. I am completely averse
to "with" and I'm not the only one with that opinion. It could/would
be the source of major agony. One of my favorite threads on
comp.lang.javascript that I have read shows how complex "with" can be

http://groups.google.com/group/comp.lang.javascript/browse_frm/thread...

I'm not sharing that link because I'm proud of my contribution to that
discussion. I would have achieved about a C grade at best on that
test.

Another issue I have with these sorts of techniques in module.js is
that ES4 is coming to Rhino by the end of the year according to Steve
Yegge. That means JavaScript's own "Packages" etc will be available
for use with xjs. If they seem appropriate in some places I would
employ them. However that could put client-server code sharing in
jeopardy for five to eight years as IE6 isn't going away fast enough.
So I would be very careful about this.

I suppose it comes down to how much a particular developer thinks the
global namespace needs to be protected. I think it needs to be
protected but using psuedo namespace objects (e.g. YAHOO, FORK, etc)
is conceptually simple, unlikely to cause difficult debugging,
performs relatively well compared to some techniques, and I don't
think it is so tricky to manage up to even a couple hundred global
objects which would only happen for an extremely huge server-side
project. Java essentially has a global namespace for packages.
Developers just use very specific package names "org.mozilla", for
example.

With all that said, if I can provide enough in the xjs core to enable
you to use your module.js as an extra module then that would be great.
I'm sure it would make the core more flexible.

As I said above, I think this is all a lot to expect someone to learn
on top of all of JavaScript's features like closures and
prototype-based inheritance.

That seems like the best place to start.

The core ones are part of the install of xjs. The shell, xmake, JSON,
the xmod utility for installing other modules (like cpan or gem).
These are the modules about which, at the very least, I need to be
completely happy. Of course I hope others like them too because they
are the common base. The idea is to keep them small so it is easier to
find them agreeable. The core modules provide just enough
functionality to make xjs extensible and to be able to install other
modules using the xmod command line utility. These core modules are
the modules that bootstrap a developer into the xjs world of modules.

The other modules, the xmod modules on the central site, are community
written and can live and die based on their popularity. I don't want
to have any control over those other than they have enough data to be
indexed on the central site. I'll offer advice if someone asks but
that is it. So very little agreement needs to be reached in these
areas. Eventually theses modules won't even need to be discussed with
me other than to have permission to upload under a particular module
name. And eventually someone else may do that job.

So as far as our discussion goes, we only really need to discuss
issues that relate to the core modules and your interests with scoping
and making sure code can be shared client/server. Other modules are
out of my control but I'd be happy to discuss them just out of
interest, of course, when there is time for such leisure. :-)

I don't see how separate install directories solve any of the
client-server sharing problems we've been discussing. If the code in
the client build directory is to be shared that is the same problem as
sharing code in any directory.

...

read more »