Date: Sat, 22 Sep 2012 10:51:40 -0700 (PDT)
From: Saleem Abdul Hamid <meel...@gmail.com>
To: nodejs@googlegroups.com
Message-Id: <d233eba5-3296-4e0e-87a2-7cac73fc6f81@googlegroups.com>
In-Reply-To: <7e8ee935-598d-4c06-b820-5249196d851d@x5g2000pbl.googlegroups.com>
References: <7e8ee935-598d-4c06-b820-5249196d851d@x5g2000pbl.googlegroups.com>
Subject: Re: Request for feedback- Yet another client-side module loader
 (it's different, I promise:))
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_380_8322247.1348336301079"

------=_Part_380_8322247.1348336301079
Content-Type: multipart/alternative; 
	boundary="----=_Part_381_17146473.1348336301079"

------=_Part_381_17146473.1348336301079
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi everyone,

This is now available as Connect middleware.

On Saturday, March 24, 2012 5:04:52 PM UTC-7, Saleem Abdul Hamid wrote:
>
> tl;dr - Client-side require with a server-side component that caches=20
> dependencies, bundles them, and caches the bundles. Need feedback on=20
> the concept, syntax. Need suggestions/contributions on implementation.=20
> Although, this works for me, it is almost just a proof-of-concept,=20
> needs work.=20
>
>
> As part of a project I'm working on, I spent a few hours writing a=20
> little client-side module loader with a server-side component enabling=20
> what I think is a pretty neat meaning to CommonJS module syntax. This=20
> morning I pulled it out of the rest of my project and attempted to=20
> package it in a useful way for others to use.=20
>
> The basic idea is this- in your client-side code, you can use require=20
> in either a "synchronous" or asynchronous fashion-=20
> module1 =3D require('some/path.js');=20
> require('some/other/path.js', function(err,result){module2 =3D=20
> result;});=20
>
> An asynchronous require makes a call to the server component to get=20
> the file in question, but before returning the file, the server parses=20
> it, finds all the synchronous require calls, loads those files as well=20
> and returning the whole thing as a package. That way, when the=20
> original file that was asynchronously loaded is executed and comes to=20
> one of those synchronous require calls, that file is already there,=20
> and the require is actually synchronous.=20
>
> At this point, maybe this screencast demo will help to clarify how it=20
> works: http://screencast.com/t/nOU53BRYUAX=20
>
> Put another way:=20
> If I async require fileA, and fileA has synchronous dependencies on=20
> fileB, and fileC, and an asynchronous dependency on fileD, the server-=20
> side component will return (in a single "bundle") and keep in memory=20
> fileA, fileB, and fileC, not fileD, and it will execute fileA.=20
> The client-side also separates fetching the files and eval'ing them=20
> (the method of getting files is xhr+eval). So, let's say fileA has=20
> require('fileB'); that executes when the file is parsed and executed=20
> on the client, but require('fileC') is inside a function somewhere.=20
> Then fileA will first be eval'ed, then fileB when it comes across=20
> that, and the text of fileC will just be in memory, not eval'ed until=20
> that function is called or some other require to it is called by any=20
> other part of the program.=20
>
> Another example-=20
> fileA has dependencies fileB, fileC, fileD, fileE, fileF=20
> fileG has dependencies fileC, fileE, fileH=20
>
> When I call require('fileA', function(err,result){return 'yay';});,=20
> the module loader will load fileA, fileB, fileC, fileD, fileE, and=20
> fileF all in a single bundle.=20
> If I, after that, call require('fileG', function(err,result){return=20
> 'yay';});, the module loader will only load fileG and fileH!=20
>
> Hopefully, that's clear....=20
>
> The advantages-=20
> Being aware of the difference in synchronous and asynchronous require=20
> in your client-side code make it extremely natural to break all your=20
> client-side code into small reusable chunks- there is no penalty and=20
> you don't have to "optimize" later by deciding what to package=20
> together and what to package separately.=20
> Handling dependencies becomes nothing. You don't have to think about=20
> it.=20
> The server can have a "deployment" mode, where it caches what the=20
> dependencies of a file are and doesn't ever need to parse that file=20
> again.=20
> In "deployment" mode, the server can also cache bundles of multiple=20
> files that are requested together, so when another client requests=20
> that same bundle, it is already in memory.=20
>
> To sum up:=20
> xhr+eval-when-necessary client-side module loader=20
> both synchronous-ish and asynchronous require in your client side-code=20
> --the synchronous require is actually a command to the server-side=20
> component to bundle=20
> server-side component=20
> --parses for dependencies and bundles them together=20
> --can cache dependency parsing results and whole bundles=20
>
>
> So- thoughts? Is this a horrible idea? Are there some gotchas that I'm=20
> missing?=20
>
> Specific advice needed-=20
> =E2=80=A2 How to package this in a way that it can be easily used in othe=
r=20
> projects? How can I make it integrate seamlessly with existing servers=20
> and make it compatible with different transport mechanisms?=20
> =E2=80=A2 How to handle path resolution?=20
> =E2=80=A2 Suggestions for licensing?=20
> =E2=80=A2 Suggestions for a name- (Mundlejs is a portmanteau of Module an=
d=20
> Bundle- didn't really think long about it)=20
>
> Things that need to be (properly)implemented:=20
> =E2=80=A2 server-side "parsing" is just a brittle regexp right now:=20
> (line.match /require\('(.*)'\)/)=20
> =E2=80=A2 neither type of server-side caching is implemented (pretty easy=
 to=20
> do)=20
> =E2=80=A2 uniquely identify clients and keep the server away of what modu=
les=20
> they already have, so we can just send the diff of cached modules-=20
> currently, I'm sending the entire list of already cached modules with=20
> every xhr call, so the server doesn't load a dependency twice.=20
> =E2=80=A2 proper compatibility with module specifications (i.e. CommonJS)=
-=20
> right now, it's just require and module.exports=20
>
>
> Code is available here: https://github.com/meelash/Mundlejs=20
> To test it:=20
> from Mundlejs/tests/, run=20
> node server.js=20
> visit http://127.0.0.1:1337/ and open your browser console.


------=_Part_381_17146473.1348336301079
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi everyone,<div><br></div><div>This is now available as Connect middleware=
.<br><br>On Saturday, March 24, 2012 5:04:52 PM UTC-7, Saleem Abdul Hamid w=
rote:<blockquote class=3D"gmail_quote" style=3D"margin: 0;margin-left: 0.8e=
x;border-left: 1px #ccc solid;padding-left: 1ex;">tl;dr - Client-side requi=
re with a server-side component that caches
<br>dependencies, bundles them, and caches the bundles. Need feedback on
<br>the concept, syntax. Need suggestions/contributions on implementation.
<br>Although, this works for me, it is almost just a proof-of-concept,
<br>needs work.
<br>
<br>
<br>As part of a project I'm working on, I spent a few hours writing a
<br>little client-side module loader with a server-side component enabling
<br>what I think is a pretty neat meaning to CommonJS module syntax. This
<br>morning I pulled it out of the rest of my project and attempted to
<br>package it in a useful way for others to use.
<br>
<br>The basic idea is this- in your client-side code, you can use require
<br>in either a "synchronous" or asynchronous fashion-
<br>module1 =3D require('some/path.js');
<br>require('some/other/path.js', function(err,result){module2 =3D
<br>result;});
<br>
<br>An asynchronous require makes a call to the server component to get
<br>the file in question, but before returning the file, the server parses
<br>it, finds all the synchronous require calls, loads those files as well
<br>and returning the whole thing as a package. That way, when the
<br>original file that was asynchronously loaded is executed and comes to
<br>one of those synchronous require calls, that file is already there,
<br>and the require is actually synchronous.
<br>
<br>At this point, maybe this screencast demo will help to clarify how it
<br>works: <a href=3D"http://screencast.com/t/nOU53BRYUAX" target=3D"_blank=
">http://screencast.com/t/<wbr>nOU53BRYUAX</a>
<br>
<br>Put another way:
<br>If I async require fileA, and fileA has synchronous dependencies on
<br>fileB, and fileC, and an asynchronous dependency on fileD, the server-
<br>side component will return (in a single "bundle") and keep in memory
<br>fileA, fileB, and fileC, not fileD, and it will execute fileA.
<br>The client-side also separates fetching the files and eval'ing them
<br>(the method of getting files is xhr+eval). So, let's say fileA has
<br>require('fileB'); that executes when the file is parsed and executed
<br>on the client, but require('fileC') is inside a function somewhere.
<br>Then fileA will first be eval'ed, then fileB when it comes across
<br>that, and the text of fileC will just be in memory, not eval'ed until
<br>that function is called or some other require to it is called by any
<br>other part of the program.
<br>
<br>Another example-
<br>fileA has dependencies fileB, fileC, fileD, fileE, fileF
<br>fileG has dependencies fileC, fileE, fileH
<br>
<br>When I call require('fileA', function(err,result){return 'yay';});,
<br>the module loader will load fileA, fileB, fileC, fileD, fileE, and
<br>fileF all in a single bundle.
<br>If I, after that, call require('fileG', function(err,result){return
<br>'yay';});, the module loader will only load fileG and fileH!
<br>
<br>Hopefully, that's clear....
<br>
<br>The advantages-
<br>Being aware of the difference in synchronous and asynchronous require
<br>in your client-side code make it extremely natural to break all your
<br>client-side code into small reusable chunks- there is no penalty and
<br>you don't have to "optimize" later by deciding what to package
<br>together and what to package separately.
<br>Handling dependencies becomes nothing. You don't have to think about
<br>it.
<br>The server can have a "deployment" mode, where it caches what the
<br>dependencies of a file are and doesn't ever need to parse that file
<br>again.
<br>In "deployment" mode, the server can also cache bundles of multiple
<br>files that are requested together, so when another client requests
<br>that same bundle, it is already in memory.
<br>
<br>To sum up:
<br>xhr+eval-when-necessary client-side module loader
<br>both synchronous-ish and asynchronous require in your client side-code
<br>--the synchronous require is actually a command to the server-side
<br>component to bundle
<br>server-side component
<br>--parses for dependencies and bundles them together
<br>--can cache dependency parsing results and whole bundles
<br>
<br>
<br>So- thoughts? Is this a horrible idea? Are there some gotchas that I'm
<br>missing?
<br>
<br>Specific advice needed-
<br>=E2=80=A2 How to package this in a way that it can be easily used in ot=
her
<br>projects? How can I make it integrate seamlessly with existing servers
<br>and make it compatible with different transport mechanisms?
<br>=E2=80=A2 How to handle path resolution?
<br>=E2=80=A2 Suggestions for licensing?
<br>=E2=80=A2 Suggestions for a name- (Mundlejs is a portmanteau of Module =
and
<br>Bundle- didn't really think long about it)
<br>
<br>Things that need to be (properly)implemented:
<br>=E2=80=A2 server-side "parsing" is just a brittle regexp right now:
<br>(line.match /require\('(.*)'\)/)
<br>=E2=80=A2 neither type of server-side caching is implemented (pretty ea=
sy to
<br>do)
<br>=E2=80=A2 uniquely identify clients and keep the server away of what mo=
dules
<br>they already have, so we can just send the diff of cached modules-
<br>currently, I'm sending the entire list of already cached modules with
<br>every xhr call, so the server doesn't load a dependency twice.
<br>=E2=80=A2 proper compatibility with module specifications (i.e. CommonJ=
S)-
<br>right now, it's just require and module.exports
<br>
<br>
<br>Code is available here: <a href=3D"https://github.com/meelash/Mundlejs"=
 target=3D"_blank">https://github.com/meelash/<wbr>Mundlejs</a>
<br>To test it:
<br>from Mundlejs/tests/, run
<br>node server.js
<br>visit <a href=3D"http://127.0.0.1:1337/" target=3D"_blank">http://127.0=
.0.1:1337/</a> and open your browser console.</blockquote></div>
------=_Part_381_17146473.1348336301079--

------=_Part_380_8322247.1348336301079--