Grouped dependencies loading

[Nickolay Platonov]

Hello all,

I've finished the adaptation of grouped dependencies loading and its now usable with local js library.
Below are the summaries. Note, that the received results are currently applicable only for Joose classes and modules.

Grouped dependencies loading.

Implementation conditions

Firstly, let me shortly present the grouped loading scheme. It was designed to implement the dependencies loading
for javascript, which will be suitable for using in production.

So it should be a) asynchronous b) should use as little http requests as possible.

Presented scheme loads any module with any number of dependencies (in-depth) with 2 http requests.
Implementation requires the server-side component - librarian.

Algorithm (non-formal)

Source data:

Module1 uses Module2, Module3
Module2 uses Module4, Module5
Module3 uses Module5
Module4 uses Module6

This dependencies information is reflected in sources

1. We need to determine the full list of "in-depth dependencies" for each module and modify its sources with computed list

After 1st step we'll receive following dependencies graph:
Module1 uses Module2, Module3, Module4, Module5, Module6
Module2 uses Module4, Module5, Module6
Module3 uses Module5
Module4 uses Module6

This step should be done statically - for example during installation of the module in the library.

2. Lets say we need to load Module1 dynamically.

2.1. There is no prior knowledge about its dependencies, so we use the 1st http request to load Module1 itself

2.2 After loading Module1 we have full info about its dependencies - [Module2, Module3, Module4, Module5, Module6] list

2.3 Lets assume, that Module5 is already loaded, so we exclude Module5 from that list: [Module2, Module3, Module4, Module6] 

2.4 We stringifying the dependencies list, and computing its MD5 hash, let it be ABC123

2.5 Then we performing 2nd request, on the url:
/libmanager/ABC123.js
with the header "X-Joose-use" = [Module2, Module3, Module4, Module6]

2.6 If the librarian already has ABC123.js file ready - its serves it as static file. 

2.7 Otherwise - librarian builds a new group, based on "X-Joose-use" header's content and saves it into ABC123.js

As an addition, during stringification, the version information is appended to each module


Implementation.

Currently, the scheme above is implemented as 2 perl distributions and appropriate js distribution:

1. Joose::Librarian

Stand-alone librarian, which can perform the 1st step of the loading scheme.
Eventually, it should be transformed into plugin for Module::Build::JSAN.

2. Joose::Library

Small catalyst app, which uses Joose::Librarian as a model.

Joose::Library will be eventually transformed into 2 different Catalyst extensions - Catalyst::Controller and Catalyst::Model

3. JooseX.Namespace.Depended

Several roles for Joose metaclasses, which make them able to asynchronously process dependencies (in both grouped/non-grouped modes)

The whole code is considered experimental. For configuration/installation/etc details, please refer to the sources.


Results.

To demonstrate the implementation results, we'll use the "stressed" test suite from JooseX.Namespace.Depended distribution.

This test suite contain about 100 inter-depended files, which are also randomly distributed across 2 roots (2 INC paths)

1. Usual (non-grouped) loading
In this mode, each module requires a separate http request to load. The results are presented on this screenshot:
http://symbie.org/sandbox/jsan/non-grouped.jpg

2. Grouped loading for the same suite.
http://symbie.org/sandbox/jsan/grouped.jpg

As you can see the whole test suite (~100 files) was loaded with 2 http requests.


Conclusion.

Was demonstrated the grouped dependencies loading approach, currently applicable for Joose classes only.
Described approach allows to load any Joose class with any number of dependencies in-depth with 2 http requests.

Comments are welcome.

Regards, Nickolay

[Nickolay Platonov]

Hi Malte,

Sure, I don't mind.
About - "demo package" - probably only as the VM image (1.3GB) Many separate things needs to be installed - like perl, Catalyst, local web server, ...

We'll be great if #serverjs can coordinate the efforts with #jsan. Though if they are a ruby fans, i think it will be hard to explain "why" to them )
Need to try anyway..

On Sat, May 23, 2009 at 2:47 PM, Malte Ubl <malt...@gmail.com> wrote:

Hey Nickolay,

this all sounds really cool. I'm gonna write it up for the blog if you
don't mind.
Could we provide a "just works" version of the backend, that people
can install without knowing CPAN, etc.?

By the way: The guys at the ServerJS mailing list
http://groups.google.com/group/serverjs are looking into a module
distribution system. Seems like currently gems is winning. Maybe JSAN
could play a role there.

Cheers
Malte