require/include HTTP URLs
ryan dahl
occasionally post little tasks to the mailing list. If you do this,
please create a patch against HEAD and post it to this mailing list.
I'll wait a week for someone to do them before doing it myself :)
Here's the first task:
Add support for including/requiring modules from HTTP URLs. This can
be done completely in javascript. It requires understanding how to use
node.http.Client and node.fs.cat(). I suggest the following path (but
feel free to come up with your own solution):
1. add a new function called node.http.cat(url, encoding, callback)
which will send a GET request to the URL, buffer the body, and return
it to the callback.
2. add another function node.cat(url_or_path, encoding, callback)
which abstracts node.http.cat and node.fs.cat().
3. change include() and require() in node.js to use node.cat() instead
of node.fs.cat() to load modules.
Urban Hafner
> Add support for including/requiring modules from HTTP URLs. This can
> be done completely in javascript. It requires understanding how to use
> node.http.Client and node.fs.cat(). I suggest the following path (but
> feel free to come up with your own solution):
Just small enough for me to get something done. Here's the first part,
the test case the code should pass:
http://github.com/ujh/node/commit/6b9e5eedf2fadbd192e9be522ea51bd8fd02bb71
Urban
ryan dahl
Looks good. I'll wait for the rest to merge it.
Urban Hafner
> Looks good. I'll wait for the rest to merge it.
http://github.com/ujh/node/commit/ce85f84d152dde9dc2cead5a8e4889710b3f0a69
This implementes node.http.cat(). It does miss a timeout, which would be
a good thing I guess. Do other parts of node.js provide timeout
functionality?
The question then becomes: Should load/require have a second argument
that is a timeout and what should the behaviour be? Should it call
onError after the timeout?
Also, is it OK that node.http.cat is inside the closure in http.js? It
doesn't seem necessary as it doesn't use any of the private stuff I guess.
Urban
tim
include/require's (pseudo) synchronous? Are there specific or
existing needs here or just following ServerJS lead? I am working on
a portable framework that intends to run in a variety of JS
environments, and synchronous module loading was a non-starter.
Instead, I have standardized on a "require(URI, callback, errback);"
construct, where the callback is passed the exports as first
argument. In my implementation, require actually returns a deferred
that can be kept around as a reference (to that "instance" of the
loaded module / URI).
Example:
var dojo131_available = require("../dojo131.js");
dojo131_available.addCallback(function(exports) {
});
dojo131_available.addErrback(function() { throw new Error("dojo131.js
is required, sorry"); });
etc.
It's easy enough to wrap various synchronous implementations, but
outside of security considerations (such as how AIR disables eval
after startup), why synchronous?
Thanks for input, -Tim
On Jun 17, 3:03 am, Urban Hafner <ur...@bettong.net> wrote:
> ryan dahl wrote:
> > Looks good. I'll wait for the rest to merge it.
>
tim
synchronous, ie: the module conceptually blocks until all require/
include's are finished and those must all be done before onLoad.
ry
open to having require(uri, callback, errback) in addition to onLoad.
Urban Hafner
> ryan dahl wrote:
>
>> Looks good. I'll wait for the rest to merge it.
Here you go:
http://github.com/ujh/node/commit/ea290e727de256a51f0a4ef9e4ff701da88f975c
The following things are of interest:
1. node.cat()'s callback has the status code as its first argument. For
files everything but 0 means failure. For HTTP everything but 200 (sort
of) means failure. Right now node.http.cat() returns the status 0 for
HTTP 200 and -1 for everything else. This of course masks the real
status code and therefore might not be ideal.
2. Module loading now uses "://" to distinguish remote modules from
local ones. Maybe someone can come up with a better distinction.
3. I've "fixed" parseUri() to exclude empty parts of the parsed URI (so
no more "" values).
I think that's it.
Urban
ry
> 1. node.cat()'s callback has the status code as its first argument. For
> files everything but 0 means failure. For HTTP everything but 200 (sort
> of) means failure. Right now node.http.cat() returns the status 0 for
> HTTP 200 and -1 for everything else. This of course masks the real
> status code and therefore might not be ideal.
Urban Hafner
> awesome. thank you.
Sure :) What's the next thing on your list? I've been thinking cookies.
Something like req.cookies() and res.setCookie(). I guess that's low
level enough to go into http.Server and http.Client.
I'm thinking of fixing it like this: node.http.cat() will return the
HTTP status code and not 0 or -1. node.cat() will
a) have add a third argument to the callback that contains stuff like
the real return code and if it was a local or remote request.
b) do away with the status callback argument as it's not clear if we
should choose the http or file return codes and add the argument from a)
which may contain an additional field called success (or so).
c) the first argument to the callback (status) could just return true or
false. Any additional information can be retrieved from argument three
(see a) ).
What do you think? Right now I'm in favour of c)
Urban
ryan dahl
>
> ry wrote:
>> awesome. thank you.
>
> Sure :) What's the next thing on your list?
Maybe deferreds (see below).
> I've been thinking cookies.
> Something like req.cookies() and res.setCookie(). I guess that's low
> level enough to go into http.Server and http.Client.
Cookies should be left to higher level code. The web server should
only deal with connection handling (e.g. keep-alive) and parsing the
message into its request line, header lines, and body. It should not
deal with header parsing (except for "Connection",
"Transfer-Encoding", and "Content-Length") or body parsing.
>>> 1. node.cat()'s callback has the status code as its first argument. For
>>> files everything but 0 means failure. For HTTP everything but 200 (sort
>>> of) means failure. Right now node.http.cat() returns the status 0 for
>>> HTTP 200 and -1 for everything else. This of course masks the real
>>> status code and therefore might not be ideal.
>>
>> Hm yeah - perhaps not ideal, but it works for now.
>
> I'm thinking of fixing it like this: node.http.cat() will return the
> HTTP status code and not 0 or -1. node.cat() will
>
> a) have add a third argument to the callback that contains stuff like
> the real return code and if it was a local or remote request.
>
> b) do away with the status callback argument as it's not clear if we
> should choose the http or file return codes and add the argument from a)
> which may contain an additional field called success (or so).
>
> c) the first argument to the callback (status) could just return true or
> false. Any additional information can be retrieved from argument three
> (see a) ).
>
> What do you think?
This the first situation I've seen where using Deferreds really makes
sense to me. Generally I've been wanting that to exist in higher level
libraries but maybe that needs to be reconsidered. Perhaps at the same
time we should think about having real DOM style events instead of
just callbacks. e.g.
https://developer.mozilla.org/en/DOM/element.addEventListener
Urban Hafner
>> I've been thinking cookies.
>> Something like req.cookies() and res.setCookie(). I guess that's low
>> level enough to go into http.Server and http.Client.
>
> Cookies should be left to higher level code. The web server should
> only deal with connection handling (e.g. keep-alive) and parsing the
> message into its request line, header lines, and body. It should not
> deal with header parsing (except for "Connection",
> "Transfer-Encoding", and "Content-Length") or body parsing.
All right. Would that be part of node itself or should that be a library?
>> I'm thinking of fixing it like this: node.http.cat() will return the
>> HTTP status code and not 0 or -1. node.cat() will
>>
>> a) have add a third argument to the callback that contains stuff like
>> the real return code and if it was a local or remote request.
>>
>> b) do away with the status callback argument as it's not clear if we
>> should choose the http or file return codes and add the argument from a)
>> which may contain an additional field called success (or so).
>>
>> c) the first argument to the callback (status) could just return true or
>> false. Any additional information can be retrieved from argument three
>> (see a) ).
>>
>> What do you think?
>
> This the first situation I've seen where using Deferreds really makes
> sense to me. Generally I've been wanting that to exist in higher level
> libraries but maybe that needs to be reconsidered. Perhaps at the same
> time we should think about having real DOM style events instead of
> just callbacks. e.g.
> https://developer.mozilla.org/en/DOM/element.addEventListener
I'm not sure I understand the connection here. Do you mean something
like having onSuccess and onError calls for the node.cat() call? And how
would you do it? I've seen stuff like node.cat(arguments, {onError:
function() {}, onSuccess: function() {}). Otherwise we need to have some
object here that can be observed (we really want the observer pattern
here, I guess).
Urban
ryan dahl
>> only deal with connection handling (e.g. keep-alive) and parsing the
>> message into its request line, header lines, and body. It should not
>> deal with header parsing (except for "Connection",
>> "Transfer-Encoding", and "Content-Length") or body parsing.
>
> All right. Would that be part of node itself or should that be a library?
That should be a library.
>> This the first situation I've seen where using Deferreds really makes
>> sense to me. Generally I've been wanting that to exist in higher level
>> libraries but maybe that needs to be reconsidered. Perhaps at the same
>> time we should think about having real DOM style events instead of
>> just callbacks. e.g.
>> https://developer.mozilla.org/en/DOM/element.addEventListener
>
> I'm not sure I understand the connection here. Do you mean something
> like having onSuccess and onError calls for the node.cat() call? And how
> would you do it? I've seen stuff like node.cat(arguments, {onError:
> function() {}, onSuccess: function() {}). Otherwise we need to have some
> object here that can be observed (we really want the observer pattern
> here, I guess).
I guess Dojo's syntax is okay
var d = node.cat("http://tinyclouds.org/mjsunit.js");
d.addCallback(onLoad);
d.addErrback(function () { puts("cannot load url"); node.exit(1); }
My timid objection to this has been that, in the name of minimalism,
it adds a lot of syntax, that for a low-level I/O infustructure, is
not necessary. However, I'm beginning to see for myself that having
Deferreds for the asynchronous functions is probably going to make
even the internal code simpler.
Urban Hafner
> I guess Dojo's syntax is okay
>
> var d = node.cat("http://tinyclouds.org/mjsunit.js");
> d.addCallback(onLoad);
> d.addErrback(function () { puts("cannot load url"); node.exit(1); }
>
> My timid objection to this has been that, in the name of minimalism,
> it adds a lot of syntax, that for a low-level I/O infustructure, is
> not necessary. However, I'm beginning to see for myself that having
> Deferreds for the asynchronous functions is probably going to make
> even the internal code simpler.
Well, how about this then:
var d = node.cat("http://tinyclouds.org/mjsunit.js");
d.addObserver({success: onLoad, error: function () { puts("cannot load
url"); node.exit(1); }})
Which is nice, because the observer can be any object that has the
required attributes.
And then there are no fixed names for the callbacks which means this can
be implemented as some kind of "mixin" (in Ruby terms) and you can use
it in all classes/objects by extending that object. Which is easy in JS
as you just have to implement something like Object.extend in Prototype.
Urban
Aaron Quint
If theres a motive to follow the EM style in some way - it would be
really helpful to have some sort of top level prototype for
Deferrables. This could lie somewhere in between an object with DOM
like events and an Event Machine style object. I like the idea of
named events much better then just defining callback - It provides an
easy syntax and structure for creating evented objects. The cool thing
about having a top level prototype (that maybe Server and Client
extend) is that you can keep 'sub' classing and basically have
something like event bubbling within node.
thoughts?
Also, I'm new to the list, but very excited about node. Thanks for all
your hard work! Its a really great piece of code.
--AQ
Aaron Quint
http://www.quirkey.com
Urban Hafner
> late chime in, but this is getting interesting . . .
> If theres a motive to follow the EM style in some way - it would be
> really helpful to have some sort of top level prototype for
> Deferrables. This could lie somewhere in between an object with DOM
> like events and an Event Machine style object. I like the idea of
> named events much better then just defining callback - It provides an
> easy syntax and structure for creating evented objects. The cool thing
> about having a top level prototype (that maybe Server and Client
> extend) is that you can keep 'sub' classing and basically have
> something like event bubbling within node.
>
> thoughts?
I was thinking of something like this:
var Observable = {
addObserver: function() {},
removeObserver: function() {},
notify: function() {}
}
Then every class that wants to have this functionality can do something
like this:
Object.extend(node.http.Server.prototype, Observable)
Assuming that Object.extend() copies the fields from argument 2 into
argument 1 (as defined here: http://prototypejs.org/api/object/extend ).
That's how we used to do it in a (client side) JavaScript project I
worked on.
Urban
Aaron Quint
Yup, Thats what I was imagining. In jQuery theres a similar convention
($.extend()). So each of those methods would take a 'name' and a
callback? One of the cool things about the jQuery Event API (I think
Prototype has this too) is the ability to pass data to the bound
callback when triggering. It makes it easy to refactor a bunch of
similar callbacks into a single callback that can switch on the
triggered data.
Something I'm curious about - my C++ is rusty so I cant dive in myself
- If node is based on a full JS interpreter (V8), doesnt it inherently
include some kind of event handling mechanism? I see the issue that
there is no real DOM in node, but couldnt you somehow implement a
virtual DOM with just a 'body' and bind/trigger events off of that,
letting V8 handle the event model?
--AQ
Urban Hafner
> Yup, Thats what I was imagining. In jQuery theres a similar convention
> ($.extend()). So each of those methods would take a 'name' and a
> callback? One of the cool things about the jQuery Event API (I think
> Prototype has this too) is the ability to pass data to the bound
> callback when triggering. It makes it easy to refactor a bunch of
> similar callbacks into a single callback that can switch on the
> triggered data.
Not really. addObserver() would register an object. Then when an event
is triggered the controlling code would call notify(eventName,
argument1, argument2). notify() would the go through all objects that
want to be notified and look for a method called eventName(). If it's
present it would be called with the arguments. If it's not there, the
object isn't interested in that event. Is this what you are looking for?
Apart from the naming of course, that's not fixed, yet.
Urban
Aaron Quint
What would happen if multiple objects include the named function?
Would notify call all of them? If so, that works, too. I guess the
benefit of a named Event approach is that a single object can have
multiple callbacks for a single Event. In pseudo code:
var MyObject = {
...
}
extend(MyObject, Observable)
MyObject.addObserver('error', function() {
// show error message
})
MyObject.addObserver('error', function() {
// send error to the log
});
MyObject.notify('error');
// triggers both callbacks.
--AQ
Urban Hafner
> What would happen if multiple objects include the named function?
> Would notify call all of them? If so, that works, too. I guess the
> benefit of a named Event approach is that a single object can have
> multiple callbacks for a single Event. In pseudo code:
What I'm suggesting here is basically the observer pattern. So yes, all
observers would get called.
Your approach is modelled after the way the events in the DOM work, I
guess. Which is natural to JS programmers. So it should definitely be
considered as well.
I'm not sure what I like better right now. Does anybody else have an
opinion?
Urban
ryan dahl
> opinion?
As I see it, there are two distinct situations in Node. They perhaps
should be handled with different APIs.
1) Simple asynchronous requests. For example, node.fs.cat (which cats
a file). The function runs in the background, returns some data or
possibly errors out. For this I think Dojo's deferreds are rather
nice.
(One use case in particular I need: a way to take N deferreds and wait
for their completion. I'm not sure how this is done in Dojo.)
2) Objects which emit events. An HTTP server gets a "request" event. A
HTTP request gets a "onBody" event. These are modeled better with
something like DOM events. The object emits events, any number of
callbacks can listen to such an event. Perhaps there could even be
event bubbling - but I can't think of what Node would use that for.
Whatever the choice, I'm most concerned about keeping it minimal.
Javascript object creation is a non-negligible performance penalty
here. I'd much rather have a fast web server than a featureful library
for event registration.
ryan dahl
> - If node is based on a full JS interpreter (V8), doesnt it inherently
> include some kind of event handling mechanism? I see the issue that
> there is no real DOM in node, but couldnt you somehow implement a
> virtual DOM with just a 'body' and bind/trigger events off of that,
> letting V8 handle the event model?
v8 doesn't have events. Node is v8's event model :)
Aaron Quint
I'm down with whatever is the most simple/fastest implementation. My
inclination is to DOM style events because it might make for cleaner
APIs if you're using the same/similar code for front-end and back-end
(my ultimate goal).
ryan dahl
use the DOM's EventTarget interface
http://www.w3.org/TR/DOM-Level-2-Events/events.html#Events-EventTarget
The web server and other network objects can implement this interface.
Asynchronous functions (mostly those which execute system calls on the
thread pool) can return an object (a "promise" or "deferred") which
implements the EventTarget interface. The returned object issues the
"complete" and "error" events.
Perhaps that returned object can have special methods for registering
completion and error handlers. e.g.
var d = node.fs.open("filename");
d.addCallback(handleSuccess);
instead of
d.addEventListener("success", function (event) {
handleSuccess.apply(this, event.args);
});
Again, I hesitate because
new node.http.Server(handleRequest).listen(80);
is much simpler than
var s = new.http.Server();
s.addEventListener("request", function (event) {
handleRequest.apply(s, [event.req, event.res]);
});
s.listen(80);
Aaron Quint
>
> Again, I hesitate because
>
> new node.http.Server(handleRequest).listen(80);
>
> is much simpler than
>
> var s = new.http.Server();
> s.addEventListener("request", function (event) {
> handleRequest.apply(s, [event.req, event.res]);
> });
> s.listen(80);
>
True, but could the first argument to server be an object that defined
the default request handlers?
var s = new.http.Server({
request: function(req, res) {
....
}
});
s.listen(80);
Something cool about this is the ability to have a defined number of
default 'lifecycle' events and be able to observe/bind to them at
will.
I implemented something similar the front-end framework I've been
working with (sammy: http://code.quirkey.com/sammy) and it allows for
doing some nice things (easy logging of events, 'before' and 'after'
triggers, etc).
--AQ
ryan dahl
> working with (sammy: http://code.quirkey.com/sammy) and it allows for
> doing some nice things (easy logging of events, 'before' and 'after'
> triggers, etc).
I really like Sammy (and Sinatra). Would be great to have it ported to Node.
Aaron Quint
Thanks! Thats my goal - I started playing around this weekend and
didnt get to far, but the dream of a single API for both front end and
backend is two awesome to sleep on.
--AQ
Urban Hafner
> 1) Simple asynchronous requests. For example, node.fs.cat (which cats
> a file). The function runs in the background, returns some data or
> possibly errors out. For this I think Dojo's deferreds are rather
> nice.
>
> (One use case in particular I need: a way to take N deferreds and wait
> for their completion. I'm not sure how this is done in Dojo.)
>
> 2) Objects which emit events. An HTTP server gets a "request" event. A
> HTTP request gets a "onBody" event. These are modeled better with
> something like DOM events. The object emits events, any number of
> callbacks can listen to such an event. Perhaps there could even be
> event bubbling - but I can't think of what Node would use that for.
Event bubbling seems to be rather useless for this level of abstraction.
If there's a higher level server interface then it might sense there.
> Whatever the choice, I'm most concerned about keeping it minimal.
> Javascript object creation is a non-negligible performance penalty
> here. I'd much rather have a fast web server than a featureful library
> for event registration.
Hm. Is there a profiler for v8? I mean how to we know if the event
handling code is too slow?
Anyway: If we can find example code for both APIs mentioned, I'll
implement them and then we can see if we need to speed them up somehow
and if they are useful. Maybe even a hybrid of both APIs should be
possible, so that you can decide what kind of style you want. At least
before we start speeding things up ;)
Urban
ryan dahl
> If there's a higher level server interface then it might sense there.
Yes, I think you're right. OTH, I like the idea of implementing w3's
event spec - perhaps that would make it possible to use browser-side
libraries more with Node.
>> Whatever the choice, I'm most concerned about keeping it minimal.
>> Javascript object creation is a non-negligible performance penalty
>> here. I'd much rather have a fast web server than a featureful library
>> for event registration.
>
> Hm. Is there a profiler for v8? I mean how to we know if the event
> handling code is too slow?
Yes: ./node --prof script.js
It creates a file called v8.log which you can analyze using
deps/v8/tools/linux-tick-processor.py
I've experienced garbage collection being triggered too often because
of creating too many objects in the HTTP server. So it went from
http://s3.amazonaws.com/four.livejournal/20090529/timeseries6.png
to
http://s3.amazonaws.com/four.livejournal/20090617/timeseries11.png
histogram view (top):
http://s3.amazonaws.com/four.livejournal/20090617/hist10.png
I'm thinking about doing an javascript object pool for connection and
request objects at some point (once node stabilizes). I think it will
speed things up.
> Anyway: If we can find example code for both APIs mentioned, I'll
> implement them and then we can see if we need to speed them up somehow
> and if they are useful. Maybe even a hybrid of both APIs should be
> possible, so that you can decide what kind of style you want. At least
> before we start speeding things up ;)
I'm going to play around with this today. It would be nice to also do
some internal plumbing at the same time for event handling.
tim
deferred/events -- how about simple default implementation wrappers
that provide a streamlined most-common-path interface? Something
like:
new nodex.http.SimpleServer(handleRequest, 80);
Re: addEventListener('request'..) -- hmm, "request" chaining could be
interesting... How would two or more registered request handlers
actually play out at runtime? This might lead to some different low-
level API requirements (if a handler needs to play nice with co-
handlers).
Re: pooling multiple Deferred's -- I don't remember off the top of my
head how this is done in Dojo or its examples, but DeferredList is how
Twisted does it:
http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto7
Re: before & after advice -- Dojo 0.4 had some AOP stuff, but it was
later removed. I'm not sure why, but my personal experience was that
it created a debugging nightmare.
http://www.dojotoolkit.org/book/dojo-book-0-4/part-5-connecting-pieces/event-system/after-and-around-advice
There's also a candidate > 0.4 port (as a "dojox.advice" module) in
the forums:
http://www.dojotoolkit.org/forum/dojox-dojox/dojox-development-discussion/dojox-advice-eta
Re: DOM events -- I've been working with server-side W3C DOM for a few
weeks now. I'm not sure what will emerge, but building on existing
browser concepts has worked well so far as a scaffolding layer.
On Jun 22, 12:10 pm, ryan dahl <coldredle...@gmail.com> wrote:
> I have to think about this a bit more but my first inclination is to
ry
> Re: addEventListener('request'..) -- hmm, "request" chaining could be
> interesting... How would two or more registered request handlers
> actually play out at runtime? This might lead to some different low-
> level API requirements (if a handler needs to play nice with co-
> handlers).
some extent, anytime you have multiple event handlers. Perhaps it
would be nice for before/after filters, for example. (Which suggests
that canceling an event before it propagates to other handlers should
be part of the API at least.)
> Re: DOM events -- I've been working with server-side W3C DOM for a few
> weeks now. I'm not sure what will emerge, but building on existing
> browser concepts has worked well so far as a scaffolding layer.
event model? Does it make sense to try and retain that functionality?
tim
> > Re: DOM events -- I've been working with server-side W3C DOM for a few
> > weeks now. I'm not sure what will emerge, but building on existing
> > browser concepts has worked well so far as a scaffolding layer.
>
> Have you experienced any uses for the capture/bubble phases of the DOM
> event model? Does it make sense to try and retain that functionality?
hard to make efficient, so maybe defer to TBD higher-level libraries
for now?
But, what might an event model might look like?
Node/TcpServer/HttpHandler/(Request|Response)/(Headers|Body) ?
Here is a quick n' dirty hypothetical DOM-like structure, for
exploring possible capture and bubbling effects:
. NodePool (OS) ?
. . NodeVM (Process) ?
. . . TCP Server (Socket) ?
. . . + on[ connect | disconnect | read | write | close ]
. . . . HTTP Server (Navigator)
. . . . + on[ *handleRequest* | load | unload | abort | error | ??]
. . . . . HTTP Req/Resp Context (Window)
. . . . . + on[ load | unload | abort | error ]
. . . . . . HttpStream (XMLHttpRequest) ?
. . . . . . + on[ readystatechange ]
. . . . . . HTTP Request (Document)
. . . . . . + on[ load | unload | stop | beforeunload |
{MutationEvents} ]
. . . . . . . HEAD (headers)
. . . . . . . BODY (entity body / payload)
. . . . . . HTTP Response (Document)
. . . . . . + on[ load | unload | stop | beforeunload |
{MutationEvents} ]
. . . . . . . HEAD (headers)
. . . . . . . BODY (entity body / payload)
....