Proposal: Timer

[Charles Jolley]

Hi everyone,

Has anyone thought yet about implementing a standard timer/timeout module yet? The browser has setTimeout(), clearTimeout() etc. but since most servers implement their own reactor loop, I assume this API is not relevant there, right?

If so, perhaps CommonJS should include a "timer" module that specifies a standard API for scheduling code to run later. In the browser this would wrap setTimeout(), clearTimeout(), setInterval(), clearInterval(). In other platforms this could be implemented as part of the reactor.

Thoughts?

-C

[Ondřej Žára]

Hi,

Has anyone thought yet about implementing a standard timer/timeout module yet?  The browser has setTimeout(), clearTimeout() etc. but since most servers implement their own reactor loop, I assume this API is not relevant there, right?

If so, perhaps CommonJS should include a "timer" module that specifies a standard API  for scheduling code to run later.  In the browser this would wrap setTimeout(), clearTimeout(), setInterval(), clearInterval().  In other platforms this could be implemented as part of the reactor.

is there a reason for *not* using the well-known and broadly used browser API for this?


O.


 

Thoughts?

-C

--

You received this message because you are subscribed to the Google Groups "CommonJS" group.
To post to this group, send email to comm...@googlegroups.com.
To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.

[Anthony Catel]

Hi,

What's wrong with this API?

Anthony

Le 10/12/09 09:10, Charles Jolley a �crit :

[Hannes Wallnoefer]

2009/12/10 Ondřej Žára <ondre...@gmail.com>:

> Hi,
>
>> Has anyone thought yet about implementing a standard timer/timeout module
>> yet?  The browser has setTimeout(), clearTimeout() etc. but since most
>> servers implement their own reactor loop, I assume this API is not relevant
>> there, right?
>>
>> If so, perhaps CommonJS should include a "timer" module that specifies a
>> standard API  for scheduling code to run later.  In the browser this would
>> wrap setTimeout(), clearTimeout(), setInterval(), clearInterval().  In other
>> platforms this could be implemented as part of the reactor.
>>
>
> is there a reason for *not* using the well-known and broadly used browser
> API for this?

I also think there's nothing wrong with the setTimeout/setInterval
API. I recently implemented this as "helma/scheduler".

+1 for defining a CommonJS standard module for this.

Hannes

[Karl Guertin]

On Thu, Dec 10, 2009 at 3:51 AM, Ondřej Žára <ondre...@gmail.com> wrote:
> is there a reason for *not* using the well-known and broadly used browser
> API for this?

The only problem I know of with setTimeout and friends is that things
get crazy in a browser when you have large numbers of timers running.
I know Sproutcore coalesces timers [1] in order to keep them
fast/synchroized. This is probably an implementation detail/problem,
but that's the only negative thing I can come up with for the timer
API.

[1] http://github.com/sproutit/sproutcore/blob/master/frameworks/foundation/system/timer.js

[Charles Jolley]

> What's wrong with this API?

I am OK with the setTimeout() API, but was proposing that CommonJS define a module ("timer"?) to contain it.

Afaict, Narwhal uses a run-to-completion model unless you enqueue() using the reactor module. Node.js also implements its own event loop. Both of these could benefit from having the timer API in a module so they can provide their own "user-space" implementations.

A simple browser implementation could just export the built-ins. Though, as Karl said, for SproutCore/Tiki I would like to use an implementation that schedules timeouts manually to provide better consistency and performance.

So my proposal would be:

---
a module called "timer" which defines:

timerId = setTimeout(func, delay); // schedules a timer
clearTimeout(timerId) ; // cancels a timer, passed a timeoutId returned by previous

timerId = setInterval(func, delay); // schedules a repeating timer
clearInterval(timerId); // cancels a repeating timer

The callback should be invoked on expiration with no parameters.

This follows the standard impl in browsers:
https://developer.mozilla.org/en/DOM/window.setTimeout

I would deviate from the standard impl though in a few ways:

- This module should NOT support passing a string for the first param or passing extra parameters. These are both complicated and error prone.

- We need to specify what "this" should be when the callback is invoked. This is definitely one place the browser impl gets it wrong, especially for the securable modules world where we want to contain leaks into a global scope. In the browser, "this" will be the window object.

-Charles

>
> Anthony
>
> Le 10/12/09 09:10, Charles Jolley a écrit :

[Anthony Catel]

Hi,

This is more or less what I've done in APE :
http://www.ape-project.org/wiki/index.php/How_to_build_a_serverside_JS_module#Timers
Internally, all timers run on top of a smaller one (1ms). That way you
can run a ton of timers without loosing accuracy.

Anthony

Le 10/12/09 16:50, Charles Jolley a �crit :

>> What's wrong with this API?
>>
> I am OK with the setTimeout() API, but was proposing that CommonJS define a module ("timer"?) to contain it.
>
> Afaict, Narwhal uses a run-to-completion model unless you enqueue() using the reactor module. Node.js also implements its own event loop. Both of these could benefit from having the timer API in a module so they can provide their own "user-space" implementations.
>
> A simple browser implementation could just export the built-ins. Though, as Karl said, for SproutCore/Tiki I would like to use an implementation that schedules timeouts manually to provide better consistency and performance.
>
> So my proposal would be:
>
> ---
> a module called "timer" which defines:
>
> timerId = setTimeout(func, delay); // schedules a timer
> clearTimeout(timerId) ; // cancels a timer, passed a timeoutId returned by previous
>
> timerId = setInterval(func, delay); // schedules a repeating timer
> clearInterval(timerId); // cancels a repeating timer
>
> The callback should be invoked on expiration with no parameters.
>
> This follows the standard impl in browsers:
> https://developer.mozilla.org/en/DOM/window.setTimeout
>
> I would deviate from the standard impl though in a few ways:
>
> - This module should NOT support passing a string for the first param or passing extra parameters. These are both complicated and error prone.
>
> - We need to specify what "this" should be when the callback is invoked. This is definitely one place the browser impl gets it wrong, especially for the securable modules world where we want to contain leaks into a global scope. In the browser, "this" will be the window object.
>
> -Charles
>
>
>> Anthony
>>

>> Le 10/12/09 09:10, Charles Jolley a �crit :

[Wes Garland]

Defining a timer module is all well and good, but in order for it to *work*, you have to either

a) define base-level support for an event loop
b) have the timer module contain the event loop and manage your code
c) implement asynchronous (pre-emptive) JavaScript events

I would be happier, BTW, seeing this as an event module where where setTimeout() etc are *one* type of event.

FWIW, gpsee has C already, but I think that is the wrong solution for the community at large.

A blend between a & b is probably the best solution: have an event module somehow enable latent base-level event loop support and implement the timer APIs.

Wes

Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Wes Garland]

Anthony:

FWIW IIUC mozilla's in-browser timer events all round to 10ms, and the minimum event delay is also 10ms.

It might be interesting to add a "now" event along with setTimeout, to all CommonJS programmers to write fast-looping programs without language-level loops.

Wes

On Thu, Dec 10, 2009 at 11:17 AM, Anthony Catel <a.c...@weelya.com> wrote:

Hi,

This is more or less what I've done in APE :
http://www.ape-project.org/wiki/index.php/How_to_build_a_serverside_JS_module#Timers
Internally, all timers run on top of a smaller one (1ms). That way you
can run a ton of timers without loosing accuracy.

Anthony

Le 10/12/09 16:50, Charles Jolley a écrit :

>> What's wrong with this API?
>>
> I am OK with the setTimeout() API, but was proposing that CommonJS define a module ("timer"?) to contain it.
>
> Afaict, Narwhal uses a run-to-completion model unless you enqueue() using the reactor module.  Node.js also implements its own event loop.  Both of these could benefit from having the timer API in a module so they can provide their own "user-space" implementations.
>
> A simple browser implementation could just export the built-ins.  Though, as Karl said, for SproutCore/Tiki I would like to use an implementation that schedules timeouts manually to provide better consistency and performance.
>
> So my proposal would be:
>
> ---
> a module called "timer" which defines:
>
> timerId = setTimeout(func, delay); // schedules a timer
> clearTimeout(timerId) ;  // cancels a timer, passed a timeoutId returned by previous
>
> timerId = setInterval(func, delay); // schedules a repeating timer
> clearInterval(timerId); // cancels a repeating timer
>
> The callback should be invoked on expiration with no parameters.
>
> This follows the standard impl in browsers:
> https://developer.mozilla.org/en/DOM/window.setTimeout
>
> I would deviate from the standard impl though in a few ways:
>
> - This module should NOT support passing a string for the first param or passing extra parameters. These are both complicated and error prone.
>
> - We need to specify what "this" should be when the callback is invoked.  This is definitely one place the browser impl gets it wrong, especially for the securable modules world where we want to contain leaks into a global scope.  In the browser, "this" will be the window object.
>
> -Charles
>
>
>> Anthony
>>

>> Le 10/12/09 09:10, Charles Jolley a écrit :

--

[Charles Jolley]

Well you could be super stupid and just idle in a while() loop until an event needs to fire; maybe with a sleep in between.  Not ideal but still easy to implement.

I think making setTimeout() part of an event module is an interesting idea too.  I was thinking maybe generalized events should be defined as a separate module to keep this one easy to comply with but a larger module makes sense to me too.

[Ash Berlin]

On 10 Dec 2009, at 16:17, Anthony Catel wrote:

> Hi,
>
> This is more or less what I've done in APE :
> http://www.ape-project.org/wiki/index.php/How_to_build_a_serverside_JS_module#Timers
> Internally, all timers run on top of a smaller one (1ms). That way you
> can run a ton of timers without loosing accuracy.
>
> Anthony
>

Timers seem to be in vogue at the moment: I am in the process of adding them to Zest (which is, or rather was, the development HTTP server shipped with JuiceJS): http://pastie.org/private/gvmxftcdcwocwbufnukvnw

I just let boost::asio do its thing. Who knows if it will need something smarter in the future or not. (Oh and that failure on clearTimeout is cos I haven't written that method yet)

As for 'a common API' - I was thinking of when the zest module is loaded it would also make it self available as require('event').

PS. Please trim posts a bit guys

[Wes Garland]

Charles:

On Thu, Dec 10, 2009 at 11:24 AM, Charles Jolley <cha...@sproutit.com> wrote:

Well you could be super stupid and just idle in a while() loop until an event needs to fire; maybe with a sleep in between.  Not ideal but still easy to implement.

That's not super-stupid, that's an event loop -- the event loop-controller, that is.

How setTimeout needs to work is to schedule a job with the event loop controller. Then, when there is no more JavaScript code left to run, the event loop controller idles in a while() loop waiting for events to fire.

I think making setTimeout() part of an event module is an interesting idea too.  I was thinking maybe generalized events should be defined as a separate module to keep this one easy to comply with but a larger module makes sense to me too.

I think we can have our cake and eat it too, by specifying timer events which load from require("events").  Then later we could have a different specification for, say, POSIX signal events.

Both of these might sit on top of a very bare-bones event specification which knows how to register "now" callbacks. "now" callbacks would be run by the event loop controller ASAP and be useful for implementing async XHR, general async I/O, etc.

I don't think setTimeout(0) is appropriate for now callbacks due to the browser implementation I mentioned earlier.  People will expect it to work the same.

The key question is how to make the event module either run an event loop, or initialize an event loop in the underlying embedding -- what has to happen is when the CommonJS program module finishes executing  (which is like finishing the SCRIPT tags in the browser) that the embedding turn around and wait for pending events in some situation. We can't "always wait", there are whole classes of CommonJS programs which are expected to terminate.

"Some situation" is interesting: in the case of timer events, we always know if there will be events in the future or not, so for timers the question is easy to answer. I see two answers:

1) If there are event handlers registered, engage the event loop, or
2) If some magic function call has been made, engage the event loop

#1 - has the advantage that it more resembles the browser environment
#2 - has the advantage that "onHUP = reloadConfigFile()" doesn't make the program not halt

If we do general events, we also need to talk about how to make programs terminate.
Do they terminate on uncaught exceptions?  (I think yes)
If no, how to they terminate?  How about throw(new require("system").SystemExit(5)) makes the the process exit with error status 5?

Wes

--

[Kris Zyp]

http://github.com/kriszyp/browserjs/ implements setTimeout and friends
(in "browser/timeout") based on Narwhal's event-queue module and works
great and properly follows the normal browser event-loop approach.

I implemented the event queue/looping module Narwhal based on
https://developer.mozilla.org/en/nsIThread ((so obviously I would
recommend it). You can see my implementation here:
http://github.com/kriszyp/narwhal/blob/master/engines/rhino/lib/event-queue.js
The key points here are:
- A separate event-queue module to allow for adding new events into the
queue and handling the event loop.
- Allow for explicitly entering the event loop (enterEventLoop()), no magic
- Allow for explicitly processing the next event so you can spin through
the event loop to wait for particular event
- Threadsafe access to enqueuing (enqueue's can happen from any thread)
- Explicit shutdown() method
- Uncaught errors don't end the event loop (just as they don't in the
browser), but the event-loop does provide a default error handler

I also updated Narwhal's REPL to be event-based, which automatically
enter the event loop and then REPL commands are executed as events,
which makes it easy to try out the setTimeout:
http://github.com/kriszyp/narwhal/blob/master/lib/narwhal/repl.js
Kris

[Kris Zyp]

> - This module should NOT support passing a string for the first param
or passing extra parameters. These are both complicated and error prone.

That is reasonable.

> - We need to specify what "this" should be when the callback is invoked. This is definitely one place the browser impl gets it wrong, especially for the securable modules world where we want to contain leaks into a global scope. In the browser, "this" will be the window object.
>

I don't believe |this| is called with the window object by setTimeout,
but rather EcmaScript defines that the callee converts an undefined
|this| to the global (window). Calling setTimeout in "use strict" mode
would not result |this| being window for the called function. "use
strict" is the pathway to securing against this vulnerability, both on
the browser and the server, and I don't think we need to do anything
differently on the server.
Kris

[Ash Berlin]

On 10 Dec 2009, at 17:08, Kris Zyp wrote:

> Someone wrote:
>> - This module should NOT support passing a string for the first param
>> or passing extra parameters. These are both complicated and error prone.
>
> That is reasonable.

Not passing strings is fine, but are passing parameters really that much of a hassle?

[Wes Garland]

On Thu, Dec 10, 2009 at 12:14 PM, Ash Berlin <ash_flu...@firemirror.com> wrote:

Not passing strings is fine, but are passing parameters really that much of a hassle?

Passing mutable parameters could certainly get entertaining. Are they evaluated when the event handler is defined, or when the event triggers?  In what context are they evaluated?

Wes
 

[Charles Jolley]

>> Someone wrote:
>>> - This module should NOT support passing a string for the first param
>>> or passing extra parameters. These are both complicated and error prone.
>>
>> That is reasonable.
>
> Not passing strings is fine, but are passing parameters really that much of a hassle?

I don't think it is a hassle, but IE does not support it. I was trying to keep the spec a subset of the existing browser implementations so that a simple browser implementation of this module could be:

exports.setTimeout = setTimeout;
exports.clearTimeout = clearTimeout;
exports.setInterval = setInterval;
exports.clearInterval = clearInterval;

If you allow parameter passing, the browser implementation would need to wrap these methods for IE. Not a big deal either way IMO.

[Charles Jolley]

Passing mutable parameters could certainly get entertaining. Are they evaluated when the event handler is defined, or when the event triggers?  In what context are they evaluated?

If you were to emulate what the browser supports, it would look like this:

exports.setTimeout = function(func, delay) {

  var callback = func;

  if (arguments.length > 2) {

    var args = Array.prototype.slice.call(arguments, 2); 

    callback = function() { func.apply(this, args); };

  }

  // pass along to "native timeout mechanism.  browser in this case

  setTimeout(callback, delay); 

};

Wes
 

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Kris Kowal]

I am in favor of providing the browser API in a module. I am
indifferent whether this is a separate module or rolled into an
"event-queue" or "reactor" module. "timer" would imply just timers.
"event-queue" would imply that it supports simple enqueuing and
possibly also timed enqueuing. "reactor" implies that it does all of
those things and also manages async IO events.

I think it would be good to add support for passing the activation
object for the event for fly-weight event handlers (as opposed to a
closure or bind). Unfortunately, setTimeout and setInterval's
argument order gets in the way. It would make much more sense for the
block, thisp, and following args to be adjacent, but we'll have to
settle for interpolating the delay or interval between them. Oh,
well. Nothing's perfect.

I also think that we need to *permit* the implementation to return an
opaque "brand" object so that timer identifiers cannot be forged.

/*@module timer, event-queue, or reactor */

setTimeout(block, delay, thisp_opt, ...args):Cookie
setInterval(block, delay, thisp_opt, ...args):Cookie
clearTimeout(cookie)
clearInterval(cookie)

I am also in favor of creating a next-generation API based on
Promises. Perhaps the next-generation API could accommodate scheduled
events, like "at this time", "starting on this date and every time
delta (for each type of time delta)", in addition to "after this
dalay", and "starting now and continuing every such interval". For
real-time assurances, the intervals and timers might accept an
"epsilon" of variance and reliable promise-breaking. But that can
wait.

Kris Kowal

[Charles Jolley]

I think the passing the activation object is really important also. Implemented properly even in the browser this can dramatically reduce memory consumption in some cases (think running animations).

The downside is that setTimeout() and setInterval() would represent a super-set of the browser native behavior. This means that a browser implementation must implement these methods. Personally that is OK by me if no one else feels strongly about it.

I would be happy to put a strawman onto the CommonJS wiki for folks to hack at. I think "timer" makes the most sense if we want to keep this module simple. Alternatively, we could use "reactor" and plan on having several levels of compliance?

Is it OK for me to add something or does someone else need to do it?

-C

[Kris Kowal]

On Thu, Dec 10, 2009 at 1:33 PM, Charles Jolley <cha...@sproutit.com> wrote:
> Is it OK for me to add something or does someone else need to do it?

Go for it. We can fiddle with organization later, but Reactor/A would
probably suffice.

Kris Kowal

[Wes Garland]

> Alternatively, we could use "reactor" and plan on having several levels of compliance?

I think in this particular instance that "level" is the wrong word: it implies that some features are at a higher level than others.  Not having setTimeout doesn't mean you don't have asynchronous I/O and vice-verse.

In POSIX, they had base-level POSIX, realtime extensions, thread extensions, etc.  Roughly the same idea here.

But other than the language, yes, I am in agreement. I also don't know if Reactor is the right word - does it have meaning outside of Narwhal?

I'd like to see a base-level module which can:

• Start an event loop (or simulate the starting thereof)
• Enqueue arbirarty events with arbitrary arguments with arbitrary conditions

Then we can look at building other options, like setTimeout on top of those. i.e. setTimeout might look like

var evm = exports;

exports.setTimeout = function setTimeout(when, fn)
{
   when = when + Date();
   evm.enqueue(
      function(){
        fn();
        evm.dequeue(this);
      },
      function () {
        return Date.now() > when;
      }
       );
}

Note: I know the code above is wrong with respect to when stuff is evaluated. I hope you gather my meaning, though.

[Kris Kowal]

On Thu, Dec 10, 2009 at 2:35 PM, Wes Garland <w...@page.ca> wrote:
> But other than the language, yes, I am in agreement. I also don't know if
> Reactor is the right word - does it have meaning outside of Narwhal?

http://en.wikipedia.org/wiki/Reactor_pattern

> I'd like to see a base-level module which can:
> Start an event loop (or simulate the starting thereof)

I think it would be nice for our users if it were at least an option
to not have to explicitly invoke the event loop. Users will not
expect to have to do anything special to make setTimeout work.

Kris Kowal

[Kris Zyp]

Can I propose the API used by the current Narwhal impl based on
https://developer.mozilla.org/en/nsIThread?
Kris

[Ash Berlin]

Assuming you are talking about the three functions and two event related functions and not the thread+thread manager itself, it seems okay, if slightly minimal - doesn't the reactor need a way to enqueue events too?

Other worthwhile reading is http://search.cpan.org/perldoc?AnyEvent and http://search.cpan.org/perldoc?AnyEvent::Intro

-ash

[Ash Berlin]

On 10 Dec 2009, at 23:38, Ash Berlin wrote:

>
> On 10 Dec 2009, at 23:19, Kris Zyp wrote:
>
>> Kris Kowal wrote:
>>> On Thu, Dec 10, 2009 at 1:33 PM, Charles Jolley <cha...@sproutit.com> wrote:
>>>
>>>> Is it OK for me to add something or does someone else need to do it?
>>>>
>>>
>>> Go for it. We can fiddle with organization later, but Reactor/A would
>>> probably suffice.
>>>
>> Can I propose the API used by the current Narwhal impl based on
>> https://developer.mozilla.org/en/nsIThread?
>> Kris
>
>
> Assuming you are talking about the three functions and two event related functions and not the thread+thread manager itself, it seems okay, if slightly minimal - doesn't the reactor need a way to enqueue events too?

s/three functions and//

[Kris Zyp]

Here is my proposal for the reactor/event-queue:
http://wiki.commonjs.org/wiki/Reactor/A
I didn't write a timer module proposal, leave that for others.

(Also, just FYI, I did some minor updates on the promise proposal
wording (making it duck-typed) and moved it to current efforts, seemed
appropriate. Additionally I created a Workers module proposal as well:
http://wiki.commonjs.org/wiki/Worker. Let me know if there are any
objections to that).
Thanks,
Kris

[Ash Berlin]

Looking at those methods, I can't implement hasPendingEvents or getNextEvent using the boost::asio event loop I use in Zest: http://github.com/ashb/Zest/tree/timers

onIdle Could do with specifying a bit more - should it happen each time after there are events?

On a general note where did these names come from? I would have gone for simpler 'run', 'post', 'stop', 'poll' etc (this is obviously just personal choice)

It might also be worth including a function to invoke all pending events without waiting.

-ash

[Kris Zyp]

Ash Berlin wrote:
> On 11 Dec 2009, at 02:14, Kris Zyp wrote:
>
>
>> Here is my proposal for the reactor/event-queue:
>> http://wiki.commonjs.org/wiki/Reactor/A
>> I didn't write a timer module proposal, leave that for others.
>>
>> (Also, just FYI, I did some minor updates on the promise proposal
>> wording (making it duck-typed) and moved it to current efforts, seemed
>> appropriate. Additionally I created a Workers module proposal as well:
>> http://wiki.commonjs.org/wiki/Worker. Let me know if there are any
>> objections to that).
>> Thanks,
>> Kris
>>
>
> Looking at those methods, I can't implement hasPendingEvents or getNextEvent using the boost::asio event loop I use in Zest: http://github.com/ashb/Zest/tree/timers
>

The proposal already defined getNextEvent as optional, and I don't have
any objections to making hasPendingEvents optional as well.

> onIdle Could do with specifying a bit more - should it happen each time after there are events?
>

Each time after there are events and nothing is in the queue.

> On a general note where did these names come from? I would have gone for simpler 'run', 'post', 'stop', 'poll' etc (this is obviously just personal choice)
>

https://developer.mozilla.org/en/nsIThread
I don't mind changing them though.

> It might also be worth including a function to invoke all pending events without waiting.
>
>

Sure, then I could eliminate the onIdle function. processAllPending()
perhaps?
Kris

[Wes Garland]

Hi, Kris;

Reactor/A looks good to me, except I have to wonder if there shouldn't be an "ready to run?" closure passed in that can be evaluated by the loop on each pass.

As it stands now, the setTimeout() implementation would have to check the time and re-enqueue itself every time it got in the event loop.  It would be cheaper for the event loop to know not to run/remove it at all, especially since in that case it could also know that since it has been idle that it can decrease its priority.

Wes

--

You received this message because you are subscribed to the Google Groups "CommonJS" group.
To post to this group, send email to comm...@googlegroups.com.
To unsubscribe from this group, send email to commonjs+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/commonjs?hl=en.

[Charles Jolley]

Thanks for putting this up Kris!

I added a draft description for setTimeout(), clearTimeout(), setInterval() and clearInterval(). I also changed the module name from "event-queue" to "reactor". Since the name of the page is Reactor, I assumed the "event-queue" name was a typo.

I have two big issues with this API that keep me from implementing it right now. I added these to the page as well. They are:

1. If a timeout timer or an interval timer is scheduled, should the process exit when the main code finishes executing? If so, how do we blend that with the event-drive API of the browser or node.js where you run an implicit event loop? A generic wait() function won't work because you can't actually block in the browser. [Charles Jolley]

2. The browser does not support explicit control over it's event loop. This means you pretty much cannot implement any of the above methods except for enqueue(), setTimeout(), clearTimeout(), setInterval(), and clearInterval(). Perhaps the loop control methods should be made optional?


-C

[Ash Berlin]

On 11 Dec 2009, at 16:50, Charles Jolley wrote:

> Thanks for putting this up Kris!
>
> I added a draft description for setTimeout(), clearTimeout(), setInterval() and clearInterval(). I also changed the module name from "event-queue" to "reactor". Since the name of the page is Reactor, I assumed the "event-queue" name was a typo.
>
> I have two big issues with this API that keep me from implementing it right now. I added these to the page as well. They are:
>
> 1. If a timeout timer or an interval timer is scheduled, should the process exit when the main code finishes executing? If so, how do we blend that with the event-drive API of the browser or node.js where you run an implicit event loop? A generic wait() function won't work because you can't actually block in the browser. [Charles Jolley]
>
> 2. The browser does not support explicit control over it's event loop. This means you pretty much cannot implement any of the above methods except for enqueue(), setTimeout(), clearTimeout(), setInterval(), and clearInterval(). Perhaps the loop control methods should be made optional?
>

re 2. Running this in the browser is a special case and I don't think making everything optional is the right way to do this. Not every thing is going to be suitable to run in a browser, and browser timers are a well known. I think this is a case for 'this module will not work in the browser -- use its built in timers -- the rest you can't do.'

-ash

[Charles Jolley]

> re 2. Running this in the browser is a special case and I don't think making everything optional is the right way to do this. Not every thing is going to be suitable to run in a browser, and browser timers are a well known. I think this is a case for 'this module will not work in the browser -- use its built in timers -- the rest you can't do.'

Hm. Well I can see this logic for something like JSGI perhaps. But browser apps and many servers are event driven. It seems reasonable that we should have a common API for scheduling things in the event loop.

But let me broaden the #2 point: not all reactor engines will expose their event loop explicitly like this API assumes. The browser is one instance of this but servers may not expose their event loop either. If we break out the event loop control methods from the scheduling methods it gives server and browser developers more freedom to implement this API in an optimal way.

-C

[Ash Berlin]

On 11 Dec 2009, at 02:14, Kris Zyp wrote:

A thought just occurred to me - we should build a table of what features the common reactors have to see if there are other good candidates to add. The list of reactors i know about is:

* POE (perl)
* Twisted (python)
* EventMachine (ruby)
* libev (C)
* libevent (C - is it worth covering this as well as livev)

Yes I realise I am volunteering myself to build up the spreadsheet - I will do it on sunday/monday.

Any other big ones which need to be added to the list?

-ash

[Wes Garland]

On Fri, Dec 11, 2009 at 12:21 PM, Charles Jolley <cha...@sproutit.com> wrote:

> re 2. Running this in the browser is a special case and I don't think making everything optional is the right way to do this.

I don't think you need to special-case any of this.

From my perspective, you could implement the Reactor in JavaScript and run it from the browser's event loop.

Of course, in the browser you would want to defer to native setTimeout and so forth, but that is clearly an implementation detail.

**** Wait, I think enterEventQueue() is specified wrong: it should not block
 a) I don't think you can't implement it in the browser
 b) I don't think you can't implement it without threads

**** Same problem with processNextEvent when maywait = true
**** Same problem with getNextEvent's blocking

I have put together a skeleton for a reactor API which uses the browser's event loop to run it.  The changes made in this skeleton from what's in the wiki right now are:

1) Problems above w.r.t. blocking avoided by not allowing blocking APIs
2) enterEventQueue() replaced with startEventQueue()
3) My suggestion for a ready-indicator function is included  (allows for non-I/O uses, like mouse clicks, time)

var events = {
  queue: [];
  shutdown: false;
  running: false;
}

function dispatchEvent(ev)
{
  var recurr = events.queue[ev].func();

  if (recurr !== true)
    events.queue.splice(ev, 1);
}

function runEventLoop() {
  events.running = true;
  for (var ev in events.queue) {
    if (events.queue.ev.ready && events.queue.ev.ready())
      dispatchEvent(ev);
  }
  if (!events.shutdown)
    setTimeout(runEventLoop, 0);
  else
    events.running = false;
}

exports.startEventLoop = function() {
  runEventLoop();
}

exports.enqueue(task, priority, readyFunc)
{
  var qev = events.queue.push();

  qev.ready = readyFunc;
  qev.priority = priority;
  qev.sort(function(a,b) { return a.priority < b.priority }));
}

exports.shutdown = function() {
  events.shutdown = true;
}

exports.hasPendingEvents = function() {
  for (var ev in events.queue) {
    if (events.queue.ev.ready && events.queue.ev.ready())
      return true;
}

exports.processNextEvent = function() {
  for (var ev in events.queue) {
    if (events.queue.ev.ready && events.queue.ev.ready())
      dispatchEvent(ev);
  } 
}

exports.getNextEvent = function() {
  for (var ev in events.queue) {
    if (events.queue.ev.ready && events.queue.ev.ready()) {
      return = events.queue.splice(ev, 1).func;
    }
  } 
}

exports.setTimeout = window.setTimeout;
exports.clearTimeout = window.clearTimeout;
exports.setInterval = window.setInterval;
exports.clearInterval = window.clearInterval;

Comments most welcome. No, I haven't tested this.

Wes

[Ryan Dahl]

On Fri, Dec 11, 2009 at 6:22 PM, Ash Berlin
<ash_flu...@firemirror.com> wrote:
>
> A thought just occurred to me - we should build a table of what features the common reactors have to see if there are other good candidates to add. The list of reactors i know about is:
>
> * POE (perl)
> * Twisted (python)
> * EventMachine (ruby)
> * libev (C)
> * libevent (C - is it worth covering this as well as livev)
>
> Yes I realise I am volunteering myself to build up the spreadsheet - I will do it on sunday/monday.
>
> Any other big ones which need to be added to the list?
>

glib, qt, tk, native select, poll, epoll, kqueue

The AnyEvent::Impl documentation is a great resource for this:
http://kobesearch.cpan.org/htdocs/AnyEvent/

[Ryan Dahl]

On Fri, Dec 11, 2009 at 6:22 PM, Ash Berlin
<ash_flu...@firemirror.com> wrote:
>
> A thought just occurred to me - we should build a table of what features the common reactors have to see if there are other good candidates to add. The list of reactors i know about is:

The common denominator is:

1. wait for I/O (either read or write) on a file descriptor
(OS-dependent what types of fds can be used)
2. wait for timeout

Everything else is built on top of these two featres. For example,
waiting for a signal involves setting up a pipe, a signal handler that
writes to that pipe, and waiting for I/O on the read end of the pipe.

[Wes Garland]

> Everything else is built on top of these two featres. For example,
> waiting for a signal involves setting up a pipe, a signal handler that
> writes to that pipe, and waiting for I/O on the read end of the pipe.

That's an awful lot of work just to check for receipt of a signal.  On my platform, I catch signals in a separate thread, and trigger an operation callback on the context that registered the signal hander. That said, I bet our EINTR semantics differ.

Is that also your solution for timer events, mouse events, etc?

[Ryan Dahl]

On Fri, Dec 11, 2009 at 8:29 PM, Wes Garland <w...@page.ca> wrote:
>> Everything else is built on top of these two featres. For example,
>> waiting for a signal involves setting up a pipe, a signal handler that
>> writes to that pipe, and waiting for I/O on the read end of the pipe.
>
> That's an awful lot of work just to check for receipt of a signal.  On my
> platform, I catch signals in a separate thread, and trigger an operation
> callback on the context that registered the signal hander. That said, I bet
> our EINTR semantics differ.

Yeah, you probably have race conditions where I do not

http://en.wikipedia.org/wiki/Event_loop#Handling_signals
http://www.cocoadev.com/index.pl?SignalSafety
http://www.skarnet.org/software/skalibs/selfpipe.html
http://cr.yp.to/docs/selfpipe.html

> Is that also your solution for timer events, mouse events, etc?

No

[Kris Kowal]

On Fri, Dec 11, 2009 at 8:50 AM, Charles Jolley <cha...@sproutit.com> wrote:
> Since the name of the page is Reactor, I assumed the "event-queue" name was a typo.

I recommended the Reactor section only because it was the most widely
encompassing term. The module could very well be called
"event-queue", and we could have specifications for additional modules
as its scope grows.

Kris Kowal

[Wes Garland]

On Fri, Dec 11, 2009 at 3:00 PM, Ryan Dahl <coldre...@gmail.com> wrote:

Yeah, you probably have race conditions where I do not

This is not constructive, particularly given that I am reasonably certain you have not spent more than a microsecond analyzing the platform in question.

> Is that also your solution for timer events, mouse events, etc?

No

 

Neither is that.

I *would* be interested in hearing how you propose handling events which are not, by nature, I/O events.

Wes

[Ryan Dahl]

On Fri, Dec 11, 2009 at 9:44 PM, Wes Garland <w...@page.ca> wrote:
> On Fri, Dec 11, 2009 at 3:00 PM, Ryan Dahl <coldre...@gmail.com> wrote:
>>
>> Yeah, you probably have race conditions where I do not
>
> This is not constructive, particularly given that I am reasonably certain
> you have not spent more than a microsecond analyzing the platform in
> question.

Sorry for the curtness. It's just that the self-pipe trick is well
known and probably only portable solution.

>> > Is that also your solution for timer events, mouse events, etc?
>>
>> No
>

> I *would* be interested in hearing how you propose handling events which are
> not, by nature, I/O events.

Timer events are primitives on all of these event loops, as stated before.

Node is not hooked to any GUI, so it does not handle mouse events. In
X-windows, at least, mouse events are I/O events.

[Charles Jolley]

> The common denominator is:
>
> 1. wait for I/O (either read or write) on a file descriptor
> (OS-dependent what types of fds can be used)
> 2. wait for timeout

Given that most platforms will ultimately need to build on a more native event callback (like epoll, select, KQUEUE, browser, etc.) how about adopting an API that leaves the actual event loop management to the implementation?

This is more in-line with most other modern reactor libraries (see Twisted, EventMachine, for ex). It would also work equally well in both browsers and servers, so that dev's can write async code for both environments.

It would look something like:

addListener(eventType, func, [context, ...params])/removeListener()
- for events

dispatchEvent(ev, [eventType])
- send an event manually - for custom events

addTimeout(func, delay, [context, ...params])/removeTimeout()
- for one-shot timers

addInterval(func, delay, [context, ...params])/removeInterval()
- for repeating timers

next(func, [context, ...params])
- inserts the passed code into the FRONT of the event queue. In other words, this will be executed after any handlers for a current event but before additional events are processed. This allows you to defer expensive updates in your code while ensuring your app is in a consistent state before processing additional events

enqueue(func, [context, ...params])
- inserts the passed code into the event queue to execute alongside any other events. This could be a shorthand for adding a custom listener, firing a custom event, then removing the listener. However this is required often enough it is usually worth optimizing.

--
Additional event loop control methods would be implementation dependent. (In other words - when it doubt, underspecify)

-C

[Hannes Wallnoefer]

2009/12/11 Charles Jolley <cha...@sproutit.com>:

>> re 2. Running this in the browser is a special case and I don't think making everything optional is the right way to do this. Not every thing is going to be suitable to run in a browser, and browser timers are a well known. I think this is a case for 'this module will not work in the browser -- use its built in timers -- the rest you can't do.'
>
> Hm.  Well I can see this logic for something like JSGI perhaps.  But browser apps and many servers are event driven.  It seems reasonable that we should have a common API for scheduling things in the event loop.
>
> But let me broaden the #2 point: not all reactor engines will expose their event loop explicitly like this API assumes.  The browser is one instance of this but servers may not expose their event loop either.  If we break out the event loop control methods from the scheduling methods it gives server and browser developers more freedom to implement this API in an optimal way.

Thanks to everybody who's pushing this forward. The new proposals are
fine with me, but I think the timer functions should be defined
independently from (and in another module than) the reactor/event-loop
functionality.

With the event-loop, at least in its current form, you need to enter
it to make it do something. I think the timer functionality should be
a level above this - it should just work, regardless of whether the
program is running an event loop or who's spinning it.

Hannes

> -C

[Ash Berlin]

On 11 Dec 2009, at 21:16, Hannes Wallnoefer wrote:

> 2009/12/11 Charles Jolley <cha...@sproutit.com>:
>>> re 2. Running this in the browser is a special case and I don't think making everything optional is the right way to do this. Not every thing is going to be suitable to run in a browser, and browser timers are a well known. I think this is a case for 'this module will not work in the browser -- use its built in timers -- the rest you can't do.'
>>
>> Hm. Well I can see this logic for something like JSGI perhaps. But browser apps and many servers are event driven. It seems reasonable that we should have a common API for scheduling things in the event loop.
>>
>> But let me broaden the #2 point: not all reactor engines will expose their event loop explicitly like this API assumes. The browser is one instance of this but servers may not expose their event loop either. If we break out the event loop control methods from the scheduling methods it gives server and browser developers more freedom to implement this API in an optimal way.
>
> Thanks to everybody who's pushing this forward. The new proposals are
> fine with me, but I think the timer functions should be defined
> independently from (and in another module than) the reactor/event-loop
> functionality.
>
> With the event-loop, at least in its current form, you need to enter
> it to make it do something. I think the timer functionality should be
> a level above this - it should just work, regardless of whether the
> program is running an event loop or who's spinning it.
>
> Hannes

'Timers should just work' is a problem. It means the event loop has to be started by the platform, which make it harder to use multiple reactors in different programs.

This is a concern for flusspferd as one of our targets is embedding into other programs which might or might not have their own event loop already, and the embedder might not care enough to expose it to javascript

In short it just makes me slightly nervous that CommonJS platform is mandating that programs must be run with/in an event loop. Take for example the Joyent Smart platform which will be getting CommonJS sometime soon - the style/current behaviour of this platform probably makes it hard/impossible to use setTimeout (I could be wrong here -- I frequently am) but it just feels a bit.... something.

I guess it depends how we view the CommonJS spec - is it one spec, or just a core spec (require, file IO) and then a set of recommendations (if you can do timers, we recommend you do it this way)?

-ash

[Kris Zyp]

Yes, while reactor is the technical term for the pattern we are
employing, I had chosen "event-queue" for the module name because it
seemed like the most appropriate and descriptive name for a module that
handles the queue of events and looping through them. I think
"event-queue" seems easily understood and recognized.
Kris

> Kris Kowal

[Kevin Dangoor]

Coming rather late to this party. Last week was not a good week for email. I don't have much to say on this topic, other than "yes, we want this"...

On Fri, Dec 11, 2009 at 5:03 PM, Ash Berlin <ash_flu...@firemirror.com> wrote:

I guess it depends how we view the CommonJS spec - is it one spec, or just a core spec (require, file IO) and then a set of recommendations (if you can do timers, we recommend you do it this way)?

Actually, we will necessarily have pieces that are optional, depending on platform capabilities. File I/O in the browser, for example. (Or threads, should we add a threading API.) We should have APIs for these things, but recognize that some of them have to be optional.

I do like the idea of having a core standard for event loop handling, but Hannes makes a good point about setTimeout "just working" without any requirement to kick the event loop to get it going. I'd be more inclined to angle toward the ability to build powerful stuff (make the event API as discussed earlier) and if a platform wants to add some conveniences the make setTimeout just work, make that non-standard for now.

Kevin


--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

[Charles Jolley]

I do like the idea of having a core standard for event loop handling, but Hannes makes a good point about setTimeout "just working" without any requirement to kick the event loop to get it going. I'd be more inclined to angle toward the ability to build powerful stuff (make the event API as discussed earlier) and if a platform wants to add some conveniences the make setTimeout just work, make that non-standard for now.

Thinking about this more over the weekend I came to the same conclusion.  If you try to wrap timers into the same spec as events, it quickly gets too complicated.  So how about we go back to the original proposal:

a "timer" module that standardizes:

setTimeout, clearTimeout,  setInterval, clearInterval

-C

Kevin


--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

[Ash Berlin]

My only concern here is how to specify exactly which reactor/event loop a program should use. Flusspferd has already been embedded into two different game engines with their own event loops.

I guess if we standardise something like this http://redmine.flusspferd.org/issues/show/152 then we could simply have a special comment:

// event-loop: my-module/name

And have it just magically work, but allow for flexibility. And it should probably be an error or be ignored for required modules (i.e. only the main script gets to control something like that.)

[Kris Kowal]

On Tue, Dec 15, 2009 at 9:59 AM, Charles Jolley <cha...@sproutit.com> wrote:
> Thinking about this more over the weekend I came to the same conclusion.  If
> you try to wrap timers into the same spec as events, it quickly gets too
> complicated.  So how about we go back to the original proposal:
> a "timer" module that standardizes:
> setTimeout, clearTimeout,  setInterval, clearInterval

It's a good start, and easy to agree on. If someone could write it up
on the wiki, that'd be good. I'll edit the proposal to represent the
idea about making the handler type unspecified so that it can in some
implementations be a brand.

Kris Kowal

[mob]

One more suggestion for http://wiki.commonjs.org/wiki/Reactor/A. It
would be nice if processNextEvent took a timeout instead of a mayWait
boolean. If the timeout is < 0, wait forever for the next event. If
zero, don't wait. Otherwise wait for the timeout.

--mob