Low-level async API proposal

44 views
Skip to first unread message

inimino

unread,
Feb 2, 2010, 1:05:08 PM2/2/10
to nod...@googlegroups.com
Based on discussions in the Promises, callbacks,... thread, I
suggest the following as an API for node on which everyone can
build their preferred abstractions easily.

Proposed low-level async file I/O API:

Each function takes two required callbacks as the first
parameters, followed by any other arguments.

Example:

posix.cat(success_callback,error_callback,path,encoding)

posix.read(success_callback,error_callback,fd,length,position,encoding)

Rationale:

Since the method-specific arguments come last, they can still
be optional.

Both callbacks are required. If arguments.length < 2 (plus
any required method-specific arguments) then the call throws
immediately.

The callbacks do not have to do any magical or error-prone
type testing to determine whether the result is an error or
success result, and there is no 'either' wrapper around the
value. It's simple and unambiguous, either one or the other
callback will be called.

No synchronous support in this API, that can be a separate
API.

Keep in mind this is a low-level API, and most people writing
e.g. web apps using node will be expected to use some kind
of higher-level abstraction such as promises, continuables,
or something else.

Thoughts?

--
http://inimino.org/~inimino/blog/

r...@tinyclouds.org

unread,
Feb 2, 2010, 1:11:50 PM2/2/10
to nod...@googlegroups.com
On Tue, Feb 2, 2010 at 10:05 AM, inimino <ini...@inimino.org> wrote:
> Based on discussions in the Promises, callbacks,... thread, I
> suggest the following as an API for node on which everyone can
> build their preferred abstractions easily.
>
> Proposed low-level async file I/O API:
>
> Each function takes two required callbacks as the first
> parameters, followed by any other arguments.
>
> Example:
>
> posix.cat(success_callback,error_callback,path,encoding)
>
> posix.read(success_callback,error_callback,fd,length,position,encoding)
>

Nah, let's just use error codes. We can rename it to 'fs' to avoid confusion.
They'll be just the process.fs methods.

inimino

unread,
Feb 2, 2010, 1:42:21 PM2/2/10
to nod...@googlegroups.com
On 2010-02-02 11:11, r...@tinyclouds.org wrote:
> Nah, let's just use error codes. We can rename it to 'fs' to avoid confusion.
> They'll be just the process.fs methods.

Clarification from IRC:

Using just one callback function, with separate arguments:

<ryah> the first arg of the callback can always be reserved for error. fs.unlink("/tmp/foo", function (error) {})

Sounds great!

--
http://inimino.org/~inimino/blog/

Karl Guertin

unread,
Feb 2, 2010, 1:59:00 PM2/2/10
to nod...@googlegroups.com
On Tue, Feb 2, 2010 at 1:42 PM, inimino <ini...@inimino.org> wrote:
> Using just one callback function, with separate arguments:
>
> <ryah> the first arg of the callback can always be reserved for error. fs.unlink("/tmp/foo", function (error) {})

So that'd be:

var cb = function(err,data) {
if(err) {
sys.write('err');
} else {
sys.write(data.slice(0,15);
}
};

posix.cat(cb, '/path/to/file', 'UTF-8');

right?

r...@tinyclouds.org

unread,
Feb 2, 2010, 2:03:47 PM2/2/10
to nod...@googlegroups.com

As of 9874412 (for process.fs methods)

Joran Greef

unread,
Feb 2, 2010, 2:04:33 PM2/2/10
to nodejs
Putting the callbacks first will have the command arguments showing up
a couple lines or so down the page after the callback definitions. The
command arguments should rather be visible right next to the command
and the callback arguments should come last. This better matches the
control flow.

A difficulty with providing fine-grained interfaces such as POSIX etc.
is that they often take a couple arguments, and one can never remember
the order. With optional arguments and callbacks it gets complex.

Prototype did something great to solve this by using a pattern where
an interface is defined as "methodName(mostCommonArgument, options =
{})". Where options is an extensible object allowing plenty of space
for options and callbacks.

Tim Caswell

unread,
Feb 2, 2010, 2:07:15 PM2/2/10
to nod...@googlegroups.com
Agreed, I really prefer the callback as the last argument. I usually inline my functions and I don't want to have to scroll 30 lines to see the filename I'm opening.

The options object as a second arguments works well.

Also it's not hard to make this second option optional with a simple type check to see if the second parameter is an object or a function.

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>

inimino

unread,
Feb 2, 2010, 2:08:50 PM2/2/10
to nod...@googlegroups.com
On 2010-02-02 11:59, Karl Guertin wrote:
> So that'd be:
>
> var cb = function(err,data) {
> if(err) {
> sys.write('err');
> } else {
> sys.write(data.slice(0,15);
> }
> };
>
> posix.cat(cb, '/path/to/file', 'UTF-8');
>
> right?

Yep.

--
http://inimino.org/~inimino/blog/

inimino

unread,
Feb 2, 2010, 2:12:01 PM2/2/10
to nod...@googlegroups.com
Sorry, I replied too soon.

On 2010-02-02 11:59, Karl Guertin wrote:

> So that'd be:
>
> var cb = function(err,data) {
> if(err) {
> sys.write('err');
> } else {
> sys.write(data.slice(0,15);
> }
> };

This is correct.

> posix.cat(cb, '/path/to/file', 'UTF-8');

The callback comes last, so:

posix.cat('/path/to/file','UTF-8',cb)

This does away with optional arguments, but that's perfectly
fine for a low-level interface.

--
http://inimino.org/~inimino/blog/

Ryan Dahl

unread,
Feb 2, 2010, 2:14:38 PM2/2/10
to nod...@googlegroups.com

(posix.cat is not a low-level function)
And if you exclude the cb then it becomes sync.

process.fs.unlink("/tmp/blah"); // sync

// async
process.fs.unlink("/tmp/blah", function (error) {
if (!error) puts("successfully deleted /tmp/blah");
}

inimino

unread,
Feb 2, 2010, 2:15:08 PM2/2/10
to nod...@googlegroups.com
On 2010-02-02 12:04, Joran Greef wrote:
> Putting the callbacks first will have the command arguments showing up
> a couple lines or so down the page after the callback definitions. The
> command arguments should rather be visible right next to the command
> and the callback arguments should come last. This better matches the
> control flow.

This is a reason why I use the Continuable style:

cat("path/to/file","optional encoding","other optional args...")
(callback)

But most people won't be using the low-level functions directly, they'll
be using the convenient APIs we build on top of them, so I think Ryan's
approach is well-suited.

--
http://inimino.org/~inimino/blog/

Joran Greef

unread,
Feb 2, 2010, 2:44:29 PM2/2/10
to nodejs
Yes. Leaving out the callback to make it synchronous is brilliant. Was
going to bring it up myself but thought it would cause a ruckus.

process.fs.unlink("/tmp/blah", function(success) {}, function(error)
{}) // Would be more awesome.

Ray Morgan

unread,
Feb 2, 2010, 2:46:52 PM2/2/10
to nod...@googlegroups.com
I worry about making the sync API so simple.. afraid that people will just use it that way.

What about something a bit more descriptive:

process.fs.unlink("/tmp/blah", "SYNC"); // sync

This way you know the intention was to make it sync, and also it kind of stands out.

-Ray

Joran Greef

unread,
Feb 2, 2010, 3:27:28 PM2/2/10
to nodejs
Re: "I worry about making the sync API so simple.. afraid that people

will just use it that way."

I was afraid someone might say that.

"There seems to be a perverse human characteristic that likes to make
easy things difficult... it's likely to continue this way. Ships will
sail around the world, but the Flat Earth Society will flourish..."

"Resist the urge to punish everyone for one person's mistake." -
http://sivers.org/punish

inimino

unread,
Feb 2, 2010, 3:32:17 PM2/2/10
to nod...@googlegroups.com
On 2010-02-02 13:27, Joran Greef wrote:
> Re: "I worry about making the sync API so simple.. afraid that people
> will just use it that way."

Just like it should be harder to use an API in a way that silently
ignores errors, it should be harder to use an API in a way that is
slow. (Not as much harder, but harder.)

> "There seems to be a perverse human characteristic that likes to make
> easy things difficult... it's likely to continue this way. Ships will
> sail around the world, but the Flat Earth Society will flourish..."

There's nothing in the least perverse about making the simple,
obvious use also the safest and most performant.

This is why I suggested making the sync API an entirely separate
module.

--
http://inimino.org/~inimino/blog/

Paul Barry

unread,
Feb 2, 2010, 8:18:23 PM2/2/10
to nod...@googlegroups.com
What am I missing here?  Doesn't making a Sync call completely defeat the purpose of using node?  Isn't this about making the wrong things difficult, to prevent people from shooting themselves in the foot?  Why is there even a Sync option?

Ray Morgan

unread,
Feb 2, 2010, 9:25:12 PM2/2/10
to nod...@googlegroups.com
There are times where a sync option is faster than the async option (when the OS caches a commonly used file for instance). But, most of the time in an app you should be using the async version.

My idea of:

process.fs.unlink("/tmp/blah", "SYNC");

Doesn't in any way (IMHO) make the program more difficult or harder to write... but it does stand out in a way that screams, "Hey FYI I am a synchronous call". I think this is a better approach than the just leaving the callback out b/c people that are learning Node might see the callback as a burden (and not realize the benefits) and opt for the sync version b/c it looks simple and clean. With the added 'SYNC' they hopefully will at least see that something is a bit strange...

Remember, we have been trained to think sync I/O so: process.fs.unlink("/tmp/blah"); feels right even though it is not usually the best method.

-Ray

Matthew Dempsky

unread,
Feb 2, 2010, 8:39:43 PM2/2/10
to nod...@googlegroups.com
On Tue, Feb 2, 2010 at 5:18 PM, Paul Barry <paulj...@gmail.com> wrote:
> What am I missing here?  Doesn't making a Sync call completely defeat the
> purpose of using node?  Isn't this about making the wrong things difficult,
> to prevent people from shooting themselves in the foot?  Why is there even a
> Sync option?

I agree. I don't like the idea of having any blocking APIs available.

If the issue is that writing code using a blocking API would be easier
than using existing non-blocking APIs, then the better solution is to
make the non-blocking APIs easier to use or to come up with syntax
extensions that let the user hide the underlying non-blocking nature
when desired.

E.g., I don't see anything wrong (at least in principle) with
automatically translating something like

var x := foo();
[...use x...]

to

foo().addCallback(function(x) {
[...use x...]
});

Tim Caswell

unread,
Feb 2, 2010, 11:49:40 PM2/2/10
to nod...@googlegroups.com
In IRC earlier, ryah and tlrobinson talked about moving to narwhal based requires. Those requires have hooks that allow for such pre-processing tricks. I've been playing with designing languages that compile to javascript, but make all this ugly async syntax prettier.

However, I do think there is value is making the API for raw JavaScript to be as elegant as possible without breaking other ideals. (Performance, correctness, elegance)

Joran Greef

unread,
Feb 3, 2010, 12:49:28 AM2/3/10
to nodejs
Re: "Isn't this about making the wrong things difficult, to prevent

people from shooting themselves in the foot?"

No. It's about designing a first class API for people who know how to
program.

Tim Caswell

unread,
Feb 3, 2010, 1:01:14 AM2/3/10
to nod...@googlegroups.com
Actually according to the nodejs.org website:

"...less-than-expert programmers are able to develop fast systems."

That paragraph is talking about the lack of threads and all the headache they bring, but I read that to mean that Ryan's goal was to make a safe framework for fast software.

I could be misinterpreting here, but that's my take on it.

Personally I don't care if there are sync functions there or not. They are much easier for people who just want to get something done and don't care about extreme speed and concurrency.

r...@tinyclouds.org

unread,
Feb 3, 2010, 1:06:37 AM2/3/10
to nod...@googlegroups.com
On Tue, Feb 2, 2010 at 10:01 PM, Tim Caswell <t...@creationix.com> wrote:
> Actually according to the nodejs.org website:
>
> "...less-than-expert programmers are able to develop fast systems."
>
> That paragraph is talking about the lack of threads and all the headache they bring, but I read that to mean that Ryan's goal was to make a safe framework for fast software.
>
> I could be misinterpreting here, but that's my take on it.
>
> Personally I don't care if there are sync functions there or not.  They are much easier for people who just want to get something done and don't care about extreme speed and concurrency.

I want to have sync methods because fs syscalls often do not spin the
disk. Sending the request to the thread pool, the context switch, and
pulling it back into the main thread is often massive overhead for
something that was a constant time calculation anyway. Thus sync
methods. I don't have any moral problems with this.

I do have major worries about continuing to support wait() and, to a
lesser extent, the synchronous require().

Matthew Dempsky

unread,
Feb 3, 2010, 2:11:01 AM2/3/10
to nod...@googlegroups.com
On Tue, Feb 2, 2010 at 10:06 PM, <r...@tinyclouds.org> wrote:
> I want to have sync methods because fs syscalls often do not spin the
> disk. Sending the request to the thread pool, the context switch, and
> pulling it back into the main thread is often massive overhead for
> something that was a constant time calculation anyway. Thus sync
> methods. I don't have any moral problems with this.

My worry is you have developers that are trying to max out performance
of their code in benchmarks, so they use these APIs because in their
synthetic benchmarks they always hit the disk cache anyway. But then
in production, there are other apps that are competing for the disk
cache, and consequently node becomes randomly unresponsive because the
main thread keeps blocking.

I'd rather developers take a hard line that their JS won't block the
main thread, suck up the fixed performance loss, and if it's *really*
an issue, push on the OS people to provide a better solution for async
and/or non-blocking disk IO. Just not providing sync APIs seems like
the best way to ensure this happens.

I'm not worried about sync APIs simply being available if people don't
misuse them, but I've just never known a language feature to go
unabused if it allowed eeking out extra performance.

Reply all
Reply to author
Forward
0 new messages