Low-level async API proposal
inimino
suggest the following as an API for node on which everyone can
build their preferred abstractions easily.
Proposed low-level async file I/O API:
Each function takes two required callbacks as the first
parameters, followed by any other arguments.
Example:
posix.cat(success_callback,error_callback,path,encoding)
posix.read(success_callback,error_callback,fd,length,position,encoding)
Rationale:
Since the method-specific arguments come last, they can still
be optional.
Both callbacks are required. If arguments.length < 2 (plus
any required method-specific arguments) then the call throws
immediately.
The callbacks do not have to do any magical or error-prone
type testing to determine whether the result is an error or
success result, and there is no 'either' wrapper around the
value. It's simple and unambiguous, either one or the other
callback will be called.
No synchronous support in this API, that can be a separate
API.
Keep in mind this is a low-level API, and most people writing
e.g. web apps using node will be expected to use some kind
of higher-level abstraction such as promises, continuables,
or something else.
Thoughts?
r...@tinyclouds.org
> Based on discussions in the Promises, callbacks,... thread, I
> suggest the following as an API for node on which everyone can
> build their preferred abstractions easily.
>
> Proposed low-level async file I/O API:
>
> Each function takes two required callbacks as the first
> parameters, followed by any other arguments.
>
> Example:
>
> posix.cat(success_callback,error_callback,path,encoding)
>
> posix.read(success_callback,error_callback,fd,length,position,encoding)
>
Nah, let's just use error codes. We can rename it to 'fs' to avoid confusion.
They'll be just the process.fs methods.
inimino
> Nah, let's just use error codes. We can rename it to 'fs' to avoid confusion.
> They'll be just the process.fs methods.
Clarification from IRC:
Using just one callback function, with separate arguments:
<ryah> the first arg of the callback can always be reserved for error. fs.unlink("/tmp/foo", function (error) {})
Sounds great!
Karl Guertin
> Using just one callback function, with separate arguments:
>
> <ryah> the first arg of the callback can always be reserved for error. fs.unlink("/tmp/foo", function (error) {})
So that'd be:
var cb = function(err,data) {
if(err) {
sys.write('err');
} else {
sys.write(data.slice(0,15);
}
};
posix.cat(cb, '/path/to/file', 'UTF-8');
right?
r...@tinyclouds.org
As of 9874412 (for process.fs methods)
Joran Greef
a couple lines or so down the page after the callback definitions. The
command arguments should rather be visible right next to the command
and the callback arguments should come last. This better matches the
control flow.
A difficulty with providing fine-grained interfaces such as POSIX etc.
is that they often take a couple arguments, and one can never remember
the order. With optional arguments and callbacks it gets complex.
Prototype did something great to solve this by using a pattern where
an interface is defined as "methodName(mostCommonArgument, options =
{})". Where options is an extensible object allowing plenty of space
for options and callbacks.
Tim Caswell
The options object as a second arguments works well.
Also it's not hard to make this second option optional with a simple type check to see if the second parameter is an object or a function.
> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
inimino
> So that'd be:
>
> var cb = function(err,data) {
> if(err) {
> sys.write('err');
> } else {
> sys.write(data.slice(0,15);
> }
> };
>
> posix.cat(cb, '/path/to/file', 'UTF-8');
>
> right?
Yep.
inimino
On 2010-02-02 11:59, Karl Guertin wrote:
> So that'd be:
>
> var cb = function(err,data) {
> if(err) {
> sys.write('err');
> } else {
> sys.write(data.slice(0,15);
> }
> };
This is correct.
> posix.cat(cb, '/path/to/file', 'UTF-8');
The callback comes last, so:
posix.cat('/path/to/file','UTF-8',cb)
This does away with optional arguments, but that's perfectly
fine for a low-level interface.
Ryan Dahl
(posix.cat is not a low-level function)
And if you exclude the cb then it becomes sync.
process.fs.unlink("/tmp/blah"); // sync
// async
process.fs.unlink("/tmp/blah", function (error) {
if (!error) puts("successfully deleted /tmp/blah");
}
inimino
> Putting the callbacks first will have the command arguments showing up
> a couple lines or so down the page after the callback definitions. The
> command arguments should rather be visible right next to the command
> and the callback arguments should come last. This better matches the
> control flow.
This is a reason why I use the Continuable style:
cat("path/to/file","optional encoding","other optional args...")
(callback)
But most people won't be using the low-level functions directly, they'll
be using the convenient APIs we build on top of them, so I think Ryan's
approach is well-suited.
Joran Greef
going to bring it up myself but thought it would cause a ruckus.
process.fs.unlink("/tmp/blah", function(success) {}, function(error)
{}) // Would be more awesome.
Ray Morgan
What about something a bit more descriptive:
process.fs.unlink("/tmp/blah", "SYNC"); // sync
This way you know the intention was to make it sync, and also it kind of stands out.
-Ray
Joran Greef
will just use it that way."
I was afraid someone might say that.
"There seems to be a perverse human characteristic that likes to make
easy things difficult... it's likely to continue this way. Ships will
sail around the world, but the Flat Earth Society will flourish..."
"Resist the urge to punish everyone for one person's mistake." -
http://sivers.org/punish
inimino
> Re: "I worry about making the sync API so simple.. afraid that people
> will just use it that way."
Just like it should be harder to use an API in a way that silently
ignores errors, it should be harder to use an API in a way that is
slow. (Not as much harder, but harder.)
> "There seems to be a perverse human characteristic that likes to make
> easy things difficult... it's likely to continue this way. Ships will
> sail around the world, but the Flat Earth Society will flourish..."
There's nothing in the least perverse about making the simple,
obvious use also the safest and most performant.
This is why I suggested making the sync API an entirely separate
module.
Paul Barry
Ray Morgan
Matthew Dempsky
> What am I missing here? Doesn't making a Sync call completely defeat the
> purpose of using node? Isn't this about making the wrong things difficult,
> to prevent people from shooting themselves in the foot? Why is there even a
> Sync option?
I agree. I don't like the idea of having any blocking APIs available.
If the issue is that writing code using a blocking API would be easier
than using existing non-blocking APIs, then the better solution is to
make the non-blocking APIs easier to use or to come up with syntax
extensions that let the user hide the underlying non-blocking nature
when desired.
E.g., I don't see anything wrong (at least in principle) with
automatically translating something like
var x := foo();
[...use x...]
to
foo().addCallback(function(x) {
[...use x...]
});
Tim Caswell
However, I do think there is value is making the API for raw JavaScript to be as elegant as possible without breaking other ideals. (Performance, correctness, elegance)
Joran Greef
people from shooting themselves in the foot?"
No. It's about designing a first class API for people who know how to
program.
Tim Caswell
"...less-than-expert programmers are able to develop fast systems."
That paragraph is talking about the lack of threads and all the headache they bring, but I read that to mean that Ryan's goal was to make a safe framework for fast software.
I could be misinterpreting here, but that's my take on it.
Personally I don't care if there are sync functions there or not. They are much easier for people who just want to get something done and don't care about extreme speed and concurrency.
r...@tinyclouds.org
> Actually according to the nodejs.org website:
>
> "...less-than-expert programmers are able to develop fast systems."
>
> That paragraph is talking about the lack of threads and all the headache they bring, but I read that to mean that Ryan's goal was to make a safe framework for fast software.
>
> I could be misinterpreting here, but that's my take on it.
>
> Personally I don't care if there are sync functions there or not. They are much easier for people who just want to get something done and don't care about extreme speed and concurrency.
I want to have sync methods because fs syscalls often do not spin the
disk. Sending the request to the thread pool, the context switch, and
pulling it back into the main thread is often massive overhead for
something that was a constant time calculation anyway. Thus sync
methods. I don't have any moral problems with this.
I do have major worries about continuing to support wait() and, to a
lesser extent, the synchronous require().
Matthew Dempsky
> I want to have sync methods because fs syscalls often do not spin the
> disk. Sending the request to the thread pool, the context switch, and
> pulling it back into the main thread is often massive overhead for
> something that was a constant time calculation anyway. Thus sync
> methods. I don't have any moral problems with this.
My worry is you have developers that are trying to max out performance
of their code in benchmarks, so they use these APIs because in their
synthetic benchmarks they always hit the disk cache anyway. But then
in production, there are other apps that are competing for the disk
cache, and consequently node becomes randomly unresponsive because the
main thread keeps blocking.
I'd rather developers take a hard line that their JS won't block the
main thread, suck up the fixed performance loss, and if it's *really*
an issue, push on the OS people to provide a better solution for async
and/or non-blocking disk IO. Just not providing sync APIs seems like
the best way to ensure this happens.
I'm not worried about sync APIs simply being available if people don't
misuse them, but I've just never known a language feature to go
unabused if it allowed eeking out extra performance.