Stream Class Proposal

3 views
Skip to first unread message

mob

unread,
Feb 3, 2010, 1:32:32 PM2/3/10
to CommonJS
I've posted a proposal for a Stream class at: http://wiki.commonjs.org/wiki/IO/C/Stream.

To date there have been several different stream proposals (see below)
with a large variety of methods and somewhat conflicting
functionality. As the foundation for most I/O including File, Http,
Socket, Http Request, Stream filters and other classes, it is time we
had a single agreed Stream class. Due to the recent focus on async
functionality, it is essential that the Stream class work well with
both sync and async programs. Existing proposals don't adequately
address async I/O and are more complex than minimally required.

This proposal differs from past proposals in a few areas:

* It supports sync and async use cases.
* It defines events and listeners using the emitter paradigm
* It is minimal and only provides the functionality needed to
interconnect streams. (It has only 8 methods) and no properties. User
facing stream classes will typically define convenience methods to
read data. For example: readLine, readString, readDate etc.
* It defines error handling for I/O errors.
* There are no timing constraints on when and how events are issued.
This makes implementation and usage easier.

Notes:
* Streams can be half duplex or full duplex. Typically streams flow in
one direction, but there are important cases where having full duplex
streams really helps.

* Streams issue events even in sync mode. This can be useful for
notification to outside observers.

* Flow control is implicit. Events are issued when the stream can
accept more data for writing and when read data becomes available. If
data is not read, then upstream streams flow control. If too much data
is written, the write call returns with a short byte count. A single
"writable" event will be issued when the stream can accept more data.

* Events are edge triggered not level. Users don't need to throttle
events.

* It is implementation dependent as to whether or how much buffering
is provided by Streams.

Prior Art:
http://wiki.commonjs.org/wiki/IO/A
http://wiki.commonjs.org/wiki/IO/B/Stream/Level0
http://wiki.commonjs.org/wiki/IO/B/Stream/Level1
http://github.com/isaacs/ejsgi
http://wiki.accessjs.org/wiki/HTTP-Gateway

Ryan Dahl

unread,
Feb 3, 2010, 3:28:25 PM2/3/10
to comm...@googlegroups.com
On Wed, Feb 3, 2010 at 10:32 AM, mob <m...@embedthis.com> wrote:
> I've posted a proposal for a Stream class at: http://wiki.commonjs.org/wiki/IO/C/Stream.

3 comments:

The event API/proposal should be decoupled from the stream API.

Write buffering is something nearly every stream user will need. It is
a stream after all.

It's not necessary to define a synchronous API.

mob

unread,
Feb 3, 2010, 4:55:38 PM2/3/10
to CommonJS
> On Wed, Feb 3, 2010 at 10:32 AM, mob <m...@embedthis.com> wrote:
> > I've posted a proposal for a Stream class at: http://wiki.commonjs.org/wiki/IO/C/Stream.
>
> 3 comments:
>
> The event API/proposal should be decoupled from the stream API.

Agree that this proposal doesn't need to define the Emitter class, but
it does of course use them pretty heavily.

>
> Write buffering is something nearly every stream user will need. It is
> a stream after all.

Yes, but it is optional. For example: a socket stream may not buffer
at all and could write directly to the socket if going from a byte
array. The proposal does not mandate buffering either way. It is up to
the concrete stream class.

>
> It's not necessary to define a synchronous API.

Agree, there should be no separate sync API. This interface will be
used in lots of sync programs, so it must be usable in a sync manner
without events and with blocking semantics. We can't force everyone to
program async ;-)

-mob

Wes Garland

unread,
Feb 3, 2010, 6:54:19 PM2/3/10
to comm...@googlegroups.com
Hi, mob;

Some comments after a first look-through;

1 - Is this intended to be a straight-up low-level Stream API, roughly analogous in CommonJS space to the file descriptor in the UNIX space?  If so, I'd suggest dropping almost everything but read, write, and close

2 - Can we move the event API somehow into another layer (reactor?)  and still have it sensible tie together with this API?

3 - How do the listeners work, at a low level?  Watch for data on the descriptor (e.g. select), kick off the callback?  What kind of event names are we talking about?

4 - What do you think about adding an argument to read which lets you read-until a particular byte is received?   Otherwise, you have the choice of reading a byte at a time or implementing a buffering layer to implement readln().  I would have this argument replace count when the third argument is typeof "string".

5 - The difference between async and sync-mode write behaviour concerns me: I am sure this would be a locus of bugs.  Can we have a blocking/non-blocking write instead?

6 - How do you see sync-mode events working?  I don't realistically see these without a reactor.

Ryan, what does Node when the underlying write() can't dump it's entire buffer?  Do you just block? Are you able to service other requests when you're writing out the buffer?  Is your stream.write method imperative or promise/callback-based?

Both the sync and async use cases are important here.  I think is a good start. My biggest goal for this type of API, though, would be to have one API with no surprising differences between the two modes of operation.  A non-blocking read and a blocking write strike me as a sweet spot there. Thoughts?

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

mob

unread,
Feb 3, 2010, 7:22:16 PM2/3/10
to CommonJS
Wes - thanks for your comments.

> 1 - Is this intended to be a straight-up low-level Stream API, roughly
> analogous in CommonJS space to the file descriptor in the UNIX space?  If
> so, I'd suggest dropping almost everything but read, write, and close

Not quite that low level, but certainly lower than prior attempts at
Streams. I think we need all 8 methods currently specified. Perhaps
the async property could go and you could specify it via the
constructor. That would reduce to 6 methods. Hard to reduce below
that. You need addListener, but I suppose you could remove
removeListener as it is rarely (if ever) used. That would make it 5.

> 2 - Can we move the event API somehow into another layer (reactor?)  and
> still have it sensible tie together with this API?

I don't think so. You need an event Emitter definition for sure, but
the addListener belongs here. Perhaps what I hear you asking is for a
real definition of Emitter -- that for sure we need. But I wanted to
tackle one piece at a time. We could do that after this?

>
> 3 - How do the listeners work, at a low level?  Watch for data on the
> descriptor (e.g. select), kick off the callback?  What kind of event names
> are we talking about?

I think this is very implementation specific. I'm sure you could do
this many ways. For us, a stream has an instance of an Emitter that
holds a map of registered listeners When an event is fired via
emitter.emit (or something similar) is invokes all the listeners that
have registered for that event name. This may be immediate or delayed
execution depending on the platforms implementation of Emitter. The
arguments provided to emit() are passed to the listener callback.
Event names are just strings.

>
> 4 - What do you think about adding an argument to read which lets you
> read-until a particular byte is received?   Otherwise, you have the choice
> of reading a byte at a time or implementing a buffering layer to implement
> readln().  I would have this argument replace count when the third argument
> is typeof "string".

I think that is the job of TextStream or any steam that wants to
implement reading by lines. At the lowest level, that would complicate
this API. I think reading by lines is less than 50% of the uses cases
so that seems a bit out of place here. If you read a block/buffer,
then TextStream or other can pretty easily scan for line terminators.


>
> 5 - The difference between async and sync-mode write behaviour concerns me:
> I am sure this would be a locus of bugs.  Can we have a
> blocking/non-blocking write instead?

Could, but then you have asyncRead/asyncWrite and read/write.
Depending on your sync/async religion, you would prefer it the other
way round.
We've found the differences between sync/async to be fairly natural.
They are present, but expected differences. Read/write blocks in sync
mode and typically listens to no events. Otherwise behavior is pretty
similar.

In sync mode, you wait for an event, then you read/write. Once you've
got the event -- the read/write operates the same. The one difference
is async write which can return short. But you have to check for this
because you may be out of memory or get an I/O error. You can't assume
that a write will always work when you are not blocking.

If you platform provides unlimited buffering, then you could just
write and hope it all works, but someday, you'll get a low memory
condition.

>
> 6 - How do you see sync-mode events working?  I don't realistically see
> these without a reactor.

These I agree are rare. We don't use them. However, it seemed
unnatural to exclude them. You could say events are only for async
mode.
What we implement is that events only get emitted if there are
listeners, so there is no cost to events if there are no listeners.

> Both the sync and async use cases are important here.  I think is a good
> start. My biggest goal for this type of API, though, would be to have one
> API with no surprising differences between the two modes of operation.  A
> non-blocking read and a blocking write strike me as a sweet spot there.
> Thoughts?

I've tried to get as close to this goal as I could. The biggest trade
off is a common read/write with an async property vs separate read/
write. Given the similarities, I went for a common read / write.

Thanks for taking the time to review.

-mob

Ryan Dahl

unread,
Feb 4, 2010, 10:47:43 AM2/4/10
to comm...@googlegroups.com
On Wed, Feb 3, 2010 at 3:54 PM, Wes Garland <w...@page.ca> wrote:
> Ryan, what does Node when the underlying write() can't dump it's entire
> buffer?  Do you just block? Are you able to service other requests when
> you're writing out the buffer?  Is your stream.write method imperative or
> promise/callback-based?

Node buffers the data. Later a 'drain' event is emitted when the
kernel send buffer is available again. I stole this API from
AnyEvent::Handle. I think it should be present in a commonjs streaming
interface.

mob

unread,
Feb 4, 2010, 1:46:59 PM2/4/10
to CommonJS

Your drain event seems to be very similar to the proposed "writable"
event. The difference is you get a writable event to begin with.
The current proposal permits your current behavior for buffering. One
difference is the return code from write permits the writer to know if
there is a write error.

So I think we already have a "drain" capability. What API did you
think should be present beyond the proposal?

-mob

Louis Santillan

unread,
Feb 4, 2010, 2:13:49 PM2/4/10
to comm...@googlegroups.com
On Wed, Feb 3, 2010 at 3:54 PM, Wes Garland <w...@page.ca> wrote:
[SNIP]

> 5 - The difference between async and sync-mode write behaviour concerns me:
> I am sure this would be a locus of bugs.  Can we have a
> blocking/non-blocking write instead?
[SNIP]

> Both the sync and async use cases are important here.  I think is a good
> start. My biggest goal for this type of API, though, would be to have one
> API with no surprising differences between the two modes of operation.  A
> non-blocking read and a blocking write strike me as a sweet spot there.
> Thoughts?

I think trying to unify an async and sync API is a going to always be
problematic. What needs to be done is to pick async or sync as a
base, low-level interface and let that API implement the other.

WRT streams in general, I think the JavaME MIDP 2.0 profile gets
streams pretty close to right and lite (at least for sync streams,
http://java.sun.com/javame/reference/apis/jsr118/java/io/package-summary.html
and http://java.sun.com/javame/reference/apis/jsr118/javax/microedition/io/package-summary.html).
I especially like how URI formatted strings are passed to
javax.microedition.io.Connector.open*() for all streams.
tcp://host:port, file://path/to/file, http://host/resource are all
valid URIs for open(). And at a bare minimum, separate a InputStream
and OutputStream seems to keep things a lot simpler too (3 methods for
Output [close,flush,write], and 7 for Input
[available,close,mark,markSupported,read,reset,skip]).

Louis Santillan

unread,
Feb 4, 2010, 2:29:36 PM2/4/10
to comm...@googlegroups.com
I'd like to point out a few things IO/C/Stream.

1) The constructor assumes that all given streams can be opened R/W.
Permissions, physical device properties (ROM-like devices), etc., may
dictate R or W only. How is the constructor to resolve this?
2) The constructor describes stacking or chaining but not in a clear
manner. Can you to create a "circular" stream, such that
StreamA.write produces data for StreamB.read and Stream.B.write
produces data for StreamA.read?
3) Having a stream being able to do read and write is a little
semantically messy. I prefer separate Input and Output Streams or a
Stream that can create in Input & Output Streams in a factory method.
4) There is no seekability defined. Is that intended to be
implemented by higher-level classes? That could force higher-level
interfaces to create buffering contracts on streams that may not
support it non-destructively.

mob

unread,
Feb 4, 2010, 3:39:18 PM2/4/10
to CommonJS
> I think trying to unify an async and sync API is a going to always be
> problematic.  What needs to be done is to pick async or sync as a
> base, low-level interface and let that API implement the other.

That is the question. Is it better to have 2 Streams, 2 Files, 2 Http,
2 Socket etc. One for async and one for sync.
The goal of this proposal is to avoid that. I agree that each Stream
interface will be simpler if you have 2, but the overall complexity I
think is worse having two worlds.

Imagine if Posix had 2 reads and writes instead of a read and write
system call that can be used with sync and async.

>
> WRT streams in general, I think the JavaME MIDP 2.0 profile gets

> streams pretty close to right and lite (at least for sync streams,http://java.sun.com/javame/reference/apis/jsr118/java/io/package-summ...
>  and  http://java.sun.com/javame/reference/apis/jsr118/javax/microedition/i...).


>  I especially like how URI formatted strings are passed to
> javax.microedition.io.Connector.open*() for all streams.

> tcp://host:port, file://path/to/file,http://host/resourceare all


> valid URIs for open().  And at a bare minimum, separate a InputStream
> and OutputStream seems to keep things a lot simpler too (3 methods for
> Output [close,flush,write], and 7 for Input
> [available,close,mark,markSupported,read,reset,skip]).

Let me read that. It has been a while since I've done MIDP
programming.

-mob

mob

unread,
Feb 4, 2010, 3:44:26 PM2/4/10
to CommonJS
> 1) The constructor assumes that all given streams can be opened R/W.
> Permissions, physical device properties (ROM-like devices), etc., may
> dictate R or W only.  How is the constructor to resolve this?

The concrete classes must provide that.

> 2) The constructor describes stacking or chaining but not in a clear
> manner.  Can you to create a "circular" stream, such that
> StreamA.write produces data for StreamB.read and Stream.B.write
> produces data for StreamA.read?

You could add code to prevent circles. That would be for the
implementation to do. This however is a classically "hard" problem.

> 3) Having a stream being able to do read and write is a little
> semantically messy.  I prefer separate Input and Output Streams or a
> Stream that can create in Input & Output Streams in a factory method.

I'll wait for more input on this from others. I prefer a single Stream
rather than having Input, Output and InputOutput. It is a trade off
having 2 Streams vs having one. Of course, if you do your earlier post
recommended Async and Sync, then you would have 4 streams. AsyncInput,
AsyncOutput, SyncInput, SyncOutput.

I think having one Stream is simpler for the whole system. But that
may just be my preference having used such a system for a while.

> 4) There is no seekability defined.  Is that intended to be
> implemented by higher-level classes?  That could force higher-level
> interfaces to create buffering contracts on streams that may not
> support it non-destructively.

Agree. There is no seek, tell, or other operations that apply only to
streams that support such features. Of course, there is nothing to
stop a stream class implementing these.

-mob

Ryan Dahl

unread,
Feb 4, 2010, 4:16:00 PM2/4/10
to comm...@googlegroups.com

Sure - your API definitely handles it - I just think it could be
easier - without the first "writable" event.

In my API I would also have write() return true or false depending on
if the entire data was flushed to the kernel or not.

mob

unread,
Feb 4, 2010, 4:20:55 PM2/4/10
to CommonJS

Having an initial writable event means you can make all writing hang
of the event if you want to.

Does the first writable event prevent your use cases working, or it
just not required in your case?

>
> In my API I would also have write() return true or false depending on
> if the entire data was flushed to the kernel or not.

How do you detect errors and what do you do if:

1. The write fails

2. There is insufficient memory to buffer the write

-mob

Louis Santillan

unread,
Feb 4, 2010, 5:19:46 PM2/4/10
to comm...@googlegroups.com
On Thu, Feb 4, 2010 at 12:39 PM, mob <m...@embedthis.com> wrote:
>> I think trying to unify an async and sync API is a going to always be
>> problematic.  What needs to be done is to pick async or sync as a
>> base, low-level interface and let that API implement the other.
>
> That is the question. Is it better to have 2 Streams, 2 Files, 2 Http,
> 2 Socket etc. One for async and one for sync.
> The goal of this proposal is to avoid that. I agree that each Stream
> interface will be simpler if you have 2, but the overall complexity I
> think is worse having two worlds.
>
> Imagine if Posix had 2 reads and writes instead of a read and write
> system call that can be used with sync and async.
>

Actually, POSIX does have duplicate read/write methods
(http://www.gnu.org/s/libc/manual/html_node/Asynchronous-Reads_002fWrites.html#Asynchronous-Reads_002fWrites
and http://www.gnu.org/s/libc/manual/html_node/Asynchronous-Reads_002fWrites.html#Asynchronous-Reads_002fWrites).
I agree with you about the rapid growth of interfaces required to
implement separate sync and async; it becomes laborious. However,
this is the path many core lib system providers have taken (Java,
POSIX, etc.). I would say there is something to learn in those
approaches if CommonJS would like to try to marry sync and async
calls.

Wes Garland

unread,
Feb 4, 2010, 5:22:17 PM2/4/10
to comm...@googlegroups.com
Food for thought --

What happens if we try to describe a bottom-most-level Stream which is non-blocking?  Other Stream types -- sync and async -- could be built on top of those primitives.

Louis Santillan

unread,
Feb 4, 2010, 5:37:40 PM2/4/10
to comm...@googlegroups.com
On Thu, Feb 4, 2010 at 1:20 PM, mob <m...@embedthis.com> wrote:
> On Feb 4, 1:16 pm, Ryan Dahl <coldredle...@gmail.com> wrote:
>> On Thu, Feb 4, 2010 at 10:46 AM, mob <m...@embedthis.com> wrote:
>> > On Feb 4, 7:47 am, Ryan Dahl <coldredle...@gmail.com> wrote:
[SNIP]

>> > So I think we already have a "drain" capability. What API did you
>> > think should be present beyond the proposal?
>>
>> Sure - your API definitely handles it - I just think it could be
>> easier - without the first "writable" event.
>
> Having an initial writable event means you can make all writing hang
> of the event if you want to.
>
> Does the first writable event prevent your use cases working, or it
> just not required in your case?

Take the approach that WebSockets does. The events are onopen, onclose, ondata.

>>
>> In my API I would also have write() return true or false depending on
>> if the entire data was flushed to the kernel or not.
>
> How do you detect errors and what do you do if:
>
> 1. The write fails
>
> 2. There is insufficient memory to buffer the write

WebSockets fail here, but you would have onerror. onerror also gives
a lead-in to the retry/flush/die issue.

Louis Santillan

unread,
Feb 4, 2010, 6:10:44 PM2/4/10
to comm...@googlegroups.com
On Thu, Feb 4, 2010 at 2:22 PM, Wes Garland <w...@page.ca> wrote:
> What happens if we try to describe a bottom-most-level Stream which is
> non-blocking?  Other Stream types -- sync and async -- could be built on top
> of those primitives.

Taking this approach, async inherits naturally from the low-level
Stream. sync might then require locking & timeOut logic to be
built-in each I/O operation (open/read/write/close). async as base
also necessitates a threads/event loop/"libevent-ish" capability for
all implementations to implement core I/O. In the land of the
multi-core, multi-cpu, multi-io card servers this will be a big
performance win. In small devices, maybe not so much.

I guess it depends on what you feel is more expensive to maintain, a
VM that is naturally threaded or a VM that threads to do I/O.

mob

unread,
Feb 5, 2010, 2:42:30 AM2/5/10
to CommonJS

On Feb 4, 2:19 pm, Louis Santillan <lpsan...@gmail.com> wrote:
> On Thu, Feb 4, 2010 at 12:39 PM, mob <m...@embedthis.com> wrote:
> >> I think trying to unify an async and sync API is a going to always be
> >> problematic.  What needs to be done is to pick async or sync as a
> >> base, low-level interface and let that API implement the other.
>
> > That is the question. Is it better to have 2 Streams, 2 Files, 2 Http,
> > 2 Socket etc. One for async and one for sync.
> > The goal of this proposal is to avoid that. I agree that each Stream
> > interface will be simpler if you have 2, but the overall complexity I
> > think is worse having two worlds.
>
> > Imagine if Posix had 2 reads and writes instead of a read and write
> > system call that can be used with sync and async.
>
> Actually, POSIX does have duplicate read/write methods

> (http://www.gnu.org/s/libc/manual/html_node/Asynchronous-Reads_002fWri...
> andhttp://www.gnu.org/s/libc/manual/html_node/Asynchronous-Reads_002fWri...).


>  I agree with you about the rapid growth of interfaces required to
> implement separate sync and async; it becomes laborious.  However,
> this is the path many core lib system providers have taken (Java,
> POSIX, etc.).   I would say there is something to learn in those
> approaches if CommonJS would like to try to marry sync and async
> calls.

Definitely always something to learn. Even bad ideas have good gems to
mine. However, the other lesson to learn from this sync/async mess in
Windows and in *NIX in general is that grafting on async I/O later
typically leads to a bag on the side. I would not regard those
examples as clean or elegant. I think we can do better if we design
sync/async in at the very lowest level and have every API above use
the same semantics.

-mob

mob

unread,
Feb 5, 2010, 2:44:56 AM2/5/10
to CommonJS

On Feb 4, 2:22 pm, Wes Garland <w...@page.ca> wrote:
> Food for thought --
>
> What happens if we try to describe a bottom-most-level Stream which is
> non-blocking?  Other Stream types -- sync and async -- could be built on top
> of those primitives.

Not sure I fully understand this. Stream is an interface not a
concrete class. If say the Socket class wants to support sync and
async capabilities in one API it should implement either:

- Two interfaces: one for sync and one for async
- One interface: for both sync and async

If you create a async only Stream, and then create a derrived
interface for Sync. I don't see how that helps create Socket and make
its API simple and easy to use?

-mob

mob

unread,
Feb 5, 2010, 2:45:45 AM2/5/10
to CommonJS

Good idea. Having an error event is a good idea and fits nicely into
the eventing paradigm. Thanks

-mob

Mike Wilson

unread,
Feb 18, 2010, 7:21:08 AM2/18/10
to comm...@googlegroups.com
[catching up... very late to the party]

mob wrote:
> I've posted a proposal for a Stream class at:
> http://wiki.commonjs.org/wiki/IO/C/Stream.
>

> This proposal differs from past proposals in a few areas:
>
> * It supports sync and async use cases.

I can see that the direction you've taken is something like
sockets (configuring them blocking or non-blocking).

For this generic stream case I'm unsure whether it is really
desired to configure the blocking behaviour on the stream
itself, as this may lead to side-effects between stream users.
Will this not lead to API docs stating "this method expects
an async stream" and stream returning methods to ask for a sync
flag, thereby putting more burden on APIs?

I interpret the goal as being able to have one stream instance
serve both sync and async clients, and for that I would let
the client somehow indicate if he wants sync or async in every
call to the stream.

Sync:
s.read() // blocking method

Async alt 1:
s.asyncRead() // non-blocking method

Async alt 2:
s.read(..., /*async*/ true) // non-blocking flag

Async alt 3:
if (s.isReadable()) s.read() // check blocking first

For sync usage it would also be good to have methods
that indicate read/writability of the stream (isReadable above,
also see Java's available), instead of having to listen for
related events.

> * Streams can be half duplex or full duplex. Typically
> streams flow in one direction, but there are important
> cases where having full duplex streams really helps.

I have to admit that I am more used to (and prefer) single-
direction streams. A dual direction channel is easily
constructed from a stream pair (cin/cout etc). Is there any
precedent for full duplex streams?

Problems I can see include:
- What to do with the write() method on a stream that can
only support reading (such as HTTP.get...)? Should it not
exist on the stream instance, or exist but always throw an
exception?
- It may in practice lead to the notion of InputStream,
OutputStream and InputOutputStream in APIs and docs, while
a split solution would have only the first two.
- There may be an explosion of method names to be able to
configure both directions of the stream:
setReadBufferSize
setWriteBufferSize
setReadEncoding
setWriteEncoding
etc...

Finally; would it be interesting to also specify how
selection (POSIX select) may be performed on a collection
of streams?

Best regards
Mike Wilson

mob

unread,
Feb 18, 2010, 12:54:24 PM2/18/10
to CommonJS

The trade-off taken in this proposal is that a stream stack tends to
(nearly always) be either async or sync. Rarely do use cases mix sync
and async on one stream at the same time. Often an entire app is async
or sync. So it seemed to make sense to have a single call read()
because the reader typically knows the context (async or sync). The
alternative, having both sync+async read and write can certainly work
as you propose, but if the proposal's assumption is true -- then a
single API may be simpler.

If using in sync mode, do you mean isReadable should mean the stream
is a read stream or that read data is present?

Full duplex streams have been used inside network protocol stacks for
a while. Less often in user space. While they may seem unfamiliar at
first, there is little mental overhead as you use them exactly the
same as single duplex if that is what you need.

Yes, you are right. On a stream that is half-duplex, the corresponding
read/write should error and return null. BTW, Http would typically be
full-duplex as you write your request and read the response.

Concrete classes can always have an explosion of names by providing
read/write APIs. But they would have that if we had Input and Output
streams anyway. So this kind of Stream isn't really the cause for
that.

You could add a select Posix style for Streams. You can also use the
listen facility to listen for events on a set of streams and use a
single function to common the events. That is in effect a multiple
select.

-mob

Mike Wilson

unread,
Feb 18, 2010, 3:36:43 PM2/18/10
to comm...@googlegroups.com
mob wrote:
> The trade-off taken in this proposal is that a stream stack
> tends to (nearly always) be either async or sync. Rarely do
> use cases mix sync and async on one stream at the same time.
> Often an entire app is async or sync. So it seemed to make
> sense to have a single call read() because the reader
> typically knows the context (async or sync).

I see, this is a different goal from my first impression.
Having the same API have different semantics in different
cases feels a little risky IMO. Practically it also doubles
the number of stream sub-types that need to be distinguished
between when deciding on APIs:
InputSyncStream
InputAsyncStream
OutputSyncStream
OutputAsyncStream
InputOutputSyncStream
InputOutputAsyncStream

> If using in sync mode, do you mean isReadable should mean
> the stream is a read stream or that read data is present?

Data is present. (like "available" in Java)

> You could add a select Posix style for Streams. You can
> also use the listen facility to listen for events on a set
> of streams and use a single function to common the events.
> That is in effect a multiple select.

Whichever solution is recommended I think it might be a good
thing to mention in the Streams proposal.

Best regards
Mike

mob

unread,
Feb 18, 2010, 4:11:47 PM2/18/10
to CommonJS

On Feb 18, 12:36 pm, "Mike Wilson" <mike...@hotmail.com> wrote:
> mob wrote:
> > The trade-off taken in this proposal is that a stream stack
> > tends to (nearly always) be either async or sync. Rarely do
> > use cases mix sync and async on one stream at the same time.
> > Often an entire app is async or sync. So it seemed to make
> > sense to have a single call read() because the reader
> > typically knows the context (async or sync).
>
> I see, this is a different goal from my first impression.
> Having the same API have different semantics in different
> cases feels a little risky IMO. Practically it also doubles

Why risky, it is no different to Posix read()?

> the number of stream sub-types that need to be distinguished
> between when deciding on APIs:
>   InputSyncStream
>   InputAsyncStream
>   OutputSyncStream
>   OutputAsyncStream
>   InputOutputSyncStream
>   InputOutputAsyncStream

Sorry, I don't understand this, can you explain?

>
> > If using in sync mode, do you mean isReadable should mean
> > the stream is a read stream or that read data is present?
>
> Data is present. (like "available" in Java)

Ah, you mean someone is using sync mode, but doesn't want to block for
some reason?
That is not very common. Probably better for concrete Steam classes to
provide that if relevant to them.

>
> > You could add a select Posix style for Streams. You can
> > also use the listen facility to listen for events on a set
> > of streams and use a single function to common the events.
> > That is in effect a multiple select.
>
> Whichever solution is recommended I think it might be a good
> thing to mention in the Streams proposal.

Definitely the event listener style is the preferred / recommended
approach. Nothing would prevent an implementation doing a select()
style API outside of the Stream.

-mob

Mike Wilson

unread,
Feb 18, 2010, 5:05:05 PM2/18/10
to comm...@googlegroups.com
mob wrote:
> On Feb 18, 12:36 pm, "Mike Wilson" <mike...@hotmail.com> wrote:
> > I see, this is a different goal from my first impression.
> > Having the same API have different semantics in different
> > cases feels a little risky IMO. Practically it also doubles
>
> Why risky, it is no different to Posix read()?

Sure, risky is probably too-a-harsh word. Either way, you can
find examples of both styles in different environments and I
prefer the more explicit "this-API-always-behaves-the-same" style,
to avoid programmer error or frustration :-).

> > the number of stream sub-types that need to be distinguished
> > between when deciding on APIs:
> >   InputSyncStream
> >   InputAsyncStream
> >   OutputSyncStream
> >   OutputAsyncStream
> >   InputOutputSyncStream
> >   InputOutputAsyncStream
>
> Sorry, I don't understand this, can you explain?

What I mean is that if you have two code modules (possibly
from different sources) where one module is supplying a stream
to the other, they need to agree on one of those six stream
types, ie they must agree on direction+duplex and on sync/async.

But then again, with this design maybe all stream creation
points will grow multiple variants:
syncstream = file.open(...)
asyncstream = file.openAsync(...)
and that may be ok too.

Best regards
Mike

mob

unread,
Feb 18, 2010, 7:04:45 PM2/18/10
to CommonJS

I think that is the key point. This proposal is a minimalist. full-
duplex Stream with async and sync combined in one read/write API.

There are other Stream proposals and you could define yet another that
does exactly what you would like. It would be better for you to create
a new proposal that does this because it is a different concept. This
is an open forum and you are more than welcome to propose an
alternative. The more ideas before we coalesce the better.

Then we can choose between the various options. I think this is better
than changing this proposal to become the opposite of what it was
meant to propose. ie. A minimalistic, single API that be a Stream
interface for sync and async I/O. Clear alternatives leads to easier
decision making.

-mob

Reply all
Reply to author
Forward
0 new messages