Issue 3807 in dart: pause/continue for throttling IO stream processing

[da...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 3807 by is...@google.com: pause/continue for throttling IO stream
processing
http://code.google.com/p/dart/issues/detail?id=3807

An IO stream consumer (e.g. callback handler of an InputStream) is not able
to throttle / pause the stream,. While processing the data, it is not able
to fire async tasks to e.g. write in a database, wait for the result and
continue the stream with the next entry, because as soon as it fires the
async event, it will receive the next entry. The only option is to keep
track of each Future reference, essentially risking (and reaching) out of
memory errors.

IO callback-driven classes, especially the following callbacks shall be
controlled through a simple pause/continue mechanism:
- DirectoryLister.onFile
- DirectoryLister.onDir
- InputStream.onData
- StringInputStream.onData
- StringInputStream.onLine

The pause() / continue() methods might the callback function's parameter,
but I think it is better to have them on the class level. In the later
case, pauseProcessing() / continueProcessing() could be a longer (more
precise) name for them.

Additional info / related examples in the following thread:
https://groups.google.com/a/dartlang.org/group/misc/browse_thread/thread/2a75f47f15a57d1f#

[da...@googlecode.com]

Comment #1 on issue 3807 by is...@google.com: pause/continue for throttling

IO stream processing
http://code.google.com/p/dart/issues/detail?id=3807

Mads added:
"The streaming interfaces already have the pausing functionality in the
sense that you can set the onData and onLine handler to null and the
data will stay in the OS buffers until you read it out. When you set
back an onData or onLine handler they will resume. I'm not entirely
sure about the directory lister, but I think it will just drop
directories and files if the handlers are set to null. It might make
sense to have some way of pausing a directory lister."

[da...@googlecode.com]

Updates:
Status: Triaged
Labels: -Type-Defect Type-Enhancement Area-IO

Comment #2 on issue 3807 by seth...@google.com: pause/continue for

throttling IO stream processing
http://code.google.com/p/dart/issues/detail?id=3807

(No comment was entered for this change.)

[da...@googlecode.com]

Comment #3 on issue 3807 by sgj...@google.com: pause/continue for

throttling IO stream processing
http://code.google.com/p/dart/issues/detail?id=3807

All IO streams have now been changed to use the dart:async Stream class.
This provides support for flow control.

Currently Directory.list will not forward the pause to the native thread
posting file/directory/link messages.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[da...@googlecode.com]

Updates:
Cc: v...@google.com kat...@google.com ag...@google.com financeC...@gmail.com
seth...@google.com

Comment #4 on issue 3807 by sgj...@google.com: pause/continue for

throttling IO stream processing
http://code.google.com/p/dart/issues/detail?id=3807

Issue 6981 has been merged into this issue.

[da...@googlecode.com]

Updates:
Status: Fixed
Owner: ajoh...@google.com

Comment #5 on issue 3807 by ajoh...@google.com: pause/continue for

throttling IO stream processing
http://code.google.com/p/dart/issues/detail?id=3807

We now support pause/resume on directory listing. However, do note that the
listing is buffered on several levels (e.g. glibc will buffer ~2048
entries).