Database with streaming blob support

[Matt]

What options are out there currently for streaming blob support in Node.js databases?

I know that MongoDB has GridFS which seems to support streams. Are there any others?

[Aria Stewart]

On May 20, 02014, at 10:00, Matt <hel...@gmail.com> wrote:

> What options are out there currently for streaming blob support in Node.js databases?
>
> I know that MongoDB has GridFS which seems to support streams. Are there any others?

LevelDB, of course.

[Matt]

On Tue, May 20, 2014 at 11:00 AM, Aria Stewart <ared...@nbtsc.org> wrote:

LevelDB, of course

I don't see the support in LevelDB for treating a single value (i.e. db.get(key)) as a true stream efficiently. It appears that one 'data' event contains the entire value. Am I reading the docs wrong? I'm looking for large blob support (tens of megabytes potentially).

[Aria Stewart]

Ah, right — it itself acts as a stream. The interface for single keys isn’t. Sorry about that!

[Luca Morandini]

On 21/05/14 00:00, Matt wrote:
> What options are out there currently for streaming blob support in Node.js databases?
>
> I know that MongoDB has GridFS which seems to support streams. Are there any others?

I suppose you mean "DBMS supported by Node,js", rather than "Node.js databases".

Anyway, I streamed successfully fairly sizeable binary attachments (600MB) to/from
CouchDB with Nano (Node.js driver for CouchDB), by piping directly to the response
(or, conversely, from the request body to the attachment):

var stream = datastore.db.attachment.get(docid, "blob",
function(errStream) {
if (errStream) {
datastore.commons.logger.error(
"Doc. %s load attachment error: %s", docid,
errStream.message);
return datastore.swagger.stopWithError(res, {
code : 500,
reason : errStream.message
});
}
});

if (stream) {
stream.pipe(res);
} else {
return datastore.swagger.stopWithError(res, {
code : 500,
reason : "Error creating stream"
});
}
...

Regards,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini

[Floby]

I'm currently streaming images and soon videos to a CouchDB database, with cradle sometimes, but mostly directly with request and/or superagent.

Putting blob attachments is a simple as `myBlob.pipe(request.put('http://my.databa.se/db/doc/attachment'))`

Getting them is even easier `request('http://my.databa.se/db/doc/attachment').pipe(myResultStream_UsuallyAnHttpResponseObject)`

If you're in the context of a web server, you get ETag caching for free.

[Alexey Petrushin]

s3? Distributed file systems?