JoinBlob & ByteStore - creating FileBlobs when data gets large?

[Brad Garcia]

As part of the blob download work, Charles had begun to implement a ByteStore.  It actually looks to be quite similar to the BlobBuilder that I've created as part of the pending JoinBlob CL.

The one main feature that ByteStore has that is currently missing from BlobBuilder is the ability to store data into a FileBlob when the data becomes too large.  In BlobBuilder, we always push (non-blob) data into a BufferBlob, and I currently have a TODO for handling data that is too large to fit into a BufferBlob.

What would everybody think about having a size threshold (most likely different for desktop/mobile platforms) above which we would create a FileBlob to hold the data?  If we did have this ability, would we want it to apply universally, or choose whether to allow it (have a BlobBuilder constructor argument, for instance)?

 -- Brad

[Nigel Tao]

On Tue, Jun 3, 2008 at 01:48, Brad Garcia <bga...@google.com> wrote:
> What would everybody think about having a size threshold (most likely
> different for desktop/mobile platforms) above which we would create a
> FileBlob to hold the data? If we did have this ability, would we want it to
> apply universally, or choose whether to allow it (have a BlobBuilder
> constructor argument, for instance)?

Sounds reasonable to me. The file-submitter stuff also has a need for
temporary files, so maybe we can share some code.

[Chris Prince]

I hadn't heard about 'ByteStore' before, and I couldn't find many
details in the changelist. What's the intended use?

[Charles Fry]

The use case is for Blob downloads. When downloading bytes from an
HTTP request (especially if Content-Length is not specified) results
in a potentially large byte stream, which may be too large to hold in
memory. The idea is to have an abstraction to which the
browser-specific classes performing the byte download can just write
blindly to, with the abstraction initially storing the bytes in
memory, and once there were "too many" swapping them out to disk. Much
of this was based on discussions with bpm before he left, and sadly
I've been distracted enough by other tasks that I haven't got back to
this one enough to even discuss it in more detail on the mailing list.
FWIW, Brad's comment originated in our Pittsburgh Gears meeting where
he realized that my long-pending CL shared a lot in common with his
current one.

Charles

[Chris Prince]

Ah, thanks, that helps a ton.

So yeah, I think it makes perfect sense for large blobs to spill over
to file-backed storage. And making that threshold be an arbitrary
number (which may differ by platform) seems fine.

Is there a plan for how to cleanup storage that is no longer
referenced? I guess spilled data only needs to last for the current
browser session.

[Charles Fry]

> Is there a plan for how to cleanup storage that is no longer
> referenced? I guess spilled data only needs to last for the current
> browser session.

Exactly. There is no specific plan yet; I wanted to be sure that there
was agreement about moving in this direction before we spent more time
brainstorming.

Charles

[Chris Prince]

BTW, I don't think we're gated on this feature, are we? It seems like
we could launch Blob support without the spill-to-disk feature.

(That doesn't mean we shouldn't be discussing this now. It's good
that we're thinking ahead. Then when the need arises, we can spend
the week or two of work and implement this without much overhead.)

[Charles Fry]

True, Blob download would just be for completeness.

That said, I think it makes sense for Brad to overflow into this as he
completes the other Blob functionality which he has begun.

Charles