Google Groups

Re: [tiddlyweb] binary content in tiddlyweb


bengillies May 3, 2012 10:34 AM
Posted in group: TiddlyWeb
I meant to reply to this a while ago but never got round to it, so...


> 1 How important is it that binary uploads and downloads in tiddlyweb
>  are efficient?

How much are binary tiddlers slowing things down by on different
TiddlyWeb instances (e.g. tiddlyspace.com) in general? If large binary
tiddlers slow things down a lot, but aren't generally PUT/GET very
often, then it's likely not that important, on the other hand, if they
are, then it is.

> 2 If that is important, how important is it that store and serialization
>  code changes made to accomodate binaries take account of existing
>  tiddlyweb plugin code and maintain backward compatibility?

I'm not sure there's _that_ much code that exists that deals with
binary tiddlers directly, so I'm not sure it would be _that_ much
effort to patch things up. From my point of view, I'm quite happy to
make any necessary changes to my various plugins and push them out at
the same time to keep things working if it's needed.

> 3 How imporant is it that binary tiddlers remain raw tiddlers or could
>  binary tiddlers be pointers to binary content?

They could be, we'd likely have to make several changes to the various
clientside plugins that deal with them (not _that_ big a deal). Though
IIRC, large binary tiddlers already appear as links anyway.

> The changes I have in mind for #2 would allow serialization as_tiddler
> to accept either a string or a file handle (currently just a string) and
> allow stores to provide or use tiddler objects (on tiddler_get and
> tiddler_put) that have a text field which can be a file handle or string
> (currently a string). The file handle would kick in when content
> matches certain types or incoming content length is over some
> configurable length.
>
> A different option (#3) would add a layer of indirection[1] for the
> creation of a binary tiddler:
>
> * Do a raw PUT of the binary content to a separate store optimized for
>  binaries (direct fd -> file streaming, no in memory middle step).
> * Get back a URI for the binary content.
> * Create a JSON tiddler that references that URI, and does tagging and
>  fields.
> * When doing a GET for the tiddler, the binary URI is dereferenced in
>  different ways (depending on the Accept header).

What do you mean binary URI? Are you talking about a separate URI, or
just treating the standard URI in different ways depending on the
Accept header (e.g. request as JSON, txt, etc -> retrieve that
standard tiddler, request as default, image/jpeg, etc -> retrieve from
the binary store). If the latter, that seems fairly sensible.

> So. To restate the question: Who cares about binary tiddlers and in
> what fashion do you care?

I care to the extent that I want to be able to upload images to
TiddlySpace without abusing things too much (that is, it's possible
now, but better to use something else and hotlink (assuming the
something else permits that of course)). I don't really do video, pdf,
doc, etc, but if TiddlyWeb is supposed to hold notes, ideas, whatever,
then I think it's important not to restrict the format that the notes
come in.


Ben