Binary Tiddlers in TiddlySpace
Ben Gillies
Paul Downey (psd)
> Referencing this thread in TiddlyWikiDevhttp://groups.google.com/group/tiddlywikidev/browse_thread/thread/540...
> TiddlySpace.
>
> As I see it, this forms 4 separate things:
>
> 1) People upload images/some other sort of binary file
> 2) People upload svg/javascript/some other sort of text based document
> 3) People upload videos/mp3s/some other sort of really big object
> 4) People upload executables/scripts/some other sort of potentially
> dangerous file
>
> I'll start with the easiest question(s) first:
>
> Q1 Regarding (4), is it a good idea to restrict the types of files people
> can upload (my opinion on this is yes, it is)? If so, which types of file do
> we explicitly allow?
others, e.g. application/atom+xml and what? application/zip
> Q2 TiddlyWeb currently doesn't work very well with large binary files (eg
> videos). Do we want to restrict the size of files we can upload, or include
> better support for large files (my opinion is better support for large
> files)
limitation explicit, and based on filesize, however this is a zero,
one, infinity problem:
http://catb.org/jargon/html/Z/Zero-One-Infinity-Rule.html
> Q3 There is currently no interface showing progress on uploaded files. Is
> this something we want, and if so, how do we accomplish it (my opinion is
> that this would be awesome, and could be tied in to answer to Q2 somehow.
http://www.flickr.com/photos/upload/
http://piotrsarnacki.com/2008/06/18/upload-progress-bar-with-mod_passenger-and-apache/
There are others ..
> Q4 There seems to be a lot of confusion over how binary tiddlers should be
> displayed/edited.
PUT http://example.tiddlyspace.com/foo.png Content-type: image/png
PUT http://example.tiddlyspace.com/foo.svg Content-type: image/svg
+xml
PUT http://example.tiddlyspace.com/foo.wav Content-type: audio/wav
PUT http://example.tiddlyspace.com/foo.ogg Content-type: audio/ogg
I then expect:
GET http://example.tiddlyspace.com/foo.png to return an image/png
GET http://example.tiddlyspace.com/foo.svg to return an image/svg
+xml
GET http://example.tiddlyspace.com/foo.mp3 to return a audio/mp3
GET http://example.tiddlyspace.com/foo.ogg to return a audio/ogg
In the TiddlyWiki I then expect to be able to add the following to my
wikitext:
[img[/foo.png]]
<<image /SiteIcon.svg>>
//actually I'd like to be able to write [img[SiteIcon.svg]]
<html>
<audio>
<source src=/foo.mp3">
<source src=/foo.ogg">
</audio>
</html>
// which similarly could become a formatter
[audio[foo.mp3,foo.ogg]]
In the TiddlyWiki if I view the tiddler:
title: "foo.png"
[the rendered image]
tags: [ .... ]
title: "foo.svg"
[the rendered image]
tags: [ .... ]
title: "foo.mp3"
[nothing, unless there is a view macro for "audio/mp3"]
tags: [ .... ]
title: "foo.ogg"
[nothing, unless there is a view macro for "audio/ogg"]
tags: [ .... ]
and if I edit the tiddler:
title: "foo.png"
data:AAFFD43243A.........
tags: [ .... ]
title: "foo.svg"
<svg><g ......./></svg>
tags: [ ....]
title: "foo.mp3"
data:AFA9234AF....
tags: [ ....]
i.e text [and XML] may be editable as text, binary as base64
In addition, I'd expect to be able to manipulate these binary tiddlers
using .tid, .json, .txt etc:
http://example.tiddlyspace.com/foo.png.tid
http://example.tiddlyspace.com/foo.png.txt
http://example.tiddlyspace.com/foo.png.json
And that's it!
Of course with some magic content-negotiation, it might be nice to:
PUT http://example.tiddlyspace.com/foo Content-type: image/png
and then be able to:
GET http://example.tiddlyspace.com/foo.png Content-type: image/png
GET http://example.tiddlyspace.com/foo.jpg Content-type: image/jpeg
GET http://example.tiddlyspace.com/foo.jpg Content-type: image/gif
etc
by writing a TiddlyWeb ImageMagick serializer, but that's beyond this
use-case.
For example, an svg image I would expect to be included in
> the TiddlyWiki as text, displayed as an image, with the text only appearing
> when you click edit; likewise, a normal image, I would expect to be included
> as a link to the image on the server, displayed as that image, with an
> upload box allowing you to upload a new version when you click edit. These
> examples are my own opinion, so may likely be wrong, but my question is
> this: Which file types do we include special behaviour for, which special
> behaviour is it exactly that we include, and what (if anything) do we fall
> back to if there is no special behaviour for said file type?
type,
which may need some new code, I'm guessing keyed off the content-type
field.
HTH
--
Paul (psd)
http://blog.whatfettle.com
Paul Downey
> /me dons asbestos suit
good luck with the asbestosis!
>> 4) People upload executables/scripts/some other sort of potentially
>> dangerous file
>> [...]
>> Q1 Regarding (4), is it a good idea to restrict the types of files people
>> can upload (my opinion on this is yes, it is)?
>
> Can you provide a specific example of what might be dangerous and why?
I think that's a good question. Maybe it's not our business to block .exe files
>> Q2 TiddlyWeb currently doesn't work very well with large binary files (eg
>> videos). Do we want to restrict the size of files we can upload, or include
>> better support for large files
>
> I consider TiddlySpace (also raw TiddlyWeb) a platform for sharing content -
> which to me is primarily textual. With that in mind, I regard binary
> tiddlers (most commonly images) as taking a supportive role.
That's an interesting POV. I see TiddlyWeb as a RESTful store for
blobs of data, akin to S3 with enough metadata to form tiddlers, and
mechanisms for assembling those tiddlers into TiddlyWikis.
I don't see anything conflicting in our views, just legs and trunk of
the same elephant.
> Thus I'm not in favor of trying to provide extensive file-hosting
> capabilities. While one might argue that a video can be just as "supportive"
> as an image, this seems to go beyond the original design. There are better
> hosts for that type of content. (It's not inconceivable to user/create a
> separate, adjacent service dedicated to file hosting.)
Right. I think I'd continue to use flickr for hosting images, vimeo
for videos, and other places for large audio files, but for reasons of
size and having an interface and UI better suited to photos, videos,
audio etc. OTOH it seems strange to preclude storing images, videos,
audio.
>> Q3 There is currently no interface showing progress on uploaded files. Is
>> this something we want, and if so, how do we accomplish it
>
> It would be nice, but I'm not sure about the complexities involved. Given
> the above, I don't see it as a priority.
It's possible. I wouldn't say it was high priority.
>> Q4 There seems to be a lot of confusion over how binary tiddlers should be
>> displayed/edited.
[snip]
> So basically, my expectation is that just TiddlyWeb provides the raw data
> and the client determines what to do with it.
Agreed.
> That leaves actual binary data, whose handling should also be determined by the client
> - which normally means it should be made available to the client in base64.
Absolutely! I really don't like the current tiddler with conjured up
non-editable wikitext.
> It might also be worth considering to serve binary tiddlers in skinny form
> in the TiddlyWiki serialization and use lazy loading - but that might then
> require special handling for the unplugged scenario (which is possibly
> inevitable anyway).
That sounds good to me.
Ben Gillies
>> 4) People upload executables/scripts/some other sort of potentiallyI think that's a good question. Maybe it's not our business to block .exe files
>> dangerous file
>> [...]
>> Q1 Regarding (4), is it a good idea to restrict the types of files people
>> can upload (my opinion on this is yes, it is)?
>
> Can you provide a specific example of what might be dangerous and why?
That's an interesting POV. I see TiddlyWeb as a RESTful store for
>> Q2 TiddlyWeb currently doesn't work very well with large binary files (eg
>> videos). Do we want to restrict the size of files we can upload, or include
>> better support for large files
>
> I consider TiddlySpace (also raw TiddlyWeb) a platform for sharing content -
> which to me is primarily textual. With that in mind, I regard binary
> tiddlers (most commonly images) as taking a supportive role.
blobs of data, akin to S3 with enough metadata to form tiddlers, and
mechanisms for assembling those tiddlers into TiddlyWikis.
> Thus I'm not in favor of trying to provide extensive file-hosting
> capabilities. While one might argue that a video can be just as "supportive"
> as an image, this seems to go beyond the original design. There are better
> hosts for that type of content. (It's not inconceivable to user/create a
> separate, adjacent service dedicated to file hosting.)
Right. I think I'd continue to use flickr for hosting images, vimeo
for videos, and other places for large audio files, but for reasons of
size and having an interface and UI better suited to photos, videos,
audio etc. OTOH it seems strange to preclude storing images, videos,
audio.
> That leaves actual binary data, whose handling should also be determined by the clientAbsolutely! I really don't like the current tiddler with conjured up
> - which normally means it should be made available to the client in base64.
non-editable wikitext.
That sounds good to me.
> It might also be worth considering to serve binary tiddlers in skinny form
> in the TiddlyWiki serialization and use lazy loading - but that might then
> require special handling for the unplugged scenario (which is possibly
> inevitable anyway).
chris...@gmail.com
> How about some sort of .exe trojan?
What about a .exe trojan that is named foo.gif?
This is one of those situations where it is very hard to tell without
a lot of serverside checking/complexity.
Avoiding complexity on both the server and the client is an important
goal here because complexity is fraught with error. If there is a
expectation that the content stored is valid and safe then the code
which checks for validity and safety needs to be correct. Making it
correct is really really hard.
> I think it's primarily textual at the moment, sure. However, I don't see why
> it has to stay that way. Some sort of wiki based on videos/images with
> tagging/metadata would be "really cool".
Storing those videos and images in the tiddlywiki would be "not cool".
If I were to personally create a wiki that is about videos and images
with metadata, I would store the metadata in the wiki, and URIs to the
videos and images located in other storage systems that are more
suited to binary content.
The content delivery systems in TiddlyWeb are designed for delivery
small chunks of content: tiddlers. Which are defined as small blocks
of microcontent.
That TiddlyWeb _can_ deliver large binaries is a side effect of the
fact that anything can be represented as a stream of bytes. However
TiddlyWeb's delivery of that stuff is not efficient in any of cpu
usage, I/O usage, or HTTP protocol usage.
(The above is just info to provide context for the discussion.)
> Some sort of co-existing service better designed to handle binary content is
> an interesting concept, yes.
In my opinion I think they call that "the web". No co-existence
required, just URIs.
> I quite like the idea of combining text/video/images into a single wiki.
> Flickr/Vimeo/YouTube/etc having a better UI is besides the point: the UI is
> exactly what we're discussing.
Well, no, not quite, I think we are talking about several things and
there is no "exactly" about it and that's part of why the conversation
won't resolve to a conclusion.
One of the things we are discussing is how a tiddlyspace hosted tiddlywiki
will allow a person to manipulate binary stuff. But right alongside that is
how a tiddlywiki will itself manipulate binary stuff. These things are
separate concerns, but require one another.
> What would the point of seeing the base64 data of a binary image upon
> clicking edit be (just curious)? How would you be expected to update that?
> What about large images? If somebody uploads a few (say 3) large images, you
> could easily end up serving a TiddlyWiki multiple MBs in size. Something
> that would presumably take quite a while.
I believe the goal is that if the base64 content is there, then an
editor for that type of content can be built into the tiddlywiki. The
tiddlywiki can, when edit is clicked, recognize the content and "do
the right thing".
That goal (above) directly conflicts with the concern represented by
your statement about "that would presumably take quite a while".
_If_ we want to allow binary editing in TiddlyWiki _and_ we want to
preserve offline capability, and we want to do it in a way that
doesn't require a bunch of extra smarts and conditionals, then we need
the binary content in the tiddlywiki.
_If_ we have binary content in the tiddlywiki, then we present the
risk that that binary content is going to create very large (and slow)
tiddlywikis.
> Isn't that similar to what already happens (aka - a link to the image)? I'm
> not sure the unplugged scenario needs excellent binary support (past
> "pseudo-binary" of course).
It's _very_ import, from the server-side perspective that the
tiddlywiki generated for the unplugged scenario is not significantly
different from the tiddlywiki generated from the plugged scenario.
Right now the difference between foo.wiki and
foo.wiki?download=foo.html is that there is a Content-Disposition
header added to the HTTP response.
If we want something different, it could potentially add a lot of code
and the value proposition doesn't feel strong enough for that (yet).
(More in response to other messages.)
--
Chris Dent http://burningchrome.com/~cdent/
[...]
chris...@gmail.com
> That's an interesting POV. I see TiddlyWeb as a RESTful store for
> blobs of data, akin to S3 with enough metadata to form tiddlers, and
> mechanisms for assembling those tiddlers into TiddlyWikis.
It's kind of like that, except it is very consciously optimized in the
direction of small textual things. The major deal here is that say you
have a giant mp3 in the store that you want to GET as JSON. The entire
mp3 is read into memory before becoming a JSON string. Both of those
collections of (many) bytes will be in memory at the same time.
Similar things if the mp3 is being encoded into a tiddlywiki.
There would be efficiency value in _only_ including the URI of the
pure content-type representation of a binary tiddler in the tiddlywiki
and JSON representations, and then optimizing the storage and delivery
of the pure[1] representation so it can be streamed direct from the
storage.
Doing that, however, interacts poorly with the "edit this binary in my
tiddlywiki" desire.
[1] By pure I mean where the resource is not processed before being
sent over the pipe, it is just delivered as is.
> Absolutely! I really don't like the current tiddler with conjured up
> non-editable wikitext.
Nor do I, but this was done explicitly to guard against the tiddlywiki
getting a tiddler which contains many megabytes of unusable stuff,
that when edited, breaks the binary tiddler (by corrupting the
content).
Until we resolve "how to deal with binary tiddlers in tiddlywiki"
we're sort of stuff.
chris...@gmail.com
> I guess it's pretty much a given that we need to distinguish "real" and
> "pseudo" binaries, the latter being text-based (i.e. content type text/*) and
> thus not too different from wikitext.
I _think_ we've resolved that text/* and */*+xml will be considered
psuedo binaries. Anybody else have additions?
> It might also be worth considering to serve binary tiddlers in skinny form in
> the TiddlyWiki serialization and use lazy loading - but that might then
> require special handling for the unplugged scenario (which is possibly
> inevitable anyway).
If we can define and strictly constrain what that special handling might
be for the unplugged scenario (which might be as simple as "don't be
lazy here if download query parameter is set) the above seems like a
good idea, as you don't need to search (in the tiddlywiki itself)
binary tiddlers so them not being present early on in the store is not
a big deal.
However, for big tiddlers, while they are lazy loading it would be
nice to have some status spinner to indicate that work is going on (as
the current message is static).
So let's have a strawman, for sake of knocking it down:
* the server continues to accept binaries like it does[1]
* the wiki serialization sends text/* and *+xml as psuedo binaries
* real binaries are sent without text set at all, thus being lazy
* if download parameter is set, include the base64 encode stuff in
* text
* the JSON serialization of binaries (the one's being lazy loaded) is
base64encoded and the client side knows to turn that into a data:uri
when lazy loading stuff
In what ways is this bad? In what ways is it good?
[1] and we worry about limits later/separately
Ben Gillies
On Thu, 15 Jul 2010, Ben Gillies wrote:What about a .exe trojan that is named foo.gif?
How about some sort of .exe trojan?
Storing those videos and images in the tiddlywiki would be "not cool".I think it's primarily textual at the moment, sure. However, I don't see why
it has to stay that way. Some sort of wiki based on videos/images with
tagging/metadata would be "really cool".
In my opinion I think they call that "the web". No co-existenceSome sort of co-existing service better designed to handle binary content is
an interesting concept, yes.
required, just URIs.
_If_ we want to allow binary editing in TiddlyWiki _and_ we want to
preserve offline capability, and we want to do it in a way that
doesn't require a bunch of extra smarts and conditionals, then we need
the binary content in the tiddlywiki.
Ben Gillies
On Thu, 15 Jul 2010, FND wrote:I _think_ we've resolved that text/* and */*+xml will be considered
I guess it's pretty much a given that we need to distinguish "real" and "pseudo" binaries, the latter being text-based (i.e. content type text/*) and thus not too different from wikitext.
psuedo binaries. Anybody else have additions?
So let's have a strawman, for sake of knocking it down:
* the server continues to accept binaries like it does[1]
* the wiki serialization sends text/* and *+xml as psuedo binaries
* real binaries are sent without text set at all, thus being lazy
* if download parameter is set, include the base64 encode stuff in
* text
* the JSON serialization of binaries (the one's being lazy loaded) is
base64encoded and the client side knows to turn that into a data:uri
when lazy loading stuff
In what ways is this bad? In what ways is it good?
[1] and we worry about limits later/separately
--You received this message because you are subscribed to the Google Groups "TiddlyWeb" group.
To post to this group, send email to tidd...@googlegroups.com.
To unsubscribe from this group, send email to tiddlyweb+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tiddlyweb?hl=en.
chris...@gmail.com
> On Thu, Jul 15, 2010 at 1:18 PM, <chris...@gmail.com> wrote:
>> On Thu, 15 Jul 2010, Ben Gillies wrote:
>> How about some sort of .exe trojan?
>>
>> What about a .exe trojan that is named foo.gif?
>
> That's a good point. How do other systems manage it?
Well on the web in general we rely on people being responsible for
protecting themselves with virus scanners or operating systems that
are fairly robust and sandboxed. Every time you get a file via
bit-torrent it could be pretty much anything, yeah?
>> In my opinion I think they call that "the web". No co-existence
>> required, just URIs.
>
> So we could provide a nice interface to flickr/picasa/vimeo/youtube/etc
> instead?
I was thinking more along the lines of letting people create
referential tiddlers. That is: tiddlers which contain URIs of remote
resources which would be treated as local resources under the right
conditions.
> Maybe the _If_ question is the thing we should be discussing? SVG, etc is
> text based, so we'd only need to deal with it in the usual way (with some
> extra stuff for displaying it properly, etc). Should we be supporting
> image/png, audio/*, etc at all?
Given that before the advent of tiddlyspace the primary binary tiddlers
were pngs and gifs then, I think, yes, we do need to support them.
Paul Downey
True, we could take the email attachment scanning POV, but that's a
very slippery slope. How about a HTML trojan?
I suspect this is a social issue, and in the hands of the downloader.
> I think it's primarily textual at the moment, sure. However, I don't see why
> it has to stay that way. Some sort of wiki based on videos/images with
> tagging/metadata would be "really cool".
yup! Though to be fair to Fred, I don't think he's saying don't do
that, only, don't host the video files on TiddlyWeb 'cos the store
isn't optimised for large files.
PMario
On Jul 15, 2:43 pm, Ben Gillies <bengill...@gmail.com> wrote:
> > On Thu, 15 Jul 2010, Ben Gillies wrote:
>
> > How about some sort of .exe trojan?
>
> > What about a .exe trojan that is named foo.gif?
>
> That's a good point. How do other systems manage it?
it :)
> > I think it's primarily textual at the moment, sure. However, I don't see
> >> why
> >> it has to stay that way. Some sort of wiki based on videos/images with
> >> tagging/metadata would be "really cool".
>
> > Storing those videos and images in the tiddlywiki would be "not cool".
>
> I'm quite open to the idea that we should be storing them in something else
> and only referencing them in TiddlyWiki. In such a case, we may want to
> optimise for that situation instead and only support text based binaries
> (svg, etc) natively. What do people think about that idea?
But it needs some work, to be done, before I can do this. Optimize the
size, compression rate, ... Import it with Erics plugins. Have a look
with TiddlerTweaker, how big it really is base64 encoded. If I think I
didn't like the size, throw the data sction away and store the links
and the description. If I could get some valid information before,
doing the work, it would be cool. (I am talking about the difference
between 20-50k byte. Not 3MB mp3). I'd store big files outside TW.
> Some sort of co-existing service better designed to handle binary content
> > In my opinion I think they call that "the web". No co-existence
> > required, just URIs.
>
> So we could provide a nice interface to flickr/picasa/vimeo/youtube/etc
> instead?
> > _If_ we want to allow binary editing in TiddlyWiki _and_ we want to
> > preserve offline capability, and we want to do it in a way that
> > doesn't require a bunch of extra smarts and conditionals, then we need
> > the binary content in the tiddlywiki.
>
> Maybe the _If_ question is the thing we should be discussing? SVG, etc is
> text based, so we'd only need to deal with it in the usual way (with some
> extra stuff for displaying it properly, etc).
> Should we be supporting image/png, audio/*, etc at all?
nice and fancy plugins. Test them. et voilá I'm done. I may have a big
TW file now, but this is my decision.
I think if you want, that people use a new system, it needs to be
better and as easy, as Erics AttachFilePlugin solution. If not,
everyone will use the well known and trusted system.
I understand, that Erics approach needs the understanding of TW's
slice and section. But _this_ is imo one thing, thats makes TW
brilliant. I also understand, that Linking to a URL, to get a big
file, causes some security issues. But I think linking to a trusted
service (youtube, flikr ..) is easyer, than to implement a trusted
service.
my 2€ents
mario
FND
> be displayed/edited.
TiddlyWebWiki 0.36 (released last Friday) enables editing for all known
textual/pseudo-binary tiddlers - i.e. anything with content-type text/*
or *+xml.
More significantly, the upcoming version 0.37 revamps handling of binary
tiddlers; their body is now always serialized as base64-encoded data,
with a new client-side plugin rendering that data:
http://svn.tiddlywiki.org/Trunk/association/plugins/BinaryTiddlersPlugin.js
For legacy browser that don't support data: URIs, images have an alt
attribute while non-image binaries are essentially non-functional links.
While I could imagine linking back to the data on the server, I'm not
keen on introducing TiddlyWeb-specific dependencies to calculate the
respective URI.
I reckon we'll also need special handling for SVGs?
-- F.
rakugo
I just tried the dev version of TiddlyWebWiki 0.37 with TiddlySpace
and uploaded a tiddler with type image/svg+xml
The tiddler is displayed as an image:
[img[http://jon.tiddlyspace.com/recipes/jon_private/tiddlers/
rosetta.svg]]
and is not editable.
Maybe I misunderstood but if this version
anything with content-type text/* or *+xml. "
editable no?
Jon
On Jul 20, 1:48 pm, FND <F...@gmx.net> wrote:
> > Q4 There seems to be a lot of confusion over how binary tiddlers should
> > be displayed/edited.
>
> TiddlyWebWiki 0.36 (released last Friday) enables editing for all known
> textual/pseudo-binary tiddlers - i.e. anything with content-type text/*
> or *+xml.
>
> More significantly, the upcoming version 0.37 revamps handling of binary
> tiddlers; their body is now always serialized as base64-encoded data,
rakugo
see things incorrectly.
Now the svg image is served up as text as it should be. However when I
click edit in TiddlySpace I am in a readonly edit so cannot edit the
tiddler and save changes.
Once this works, I think we have a good story for svg. I am updating
the ImageMacro as we speak to display the image in view mode
correctly.
Jon
FND
> cannot edit the tiddler and save changes.
Please raise an issue for this, also to avoid derailing this thread with
implementation details.
Thanks.
-- F.
PMario
> Okay, it seems I was running two servers at the same time making me
> see things incorrectly.
> Now the svg image is served up as text as it should be. However when I
> click edit in TiddlySpace I am in a readonly edit so cannot edit the
> tiddler and save changes.
>
> Once this works, I think we have a good story for svg. I am updating
> the ImageMacro as we speak to display the image in view mode
> correctly.
If I include a svg from a text editor with copy/paste into TW, <<image
copyPaste.svg>> doesn't work anymore. Error "server.content-type" is
not defined. Could you please make a fallback, that checks the svg
header info, that is contained in the tiddler.
regards Mario
btw: If I upload a svg file with BinaryUpload macro it displays.
myTiddler.svg
[img[http://..]]
which doesn't work.
and <<image myTiddler.svg>> doesn't work either.
I suppose, this is work in progress. right?
rakugo
The latest version of the ImageMacro and TiddlyWebWiki show solve
these problems.
You can get the latest ImageMacro plugin here: (which should solve
your server.content-type error) @
http://svn.tiddlywiki.org/Trunk/contributors/JonRobson/plugins/ImageMacroPlugin/plugins/ImageMacroPlugin.js
When TiddlyWebWiki is updated you ***should*** find uploading svg
files works as might be expected, with the svg image shown in view
mode but the source shown in edit mode... There are still a few issues
to iron out (eg. editing those svg files in TiddlySpace) but I'm
hoping these will get resolved this week.
Jon
EduardWagner
just set up a new server to test tiddlyweb 1.0.3.
After migrating our operative store to the new instance it was
surprising.
As You should know:
we use a lot of (1000s) binary pictures, that are shown in different
tiddlers.
AND NOW LOADING IS SOOOOOOO LAZY THAT YOU CAN DRINK SOME CUPS OF
COFFEE FOR A WHILE
So one BIG point why we are using tiddlyweb is to be VERY QUICK in
writing things down, so please
is there a way to get binary tiddlers like it was done before?
Thanks Edi
FND
> AND NOW LOADING IS SOOOOOOO LAZY THAT YOU CAN DRINK SOME CUPS OF
> COFFEE FOR A WHILE
That seems unexpected - what version of TiddlyWeb (and TiddlyWebWiki)
are you currently using?
You could try the current development build:
$ wget http://fnd.lewcid.org/tmp/tiddlyweb-1.1.dev5.tar.gz
$ tar -xvf tiddlyweb-1.1.dev5.tar.gz
$ mv tiddlyweb-1.1.dev5/tiddlyweb myInstance
# restart myInstance server
> we use a lot of (1000s) binary pictures, that are shown in different
> tiddlers
Can you elaborate on this a bit? Thousands of tiddlers in a single bag?
Is there any kind of filtering or on-demand loading? What do your
recipes look like?
While huge collections are a known design limitation of TiddlyWeb, it
should not take that long...
-- F.
EduardWagner
a complete new installation with pip instead of easy_install.
The testing instance is
This is TiddlyWeb version 1.0.3.
The current store is: text.
System Plugins:
tiddlywebwiki (0.38.0)
tiddlywebplugins.form (unknown)
tiddlywebplugins.pathinfohack (unknown)
tiddlywebplugins.reflector (unknown)
tiddlywebplugins.logout (unknown)
tiddlywebplugins.env (unknown)
together with wsgi and apache.py
our operative instance lives since 0.5... (old baby) and got updated
from time to time an know we have:
This is TiddlyWeb version 1.0.2.
The current store is: text.
System Plugins:
tiddlywebwiki (0.28.0)
tiddlyeditor (unknown)
logout (unknown)
tiddlywebplugins.form (unknown)
tiddlywebplugins.socialusers (unknown)
tiddlywebplugins.pathinfohack (unknown)
tiddlywebplugins.reflector (unknown)
env (unknown)
like (unknown)
together with wsgi and apache.py
Our recipes always use these bags (work = name of project)
the structure is a little bit like the nice tiddlyspace, but a little
bit different.
drwxrwxr-x 3 webservd webservd 5 Aug 2 14:55 system/
drwxrwxr-x 3 webservd webservd 5 Dec 15 2009 int/
drwxrwxr-x 3 webservd webservd 5 Dec 16 2009 core/
dwxrwxr-x 3 webservd webservd 5 May 14 12:14 work/
drwxrwxr-x 3 webservd webservd 5 Mar 12 12:10 work_bin/
drwxrwxr-x 3 webservd webservd 5 Mar 12 12:10 work_int/
eg. work policy looks like
{"read": ["R:TUP", "R:GROUP"], "create": ["R:TUP", "R:GROUP.USE"],
"manage": ["R
:ADMIN"], "accept": [], "write": ["R:TUP", "R:GROUP.USE"], "owner":
null, "delet
e": ["R:TUP", "R:GROUP.USE"]}
and a work recipe looks like
desc:
policy: {"read": ["R:TUP"], "create": [], "manage": [], "write": [],
"owner": nu
ll, "delete": []}
/bags/system/tiddlers (5tiddlers) read only role TUP
/bags/core/tiddlers (plugins for everyone 124tiddlers) read only TUP
/bags/int/tiddlers (plugins for us 75tiddlers) read only
/bags/work/tiddlers (projecttext 100-1500 depends on project) full
access (public) TUP
/bags/work_int/tiddlers (projectinternal 100-200) full access
(private) TUP
/bags/work_bin/tiddlers (projectbinarys 100-300) only create and
delete TUP
there are 50 recipes now and each of them serves about 200-1500
tiddlers.
uploading of binarys is managed via client side plugins und targets
workspace + '_bin', so every team has it's own work and binary baga.
The next bag in our structure will be "work_comments" (create
external, delete internal).
Our Use case is a little bit different to tiddlyspace:
External recipes use the same bags, but the work tiddlers get filtered
via tag 'ext', so we don't have to "publish", just tag.
So for sure this is not secure, but this doesn't matter, things you
don't want to see external get moved to "_int", and external recipes
don't have this work_bag.
The operative instance works without any problems, the wikis load fast
and the binarys aren't loaded within the wiki (just generated url-
links within tiddlers).
E.g. today i'm working with a recipe with 300 binarys, 1200
worktiddlers, and 200 systemtiddlers, and firefox takes the first load
within 20 seconds! Thats fairly OK.
We never had the wish to download the wikis, because internet is
everywhere now. So we don't care if binarys are loaded within the
wiki.
But one thing every user cares is FAST creating of tiddlers and even
FASTER searching them!!! And everyone hates LAZY loading!!!
As i migrated the store to the other instance in another container
serving 1.0.3 it took 10 Minutes to load the same recipe, and now the
behaviour of binary tiddlers is changed. The System bag has an
additional plugin regarding Binarys now und the binarys are not just
links any more.
SOME questions:
*you said "loading on demand", is there a concept or testcase for? We
think of bags everyone could need, but doesn't have to load, if not
wanted.
*you said "huge collection", how much is huge?
*tagging is essential, so will i be able to tag binarys later someday
(not only once during upload)?
*as mentioned we use a lot of plugins, so are there known limitations
in tiddlyweb 1.1, or can we use these plugins further?
I did't know it was recommended to take even 1.1 now?
Maybe there is another reason for lazy loading? Should we install
again?
Thanks Edi
On 2 Aug., 19:31, FND <F...@gmx.net> wrote:
> > just set up a new server to test tiddlyweb 1.0.3. [...]
> > AND NOW LOADING IS SOOOOOOO LAZY THAT YOU CAN DRINK SOME CUPS OF
> > COFFEE FOR A WHILE
>
> That seems unexpected - what version of TiddlyWeb (and TiddlyWebWiki)
> are you currently using?
>
> You could try the current development build:
>
FND
How big are your binary tiddlers, approximately? That is, are they
generally just small image files, or big PDF documents?
> That seems unexpected
I lied - since the TiddlyWiki serialization now includes the raw data
for all binary tiddlers, a large number of binary tiddlers (or even a
few large binary tiddlers) can have a significant impact.
So we'll probably make this a configuration option (in a week or two),
giving instance owners the option to fall back to the original handling,
replacing the actual data in (non-pseudo-)binary tiddlers' body with a
link to the resource on the server.
> *you said "loading on demand", is there a concept or testcase for?
There have been various experiments and huge discussions (see group
archives), but as far as I'm aware, there's no simple solution that can
just be picked up and used.
> *you said "huge collection", how much is huge?
Well, that depends on a number of factors - pulling a number out of the
air, I'd say five figures perhaps? At various times, Chris has written
about this topic, and how particular applications work around it, e.g.
by using a store implementation tailored to the respective use case.
> *tagging is essential, so will i be able to tag binarys later someday
Well, in theory, right now:
http://trac.tiddlywiki.org/changeset/12328
http://svn.tiddlywiki.org/Trunk/association/plugins/TiddlyWebConfig.js
However, this requires the new way of handling binary tiddlers, and thus
conflicts with what I said above about optionally reverting to the old
behavior. At that point, we'll have to introduce special handling for
such mangled tiddlers, ensuring that the original body is retrieved
before the modified tiddler can be PUT...
> *as mentioned we use a lot of plugins, so are there known limitations
> in tiddlyweb 1.1, or can we use these plugins further?
Some server-side plugins will have to be updated for compatibility with
tiddlyweb 1.1 (to be released as 1.2), but this is generally not a big
deal and many of the common plugins have already been updated for use in
TiddlySpace.
> I did't know it was recommended to take even 1.1 now?
It's not - I was just trying to get more data to get to the bottom of
the issue, but that's become irreleavant now.
Aside: "lazy loading" is commonly used for dynamic/on-demand loading -
your use of it for slow loading might be confusing.
-- F.
EduardWagner
> generally just small image files, or big PDF documents?
zipped files for other types (pdf, xls, doc, jnt, vsd, ...)
>
> > That seems unexpected
>
> I lied - since the TiddlyWiki serialization now includes the raw data
> for all binary tiddlers, a large number of binary tiddlers (or even a
> few large binary tiddlers) can have a significant impact.
>
> So we'll probably make this a configuration option (in a week or two),
> giving instance owners the option to fall back to the original handling,
> replacing the actual data in (non-pseudo-)binary tiddlers' body with a
> link to the resource on the server.
> > *you said "loading on demand", is there a concept or testcase for?
>
> There have been various experiments and huge discussions (see group
> archives), but as far as I'm aware, there's no simple solution that can
> just be picked up and used.
>
> > *you said "huge collection", how much is huge?
>
> Well, that depends on a number of factors - pulling a number out of the
> air, I'd say five figures perhaps? At various times, Chris has written
> about this topic, and how particular applications work around it, e.g.
> by using a store implementation tailored to the respective use case.
>
> > *tagging is essential, so will i be able to tag binarys later someday
>
> However, this requires the new way of handling binary tiddlers, and thus
> conflicts with what I said above about optionally reverting to the old
> behavior. At that point, we'll have to introduce special handling for
> such mangled tiddlers, ensuring that the original body is retrieved
> before the modified tiddler can be PUT...
>
> > *as mentioned we use a lot of plugins, so are there known limitations
> > in tiddlyweb 1.1, or can we use these plugins further?
>
> Some server-side plugins will have to be updated for compatibility with
> tiddlyweb 1.1 (to be released as 1.2), but this is generally not a big
> deal and many of the common plugins have already been updated for use in
> TiddlySpace.
> > I did't know it was recommended to take even 1.1 now?
>
> It's not - I was just trying to get more data to get to the bottom of
> the issue, but that's become irreleavant now.
>
> Aside: "lazy loading" is commonly used for dynamic/on-demand loading -
> your use of it for slow loading might be confusing.
i meant "SLOW" loading.
Thanks for advice.
It's always nice to see, that tiddlyweb development is going on.
For us it's the best wiki out there!!! Thanks for making this happen.
Bye Edi
chris...@gmail.com
> > So we'll probably make this a configuration option (in a week or two),
> > giving instance owners the option to fall back to the original handling,
> > replacing the actual data in (non-pseudo-)binary tiddlers' body with a
> > link to the resource on the server.
>
> awaiting it ;)
for a configuration item named tiddlywebwiki.binary_limit. If you
upgrade TiddlyWebWiki and do a pydoc tiddlywebwiki.serialization
you'll read this:
For those tiddlers which are considered binary (e.g. contain an
image, an application, etc) the contents of tiddler.text is sent
as the base64 encoding of that text. Client side plugins can
turn that into a data: style URI and use the content.
If config['tiddlywebwiki.binary_limit'] is set to some integer
value that value sets a limit above which the base64 content is
_not_ sent. Instead a link is made back to the server. If
tiddler.type matches 'image/' then the link is an <img> tag.
Otherwise an anchor.
If you want all binaries to be links back to the server then in
tiddlywebconfig.py set:
'tiddlywebwiki.binary_limit': 1 # not 0!
Edi, let us know how this works out for you.
PMario
> If config['tiddlywebwiki.binary_limit'] is set to some integer
> value that value sets a limit above which the base64 content is
> _not_ sent. Instead a link is made back to the server. If
> tiddler.type matches 'image/' then the link is an <img> tag.
> Otherwise an anchor.
-m
chris...@gmail.com
Not at the moment, no, but we could probably investigate adding a
query parameter (similar to "fat" used with JSON tiddler
collections) that would make it possible to control it per request
(presumably lowering but not raising the limit set by the server).
Would that be useful?
--
Chris Dent http://burningchrome.com/
[...]
PMario
-m
EduardWagner
back from holidays again (Cyprus is really hot and humid!).
I was testing around a bit:
This is TiddlyWeb version 1.2.1.
System Plugins:
tiddlywebplugins.pathinfohack (unknown)
tiddlywebplugins.reflector (unknown)
tiddlywebplugins.logout (unknown)
tiddlywebplugins.env (unknown)
apache with mod_wsgi
1. Update to 1.2.1 worked all fine ;)
- pip install -U tiddlywebwiki
- twanager update
- using BinaryUploadPlugin Version 0.3.6
2. base64 content is sent
- 'tiddlywebwiki.binary_limit': 0 # to test normal behaviour
- everything is fine, but the wiki is getting really slow!
- it's nice that it's possible to change tags und titles of binary
files also!!
3. base64 content is _not_ sent
- 'tiddlywebwiki.binary_limit': 1 #
having various problems
a) uploading an image test.jpg
using <<binaryUpload edit:title edit:tags>>
works fine and now you are even informed that an uploding prozess is
pending "Please wait while the file is uploaded...".
But i miss, that the uploaded tiddler is not shown afterwards like it
has been in
BinaryUploadPlugin Version 0.1, i would appreciate this.
After reloading the tiddler is shown, but the image isn't rendered,
the tiddler looks empty?
BUT: if i create a new tiddler and link to [img[/tiddlyweb/bags/
sandbox/tiddlers/test.jpg]] the image is shown.
If i edit the image tiddler the editor division is'nt shown also?
Why, i would like to be able to copy the link inside the tiddler.
Then i tried to change a tag and got the following error
2010-09-14 08:59:56,554 DEBUG starting "PUT" request with uri "/
tiddlyweb/bags/sandbox/tiddlers/test.jpg", script_name "/tiddlyweb",
patinfo "/bags/sandbox/tiddlers/test.jpg" and query ""
2010-09-14 08:59:56,555 DEBUG simple_cookie looking at cookie
string: tiddlyweb_user="EduardWagner:....
2010-09-14 08:59:56,556 DEBUG negotiating for content-type
application/json; charset=UTF-8
2010-09-14 08:59:56,720 WARNING Traceback (most recent call last):
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
http.py", line 154, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
wsgi.py", line 242, in __call__
output = self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
wsgi.py", line 47, in __call__
output = self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
serve.py", line 110, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
serve.py", line 95, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
query.py", line 26, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlywebplugins/
pathinfohack.py", line 40, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
wsgi.py", line 202, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
extractor.py", line 29, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
wsgi.py", line 33, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
negotiate.py", line 25, in __call__
return self.application(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/selector.py", line
161, in __call__
return app(environ, start_response)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
handler/tiddler.py", line 74, in put
control.determine_bag_for_tiddler)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/web/
handler/tiddler.py", line 162, in _determine_tiddler
serializer.from_string(content.decode('utf-8'))
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/
serializer.py", line 81, in from_string
return object_func(self.object, input_string)
File "/opt/coolstack/lib/python2.5/site-packages/tiddlyweb/
serializations/json.py", line 120, in as_tiddler
tiddler.text = b64decode(tiddler.text)
File "/opt/coolstack/lib/python2.5/base64.py", line 76, in b64decode
raise TypeError(msg)
TypeError: Incorrect padding
2010-09-14 08:59:56,720 INFO 172.31.11.48 - EduardWagner [14/Sep/
2010:08:59:56 ] "PUT /tiddlyweb/bags/sandbox/tiddlers/test.jpg HTTP/1.
500 - "https://tiddler.ka.tup.com/tiddlyweb/recipes/sandbox/
tiddlers.wiki" "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:
1.9.2.8) Gecko0100722 Firefox/3.6.8"
b) uploading test.pdf
uploading is fine.
The uploaded tiddler shows a link
data:application/pdf;base64,<html><ahref="https://tiddler.ka.tup.com/
tiddlyweb/bags/sandbox/tiddlers/test.pdf">test.pdf</a></html>
,but the link is'n working. I noticed that a space is missing
also ...<html><a href=......
If i edit this tiddlers tags, i get the same error as above.
So i created a new tiddler linked to <html><a href="https://
tiddler.ka.tup.com/tiddlyweb/bags/sandbox/tiddlers/test.pdf">test.pdf</
a></html> and the binary file opens correctly!
Maybe some circumstances addicted to tiddlyweb 1.2.1 and
tiddlywebwiki 0.44?
Any hints?
Thanks Edi
On 19 Aug., 18:50, "cd...@peermore.com" <chris.d...@gmail.com> wrote:
> On Aug 3, 3:26 pm, EduardWagner <ugligats...@googlemail.com> wrote:
>
> > > So we'll probably make this a configuration option (in a week or two),
> > > giving instance owners the option to fall back to the original handling,
>
> > awaiting it ;)
>
> This has been done. The 0.43 release of TiddlyWebWiki includes support
> for a configuration item named tiddlywebwiki.binary_limit. If you
> upgrade TiddlyWebWiki and do a pydoc tiddlywebwiki.serialization
> you'll read this:
>
chris...@gmail.com
> I was testing around a bit:
Thanks for that.
> After reloading the tiddler is shown, but the image isn't rendered,
> the tiddler looks empty?
> BUT: if i create a new tiddler and link to [img[/tiddlyweb/bags/
> sandbox/tiddlers/test.jpg]] the image is shown.
The BinaryTiddlersPlugin has not yet been updated to handle the
tiddlers which have the <html><img ></html> style contents that get
sent when binary content is not sent.
As a result the links get treated as base64 content and do not
decode to correct images, thus resulting in no image.
> If i edit the image tiddler the editor division is'nt shown also?
> Why, i would like to be able to copy the link inside the tiddler.
This is related to the above. If there is binary content in the
tiddler, then we don't want people messing with it, as it will
result in corrupting the binary data.
So, short story: You've found a bug in the BinaryTiddlersPlugin
which will be fixed soon. When it is fixed these issues should be
resolved.
> File "/opt/coolstack/lib/python2.5/base64.py", line 76, in b64decode
> raise TypeError(msg)
> TypeError: Incorrect padding
This also is the same problem as above. The <html><img> stuff was
sent back to the server as if it is base64, which it is not, of
course, and thus could not be decoded.
> Maybe some circumstances addicted to tiddlyweb 1.2.1 and
> tiddlywebwiki 0.44?
I'm pretty sure all of these things are related to BinaryTiddlersPlugin
around here:
http://trac.tiddlywiki.org/browser/Trunk/association/plugins/BinaryTiddlersPlugin.js#L47
And FND or I will work it out soon, unless somebody else gets there
first.
chris...@gmail.com
> Hi Chris,
> back from holidays again (Cyprus is really hot and humid!).
I've made a quick hacky patch to BinaryTiddlersPlugin which makes
the images at least display (when binary content is not included in
the wiki). It does not address the problems when editing the
tiddlers.
A diff is attached.
EduardWagner
> 1KAnzeigenHerunterladen