multi record tiddlers
xen
I would like to ask a question about something. It's not something I
am already considering to implement or build, but it'd like to know
what is possible; what the context is.
I have this idea about multi-record tiddlers. I know the slicing
mechanism treats the tiddler as a record so there is a 1:1 mapping.
But there is a drawback to this. First, you have to give every record
in your database a unique name. For certain data types (applications)
that can be quite annoying. Second, a record is something very stale,
much unlike a normal wiki page that can have text, markup, headings
and so on. From an end-user perspective, being able to put multiple
records into a single tiddler and add some descriptive free-form text
to it, or even a discussion or whatever, can be a very nice thing.
One example, suppose I create a database of small notes about articles
I've read. These notes are like bookmarks, they are stubs, points of
access to these articles for my own recollection. Now I might read an
article on cultural relativism and I might add it to a tiddler called
just that way "CulturalRelativism". Then I read another article about
it. It's not like I'm doing a complete study of everything on the
subject, so I add it to the same tiddler. I write my summary (part of
the record) and add some thoughts (free-flow text). Maybe I add some
text that describes both articles something I cannot do in a single
record. As I add more and more bookmarks on this subject, it might get
too much, so I split it into two or three tiddlers with more
descriptive titles. I dunno. It's all organic. I'm organically
organizing my data. I don't plan ahead, I just reorganize when I like
to.
But because we have a data application framework that can collect
records from tiddlers in whatever way they are distributed, I can
still harness some automation power. I can list all of the tiddlers I
have in my system (with a certain tag) and all of the records
contained in them (that don't have independant tags, after all, they
are under the same topic and title). I can do everything a standard
tiddly data application could do, except for deleting records from the
"management interface". The records are contained in free-flow text it
doesn't make sense to delete them individually.
The records would have to be identified by some wrapper such as
{{record{ ... }}} which would also immediately style the slice-table
or hide it from default view, depending on the implementation.
TaggedTemplateTweak would be used to render each of the record
sections individually and combine it with the rendering of the
surrounding wikitext, presenting a page-like tiddler.
So basically the argument is: wiki's are organic; imposing strict
rules on "data tiddlers" by turning them into records is
counterintuitive. A wiki can be a database, yes. But a wiki must
remain a wiki. The foundation is organic organisation and
reorganisation of text data that can contain structured data as well.
I'm not saying all applications would adhere to this standard, but
there are some that would. For a pure database it makes no sense. But
for a personal collection of artifacts it makes a lot of sense, to me.
----------
Now the details. As I understand it the slicing mechanism operates out
of the TiddlyWiki (store) object, and not out of Tiddler itself. The
supposed reason for this is that shadow tiddlers are not special
Tiddler objects but simple hashmaps to text, and for all of the
slicing functionality to work uniformly across these two incompatible
types of objects, it would have to operate out of the wrapper that
contains them. Seems like a bad design choice to begin with, after
all, why /not/ make a cool object out of them? There are not and never
will be that many shadow tiddlers to begin with, so it would make the
design more elegant without sacrificing performance. But, of course, I
can't argue with you on that score.
The problem is that for a slicing mechanism to work on records, I must
be able to extract the records from the tiddler and call .getSlice()
on them. The records I extract can either be real Record objects or
just nameless Tiddlers. If Tiddler has Tiddler.getSlice(), then I can
just call a new method Tiddler.getRecords() and iterate the nameless
Tiddlers that are returned to me. (These Tiddlers would simply contain
the {{record{ ... }}} section of text).
It would add a new layer of functionality to the tiddler structure but
it would be completely transparent, that is to say, nothing would
change, and only the multi-record aware code would even know of the
getRecords() method. The normal single-record getSlice() and the multi-
record getRecords()[i].getSlice() would coexist peacefully.
The tiddler::key format would still only reference single-record
slices. Only data macros and scripts would know how to traverse these
records and how to display them. Default ViewTemplates would simply
display the tables (provided they haven't been hidden in CSS) and
application-specific ViewTemplates would know how to render them. The
{{record{...}}} wrapper could also be made flexible by allowing the
getRecords() method to receive a parameter "datatype" that's based on
the predefined tag for the application and is thus known by the data-
plugins responsible for traversal and display. A tiddler tagged as
"bookmark" would define its records by {{bookmark{ ... }}}. This
provides for flexible default styling and hiding of the table, whereas
application-specific plugins would provide viewtemplates and
edittemplates. The macro code would mostly be generic for the data
application framework (and I'm sure that that's how it's already being
done).
So, to wrap it up....
1. Tiddler.getRecords()
2. Tiddler.getSlice()
3. the ability for getRecords() to create nameless tiddlers that are
not entered into the store.
and we're set to go ;-).
Let me know what you think. I'm probably missing something.
Alex Hough
Are these ideas that could feed into TW 5? [1]
I am wondering that some problems will take considerable time to solve and get working and by the time they are solved they will have been removed by comprehensive re-designs.
I was very excited to read about TW 5 and would like to start messing about with it. It seems to be more profitable ( and exciting?) for me to learn the new TW5 and the html5, jQuery and jQuery UI stuff rather than continuing to get to grips with TW 2.6.
Alex
[1] http://www.tiddlywiki.com/tiddlywiki5/
Jeremy Ruston
would make sense to handle your scenario by structuring your records
as unobtrusively marked up HTML, and then using jQuery selector-style
or even XSLT syntax to chop each tiddler up into it's constituent
parts. There would some equivalent of TiddlerName##Selector to
retrieve one of these new structured slices.
Cheers
Jeremy
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>
--
Jeremy Ruston
mailto:jer...@osmosoft.com
http://www.tiddlywiki.com
xen
I must say, now that I have heard of it, that I *immensely* dislike
the idea of losing wikitext. You can say that wikitext will still be
supported but clearly if plugins are going to be built around html
tags and direct jQuery style DOM traversal, then using wikitext is
going to suffer an immense loss of popularity, making it de facto
impossible to keep using it.
I like the idea of a WYSIWYG editor but look at Wikispaces.com (imo
the best online online wiki hoster, even if it lacks a lot of power):
they implemented a beautiful wysywig editor without letting go of
wikitext. Wikispaces + transclusion + macros == my ideal *simple* wiki
site. ((Wikispaces can never be as powerful as TiddlyWiki, but it
makes up for it with great themes, reasonable documentation, lots of
widgets that you can include via a friendly GUI, and a great admin
interface (much like Mac OSX))). Wikispaces is not an application
platform like TW, its a simple wiki hosting site. But one thing they
did right: they allow for *both* wysiwyg AND wikitext. Nobody is going
to be writing in HTML once the wysiwyg editor is there. Writing in
HTML sucks ass. It is unnecessarily verbose (for the context of a
handwritten document) which is the very reason that wiki's became so
popular.
Now I understand that a regular wiki doesn't expose its wikitext but
feeds everything through a rendering engine, that is, the
representation is always html, but for TiddlyWiki this isn't possible
since a search engine will simply traverse the entire html file
without caring about javascript rendering. So for search engines it
makes sense to keep it in html. But for a user it doesn't make sense.
However, to decide on the basis of search engine coverage to change
your internal //editing style// seems like a baaad thing. (And
remember, when a search engine indexes that precious HTML, the link to
the html file will open the default view, and you are not going to see
the tiddlers that you want unless a search engine magically makes use
of anchors and the like.)
In short, if we going to lose wikitext, you are probably going to lose
me (although I don't see where I'll go, except sticking tenaciously to
the old tikiwiki.) For TW5 probably all of the plugins will have to be
rewritten (??) so migrating to TW5 is going to be a pain. Not that it
matters, just giving it as input.
~Xen
xen
http://tidmanual.wikispaces.com/
You can anonymously edit the page to check out the editors.
Jeremy Ruston
> the idea of losing wikitext. You can say that wikitext will still be
> supported but clearly if plugins are going to be built around html
> tags and direct jQuery style DOM traversal, then using wikitext is
> going to suffer an immense loss of popularity, making it de facto
> impossible to keep using it.
There is no question in my mind of losing wikitext.
My personal belief is that wikitext is a beautiful tool, and a
fundamental advance in how we think about formatting, moving it from
the domain of popup windows and toolbars and into the domain of things
you can type. I love that wikitext elevates links to be part of the
punctuation, and as a result I see people using wikitext using linking
much more densely and usefully.
First, as you note, TW5 will support wiki text directly, and will
support plugins for rendering other wiki formats. This isn't some
little nod to backwards compatibility, though. The overall design is
the same as with classic TW: macro rendering is done by converting
text to DOM elements, and macro refreshing remains, as it has always
been, a DOM operation, not a text processing operation.
Secondly, my objective with the WYSIWYG editor is to be able to bring
the best of wikitext to wysiwyg. As you can see if you switch a
tiddler in the TW5 demo to edit mode, the macros are represented as
proxy elements that can (eventually) be typed, dragged and dropped and
so on, treated as if they were characters in the text flow of the
tiddler. So it won't be necessarily to invoke some funky little pull
down menu to select a macro, you'll be able to just type it, and the
editor will automatically macro-ize it.
> I like the idea of a WYSIWYG editor but look at Wikispaces.com (imo
> the best online online wiki hoster, even if it lacks a lot of power):
> they implemented a beautiful wysywig editor without letting go of
> wikitext. Wikispaces + transclusion + macros == my ideal *simple* wiki
> site. ((Wikispaces can never be as powerful as TiddlyWiki, but it
> makes up for it with great themes, reasonable documentation, lots of
> widgets that you can include via a friendly GUI, and a great admin
> interface (much like Mac OSX))). Wikispaces is not an application
> platform like TW, its a simple wiki hosting site. But one thing they
> did right: they allow for *both* wysiwyg AND wikitext. Nobody is going
> to be writing in HTML once the wysiwyg editor is there. Writing in
> HTML sucks ass. It is unnecessarily verbose (for the context of a
> handwritten document) which is the very reason that wiki's became so
> popular.
I also like WikiSpaces, we use it a lot at unamesa.
I'm not sure why you're against being able to edit the HTML directly
as well as using the WYSIWYG editor.
> Now I understand that a regular wiki doesn't expose its wikitext but
> feeds everything through a rendering engine, that is, the
> representation is always html, but for TiddlyWiki this isn't possible
> since a search engine will simply traverse the entire html file
> without caring about javascript rendering. So for search engines it
> makes sense to keep it in html. But for a user it doesn't make sense.
> However, to decide on the basis of search engine coverage to change
> your internal //editing style// seems like a baaad thing. (And
For me the key driver wasn't search engines, but rather the
observation, again and again, that the lack of WYSIWYG editing is an
obstacle to people trying out TiddlyWiki.
> remember, when a search engine indexes that precious HTML, the link to
> the html file will open the default view, and you are not going to see
> the tiddlers that you want unless a search engine magically makes use
> of anchors and the like.)
The way that other applications resolve this is to have client side
code that looks at the referrer, and detects the search engine search
string, and then does a TiddlyWiki search for the same search term. I
believe there is a classic TiddlyWiki plugin that does this, too.
> In short, if we going to lose wikitext, you are probably going to lose
> me (although I don't see where I'll go, except sticking tenaciously to
> the old tikiwiki.) For TW5 probably all of the plugins will have to be
> rewritten (??) so migrating to TW5 is going to be a pain. Not that it
> matters, just giving it as input.
Plugins will indeed have to be rewritten for TW5. Hence the discussion
about striking a balance between backwards compatibility, and the
opportunities offered by breaking it.
Best wishes
Jeremy
xen
> There is no question in my mind of losing wikitext.
>
> My personal belief is that wikitext is a beautiful tool, and a
> fundamental advance in how we think about formatting, moving it from
> the domain of popup windows and toolbars and into the domain of things
> you can type. I love that wikitext elevates links to be part of the
> punctuation, and as a result I see people using wikitext using linking
> much more densely and usefully.
Yes well naturally I don't know yet what the HTML is going to look
like. If writing html involves writing <p>paragraphs</p> and <br/>
line breaks, and if creating lists involve wrapping stuff in <ul> and
<li>, and if tables involve writing <table> and <tr> and <td>, then
clearly it is not an advance at all.
It might be a solid idea that the user is able to combine HTML and
wikitext in the same tiddlers, but wouldn't this add to the
complexity? Your wysiwyg editor is going to have to support both html
and wikitext, if that is the case.
> First, as you note, TW5 will support wiki text directly, and will
> support plugins for rendering other wiki formats.
Seems like a good thing. Take care though that are not losing
interopability because people are going to combine all sorts of markup
and possibly lose sight of what is the default code.
> and macro refreshing remains, as it has always
> been, a DOM operation, not a text processing operation.
Okay.. I don't know what you mean by macro refreshing though.
> Secondly, my objective with the WYSIWYG editor is to be able to bring
> the best of wikitext to wysiwyg.
How does it still involve wikitext then?
> As you can see if you switch a
> tiddler in the TW5 demo to edit mode, the macros are represented as
> proxy elements that can (eventually) be typed, dragged and dropped and
> so on, treated as if they were characters in the text flow of the
> tiddler. So it won't be necessarily to invoke some funky little pull
> down menu to select a macro, you'll be able to just type it, and the
> editor will automatically macro-ize it.
This is a very cool thing indeed. I'm not at all opposed to wysiwyg. I
think it is a great advancement.
> I'm not sure why you're against being able to edit the HTML directly
> as well as using the WYSIWYG editor.
I'm not against being able to edit the HTML, I'm against editing
(writing) HTML in the first place. I want to be able to switch between
pure wikitext and the wysiwyg editor.
Depending on the format of the html we would have to write - if it is
anything like editing a normal html page - I wouldn't be using it.
Yes, now and then. When needed. Personally, if I am given the choice
between HTML and WYSIWYG, I will choose wysiwyg most of the time. This
is, if I have some long document to write, I will use a wysiwyg editor
and not a text editor. But given the choice between WYSIWYG and
wikitext, then I will choose wikitext at any time! Especially if my
editor supported dragging and dropping images, or including image
links through a dialog box.
So I'm not against wysiwyg, and I'm not against being able to write
the HTML, I'm against writing HTML //as a language for markup.//
It's not clear to me how you would combine html and wikitext in the
same tiddlers. Or how you would have plugins that support both.
You don't need html for wysiwyg, that's my point.
> For me the key driver wasn't search engines, but rather the
> observation, again and again, that the lack of WYSIWYG editing is an
> obstacle to people trying out TiddlyWiki.
That's a solid observation, and a solid reason for implementing
wysiwyg.
> The way that other applications resolve this is to have client side
> code that looks at the referrer, and detects the search engine search
> string, and then does a TiddlyWiki search for the same search term. I
> believe there is a classic TiddlyWiki plugin that does this, too.
Okay, fair enough.
> Plugins will indeed have to be rewritten for TW5. Hence the discussion
> about striking a balance between backwards compatibility, and the
> opportunities offered by breaking it.
Rewriting plugins is also a sacrifice I would make. Some of it will be
easier because many existing plugins would start using jQuery
properly. (Another language to learn :P).
I'm looking forward to your new architecture, but I'm keeping my
fingers crossed whether I can still keep using wikitext properly.
~Xen
Xen
> Parenthetically, and I should have replied to the original thread,
> I've come to believe that aggregated tiddlers can be restrictive.
They add some restriction to the code.. but also a lot of flexiblity
to the user. Are we a database or a wiki? And can't we be both?
> In many cases the best way around seems to be to use synthetic
> names (much as a database assigns numerical ids for records in
> some situations).
True, but think of a regular document, you would never do that to your
page titles. Especially considering that titles are more than just
titles for a tiddler, they are also headings and it identifies them in
the mind of the user.
> It sounds like some of the complexity you've got comes from the
> nesting of tiddlers inside other tiddlers, and the need to re-invent
> how to encode the metadata. I've found that pulling back to a simpler,
> non-hierarchical model avoids this problem.
If indeed there was _never_ any other way that a macro wants to
process the tiddler text apart from (a) slices and (b) sections, this
would be a conceptual departure from the current model. Slices are one
sublevel of a tiddler and sections are also one sublevel of a tiddler.
You can't have slices-per-section. Having slices per record per
tiddler (or slices per section per tiddler) indeed adds a layer to
everything involving this functionality.
But the complexity regarding the supporting code lies not in the extra
layer. All plugins that add formatters in order to look for patterns
in the wikitext in order to add new syntax, will in fact need to be
rewritten for EVERY text format you support. Other plugin examples
closer to my topic would be plugins that list all of the outgoing
links in the documents, or all images in tiddlers tagged "photoshop".
It's the multi-format-support that adds difficulty in this case.
The complexity then lies in recognizing the special tag I include in
my tiddlers that define my records. You call it metadata. It is an
addition to the four types of metadata called tags, fields, slices and
sections. In the case of slices and fields, will there be separate
code for each text-format? This is most likely.
If so, it would not be hard for my record-plugin to write custom code
too for all the supported formats.
This could be a plugin that really lays a foundation for other plugins
to build on. Record-aware Listing macro's would call getRecord before
calling getSlice on it. Others would simply call getSlice directly.
The display of tiddlers does become more complex. There would be two
modes:
1) direct rendering of the tiddler using a view-template that calls
several macros to deal with tags, fields and text.
2) rendering of the tiddler using an standard view-template that calls
an iteration macro that utilizes a special view-template to render the
records and wikifies the remaining content
These two paths would not always produce identical results for single-
record tiddlers.
A data application would likely choose one of the two approaches, not
both.
I agree that this is not a very beautiful approach. It is said that
the greatest complexity is the greatest simplicity but in this case
the complexity is rather ugly. It is not something to include in the
core. Except for the required method Tiddler.prototype.getSlice().
That is beautiful.
I would still personally take this approach for my project because the
added ugliness for the code is balanced by the added beauty for the
user imo. What you get is this: _a platform for data applications that
is layered on top of page-formatted tiddlers_. This would not be
something exclusive to TiddlyWiki, in principle. But tiddlers are more
suited for this then normal pages, because tiddlers are small, they
exist in memory, they will never contain more than say 20 records
depending on view-format, and they already have slicing mechanisms in
place which would be a format completely alien to other wikis. It also
somewhat bridges the gap between all-data-in-a-table and every-row-in-
a-tiddler.
But the plugin would have to account for all of the possible text-
format-plugins and provide regexp patterns for all of them. This is no
different from other plugins that add syntax though!
One could argue that if I wanted to include tiddlers in tiddlers, I
should use TagglyTagging. But that doesn't take away from the
difficulty of naming them and also doesn't provide a page-like
collection of records. On the other hand, I could hand-create a custom
index page that includes tiddlers with custom view-templates and view-
editing-controls for all fields, but this cannot work for slices
(readonly), only for fields. I would need to create unique names for
all the tiddlers but I would manage and view them on the index-page.
The index pages would simply call <<tiddler>> macros on the individual
records. Then I need not deal with the individual tiddlers anymore. I
can even excludeList them. I could even create them and give them a
synthetic name from the management interface at the top/bottom of the
tag-page.
But all of this would be a solution - although very simple in its
design - a lot less powerful. And, not less importantly, it would
never be a foundation for building other multi-record-aware
applications, except that it will be a final solution in itself.
Reorganizing those index-pages would not be any different than copying
links around, except that the "<<tiddler>>" calls would reference non-
discriminate tiddler-names to other tag-pages so it becomes somewhat
messy.
I bet this is how most tiddlywiki databse adaptations do it. Just
don't let the user see the tiddlers in the first place. Provide an
interface for management and creation and display. Don't use slices,
use fields.
Agh but it SO takes away from the tiddler-is-a-page concept. I
understand that true tailormade specialized sophisticated special-
purpose applications do it in this way. Just records. No pages. No
free flow text. No organic organizing.
I believe multi-record tiddlers would be more powerful in providing
applications that are not entirely specialized but which combine page-
like editing and display with added data automation abilities. The
data automation would be a layer "above" the page-editing, not
replacing it. It would be the symbiosis between wiki-text editing and
automated applications.
Let me know what you think.
~Xen
[1] http://groups.google.com/group/tiddlywiki/browse_thread/thread/33f9cb46d5d15566
FND
> the HTML, I'm against writing HTML //as a language for markup.//
FWIW, I guess there's a misunderstanding here.
I don't think anyone will be expected to compose tiddlers in raw HTML
(unless they want to). Instead, you'll be able to use wiki markup just
like you do in TiddlyWiki 2.x.
IIRC, Jeremy said that such tiddlers would be saved with their original
markup rather than persisting* as HTML, due to the issue of
round-tripping (wiki markup -saving-> HTML -editing-> wiki markup).
I'd prefer using HTML as common denominator, but I'm not sure how
realistic that is. (Sorry if this issue has been resolved already, it
might be buried somewhere in this rather lengthy thread.)
-- F.
* the markup would still be rendered as HTML when displaying a tiddler
Xen
the usergroup. That discussion is more directly about html versus
wikitext.
Please everyone if you reply on the html versus wikitext topic, go
there. This thread is for the concept of multi-record tiddlers.