Options for Forms and Structured Data Storage in Tiddlywiki

[ian]

I am trying to determine if I'm properly evaluating the many
alternatives when considering forms and structured data storage in
Tiddlywiki. As with my templated html publishing notes, I will add any
notes to the wiki - after I get to a working solution.

the problem I am trying to solve:
I need to display a few form elements for an adoption record
(multiselect listboxes, repeating form fields e.g.
http://javascript.neyric.com/inputex/examples/list_field.html )
I would also like to eventually wire one of the many form building
UI's so users may create their own form templates, and possibly other
javascript libraries to allow a more interactive display in Tiddlywiki
(json handlers seem to be common in these libraries).

All of the methods listed below can provide data to the ForEach
plugin, so retrieval is the same - but are there any performance pros/
cons for the various formats?

Udo's FormTiddlerPlugin and DataTiddlerPlugin
---------------------------------------------------------
-data stored as json objects which should make interfacing with
external libraries easier
-appears to be the best plugin combo for using an external forms
builder toolkit since the html form templates are separate tiddlers
and just plain ol' html

con: it would be nice to have the full power of the wiki editor and
preview plugin in a form field- e.g to see layout of images
but this is still possible with a compound tiddly document (?) - i.e.
tiddler=form + (other included tiddlers)
con: Tiddler slices are a more user friendly representation

-examples-
http://members.bbnet.com.au/~kwright/overtime.html
http://xdexavier.googlepages.com/FormSample.html
-adds a prefix to be used when storing <data>
-disables the autosave of the tiddler when changing focus
http://www.giffmex.org/bibblywiki.html
http://tiddlywiki.abego-software.de/#%5B%5BDecember%20Expenses%20(with%20sum)%5D%5D
http://www.rogermarrs.com/tw/DigitalCaseFileDemo.html
http://www.tiddly-twab.com/
-example contact database using tabbed forms, support for csv import/
exports including Microsoft Outlook, Google, and Yahoo

Tiddler Slices and Tiddlywiki Tables
----------------------------------------------
http://tiddlywiki.org/wiki/Tiddler_Slices
Each slice is a line consisting of a name-and-value pair, which can be
specified anywhere within a tiddler, in any of these formats:
theName: textSlice
|theName:| textSlice |
|theName| textSlice |

basic way to retrieval of tiddler slices:
* using the [[Tiddler::Slice]] notation (only works in tiddlers
evaluated by TiddlyWiki, e.g. PageTemplate)
* using the tiddler macro: <<tiddler [[tiddlerName::sliceName]]>>

cons:
-Question: when editing generated tiddler, there is no way to get back
to the options that were in drop down list in the form(using Eric's
FAQ example)...is there a way to apply the original form used to
create the tiddler so users can see the various options in a select
field?
-using Eric's example, it seems the mix of js code with the html form
template means it may be harder to interface with an external form
building UI library(?)

-examples-
Eric Shulman's example to create a tiddler table from a form
http://www.TiddlyTools.com/#FAQ_CreateTiddlersFromForms
An extension ofEric's example- includes GPXReader import too
http://trey.arthur.tiddlyspot.com/#WebTripReport_FromFormsToGeocachingTiddlers
http://groups.google.com/group/TiddlyWiki/browse_thread/thread/527e7080998bf3c8/4f4b2b42427c6a61

Tiddler Fields data structure
-------------------------------------
http://www.tiddlywiki.org/wiki/Getting_started_with_custom_tiddler_fields
case-sensitive and must be all lower case for their values to be
stored/retrieved sucessfully

cons:
easy to access data objects from external library?

-examples-
http://tiddlyforge.com/tws/geocache.html
http://no-sin.com/wiki/WorkTracker.html
example showing the edit of some fields in view mode, and others in
edit mode
http://www.hawksworx.com/playground/TeamTasks/roadmap.html
http://www.hawksworx.com/playground/TeamTasks/
http://curiousjames.wordpress.com/2008/05/29/track-choose-v02/ - built
on the TeamTasks skeleton

[ian]

a few updates based on further searching the mailing list

via this thread:
http://groups.google.com/group/TiddlyWiki/browse_thread/thread/936fed2828836ff1
slices are not designed to handle "extended data" types - but a syntax
could be used for multi valued fields
e.g. an alternative slices plugin that handles multi valued fields:
http://remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html#ImprovedSlicesPlugin

custom tiddler fields can handle multi valued fields, but do not seem
able to handle a 2x2 array, tiddler fields also need a customized view/
edit template to be useful
Eric's http://www.tiddlytools.com/#CDSample is an example of custom
fields with slices in a single tiddler to show mixing of the two
approaches

json seems to give the most flexibility for managing complex data
structures in a single tiddler

am I missing anything?

ian

> -examples-http://members.bbnet.com.au/~kwright/overtime.htmlhttp://xdexavier.googlepages.com/FormSample.html

> -adds a prefix to be used when storing <data>

> -disables the autosave of the tiddler when changing focushttp://www.giffmex.org/bibblywiki.htmlhttp://tiddlywiki.abego-software.de/#%5B%5BDecember%20Expenses%20(wit...http://www.rogermarrs.com/tw/DigitalCaseFileDemo.htmlhttp://www.tiddly-twab.com/

> -example contact database using tabbed forms, support for csv import/
> exports including Microsoft Outlook, Google, and Yahoo
>
> Tiddler Slices and Tiddlywiki Tables

> ----------------------------------------------http://tiddlywiki.org/wiki/Tiddler_Slices

> Each slice is a line consisting of a name-and-value pair, which can be
> specified anywhere within a tiddler, in any of these formats:
> theName: textSlice
> |theName:| textSlice |
> |theName| textSlice |
>
> basic way to retrieval of tiddler slices:
> * using the [[Tiddler::Slice]] notation (only works in tiddlers
> evaluated by TiddlyWiki, e.g. PageTemplate)
> * using the tiddler macro: <<tiddler [[tiddlerName::sliceName]]>>
>
> cons:
> -Question: when editing generated tiddler, there is no way to get back
> to the options that were in drop down list in the form(using Eric's
> FAQ example)...is there a way to apply the original form used to
> create the tiddler so users can see the various options in a select
> field?
> -using Eric's example, it seems the mix of js code with the html form
> template means it may be harder to interface with an external form
> building UI library(?)
>
> -examples-

> Eric Shulman's example to create a tiddler table from a formhttp://www.TiddlyTools.com/#FAQ_CreateTiddlersFromForms
> An extension ofEric's example- includes GPXReader import toohttp://trey.arthur.tiddlyspot.com/#WebTripReport_FromFormsToGeocachin...http://groups.google.com/group/TiddlyWiki/browse_thread/thread/527e70...
>
> Tiddler Fields data structure
> -------------------------------------http://www.tiddlywiki.org/wiki/Getting_started_with_custom_tiddler_fi...

> case-sensitive and must be all lower case for their values to be
> stored/retrieved sucessfully
>
> cons:
> easy to access data objects from external library?
>

> -examples-http://tiddlyforge.com/tws/geocache.htmlhttp://no-sin.com/wiki/WorkTracker.html

> example showing the edit of some fields in view mode, and others in

> edit modehttp://www.hawksworx.com/playground/TeamTasks/roadmap.htmlhttp://www.hawksworx.com/playground/TeamTasks/http://curiousjames.wordpress.com/2008/05/29/track-choose-v02/- built
> on the TeamTasks skeleton

[ian]

just adding a quick update to this thread:

I missed another recent discussion on forms/data in TW that took place
here:
http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/59952b0ef80802f2
http://groups.google.com/group/TiddlyWiki/browse_thread/thread/1986179b33c278a6

There was some discussion as to whether DataTiddler or tiddler fields
provided faster retrieval, with the outcome being that it did not seem
to matter.
Regarding speed:
"I was a bit quick about the cause of those performance
issues... it is not that there aren't any, but rather that they don't
seem to depend too much on whether you use fields or
DataTiddlerPlugin. "

Regarding why Paul started with DataTiddler:
"As for why this, originally, used the DataTiddler plugin. I
wanted to see if I could do this plugin using a form from Udo's
FormTiddler plugin. From there it grew into an investigation of how
to reference objects from various places within the document.
Needless to say it got too complex and, being in the midst of it, I
needed another mind to tell me that I was going down a needlessly
complicated path."

I wasn't clear on the complexities experienced by Paul with the
DataTiddlerPlugin though.

Since I also started my investigations with DataTiddlerPlugin, I will
stick with it for now. Eventually, I will likely create a test case to
see where the walls are with the different options. In a week or two,
after I've gained more experience with data/forms in TW, I will
consolidate the contents of this post and my experiences into this
page (created only as placeholder for now): http://www.tiddlywiki.org/wiki/Forms_and_Data

ian

On Jul 3, 8:59 am, ian <lister...@gmail.com> wrote:
> a few updates based on further searching the mailing list
>

> via this thread:http://groups.google.com/group/TiddlyWiki/browse_thread/thread/936fed...

> slices are not designed to handle "extended data" types - but a syntax
> could be used for multi valued fields

> e.g. an alternative slices plugin that handles multi valued fields:http://remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html#Impr...

>
> custom tiddler fields can handle multi valued fields, but do not seem
> able to handle a 2x2 array, tiddler fields also need a customized view/
> edit template to be useful

> Eric'shttp://www.tiddlytools.com/#CDSampleis an example of custom

> > -examples-http://members.bbnet.com.au/~kwright/overtime.htmlhttp://xdexavier.go...

> > -adds a prefix to be used when storing <data>

> > -disables the autosave of the tiddler when changing focushttp://www.giffmex.org/bibblywiki.htmlhttp://tiddlywiki.abego-softwar...

> > -example contact database using tabbed forms, support for csv import/
> > exports including Microsoft Outlook, Google, and Yahoo
>
> > Tiddler Slices and Tiddlywiki Tables
> > ----------------------------------------------http://tiddlywiki.org/wiki/Tiddler_Slices
> > Each slice is a line consisting of a name-and-value pair, which can be
> > specified anywhere within a tiddler, in any of these formats:
> > theName: textSlice
> > |theName:| textSlice |
> > |theName| textSlice |
>
> > basic way to retrieval of tiddler slices:
> > * using the [[Tiddler::Slice]] notation (only works in tiddlers
> > evaluated by TiddlyWiki, e.g. PageTemplate)
> > * using the tiddler macro: <<tiddler [[tiddlerName::sliceName]]>>
>
> > cons:
> > -Question: when editing generated tiddler, there is no way to get back
> > to the options that were in drop down list in the form(using Eric's
> > FAQ example)...is there a way to apply the original form used to
> > create the tiddler so users can see the various options in a select
> > field?
> > -using Eric's example, it seems the mix of js code with the html form
> > template means it may be harder to interface with an external form
> > building UI library(?)
>
> > -examples-
> > Eric Shulman's example to create a tiddler table from a formhttp://www.TiddlyTools.com/#FAQ_CreateTiddlersFromForms

> > An extension ofEric's example- includes GPXReader import toohttp://trey.arthur.tiddlyspot.com/#WebTripReport_FromFormsToGeocachin......

>
> > Tiddler Fields data structure
> > -------------------------------------http://www.tiddlywiki.org/wiki/Getting_started_with_custom_tiddler_fi...
> > case-sensitive and must be all lower case for their values to be
> > stored/retrieved sucessfully
>
> > cons:
> > easy to access data objects from external library?
>

> > -examples-http://tiddlyforge.com/tws/geocache.htmlhttp://no-sin.com/wiki/WorkTr...

> > example showing the edit of some fields in view mode, and others in

> > edit modehttp://www.hawksworx.com/playground/TeamTasks/roadmap.htmlhttp://www....built
> > on the TeamTasks skeleton

[wolfgang]

Hi Ian,

when I tried to create a simple dictionary I soon met TiddlyWiki's
database limits. This caused me to plea for "an incredible better
DataTiddlerPlugin" in the following thread:
http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/b0527a9f98775ea0/55c473ca876b1465?lnk=gst&q=pali+dictionary#55c473ca876b1465

The awfully slow loading dictionary- TiddlyWiki I created you could
download from here:
http://wolfi.110mb.com/pali.html

Regards,

W.

[ian]

Hi Wolfgang,

I missed this post too! Thanks for the pointer to your experience, I
found it very helpful.

For my needs, I will have hundreds of structured datasources roughly
the size of your dictionary, usually representing the metadata
attached to an experiment log. I still want to easily link to parts of
the datasources when I write something, or search, but I'm not sure
the resources need to be in a tiddler format (TW file) unless I update/
add to the datasource.

To continue your investigation, at an attempt to make TW useful for
working with larger data structures, I wonder if anyone has any
comments on the following approaches:

*Storing data in SQLite (database included with Firefox):
zotero.org a references datastore with page scrapers already used by
many academics could probably handle your dictionary example, I once
considered creating a TW interacting with the zotero tables using
Jack's SQLQueryPlugin, but some scientists I know have complained the
growing volume of their references has considerably slowed their
interactions with the tool, so I did not bother - my interests have
moved toward Solr, or creating a Tiddlywiki version that shards data
according to the domains I work between

*Storing data in Solr:
This will easily handle your dictionary and several hundred more
dictionaries, can pass json structures, and simple to batch updates to
Solr with the TW RSS feed. Solr can be thought of as a flat database,
tiddler objects easily map to it. I have some bias though, since I
already invested considerable time with the project.
One issue with Solr is the requirement for a Java runtime - and for
part of the user base I would like to help, it is probably not
feasible to expect a java runtime.

*Enhance TW to actively partition notebooks and create/load
precomputed indexes based on rule sets derived from tags, alphanumeric
range, username, date, etc.:
As noted in the Pali dictionary example, loads and edits are slow with
larger collections, but searches are relatively fast, this has been my
experience too, even with the 8MB references file I maintain. I rarely
want to edit this file, just search it. So, when priming the index for
the TW search command, perhaps a parameter can be referenced that
defines a set of additional pre computed index structures to load into
scope. The indexes are derived from a datasource such as other TW
files (e.g. pali dictionary), or another file format (e.g. csv). If I
need to create/update a reference (e.g. pali dictionary entry), a
tiddler object is created in my primary notebook as needed. Changes
are then propagated back to the datasource (e.g. a TW document, csv
file, or server) based on user preference (batched as a monthly sync,
lazy loaded during minimal activity, etc.).

The idea is to keep your primary notebook small, and thus able to
persist edits quickly. It is a short term memory of sorts always in
RAM - a bag of essential objects(tasks, reminders, etc. ), plugins,
and themes. This primary notebook would also manage profiles pointing
to sets of index structures appropriate for the various realms I may
exist in e.g. all tiddlers I authored over the last year, my wife's
website, my research and experiments, tiddlywiki dev bits - perhaps
using iframes to load a set of objects as a new TW file. This also
sounds a little like what the CognyWiki authors had in mind, in that
at any given point in time, my brain is likely pointed along some task
axis, it's just that occasionally my tasks have large collections of
connected objects I would like to maintain local references to.

Does it seem like this might improve user interaction when working
with sets of large objects?
It is likely more than you need for the dictionary example, but it
should scale to handle many more dictionaries.

To further explain my reasoning:
Even for a very large lexical and data rich domain as the life
sciences, one is provided many reasonable dimensions upon which to
split information into smaller chunks. There are many molecules in
nature, but I am often only interested in molecules of a particular
mass range (e.g. 'small molecules', or proteins), and usually just the
set that influences the particular biochemical pathways I am working
with. Or, if I am looking at a mass spec dataset for biomarker search
of patient plasma, I might only need the small peptide part of my
molecule datastore - where I may have entered several new peptide
entries based on some external tool's sifting of the dataset. Later, I
might want to sync the new entries with some data authority after I
publish a paper. The algorithms in the external tool may have actually
identified a few hundred different peptides in the sample, in addition
to my three new unidentified peptides - all of which are now
referenced in my molecule datastore. Other times, when I am in my
HomeRealm, I have MB's of statistics and clippings on electric
tankless water heaters and other energy topics, notes, emails from a
neighbor, etc. - I do not need this information in RAM, when I am in
the biomedical branch of my task realm.

Based on a quick review of the plugins available, perhaps this might
serve as the foundation for the solution I am proposing (again
comments welcome):
I wasn't entirely clear from the post here:
http://groups.google.com/group/TiddlyWiki/browse_thread/thread/640aba056052fffc
but I think this is the direction the author was headed in, to persist
index structures - maybe some mix of this plugin, SearchPlusPlugin and
the Includes/MasterIncludes plugin will provide the building blocks
for a solution (?)
I wonder at what point on such a path, I am recreating a large part of
Solr, but in Javascript - not something I want to do. So, maybe it is
better to stuff everything into Solr and only use TW for the things I
regularly edit (appointments, etc.)? I'd still apply the concept of a
short term memory mapped to a TW file, but as my task realms change,
alternatively load and query various Solr indexes with http/json
(localhost.molecules, localhost.patients, university_server, etc.).

ian

On Jul 5, 5:11 pm, wolfgang <wolfgangl...@gmail.com> wrote:
> Hi Ian,
>
> when I tried to create a simple dictionary I soon met TiddlyWiki's
> database limits. This caused me to plea for "an incredible better

> DataTiddlerPlugin" in the following thread:http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/b05...

[ian]

actually, since this post touches on development issues, it is likely
better that I continue any discussion on the dev board in the thread
you created:
http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/b0527a9f98775ea0

I will copy the contents of my previous post there

> I wasn't entirely clear from the post here:http://groups.google.com/group/TiddlyWiki/browse_thread/thread/640aba...