Request for an incredible better DataTiddlerPlugin

[wolfgang]

Greetings to all developers - especially to Udo,

In the following thread the possibility for a bidirectional and
multilingual dictionary in a TiddlyWiki was inquired by Abd Shomad:

http://tinyurl.com/32mam4
http://groups.google.com/group/TiddlyWiki/browse_thread/thread/9c672c0435cc25e2#

>>> quote - wolfgang:
Ok, I gave it a try and created a proof of concept - if and how
TiddlyWiki would be able to handle internal database functions like a
simple Pali-English dictionary.

For this I pasted 36 word lists from http://www.budsas.org/ebud/dict-pe/index.htm
into the text editor Pspad ([1] which has easy to use reg-ex search/
replace [2]) to create wiki-format italics and line-separators for the
further procession of these word lists into separate tiddlers - one
for each word - with SplitTiddler script [3] within the
TiddlyWiki itself. Which worked quite straight forward - thanks to
these 2 tools.

Here the description and comparison of the TiddlyWiki's created en
route:
---------------------------------------
Dictionary Creator TiddlyWiki:
word-list tiddlers: 36
file size: 1229 KB
saveChanges: 3 sec.
reload: 2 sec.
---------------------------------------
Pali-English Dictionary TiddlyWiki:
word tiddlers: 20137
size: 3.140 KB
saveChanges: 40 sec.
reload: 60 sec.
example word search (SearchPlusPlugin [4]): 8 sec.
example word goto (GotoPlugin [3]): 8 sec.
---------------------------------------

That means dividing word lists into separate tiddlers can't be the way
to create such huge databases with TiddlyWiki - because the meta data
of tiddlers is bloating it to almost 3 times the necessary file size!

Though with ScrubTiddlerFields and SetUserName bookmarklets [5] I
could
reduce this meta data overhead - still it remained double its original
size:
---------------------------------------
Pali-English Dictionary TiddlyWiki with reduced meta data:
word tiddlers: 20137
size: 2.490 KB
saveChanges: 30 sec.
reload: 50 sec.
example word search (SearchPlusPlugin): 8 sec.
example word goto (GotoPlugin): 8 sec.
---------------------------------------

... And by limiting the search to those with the first letter of the
words (by which I tagged them) the search is reduced to milliseconds.
(with SearchPlusPlugin)
>>>


This brought me to the following suggestions:

A DataTiddlerPlugin [6] for such huge databases - as multilingual
dictionaries are - in a TiddlyWiki would have to be able to:

# to store its data in one or few tiddler only
# for each set of data using one line in this tiddler only
# each line divided further into fields by something like
ImprovedSlicesPlugin [8]
# each field has a first word, and then further definitions (ie. |
easy //[adj.]// ... |facil ... |leicht ...|)
# provide a input panel which recognizes, chooses and creates such
improved slice table tiddlers
# in the input panel define the number and kind of table columns for
newly created database tiddlers
# automatically add the specified input fields for entering data
# automatically add non-wikify delimiters for table syntax entered in
these fields (ie. ||)
# append such lines in the improved slices format to the existing
database tiddler
# sorting the lines in database tiddlers alphabetically
# spitting database tiddler up - upon exceeding an optimal size
# a search panel with checkboxes to define which slice columns to
search and which to display
# showing a result with the searched word (without definitions), and
the words equivalent in the searched for language, bold - with the
additional definitions in that language appended
# a search recognizing special characters of foreign languages (ie. n
searches also for ñ)
# an edit mode, where the found data set is opened in input fields
again
# an import function for data created by the existing
DataTiddlerPlugin (optionally)
# a convert tiddlers function, which converts all existing tiddlers
below a certain size into one line of a database tiddler with <b>
syntax for new lines, adding the meta data in its further columns. And
the recognition and display of such rows - as if they were real
tiddlers! (for TWs exceeding a certain size slowing them considerably
down - optionally ;-)

Such a DatabasePlugin would eliminate all hindrances to huge databases
or dictionaries with - and the tiddler_number_slowing_down_factor, in
TiddlyWikis!

Regards,

W.

PS: I apologize if this is seen as double posting. However, the
unbelievable added potential to TiddlyWikis with such a incredible
better DataTiddlerPlugin was enough justification for me to post this
summary :-)

[1] Pspad Editor: http://www.pspad.com/
[2] Multiline Search/Replace: http://www.phdesign.com.au/document.php?page=software_phreplace
[3] SplitTiddlers, InlineJavascriptPlugin, ExportTiddlersPlugin &
GotoPlugin: http://www.TiddlyTools.com/#SplitTiddler
[4] SearchPlusPlugin & PreferenceSaverLib: http://tw.lewcid.org/svn/plugins/
[5] ScrubTiddlerFields & SetUserName bookmarklets:
http://lewcid.org/tiddlywiki-bookmarklets/
[6] DataTiddlerPlugin & FormTiddlerPlugin: http://tiddlywiki.abego-software.de/
[7] http://www.remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html
[8] "FAQ_Create tiddlers using custom-built HTML forms" from
TiddlyTools