TiddlyWiki as a bidirectional dictionary (later will be multilingual)

[abd shomad]

Dear all,

I have an idea to make a dictionary out of TiddlyWiki document, and
wants to know what other think on how to implement this idea. Whether
it is possible, what are the limitations and what are the possible
workarounds, if any.

! TiddlyWiki as a bidirectional dictionary

The first problem I can imagine is on the size of the dictionary,
which I think can be solved using LocalAdapterPlugin, by having
multiple tidlers on its separate files.

The second problem is the javascript performance to wikify all the
loaded contents, which can be solved possibly by using
LazyLoadingPlugin (Is there any?).
The LazyLoading idea can be found here: http://tinyurl.com/ys7olu

The third problem is that every word should be clickable. If we have
to wrap all words inside <a href></a> tags, I am afraid this will also
degrade performance because of javascript processing time to wikify
and WikiWording all the word. And also, I prefer not to enclose all
words with double square brackets [[theword]] to make the word
clickable.

I think this is the time for someone to develop a LinkAllWordPlugin,
hopefully by using eventBubble?

Since I am not (yet) literate in javascript development, and try only
few of the available plugins, I need some advice on what available
plugins do I need for the above functionality.

Use Cases:
* User select translate to Indonesian : (xlat_to = "id")
* User click any of the displayed word (eg: the word "simple")
TiddlyWiki will LazyLoad from file "simple.id.tiddler"
(Loaded file : ClickedWord (dot) xlat_to (dot) tiddler)
* Manual operation:
User can manually commit all the updated / createad tiddlers to SVN
repository

This dictionary is a simple dictionary, not the complex thesaurus
found on this thread ( http://tinyurl.com/ynugkl )

I also prefer to make this dictionary as simple as possible (no tags,
for example), so that everybody can collaborate on completing the
contents by adding their definitions for every possible words.

Just word linked to its translation.

Regards,
Abd Shomad

[wolfgang]

Hi abd,

I just started to play with the same idea of creating a dictionary
with a TiddlyWiki.

But honestly - I guess soon to meet the limits of a TiddlyWiki in this
area - alone the words beginning with an 'A' amounts to 2500 in the
language I'm interested!

Making it bidirectional - with 2 tiddlers for each word - we are
talking about 100.000 tiddlers! That can't work - I suppose.

However, by creating 1 tiddler per word in you mother tongue - and
then use slices as content, you can cut the total amount of tiddlers
at least by its half:

English: easy
Español: facil
Français: facile
Português: simples
Deutsch: leicht

And using the search function enhanced with a search plugin and
limiting the search to the title or text only - for finding a word's
translation.

Such non-bidirectional linking, but possibly multilingual, would make
the most sense, I guess.


LocalAdapterPlugin would not help you to reduce the size of such a TW.
Though it saves tiddler revisions external - it also and still keeps
them within the storeArea of the TiddlyWiki.

ArchivePlugin similarly saving tiddlers separate but keeping only the
meta data in the storeArea - would indeed reduce the size of the TW.
However, search with only 2000 external tiddlers already slows down
considerably in this case.

Remains only IncludePlugin, which seems to make the most sense for
this particular use case, and for keeping the TiddlyWiki small. And
fast enough on a saveChanges.

Regards,

W.

abd shomad schrieb:

[Kashgarinn]

dictionaries and direct manipulation/indexing of data is really the
realm of databases.

Tiddlywiki behaves alot like a relational database, but the overhead
of finding and grabbing the data, either directly or from slices means
it's a bad database for any wiki which is close to or over 1MB.

It's a nice idea, but tiddlywiki is not the answer, or rather,
tiddlywiki as it is now is not the answer. There's some cool
developments going on over at CCTW, a tiddlywiki based on a database
server, hopefully database requests will in the future be optimized
for tiddlers and slices so minimum to no overhead will be from having
to manipulate the data in javascript... although that's probably hard
to do as all macros are in javascript. But the benefits would be
great, and would probably fit your idea of a relational dictionary.
But when I say in the future, it's a long way away in the future,
months perhaps.

K.

> > found on this thread (http://tinyurl.com/ynugkl )

[wolfgang]

Hi Kashgarinn,

> dictionaries and direct manipulation/indexing of data is really the
> realm of databases.
>
> Tiddlywiki behaves alot like a relational database, but the overhead
> of finding and grabbing the data, either directly or from slices means
> it's a bad database for any wiki which is close to or over 1MB.

I'm not really sure if this is so, unless I really reached the limit -
and I'm ain't there yet - with my biggest TW of almost 10 MB!
Of course, I had to keep some rules to keep this TW at that size
reasonable responsive, like disabling SidebarTabs if not explicitly
needed, limit plugins to a few, the most effective, externalizing them
or loading them on demand...

However, no non-commercial dictionary program gives me the freedom a
TiddlyWiki gives in a way supporting the learning experience of a
foreign language already now. And very few would start to learn a
language with a vocabulary of 50 thousands words anyway...

> But when I say in the future, it's a long way away in the future,
> months perhaps.

Again not sure. If I only knew how to use SQLQueryPlugin... Anyone???

http://jackparke.googlepages.com/jtw.html#SQLQueryPlugin

Regards,

W.

[wolfgang]

> The third problem is that every word should be clickable. If we have
> to wrap all words inside <a href></a> tags, I am afraid this will also
> degrade performance because of javascript processing time to wikify
> and WikiWording all the word. And also, I prefer not to enclose all
> words with double square brackets [[theword]] to make the word
> clickable.
>
> I think this is the time for someone to develop a LinkAllWordPlugin,
> hopefully by using eventBubble?

AutoWeavePlugin automatically creates double-brackets links out of
word existing as a title in a TiddlyWiki on clicking a 'done' of a
tiddler.

Would it be possible to adapt AutoWeavePlugin in a way so that it
creates such links in all tiddlers of a TiddlyWiki on clicking a
saveChanges?

Regards,

W.

[Daniel Baird]

On Thu, Mar 20, 2008 at 7:46 AM, wolfgang <wolfga...@gmail.com> wrote:
>
> Again not sure. If I only knew how to use SQLQueryPlugin... Anyone???
> http://jackparke.googlepages.com/jtw.html#SQLQueryPlugin

Looks like it works in IE only, using some kind of ActiveX object, and
only works when the thing is saved locally.

If that's all OK with you, you might want to email jack and ask for a
demo or something..

;D

--
Daniel Baird
i neeber olok at ym kyebord wen i tpey

[abd shomad]

On 3/20/08, wolfgang <wolfga...@gmail.com> wrote:
>
> > The third problem is that every word should be clickable. If we have
> > to wrap all words inside <a href></a> tags, I am afraid this will also
> > degrade performance because of javascript processing time to wikify
> > and WikiWording all the word. And also, I prefer not to enclose all
> > words with double square brackets [[theword]] to make the word
> > clickable.
> >
> > I think this is the time for someone to develop a LinkAllWordPlugin,
> > hopefully by using eventBubble?
>
> AutoWeavePlugin automatically creates double-brackets links out of
> word existing as a title in a TiddlyWiki on clicking a 'done' of a
> tiddler.
>
> Would it be possible to adapt AutoWeavePlugin in a way so that it
> creates such links in all tiddlers of a TiddlyWiki on clicking a
> saveChanges?
>
> Regards,
>
> W.

Thank you for the suggestion. I will (also) try this plugin :)
Has anyone try the "double-click" feature @ Answer (dot) com? I think
that feature suite the need to implement this dictionary, but I have
no knowledge (yet) as to how to create the plugin.

Abd Shomad

[abd shomad]

On 3/19/08, Kashgarinn <stei...@gmail.com> wrote:
>
> dictionaries and direct manipulation/indexing of data is really the
> realm of databases.
>
> Tiddlywiki behaves alot like a relational database, but the overhead
> of finding and grabbing the data, either directly or from slices means
> it's a bad database for any wiki which is close to or over 1MB.

Hi Kashgarinn,

What do you think if we don't load all the tiddlers? I mean, we can
load the "requested" tiddler on demand (Lazy Loading).
On the very first screen, we display only the list of the alphabet (A
~ Z), and one search box.
When user click on "A", TW will loads the "A.[lang-selected].tiddler"
on demand (Lazy Loading), which can contain as many words starting
with letter "A" (as the index).
And suppose that user clicks on "abacus", then TW will loads the
"abacus.[lang-selected].tiddler" on demand (Lazy Loading, again...).

This can also be applied to the inputted word in the search box. If
user just type "abacus" in the search box, TW will loads the
"abacus.[lang-selected].tiddler" on demand (Lazy Loading, again and
again... ).

I will try this approach and let others know the result (maybe after
some weeks, since I am not yet fluent in TW, javascript and plugin
development).

I think, the key to the performance problem is on the Lazy Loading
mechanism here.

The reason TW is probably a good solution for creating Collaborative
Dictionary is that with LocalAdapterPlugin, other contributors can
submit their "revisions", which is a collection of small tiddlers.
This will attract more users / contributors, since they can work
"offline" and submit their changes / additions only when connected to
the internet.

Regards,
Abd Shomad

[wolfgang]

HI everyone,

> Looks like it works in IE only, using some kind of ActiveX object, and
> only works when the thing is saved locally.

:-(
IE - now way! thanks for that info, Daniel.

> The reason TW is probably a good solution for creating Collaborative
> Dictionary is that with LocalAdapterPlugin, other contributors can
> submit their "revisions", which is a collection of small tiddlers.

Abd, LocalAdapterPlugin wont simplify the submission of revisions in
any other way as if they were submitted as simple text files!

You still would have to open either with an editor, and then copy/
paste their contents into your TiddlyWiki. Nor does it reduce the file
size at all - size all externalized tiddlers still are stored and
remain in the TiddlyWiki!

Regards,

W.

On 21 Mrz., 02:58, "abd shomad" <abd.sho...@gmail.com> wrote:

[wolfgang]

Ok, I gave it a try and created a proof of concept - if and how
TiddlyWiki would be able to handle internal database functions like a
simple Pali-English dictionary.

For this I pasted 36 word lists from http://zencomp.com/greatwisdom/ebud/dict-pe/index.htm
into the text editor Pspad (which has easy to use reg-ex search/
replace) to create wikiformat italics and line-separators for the
further procession of these word lists into separate tiddlers - one
for each word - with SplitTiddler script (TiddlyTools) within the
TiddlyWiki itself. Which worked quite straight forward - thanks to
these 2 tools.

Here the description and comparison of the TiddlyWiki's created en
route:
---------------------------------------
Dictionary Creator TiddlyWiki:
word-list tiddlers: 37
file size: 1229 KB
saveChanges: 3 sec.
reload: 2 sec.
---------------------------------------
Pali-English Dictionary TiddlyWiki:
word tiddlers: 20137
size: 3.140 KB
saveChanges: 40 sec.
reload: 60 sec.
example word search (SearchPlusPlugin): 8 sec.
example word goto (GotoPlugin): 8 sec.
---------------------------------------

That means dividing word lists into separate tiddlers can't be the way
to create such huge databases with TiddlyWiki - because the meta data
of tiddlers is bloating it to almost 3 times the necessary file size!

Though with ScrubTiddlerFields and SetUserName bookmarklets I could
reduce this meta data overhead - still it remained double its original
size:
---------------------------------------
Pali-English Dictionary TiddlyWiki with reduced meta data:
word tiddlers: 20137
size: 2.490 KB
saveChanges: 30 sec.
reload: 50 sec.
example word search (SearchPlusPlugin): 8 sec.
example word goto (GotoPlugin): 8 sec.
---------------------------------------

Even so - surprisingly - this way is a possible approach. Because what
Abd Shomad is envisioning: loading of tiddlers on demand - concerning
the search - is already possible with SearchPlusPlugin. By limiting
the search to those with the first letter of the words (by which they
have been tagged) the search is reduced to milliseconds.

> Remains only IncludePlugin, which seems to make the most sense for
> this particular use case, and for keeping the TiddlyWiki small. And
> fast enough on a saveChanges.

With IncludePlugin one is already able to load specific content on
demand, and create a TiddlyWiki StoreArea html file for each letter of
the alphabet. Further, by creating links invoking the include macro
for each, one is able to load only those tiddlers.

However, practically the loading process of those words (beginning
with one particular letter only) with IncludePlugin also takes in
avarage 5 seconds, and that is not satisfactory enough to me.
..

Remains only one solution :-)

A DatabasePlugin - which is able to store its data in one tiddler
only, each bit using one line each, with a search function that
displays only those relevant lines. Like tiddler sections.

For a multilingual bidirectional dictionary it would have to be able
to read each line of the one store tiddler like a tiddler, or a
section - and within each line create double (or multiple)
compartments of data as slices, or ImprovedSlicesPlugin does. Such a
line could for example look like:

!easy //[adj.]// ... |facil ... |facile ... |leicht ...

where each subsection could be defined (English, Spanish, French,
German) by its order. The first word of each subsection would display
on a search as header - and the predefined search result (ie., which
language) would not only display its first word, but also any
additional definitions.

Such a DatabasePlugin would eliminate all hindrances to huge databases
or dictionaries within TiddlyWikis!

Anyone up for the challenge of writing such a plugin ;-)

Regards,

W.

[wolfgang]

Forgot to give the sources for all the stuff mentioned:

Concise Pali-English Dictionary, A.P. Buddhadatta Mahathera:
http://www.budsas.org/ebud/dict-pe/index.htm
Pspad Editor: http://www.pspad.com/
Multiline Search/Replace: http://www.phdesign.com.au/document.php?page=software_phreplace
DejaVu Sans Mono: http://dejavu.sourceforge.net/wiki/index.php/Download
SplitTiddlers, InlineJavascriptPlugin & ExportTiddlersPlugin:
http://www.TiddlyTools.com/#SplitTiddler
ScrubTiddlerFields & SetUserName bookmarklets: http://lewcid.org/tiddlywiki-bookmarklets/
SearchPlusPlugin & PreferenceSaverLib: http://tw.lewcid.org/svn/plugins/
IncludePlugin & DataTiddlerPlugin: http://tiddlywiki.abego-software.de/
AutoWeavePlugin: http://weave.tiddlyspot.com/index.html#AutoWeavePlugin
LocalAdaptorPlugin & AdaptorCommandsPlugin:
http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/adaptors/
SQLQueryPlugin & ArchivePlugin: http://jackparke.googlepages.com/jtw.html

W.

On 21 Mrz., 17:28, wolfgang <wolfgangl...@gmail.com> wrote:
> Ok, I gave it a try and created a proof of concept - if and how
> TiddlyWiki would be able to handle internal database functions like a
> simple Pali-English dictionary.
>

> For this I pasted 36 word lists fromhttp://zencomp.com/greatwisdom/ebud/dict-pe/index.htm

> ...
>
> Erfahren Sie mehr »

[wolfgang]

> For a multilingual bidirectional dictionary it would have to be able
> to read each line of the one store tiddler like a tiddler, or a
> section - and within each line create double (or multiple)
> compartments of data as slices, or ImprovedSlicesPlugin does. Such a
> line could for example look like:
>
> !easy //[adj.]// ... |facil ... |facile ... |leicht ...

Actually ImprovedSlicePlugin already could accomplish half of the
whole job:
http://www.remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html#ImprovedSlicesPlugin

All what would be needed is a search plugin which is able to interpret
such ImprovedSlices, and displays only the line where the word is
found in the specified way:

> .. The first word of each subsection would display

> on a search as header - and the predefined search result (ie., which
> language) would not only display its first word, but also any
> additional definitions.

W.

> ...
>
> Erfahren Sie mehr »

[wolfgang]

> All what would be needed is a search plugin which is able to interpret
> such ImprovedSlices, and displays only the line where the word is
> found in the specified way:
> >
> > .. The first word of each subsection would display
> > on a search as header - and the predefined search result (ie., which
> > language) would not only display its first word, but also any
> > additional definitions.

Actually I believe it would not take much to adapt YourSearchPlugin to
display search results in the specified way - it already shows a
preview of the line where the word is found.

It comes with a shadowed YourSearchResultTemplate, which would be to
difficult to adapt by someone knowledgeable, I guess.

Anyone?

W.

On 21 Mrz., 19:41, wolfgang <wolfgangl...@gmail.com> wrote:
> > For a multilingual bidirectional dictionary it would have to be able
> > to read each line of the one store tiddler like a tiddler, or a
> > section - and within each line create double (or multiple)
> > compartments of data as slices, or ImprovedSlicesPlugin does. Such a
> > line could for example look like:
>
> > !easy //[adj.]// ... |facil ... |facile ... |leicht ...
>
> Actually ImprovedSlicePlugin already could accomplish half of the

> whole job:http://www.remotely-helpful.com/TiddlyWiki/ImprovedSlicesPlugin.html#...

> ...
>
> Erfahren Sie mehr »

[wolfgang]

Of course, a specific DatabasePlugin able to manage such database
tiddlers would be much more useful.

Optimally it would provide a input panel with fields, where one could
give the specifics of such a slices database in a tiddler. Like the
kind (address book, dictionary, etc.) and which fields to add
(english, french, german .../ name, address, phone, etc.). And then
creates such a tiddler database as "Create tiddlers using custom-built
HTML forms" from TiddlyTools does.

When opening such a panel next, one could choose which database (ie.
tiddlers: dictionary or address book; all tagged with excludeLists
and -Search) to choose from, and add some additional data.
DatabasePlugin would append in that tiddler's database. And exceeding
a certain size, splitting them further up.

In the query panel one could choose - by checking checkboxes - which
fields to search, and which to display. And instead of limiting the
search to tags, the first letter of a searched word could take a
similar function in very rapidly and efficiently locating the data in
the database tiddler (with lines sorted alphabetically). By changing
to edit mode of such a set of found data with input fields, it could
be easily changed again.

Goosh, such a DataQueryPlugin - overcoming all limitations to size -
seems so close in reach...

W.

> ...
>
> Erfahren Sie mehr »

[wolfgang]

> ... And then creates such a tiddler database as "Create tiddlers

> using custom-built HTML forms" from TiddlyTools does.

> http://www.tiddlytools.com/#FAQ_CreateTiddlersFromForms

Of course, it should be able to recognize existing tiddlers in this
improved slice format. And an import function for data created by
DataTiddlerPlugin would be a further bonus.

For escaping table syntax entered into a field box: ||
- it would be very convenient if this DataQueryPlugin would
automatically add non-wikify delimiters in text containing such.

Then, for example for languages like Pali, the search function should
be able to adapt to special characters and search:
a i u d l m m n n t n
- also as:
ā ī ū ḍ ḷ ṁ ṃ ṅ ṇ ṭ ñ
too.

Regards,

W.