performance of a large tiddlywiki

[miles]

Tiddlywiki seems to meet all my requirements for producing an
undergraduate medicine teaching aide. However, before I commit to
(many hours) of data entry I'd like to feel a bit more reassured about
performance...
The tiddlywiki I'm planning will probably have ~3000 tiddlers and ~500
tags with 100-200 words in each tiddler. How would a tiddlywiki of
that size perform ? Does anybody know of any tiddlywiki instances on
the internet of this sort of size (content doesn't matter) that I
could look at as an example ??. Any help much appreciated.

[wolfgang]

Once upon a time I made a TiddlyWiki in which I imported anything
relating to TiddlyWiki: plugins, scripts, help texts.. for having a
searchable TiddlyWiki repository. And avoiding never finding the
things again I knew existed.. - but where had I seen?

A bid a year ago the first alpha of a searchable TW plugin repository
automatically and directly loading on demand anything ever created on
a TiddlyWeb instance made me quit my version. Manually updating was
just to work intense in comparison, beside its biggest drawback being
a loading and 'save changes' time of around 1 minute with FF2.

Once the whole file - about 13 MB with almost 4000 tiddlers - was
loaded, it was still responsive for editing by applying following
important rules:

* only the least and most essential plugins installed - or loaded on
demand via RunTiddlerPlugin (by TiddlyTools)
* tabs which cruise and update with each editing every existing
tiddlers - as in the right sidebar - hidden, and only opened when
really needed

A secondary reason for quiting to update this huge TW was that FF2 a
few times frustratingly failed to save changes after hours of work..

However, to make at least a part of this TiddlyWiki wealth available I
created 'Changing Themes' into which I distilled every theme or style
customizations available then:

http://change.tiddlyspot.com/download

.. which, despite of containing almost 1500 tiddlers at 3,7 - is much
less responsive due to all main plugins installed. As far as I know,
this remains the largest online TW (due to possibe slow connection
speed above link will start a download to view and edit it locally).

Just yestersday I was really surprised to find that with a portable
version of FF3.5 (still only used FF2) the loading and saving time of
the 13 MB TW has amazingly decreased below 10 seconds!

However, FF3.5 versions still seems a bid buggy to me (much
intermittent freezing) I would recommend making frequent backups
during working with huge TiddlyWiki, which due to the acceptable
saving speed, is a viable option now. :-)

Hope that helps a bid further..

[miles]

Many thanks for info. and advice.
I downloaded the change.html tiddlywiki
My immediate impressions are that:
- simple operations (opening tiddlers etc) work okay
- jscript freezes do occur
- it may be best to avoid plugins if possible

The demands for the wiki are not great: the end product would
hopefully look like a straightforward instance such as
http://reasoningwell.tiddlyspot.com/, just a lot larger !!

I would still *much* rather go via the tiddlywiki route than consider
the various other php/mysql hosted wiki engines.

It would be great for the completed wiki to be on the university site
and then students can get a dump of the wiki plus the data by just
saving the tiddlywiki html locally...

[Anthony Muscio]

Miles,

Couple of suggestions on taking this path. Use some kind of batch process to generate a file of dummy tiddlers of the size you want without regard to the actual content and import them all, and actualy test the scale you are trying to use.

Use the tips in;
http://www.giffmex.org/twfortherestofus.html#[[Tips%20for%20speeding%20up%20performance%20on%20large%20TiddlyWikis]]

And also note that dependent upon the nature of the data in your tiddlywiki using external, temporary tiddler etc I am sure you can do a lot more if you are not wedded to a single file.

You may need to alter the firefox or browser time out on Java to allow it to run a little longer so you d not need to press ok too many times.


Regards Tony


TonyM

If you have not found an easy way to do it with TiddlyWiki, you have missed something.
www.tiddlywiki.com

[Måns]

Hi Miles

"Use some kind of batch process to
> generate a file of dummy tiddlers of the size you want without regard to the
> actual content and import them all, and actualy test the scale you are
> trying to use."

How I would generate 3000 dummy tiddlers with 2-300 words each and
optionally tag them...

First download an empty tw-document - install Erics
ImportTiddlersPlugin. http://www.tiddlytools.com/#ImportTiddlersPlugin
Create a spreadsheet with these column headlines: title text tags
field1 field2 field 3 etc..
For the titles column I would simply drag a numbered list (up to 3000
if you like), for the text column I would copy/paste a textsnip in
every individual cell (drag or copy into all at once) with a lenght of
2 - 300 words (be carefull with commas! - see plugininformation)
optionally tag with whatever tag you'd like - to be able to group
tiddlers by category - and - again optional - create custom fields for
each tiddler...

Save the spreadsheet to a csv-file and import all tiddlers to the
empty TW at once...

I haven't actually tried this myself - but theoretically it should be
possible..

Regards Måns Mårtensson

[Måns]

I've just tested the import of 3000 tiddlers with a text of 300 words
each vis csv - and it completed the action in less than 15 sec!!
The csv-file is almost 5 megs (4.6) - and the TW is little bigger.
(5.3)
You can check them both out here: csv: http://tinyurl.com/ydb5bqt
TiddlyWiki with 3000 tiddlers - all tagged "importedTiddlers" :
http://tinyurl.com/y86y9of

Regards Måns Mårtensson

On 30 Sep., 14:03, Måns <humam...@gmail.com> wrote:
> Hi Miles
>
> "Use some kind of batch process to
>
> > generate a file of dummy tiddlers of the size you want without regard to the
> > actual content and import them all, and actualy test the scale you are
> > trying to use."
>
> How I would generate 3000 dummy tiddlers with 2-300 words each and
> optionally tag them...
>
> First download an empty tw-document - install Erics

> ImportTiddlersPlugin.http://www.tiddlytools.com/#ImportTiddlersPlugin

[mck]

Hi Miles,

With much assistance from this group, I've developed a tiddlywiki to
keep track of over 1400 master's theses in counseling psychology:

http://www.pacifica.edu/lib/theses.html

The current size is 648 kb but this is a bit deceptive as the site
pulls in abstract information from a separate tiddlywiki using Eric's
ExternalTiddlersPlugin:

http://www.tiddlytools.com/#ExternalTiddlersPlugi

The second tiddlywiki containing the abstracts is at

http://www.pacifica.edu/lib/abs.html

and is currently 511 kb in size.

So far, only the 2008-2009 theses contain abstracts and there is a
very brief delay before the abstract information appears.

I've used tinytiddly to further reduce the overall file size:

http://tinytiddly.tiddlyspot.com/

As others have indicated, the performance is improved by hiding the
index and timeline and by turning off the animation.

I've created a CookieJar tiddler tagged with systemConfig to fix the
animation setting:

{
config.options.chkAnimate=false;
}

Eric recently suggested some great code to efficiently generate the
"inline tabs using only TW-native syntax, all in one tiddler without
any plugins or scripts":

http://groups.google.com/group/tiddlywiki/msg/68e8958bcff5809f

This works well in conjunction with the ForEachTiddlerPlugin to
generate the tabbed content: by Author, by Title, by Year, etc.

http://tiddlywiki.abego-software.de/#ForEachTiddlerPlugin

I believe this tabbed interface further improves performance as only a
fraction of the tiddlers are invoked at any one time.

I've also used the HideWhenPlugin to add content to the ViewTemplate
that otherwise would have to be repeated in every tiddler:

<div macro="showWhen tiddler.tags.contains('cp')">Unpublished master's
thesis. Pacifica Graduate Institute, Carpinteria, CA.</div>

http://mptw.tiddlyspot.com/#HideWhenPlugin

These ViewTemplate tweaks combined with the use of custom fields have
enabled me to generate the tiddler citation information on the fly:

Circo, C. J. (2009).
Dancing with the trickster: Expressive arts therapy and the trickster
dynamic.
Unpublished master's thesis. Pacifica Graduate Institute, Carpinteria,
CA.

Also, whenever I found repeatable content such as:

"The electronic copy of this thesis is on a disc containing a single
PDF file that can be opened with Adobe Acrobat Reader."

I would move such content into a separate tiddler (ec) and then use
the tiddler macro to invoke it:

<<tiddler ec>>

Overall, the site performs quite well and I trust I will be able to
scale it up as I add more content.

Yet, I am always open to new suggestions on how to keep the size small
and the performance snappy.

Mark

[mck]

[Dave Gifford]

An example of a library database TiddlyWiki with 3500 entries, each
with several form fields, that runs quickly, and has a large number of
tags:

http://www.giffmex.org/bibliotecaSTRM.html

Dave Gifford

[AlfonsoReyes]

Hello,
could you point me please to an example of the use of
RunTiddlerPlugin?
I have a 1.1 MB TW that is tuning slow because of the use of plugins
like jsMath, IntelliTagger, PartTiddler, SyntaxHighlighter,
YourSearch and fifteen more.
Some of my tiddlers would need to use SyntaxHighlighter but not called
most of the time.
Thanks,
Alfonso

[wolfgang]

Hi Alfonso,

for example at it's original location you will find a 'run' command in
every tidders toolbar. So if it would be a systemConfigDisable tagged
tiddler clicking on 'run' would load it as a plugin (as it usually
does on every startup dependent on tag). However, this doesn't works
with every plugin, so just try them out and find with which it works.

http://www.tiddlytools.com/#RunTiddlerPlugin

An other way is to have RunTiddlerPlugin run automatically once a
tiddler dependent on a disabled plugin displays. In the following
tidder you have an example of TreeViewPlugin loaded only once this
tiddler loads (open in edit mode to see):

http://menuflex.tiddlyspot.com/#TreeMenu

Regards..

[miles]

**Many thanks** everyone.

Måns's dummy file http://dl.getdropbox.com/u/1064531/CsvTest/empty.html
is pretty close to what I'm looking for and seems to work reasonably
well.

I'm going to ahead and show some of this stuff to the tech bods at
work and see what they think.

I'll report back if / when things go ahead.

Miles