TiddlyWeb: Lazy loading and load-on-demand

[Oveek]

I'm following up a quick chat with Chris, and FND, on #tiddlyweb,
about delayed loading of tiddlers. The aim is to discuss possible
implementations in TiddlyWeb and get started working on one or more of
them.

This is by no means a new topic and FND pointed me at a thread
summarizing an experimental implementation in ccTiddly:
http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/6176bd7c7ae2400f/e02077f7ccfa3353?lnk=gst&q=skinny+tiddlers#e02077f7ccfa3353

In fact, more recent topics on the ccTiddly group suggest Simon is
continuing to tackle this problem; so there is already an approach we
can gain insights from.

Just to restate the point of this: the primary motivation is to reduce
the initial load time of a large TiddlyWiki, and generally to improve
scalability. I roughly think of initial load time as the time elapsed
between hitting go in your browser and seeing a fully rendered
tiddlywiki with all default tiddlers displayed.

It might help to distinguish two approaches to delayed loading of
tiddlers (corrections appreciated):

*lazy loading: After first opening the wiki, the starting tiddlers are
loaded, and additionally the client is sent the names of all the other
tiddlers on the server.
-This is done using skinny tiddlers which are essentially just the
title (and meta data?). It amounts to sending a table of contents to
the user when they first load the wiki, and later filling in tiddler
bodies upon request.
-The initial load time depends on the number of tiddlers in the wiki.

*load on demand: After first opening the wiki, the starting tiddlers
are loaded, but no information about other tiddlers on the server is
sent to the client.
- How this would be done remains to be seen.
- Initial load time is independent of the number of tiddlers.

Both approaches ought to yield major performance improvements. As
noted, lazy loading will incur a small increase in initial load time
as tiddlers are added to the wiki, but this increase should only start
to even be noticeable after adding a large number of tiddlers.

Load on demand should provide constant load time no matter how many
tiddlers are in the document. Theoretically Wikipedia could be stored
in a TW and the initial load would still be snappy.


I think the problem of implementing either of these methods can be
divided into two main questions:

1) What gets loaded initially?
Default tiddlers, plugins, etc,.

2) How are requests handled after the initial load is done and the
user starts interacting with the wiki?
This concerns retrieval of additional tiddlers, searching, etc,.

Question 2 brings up the trade off that comes with delaying tiddler
loading: namely waiting for tiddlers to load when links to tiddlers
are clicked.

In general it would be nice to get the benefit of a quick "boot up",
while still preserving the seamless navigation we get with a
standalone TiddlyWiki (tiddlers show up the moment you click them).

I noticed in FND's thread the nice idea of streaming tiddlers in the
background. I think a cool variation would be to stream or "prefetch"
only tiddlers that are linked from the currently open tiddler(s). This
might achieve a compromise between not transferring unrequested data
and trying to maintain smooth navigation. Anyway that's for later
consideration.

For now I've conveniently ignored any painful realities and stuck to
concepts. Please correct anything you find inaccurate, I've actually
written this post while knowing very few details.

Hopefully the core devs and others can give their input and we can
move towards some concrete aspects of developing the feature in
TiddlyWeb.

[tony]

On May 12, 9:34 am, Oveek <mov...@gmail.com> wrote:
> I'm following up a quick chat with Chris, and FND, on #tiddlyweb,
> about delayed loading of tiddlers. The aim is to discuss possible
> implementations in TiddlyWeb and get started working on one or more of
> them.

Hi,

a wee bit of user input:

One helpful idea I've seen is an initial splash screen indicating to
the user that load time is occuring or whatever message ie just
tiddlers titles are loading.
While it doesn't contribute anything to the solution, this small
anticipatory guidance may be helpful user experience-wise regardless
of the load approach.
A good example is Eric's www.tiddlytools.com which is pretty massive
but yet that initial screen reassures me (user) that I need to 'wait'
so I don't think something is amiss.
I'm glad there is rumination on the issue as I initially checked out
TiddlyWeb to see if it would help with scability which it does well
with the filters and revisions but when a recipe is large, rendering
as tiddlers.wiki view acts just like a regular tw.
In practice, however I've learned to chunk out/split my content better
and with the newer browsers, I rarely encounter prolonged load times
anymore.
But I'm really glad you all are looking into it just in case!

Best,
tony

[rakugo]

I would personally like to see a solution that starts with ALL user
content loaded in a skinny form eg. title and tags, possibly fields,
with the exception of systemConfig tagged tiddlers and maybe overriden
shadow tiddlers. I think this is important as a lot of plugins make
use of tagging and it might be important to load all tags firsthand.

A javascript dictionary object is also created that indicates the
status of whether content has loaded we'll call this tiddler_status.
I then imagine a world where before any tiddler is used (possibly this
would be in store.getTiddler(), a check is made to see whether the
content has been loaded yet and if not it is pulled in eg. if(!
tiddler_status[foo]) load the tiddler using ajax.

It could be interesting if random tiddlers are loaded during times of
inactivity while the user browses the tiddlywiki, I think the
important thing is the initial load time, as I am greatly put off by
TiddlyWiki sites that take a long time to load. Mine using ccTiddly is
terrible and I'm desperate for a decent solution here.

As another talking point, maybe a stupid suggestion (it's getting late
and I'm maybe not thinking clearly after a glass of wine) would it
yield faster results if tiddler data instead of being written to the
dom was written to the store in javascript world in the head of the
document as packed javascript and added to the storeArea div when
required by displayTiddler? I know writing lots of dom elements causes
quite a big slow down so this might speed loading time up but I'm not
sure whether this would be easy to modify TiddlyWiki to do. Anyway
probably stupid... :)

That's my 2 cents.

On May 12, 5:34 pm, Oveek <mov...@gmail.com> wrote:
> I'm following up a quick chat with Chris, and FND, on #tiddlyweb,
> about delayed loading of tiddlers. The aim is to discuss possible
> implementations in TiddlyWeb and get started working on one or more of
> them.
>
> This is by no means a new topic and FND pointed me at a thread

> summarizing an experimental implementation in ccTiddly:http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/617...

[chris...@gmail.com]

On May 12, 5:34 pm, Oveek <mov...@gmail.com> wrote:

> I'm following up a quick chat with Chris, and FND, on #tiddlyweb,
> about delayed loading of tiddlers. The aim is to discuss possible
> implementations in TiddlyWeb and get started working on one or more of
> them.

Glad you started this thread. I think you're smart to try and define
the terms up front as there is a lot of ambiguity and in order to
really talk about this stuff we need to know what we're talking about.

> Just to restate the point of this: the primary motivation is to reduce
> the initial load time of a large TiddlyWiki, and generally to improve
> scalability. I roughly think of initial load time as the time elapsed
> between hitting go in your browser and seeing a fully rendered
> tiddlywiki with all default tiddlers displayed.

In this use case do you assume that the server is always going to be
available; available for at least the initial load time, but maybe not
later; or server availability is an unknown. The first option is
clearly the easiest.

> It might help to distinguish two approaches to delayed loading of
> tiddlers (corrections appreciated):

[lazy loading and on demand]

What, in your estimation, are things that are gained by sending the
list of tiddlers at load time?

It does lots of things, but it is also possible to get by without it.
Try this URL, clicking on some of the links in the list in the
TiddlyWeb Documentation tiddler:

http://tiddlyweb.peermore.com/wiki/recipes/docs/tiddlers.wiki?mselect=bag:system,title:TiddlyWeb%20Documentation#[[TiddlyWeb%20Documentation]]

If it's working right, what should happen is the tiddlers are loaded
from the server, as you navigate along them.

(I'm just at the end of a train ride, so won't stop to explain what's
going on here in this message, but will write more later.)

Presumably things like the time line and tag handling are the main
wins to be had by providing the tiddler list and metadata.

> Both approaches ought to yield major performance improvements. As
> noted, lazy loading will incur a small increase in initial load time
> as tiddlers are added to the wiki, but this increase should only start
> to even be noticeable after adding a large number of tiddlers.

This is an assumption that probably deserves some testing to establish
if it is true or not. It is going to depend a lot on different
variables:

* the bandwidth of the user's connection
* the processing speed available to the browser when the tiddlywiki is
loaded
* how long it takes a server-side to "compose" the tiddlywiki server
side
* whether being lazy or on demand has much impact on that composition
time server side
* whether the server side is able to do any caching[1]
* other things I can't think of right this minute

[1] See the caching hoster plugin: http://tiddlyweb.peermore.com/wiki/#cachinghoster

> 1) What gets loaded initially?
> Default tiddlers, plugins, etc,.
>
> 2) How are requests handled after the initial load is done and the
> user starts interacting with the wiki?
> This concerns retrieval of additional tiddlers, searching, etc,.

Yes.

> I noticed in FND's thread the nice idea of streaming tiddlers in the
> background. I think a cool variation would be to stream or "prefetch"
> only tiddlers that are linked from the currently open tiddler(s). This
> might achieve a compromise between not transferring unrequested data
> and trying to maintain smooth navigation. Anyway that's for later
> consideration.

That's a good idea.

[Oveek]

This post became pretty big, hope you guys can stay awake :)

> In this use case do you assume that the server is always going to be
> available; available for at least the initial load time, but maybe not
> later; or server availability is an unknown. The first option is
> clearly the easiest.

To begin with I assumed the server is always going to be there. At the
least I think we have to assume the server is available for the
initial load, else the user won't even be able to start doing
anything; that is unless there's a provision for working offline with
a local standalone copy. If server availability is a question mark
then the ability to fall back to standalone mode would allow work to
continue till the server is accessible again.

> What, in your estimation, are things that are gained by sending the
> list of tiddlers at load time?

As you noted, the tags and timeline are big ones. Both could be
especially important for newcomers unfamiliar with what content is
available on the site. That would then require skinny tiddlers (title
plus field meta data).

If we're absolutely focused on the fastest possible load, maybe we
could do a hybrid loading process, something like:
* First load the default tiddlers / initial tiddlers
* Then proceed with background loading:
** Prefetch tiddlers linked from currently open tiddlers. This step
would repeat each time a new tiddler is opened.
** Proceed to download the remaining tiddlers in "skinny" form
allowing the remainder of the timeline and list of tags to be filled
in.

Or a slower simplification of the above:
* First load the default tiddlers in full along with the rest of the
tiddlers in skinny form.
* Prefetch tiddlers as necessary.

Depending on the size of the wiki, and assuming we want the entire
timeline available, the hybrid method may noticeably speed up the
initial load by allowing the full "table of contents" to stream in the
background in the form of skinny tiddlers while the user starts
looking at the content in the initial tiddlers.

> > Both approaches ought to yield major performance improvements. As
> > noted, lazy loading will incur a small increase in initial load time
> > as tiddlers are added to the wiki, but this increase should only start
> > to even be noticeable after adding a large number of tiddlers.
>
> This is an assumption that probably deserves some testing to establish
> if it is true or not. It is going to depend a lot on different
> variables:

Absolutely. That was an optimistic assumption made to cast lazy
loading in a more favorable light when compared to on demand loading
in terms of performance. That was when I thought lazy loading was
going to be the easier one to implement.

I think the effects of the variables you mentioned bears thinking. I
haven't had a chance to look at the caching hoster plugin so I'm not
sure what role that would play. It would be nice to run some numbers
to get ballpark figures on load times for various sized wikis. I'm
curious how many tiddlers it would take before transferring a full
list of skinny tiddlers starts becoming really noticeable.


> http://tiddlyweb.peermore.com/wiki/recipes/docs/tiddlers.wiki?mselect...[[TiddlyWeb%20Documentation]]

>
> If it's working right, what should happen is the tiddlers are loaded
> from the server, as you navigate along them.

That's awesome! It's working beautifully! I probably find seeing that
in action more exciting than I should.

I jumped around clicking several of the tiddly links trying to get a
sense of how disruptive the load times are compared to the normal
uninterrupted flow of navigation in an offline TiddlyWiki. It's a
little hard to assess objectively right now becauses part of me is
just excited to see on demand loading working before my eyes.

Apart from the linked example I tried using the "filter" tiddler as a
starting point which provided a nice avenue for exploration. I kept
Firebug open and watched the latencies of the GET requests. On average
they were below 1 second, most in the range of 600 to 800 ms. My
initial impression is that load times in that range are acceptable,
but I don't know whether that will hold up in the long run.

Based on the definitions from the initial post this is really a fully
implemented form of load on demand functionality. Contrary to what I
originally thought lazy loading is actually the option that would
require more work, is that right?

I looked in the core to try to get a more concrete picture of how
these things are implemented in TiddlyWiki.

If I'm not mistaken, load on demand is actually the natural mode of
operation for TiddlyWiki's core adaptor mechanism. Clicking a tiddly
link starts a chain of function calls beginning with the
onClickTiddlerLink() event handler. If the tiddler isn't found in the
local store area, the Story object's loadMissingTiddler() function
gets called which in turn invokes the TiddlyWebAdaptor via the
SyncMachine. The SyncMachine performs a few steps; it executes some of
the adaptor's methods including the adaptor's getTiddler() method
which sends an HTTP request to the TiddlyWeb server. If the server
finds the requested tiddler it is sent back in JSON serialized format.
That's roughly my understanding of what happens on the client-side.

It appears load on demand is a byproduct of implementing the adaptor
interface.

On a related note, I did see one issue. There's no notification when a
tiddler doesn't exist. Upon opening a tiddler it initially appears
with the message "Attempting to retrieve the tiddler..." and even when
the GET returns a 404, the same message continues to display with no
indication that anything happened.

To summarize:

As far as load on demand is concerned it seems what actually needs
work is how to send just a subset of tiddlers initially (and which
ones to send). Is that accurate?

For lazy loading it's additionally necessary to add support for
handling skinny tiddlers because the adaptor does not handle or
recognize them by default. Also using the hybrid loading approach I
think the perceived performance of lazy loading could match that of
load on demand.

[chris...@gmail.com]

On May 15, 8:37 pm, Oveek <mov...@gmail.com> wrote:
> > What, in your estimation, are things that are gained by sending the
> > list of tiddlers at load time?
>
> As you noted, the tags and timeline are big ones. Both could be
> especially important for newcomers unfamiliar with what content is
> available on the site. That would then require skinny tiddlers (title
> plus field meta data).

This makes sense.

> If we're absolutely focused on the fastest possible load, maybe we
> could do a hybrid loading process, something like:
> * First load the default tiddlers / initial tiddlers

Which means we need some way to indicate those initial tiddlers. In a
TiddlyWeb environment we could do this by indicating a bag, a set of
bags, or some other collection via a filter on the recipe. Another
option is to extend the wiki serialization so it takes some additional
arguments that allow the declaration of the defaults by some other
parameter string.

> * Then proceed with background loading:
> ** Prefetch tiddlers linked from currently open tiddlers. This step
> would repeat each time a new tiddler is opened.

This would require adaptation (as you've discovered) of the
SyncMachine/ServerAdaptor code.

> ** Proceed to download the remaining tiddlers in "skinny" form
> allowing the remainder of the timeline and list of tags to be filled
> in.

Again, in the TiddlyWeb context this would be a matter of getting the
JSON list of the tiddlers generated by the recipe being accessed.
These results are skinny by default (passing fat=1 includes the text).
Making that request and then processing the data would require either
a new plugin in TiddlyWiki or adaptations to existing.

> >http://tiddlyweb.peermore.com/wiki/recipes/docs/tiddlers.wiki?mselect...[[TiddlyWeb%20Documentation]]
>
> > If it's working right, what should happen is the tiddlers are loaded
> > from the server, as you navigate along them.
>
> That's awesome! It's working beautifully! I probably find seeing that
> in action more exciting than I should.

When I first did it I got all excited too :)

> Apart from the linked example I tried using the "filter" tiddler as a
> starting point which provided a nice avenue for exploration. I kept
> Firebug open and watched the latencies of the GET requests. On average
> they were below 1 second, most in the range of 600 to 800 ms. My
> initial impression is that load times in that range are acceptable,
> but I don't know whether that will hold up in the long run.

There are a variety of ways in which this could probably be sped up.
TiddlyWeb, in its current form, is almost completely non-optimized.
For those situations where speed is required there are likely many
things that can be done.

> Based on the definitions from the initial post this is really a fully
> implemented form of load on demand functionality. Contrary to what I
> originally thought lazy loading is actually the option that would
> require more work, is that right?

Seems so, at least in terms of the work required in TiddlyWiki to make
things go.

> If I'm not mistaken, load on demand is actually the natural mode of
> operation for TiddlyWiki's core adaptor mechanism.

Yes.

> On a related note, I did see one issue. There's no notification when a
> tiddler doesn't exist. Upon opening a tiddler it initially appears
> with the message "Attempting to retrieve the tiddler..." and even when
> the GET returns a 404, the same message continues to display with no
> indication that anything happened.

Yeah, this is a bug in, I think, the sync machine which is not as
savvy about HTTP response codes as it could be. I'll make a ticket.

> As far as load on demand is concerned it seems what actually needs
> work is how to send just a subset of tiddlers initially (and which
> ones to send). Is that accurate?

Yes.

> For lazy loading it's additionally necessary to add support for
> handling skinny tiddlers because the adaptor does not handle or
> recognize them by default. Also using the hybrid loading approach I
> think the perceived performance of lazy loading could match that of
> load on demand.

This sounds right too.

[FND]

> the primary motivation is to reduce the initial load time of a
> large TiddlyWiki, and generally to improve scalability

Personally, I'm more concerned about scalability than initial load time
(gzip seems to do a good job reducing that).
The TiddlyWiki client hits a wall if there's a massive amount of
tiddlers to be processed (think for-each-tiddler operations like
calculating the timeline, tags etc.).

Those should probably be considered two separate, though related issues.
I guess the way to approach this is tackling load time first, then see
what we've learned about scalability.

> distinguish two approaches to delayed loading of tiddlers

As discussed, I think that distinction is correct and important.
Even though one might argue about the semantics of "lazy" vs.
"on-demand" loading, let's just stick to those labels for now and use
"dynamic" loading for the more general concept.

I've tried to summarize the basic points raised so far on our wiki:
http://tiddlywiki.org/wiki/Dev:Dynamic_Loading

> would it yield faster results if tiddler data instead of being
> written to the dom was written to the store in javascript world in
> the head of the document as packed javascript and added to the
> storeArea div when required by displayTiddler?

Well, you seem to be confusing store and story there.
However, you're indirectly raising the issue of a different store
implementation (using JSON instead of HTML) - which might very well be
worth discussing, though preferably in a separate thread.

-- F.

[FND]

> do you assume that the server is always going to be available

I think that's a fair assumption when dealing with dynamic loading -
provided, of course, that it's optional and you can still get the full
document for taking it offline.

> If it's working right, what should happen is the tiddlers are loaded
> from the server, as you navigate along them.

[...]

> If I'm not mistaken, load on demand is actually the natural mode of
> operation for TiddlyWiki's core adaptor mechanism.

Indeed, TiddlyWiki already has basic support for loading tiddlers on
demand. However, that (currently) only works if a tiddler is requested
explicitly through the UI, so it's incompatible with many core functions
(e.g. search, references) and plugins.

> There's no notification when a tiddler doesn't exist.

We're aware of that (the mechanism is really very basic at the moment).
I don't think there's a ticket for this though - could you report the
issue on Trac?

So, now that we've outlined the issues, where does this leave us - what
could/should we start with to create a simple proof of concept?

-- F.

[Oveek]

On May 18, 12:01 am, FND <F...@gmx.net> wrote:
> As discussed, I think that distinction is correct and important.
> Even though one might argue about the semantics of "lazy" vs.
> "on-demand" loading, let's just stick to those labels for now and use
> "dynamic" loading for the more general concept.
>
> I've tried to summarize the basic points raised so far on our wiki:
> http://tiddlywiki.org/wiki/Dev:Dynamic_Loading

Thanks for boiling the fat off of this discussion and writing down
that summary. On a related note I have to commend you guys on the
documentation here as well: http://tiddlyweb.peermore.com/wiki/, it's
really exemplary. Keep up the great work. I've referred to it
countless times already and have benefited greatly from the clarity
and throughness of the information on everything from the API to
filters. Tiddlyweb is doing an admirable job of serving as a platform
for documenting itself.

> We're aware of that (the mechanism is really very basic at the moment).
> I don't think there's a ticket for this though - could you report the
> issue on Trac?

Chris mentioned making a ticket... I'm assuming he was going post on
Trac.

[Oveek]

By the way, I think dynamic loading is a good umbrella term for lazy
loading, on demand loading, and any other variation that may come up.

On May 18, 12:21 am, FND <F...@gmx.net> wrote:
> So, now that we've outlined the issues, where does this leave us - what
> could/should we start with to create a simple proof of concept?
>
> -- F.

I've taken a stab at creating a basic load on demand setup and just
pushed two plugins to my fork of tiddlyweb-plugins:

* loadinitial: http://github.com/moveek/tiddlyweb-plugins/tree/8e7421acb8dc6a242dfec246e7d8e5410afd9d38/loadinitial
* tiddlylinkfilter:
http://github.com/moveek/tiddlyweb-plugins/tree/8e7421acb8dc6a242dfec246e7d8e5410afd9d38/tiddlylinkfilter

To try this out, grab both plugins, copy the config from the
tiddlywebconfig.py supplied with loadinitial, and optionally tweak the
list of initial tiddlers. The list of initial tiddlers in the config
file is just an mselect query expressed as a dictionary (more on this
below and in the README). The tiddlylinkfilter plugin extends the
filter syntax with a new select attribute that is used in specifying
some of the intial tiddlers in the aforementioned dictionary. I should
note loadinitial requires the presence of the mselect plugin.

It should be possible to pretty much plug and play. Install the
plugins, and next time you access a tiddlywiki via a recipe, only the
tiddlers listed in the config dictionary should load, the rest will be
retrieved on demand. This basically amounts to duplicating the example
Chris linked earlier, but doing it on the serverside, and adding a
simple mechanism for controlling the select filter from the TiddlyWeb
config file.

Actually there's one minor issue that needs fixing before you can plug
and play. The loadinitial plugin makes use of a custom key in the
config dictionary called 'initial_tiddlers'. On encountering this
custom key, the try block in the read_config() function (in config.py)
throws a KeyError that is currently unhandled. Simply adding it to the
except clause solves this problem.

Both plugins have detailed READMEs, but I'll put a little explanatory
info here:

Since the question of what tiddlers are essential at load time is a
bit open ended, the idea was to allow control over this aspect from
tiddlywebconfig.py. Among the candidates for initial load are:

* the tiddler DefaultTiddlers
* tiddlers tagged systemConfig,
* StyleSheet and other overridden shadow tiddlers

All of those can contribute to the initial state of a tiddlywiki.
Arguably the most obvious one is DefaultTiddlers.

The plugins aim to provide a logical default selection based on the
tiddlers listed above, and provide an easy way to add or remove
tiddlers as desired.

loadinitial makes it possible to list the tiddlers intended for
initial load as a dictionary in tiddlywebconfig.py. Example:

'initial_tiddlers': {
'in': ['DefaultTiddlers'],
'tag': ['systemConfig'],
'title': ['SiteTitle', 'SiteSubtitle']
}

The plugin takes the dictionary and expands it into an mselect query.
The code that converts the dictionary is quite flexible: any number of
attributes supported by the select filter can be used in the
dictionary, and each attribute can take a list of any number of
values. For instance to preserve only tiddlyweb's serverside
functionality, the 'tag':[systemConfig'] line could be replaced with
'bag':['system']. For the sake of illustration, the above dictionary
is expanded into the following filter query:


mselect=title:SiteTitle,title:SiteSubtitle,tag:systemConfig,in:DefaultTiddlers

The custom select attribute 'in' is provided by the tiddlylinkfilter
plugin. It takes the name of a tiddler (call it the condition tiddler)
and selects the condition tiddler and all tiddlers referenced in the
condition tiddler. If the condition tiddler links to 20 other
tiddlers, then all 20 of those tiddlers plus the condition tiddler are
selected. In the example above, the filter first creates a list of all
the tiddlylinks in the body of DefaultTiddlers; the compiled list of
tiddlers is then used for selection.
The filter was specifically written to handle the DefaultTiddlers case
as TiddlyWiki expects all tiddlers listed in DefaultTiddlers to be
available on load. This kind of touches on the concept FND mentions in
his thread summary about determining essential tiddlers recursively.
It might be termed something like "deep select."

Well that's about it. I'm running this setup right now and noting
problems that come up. This is just experimental and of course doesn't
solve any other issues like handling of searches, but it provides a
pretty good way of further testing a load on demand setup.

[chris...@gmail.com]

On May 22, 8:31 pm, Oveek <mov...@gmail.com> wrote:
> I've taken a stab at creating a basic load on demand setup and just
> pushed two plugins to my fork of tiddlyweb-plugins:
>

> * loadinitial:http://github.com/moveek/tiddlyweb-plugins/tree/8e7421acb8dc6a242dfec...
> * tiddlylinkfilter:http://github.com/moveek/tiddlyweb-plugins/tree/8e7421acb8dc6a242dfec...

Well this is just awesome. Very happy to see this.

By the way, if you haven't already done it, it's probably best to make
your fork for tiddlyweb-plugins have tiddlyweb/tiddlyweb-plugins as
its upstream, not cdent/tididlyweb-plugins.

> To try this out, grab both plugins, copy the config from the
> tiddlywebconfig.py supplied with loadinitial, and optionally tweak the
> list of initial tiddlers. The list of initial tiddlers in the config
> file is just an mselect query expressed as a dictionary (more on this
> below and in the README). The tiddlylinkfilter plugin extends the
> filter syntax with a new select attribute that is used in specifying
> some of the intial tiddlers in the aforementioned dictionary. I should
> note loadinitial requires the presence of the mselect plugin.

This is a nicely clever idea. Takes advantage of existing parts of the
system very well.

> Actually there's one minor issue that needs fixing before you can plug
> and play. The loadinitial plugin makes use of a custom key in the
> config dictionary called 'initial_tiddlers'. On encountering this
> custom key, the try block in the read_config() function (in config.py)
> throws a KeyError that is currently unhandled. Simply adding it to the
> except clause solves this problem.

I've got a fix for this that I'm just about to check in. Guess I never
tried to add a dict to the config for non-existent key.

> The plugins aim to provide a logical default selection based on the
> tiddlers listed above, and provide an easy way to add or remove
> tiddlers as desired.

Is nice.

> The custom select attribute 'in' is provided by the tiddlylinkfilter
> plugin. It takes the name of a tiddler (call it the condition tiddler)
> and selects the condition tiddler and all tiddlers referenced in the
> condition tiddler. If the condition tiddler links to 20 other
> tiddlers, then all 20 of those tiddlers plus the condition tiddler are
> selected. In the example above, the filter first creates a list of all
> the tiddlylinks in the body of DefaultTiddlers; the compiled list of
> tiddlers is then used for selection.

This is a very useful filter, for a variety of things. It will be
handy for abstracting a lot of duplicated code in some of the plugins
that already exist.

> Well that's about it. I'm running this setup right now and noting
> problems that come up. This is just experimental and of course doesn't
> solve any other issues like handling of searches, but it provides a
> pretty good way of further testing a load on demand setup.

How's it working out so far? Any surprises?

It's really great to see you doing this work. I'm especially glad that
you've posted up the code so people can have a poke themselves.

The way you've created your plugins is cool to see and really shows
off some of the flexibility I hoped, but wasn't sure, that I was
building into the system. You've made your plugins in a way that is
almost entirely different from how I would do it, but in a perfectly
reasonable and potentially better way.

One of the side effects of on demand loading is that the server will
be respond to [...]/tiddlers/{tiddler_name} URLs more than it has been
in the past, where lists of tiddlers were more common. Work I've been
doing recently should make it possible to make these operations quite
fast. So nice overlap. I'll post about some of that work shortly.

[Oveek]

Been trying to get around to replying...

On May 23, 10:52 pm, "cd...@peermore.com" <chris.d...@gmail.com>
wrote:

>By the way, if you haven't already done it, it's probably best to make
>your fork for tiddlyweb-plugins have tiddlyweb/tiddlyweb-plugins as
>its upstream, not cdent/tididlyweb-plugins.

Yea I later realized most of the commit activity is in tiddlyweb/
tiddlyweb-plugins. Is cdent/tiddlyweb mainly for releases?

> This is a very useful filter, for a variety of things. It will be
> handy for abstracting a lot of duplicated code in some of the plugins
> that already exist.

One point about this filter (which I clumsily attempted to explain in
the readme). The filter interface basically says that filters only
know about one tiddler at a time. The tiddlylinkfilter has an extra
requirement due to the initial step of accessing the store to pick up
the condition tiddler. The way this is currently done is kind of
hacky, and runs into trouble in a special case.

There's a potential problem when the filter is applied to recipes that
have multiple copies of the condition tiddler. If I understand
correctly, when there are multiple copies of a tiddler in the list of
bags making a recipe, the recipe uses whichever one appears in the
later bag. To give the right results the tiddlylinkfilter should use
the same condition tiddler used in the recipe after it is
'uniquified', but as far as I can see, to do that requires knowing the
bags used by the recipe (and the order they occur in).

> > Well that's about it. I'm running this setup right now and noting
> > problems that come up. This is just experimental and of course doesn't
> > solve any other issues like handling of searches, but it provides a
> > pretty good way of further testing a load on demand setup.
>
> How's it working out so far? Any surprises?

It does a decent job fulfilling the basic on demand loading
requirement. I'm finding the lack of a complete timeline and taglist
to be kind of inconvenient. Also because of the flexibility in styling
TiddlyWikis, the initial tiddlers that need to be available can vary a
lot. I think skinny tiddlers / lazy loading will prove to be a more
useful approach in most cases. I've been kicking around some
ideas...I'll post an outline of that in a bit.

I'd like to work out a skinny tiddler solution with you guys, and then
focus on the plugin and search issues. I have a feeling that dealing
with skinny tiddlers is going to require dipping into TiddlyWiki's
core at a fairly basic level.

> It's really great to see you doing this work. I'm especially glad that
> you've posted up the code so people can have a poke themselves.
>
> The way you've created your plugins is cool to see and really shows
> off some of the flexibility I hoped, but wasn't sure, that I was
> building into the system.

The design of TiddlyWeb is really impressive. You've done a remarkable
job in making virtually every component extensible. Combined with the
extensibility of TiddlyWiki itself the possibilities are mind
boggling.

> You've made your plugins in a way that is
> almost entirely different from how I would do it, but in a perfectly
> reasonable and potentially better way.

I'm very curious how you might have made the plugins. What I ended up
with was not really what I expected at the outset.

[chris...@gmail.com]

On Jun 1, 7:07 am, Oveek <mov...@gmail.com> wrote:
> On May 23, 10:52 pm, "cd...@peermore.com" <chris.d...@gmail.com>
> wrote:
> >By the way, if you haven't already done it, it's probably best to make
> >your fork for tiddlyweb-plugins have tiddlyweb/tiddlyweb-plugins as
> >its upstream, not cdent/tididlyweb-plugins.
>
> Yea I later realized most of the commit activity is in tiddlyweb/
> tiddlyweb-plugins. Is cdent/tiddlyweb mainly for releases?

tiddlyweb/tiddlyweb is for development of the core

tiddlyweb/tiddlyweb-plugins is for development of plugins that are
somehow "blessed" into the mainstream. At the moment blessed basically
means I wrote them, or I said, "that's cool". I'm sure as tiddlyweb
matures, plugin handling will need to be a bit more formal, but I'd
rather pave the cow paths than whatever the opposite of that is.

The cdent stuff is basically an orphan.

With your plugins I can either pull them into tiddlyweb/tiddlyweb-
plugins if you like, or just point to them from http://tiddlyweb.peermore.com/
(from its plugins section).

> There's a potential problem when the filter is applied to recipes that
> have multiple copies of the condition tiddler. If I understand
> correctly, when there are multiple copies of a tiddler in the list of
> bags making a recipe, the recipe uses whichever one appears in the
> later bag. To give the right results the tiddlylinkfilter should use
> the same condition tiddler used in the recipe after it is
> 'uniquified', but as far as I can see, to do that requires knowing the
> bags used by the recipe (and the order they occur in).

I'm not sure I'm entirely following, but I'm short of coffee so far
today. If the condition tiddler is a part of the recipe which it is
then filtering, then once you get to the filtering stage you have the
correct condition tiddler (because the recipe has already been
processed) and each tiddler being processed has attributes 'bag' and
'recipe' which can be used to do some further inspection to find the
right bag in which to find the tiddlers it is linking to.

In control is a method called 'determine_tiddler_bag_from_recipe'
which is probably useful here: for each link you can ask that method
to tell you which bag to find that tiddler in.

That could get time consuming, so the other option is to generate the
list of tiddlers produced by the recipe the first time the filter is
called, and then compare against that with subsequent filterings.

I hope I'm making some sense here. Generally what I'm trying to say is
that the tiddlers probably have the information you need to make the
right thing happen, just dig into their attributes and see where you
end up.

> > How's it working out so far? Any surprises?
>
> It does a decent job fulfilling the basic on demand loading
> requirement. I'm finding the lack of a complete timeline and taglist
> to be kind of inconvenient. Also because of the flexibility in styling
> TiddlyWikis, the initial tiddlers that need to be available can vary a
> lot. I think skinny tiddlers / lazy loading will prove to be a more
> useful approach in most cases. I've been kicking around some
> ideas...I'll post an outline of that in a bit.

One option with timeline and taglist is to completely replace the
existing timeline and taglist ui elements with things that query the
server for the information. This is probably not ideal, but is an
option.

I agree that skinny stuff is probably a better option for people
working with TiddlyWeb content from TiddlyWiki.

> I'd like to work out a skinny tiddler solution with you guys, and then
> focus on the plugin and search issues. I have a feeling that dealing
> with skinny tiddlers is going to require dipping into TiddlyWiki's
> core at a fairly basic level.

I think you are probably right.

> > The way you've created your plugins is cool to see and really shows
> > off some of the flexibility I hoped, but wasn't sure, that I was
> > building into the system.
>
> The design of TiddlyWeb is really impressive. You've done a remarkable
> job in making virtually every component extensible. Combined with the
> extensibility of TiddlyWiki itself the possibilities are mind
> boggling.

Thanks. Python makes making things extensible quite easy. TiddlyWeb is
also my first major project after being a different project when
extensibility was way too hard, so I'm having a bit of a compensating
response.

> > You've made your plugins in a way that is
> > almost entirely different from how I would do it, but in a perfectly
> > reasonable and potentially better way.
>
> I'm very curious how you might have made the plugins. What I ended up
> with was not really what I expected at the outset.

Well the main difference is that I first noticed is you tend to be
class oriented. Not a bad thing as it lets you carry around a fair bit
of information where my usual approach (psuedo-functional) would not.

I often have that "not really what I expected at the outset" with
TiddlyWeb stuff.