Re: Lazy loading. Any news?

[Poul]

Not sure if it matches your requirements, but giewiki certainly does have the ambition to scale much better content-wise, including support for lazy-loading tiddlers and server-side search. 

It builds on Google's App Engine, and I do not recommend using the SDK for storage of thousands of tiddlers, but if the advantages of a hosted solution outweighs the disadvantages for you, a hosted giewiki is definitely my recommendation.

:-) Poul

http://giewiki.appspot.com

Den onsdag den 17. oktober 2012 16.23.23 UTC+2 skrev Ruler11:

I haven't been watching at TW-development for a long time due to the lack of a mechanism of storing thousands of tiddlers.

So is there a plugin for lazy loading of tiddlers or connecting TW to a MySQL database or even Mediawiki?

[Jeremy Ruston]

I believe that there is also an implementation of lazy loading for TiddlySpace, but I don't have the details to hand. TW5 will have lazy loading built in as part of the core.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To view this discussion on the web visit https://groups.google.com/d/msg/tiddlywiki/-/6T8JpdsgMTEJ.

To post to this group, send email to tiddl...@googlegroups.com.
To unsubscribe from this group, send email to tiddlywiki+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/tiddlywiki?hl=en.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[chris...@gmail.com]

On Thu, 18 Oct 2012, Jeremy Ruston wrote:

> I believe that there is also an implementation of lazy loading for
> TiddlySpace, but I don't have the details to hand. TW5 will have lazy
> loading built in as part of the core.

There are a few different implementations but none was ever blessed as
the official solution.

Part of it was that things just keep getting faster on TiddlySpace, so
a sizable number of tiddlers, under the right conditions, can load
quite quickly, if they are coming from cache.

Another part is that there was insufficient feedback and development.

This is because the people who expresssed interest in having large
collections of tiddlers were in three groups:

* Didn't mind the slower load times: once stuff is loaded it is there
and future operations are quite fast. Plugins exist that will load
up new content as it "arrives" in the space.

* Wanted lots of tiddlers, but didn't care about TiddlyWiki, moved to
using the HTML representation.

* Liked the concept of lazy loading, but didn't like the way in which
timeline, search and tags were affected by it.

Identifying a larger group of participants from this latter set would
be useful as it could provide some robust feedback and other
contributions.

Without those contributions it is unlikely anything will happen.

In any case:

My space, which has 1099 tiddlers, takes about 3 seconds to load in
standard wiki mode (no laziness).

The /_tiddlywiki app (available from the @apps space[1]) is a special
customization of tiddlywiki that loads an empty tiddlywiki html +
javascript and then bootstraps itself by loading content after initial
start. To go from starting it to working TiddlyWiki takes longer than
3 seconds.

By the way, in total there about 825,000 revisions of 355,000 tiddlers
on http://tiddlyspace.com/

Of course all this stuff is on serversides. If what you're after is
standalone, then I think TiddlyWiki5 is probably going to be the
tool needed. For seriously large instances it will be important for
whatever the tool is to do what amounts to memory management on
tiddlers: disposing the least recently used.

[1] http://apps.tiddlyspace.com/
--
Chris Dent http://burningchrome.com/
[...]

[Ruler11]

On Thursday, October 18, 2012 11:55:42 AM UTC+4, Jeremy Ruston wrote:

I believe that there is also an implementation of lazy loading for TiddlySpace, but I don't have the details to hand. TW5 will have lazy loading built in as part of the core.

So any news on lazy loading now? 

[Jeremy Ruston]

I believe that there is also an implementation of lazy loading for TiddlySpace, but I don't have the details to hand. TW5 will have lazy loading built in as part of the core.

So any news on lazy loading now? 

The lazy loading mechanism is now implemented in TW5. It's used currently only in the TiddlyWeb edition; in that edition all tiddlers are loaded lazily.

Cheers

Jeremy

 

Best wishes

Jeremy

On Thu, Oct 18, 2012 at 8:32 AM, Poul <poul.st...@gmail.com> wrote:

Not sure if it matches your requirements, but giewiki certainly does have the ambition to scale much better content-wise, including support for lazy-loading tiddlers and server-side search. 

It builds on Google's App Engine, and I do not recommend using the SDK for storage of thousands of tiddlers, but if the advantages of a hosted solution outweighs the disadvantages for you, a hosted giewiki is definitely my recommendation.

:-) Poul

http://giewiki.appspot.com

Den onsdag den 17. oktober 2012 16.23.23 UTC+2 skrev Ruler11:

I haven't been watching at TW-development for a long time due to the lack of a mechanism of storing thousands of tiddlers.

So is there a plugin for lazy loading of tiddlers or connecting TW to a MySQL database or even Mediawiki?

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.

To view this discussion on the web visit https://groups.google.com/d/msg/tiddlywiki/-/6T8JpdsgMTEJ.

To post to this group, send email to tiddl...@googlegroups.com.
To unsubscribe from this group, send email to tiddlywiki+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/tiddlywiki?hl=en.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[Tobias Beer]

Unless you have thousands of alike entries, one approach might be to actually have a number of either TiddlyWikis, TiddlySpaces, Bags, etc... into which you cluster tiddlers depending on topic / context / etc — which shifts the problem / focus to...

• establishing a consistent and manageable way for representation throughout the entirety of your content
  
• making this collection of Tiddly* instances explorable via search
• or even manageable via some form of remote access / editing

Cheers, Tobias.

[chris...@gmail.com]

On Sat, 27 Apr 2013, Ruler11 wrote:
> On Thursday, October 18, 2012 11:55:42 AM UTC+4, Jeremy Ruston wrote:
>>
>> I believe that there is also an implementation of lazy loading for
>> TiddlySpace, but I don't have the details to hand. TW5 will have lazy
>> loading built in as part of the core.
>
> So any news on lazy loading now?

The most recent lazy loading version of tiddlywiki for tiddlyspace
remains:

http://apps.tiddlyspace.com/_tiddlywiki

What it does is load up an empty framing for tiddlywiki and then loads
in the necessary parts from the server, a tiddler at a time.

There hasn't been much work on it because the tiddlyspace server itself
has increased in speed and except for really big spaces, or really
slow connections the win with using lazy loading is not that
significant.

Or perhaps a better way to put it is that the people who are really
concerned about speed have simply removed TiddlyWiki from the equation
and are working with tiddlers directly.

[Yakov]

Hi Ruler11,

now sure if this helps, but you can try SharedTiddlersPlugin [1]. It's not exactly what you want: it can be used for lazy loading (including), but doesn't allow to edit external tiddlers inside including TW (although, import:4 option can do the job to some extent).

[1] http://yakovl.bplaced.net/TW/STP/STP.html

суббота, 27 апреля 2013 г., 21:02:37 UTC+4 пользователь Ruler11 написал:

[Jeremy Ruston]

Lazy loading without using a server with TW5 is planned but will require TiddlyFox, or another means of allowing the browser to save arbitrary files.

Best wishes

Jeremy

On Fri, May 10, 2013 at 8:30 AM, Ruler11 <ruler...@gmail.com> wrote:

On Thursday, May 2, 2013 12:43:21 AM UTC+4, Yakov wrote:

Hi Ruler11,

now sure if this helps, but you can try SharedTiddlersPlugin [1]. It's not exactly what you want: it can be used for lazy loading (including), but doesn't allow to edit external tiddlers inside including TW (although, import:4 option can do the job to some extent).

[1] http://yakovl.bplaced.net/TW/STP/STP.html

Thanks. I'll have a look..

What I want is (preferably) local tiddlywiki without the need for a server. Not using a .jar would be an ideal solution but I understand...

I have thousands of entries to import into TW and it obviously freezes so I can't use TW for managing my content.

I really miss TW because SplitTiddler was a very nice tool for me.

[G.J.Robert]

Sorry I'm giving feedback way too late. I am one of those who has really big spaces (I have a private notebook of learning laws, about 15~17MB now and still counting). Working lazy loading mechanism would really help me a lot especially when I do layout/style editions and need to reload the page frequently, or when my Firefox blocking drag 'n' drops in textareas I need to restart even the whold browser
again and again.

I depend on the TW environment too much so working directly on tiddlers in the static HTML level is not adequate for me. I have abandoned working online on TiddlySpace a couple of month ago and has been working on offline TSpace files and do importing back every couple week.

But any news to the lazy loading for TSpace/TW classic would be appreciated, since I still love the feeling to retrieve and save every tiddler instantly on the web. I'm also depending too much on currently so many plugins of TW classic, so I'm not sure if switching to TW5 is a suitable option for me.

Again many thanks to so many creators/contributors of TW/TSpace/plugins like Jeremy, Eric, Chris, Tobias, Simon Baird, Saq (just to name a few) etc. etc.

Chris Dent於 2013年4月29日星期一UTC+8下午9時45分34秒寫道：

[G.J.Robert]

Adding a couple of remarks: I do sense the increase of TSpace server in last months (also my ISP has also improved their cables...), and today my 15~17MB space loads in just 20s on my home laptop, it's really a great improvement! I guess I can reconsider to move back to online editing. I'm not sure how fast can it load in less stable and firm environment like mobile networks (sadly we don't have LTE in Taiwan yet, and network speed here is never comparable to those in the States and Europe)

I have just also tried "_tiddlywiki" of @apps on one of my smaller spaces. It loads fine, only that it seems to break some functions or some plugins, so my space looks a bit different in the top bar (it seems that the top bar from other included spaces gets rendered first instead of my customizations), and I've got "ReferenceError: hasClass is not defined" every time a tiddler is opened.

I also found that @apps seems to have altered my sidebar on that space when viewing the normal URL of the space (without "_tiddlywiki")... I should have used my sandbox space to try it...

G.J.Robert於 2013年5月20日星期一UTC+8上午6時48分57秒寫道：

[G.J.Robert]

Oh, I'm sorry, it's not @apps that was changing my sidebar. That's from @mobile space.

G.J.Robert於 2013年5月20日星期一UTC+8上午7時14分30秒寫道：

[chris...@gmail.com]

On Sun, 19 May 2013, G.J.Robert wrote:

> Adding a couple of remarks: I do sense the increase of TSpace server in
> last months (also my ISP has also improved their cables...), and today my
> 15~17MB space loads in just 20s on my home laptop, it's really a great
> improvement! I guess I can reconsider to move back to online editing. I'm
> not sure how fast can it load in less stable and firm environment like
> mobile networks (sadly we don't have LTE in Taiwan yet, and network speed
> here is never comparable to those in the States and Europe)

We've recently discovered the extent to which reaching half way
round the world has an impact as well.

There's a change coming[1] in TiddlyWeb which may speed things up a
bit more (with standard TiddlyWiki) but is unlikely to make a huge
difference in a space your size: The sheer number of bytes that need
to be transmitted in a row is the controlling factor (thus the
desire for some kind of lazy solution).

> I have just also tried "_tiddlywiki" of @apps on one of my smaller spaces.
> It loads fine, only that it seems to break some functions or some plugins,
> so my space looks a bit different in the top bar (it seems that the top bar
> from other included spaces gets rendered first instead of my
> customizations), and I've got "ReferenceError: hasClass is not defined"
> every time a tiddler is opened.

Yes, there's definitely some work that needs to be done to make
_tiddlywiki work better. It needs more brains. If there were
unlimited time and energy I'd really love to get it correct and make
it the default way to view a tiddlywiki produced by TiddlySpace,
when online.

If you have a chance could you describe how you imagine the ideal
lazy system for tiddlyspace + tiddlywiki would work?

Thanks.

[1]
https://github.com/tiddlyweb/tiddlyweb/commit/de43319c459e5ea873f1129352a47f24c75c92f1

[G.J.Robert]

Chris Dent於 2013年5月21日星期二UTC+8上午3時34分33秒寫道：

Hi Chris,

Thanks for your kind listening and responding. I'm not a coder at all but I'll try my best on this.

Let's recall when TiddlyWiki just get so well noticed in it's 1.x age. There was an adaptation called "LesserWiki" if some people remember it. (discontinued long ago and sadly I can't even find it's official site now) It seemed to use AJAX methods behind the curtain so every tiddler was retrieved when a link is clicked (a throbber indicating the loading is in progress). Ever since seeing it, I've always hoped that TiddlyWiki itself has this kind of serverside backend. I imagined that on page loading only necessary frame codes and plugins are loaded and rendered, with of course the DefaultTiddlers and tiddler transcluded on Sidebar, MainMenu, Topbar etc. are loaded at the first place. The rest of all the tiddlers are treated like a backend database and are only retrieved when being clicked/searched/inquired for.

Of course as I further used and got knowledge about a little how TW works, I realized that making modern TW (even the classic edition) load lazily with so many transclusions, plugins, layout and environment settings is definitely not easy. For example, how will ForEachTiddlerPlugin behave when most of the tiddlers are not loaded at all? Will it throw complete and desirable output when the macro is executed? Will it take too long as loading the whole TiddlyWiki? And what I'm thinking may go way too far from the original design of TiddlyWiki as a single HTML file with all tiddler data as loaded but hidden divs.

Wait a moment, am I talking about "loading on demand," not "lazy loading"? Oh sweet Jesus. Pardon me if my concepts are so confused.

My biggest wish is the AJAX "on demand" way, but at least, I like enough the way "@apps _tiddlywiki" loads the spaces. Although it is not "on demand loading", but it does reduce the initial time to render the page frame and does load all the data eventually after very acceptable delay. If I can find a direction to pay suggestions, how about distinguishing more clearly between those tiddlers definitely needed initially (all the plugins, stylesheets and tiddlers making up the whole layout) and those irrelevant to the structure (only playing as data), and making sure that the necessary ones get loaded first, so that the initial page is 100% usable and correctly operational? Pardon me if this is not the problem now.

And also in my guess, whether we are doing "load on demand" or "lazy loading", the essential set of tiddlers consisting of the page may be the same, isn't it? Is the difference only in whether the page keeps loading all the other tiddlers until finished, or the page enters the standby mode awaiting user actions to retrieve the other tiddlers?

(Btw, I have also just tested "@apps _tiddlywiki" lazy loading on my big big 15MB space. Besides of the incomplete structure got rendered, this time the biggest problem is that the page data never finished loading. The percentage is always at 81%. Does the server timed out because of the size of my space?)

[chris...@gmail.com]

On Wed, 22 May 2013, G.J.Robert wrote:

> Let's recall when TiddlyWiki just get so well noticed in it's 1.x age.
> There was an adaptation called "LesserWiki" if some people remember it.
> (discontinued long ago and sadly I can't even find it's official site now)
> It seemed to use AJAX methods behind the curtain so every tiddler was
> retrieved when a link is clicked (a throbber indicating the loading is in
> progress). Ever since seeing it, I've always hoped that TiddlyWiki itself
> has this kind of serverside backend. I imagined that on page loading only
> necessary frame codes and plugins are loaded and rendered, with of course
> the DefaultTiddlers and tiddler transcluded on Sidebar, MainMenu, Topbar
> etc. are loaded at the first place. The rest of all the tiddlers are
> treated like a backend database and are only retrieved when being
> clicked/searched/inquired for.

This sort of thing is technically possible, and is how I always
hoped/imagined dynamic or lazy loading would work but it has all the
problems you imply here

> Of course as I further used and got knowledge about a little how TW works,
> I realized that making modern TW (even the classic edition) load lazily
> with so many transclusions, plugins, layout and environment settings is
> definitely not easy. For example, how will ForEachTiddlerPlugin behave when
> most of the tiddlers are not loaded at all? Will it throw complete and
> desirable output when the macro is executed? Will it take too long as
> loading the whole TiddlyWiki? And what I'm thinking may go way too far from
> the original design of TiddlyWiki as a single HTML file with all tiddler
> data as loaded but hidden divs.

as well as issues with search, timeline (and other info tabs).

As a result things rather stall out. The basic situation is that it is
possible to do a dynamic (and yes that is a better term) loading
TiddlyWiki but only if you want to throw away a lot of the features
that make TiddlyWiki special.

Therefore it becomes necessary to explore the idea of loading an
initial basic TiddlyWiki and then loading in the rest of the content
as individual tiddlers, and then starting the app.

> My biggest wish is the AJAX "on demand" way, but at least, I like enough
> the way "@apps _tiddlywiki" loads the spaces. Although it is not "on demand
> loading", but it does reduce the initial time to render the page frame and
> does load all the data eventually after very acceptable delay. If I can
> find a direction to pay suggestions, how about distinguishing more clearly
> between those tiddlers definitely needed initially (all the plugins,
> stylesheets and tiddlers making up the whole layout) and those irrelevant
> to the structure (only playing as data), and making sure that the necessary
> ones get loaded first, so that the initial page is 100% usable and
> correctly operational? Pardon me if this is not the problem now.

I think it doesn't load as effectively as it should. There's
definitely room for improvement. Looking at the code it seems to not
be taking as advantage of potential parallelisms in the browser as it
could.

If that were fixed it is likely the lazy loading might be relatively
effective, assuming we define lazy as:

1 load the app framing
2 load the content in parallel
3 start the app

That's different from what's currently being done, which is ordered as
1, 3, 2 (with bad parallelism).