dependencies in tw5/cook.js WikiTextRules.js

[chris...@gmail.com]

Jeremy,

As you know from
http://groups.google.com/group/tiddlywiki/browse_frm/thread/b272acab87b70f51/4048693aabf15ca6#4048693aabf15ca6

I've been playing with the new parsing and rendering code you've
been working on.

One thing I've noticed that might be improved or changes is that
currently dependencies are managed as a simple list of link targets.
This means that when you query the dependencies of a single tiddler
you might get output like this:

[ 'TiddlySpace',
'TiddlySpaceSockets',
'https://github.com/cdent/tapas',
'http://tiddlyspace.com/bags/tapas_extra/tiddlers/tiddlersocket.js',
'https://github.com/cdent/tapas/issues',
'@tapas',
'TiddlySpaceSockets#d75673c75548023b8b8acf782a002b85' ]

(these are the dependencies from my http://cdent.tiddlyspace.com/tapas
tiddler)

We've got basically two types of links here: a wikilink or an
external link. I haven't inspected WikiTextRules in detail but I'm
guess there could be more and there might be others that result from
plugins, etc.

In my experiments I now need to branch on whether the link is a Wiki
link (free or camel) or something else to choose whether to pull an
additional tiddler into the store. Is this what you're
intending/expected, or is this an interim state of affairs?

The gravy train version of dependencies, for my use case, would be
something like this: A dictionary with keys representing types of
links and values that are the links themselves. The types would be
things like:

1 link to tiddler no text required
2 link to tiddler requiring text
3 url

I recognize this may be a bit outside the immediate remit, but if
you're still fleshing it out, there are some ideas.

For the time being I can still do what I need to do with stuff as
is.

--
Chris Dent http://burningchrome.com/
[...]

[Jeremy Ruston]

> One thing I've noticed that might be improved or changes is that
> currently dependencies are managed as a simple list of link targets.
> This means that when you query the dependencies of a single tiddler
> you might get output like this:
>
> [ 'TiddlySpace',
>  'TiddlySpaceSockets',
>  'https://github.com/cdent/tapas',
>  'http://tiddlyspace.com/bags/tapas_extra/tiddlers/tiddlersocket.js',
>  'https://github.com/cdent/tapas/issues',
>  '@tapas',
>  'TiddlySpaceSockets#d75673c75548023b8b8acf782a002b85' ]
>
> (these are the dependencies from my http://cdent.tiddlyspace.com/tapas
> tiddler)
>
> We've got basically two types of links here: a wikilink or an
> external link. I haven't inspected WikiTextRules in detail but I'm
> guess there could be more and there might be others that result from
> plugins, etc.

The behaviour you're seeing is intentional but provisional (as in, I'm
open to suggestions to change it).

At the moment, the wikification engine doesn't try to determine
whether a link is internal or external right until it is rendered,
when the function WikiStore.adjustClassesForLink assigns classes to
the link according to whether the target exists as the title of a
tiddler. It is done this way so that if a link needs to be updated
because a target tiddler has changed its status, it becomes possible
to just rerender the link, rather than having to reparse the tiddler
too.

Back in the parsing processing, when the dependencies are computed,
I'm trying to avoid accessing any other tiddlers in the store because
then we'd have to reparse when the store changes. Thus the only
mechanism available for determining whether a link is external would
be for the parsing stage to apply regexps to the target string. We
found with TiddlyWiki that it's very hard to come up with a good
general purpose regexp for detecting external URIs and so I felt it
was better to let the outside world apply it's own logic to determine
external links.

As I may have mentioned, I am thinking that the representation of
dependencies needs to be updated to distinguish between tiddlers that
are linked to from those that are transcluded.

> In my experiments I now need to branch on whether the link is a Wiki
> link (free or camel) or something else to choose whether to pull an
> additional tiddler into the store. Is this what you're
> intending/expected, or is this an interim state of affairs?

It does seem counter-intuitive, but as I say, it does give you a great
deal of control over the policies governing how a URI is parsed.

> The gravy train version of dependencies, for my use case, would be
> something like this: A dictionary with keys representing types of
> links and values that are the links themselves. The types would be
> things like:
>
> 1 link to tiddler no text required
> 2 link to tiddler requiring text
> 3 url
>
> I recognize this may be a bit outside the immediate remit, but if
> you're still fleshing it out, there are some ideas.

What's the distinction between types 1 and 2? Links in TW5 are coded
as macros, and macros have been extended so that they can have
children. Thus, the link macro can contain arbitrary content, not just
text.

You may find it useful to dig around in the internal structures by
clicking on the "parse tree" and "render functions" buttons at the
bottom of each tiddler here:

http://tiddlywiki.com/tiddlywiki5/

I'm pleased it's coming together,

Best wishes

Jeremy

> For the time being I can still do what I need to do with stuff as
> is.
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>

[chris...@gmail.com]

On Tue, 31 Jan 2012, Jeremy Ruston wrote:

> At the moment, the wikification engine doesn't try to determine
> whether a link is internal or external right until it is rendered,
> when the function WikiStore.adjustClassesForLink assigns classes to
> the link according to whether the target exists as the title of a
> tiddler. It is done this way so that if a link needs to be updated
> because a target tiddler has changed its status, it becomes possible
> to just rerender the link, rather than having to reparse the tiddler
> too.

That makes sense.

>> The gravy train version of dependencies, for my use case, would be
>> something like this: A dictionary with keys representing types of
>> links and values that are the links themselves. The types would be
>> things like:
>>
>> 1 link to tiddler no text required
>> 2 link to tiddler requiring text
>> 3 url
>>
>> I recognize this may be a bit outside the immediate remit, but if
>> you're still fleshing it out, there are some ideas.
>
> What's the distinction between types 1 and 2? Links in TW5 are coded
> as macros, and macros have been extended so that they can have
> children. Thus, the link macro can contain arbitrary content, not just
> text.

The distinction is a way of saying: do I just need the metadata of
the tiddler to carry on with the render or do I also need its text
field.

For example if the tiddler being rendered does a <<tiddler>> macro,
then it needs the text of that tiddler.

However if there is just some kind of <<list>> macro, then you
wouldn't need text, you'd just need titles and a bit of meta.

It's essentially a premature optimization gesture on my part: Can I
avoid getting fat tiddlers?

As it turns out the code I have rolling now get tiddlers
individually (not en-masse) so it may not be an issue.

That commit coming in a moment on the wikify branch of my fork of
cook.js.

[Jeremy Ruston]

> The distinction is a way of saying: do I just need the metadata of
> the tiddler to carry on with the render or do I also need its text
> field.
>
> For example if the tiddler being rendered does a <<tiddler>> macro,
> then it needs the text of that tiddler.
>
> However if there is just some kind of <<list>> macro, then you
> wouldn't need text, you'd just need titles and a bit of meta.
>
> It's essentially a premature optimization gesture on my part: Can I
> avoid getting fat tiddlers?
>
> As it turns out the code I have rolling now get tiddlers
> individually (not en-masse) so it may not be an issue.
>
> That commit coming in a moment on the wikify branch of my fork of
> cook.js.

Aha, that makes sense. I think you're talking about the same
distinction as me with my comment about distinguishing "tiddlers that
are linked to from those that are transcluded. The underlying issue
can indeed be seen as whether one needs the skinny or thick version of
the tiddler.

So, I'll explore making the dependencies be a hashmap of
{title:,type:}, where type would be "skinny" or "fat" (or something
less confusing).

There's also some housekeeping that I need to attend to: one is moving
the code over to my TiddlyWiki5 repo, because it's poorly named where
it is. When and if you get closer to production then I think I'll also
need to introduce separate dev/stable branches.

Best wishes

Jeremy.

[chris...@gmail.com]

On Jan 31, 4:29 pm, Jeremy Ruston <jeremy.rus...@gmail.com> wrote:
> Aha, that makes sense. I think you're talking about the same
> distinction as me with my comment about distinguishing "tiddlers that
> are linked to from those that are transcluded. The underlying issue
> can indeed be seen as whether one needs the skinny or thick version of
> the tiddler.
>
> So, I'll explore making the dependencies be a hashmap of
> {title:,type:}, where type would be "skinny" or "fat" (or something
> less confusing).

My latest code[1] is now relatively close to being on par with
twikifier. It listens on a socket for info about what to render
(tiddler title and context), resolves dependencies, gets them,
renders, returns the output.

Things I've noticed:

* As expected, having the type info discussed above would be useful,
most especially in the case where null is returned. If that null can
be qualified as not needing text, then a single request (for skinny
json of the context collection) can be made rather than either a fat
request or a many single requests. The current code does the latter
and when the number of tiddlers in the context gets high, up over a
few hundred, it is noticed.

* This may have changed already, but it appears that when generating
links to tiddlers the href is not getting an encodeURIComponent
applied, thus [[foo bar]] becomes href="foo bar" where it presumably
should be "foo%20bar".

The next things to do for me are:

* get caching of remote tiddlers and/or stores back in
* translate the tiddlyspace link syntax and add it to macros

[1] https://github.com/cdent/cook.js/tree/wikify

[Jeremy Ruston]

> My latest code[1] is now relatively close to being on par with
> twikifier. It listens on a socket for info about what to render
> (tiddler title and context), resolves dependencies, gets them,
> renders, returns the output.

That's great. Do you have any impression of the performance compared
to the old code?

> Things I've noticed:
>
> * As expected, having the type info discussed above would be useful,
> most especially in the case where null is returned. If that null can
> be qualified as not needing text, then a single request (for skinny
> json of the context collection) can be made rather than either a fat
> request or a many single requests. The current code does the latter
> and when the number of tiddlers in the context gets high, up over a
> few hundred, it is noticed.

Yup, I'll be fixing that up in the next day or two.

> * This may have changed already, but it appears that when generating
> links to tiddlers the href is not getting an encodeURIComponent
> applied, thus [[foo bar]] becomes href="foo bar" where it presumably
> should be "foo%20bar".

I'm mid horrendous refactoring, but have added a test for this.

> The next things to do for me are:
>
> * get caching of remote tiddlers and/or stores back in
> * translate the tiddlyspace link syntax and add it to macros

I'd add that there are a lot of standard TiddlyWiki macros that still
need to be implemented. I'm currently doing a lot of rework around the
interactive aspects of macros (selective refresh and macro event
dispatch), and so there is a small chance that the macro API will
change over the next few days.

I'm also in the process of transferring the code to the
Jermolene/TiddlyWiki5 repository.

Best wishes

Jeremy

> [1] https://github.com/cdent/cook.js/tree/wikify

[Jeremy Ruston]

>> * This may have changed already, but it appears that when generating
>> links to tiddlers the href is not getting an encodeURIComponent
>> applied, thus [[foo bar]] becomes href="foo bar" where it presumably
>> should be "foo%20bar".
>
> I'm mid horrendous refactoring, but have added a test for this.

I've realised that this is slightly more complicated than it first
appears. I can't just encodeURIComponent every href, because then
absolute URIs like "http://google.com/" would get converted. I guess
the conversion isn't part of the translation to HTML so much as
processing applied to a tiddler title to turn it into a URI.

So, I think that the processing applied to the href needs to depend on
the link. This could be done in the same way that I'm whacking in the
appropriate classes on the links. So, the encodeURIComponent would be
applied to tiddler links, with external links unmolested. Annoyingly,
it piles more consequences on the logic I use to distinguish internal
and external links.

Best wishes

Jeremy