Updated dependency handling in TiddlyWiki5

[Jeremy Ruston]

This is mainly for @cdent's benefit, but others may be interested too.

I've just changed the way that dependencies are tracked:

https://github.com/Jermolene/TiddlyWiki5/commit/95ab295fbb8a8a65b73eac3f309b09ba63c18564

Now, the WikiTextParseTree object's "dependencies" member is a hashmap
as follows:

{
link: {tiddlerTitle: 1, tiddlerTitle2: 2},
include: {tiddlerTitle3: 7, tiddlerTitle4: 5},
dependentAll: true,
}

The idea is to distinguishing different "relation types", which are
currently "link" and "include", and to keep a count of how many times
each tiddler is so referenced. A "link" reference is just a dependency
on whether the tiddler exists or not (and so would imply that the
target tiddler can be loaded skinny), while an "include" reference
means that the value of the target tiddler directly impacts the
rendering of the tiddler (and so would imply that the target tiddler
should be loaded fat).

The "dependentAll" flag, as before, indicates that the tiddler's
content depends on a scan of all the loaded tiddlers; it is triggered
by the use of the 'list' macro, or the "with" parameter with the
'tiddler' macro. The implication is that skinny versions of all
relevant tiddlers should be loaded.

I'm not happy with the irregular way that the "dependentAll" flag is
currently handled, and so there may well be further changes to the
interface. Let me know how the changes work for you,

Best wishes

Jeremy

[chris...@gmail.com]

On Tue, 7 Feb 2012, Jeremy Ruston wrote:

> This is mainly for @cdent's benefit, but others may be interested too.

Thanks, this makes sense.

I've moved my related work out of a fork into its own thing:

https://github.com/cdent/tw5ikifier

Doesn't currently work as I've not yet updated the code to reflect
the new dependency handling.

Will report on how they feel once I do.

--
Chris Dent http://burningchrome.com/
[...]

[chris...@gmail.com]

On Feb 9, 5:09 pm, chris.d...@gmail.com wrote:
>    https://github.com/cdent/tw5ikifier
>
> Doesn't currently work as I've not yet updated the code to reflect
> the new dependency handling.
>
> Will report on how they feel once I do.

I have it working. Initially it was a bit of a pain as I needed to
account non-existent keys. It _might_ (but I'm not sure) be useful to
make the returned object have default empty values.

In any case once I can that going and worked out how to effectively
deal with the usual asynch whooha things seemed pretty okay. Couple of
things noticed:

* For now include and link mean the same thing to me as there's no way
in TiddlyWeb to get a single tiddler without it's text, only with a
collection. Changing this has been discussed in the past, so it might
be time.

* When rendering a tiddler that uses the <<tiddler>> macro, whereas
tiddler links in the outer tiddler are properly managed for resolution
or missing, links in the inner tiddler are not. For example if both
the inner and outer tiddler's reference "TiddlySpace" (and TiddlySpace
exists in the context), then the outer tiddler will class the link "tw-
tiddlylink-resolves" but the inner will class it "tw-tiddlylink-
missing'. Is that something my code gets wrong, or something deeper?

Still no clear picture on performace. At the moment the vast majority
of time is spent satisfying the require statements, which is why my
code moves all of those outside the Store constructor.

[Jeremy Ruston]

> I have it working. Initially it was a bit of a pain as I needed to
> account non-existent keys. It _might_ (but I'm not sure) be useful to
> make the returned object have default empty values.

That makes sense. The reason they don't have empty values is that I
wanted the code to be able to extend gracefully to cover other types
of parsed relationship between tiddlers (not that I can think of any
besides "link" and "include"). The alternative would imply that the
parsing mechanism is aware of which relationship types are used by
macros, so that it can pre-create the fields of the "dependencies"
structure.

> In any case once I can that going and worked out how to effectively
> deal with the usual asynch whooha things seemed pretty okay. Couple of
> things noticed:
>
> * For now include and link mean the same thing to me as there's no way
> in TiddlyWeb to get a single tiddler without it's text, only with a
> collection. Changing this has been discussed in the past, so it might
> be time.

Right. Perhaps in this situation it would be convenient to be able to
retrieve the skinny JSON of a list of tiddlers enumerated by title?

> * When rendering a tiddler that uses the <<tiddler>> macro, whereas
> tiddler links in the outer tiddler are properly managed for resolution
> or missing, links in the inner tiddler are not. For example if both
> the inner and outer tiddler's reference "TiddlySpace" (and TiddlySpace
> exists in the context), then the outer tiddler will class the link "tw-
> tiddlylink-resolves" but the inner will class it "tw-tiddlylink-
> missing'. Is that something my code gets wrong, or something deeper?

Ouch. That sounds like a bug, I'll investigate.

> Still no clear picture on performace. At the moment the vast majority
> of time is spent satisfying the require statements, which is why my
> code moves all of those outside the Store constructor.

Yup, I've noticed that node.js stuff does end up having glacially slow
startup. A great nuisance when you're running lots of separate node
tasks in a shells script.

The interface to the dependency stuff is going to have to change a
little bit when I implement slices, sections and inter-space links.
I'll be parsing links into a structure something like:

{target: <title>, section: <section>, slice: <slice>, space: <space>}

It means that I'll no longer be able to use the link target as a
hashmap key, and probably will abandon the current approach of
counting references to each link destination.

Best wishes

Jeremy

>
> --
> You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to tiddlywikide...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/tiddlywikidev?hl=en.
>

[Ben Gillies]

On Feb 10, 2012 9:21 AM, "Jeremy Ruston" <jeremy...@gmail.com> wrote:
>
> > I have it working. Initially it was a bit of a pain as I needed to
> > account non-existent keys. It _might_ (but I'm not sure) be useful to
> > make the returned object have default empty values.
>
> That makes sense. The reason they don't have empty values is that I
> wanted the code to be able to extend gracefully to cover other types
> of parsed relationship between tiddlers (not that I can think of any
> besides "link" and "include").

Maybe it would make more sense to provide a getattr style helper function than to provide default values to everything.

[Jeremy Ruston]

>> > I have it working. Initially it was a bit of a pain as I needed to
>> > account non-existent keys. It _might_ (but I'm not sure) be useful to
>> > make the returned object have default empty values.
>>
>> That makes sense. The reason they don't have empty values is that I
>> wanted the code to be able to extend gracefully to cover other types
>> of parsed relationship between tiddlers (not that I can think of any
>> besides "link" and "include").
>
> Maybe it would make more sense to provide a getattr style helper function
> than to provide default values to everything.

You mean as a method on the parser or dependencies object? Yes, that
makes sense.

Best wishes

Jeremy

[Jeremy Ruston]

> * When rendering a tiddler that uses the <<tiddler>> macro, whereas
> tiddler links in the outer tiddler are properly managed for resolution
> or missing, links in the inner tiddler are not. For example if both
> the inner and outer tiddler's reference "TiddlySpace" (and TiddlySpace
> exists in the context), then the outer tiddler will class the link "tw-
> tiddlylink-resolves" but the inner will class it "tw-tiddlylink-
> missing'. Is that something my code gets wrong, or something deeper?

I think I have fixed this problem in this commit:

https://github.com/Jermolene/TiddlyWiki5/commit/7fdd8985ef5e898b92a46af727ad13e8edcfc52f

Can you give it a try and let me know?

Many thanks,

Jeremy

[Ben Gillies]

Yes, either that or more generally within TW5 somewhere itself. It
seems like a fairly common problem (CoffeeScript has a dedicated
operator for it for example)

[chris...@gmail.com]

So I narrowed it down a bit using two tiddlers as tests:

http://cdent.tiddlyspace.com/fiertest1
http://cdent.tiddlyspace.com/HelloThere

The second one transcludes SiteInfo as the first thing in the
tiddler. That "TiddlySpace" is claimed to be missing. Later ones are
not.

The first one transcludes later in the tiddler and the "TiddlySpace"
is not missing.

I can paste the output somewhere if that would be useful.

(This is using your latest master code)

[Jeremy Ruston]

> So I narrowed it down a bit using two tiddlers as tests:
>
>    http://cdent.tiddlyspace.com/fiertest1
>    http://cdent.tiddlyspace.com/HelloThere
>
> The second one transcludes SiteInfo as the first thing in the
> tiddler. That "TiddlySpace" is claimed to be missing. Later ones are
> not.
>
> The first one transcludes later in the tiddler and the "TiddlySpace"
> is not missing.
>
> I can paste the output somewhere if that would be useful.
>
> (This is using your latest master code)

Thanks. I can't duplicate the problem in my test rig, so I've been
looking at the tw5ikifier code. I'm probably being very stupid and
misreading the code, but I'm wondering if the problem isn't that it
doesn't seem to be processing the sub-dependencies of the tiddlers
that get loaded as primary dependencies. You don't need to do it for
"link" tiddlers, but I think you do need to do it for "include"
tiddlers.

Best wishes

Jeremy

[chris...@gmail.com]

On Fri, 10 Feb 2012, Jeremy Ruston wrote:

> Thanks. I can't duplicate the problem in my test rig, so I've been
> looking at the tw5ikifier code. I'm probably being very stupid and
> misreading the code, but I'm wondering if the problem isn't that it
> doesn't seem to be processing the sub-dependencies of the tiddlers
> that get loaded as primary dependencies. You don't need to do it for
> "link" tiddlers, but I think you do need to do it for "include"
> tiddlers.

Hmm. Doesn't the rendering happen after the store has been filled up,
and sub dependencies are looking at the same store as primary
dependencies?

In my examples, in any case, the sub-dependencies are tiddlers in
the same set as the primary dependencies, so presumably they shouldn't
need to be resolved again?

And finally: My expectation would be that the if I ask for the
dependencies for tiddler X, I will get _all_ dependencies for it.

[Jeremy Ruston]

It may be easier to tackle your last point first:

And finally: My expectation would be that the if I ask for the
dependencies for tiddler X, I will get _all_ dependencies for it.

The reason why you are only getting the immediate dependencies of the tiddlers you parse is that the further dependencies can't be computed until those immediate dependencies have been loaded and parsed. Your code is part of the process of progressively loading those tiddlers, reading out their dependencies, and recursing back to load them. The parser can't give you all the dependencies of a tiddler until it's examined all the other tiddlers that it references with the "include" dependency.

There will eventually be a method on the WikiStore object that aggregates all the dependencies of a tiddler, but it will be synchronous, and so won't be any use to you.

Best wishes

Jeremy

--

http://jermolene.com

http://tiddlywiki.com

http://osmosoft.com

[chris...@gmail.com]

On Fri, 10 Feb 2012, Jeremy Ruston wrote:

> The reason why you are only getting the immediate dependencies of the
> tiddlers you parse is that the further dependencies can't be computed until
> those immediate dependencies have been loaded and parsed. Your code is part
> of the process of progressively loading those tiddlers, reading out their
> dependencies, and recursing back to load them. The parser can't give you
> all the dependencies of a tiddler until it's examined all the other
> tiddlers that it references with the "include" dependency.

Ah, yeah, of course. I was having a mental conflation of parsing and
rendering etc.

I'll do some more poking about.

[Jeremy Ruston]

Hi Chris

I've slightly changed the format for representing dependencies. They
are now encapsulated in a class:

https://github.com/Jermolene/TiddlyWiki5/blob/master/js/Dependencies.js

As you can see, you can now enumerate the members of
parseTree.dependencies.tiddlers, which will have the value 'true' if
the dependency is on the fat version of the tiddler, or 'false' if it
is just on the skinny version.

I hope that makes things a bit more convenient,

Best wishes

Jeremy

[chris...@gmail.com]

On Fri, 17 Feb 2012, Jeremy Ruston wrote:

> Hi Chris
>
> I've slightly changed the format for representing dependencies. They
> are now encapsulated in a class:
>
> https://github.com/Jermolene/TiddlyWiki5/blob/master/js/Dependencies.js

That seems useful. Should make it easier to work with.

> As you can see, you can now enumerate the members of
> parseTree.dependencies.tiddlers, which will have the value 'true' if
> the dependency is on the fat version of the tiddler, or 'false' if it
> is just on the skinny version.

I think I'm seeing the opposite logic. The following is a console
log of the depenencies on my http://cdent.tiddlyspace.com/HelloThere
tiddler. For that SiteInfo ought to be a fat dependency and the
others skinny. Bug or misrepresentation of the logic?

{ tiddlers:
{ SiteInfo: undefined,
TiddlySpaceTips: true,
'http://tiddlyspace.com/profiles/cdent.atom': true,
Spaces: true },
dependentAll: undefined }

[Jeremy Ruston]

Hi Chris

Apologies, that was a stupid mistake on my part, fixed here:

https://github.com/Jermolene/TiddlyWiki5/commit/470b622bb15bd6e4e4076a019be831f9b07b7e77

I'll add some tests around this.

Best wishes

Jeremy