cookbook for porting tw2.x formatter plugin to tw5?

[chris...@gmail.com]

Is there yet a cookbook or rules of thumb for porting a plugin that
adds to the formatter of tw2 to one that adds rules to the tw5
wikitextrules?

I've managed to get my tw5ikifier[1] code (mostly) correctly handing
dependencies so I'd now like to add TiddlySpaceLinkPlugin[2] into the
mix.

It looks relatively straightforward, but before I tuck in I thought
I'd ask.

Thanks.

[1] https://github.com/cdent/tw5ikifier
[2]
https://github.com/TiddlySpace/tiddlyspace/blob/master/src/plugins/TiddlySpaceLinkPlugin.js

--
Chris Dent http://burningchrome.com/
[...]

[Jeremy Ruston]

Hi Chris

> Is there yet a cookbook or rules of thumb for porting a plugin that
> adds to the formatter of tw2 to one that adds rules to the tw5
> wikitextrules?

No, not yet. I'm still expecting to do a fairly substantial
refactoring of the wikifier to make it properly re-entrant, which will
have an impact on how formatters are coded. Worse, I have yet to
properly document the new renderer architecture.

The easiest way to cope with the uncertainty might be to maintain the
new formatters as part of the core TW5 code base, and then I can
refactor them easily alongside the others. I plan to re-use the @space
syntax in standalone tiddlywiki, so I think it would make sense.

I think there are two jobs:

1) make the <<link>> macro understand how to make inter-space links
2) add/modify a new formatter to detect the @space link syntax and
emit the required <<link>> macro

> I've managed to get my tw5ikifier[1] code (mostly) correctly handing
> dependencies so I'd now like to add TiddlySpaceLinkPlugin[2] into the
> mix.

Great stuff, let me know if I can do anything to help.

Cheers

Jeremy

> It looks relatively straightforward, but before I tuck in I thought
> I'd ask.
>
> Thanks.
>
> [1] https://github.com/cdent/tw5ikifier
> [2]
> https://github.com/TiddlySpace/tiddlyspace/blob/master/src/plugins/TiddlySpaceLinkPlugin.js
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>

--
Jeremy Ruston
mailto:jeremy...@gmail.com
http://www.tiddlywiki.com

[Chris Dent]

On Thursday, March 8, 2012 12:36:29 PM UTC, Jeremy Ruston wrote:

1) make the <<link>> macro understand how to make inter-space links

2) add/modify a new formatter to detect the @space link syntax and
emit the required <<link>> macro

So to explore the issues I made these changes:

   https://gist.github.com/2149828

which is basically a straight port of the rules into WikiTextRules .js, replacing the link generation with creation of a link macro.

The main things this points out are:

* If/when the link macro understands inter-space linking, there needs to be some global-ish way to tell it the domain or domains that are being worked with. Right now I just hard code it. Obviously not great.

* The classic version of the plugin puts some metadata attributes on the generated links, to do things like name the tiddler being targeted (if indeed it is a tiddler). I don't think this is strictly necessary for tw5ikifier, but the idea may have merit for making the link macro more general: parameters that let you say "here, add this stuff".

The tw classic-based twikifier seems to be getting munged up about once a day on tiddlyspace.com lately. Not terminally so, just a few minutes of not working then it recovers. It would be great if I could replace it soon with the much more inspectable tw5-based stuff.

Thanks.

[Tobias Beer]

Hi Jeremy,

>> I plan to re-use the @space syntax in standalone tiddlywiki,
>> so I think it would make sense.

Do you mean in the sense where the link points to a space in (a)
TiddlySpace (environment)?!?

Tobias.

[Jeremy Ruston]

I'm interested in generalising the link notation so that it can be
used as a general purpose mechanism to reference tiddlers from a
different source than the current document.

One simple way of doing it would be for a reference like 'alpha@gamma'
to be interpreted as a reference to the tiddler called 'alpha' coming
from the space defined by the tiddler called 'gamma'. Then the content
of the tiddler 'gamma' would define the origin space, perhaps with a
type field and a base URL. The type field would allow for tiddlers to
be retrieved from instances of TiddlyWeb/TiddlySpace, but could also
be used to access other types of data.

For instance, with the appropriate definitions, we might have:

- 02747527426@twitter to identify a particular tweet
- Lettuces@wikipedia to identify a Wikipedia article
- Agenda@work to identify a tiddler in a separate TiddlyWiki document

I've also been researching oembed, a protocol designed to simplify
embedding content from one website in another, but I note that it is
actually a pretty good implementation of what we'd need for a general
purpose import mechanism.

Best wishes

Jeremy

>
> Tobias.

[Jeremy Ruston]

I'm away from a proper computer until next week, but this is great stuff, and I'll look forward to picking it up on my return,

Best wishes

Jeremy

--

Jeremy Ruston

jeremy...@gmail.com

[Chris Dent]

On Thursday, March 22, 2012 5:30:44 PM UTC, Jeremy Ruston wrote:

I'm away from a proper computer until next week, but this is great stuff, and I'll look forward to picking it up on my return,

Any thoughts or headway? 

[Jeremy Ruston]

Thanks Chris.

I'm tending towards the idea of being able to register a link
generator function with the store object. The function would be passed
the parameters of the link macro and return a Renderer.ElementNode
representing the generated <a> tag. It might be less brittle if the
function were passed a default <a> element which it could modify,
rather than generating one from scratch. Does that sound reasonable?

Best wishes

Jeremy

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/tiddlywikidev/-/_cWUg749cgMJ.

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

> I'm tending towards the idea of being able to register a link
> generator function with the store object. The function would be passed
> the parameters of the link macro and return a Renderer.ElementNode
> representing the generated <a> tag. It might be less brittle if the
> function were passed a default <a> element which it could modify,
> rather than generating one from scratch. Does that sound reasonable?

Are you thinking that there would be a mapping of link types to
various functions (which return the ElementNode) or something within a
single function that handles different types of links?

The former seems in keeping with the usual patterns.

So a "formatter" would result in a link macro of type X. Macro
processing would say "oh yeah, that's X, funcX deals with that".

[Jeremy Ruston]

> Are you thinking that there would be a mapping of link types to
> various functions (which return the ElementNode) or something within a
> single function that handles different types of links?
>
> The former seems in keeping with the usual patterns.
>
> So a "formatter" would result in a link macro of type X. Macro
> processing would say "oh yeah, that's X, funcX deals with that".

Can you clarify what sort of link types you'd expect to see?

Links in TW5 are parsed as a simple link target string. The link
target may be a URI or a tiddler reference, and in the latter case
we're talking about adding an optional space component to the link.
Currently, heuristics are applied at macro execution time to determine
if the link target is an external link or a tiddler link.

I don't think it makes sense to have separate callbacks for massaging
external links, tiddler links, intraspace vs. interspace links because
one of the things we want to be able to override is those heuristics
that are used to classify and render links.

Therefore, I see this mechanism more as a way for the link macro to
delegate some of its work to an external callback, and would be
inclined structure it in those terms, as a single callback.

Cheers

Jeremy

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

>> Are you thinking that there would be a mapping of link types to
>> various functions (which return the ElementNode) or something within a
>> single function that handles different types of links?
>>
>> The former seems in keeping with the usual patterns.
>>
>> So a "formatter" would result in a link macro of type X. Macro
>> processing would say "oh yeah, that's X, funcX deals with that".
>
> Can you clarify what sort of link types you'd expect to see?

Well the original problem I mentioned was that we need a way to
resolve space names (or that kind of syntax) which in the most basic
case means being able to say "prepend the space name onto the string
'.tiddlyspace.com' to make the hostname portion of the link".

In my gist[1] that's currently a static string, but what we want is to
be able to declare, somewhere, how space names are resolved,
preferably in a programatic way.

My understanding of what you were suggesting was that link macro,
knowing that it is of space type, would look for a function to perform
the resolution, thus being able to create a correct target.

The main problem here is that the formatter rule and its handler itself
are (correctly) encapsulated and thus don't know that in whatever
context we happen to be in the space domain '.tiddlyspace.com'. In
some _other_ context (running the same code, in the same process)
it might be something else.

It matters naught to me whether there are many or one callback, but
from an extensibility standpoint, assuming you want a single link macro,
and the inevitability of wanting to massage links (beyond simple internal
external-ness), from a writer-of-formatter-rules it would be nice to
be able to declare, in my formatter two things:

* that the link I'm creating is a special
* that this (adjacent piece of code) is the massager to make it behave

All that send I'm not wed to any particular way of doing this. I just
want to be able to get space links working. How is an interesting
concern, but secondary for me. I _really_ want to move on from
the 2.6.x-based twikifier.

[1] https://gist.github.com/2149828

[Jeremy Ruston]

> Well the original problem I mentioned was that we need a way to
> resolve space names (or that kind of syntax) which in the most basic
> case means being able to say "prepend the space name onto the string
> '.tiddlyspace.com' to make the hostname portion of the link".
>
> In my gist[1] that's currently a static string, but what we want is to
> be able to declare, somewhere, how space names are resolved,
> preferably in a programatic way.
>
> My understanding of what you were suggesting was that link macro,
> knowing that it is of space type, would look for a function to perform
> the resolution, thus being able to create a correct target.

In fact, I think you do want to process all links, not just space links.

The reason is that I think that the encodeURIComponent() that I added
a few weeks ago is part of the same bundle of special logic: how to
treat TiddlySpace links. I can imagine other hosts that might use a
different way of encoding tiddler titles into URIs (perhaps using
dashes for spaces).

But otherwise, yes, I think we're seeing the same problem and the same solution.

> The main problem here is that the formatter rule and its handler itself
> are (correctly) encapsulated and thus don't know that in whatever
> context we happen to be in the space domain '.tiddlyspace.com'. In
> some _other_ context (running the same code, in the same process)
> it might be something else.

Right. I think the formatter can just shovel the space component into
the parameters for the link macro, and let the link macro invoke the
link generator callback to generate the actual link.

> It matters naught to me whether there are many or one callback, but
> from an extensibility standpoint, assuming you want a single link macro,
> and the inevitability of wanting to massage links (beyond simple internal
> external-ness), from a writer-of-formatter-rules it would be nice to
> be able to declare, in my formatter two things:
>
> * that the link I'm creating is a special
> * that this (adjacent piece of code) is the massager to make it behave

I think this might be where we're differing. I don't see such a tight
coupling between the special formatter and the link massaging rules. I
think that's partly because as I mentioned above, I think it's useful
for the syntax for space links to be part of the TW5 core. This means
that the formatters you've written would become part of the core, and
not be packaged along with the massager.

> All that send I'm not wed to any particular way of doing this. I just
> want to be able to get space links working. How is an interesting
> concern, but secondary for me. I _really_ want to move on from
> the 2.6.x-based twikifier.

Terrific. I'll prototype up some code and get back to you,

Best wishes

Jeremy

> [1] https://gist.github.com/2149828
>
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.

> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>

--

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

> Right. I think the formatter can just shovel the space component into
> the parameters for the link macro, and let the link macro invoke the
> link generator callback to generate the actual link.

[...]

> for the syntax for space links to be part of the TW5 core. This means
> that the formatters you've written would become part of the core, and
> not be packaged along with the massager.

This still leaves me not yet understanding where the space component
is coming from?

> Terrific. I'll prototype up some code and get back to you,

Cool, looking forward to it.

[Jeremy Ruston]

>> Right. I think the formatter can just shovel the space component into
>> the parameters for the link macro, and let the link macro invoke the
>> link generator callback to generate the actual link.
>
>
> [...]
>
>
>> for the syntax for space links to be part of the TW5 core. This means
>> that the formatters you've written would become part of the core, and
>> not be packaged along with the massager.
>
>
> This still leaves me not yet understanding where the space component
> is coming from?

The new link formatters would detect the space links, and generate a
link macro with a "space" parameter containing the space, analogous to
the current "target" parameter.

The link massager, via the link macro, would see the "space" parameter
and generate appropriate markup.

Is that

Cheers

Jeremy

>
>
>> Terrific. I'll prototype up some code and get back to you,
>
>
> Cool, looking forward to it.
>
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

> The new link formatters would detect the space links, and generate a
> link macro with a "space" parameter containing the space, analogous to
> the current "target" parameter.
>
> The link massager, via the link macro, would see the "space" parameter
> and generate appropriate markup.

I get that, but the space is 'cdent'. Where does tiddlyspace.com come
from?

[Jeremy Ruston]

>> The new link formatters would detect the space links, and generate a
>> link macro with a "space" parameter containing the space, analogous to
>> the current "target" parameter.
>>
>> The link massager, via the link macro, would see the "space" parameter
>> and generate appropriate markup.
>
>
> I get that, but the space is 'cdent'. Where does tiddlyspace.com come
> from?

The massager would insert it, I was thinking.

Cheers

Jerm

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

>>> The new link formatters would detect the space links, and generate a
>>> link macro with a "space" parameter containing the space, analogous to
>>> the current "target" parameter.
>>>
>>> The link massager, via the link macro, would see the "space" parameter
>>> and generate appropriate markup.
>>
>>
>> I get that, but the space is 'cdent'. Where does tiddlyspace.com come
>> from?
>
> The massager would insert it, I was thinking.

I'll wait to see the prototype code, but this is what I was thinking
as well and why I thought there would multiple (and extensible) massagers:
Because you only want to make 'cdent' into 'cdent.tiddlyspace.com' if
the link in question is:

* a space link
* a space link in the tiddlyspace.com domain

[Jeremy Ruston]

OK, I've checked in some code:

https://github.com/Jermolene/TiddlyWiki5/commit/d13e2bacaf99c93f240390728054c78a77b8a49b

Let me know how it goes,

Best wishes

Jeremy

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>

--

[chris...@gmail.com]

On Tue, 3 Apr 2012, Jeremy Ruston wrote:

> OK, I've checked in some code:
>
> https://github.com/Jermolene/TiddlyWiki5/commit/d13e2bacaf99c93f240390728054c78a77b8a49b
>
> Let me know how it goes,

I feel like I'm being a bit thick or something, because I'm not sure
I get it.

First off you've got 'href' and 'target' as input parameters. But
then later when generating the Element node, the attribute 'href'
takes the value of 'linkInfo.target', whereas it seems quite likely
one might like to create a link which uses the HTML target attribute, in
which case linkInfo might need a thing for that.

Given that, what are the meaning of: target, space and href;
especially if one's starting point is foo@bar?

My intuition, but I don't see how the code would support this is
that target = 'foo', space='bar', and href would be constructed in
the linkMassager, which is a closure with the upvariable of the space
domain (perhaps 'tiddlyspace.com').

Secondly, I'm not sure how to move forward with the change. You have
installed a default linkMassager in js/App.js.

Presumably in my code[1] when creating the store, a different
linkMassager would be registered. That one would need to:

* duplicate the "linkInfo.target = encodeURIComponent(linkInfo.target);"
in js/App.js, for the sake of correct tiddlywiki-style links

* add code which did something like this:

if (linkInfo.space) {
linkInfo.target = 'http://'
+ linkInf.space
+ '.' + theEnclosedSpaceDomain
+ '/' + encodeURIComponent(linkInfo.target);
}

Is this what you were thinking, or something else?

Thanks.

[1] https://github.com/cdent/tw5ikifier/blob/master/Wikifier.js

[Jeremy Ruston]

> I feel like I'm being a bit thick or something, because I'm not sure
> I get it.

> First off you've got 'href' and 'target' as input parameters. But
> then later when generating the Element node, the attribute 'href'
> takes the value of 'linkInfo.target', whereas it seems quite likely
> one might like to create a link which uses the HTML target attribute, in
> which case linkInfo might need a thing for that.
>
> Given that, what are the meaning of: target, space and href;
> especially if one's starting point is foo@bar?

OK, you're not being stupid: part of the confusion is that several TW5
macros use the term "target" for the title of the tiddler that they
refer to (for instance, <<tiddler target:SomeTiddler>>). Accordingly,
the link macro uses the "target" parameter for the title of the
tiddler being linked to (or the URL for an external link).

The other confusion is that the mechanism doesn't, as it should,
permit the linkMassager to manipulate the HTML "target" property. My
first thought for the callback was for it to process the actual
ElementNode generated by the link macro; I did it the way that you see
in an attempt to try to shield the linkMassager from the
implementation of the ElementNode.

> My intuition, but I don't see how the code would support this is
> that target = 'foo', space='bar', and href would be constructed in
> the linkMassager, which is a closure with the upvariable of the space
> domain (perhaps 'tiddlyspace.com').
>
> Secondly, I'm not sure how to move forward with the change. You have
> installed a default linkMassager in js/App.js.
>
> Presumably in my code[1] when creating the store, a different
> linkMassager would be registered. That one would need to:
>
> * duplicate the "linkInfo.target = encodeURIComponent(linkInfo.target);"
>  in js/App.js, for the sake of correct tiddlywiki-style links

Yes, that is correct. I'm working on the basis that App.js is the "TW5
app", and that other systems using TW5 components (such as tw5ikifier)
would derive their own equivalents.

The idea is that mapping the link target to the URI using
encodeURIComponent() could be replaced with a slug generator that e.g.
uses dashes instead of spaces.

> * add code which did something like this:
>
>    if (linkInfo.space) {
>        linkInfo.target = 'http://'
>            + linkInf.space
>            + '.' + theEnclosedSpaceDomain
>            + '/' + encodeURIComponent(linkInfo.target);
>    }

> Is this what you were thinking, or something else?

Exactly that, modulo the points above about confusing parameter names,
and the need to re-engineer the callback to permit it to manipulate
the HTML target attribute.

I'll review the design tomorrow and ping you when I've got something better,

Best wishes

Jeremy

> Thanks.
>
> [1] https://github.com/cdent/tw5ikifier/blob/master/Wikifier.js
>
>
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[chris...@gmail.com]

On Thu, 5 Apr 2012, Jeremy Ruston wrote:

> Exactly that, modulo the points above about confusing parameter names,
> and the need to re-engineer the callback to permit it to manipulate
> the HTML target attribute.
>
> I'll review the design tomorrow and ping you when I've got something better,

I see that you've done some refactoring on the LinkMassager stuff[1]. Can
you provide some explanation of how tw5ikifier might make use of it
for space stuff?

Thanks.

[1]
https://github.com/Jermolene/TiddlyWiki5/commit/af7b69e778c1c11fc8edfc67b2323d39c8b79a13

[Jeremy Ruston]

Hi Chris

> I see that you've done some refactoring on the LinkMassager stuff[1]. Can
> you provide some explanation of how tw5ikifier might make use of it
> for space stuff?

Sorry about that.

There are docs for the new design at the top of the source for the link macro:

https://github.com/Jermolene/TiddlyWiki5/blob/master/js/macros/link.js

It's now possible for the massager to do whatever it wants to the HTML
attributes of the <a> element, or indeed to completely suppress the
link (I use this option to prevent broken links appearing in the
generated readme.md file on GitHub).

Does that make a bit more sense?

Best wishes

Jeremy

>
> Thanks.
>
> [1]
> https://github.com/Jermolene/TiddlyWiki5/commit/af7b69e778c1c11fc8edfc67b2323d39c8b79a13
>
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[Jeremy Ruston]

Hi Chris

Just to let you know that I'm now going ahead and implementing
TiddlySpace links directly in TW5, hopefully will be done later today.
You may want to wait until those commits before you pick this up,

Best wishes

Jeremy

[chris...@gmail.com]

On Fri, 20 Apr 2012, Jeremy Ruston wrote:

> Just to let you know that I'm now going ahead and implementing
> TiddlySpace links directly in TW5, hopefully will be done later today.
> You may want to wait until those commits before you pick this up,

Oh cool, glad to hear it. What's the status now?

[Jeremy Ruston]

Hi Chris

I'm afraid that I ended up going down a rather long rabbit hole there.
I hope to be committing a major rewrite in the next day or two.

The background is that I realised there was a common element between
your requirements for customising the link behaviour and some
requirements that came out of my consultancy work. In both cases, I
wanted to be able to make it easier to customise the behaviour of
components within TiddlyWiki.

So, I've radically re-engineered the current code to incorporate a
flexible mechanism for plugins. In fact, TW5 is now a 500 line
micro-kernel that runs on node.js or in the browser, and everything
else is plugins. The kernel boots just enough of the TiddlyWiki
environment to allow it to load tiddlers as plugins and execute them
(a barebones tiddler class, a barebones wiki store class, some
utilities etc.). Plugin modules are written like node.js modules; you
can use require() to invoke sub components and to control load order.

There are several different types of plugins: parsers, serializers,
deserializers, macros etc. To my pleasure, I've been able to go much
further than I expected. For example, individual tiddler fields are
plugins, too: there's a plugin that knows how to handle the tags
field, and another that knows how to handle the special behaviour of
the modified and created fields.

Some plugins have further sub-plugins; the wikitext parser, for
instance, accepts rules as individual plugins.

It's been fun doing such a major refactoring. I ended up starting from
scratch, writing the new boot kernel first and then one by one
transferring modules from the old code base to the new, refactoring as
I've gone. One quite satisfying aspect is that I'm using the previous
version of TW5 to build the new version.

I've been thinking about this approach for a while. I'd consciously
avoided implementing anything until I felt I understood enough of the
requirements to be confident that any new mechanism was elegant and
sufficient. It feels absolutely right, and is definitely a step up
from the rather crude plugin model in classic TW.

I'm now at the point where everything is up and running again, I'm
just fixing some typos and fixing some bugs and then I'll commit the
code.

Once that's done I'll create a starter TiddlySpace plugin that we can
use to refine tw5ikifier.

Best wishes

Jeremy


--
Jeremy Ruston
jeremy...@gmail.com

[Chris Dent]

On Sunday, April 29, 2012 2:52:29 PM UTC+1, Jeremy Ruston wrote:

I'm now at the point where everything is up and running again, I'm
just fixing some typos and fixing some bugs and then I'll commit the
code.

Once that's done I'll create a starter TiddlySpace plugin that we can
use to refine tw5ikifier.

How are things now? I've seen that there have been a lot of commits refactoring stuff and I've managed to lose track of the entry point that I'll want to use to hook my code in with your new stuff. I'm assuming this is no longer in the proper form:

    https://github.com/cdent/tw5ikifier/blob/master/Wikifier.js 

[Jeremy Ruston]

> How are things now? I've seen that there have been a lot of commits
> refactoring stuff and

I've been rather heads down these last few weeks, and not reported
back on progress. The big news since we last spoke is that I've
completed two major refactorings:

* New microkernel architecture, with everything being plugins all the way down
* New wikitext parser, which properly parses paragraphs, and includes
many refinements such as nested macros

> I've managed to lose track of the entry point that
> I'll want to use to hook my code in with your new stuff. I'm assuming this
> is no longer in the proper form:
>
>     https://github.com/cdent/tw5ikifier/blob/master/Wikifier.js

No, the new plugin architecture has changed all the details fairly
radically, but the broad approach remains the same.

I think I need to do two things to let you proceed:

* Document/demonstrate how to use the new code as a component in a
bigger node.js application
* Specify a new class of link massager plugins

We'd also need to figure out a way for TiddlySpace users to configure
which wikitext parser they're going to get. Having worked with the new
parser for a week or so, I'm finding that it leads to much cleaner,
more spacious, easier to read wikitext -- mainly by adopting the
markdown convention of double-newline to end a paragraph, and treating
single newlines as whitespace. And of course it's much, much nicer to
style than the old <br> tags.

Anyway, I remain very keen to get TW5 on TiddlySpace, and hope I can
get those bits done swiftly.

Best wishes

Jeremy




--
Jeremy Ruston

mailto:jeremy...@gmail.com

[chris...@gmail.com]

On Mon, 11 Jun 2012, Jeremy Ruston wrote:

> We'd also need to figure out a way for TiddlySpace users to configure
> which wikitext parser they're going to get. Having worked with the new
> parser for a week or so, I'm finding that it leads to much cleaner,
> more spacious, easier to read wikitext -- mainly by adopting the
> markdown convention of double-newline to end a paragraph, and treating
> single newlines as whitespace. And of course it's much, much nicer to
> style than the old <br> tags.

I've been thinking about this as well. There are a few variables
which can control that:

* The rendering subsystem of TiddlyWeb can apply different render()
methods to a tiddler to make it into HTML based on the 'type'
attribute on the tiddler. At the moment if this is "None" or a few
variations on that theme, it is considered tw2 wikitext. The other
currently supported type is markdown.

So, if your new wikitext is of a different type, then applying a
different renderer is no problem: Server side configuration
pointing to a handler.

I think your code does something a bit like this, but I may be
mis-remembering the glance I made through it.

* A space ought to be able to declare which renderer it uses for the
same type. ServerSettings can be extended to do that[1].

So, if the wikitext type is the same, but people might prefer a
different rendering engine (wikklytext, twikifier, tw5ikifer, etc)
they ought to be able to declare that.

It's not (yet) clear whether the new text has a different type, so
which way it goes will have to wait and see.

> Anyway, I remain very keen to get TW5 on TiddlySpace, and hope I can
> get those bits done swiftly.

Getting tw5 at large (the entire thing/concept, not just a renderer) onto
TiddlySpace ought not be too much of an ordeal. Again there are a
few different possible directions:

* Do something akin to the tw2 serialization: push tiddlers into an
HTML doc which is the tiddlywiki5 thing.

* Do something akin to the /_tiddlywiki[2] app which is on
TiddlySpace. That is: load a kernel, and then calculate (based on
the URI) which tiddlers to load in.

@dash and @mosaic[3] do similar things.

The latter seems in keeping with what it has sounded like your goals
may be.

It's also in keeping with the goals we have for TiddlySpace: a host
for tiddlers with lots of different tools that work with them. Some
of those tools work with aggregations of tiddlers, others with just
one. This ought to mean that tw2, tw5 and other tools can cooperate
over the same corpus.

[1] It has recently been extended to allow setting an 'editor' for the
space.

[2] http://apps.tiddlyspace.com/_tiddlywiki.txt (It's a tiddlwiki
cooked up to bootstrap itself with server side stuff)

[3] http://dash.tiddyspace.com/ http://mosaic.tiddlyspace.com/

[Jeremy Ruston]

> I've been thinking about this as well. There are a few variables
> which can control that:
>
> * The rendering subsystem of TiddlyWeb can apply different render()
>  methods to a tiddler to make it into HTML based on the 'type'
>  attribute on the tiddler. At the moment if this is "None" or a few
>  variations on that theme, it is considered tw2 wikitext. The other
>  currently supported type is markdown.
>
>  So, if your new wikitext is of a different type, then applying a
>  different renderer is no problem: Server side configuration
>  pointing to a handler.
>
>  I think your code does something a bit like this, but I may be
>  mis-remembering the glance I made through it.

Yes, that makes sense. Currently, the new parser is text/x-tiddlywiki
and the old parser is text/x-tiddlywiki-old.

Parenthetically, I'm very hopeful that we will be able to engineer
pretty good converters that convert tiddlers from old to new format,
pretty much by judicious insertion of additional line breaks. If that
is possible, I'll include it in the TW5 import engine, and we may want
to explore building a little 'upgrade my space' app for TiddlySpace.

> * A space ought to be able to declare which renderer it uses for the
>  same type. ServerSettings can be extended to do that[1].
>
>  So, if the wikitext type is the same, but people might prefer a
>  different rendering engine (wikklytext, twikifier, tw5ikifer, etc)
>  they ought to be able to declare that.
>
> It's not (yet) clear whether the new text has a different type, so
> which way it goes will have to wait and see.

Yes, likewise, that sounds sensible.

> Getting tw5 at large (the entire thing/concept, not just a renderer) onto
> TiddlySpace ought not be too much of an ordeal. Again there are a
> few different possible directions:
>
> * Do something akin to the tw2 serialization: push tiddlers into an
>  HTML doc which is the tiddlywiki5 thing.

Currently TW5 packs tiddlers in the same <div> format as TW2. I plan
to introduce a new, preferred JSON-in-a-script-tag format which should
be a bit easier in terms of encodings and so on.

> * Do something akin to the /_tiddlywiki[2] app which is on
>  TiddlySpace. That is: load a kernel, and then calculate (based on
>  the URI) which tiddlers to load in.

I've not done it yet, but the plan is to have lazy loading built into
TW5, very much along the lines that we've discussed in the past:
optionally pack skinny tiddlers in the HTML file, and reach out over
HTTP to get the content on first access. Image references to lazily
loaded tiddlers would just generate a <img> tag with the src pointing
to the URL of the image, which I think would be useful for TiddlySpace
too.

>  @dash and @mosaic[3] do similar things.

I've been watching these developments with interest.

> The latter seems in keeping with what it has sounded like your goals
> may be.
>
> It's also in keeping with the goals we have for TiddlySpace: a host
> for tiddlers with lots of different tools that work with them. Some
> of those tools work with aggregations of tiddlers, others with just
> one. This ought to mean that tw2, tw5 and other tools can cooperate
> over the same corpus.

Yes, exactly that. TW5 is all the stronger for not having to worry
about serious HTTP serving issues. Hopefully I'm skating towards the
same puck as you.

Best wishes

Jeremy

> [1] It has recently been extended to allow setting an 'editor' for the


> space.
>
> [2] http://apps.tiddlyspace.com/_tiddlywiki.txt (It's a tiddlwiki
> cooked up to bootstrap itself with server side stuff)
>
> [3] http://dash.tiddyspace.com/ http://mosaic.tiddlyspace.com/
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWikiDev" group.
> To post to this group, send email to tiddly...@googlegroups.com.
> To unsubscribe from this group, send email to
> tiddlywikide...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tiddlywikidev?hl=en.
>

[chris...@gmail.com]

On Mon, 11 Jun 2012, Jeremy Ruston wrote:

> Yes, that makes sense. Currently, the new parser is text/x-tiddlywiki
> and the old parser is text/x-tiddlywiki-old.

Over the longer term that's probably not ideal. On tiddlyspace.com
there are some very large number of tiddlers extant which are already
treated as text/x-tiddlywiki.

That is: The currently assumed MIME type for tiddlywiki 2 tiddlers is
text/x-tiddlywiki, so tiddlywiki 5 tiddlers, in a different format,
probably want a different type.

> Parenthetically, I'm very hopeful that we will be able to engineer
> pretty good converters that convert tiddlers from old to new format,
> pretty much by judicious insertion of additional line breaks. If that
> is possible, I'll include it in the TW5 import engine, and we may want
> to explore building a little 'upgrade my space' app for TiddlySpace.

That will be cool and useful, but for as long as tiddlywiki2 remains
the primary entry point that will need support. Log analysis shows
that thus far exploration of paths other than the default entry point
is very very very very light.

Unless of course the wikifier for the newer tw5 syntax gets backported
(as a plugin) to tw2. Which would be handy.

[Jeremy Ruston]

>> Yes, that makes sense. Currently, the new parser is text/x-tiddlywiki
>> and the old parser is text/x-tiddlywiki-old.
>
>
> Over the longer term that's probably not ideal. On tiddlyspace.com
> there are some very large number of tiddlers extant which are already
> treated as text/x-tiddlywiki.
>
> That is: The currently assumed MIME type for tiddlywiki 2 tiddlers is
> text/x-tiddlywiki, so tiddlywiki 5 tiddlers, in a different format,
> probably want a different type.

Yes, to start with I had the new parser as text/x-tiddlywiki-new, but
I'm not too keen on that, because I want to look forwards to the new
syntax being 100 times more popular than the old, and the old to be
consigned to being a historical footnote. I suppose calling it
text/x-tiddlywiki5 might be better.

One possibility might be to use text/x-tiddlywiki5 for the new format,
and text/x-tiddlywiki for the old, but to deprecate the latter in
favour of text/x-tiddlywiki2. Then we could declare that at some
future coordinated release of TiddlyWiki and TiddlyWeb we'll switch
the default interpretation of text/x-tiddlywiki to text/x-tiddlywiki5.

> That will be cool and useful, but for as long as tiddlywiki2 remains
> the primary entry point that will need support. Log analysis shows
> that thus far exploration of paths other than the default entry point
> is very very very very light.

Right, I'd agree that whatever we do we need to preserve tiddlywiki2
spaces for the indefinite future. But I'd hope that tw5 spaces will
outnumber the tw2 ones quite quickly.

> Unless of course the wikifier for the newer tw5 syntax gets backported
> (as a plugin) to tw2. Which would be handy.

I think it would also break rather a lot of things. I'd hate to have
to support tw2/5 frankenstein spaces as well as tw2 spaces and tw5
spaces; surely it's simpler to focus on good support for both of the
latter?

I'd love quite soon to be able to offer a link on
tiddlywiki.com/tiddlywiki5 that takes people to a TiddlySpace page
that enables them to register if necessary, and create a tw5 space in
a single step. Done right, this would be the smoothest easiest path
for a visitor to tiddlywiki.com to get up and running.

As I've said, my goal with tw5 is to make it much, much simpler and
easier to use. My hope is that this can make the thing 100 times more
popular than in the past. tw2 users will be our valued pioneers that
helped shape the product, but the future will belong to tw5. None of
that undermines the commitment above to keeping tw2 working, but it's
helpful for me in figuring out where to put my attention.

Best wishes

Jeremy

>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[FND]

> One possibility might be to use text/x-tiddlywiki5 for the new format,
> and text/x-tiddlywiki for the old, but to deprecate the latter in
> favour of text/x-tiddlywiki2. Then we could declare that at some
> future coordinated release of TiddlyWiki and TiddlyWeb we'll switch
> the default interpretation of text/x-tiddlywiki to text/x-tiddlywiki5.

Who benefits from such a dance? I'd rather stick to something
stable and universal.

I'm don't run with the hardcore REST crew, but those people have
discussed media types at length - often in the context of versioning
HTTP APIs. If I recall correctly, they usually recommend custom media
types to look like this: application/vnd.<...>;version=<...>
(personally, I abhor[1] the "vendor" thing, and would rather see "text"
than "application", but I'm not sure whether there's actual consensus
around those things).

I look forward to TW5 joining the ecosystem as Yet Another Tiddler Handler.


[1] http://fnd.tiddlyspace.com/136582fd473

[Jeremy Ruston]

I think the nub for me is that tw2 and tw5 wikitext don't really feel
like entirely separate content types. tw5 text is largely backwards
compatible out of the box, and as I've said, there is reason to be
hopeful that the details that have changed are amenable to automated
conversion.

So, I see the logic in this discussion as it would apply to two
utterly different wikitext formats, but this feels more like the
change from HTML4 to HTML5; no-one expected that to be done with a new
content type "text/html5".

Here, as Chris has already suggested, we could keep both wikitexts as
text/x-tiddlywiki, and make it a per-space setting to control which
parser is used.

On Tue, Jun 12, 2012 at 12:35 PM, FND <FN...@gmx.net> wrote:
>> One possibility might be to use text/x-tiddlywiki5 for the new format,
>> and text/x-tiddlywiki for the old, but to deprecate the latter in
>> favour of text/x-tiddlywiki2. Then we could declare that at some
>> future coordinated release of TiddlyWiki and TiddlyWeb we'll switch
>> the default interpretation of text/x-tiddlywiki to text/x-tiddlywiki5.
>
> Who benefits from such a dance? I'd rather stick to something
> stable and universal.

Just to be clear, the benefit would be that the majority of TiddlyWiki
users (ie those people who are not yet using it) would see a nice
simple world with a content type that just means "wikitext". They
wouldn't see any residue showing that there was once two different
wikitext formats.

In other words I'm suggesting a policy for this issue with two parts:
(a) don't break existing stuff and (b) polish the cowpaths for new
users.

> I'm don't run with the hardcore REST crew, but those people have
> discussed media types at length - often in the context of versioning
> HTTP APIs. If I recall correctly, they usually recommend custom media
> types to look like this: application/vnd.<...>;version=<...>
> (personally, I abhor[1] the "vendor" thing, and would rather see "text"
> than "application", but I'm not sure whether there's actual consensus
> around those things).

Versioned content types seem like a nightmare to me, I haven't
actually come across them in the wild.

> I look forward to TW5 joining the ecosystem as Yet Another Tiddler Handler.

Yes, but this discussion strays into something else, much more
intricate and interesting: how do we upgrade what constitutes a
wikitext tiddler?

Best wishes

Jeremy

>
> [1] http://fnd.tiddlyspace.com/136582fd473

[chris...@gmail.com]

On Tue, 12 Jun 2012, Jeremy Ruston wrote:

> Here, as Chris has already suggested, we could keep both wikitexts as
> text/x-tiddlywiki, and make it a per-space setting to control which
> parser is used.

This actually isn't my preferred solution. A preferred solution is
that each space declares which mime type they want their (new) tiddlers
to default to.

The global view on tiddyspace is that it just has tiddlers and a
tiddler can be _anything_. How they are presented is dependent on their type.

That those tiddlers that currently exist which have no type are treated
as tiddlywiki2text is an accident of history.

To avoid continuing that accident in the future, when a new syntax is
introduced, tiddlers using that syntax should get a mime type and be
handled explicitly. That the change from tw2 to tw5 syntax is small
doesn't much matter: it still will require a transform and that
transform process can (and should) set the mime type.

This is all in alignment with tiddlyweb raising the tiddler to the
primary entity. It's not the application (tiddlywiki etc) that
matters, it's the tiddler.[1]

This means, effectively, that a user can come along and say "I'm going
to start creating my textual tiddlers using tw5 syntax" _and_ they can
say "I'm going to use this app to do a one time transform of my
existing tiddlers to the new syntax" _and_ they can say "For these
tiddlers I'm going to use markdown".

And from an administrative side there is a time when we _could_ say
"The default app for new spaces creates tiddlywiki5 syntax tiddlers
(because it _is_ tiddywiki5)".

> Just to be clear, the benefit would be that the majority of TiddlyWiki
> users (ie those people who are not yet using it) would see a nice
> simple world with a content type that just means "wikitext". They
> wouldn't see any residue showing that there was once two different
> wikitext formats.

This seems at least spritually aligned with what I'm saying above.

> Yes, but this discussion strays into something else, much more
> intricate and interesting: how do we upgrade what constitutes a
> wikitext tiddler?

If we treat a tiddler as a thing with a type, this doesn't seem all
that hard to me. The details are in how we present to the existing
tiddler keepers that they have the option to change if they want.

[1] http://cdent.tiddlyspace.com/Evolution

[chris...@gmail.com]

On Tue, 12 Jun 2012, Jeremy Ruston wrote:

> Right, I'd agree that whatever we do we need to preserve tiddlywiki2
> spaces for the indefinite future. But I'd hope that tw5 spaces will
> outnumber the tw2 ones quite quickly.

I think this paragraph may highlight where our views of the future are
slightly divergent. I don't perceive a future where there are tw2 or
tw5 spaces. I just see spaces with tiddlers in them.

Some markdown, some tiddlywiki2, some tiddlywiki5, some html, some
creole, some etc etc.

Of course it will be good form to choose a syntax that allows good
linking between tiddlers (if you want that sort of thing).

[Jeremy Ruston]

>> Here, as Chris has already suggested, we could keep both wikitexts as
>> text/x-tiddlywiki, and make it a per-space setting to control which
>> parser is used.
>
> This actually isn't my preferred solution. A preferred solution is
> that each space declares which mime type they want their (new) tiddlers
> to default to.

Well I was underlining my key point, which is that for me it's not yet
clear that having two different content types is necessarily the best
solution. Or at least, that I acknowledge that it is the prima facie
obvious solution, but am keen to explore the ramifications on end
users. I return to my analogy with HTML4 vs HTML5, which are
incompatible but served with the same content type: one of the things
we might explore is simple heuristics to allow the parser to determine
which syntactic variant to use, allowing us to use a single content
type. Right now, after all, everything in TW5 is entirely mutable.

Being far from an expert on content types, I've wondered about using
parameters like `text/x-tiddlywiki; newlines=old`. Presumably that's
evil?

Your second sentence about the preferred solution being that each
space declares a default content type for new tiddlers would still
hold true for me.

> The global view on tiddyspace is that it just has tiddlers and a
> tiddler can be _anything_. How they are presented is dependent on their
> type.

Agreed.

> That those tiddlers that currently exist which have no type are treated
> as tiddlywiki2text is an accident of history.

Yes, although conceivably it may help us, in that there is little
material out there that is yet explicitly tagged with a TiddlyWiki
wikitext content type. On the TiddlyWiki side there is none.

> To avoid continuing that accident in the future, when a new syntax is
> introduced, tiddlers using that syntax should get a mime type and be
> handled explicitly. That the change from tw2 to tw5 syntax is small
> doesn't much matter: it still will require a transform and that
> transform process can (and should) set the mime type.

Yes, and that was my starting position too. My concern when we kicked
off this discussion was just about the best names for the content
types, and trying to focus things so that the clearest user experience
is for users who arrive after the TW5 transition.

> This is all in alignment with tiddlyweb raising the tiddler to the
> primary entity. It's not the application (tiddlywiki etc) that
> matters, it's the tiddler.[1]

Yes, agreed.

> This means, effectively, that a user can come along and say "I'm going
> to start creating my textual tiddlers using tw5 syntax" _and_ they can
> say "I'm going to use this app to do a one time transform of my
> existing tiddlers to the new syntax" _and_ they can say "For these
> tiddlers I'm going to use markdown".

Yes, agreed.

> And from an administrative side there is a time when we _could_ say
> "The default app for new spaces creates tiddlywiki5 syntax tiddlers
> (because it _is_ tiddywiki5)".

Yes, agreed.

> This seems at least spritually aligned with what I'm saying above.

Yes, I think so.

> If we treat a tiddler as a thing with a type, this doesn't seem all
> that hard to me. The details are in how we present to the existing
> tiddler keepers that they have the option to change if they want.

Exactly.

Best wishes

jeremy

> [1] http://cdent.tiddlyspace.com/Evolution
>
>
> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[Jeremy Ruston]

>> Right, I'd agree that whatever we do we need to preserve tiddlywiki2
>> spaces for the indefinite future. But I'd hope that tw5 spaces will
>> outnumber the tw2 ones quite quickly.
>
> I think this paragraph may highlight where our views of the future are
> slightly divergent. I don't perceive a future where there are tw2 or
> tw5 spaces. I just see spaces with tiddlers in them.

Well, I think you might be reading too much into that. Fairly
obviously all I mean by "tw2 spaces" is "spaces that contain wikitext
content in tw2 format", and so I'm articulating the hope that if one
were to classify spaces in that way, one would see a the "tw5 spaces"
quite quickly outnumber the "tw2 spaces". I'm just again reiterating
my belief that we should be focussed on what we want the world to look
like post-transition.

Anyhow, in practice, I'd consider it very important to be able to have
spaces that mixed tw2 and tw5 content, as I'm sure you would too.

> Some markdown, some tiddlywiki2, some tiddlywiki5, some html, some
> creole, some etc etc.
>
> Of course it will be good form to choose a syntax that allows good
> linking between tiddlers (if you want that sort of thing).

Are you talking about the linking syntax in TW5? At the moment it's
the same as tw2 (with plans for the TiddlySpace extensions), but I'm
open to suggestions for improvements.

Best wishes

Jeremy

> --
> Chris Dent                                   http://burningchrome.com/
>                                [...]
>

[chris...@gmail.com]

On Tue, 12 Jun 2012, Jeremy Ruston wrote:

>> Of course it will be good form to choose a syntax that allows good
>> linking between tiddlers (if you want that sort of thing).
>
> Are you talking about the linking syntax in TW5? At the moment it's
> the same as tw2 (with plans for the TiddlySpace extensions), but I'm
> open to suggestions for improvements.

The syntax of indicating a link isn't really what I'm remarking on
there. More the awareness (in the rendering engine) of there being a
domain (recipe/bag) in which tiddlers can link to one another.

For example in tiddyspace, now, it is easy to link between tiddlers if
they are in tiddlywiki(2) text. That's "built in" with the various
renderers that can handle that type.[1]

The markdown renderer, however, had to be modified to be able to allow
links between tiddlers and it is flaky.[1]

What I'm saying with that "of course" statement can be translated as
"if you want your space to be a wiki, you'll likely want to choose one
syntax, and one that does wiki linking". Waving my flag basically.

[1] One of the reasons for tw5ikifier over twikifier is that the newer
one provided much more robust classing of the HTML for doing styling
based on whether linked-to tiddler exists and such.

[Jeremy Ruston]

Aha, that all makes sense.

Cheers

Jeremy

[PMario]

Hi folks,
I'd like to know a number, how many tiddlers actually contain a custom
type field "server.content-type" that is "text/x-tiddlywiki"?
I wasn't able to create a query string that worked. @cdent :)

If there is an easy migration path, I'd be willing to "manually"
change my tiddlers, to use TW5 renderer (when it's finished). I
personally wouldn't use a fully automatic update mechanism, since I'm
sure it will break stuff.

I could imagine a semiautomatic plugin/SPA, that has a TW2/TW5 side by
side view plus 2 buttons [accept update] [log for review]. There will
be some time involved, but afterwards I'm sure, I'm fine.

====

bengillies bookmarked [1] a very interesting article [2] about minting
mime types.

imo an IANA [3] registration, where "tiddler" is the thing, would look
like:

text/vnd.tiddler.tw2
text/vnd.tiddler.tw5 or even
text/vnd.tiddler.tw

I'd even keep the version numbers, if possible.

On Jun 11, 11:46 pm, Jeremy Ruston <jeremy.rus...@gmail.com> wrote:
>Currently TW5 packs tiddlers in the same <div> format as TW2. I plan
>to introduce a new, preferred JSON-in-a-script-tag format which should
>be a bit easier in terms of encodings and so on.

I'm not sure, if this would be a new mime-type or just a new
serverside serialisation.

-mario

[1] http://bengillies.tiddlyspace.com/Minting%20new%20Internet%20Media%20Type%20Identifiers%20|%20Sebastien%20Lambla
[2] http://codebetter.com/sebastienlambla/2011/02/01/minting-new-internet-media-type-identifiers/
[3] http://www.iana.org/assignments/media-types/index.html

[chris...@gmail.com]

On Tue, 12 Jun 2012, PMario wrote:

> I'd like to know a number, how many tiddlers actually contain a custom
> type field "server.content-type" that is "text/x-tiddlywiki"?
> I wasn't able to create a query string that worked. @cdent :)

There aren't any public ones:

http://tiddlyspace.com/search?q=type:text/x-tiddlywiki

(type on the server gets translate to server.content-type in
tiddlywiki).

There is 1 private one, which is yours.

However, as I've said elsewhere, if type isn't set it _means_, as far
as the rendering system is concerned, text/x-tiddlywiki. So any
transformation engine would have to account for that.

> imo an IANA [3] registration, where "tiddler" is the thing, would look
> like:
>
> text/vnd.tiddler.tw2
> text/vnd.tiddler.tw5 or even
> text/vnd.tiddler.tw

This would be right, because imposes on the tiddler that it _is_ a
tiddler, which we don't want to do. The fact it is a tiddler is
ancillary. Anything can be a tiddler a tiddler can be anything. So a
tiddler which happens to be tiddlywikitext needs to say _that_, not
that it is a tiddler.

This distinction is critical to understanding the whole system.