Trying to move TiddlyTemplating into the core

[jnthnlstr]

Hello,

TiddlyTemplating has been around for a little while now and there have
been a few discussions about it and suggestions that people use it to
solve problems - this is a search on just this group:

http://groups.google.com/group/TiddlyWikiDev/search?q=TiddlyTemplating&start=0&scoring=d&

I have been speaking with Martin today about pushing the basic
mechanism of templating into the core and what is required for that to
happen, as well as what outstanding questions exist.

To recap: the underlying idea behind TiddlyTemplating is that you can
create HTML templates and apply them to each of a set of tiddlers to
create an output. A use case that has existed right from the start is
a replacement for the various file saves that can be triggered from a
TiddlyWiki, namely: saving the file itself, taking a backup of the
original file, saving an empty TiddlyWiki, publishing an RSS feed of
changed tiddlers. Apart from consolidating the mechanism, it would
also reduce the time taken to save since it would obviate the need to
load the file from the file system first. Other use cases include
publishing ATOM feeds and web pages.

It looks like there are two stages to supporting a generic templating
model, with the existing save/publish mechanism replaced. I'll outline
them below. I apologise in advance for any over-lengthiness, but I
wanted to get it all down somewhere.

1) Core templating mechanism

This involves supporting the processing of templates and, if done
using current implementation, would involve integrating (most of) the
following code:
TemplatePlugin [1] - central function for expanding templates and two
macros for hooking into this
TemplateFormatterPlugin [2] - a special formatter used by the wikifier
to process templates

This would allow anyone to create macros/plugins that used the
templating process, but doesn't explicitly add any new functionality
as such. Martin pointed out that putting something in the core that
doesn't expose anything might be better off sitting in a plugin until
it's needed.

[1] http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/TemplatePlugin.js
[2] http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/experimental/TemplateFormatterPlugin.js

2) Replacing existing file saving/publishing

The implementation of this is incomplete, but at the moment the
following plugins and templates exist:
TiddlyTemplatingMacro [3] - actually this was made for the website
publishing, but uses generic methods and illustrates how to template
and publish files
RssTemplate, RssItemTemplate, RssItemCategoryTemplate [4],[5],[6] -
templates used to output a RSS feed

[3] http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/TiddlyTemplatingMacro.js
[4],[5],[6] http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/templates/

Unfortunately, three problems reared their head when discussing this
today:

a) Saving the backup

This saves the TiddlyWiki file as it existed *before* it was edited,
so is effectively a copy of what is on the file system. We're trying
to avoid loading from the file system before saving, so this will need
to be worked around.

b) Saving the TiddlyWiki

Martin's FastSavePlugin [7] extends the TemplateFormatterPlugin [2] to
process the TiddlyWikiTemplate [8]. However, this is very much
handling the file saving as a special case and it seems to me that
using any standard templating mechanism would be more stable. However,
the counter-argument is that the macros that would need to be written
and would sit in the new TiddlyWikiTemplate would only be used for
that use case, so from a code size point of view, it is not optimal.

[7] http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/experimental/FastSavePlugin.js
[8] http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/tiddlers/TiddlyWikiTemplate.tiddler

c) Default Templates

What is the best way to include any default templates in the system?
When they are integral to the fundamental feature of being able to
save your TiddlyWiki, is it important to make them hard/impossible to
change? Does this mean an implementation of read-only shadow tiddlers,
the use of internal variables, or something else?

Thanks for your patience if you've made it this far. I'd welcome
comments on anything mentioned so far, but particularly part 2. I hope
to have part 1 ready to go (as core patches) over the next day or two.



J.

[Anthony Muscio]

J,

As an advanced user (rather than Developer) my primary use of the word
"template" in tiddlywiki is to create tiddlers with "customised
content" and the use of tags that allow custom ViewTemplates, First
EditTemplates and subsequent editTemplates. that is the process of
having templated tiddlers and views. I simply mention this to assist
in future clarity.

Perhaps we can talk of;
TiddlerTemplates
TiddlyWikiTemplates
File, output, export or save templates

I presume you are talking about the last one in the list. Which to a
user of tiddly wiki does not feature, the shadow tiddlers they know
called templates are the view and edit templates only.

Personally I think this issue of templating is all part of the same
big question facing the tiddlyWiki environment today. I have been
reading in this and the user forums and experimenting with tiddler
Import, Sync the plugins such as Archive, SnapShot, RSS, Import/
Export, Publish, Include and others, none yet seems to deliver exactly
what I want.

I think a generalised approach is required to match the quality of the
core tiddlyWiki concepts that already exist. It is tiddly wiki's
greatest feature and handicap that it is a single file and, given the
inspiration this quality tool gives us, we all want to see it over
come it's only handicap as well - connectivity with external files
even if these are self referential as in the case of templates.

I will watch your progress with a keep interest and contribute if I
see I can help.

Perhaps a clear statement of requirements is the first step. If I have
this I am sure I can add value.

Tony

On Jul 9, 2:32 am, jnthnlstr <jnthnl...@googlemail.com> wrote:
> Hello,
>
> TiddlyTemplating has been around for a little while now and there have
> been a few discussions about it and suggestions that people use it to
> solve problems - this is a search on just this group:
>

> http://groups.google.com/group/TiddlyWikiDev/search?q=TiddlyTemplatin...

> [1]http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/T...
> [2]http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/experimenta...

>
> 2) Replacing existing file saving/publishing
>
> The implementation of this is incomplete, but at the moment the
> following plugins and templates exist:
> TiddlyTemplatingMacro [3] - actually this was made for the website
> publishing, but uses generic methods and illustrates how to template
> and publish files
> RssTemplate, RssItemTemplate, RssItemCategoryTemplate [4],[5],[6] -
> templates used to output a RSS feed
>

> [3]http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/T...

> [4],[5],[6]http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/templates/
>
> Unfortunately, three problems reared their head when discussing this
> today:
>
> a) Saving the backup
>
> This saves the TiddlyWiki file as it existed *before* it was edited,
> so is effectively a copy of what is on the file system. We're trying
> to avoid loading from the file system before saving, so this will need
> to be worked around.
>
> b) Saving the TiddlyWiki
>
> Martin's FastSavePlugin [7] extends the TemplateFormatterPlugin [2] to
> process the TiddlyWikiTemplate [8]. However, this is very much
> handling the file saving as a special case and it seems to me that
> using any standard templating mechanism would be more stable. However,
> the counter-argument is that the macros that would need to be written
> and would sit in the new TiddlyWikiTemplate would only be used for
> that use case, so from a code size point of view, it is not optimal.
>

> [7]http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/experimenta...
> [8]http://svn.tiddlywiki.org/Trunk/contributors/MartinBudden/tiddlers/Ti...

[Jonathan Lister]

Tony,

Thanks for your comments. Apologies for creating confusion. I can see the separate use case you are describing, and I admit that I haven't focussed on the custom views of tiddlers. However, we've used TiddlyTemplating (to use my term) to create custom views of tiddlers before, so what I'm saying is that the mechanism is general purpose enough to be used both within a TiddlyWiki and for publishing to external files.

For example, for the RippleRap BlogTalk edition, we had different views for tiddlers that showed Flickr, Twitter and Blog imports e.g:

http://www.ripplerap.com/BlogTalk2008/blogtalk.html#Blogs

I agree with you that TiddlyWiki's nature as a single file is one of the major elephants in the room...

J.

--
http://www.jaybyjayfresh.com

[Daniel Baird]

On Wed, Jul 9, 2008 at 2:32 AM, jnthnlstr <jnth...@googlemail.com> wrote:
>
> Hello,
> [...]

> I have been speaking with Martin today about pushing the basic
> mechanism of templating into the core

Will that make the core much bigger? Templating seems too specialised
to be a core feature.

Perhaps I've misunderstood something..

;D

--
Daniel Baird
/to be or not to be/ => /(2b|[^2]b)/ => /(2|[^2])b/ => /.b/
...optimise your regexes, people!

[jnthnlstr]

Tony,

I've just reworked most of the TiddlyTemplating documentation on the
community wiki, to try and make the explanation of what it can do
clearer.

Please let me know whether this makes sense to you and what you'd like
explaining further or in a different way.

http://tiddlywiki.org/wiki/TiddlyTemplating



J.

[jnthnlstr]

The use case for the core is as part of the reworking of the saving
process, which consumes a lot of code and is not standardised e.g.
saving an RSS feed is wildly different from saving an empty
TiddlyWiki. So, we hope to actually reduce the size of the core by
adding templating in as the standard interface for creating external
files.

The other part of the reworked saving process, which hasn't had any
development work done on it yet, is the idea of creating a list of
save targets, each of which is probably a systemServer-like tiddler
that describes the filter to use, the template to use and the file
name to save to.

You could replace the word "save" with "publish" and see that this is
a specific case of a very general mechanism.


J.

On Jul 9, 1:18 pm, "Daniel Baird" <danielba...@gmail.com> wrote:

[Anthony Muscio]

J,

Thanks for that, I will work through all the text before I give a fuller review.

Tony M

--
Anthony Muscio

[jnthnlstr]

Update: I've submitted the base code for templating as a core
enhancement.

http://trac.tiddlywiki.org/ticket/700

Description:
This is the minimal set of code to add Templating into the core, where
Templating refers to TiddlyTemplating? as described at:
http://tiddlywiki.org/wiki/TiddlyTemplating

This defines a new core file, Templating.js, which contains code for
the following:
* expandTemplate - a function to render an array of tiddlers with a
template
* templateFormatter - simple formatter for processing templates
* templateTiddlers - macro to find a set of tiddlers and render them
once each through a template
* templateTags - macro to render a tiddler's tags through a template

This patch does not add in an actual use case of templating, but
leaves that to future patches, instead preferring to bake the core
mechanism into the core.



J.