TiddlyWiki Development News
Jeremy Ruston
time [1], my plan was to strike a balance between consultancy work and
putting significant effort into bringing TiddlyWiki up to date, and
making it more useful. I've now reached a bit of a milestone that is
worth explaining. It's technical and confusing, but rewardingly
mind-blowing, hence my posting in this group.
At the risk of sowing even more confusion, I'll try a metaphorical
explanation first.
----
TiddlyWiki is a car with the unusual ability of creating copies of
itself. The car is so complex and fiddly to work on that we keep all
the components spread out on the floor of our workshop. When we want
to test the car, we have a robot called Ruby that whizzes around the
factory picking up the right pieces, and slotting them together to
make a functioning TiddlyWiki car. The trouble is that Ruby isn't as
good at making TiddlyWiki cars as TiddlyWiki itself; Ruby makes
mistakes. So, we can't sell the cars that Ruby produces directly.
Instead, as each car rolls out of the factory gates, we manually lean
in and press the button marked "regenerate". The TiddlyWiki car
promptly rebuilds itself perfectly, without the defects that Ruby
introduced.
So, what you've got there is something awesome (a self building car!)
that is dependent on something awful (a factory that makes dodgy
cars!). The answer, obviously, is to improve the factory so that it
can create defect-free TiddlyWiki cars.
Out goes Ruby, to be replaced by a new robot called Node, which does a
much better job. To make sure that it makes cars just as good as the
ones that TiddlyWiki makes itself, Node was built out of pieces taken
from the TiddlyWiki car with lots of polishing and refurbishment.
Hurray, finally, we've got a factory that can reliably turn out the
tried and trusted TiddlyWiki car.
But now that we've got the new cars whizzing out of the factory, we
see that actually the TiddlyWiki car isn't as wonderful as it once
was. The design is showing its age, and it doesn't seem to be as well
adapted to modern roads. And modern mechanics take one look at it and
deride its antique internals.
Sticking with the tortured metaphor, the breakthrough that has me
excited is that I've (in an experimental way) got this new Node robot
to produce a shiny new model 5 of the TiddlyWiki car. The new car
isn't built from the components of the classic TiddlyWiki car, but
rather Node pulls its own components out of it's own body to construct
TiddlyWiki5. So, in fact, there is no difference between TiddlyWiki5
and Node; the car has become its own factory robot.
----
Abandoning the metaphorical...
The problem I set out to tackle first was to replace TiddlyWiki's
build tools. Originally, back in 2004, I wrote TiddlyWiki as one big
file. It became increasingly hard to manage it as the product grew and
so by the time we set up the first official source code repository on
Trac, we decided to split the source code up into separate files, and
a tool called cook.rb was created in Ruby which could stitch the
individual files back together into a functioning TiddlyWiki.
In this respect, cook.rb was like TiddlyWiki: it was a blob of code
that could generate TiddlyWiki HTML files. However, cook.rb didn't
have all the capabilities of TiddlyWiki (such as the ability to wikify
text), and so has never been able to precisely duplicate the files
that TiddlyWiki itself generates. So, with cook.rb the build process
for TiddlyWiki was convoluted: use cook.rb to stitch the files
together into a not-quite-correct TiddlyWiki file, and then load that
TiddlyWiki up in Firefox and click 'save changes' to generate a true
TiddlyWiki.
The awkwardness and fragility of this process makes it harder than
necessary to work on the development of TiddlyWiki itself. So, I set
out to create a replacement tool that I called cook.js [2]. The idea
was that it would run on the command line under node.js, and do the
same things that cook.rb used to do, but, crucially, also to
incorporate code from TiddlyWiki itself so that it could, for example,
wikify text.
So, the first phase of my work was to analyse the existing cook.rb
tool, and update its documentation [2]. I then built up new JavaScript
code to replicate the key bits of functionality of the old tools - you
can see the code at [3].
To begin with, the new code was about understanding the `.recipe`
files used by cook.rb, and figuring out how to load the required
files, and if necessary massage them as they get spliced into the main
template. By the end of November, I had code that could do all the
basic file stitching, and did in fact create a functioning TiddlyWiki.
At that point, the outstanding issues preventing cook.js from doing
everything that TiddlyWiki could do were all around wikifying. For
example, TiddlyWiki updates the <title> tag of the HTML page according
to the plain text rendered content of the WindowTitle shadow tiddler,
which is normally defined as "<<tiddler SiteTitle>> - <<tiddler
SiteSubtitle". So, for cook.js to create the correct text for the
<title> tag, it would have to be able to convert raw wiki text into
HTML, and handle transclusion and other macros. Similarly, TiddlyWiki
can generate an RSS file that includes HTML representations of recent
tiddlers.
So, at the beginning of December I took a deep breath and
transliterated the existing TiddlyWiki wikifier code into cook.js. In
the process, I massively refactored it, and structured it much more
sensibly. Instead of using jQuery and the DOM, it now parses the
wikitext into a generic tree format, and then renders that into
textual HTML.
Thus, a week ago, I got to the point where cook.js contained a fully
functioning wikifier (albeit currently implementing only a handful of
macros).
I spent some time refining the command line interface so that it was
easy to use it to perform a bunch of tiddler and TiddlyWiki
operations:
* Splitting tiddlywiki files into tiddlers
* Constructing tiddlywiki files from recipes
* Dynamically serving tiddlywiki files over http
* Dynamically serving individual tiddlers in HTML over http
There's room to add more, too: for instance, it could read a
TiddlyWiki file and then generate a directory with a wikified .HTML
file for each tiddler, all cross linking to each other. The result
would be an entirely static, but browsable version of the content from
a TiddlyWiki. And a little more polishing would let it be used in
TiddlySpace, for improved server-side wikification support.
But, yesterday I tried an experiment that I think is even more
exciting: to take all the new code in cook.js for handling tiddlers
and wikifying, and make it run in the browser. In effect, cook.js has
now reached the point where it can do (almost) everything that
TiddlyWiki can do - and so, the new code can completely supplant the
old.
For the moment, I've reused the name TiddlyWiki5 for this new school
TiddlyWiki. What I did was to build up a recipe that referenced the
new JavaScript modules that I had built for cook.js, and pack them
into an HTML file, along with some tiddlers and some boot code.
Node.js modules don't work directly in a browser, so I had to write a
little bootloader that replicates enough of the node.js module
environment. I had to add a bit of code to navigate to tiddlers when
links are clicked. I barely touched the main cook.js code itself, but
the thing worked pretty much first time.
So, what we've got is a primordial TiddlyWiki5, built by a version of
itself running on the server. You can see it at [4]. If you click on a
tiddler link then the new tiddler is shown at the bottom of the page.
If you click on an external page then it opens in a new window. That's
it; there's no title shown, no styling, and of course no editing
capabilities. All of that is to come.
The reason why this is a big deal is that it brings all the TiddlyWiki
developments together. We've got a build tool that can uniformly build
old and new TiddlyWikis. We've got a single wikifier code base that
can be used on the server, in the browser and as a command line tool.
So, we can efficiently maintain the old 2.x codebase while charging
ahead with the new code.
The new code will have quite an impact on the capabilities of
TiddlyWiki. For example, it now formally separates parsing wiki text
into a structured tree from rendering it into HTML. The structured
tree representation will allow TiddlyWiki to finally properly
implement features like sections and slices, orphans and missing
links.
I'm very excited about TiddlyWiki5. I've been thinking about
TiddlyWiki for a long time now, and have many ideas built up. Today's
breakthrough isn't really visible to end users, which is why I've
tried to explain it here. But hopefully you can see that this train is
starting to pick up steam and get moving.
If you want to find out more, dive into the code at [3] - there's a
long readme file. To make the code work, you'll need to install
node.js, but happily that is very easy these days - see
http://nodejs.org/ for instructions.
For the most part, I shall continue to post development updates to the
TWDev group. I'm hoping to get quite a lot done over the next few
weeks, and will post in this group when I've got something that's
worth having a look at.
As ever, I'll do my best to answer any questions,
Best wishes
Jeremy
[1] http://groups.google.com/group/tiddlywiki/browse_thread/thread/7ea26be3f6b76698/38e167b0e73b6c5a
[2] https://github.com/TiddlyWiki/cooker
[3] https://github.com/Jermolene/cook.js
[4] http://jermolene.github.com/tiddlywiki5/index.html
Jarrod
said "this train is starting to pick up steam and get moving". I'm
looking forward to watching and eventually using the new developments
that your cooking up. Keep up the good work.
Jarrod
perlguy
node.js (or any js runtime/server) with tiddlywiki, the synchronicity
would be scary.
I've been mulling over (for a couple of months) the idea of using
node.js to implement a store-and-forward message queue to use with TW,
for many of the same reasons you selected it, I'm sure. Mainly I
wanted to be able to use the same code/modules in the node.js server
as I would in the tiddlywiki plugin that would send/receive the
messages.
node.js runs on the 2 main O/Ses I care about (Linux and XP), and
javascript is now an integral part of the Gnome desktop. That makes a
pretty level playing field for better integration of TW based apps
with my personal desktop, and still provding for a relatively simple
connection path from my desktop/TW (F15/Gnome) talk to my laptop/TW
(XP).
My initial goal for a node.js mq process is for a tiddlysnip
alternative, that can run outside the browser, and is platform
agnostic - as far as node.js is, at least. I watched the discussion
over formalizing standard tiddler fields and content with hope and
fear... hope that it would lead to interesting new developments, and
fear that the core tiddler might get bogged down with too much baggage
(the latter was a pretty remote possibility, I know, if you project
forward based on TW history so far). I'm glad the status quo appears
to be unchanged, leaving the core tiddler untouched, and letting
(requiring?) anyone implementing sync or external stores to add on
their own metadata to facilitate passing tiddlers back and forth. I
think that's one of the major strengths TW has - it's extremely open.
and simple, at it's core. Only the thing that must exist to make the
core work, are in the core.
Anyway, I'm really excited to hear about the progress you've made -
and I can tell you're pretty excited about it too - I totally get it.
Hopefully, I can find/make some time to actually put some code
together for my node-mq project... so far, I've only had time to
research existing modules that I might be able to press into service,
that work under node.js AND brower runtimes (I figure if both those
work - integrating to the desktop - directly via desktop extension, or
via standalone applet - should be a snap).
The structured format for the intermediate tiddler state (parsed, but
not yet rendered) would likely make tiddler exchange (peer-2-peer,
client/server, etc) simpler and more portable.
This is a more momentous anncounment - to me anyway - than when you
made the *first* TW5 available for preview. I can't wait to see what's
next!
Sounds like it's time for me to sub to the TWDev group... even if it's
just to listen in.
Kudos all around.
--
Jim
cdent
> So, at the beginning of December I took a deep breath and
> transliterated the existing TiddlyWiki wikifier code into cook.js. In
> the process, I massively refactored it, and structured it much more
> sensibly. Instead of using jQuery and the DOM, it now parses the
> wikitext into a generic tree format, and then renders that into
> textual HTML.
\o/ +1
> The reason why this is a big deal is that it brings all the TiddlyWiki
> developments together. We've got a build tool that can uniformly build
> old and new TiddlyWikis. We've got a single wikifier code base that
> can be used on the server, in the browser and as a command line tool.
Very curious to see the wikifier in action. The twikifier code works
relatively well, but is not ideal, and is fairly inscrutable.
Jeremy Ruston
> node.js to implement a store-and-forward message queue to use with TW,
> for many of the same reasons you selected it, I'm sure. Mainly I
> wanted to be able to use the same code/modules in the node.js server
> as I would in the tiddlywiki plugin that would send/receive the
> messages.
Cool, that sounds like a good plugin for TW5. I plan to add plugin
support pretty soon so that we can package up the TiddlySpace specific
extensions like tiddler@user and @user.
> node.js runs on the 2 main O/Ses I care about (Linux and XP), and
> javascript is now an integral part of the Gnome desktop. That makes a
> pretty level playing field for better integration of TW based apps
> with my personal desktop, and still provding for a relatively simple
> connection path from my desktop/TW (F15/Gnome) talk to my laptop/TW
> (XP).
Yes indeed. I've been reading that there is now deep support for
node.js on Windows, which is pretty cool from the point of view of
TiddlyWiki5 going mainstream.
> My initial goal for a node.js mq process is for a tiddlysnip
> alternative, that can run outside the browser, and is platform
> agnostic - as far as node.js is, at least.
Got you. I've been looking at browser extension APIs as I think it
would make sense for TiddlyWiki 5 to have the option of building a
browser extension. One could imagine an entirely internal
implementation that uses the browsers SQL engine for storage, but I
prefer the model of having a separate node.js instance running a local
web server that the browser extension talks to. That sounds a bit
complicated for end users, but I think we'll see the availability of
tools that package node.js applications into executables for Windows
and OS X (I think I've seen such a packager, but I can't find it now).
> Anyway, I'm really excited to hear about the progress you've made -
> and I can tell you're pretty excited about it too - I totally get it.
> Hopefully, I can find/make some time to actually put some code
> together for my node-mq project... so far, I've only had time to
> research existing modules that I might be able to press into service,
> that work under node.js AND brower runtimes (I figure if both those
> work - integrating to the desktop - directly via desktop extension, or
> via standalone applet - should be a snap).
Yup, I'm finding that lots of modules that I'm interested in are
already built and tested both for node and the browser (for example,
peg.js, which I'm planning to use for parsing CSS and so on).
> This is a more momentous anncounment - to me anyway - than when you
> made the *first* TW5 available for preview. I can't wait to see what's
> next!
Thank you, much appreciated.
> Sounds like it's time for me to sub to the TWDev group... even if it's
> just to listen in.
Keep an eye on my github account, too: https://github.com/Jermolene
Cheers
jeremy
> Kudos all around.
>
> --
> Jim
>
> --
> You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
> To post to this group, send email to tiddl...@googlegroups.com.
> To unsubscribe from this group, send email to tiddlywiki+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/tiddlywiki?hl=en.
>
Ben Gillies
> transliterated the existing TiddlyWiki wikifier code into cook.js. In
> the process, I massively refactored it, and structured it much more
> sensibly. Instead of using jQuery and the DOM, it now parses the
> wikitext into a generic tree format, and then renders that into
> textual HTML.
+1. This is something I think we've all wanted for quite a long time.
Is there a description of the tree format anywhere (I couldn't find
one in the repo)? How are macros intended to fit into this tree?
(These may be questions better asked in a twdev thread instead).
> Thus, a week ago, I got to the point where cook.js contained a fully
> functioning wikifier (albeit currently implementing only a handful of
> macros).
Does this mean that the new cook is sufficiently complete to
officially replace the old one?
Question: Would it be possible to use the new cook.js as a more
generic (i.e. non TiddlyWiki specific) build tool, in the same vein as
the RequireJS optimizer (but less RequireJS dependent)?
Jeremy Ruston
>> So, at the beginning of December I took a deep breath and
>> transliterated the existing TiddlyWiki wikifier code into cook.js. In
>> the process, I massively refactored it, and structured it much more
>> sensibly. Instead of using jQuery and the DOM, it now parses the
>> wikitext into a generic tree format, and then renders that into
>> textual HTML.
>
> \o/ +1
It was surprisingly easy to get the basic transliteration up and
running, and of course very satisfying to thrash it into some sort of
respectable shape. I'm very excited about the way that it makes lots
of things easy and elegant that were frightful and tacky in the old
code.
>> The reason why this is a big deal is that it brings all the TiddlyWiki
>> developments together. We've got a build tool that can uniformly build
>> old and new TiddlyWikis. We've got a single wikifier code base that
>> can be used on the server, in the browser and as a command line tool.
>
> Very curious to see the wikifier in action. The twikifier code works
> relatively well, but is not ideal, and is fairly inscrutable.
The new wikifier is currently synchronous, but I'm planning to convert
it to synchronous operation, which I think might simplify things on
your side a bit.
One of my next steps is to rewrite the wikifier as a peg.js grammar,
developing a revised TiddlyWiki wiki format that is largely backwards
compatible but handles paragraphs sensible, and brings in the best of
markdown (I'm very used to writing `code like this` now, for
instance).
Cheers
Jeremy
Ben Gillies
>> javascript is now an integral part of the Gnome desktop. That makes a
>> pretty level playing field for better integration of TW based apps
>> with my personal desktop, and still provding for a relatively simple
>> connection path from my desktop/TW (F15/Gnome) talk to my laptop/TW
>> (XP).
>
> Yes indeed. I've been reading that there is now deep support for
> node.js on Windows, which is pretty cool from the point of view of
> TiddlyWiki5 going mainstream.
I'm not sure it's relevant, but there's now a few SSJS engines
knocking around (e.g. SilkJS http://silkjs.org/)
> Yup, I'm finding that lots of modules that I'm interested in are
> already built and tested both for node and the browser (for example,
> peg.js, which I'm planning to use for parsing CSS and so on).
Have you seen language.js as well
(https://github.com/tolmasky/language)? It has similar syntax to
peg.js, but seems somewhat better at error handling.
Jeremy Ruston
> Is there a description of the tree format anywhere (I couldn't find
> one in the repo)? How are macros intended to fit into this tree?
> (These may be questions better asked in a twdev thread instead).
The tree format is documented at the top of WikiTextParser.js:
https://github.com/Jermolene/cook.js/blob/master/js/WikiTextParser.js
I need to add an update about macros, which are just modelled as a node:
{type:"macro",name: "tiddler", params: "Tiddler2 with:Hello"}
When a macro is rendered it generates a sub-parse tree in the "output" member.
(Parenthetically, I'm planning to resolve the tension between docs in
the source files and in the readme.md by auto generating the readme
from the code).
>> Thus, a week ago, I got to the point where cook.js contained a fully
>> functioning wikifier (albeit currently implementing only a handful of
>> macros).
>
> Does this mean that the new cook is sufficiently complete to
> officially replace the old one?
Not quite. I haven't got round to adding the renderers for most of the
macros. I did the "tiddler" macro first because obviously it's the one
that has deep structural implications.
> Question: Would it be possible to use the new cook.js as a more
> generic (i.e. non TiddlyWiki specific) build tool, in the same vein as
> the RequireJS optimizer (but less RequireJS dependent)?
Yes, I think so. Right now the recipe file format is still a bit
inadequate, and tied up to TiddlyWiki concepts.
Cheers
Jeremy
Jeremy Ruston
> knocking around (e.g. SilkJS http://silkjs.org/)
Interesting. My first reaction is not to like the JST file idea, but
I'll dig around a bit.
>> Yup, I'm finding that lots of modules that I'm interested in are
>> already built and tested both for node and the browser (for example,
>> peg.js, which I'm planning to use for parsing CSS and so on).
>
> Have you seen language.js as well
> (https://github.com/tolmasky/language)? It has similar syntax to
> peg.js, but seems somewhat better at error handling.
Brilliant, I hadn't seen that, very interesting, thanks.
Best wishes
Jeremy
Ben Gillies
>> +1. This is something I think we've all wanted for quite a long time.
>> Is there a description of the tree format anywhere (I couldn't find
>> one in the repo)? How are macros intended to fit into this tree?
>> (These may be questions better asked in a twdev thread instead).
>
> The tree format is documented at the top of WikiTextParser.js:
>
> https://github.com/Jermolene/cook.js/blob/master/js/WikiTextParser.js
Right, sorry. Must have missed that.
>> Question: Would it be possible to use the new cook.js as a more
>> generic (i.e. non TiddlyWiki specific) build tool, in the same vein as
>> the RequireJS optimizer (but less RequireJS dependent)?
>
> Yes, I think so. Right now the recipe file format is still a bit
> inadequate, and tied up to TiddlyWiki concepts.
Cool. Being able to do something like:
node tiddlywiki.js *.js *.html *.css > spa.html
or whatever would be really useful.
Jeremy Ruston
>
> node tiddlywiki.js *.js *.html *.css > spa.html
>
> or whatever would be really useful.
Yes, that's very nice. I do intend to add wildcard support. Earlier
versions of the code worked by writing to stdout, which does integrate
with the shell wonderfully. But of course it's less useful when you're
generating a bundle of files, as with cooking TiddlyWikis.
Cheers
Jeremy
Ben Gillies
>> Cool. Being able to do something like:
>>
>> node tiddlywiki.js *.js *.html *.css > spa.html
>>
>> or whatever would be really useful.
>
> Yes, that's very nice. I do intend to add wildcard support. Earlier
In Bash at least, the wildcards will complete before they reach your
script so you don't need to do anything. Of course, that says nothing
about Windows.
> versions of the code worked by writing to stdout, which does integrate
> with the shell wonderfully. But of course it's less useful when you're
> generating a bundle of files, as with cooking TiddlyWikis.
Indeed.
PMario
> (Parenthetically, I'm planning to resolve the tension between docs in
> the source files and in the readme.md by auto generating the readme
> from the code).
Hi Jeremy,
Nice work !!
Have you seen dox [1] project. It can handle markdown and JSDoc style
tags. The big difference to other doc generators is, that it produces
JSON output, that can be rendered by any templating engine.
-m
Jeremy Ruston
> Nice work !!
>
> Have you seen dox [1] project. It can handle markdown and JSDoc style
> tags. The big difference to other doc generators is, that it produces
> JSON output, that can be rendered by any templating engine.
Thank you, I hadn't seen that. I'm intending to use a full JavaScript
parser for javascript tiddlers, and hoping that it will integrate with
any of the TextProcessors.
Cheers
Jeremy
> [1] https://github.com/visionmedia/dox
HansBKK
> One of my next steps is to rewrite the wikifier as a peg.js grammar,
> developing a revised TiddlyWiki wiki format that is largely backwards
> compatible but handles paragraphs sensible, and brings in the best of
> markdown (I'm very used to writing `code like this` now, for
> instance).
I've been doing a lot of research and playing around in the "tools
that transform markup" space recently, and would **implore** you to
consider choosing one of the more mainstream cross-platform syntaxes
rather than re-implementing a proprietary TWmarkup, even if it may be
"based on" one of them.
Markdown is a popular choice, lots of active development in the
Pandocs project - their "extensions" may be a bit proprietary, but
since the focus of the whole project is interoperability, it's for a
good cause.
Other candidates are txt2tags and reST/Sphinx, with the former having
the edge in a huge number of output formats currently supported. Both
of these are implemented in Python, while Pandocs is Haskell, if that
means anything.
IMO the key is to **not** pick and choose bits and pieces, but to take
on an entire **syntax spec as a whole**.
This would enable plugging TW into a standardized toolchain so that it
can be either a "publishing and distribution target", along with say
EPUB, AsciiDoc and HTMLhelp, or perhaps even the location where the
"master source" text is edited, to then be able to output and
transform to such formats. NB I consider TW's forte to be the former
rather than the latter, but something like Pandocs would give the
flexibility to go in either direction.
But please **please** don't just throw another proprietary wikiMarkup
syntax set into the mix, this is an historic opportunity to contribute
to the idea of "open data" in the larger ecosystem rather than
continuing to view tiddlers as an isolated silo needing custom coding
to extract inline markup semantics.
Thanks in advance for at least considering these ideas. . .
/rant
Jeremy Ruston
> that transform markup" space recently, and would **implore** you to
> consider choosing one of the more mainstream cross-platform syntaxes
> rather than re-implementing a proprietary TWmarkup, even if it may be
> "based on" one of them.
The primary constraint I am embracing is to retain compatibility as
far as possible with existing TiddlyWIki markup. I think that that
rules out the wholesale adoption of a standard format - for instance,
TiddlyWiki gets [[pretty|links]] the wrong way round from most wikis.
To get around the limitations imposed by that constraint, we have the
idea of pluggable parsers and renderers, so that it is possible to
adopt other formats, and intermingle them and so on. If you can find a
JavaScript parser for it, then hopefully you'll be able to use it.
> Markdown is a popular choice, lots of active development in the
> Pandocs project - their "extensions" may be a bit proprietary, but
> since the focus of the whole project is interoperability, it's for a
> good cause.
Adopting MarkDown in its entirety is a bit troublesome from my
perspective; it's not very formally defined, and the only
implementations that I've seen are even hairier balls of regexps than
the old TiddlyWIki wikifier. It also lacks what I'd have thought of as
basic features, like tables, arguing that users should write HTML tags
for them.
> Other candidates are txt2tags and reST/Sphinx, with the former having
> the edge in a huge number of output formats currently supported. Both
> of these are implemented in Python, while Pandocs is Haskell, if that
> means anything.
I need JavaScript code, obviously. There may be bits and pieces worth
taking from those projects, but a brief glance shows that they take in
concerns that don't entirely match TiddlyWiki, so I don't see a way to
wholesale adopt those syntaxes.
> IMO the key is to **not** pick and choose bits and pieces, but to take
> on an entire **syntax spec as a whole**.
I only see that as feasible if there was a spec out there that was (a)
compatible with TiddlyWiki and (b) did everything that TiddlyWiki
needs and (c) didn't include lots of features that aren't relevant for
TiddlyWiki and (d) had usable JavaScript code.
> This would enable plugging TW into a standardized toolchain so that it
> can be either a "publishing and distribution target", along with say
> EPUB, AsciiDoc and HTMLhelp, or perhaps even the location where the
> "master source" text is edited, to then be able to output and
> transform to such formats. NB I consider TW's forte to be the former
> rather than the latter, but something like Pandocs would give the
> flexibility to go in either direction.
As I say, the ability to have pluggable parsers and renderers should
give you the capabilities you want. For you, the native TiddlyWiki
format might just be what gets used for the application plumbing, with
all your content being in other formats.
> But please **please** don't just throw another proprietary wikiMarkup
> syntax set into the mix, this is an historic opportunity to contribute
> to the idea of "open data" in the larger ecosystem rather than
> continuing to view tiddlers as an isolated silo needing custom coding
> to extract inline markup semantics.
I'm not planning to throw another proprietary wiki markup into the
mix. I'm planning to improve TiddlyWiki's existing markup so that it
isn't broken with respect to paragraphs, and to introduce some
alternative syntaxes for some simple formatting.
> Thanks in advance for at least considering these ideas. . .
As I say, I think the kinds of things you're interested in are enabled
by the parsing/rendering architecture. I would very much appreciate
feedback on the improvements to the wikitext.
Best wishes
Jeremy
> /rant
Martin Budden
TW 5. I differ from in that I think you should adopt WikiCreole see:
http://www.wikicreole.org/wiki/Creole1.0
and
http://www.wikicreole.org/wiki/CreoleAdditions
The main reason for this recommendation is the WikiCreole markup is
*very* similar to TW markup. The main differences being the handling
of bold, headings and links. WikiCreole even recommends using double
angle brackets for plugins.
Since TW 5 is going to break compatibility, then I think it should
also take the opportunity to move to a standard wiki format. I know
text compatibility and code compatibility are different things, but
nevertheless I believe this is the correct way forward.
Note TW 5 could support standard TW wikiformat using a plugin, and
also could provide a converter so that people can easily move their
text from TW format to WikiCreole format. Since the formats are so
similar, conversion would be fairly straightforward.
Martin
Julio
' hope everyone is having a good day.
Although much of this thread goes over my head, I am glad that Jeremy
is again working on Tiddliwiki. As a regular Joe, however, I would
like to state that if indeed one were to go with a syntax change I
would concur with Martin above. I would definitely be able to adapt to
the Creole syntax for the same reason he stated...it's very similar to
Tiddlywiki.
I do have a question:
* If indeed this were to be the case, can one adapt Creole yet keep
some syntax that would be "unique" to Tiddlywiki without the use of
plugins or converters?
* Or would that be too much work for development? Just trying to
educate myself here.
This probably should be discussion for another thread.
I thought I'd just throw the question out there while it's still fresh
in my mind.
Thanks all,
Julio
On Dec 15, 6:15 am, Martin Budden <mjbud...@gmail.com> wrote:
> I concur with HansBKK that you should adopt a standard wiki markup for
> TW 5. I differ from in that I think you should adopt WikiCreole see:
>
> http://www.wikicreole.org/wiki/Creole1.0
> andhttp://www.wikicreole.org/wiki/CreoleAdditions
>
> The main reason for this recommendation is the WikiCreole markup is
> *very* similar to TW markup. The main differences being the handling
> of bold, headings and links. WikiCreole even recommends using double
> angle brackets for plugins.
>
> Since TW 5 is going to break compatibility, then I think it should
> also take the opportunity to move to a standard wiki format. I know
> text compatibility and code compatibility are different things, but
> nevertheless I believe this is the correct way forward.
>
> Note TW 5 could support standard TW wikiformat using a plugin, and
> also could provide a converter so that people can easily move their
> text from TW format to WikiCreole format. Since the formats are so
> similar, conversion would be fairly straightforward.
>
> Martin
>
> >> For more options, visit this group athttp://groups.google.com/group/tiddlywiki?hl=en.
PMario
I did re/search a bit for nicely documented wikitext grammers lately.
I only found one for wiki creole. The good thing is, that it imo is a
good "subset" of the TW grammar.
There is a short article from:
> Authors: Martin Junghans, Dirk Riehle, Rama Gurram, Matthias Kaiser, Mário Lopes, Umit Yalcinalp
at: http://dirkriehle.com/2008/07/19/a-grammar-for-standardized-wiki-markup/
that links to: (see "available as a PDF file")
http://dirkriehle.com/wp-content/uploads/2008/07/ws2008-wiki-creole-grammar-final-for-web.pdf
which talks about there intentions.
=====
Searching Dirk Riehle's site a bit more I found:
http://dirkriehle.com/publications/2008-2/wiki-creole/
and
http://dirkriehle.com/wp-content/uploads/2008/01/creole10_with_extension.g
Which is a full EBNF grammar description for wiki Creole. (using some
ANTLR specific commands).
Since the source (see: creole10_with_extension.g link) is copyrighted,
I did contact Mr. Riehle to get a less restrictive license (just to be
sure). Since the creole project is CreativeCommons, I did get an
emailed permission to use the most open CC-BY - http://creativecommons.org/licenses/by/3.0/
license to use with TW.
@Jeremy
Just linking some resources, that may be usefull.
-m
David Gifford
comes along, to see whether you all are going to change the wikitext
to some other format...
Kudos to you, Jeremy, for taking time to fiddle with the tiddlers
again. I will be looking forward to what comes of the cooks and nodes.
Dave
Jeremy Ruston
plan is that it should be as compatible as possible with existing
TiddlyWiki wikitext. I do want to take the opportunity to extend the
format, though, which I think can mostly be done without breaking
backwards compatibility.
WikiCreole is interesting. I hadn't seen the amendments section. I'd
be interested in trying to make TiddlyWiki5's format compatible with
WikiCreole, but only if it can be done without substantially impacting
existing texts.
The primary area where backwards compatibility is a concern is the
handling of paragraphs. TiddlyWiki's use of <br> tags is just plain
wrong, and I think it's worth taking a minor compatibility hit in
order to gain the advantage of producing semantically proper HTML,
which is generally much more useful.
The reason that I'm so keen on backwards compatibility is that there
is a lot of text out there in TiddlyWiki format; much of it on
people's hard drives. I like the idea of enhancing the value of that
existing investment by making it super easy for existing users to take
advantage of the new features of the new software.
Best wishes
Jeremy
Ben Gillies
> comes along, to see whether you all are going to change the wikitext
> to some other format...
I don't see why we can't support both a new and an old wikitext syntax
(along with others), as Jeremy suggests.
David Gifford
Dave
HansBKK
On Friday, December 16, 2011 9:55:25 PM UTC+7, Jeremy Ruston wrote:
WikiCreole is interesting. I hadn't seen the amendments section. I'd be interested in trying to make TiddlyWiki5's format compatible with WikiCreole, but only if it can be done without substantially impacting existing texts.
IMO **Any** standardized syntax would be better than keeping TW's unique. That way at least there's a way to get plugged into a toolchain down the road without having to write a specific converter template/parser for TW.
Converting to or from the "standardX" markup used by TW is then more than likely to be supported by converter/parser engines like Pandocs and docutils as they continue to evolve, or at least if a new "tangler/template" whatever needs to be written for it that's useful to the whole ecosystem, not just the one community, and therefore more likely to be supported and maintained over time.
The primary area where backwards compatibility is a concern is the handling of paragraphs. TiddlyWiki's use of <br> tags is just plain wrong, and I think it's worth taking a minor compatibility hit in order to gain the advantage of producing semantically proper HTML, which is generally much more useful.
The reason that I'm so keen on backwards compatibility is that there is a lot of text out there in TiddlyWiki format; much of it on
people's hard drives. I like the idea of enhancing the value of that existing investment by making it super easy for existing users to take advantage of the new features of the new software.
If you're fixing the paragraph syntax you'll have to do something like that to accomplish the above goal anyway.
On Thursday, December 15, 2011 4:22:51 PM UTC+7, Jeremy Ruston wrote:
> I've been doing a lot of research and playing around in the "tools that transform markup" space recently, and would **implore** you to consider choosing one of the more mainstream cross-platform syntaxes rather than re-implementing a proprietary TWmarkup, even if it may be "based on" one of them.
Sorry for the use of the word "proprietary", very rude in the FOSS world I know - how about "unique" syntax?
the idea of pluggable parsers and renderers, so that it is possible to
adopt other formats, and intermingle them and so on. If you can find a
JavaScript parser for it, then hopefully you'll be able to use it.
>> Adopting MarkDown in its entirety is a bit troublesome from my perspective; it's not very formally defined
I need JavaScript code, obviously. There may be bits and pieces worth taking from those projects, but a brief glance shows that they take in concerns that don't entirely match TiddlyWiki, so I don't see a way to wholesale adopt those syntaxes.
The whole point I'm getting at is to be able to keep our data in a canonical format as plain text, for transfer to whatever "container", files in folders on whatever storage medium, database, CMS, wiki, and easily get it out there as EPUB, static HTML, DocBook whatever. Obviously there isn't currently such a beast with parsers/converters/renderers etc implemented in every mainstream language, or everyone would be using it. I'm just doing my bit to advocate that one of the cool tools I use help move in that desirable direction.
If TiddlyWiki can help facilitate that goal it's that much more valuable as one of those platform choices, whether used as a source-file editor or target publishing format for output from other parts of the toolchain.
> IMO the key is to **not** pick and choose bits and pieces, but to take on an entire **syntax spec as a whole**.
As I say, the ability to have pluggable parsers and renderers should give you the capabilities you want. For you, the native TiddlyWiki format might just be what gets used for the application plumbing, with all your content being in other formats.
What TW needs internally is a black box AFAIC, although having a simple intra-file link, even if only to tiddlerTitle would be a bonus.
Sorry to keep ranting on, I'll try to have this be 'nuff said. . .
rakugo
I'd assume that like existing TiddlyWiki one could swap in/out
syntaxes (but probably even more easily) so I could imagine supporting
a plugin to help with this transition.
I would assume it would be easy to convert from one format to another,
since TiddlyWiki markup can be converted to html then from that into
WikiCreole. I could imagine a transition tool that takes an old
TiddlyWiki and converts it into a new TiddlyWiki.
Martin Budden
TiddlyWiki5's format compatible with WikiCreole, but only if it can be
done without substantially impacting existing texts."
I've started a new discussion on this topic. I think it is certainly
possible to make TW more compatible with WikiCreole.
Martin
HansBKK
http://wiki.txt2tags.org/index.php/Main/TiddlywikiPlugin
cdent
> > The reason why this is a big deal is that it brings all the TiddlyWiki
> > developments together. We've got a build tool that can uniformly build
> > old and new TiddlyWikis. We've got a single wikifier code base that
> > can be used on the server, in the browser and as a command line tool.
>
> Very curious to see the wikifier in action. The twikifier code works
> relatively well, but is not ideal, and is fairly inscrutable.
in a standalone kind of way? If so where and how?
The twikifier code used with TiddlySpace is proving to be a bit heavy
(cpu-use-wise) and it would be useful to experiment with something a
bit more modern and hopefully more responsive to tweaking, inspection
and improvement.
Thanks.
Jeremy Ruston
> in a standalone kind of way? If so where and how?
With some caveats I think this is probably a good time to start
investigating. I think the code is capable of what you need, but it's
still rather in flux, and so can be changed in response to your
requirements.
The code is still in the wrong repo:
https://github.com/Jermolene/cook.js
The main issues that might affect the decision about timing adoption:
- The new code doesn't currently implement all of the core TiddlyWiki
macros. I've focussed on the structural ones (like view, story, and
tiddler) so far. I don't see any particular obstacles to hoiking the
remainder in.
- The new code doesn't yet properly sanitise HTML and JavaScript
content within tiddlers. In the case of JavaScript, it gets halfway
there: evaluated macro parameters are parsed and recompiled, giving an
opportunity to check for e.g. access to dangerous globals, but this
doesn't happen at the moment
- There is not yet support for sections and slices. This is next on my
list, and I think probably necessary
- There's probably other limitations; I've been focussing on getting a
solid architecture for the representation transformation engine at the
heart of TiddlyWiki
- Error handling is probably a bit inadequate
> The twikifier code used with TiddlySpace is proving to be a bit heavy
> (cpu-use-wise) and it would be useful to experiment with something a
> bit more modern and hopefully more responsive to tweaking, inspection
> and improvement.
Throughout the development I've kept in mind your comment last year
that you'd like wikification to be exposed as an asynchronous
function, presumably with a signature something like this:
store.wikify(tiddler,callback)
And then have the callback invoked with an error code and the HTML
representation of the tiddler.
It turns out that that style interface isn't ideal for the client
code. In the browser, the asynchronicity is around async events
triggering changes to the values of tiddlers which trigger highly
selective changes to the DOM. The actual wikification process is
synchronous; in the client if we're waiting for a tiddler to load from
the server we'd generally want attempts to transclude that tiddler to
succeed, but to render a spinner. The eventual arrival of the tiddler
automatically triggers the DOM processing to refresh the tiddler that
did the transcluding.
So, I've tried to arrive at an implementation that supports both
approaches. One of the hallmarks of the design is the aggressive use
of caching to support efficient selective updating of the DOM. I think
a big area of discussion around this integration is how we can ensure
that our respective caching strategies work well together.
Using the existing implementation, an asynchronous function to wikify
a tiddler goes like this:
- Initialise a WikiStore object
- Asynchronously load the tiddler into the wikistore
- When it's loaded, parse the tiddler to obtain a list of its
dependencies (this is a synchronous operation)
- Loop back to recursively load the dependent tiddlers
- Call the store.renderTiddler() function to obtain an HTML rendering
of the tiddler, which is also a synchronous operation
The WikiStore object is the equivalent of the TiddlyWiki() object in
the old code; it's a collection of tiddlers. The Tiddler object is
intentionally simple: just an immutable collection of fields. The
WikiStore also needs be configured with the parsers and macros that
you want to use. I'm currently using a module called App.js to
centralise the WikiStore configuration steps for both browser and
node.js configurations of the code:
https://github.com/Jermolene/cook.js/blob/master/js/App.js
You'd basically need to do everything that isn't in one of the
if(isBrowser) conditionals.
Loading a new tiddler into the store is easy:
store.addTiddler(new Tiddler({title: "HelloThere", text: "Some text",
type: "text/x-tiddlywiki"}));
Getting the list of dependencies for a tiddler is:
store.parseTiddler(title).dependencies
It will either be an array of tiddler title strings, or the special
value null. This means that the tiddler is dependent on all the other
tiddlers it can get it's hands one - for example, using <<list all>>
within a tiddler will set the dependencies to null. Another issue we
might need to look at is that there is currently no distinction
between being dependent on a tiddler because we transclude it, or just
because we link to it. You might prefer to avoid loading tiddlers that
are only linked.
Rendering the tiddler in HTML is:
store.renderTiddler("text/html",title);
You can also render the specified tiddler as though it were another
one; this is useful for rendering tiddlers as templates. For example:
store.renderTiddler("text/html","MyTemplate","HelloThere");
This example would render the tiddler HelloThere using the tiddler
MyTemplate as a template. So things like <<view title>> within
MyTemplate would render the tiddler HelloThere.
That's the basic outline. There's a lot more to be said, particularly
about caching, but it should let you rootle around in the code a bit
more easily.
Best wishes
Jeremy
dave
javascript code, that contains a mess of regex's to implement
txt2tags. There are a few gaps in the syntax, and it borrows some
tiddlywiki syntax (I copied the <<macro>> syntax as txt2tags has no
concept of that feature). I've coincidentally been fixing some bugs
and am posting an update over the weekend.
HansBKK
Yeah, that implements a txt2tags formatter in tiddly form. If you need javascript code, that contains a mess of regex's to implement txt2tags. There are a few gaps in the syntax, and it borrows some tiddlywiki syntax (I copied the <<macro>> syntax as txt2tags has no concept of that feature). I've coincidentally been fixing some bugs and am posting an update over the weekend.
> http://wiki.txt2tags.org/index.php/Main/TiddlywikiPlugin
And may be wildly irrelevant at this point, but note that the developer of Zim (desktop wiki + some calendar+to-do) has set a goal of transitioning from supporting only its proprietary syntax (very close to DokuWiki's) to the Pandoc flavor of extended Markdown. To start with he's writing an export routine, but plans to support it internally in the future, perhaps with other more open syntax choices as well.
I've found that Pandoc is *extremely* actively developed, much more so than txt2tags, and another big advantage is that it supports two-way transforms for many of its supported formats, whereas your plugin is the *only* example I've come across of anything going *to* txt2tags syntax.
For those interested in this stuff, I'm maintaining a pivot-table matrix in gDocs
Jeremy Ruston
The big difference I notice with these tools is that they are usually
designed for batch processing. With TiddlyWiki, a big part of the
challenge of rendering wikitext to HTML is to be able to efficiently
selectively update the DOM nodes that need regenerating when an
underlying data item changes. This is necessary for the familiar
interactive features to work properly (eg, an italicised link to a
missing tiddler automatically changing to a bold, non-italic link when
the tiddler is created). With TW5 I'm hoping I've been able to do that
in a way that will still be familiar to people who've worked on other
text processing systems.
Cheers
Jeremy
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/tiddlywiki/-/HFphD2Rv1poJ.
HansBKK
Pandoc is very interesting, thanks for pointing it out.The big difference I notice with these tools is that they are usually
designed for batch processing. With TiddlyWiki, a big part of the
challenge of rendering wikitext to HTML is to be able to efficiently
selectively update the DOM nodes that need regenerating when an
underlying data item changes.
Keep in mind the syntax aspect is separate from any software that may leverage it, so although the Pandoc software may be written in Haskell, if Zim ends up supporting that (markdown+pandoc) syntax, Jaap will be writing code from scratch in Python to do so.
The author of Pandoc has also written Gitit, an interactive wiki that can use multiple DVCS's as the storage back-end, but it's the syntax of the plain-text "sea of nodes" that gives it the ability to be transformed into a website, ebook, pretty PDF or dead-trees via LaTeX. . .
PMario
didn't want to use it with markdown, but with TW syntax. So I needed
to render HTML output (using TiddlySpace twikifyer) and feed it to
pandoc. The problem is that there are no paragraph <p> tags in TW HTML
output, which makes stuff complicated for propper PDF generation.
So I stopped testing.
[1] http://fnd.tiddlyspace.com/#3283b3cf0e9782f4d707825a80224086%2068ecefe7437c53003d5a8045bd812940
HansBKK
I did test pandoc some time ago. FND posted some usefull links [1]. I didn't want to use it with markdown, but with TW syntax. So I needed to render HTML output (using TiddlySpace twikifyer) and feed it to pandoc. The problem is that there are no paragraph <p> tags in TW HTML output, which makes stuff complicated for propper PDF generation.
So I stopped testing.
The absolute ideal IMO would be a core TW architecture that allowed for user choice of internally-stored syntax, with or without appropriate rendering and input assistance. Zim's author is intending to head down that road in future.
Second best is export capability to one or more "standard" syntaxes to support Txt2tags and/or Pandoc - I'm now leaning toward the latter, so leading candidates are extended markdown or reST/Sphinx.
Worst case is adding a "reader" for TW-specific syntax to Pandoc, which will then be able to output to any of its dozen+ target output formats. In which case a Pandoc "writer" for txt2tags and/or Asciidoc (which I believe is in the dev version) would make things pretty complete.
My understanding is that these wouldn't be too hard for an experienced programmer. If I need to learn to program, I'm not sure if Haskell's the language to start with but they say most important is a relevant real-world project to motivate you. 8-)
Jeremy Ruston
> issues, doesn't that make use of <p> tags?
As I've said before, I do plan to explore "fixing" the standard
TiddlyWiki wikifier in TW5 so that it does emit the expected <p> tags.
> The absolute ideal IMO would be a core TW architecture that allowed for user
> choice of internally-stored syntax, with or without appropriate rendering
> and input assistance. Zim's author is intending to head down that road in
> future.
I think that's where TW5 is. You can store tiddlers in what ever type
you like. The system looks for a parser that can convert each
particular MIME type into HTML. So, you'll be able to, for instance,
whip up a MarkDown parser (perhaps based on ShowDown), and then store
all your stuff in "text/x-web-markdown" (if that is the right mime
type; it seems contentious). Of course, MarkDown would need to have
macros added somehow.
> Second best is export capability to one or more "standard" syntaxes to
> support Txt2tags and/or Pandoc - I'm now leaning toward the latter, so
> leading candidates are extended markdown or reST/Sphinx.
For general, interoperable export I was thinking that HTML would be useful.
> Worst case is adding a "reader" for TW-specific syntax to Pandoc, which will
> then be able to output to any of its dozen+ target output formats. In which
> case a Pandoc "writer" for txt2tags and/or Asciidoc (which I believe is in
> the dev version) would make things pretty complete.
I imagine it might be useful if Pandoc were able to support TW5 wikitext.
> My understanding is that these wouldn't be too hard for an experienced
> programmer. If I need to learn to program, I'm not sure if Haskell's the
> language to start with but they say most important is a relevant real-world
> project to motivate you. 8-)
On those grounds it might be reasonable to learn JavaScript?
best wishes
Jeremy
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/tiddlywiki/-/XXVlyMY7oAIJ.
HansBKK
On Tuesday, January 31, 2012 4:43:56 PM UTC+7, Jeremy Ruston wrote:
> I thought I remember coming across a TW plugin that handled hard return
> issues, doesn't that make use of <p> tags?As I've said before, I do plan to explore "fixing" the standard
TiddlyWiki wikifier in TW5 so that it does emit the expected <p> tags.
> The absolute ideal IMO would be a core TW architecture that allowed for user
> choice of internally-stored syntax, with or without appropriate rendering
> and input assistance. Zim's author is intending to head down that road in
> future.I think that's where TW5 is. You can store tiddlers in what ever type
you like. The system looks for a parser that can convert each
particular MIME type into HTML. So, you'll be able to, for instance,
whip up a MarkDown parser (perhaps based on ShowDown), and then store
> Second best is export capability to one or more "standard" syntaxes to support Txt2tags and/or Pandoc - I'm now leaning toward the latter, so leading candidates are extended markdown or reST/Sphinx.
For general, interoperable export I was thinking that HTML would be useful.
> Worst case is adding a "reader" for TW-specific syntax to Pandoc, which will then be able to output to any of its dozen+ target output formats. In which case a Pandoc "writer" for txt2tags and/or Asciidoc (which I believe is in the dev version) would make things pretty complete.
I imagine it might be useful if Pandoc were able to support TW5 wikitext.
> My understanding is that these wouldn't be too hard for an experienced
> programmer. If I need to learn to program, I'm not sure if Haskell's the
> language to start with but they say most important is a relevant real-world
> project to motivate you. 8-)On those grounds it might be reasonable to learn JavaScript?
Jeremy Ruston
about tiddlers.
For interchange of richly structured documents, a JSON format would be
quite useful, so I'd be interested to understand Pandoc's support
better.
Lots of people love Python, I think it makes a good choice.
Best wishes
Jeremy
chris...@gmail.com
>> Hey Jeremy, what's the status of your wikifier. It's it play-with-able
>> in a standalone kind of way? If so where and how?
>
> With some caveats I think this is probably a good time to start
> investigating. I think the code is capable of what you need, but it's
> still rather in flux, and so can be changed in response to your
> requirements.
Thanks for the detailed info. I shall attempt to dig. Some comments
within.
> - Error handling is probably a bit inadequate
I don't know about you but I've found effective error trapping and
(especially) awareness quite challenging with node.
> So, I've tried to arrive at an implementation that supports both
> approaches. One of the hallmarks of the design is the aggressive use
> of caching to support efficient selective updating of the DOM. I think
> a big area of discussion around this integration is how we can ensure
> that our respective caching strategies work well together.
The finally implementation of twikifier has just one cache: it caches
(in memcache) the (skinny) JSON that is uses to makes its pseudo store
(see below).
> Using the existing implementation, an asynchronous function to wikify
> a tiddler goes like this:
>
> - Initialise a WikiStore object
> - Asynchronously load the tiddler into the wikistore
> - When it's loaded, parse the tiddler to obtain a list of its
> dependencies (this is a synchronous operation)
> - Loop back to recursively load the dependent tiddlers
> - Call the store.renderTiddler() function to obtain an HTML rendering
> of the tiddler, which is also a synchronous operation
This is quite similar to what twikifier does, the differences coming
about mainly because twikifier is accepting requests for rendering over
a unix domain socket:
* on the socket accept a string which is
** the current collection (bag or recipe in which the tiddler lives)
** the text of that tiddler
** the cookie of the current user (in case the bag or recipe needs
auth to access the collection)
* if the text has a macro, load the skinny JSON
** make a store with tiddlers without text (supporting timeline
style macros, <<tiddler>> type transclusion is done afterwards,
recursively from the other side of the socket)
* call wikify on the text faking in a dom thing from jsdom
The part of your description that is most interesting is the
dependency resolution.
> The WikiStore object is the equivalent of the TiddlyWiki() object in
> the old code; it's a collection of tiddlers. The Tiddler object is
> intentionally simple: just an immutable collection of fields. The
> WikiStore also needs be configured with the parsers and macros that
> you want to use. I'm currently using a module called App.js to
> centralise the WikiStore configuration steps for both browser and
> node.js configurations of the code:
>
> https://github.com/Jermolene/cook.js/blob/master/js/App.js
so:
* make a store
* register parsers
* What are tiddlerInput and tiddlerOutput?
* install macros
* addTiddler
* get its dependencies
** if null: load all tiddlers into store
** if list: load each of those tiddlers into store
* renderTiddler
That sound about right?
--
Chris Dent http://burningchrome.com/
[...]
Jeremy Ruston
> within.
>
>
>> - Error handling is probably a bit inadequate
>
>
> I don't know about you but I've found effective error trapping and
> (especially) awareness quite challenging with node.
Yes, quite. I've settled on using the callback(err) pattern, and only
throw exceptions right up in the command line handling of
tiddlywiki.js.
A particularly troublesome issue is making sure that errors in
compiled (ie eval-ed) code are both properly trapped and helpfully
reported.
> The finally implementation of twikifier has just one cache: it caches
> (in memcache) the (skinny) JSON that is uses to makes its pseudo store
> (see below).
Cool. That would still be useful, and you could consider caching the
WikiStore and tiddler objects perhaps later.
> The part of your description that is most interesting is the
> dependency resolution.
Yes, it makes a big difference, and is very easy to do once one has
the parse tree lying around.
> so:
>
> * make a store
> * register parsers
> * What are tiddlerInput and tiddlerOutput?
(they handle deserialisation and serialisation of tiddlers; just a
pile of static functions, and I imagine you won't need them)
> * install macros
> * addTiddler
> * get its dependencies
> ** if null: load all tiddlers into store
> ** if list: load each of those tiddlers into store
> * renderTiddler
>
> That sound about right?
Bang on, yes.
Do you have many tests for twikifier? It'd be cool for TW5 to be able
to adopt/adapt them.
Best wishes
Jeremy
> --
> Chris Dent http://burningchrome.com/
> [...]
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
chris...@gmail.com
> Do you have many tests for twikifier? It'd be cool for TW5 to be able
> to adopt/adapt them.
Sadly no. I was in such a rush and flailing around with such ignorance
when I was making twikifier that I was mostly just throwing code and
hoping. The irony is that that's a great time to write tests, but you
know...
Jeremy Ruston
>> to adopt/adapt them.
>
>
> Sadly no. I was in such a rush and flailing around with such ignorance
> when I was making twikifier that I was mostly just throwing code and
> hoping. The irony is that that's a great time to write tests, but you
> know...
TW5 has a very crude test rig that could be adapted to test over
twikifier, I think. It's just a folder of *.tid files alongside the
.html and .txt files that they wikify to:
https://github.com/Jermolene/cook.js/tree/master/test/wikitests
Best wishes
Jeremy
chris...@gmail.com
>> That sound about right?
>
> Bang on, yes.
https://github.com/cdent/cook.js/commit/961713675461c93668b385e731ede015622b5360
Has a very simple prove to myself that I get it.
Jeremy Ruston
support from App.js as well at some point.
You may also be interested in the new disableHtmlWrapperNodes option
for the WikiStore constructor; it suppresses the DIVs that are
otherwise inserted to help the refresh mechanism. In your environment
you probably won't want them.
Best wishes
Jeremy
chris...@gmail.com
> Great, that looks right. I think you'll need to add the shadow tiddler
> support from App.js as well at some point.
What for? The usual HTML rendering for Tiddly{Web,Space} ignores
shadow tiddlers as artifacts of TiddlyWiki, not tiddlers in general.
> You may also be interested in the new disableHtmlWrapperNodes option
> for the WikiStore constructor; it suppresses the DIVs that are
> otherwise inserted to help the refresh mechanism. In your environment
> you probably won't want them.
Oh yes, that sounds useful.
Jeremy Ruston
>> support from App.js as well at some point.
>
>
> What for? The usual HTML rendering for Tiddly{Web,Space} ignores
> shadow tiddlers as artifacts of TiddlyWiki, not tiddlers in general.
Right; some TiddlyWiki content assumes the existence of the shadow
tiddlers by doing things like transcluding TabAll, or rendering
WindowTitle to get the page title. It's good if you've not been
running into cases where that matters.
Best wishes
Jeremy
HansBKK
For interchange of richly structured documents, a JSON format would be quite useful, so I'd be interested to understand Pandoc's support better.
% pandoc -t json1. *hi*2. there^D
[{"docTitle":[],"docAuthors":[],"docDate":[]},[{"OrderedList":[[1,"Decimal","Period"],[[{"Plain":[{"Emph":[{"Str":"hi"}]}]}],[{"Plain":[{"Str":"there"}]}]]]}]]
Jeremy Ruston
>
> store.renderTiddler("text/html",title);
>
> You can also render the specified tiddler as though it were another
> one; this is useful for rendering tiddlers as templates. For example:
>
> store.renderTiddler("text/html","MyTemplate","HelloThere");
>
> This example would render the tiddler HelloThere using the tiddler
> MyTemplate as a template. So things like <<view title>> within
> MyTemplate would render the tiddler HelloThere.
Mainly for Chris's benefit:
The signature for store.renderTiddler() has just changed. It's now:
renderTiddler(targetMimeType,tiddlerTitle,templateTitle,options)
As you would expect, you can leave the template parameter null or
undefined to render the tiddler directly without a template.
Cheers
Jeremy