pandoc and markdown to TW format

[Shay Shaked]

So I installed the markdown plugin, works ok, but the problem is that it doesn't play nice inside TW. some markings (like strikethrough) and my own Macros, highlight to text, journal entries and the like do not play nice in TW. So, I was looking at pandoc to help me convert markdown to WikiText, which exists, but then TW is not exactly WikiText

Did anyone build a TW extension to pandoc perhaps? Can this be done? Or, is there a plugin to TW that understands WikiText? 

[PMario]

On Friday, November 17, 2017 at 6:16:46 PM UTC+1, Shay Shaked wrote:

... but then TW is not exactly WikiText

That's right. TiddlyWiki has its own syntax. The TiddlyWiki syntax, which has much more possibilities than most other wiki system.

TiddlyWiki wikitext exists since 2004. That's much longer then many other wiki systems. ... But as so often, it's not the the most powerfull wikitext, "that dominates the world". It's the one, that is "just good enough" to be usefull. ...

Did anyone build a TW extension to pandoc perhaps?

I don't know one.

 

Can this be done?

Sure, If someone, who knows how to program in Haskell, does it.  

-mario

[TonyM]

Shay,

There are so many formats there, I imagine it could be easy to add TiddlyWiki markup to the set.

Of more interest to me would be being able to import/export tiddlers in anyone of these nominated markups.

However I often find value customising the output differently to the source text anyway.

Regards
Tony

[@TiddlyTweeter]

Ciao Shay

IF you understand how to write complex regular expressions I highly recommend BJ's FLEXITY PLUGIN that will allow you to create **your own parser**.

In a previous post I adumbrated that TiddlyWiki has the potential to be a Universal Markup Engine--at least for the simpler markups and downs. I think there is a lot of mileage in that idea for inter-conversion of the simpler markup systems.

The older systems of markup are looking a bit tired. There is no real reason that they should not be inter-changeable easily. Its more an inertia to vision forward than any technical obstacle.

Pandoc is something else. Its scope is huge and its DEDICATED to the one purpose of complex conversion. Unless you need complex XML etc I think its overkill.

Best wishes
Josiah

[Shay Shaked]

Here's my naaive, newbie theory:

I use Typora (Markdown) and TiddlyWiki (TW text). Typora had panodoc built into it, so it can convert to HTML, Wiki text, and bunch of other things. Pandoc has "extensions" (for lack of a better term) that "translate" Markdown to Wikitext.

Ok, so, why not find the code that translate md to Wikitext, and change it so instead of translating # to <h1>, translate it to <li> for example. Continue with the rest.

I _know_ it can't be that simple, but don't know why. Also, TW understands HTML, so that's a work around, but not exactly what I want.

Can you, wise people, explain it to me in terms that I can grasp maybe? Thanks :-)

[PMario]

On Sunday, November 19, 2017 at 1:30:19 AM UTC+1, Shay Shaked wrote:

I _know_ it can't be that simple, but don't know why.

You are absolute right. ... The basic parts are probably simple eg:

static lists in markdown

 - element 1

 - element 2

static lists in tiddlywiki

* element 1

* element 2

That is simple, but boring, cumbersome and errorprone.

Whenever a new element should be added, you have to search all the different occurrencies and edit them.

We use dynamic lists, built with tags, fields, .... like so:

<<list-links filter:"[tag[myList]]">> ... TW wikitext macros

{{{"[tag[myList]]"}}} ... which is a shorthand for: <$list filter="[tag[myList]]"/>

<$list filter="[tag[myList]]" template="myCustomListTemplate"/>

and a lot more: https://tiddlywiki.com/#ListWidget

That's just one reason why it's not simple. 

While pandoc is an exceptional piece of work, it only translates simple text a to simple text b. 

With TW we use transclusions a lot. see: https://tiddlywiki.com/#Transclusion
And that's the second reason why it isn't simple.

and so on, and so on, ...

have fun!
mario

[@TiddlyTweeter]

Ciao Shay & PMario

I'm finding this thread both really interesting AND confusing. We need be clear how TW parsing works.

Let's take the case that you want a non-TiddlyWiki markup system converting to Tiddly-Wiki markup. The issue could be thought about at least two ways ...

1 - you PRESERVE the original markup of your source and dynamically convert it to TW markup. For that the BJ plugins are relevant. In the current TW architecture it is a reasonable route. Both Flexity (mentioned before, that uses regular expressions) and the Flexible Types plugin that requires you to be able to write new JavaScript components would work. The point is both take hold of the parsing system and allows you to pre-run your own markup conversions.

2 - you CONVERT (permanently) the original markup to TW markup. Either via an import mechanism
or via (1) saving to a new Tiddler a rendered version of the final code.

A lot of the work in all such is done via regular expressions. For simple substitution, like "^ *-" (meaning "-" is the first non-space character in the line) for "*" are quite easy. The in-built parsers in TW have more sophistication allowing for more complex situations--but I think most simple markup can be "translated" dynamically easily to its TW equivalent and let TW then do the HTML render in the normal way. Or you could directly convert to HTML (more complex).

Best wishes
Josiah

[TonyM]

Josiah,

Is it a third approach, or one of your two to use tiddler type to keep and edit in any desired markup standard and only render it as needed to html as currently happens with wikitext?

regards tony

[@TiddlyTweeter]

Ciao TonyM

TonyM wrote:

Is it a third approach, or one of your two to use tiddler type to keep and edit in any desired markup standard and only render it as needed to html as currently happens with wikitext?

For some types of specialised Markup, closely linked to application types, it makes most sense to create a new Content Type and go much of the way to creating final HTML for the rendered version.

FOUNTAIN markup is an example that is widely used in writing screenplays. It makes much sense to author using Fountain when writing screenplays inTW. But its markup conflicts with TW standard markup. https://fountain.io/syntax

There are TWO ways forward (which you can mix) ... you convert for TW compliant markup where you can (in other words change F. markup to compliant TW markup in a pre-parser before the TW parser runs--after that you get what you'd expect) AND/OR directly convert F. Markup to HTML.

Fountain markup is interesting because its largely IMPLICIT (how you space paragraphs & how you start them) whilst most normal wiki markup is initiated by explicit characters (*/-/! etc). F. is great for screenplay authors because you rarely ever have to enter any explicit markup if you have your line layout right.

But to use the F. approach in TW you need more than straight conversion of markup systems because the "implicit" markup of F. TW has no equivalents for. Its still pretty easy once you can access a pre-parser just using regular expressions to do it though.

All of this works under point (1) of my last post.

Point (2) about CONVERSION is more a one-way-ticket. Its for cases where you don't need to retain original markup, just permanently convert for native TW normal.

Best wishes
Josiah

[Mark S.]

Looking at pandoc, it looks like one might have it do markdown to plain text, and then create filters in between for each of the markdown types to wrap the plain text the way TW text wants.

But how much markdown to have to convert?

-- Mark

[TonyM]

Josiah,

The following is possible more for my own understanding, but may help describe the situation.

When it comes to IMPLICIT rather than explicit there are hidden characters in any text be it new lines, a pair of new lines and more. It seems to me that along with some smart CSS determining if you can easily process an alternate markup can only be determined in its details, ie stepping through each of the detailed markup features and finding a way to display according the required standard. The idea would be to always retain the text in its custom markup and only display it according to standards just in time. This allows it to be imported/exported/viewed and edited in its native form. Since the act of displaying it, is one of the standards, HTML the fact is we can copy or save it in this display format as well for transfer to other documents or environments.

The other approach would be to use tiddlywiki markup and macros as needed to display (not in finished presentation format) but in fountain markup which can be copied and pasted into a converter or tool for the final display.

I suppose it all depends on which part(s) of the workflow you want tiddlywiki to be responsible.

Regards

Tony

[Shay Shaked]

This is all new stuff to me. I'm reading up on parsers, so this is beyond me at this point.

It's simple enough to go over the md file for now and put in the formatting myself. Since I know I will be doing this, I don't bother too much with formatting; i create a list at the button of the entry with notes for format, usually notes to what I want to link or what picture to add, and that's it. I also use the search function in the browser to highlight underscores when adding an article, so I known to use // for example.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/QcBqV2Cvxrw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/649b92f5-274a-431d-91ee-526295276c1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[TonyM]

Mark,

I would think changing // to __ as an example is trivial. As long as one can reliable search and replace one markup for another in a set of search and replace steps you could build a macro for this purpose in NotePadd++ and run it against material constructed in TiddlyWiki mark(down). Even leading characters can be replaced using find "\n!!!" replace with "\n###", of course one should replace longer strings before shorter ones like find "\n!" replace with "\n#" after the previous example. Oh and line Trailing "]]\n".

Unfortunately you need to step through every conversion step before you will know if it is easy or not.

Regards

Tony