What if?

[PMario]

What if ...

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

-m

[1] http://commonmark.org/
[2] http://spec.commonmark.org/0.28
[3] https://en.wikipedia.org/wiki/Subset

[Riz]

Strongly supports the motion.

[@TiddlyTweeter]

Ciao PMario

I am very much of the view that TiddlyWiki could become a Universal Markup handler (for the simpler markup systems).

Experimenting with BJ's http://flexibility.tiddlyspot.com/ ... a variant on his Flexitypes that allows use of RegEx has made me very aware of a number of issues ...

1 - IF you can access a PRE-PARSER then you can take any alien (simple) Markup and convert it dynamically to TW markup (BUT only IF TW has an equivalent). It is almost trivially easy. You do the conversion in the pre-parse and then it passed through the normal rendering.

2 - That opens the possibility of using MANY types of naked markup, including USER DESIGNED markup systems.

3 - COMPLEXITIES arise only when the originating markup has no TW direct equivalent. For instance much of "Fountain Markup" does not have TW equivalents because its IMPLICIT markup and TW mainly manages EXPLICIT markup.

Underlying my points here is a question about HOW much JavaScript you'd actually need? A RegEx interface for pre-parsing seems to be able to do a hell of lot (and without degrading performance).

Thoughts. With a bit experience in them.

Best wishes
Josiah

[Diego Mesa]

Strong second! 

On Friday, December 8, 2017 at 7:15:52 AM UTC-6, Riz wrote:

Strongly supports the motion.

[@TiddlyTweeter]

Further to my last.

I want to MOAN.

Moan that markup systems have emerged that differ but should be the same.

Moan that a universal etiquette on simple markup is not there. It unnecessarily complicates things that the simpler markups are not orthographically congruent.

Moan that TiddlyWiki isn't providing the needed Multi-Markup support is not there yet.

Hoping that TiddlyWiki will take simple Markup and downs out of the dark ages into a universal step forward.

Josiah, x  

On Friday, 8 December 2017 13:56:31 UTC+1, PMario wrote:

[PMario]

On Saturday, December 9, 2017 at 1:55:26 PM UTC+1, @TiddlyTweeter wrote:

I want to MOAN.

q:)

 

Moan that markup systems have emerged that differ but should be the same.

... We are lucky to have a choice! ... Otherwise everyone would have to use what I prefer ;)

 

Moan that a universal etiquette on simple markup is not there. It unnecessarily complicates things that the simpler markups are not orthographically congruent.

Yea. ... Just "good enough" rules the world. You can see this everywhere.

Moan that TiddlyWiki isn't providing the needed Multi-Markup support is not there yet.

Oh, ... we are close. ... BUT if we would make commonmark a first class citicen, it would break backwards compatibility.

 

Hoping that TiddlyWiki will take simple Markup and downs out of the dark ages into a universal step forward.

No hope here. If TW syntax would be a superset of MD it needs to be more advanced and therefor more "complicated".

-m

[@TiddlyTweeter]

Josiah: TW isn't providing the needed Multi-Markup support is not there yet.

 

PMario wrote:

Oh, ... we are close. ... BUT if we would make commonmark a first class citicen, it would break backwards compatibility.

Initially, EEK! I looked at the 622 example cases of the CommonMark Spec. But actually many of them are easy. If I had a spare month I might have a stab at it (in regex) :-) Its the multi-line nested blocks (like lists) that look most difficult.

Best wishes
Josiah

[Jeremy Ruston]

Mario said:

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

I’d be very interested if that were possible. It would potentially allow us to dispense with the current Markdown parser, which has significantly limited capabilities compared to native wikitext.

I also agree with Josiah’s point that a worthy longer term goal is to make TiddlyWiki a meta-markup system that allows end users to create their own markup parse rules. However, the complexity of the CommonMark specification suggests to me that it wouldn’t be possible to create a practical parser entirely declaratively. In other words, I think the complexities of arbitrary markup may well require arbitrary computational capabilities to parse.

But, of course, even if we couldn’t create a 100% spec compliant CommonMark parser declaratively, it would still be a very useful thing to have.

Best wishes

Jeremy

[@TiddlyTweeter]

Ciao Jeremy

I'm a pragmatist.

I simply do NOT believe that the 600 or so strictures that CommonMarkup syntax say are essential are actually essential for most practical work. It looks like a tall-story. When you look into it is boils down to something more cope-able, I think.

I am pretty convinced that TW can get very close to a "universal markup" for coping with most of the the variant Wiki-text versions in real use. And I already know from playing with BJ's "pre-parser" that that there is immense flex in what TW can do towards creating "User Defined Markup" quite easily.

On this particular issue I'm happy to help as its one of the few things I have competence on.

Best wishes
Josiah

[Jeremy Ruston]

Hi Josiah

I'm a pragmatist.

I simply do NOT believe that the 600 or so strictures that CommonMarkup syntax say are essential are actually essential for most practical work. It looks like a tall-story. When you look into it is boils down to something more cope-able, I think.

That’s rather my point: a putative declarative meta-markup couldn’t achieve 100% compatibility, but that even partial compatibility would be useful. But that’s not what Mario originally proposed.

I am pretty convinced that TW can get very close to a "universal markup" for coping with most of the the variant Wiki-text versions in real use. And I already know from playing with BJ's "pre-parser" that that there is immense flex in what TW can do towards creating "User Defined Markup" quite easily.

On this particular issue I'm happy to help as its one of the few things I have competence on.

Great, some concrete proposals would be useful at this point

Best wishes

Jeremy.

Best wishes
Josiah

On Tuesday, 12 December 2017 16:42:00 UTC+1, Jeremy Ruston wrote:

Mario said:

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

I’d be very interested if that were possible. It would potentially allow us to dispense with the current Markdown parser, which has significantly limited capabilities compared to native wikitext.

I also agree with Josiah’s point that a worthy longer term goal is to make TiddlyWiki a meta-markup system that allows end users to create their own markup parse rules. However, the complexity of the CommonMark specification suggests to me that it wouldn’t be possible to create a practical parser entirely declaratively. In other words, I think the complexities of arbitrary markup may well require arbitrary computational capabilities to parse.

But, of course, even if we couldn’t create a 100% spec compliant CommonMark parser declaratively, it would still be a very useful thing to have.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/94321674-7dca-464a-abb2-aee48340beea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PMario]

On Tuesday, December 12, 2017 at 5:08:46 PM UTC+1, Jeremy Ruston wrote:

I simply do NOT believe that the 600 or so strictures that CommonMarkup syntax say are essential are actually essential for most practical work. It looks like a tall-story. When you look into it is boils down to something more cope-able, I think.

That’s rather my point: a putative declarative meta-markup couldn’t achieve 100% compatibility, but that even partial compatibility would be useful. But that’s not what Mario originally proposed.

I did have a closer look at the commonmark rules. ... They did standardize some strange decisions. Mainly because of very popular existing MD software and variants, such as pandoc ..

The new markdown spec allows the registration and usage of a variants hint. So if we do it right, it might be possible to use

text/markdown for 100% commonMark compatible stuff and
text/markdown; variant=tiddlywiki   ... for our TW stuff. ... Only possible if we manage to register it!!


@Jeremy btw: We should register text/vnd.tiddlywiki to be listed by the IANA. ...


We may be able to use: text/vnd.tiddlywiki; variant=commonmark  or variant=markdown-it or ...

So imo it's not really needed to get a 100% compatibility. .. As long as we can manage the MD-zoo we would be able to create

have fun!
mario

[@TiddlyTweeter]

PMario wrote:

So imo it's not really needed to get a 100% compatibility. .. As long as we can manage the MD-zoo we would be able to create

Right. A likely ZOO. With likely cross-breeding over time. And demands to allow some Darwin nightmare to live that should have been put-down.

Immediately for me come three potential tickets ...

TICKET #1 - The ONE-WAY TICKET - You join TW and it CONVERTS your markup to TW markup. TW from now on, mate.

TICKET #2 - The TWO-WAY TICKET - You join TW and it CONVERTS your markup to TW markup but lets you also to EXPORT it to that terrible system you used before. You are converted so now need to use the Catholic TW system but if you want to go back to your previous bad habits you can.

TICKET #3 - The OPEN TICKET - You join TW and continue to use the markup system you love already. Advantage: garnering the lost souls of other wiki systems. Disadvantage: the garnering of lost souls of other wiki systems.

Just some thoughts.
Josiah

[TonyM]

Josiah,

Apart from a desire to use TiddlyWiki such that I can secure content from any sources, markup alternatives etc.. I support multiple markups. 

However I think a brainstorm could Identify specific uses for alternate markups.

• I for one would love to be able to freely move content between tiddlywiki and MediaWiki including Wikipedia such that I do not need to know both in detail.
  
• I would like to be able to build scripts and batches, sql commands and export them to their required file types which are basically plain text.
• I would like to define ad hoc markup to interrogate logs files and other standard data sources.

I do however think HTML should be a 1st Class Citizen since regardless of a markup standard, it is often displayed as html, almost every piece of content can be captured and rendered this way.

Just some thoughts to test if your tickets are comprehensive enough.

Regards

Tony

[@TiddlyTweeter]

Ciao PMario et al,

PMario wrote:

What if ... we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

Rather than CommonMark wouldn't Github Flavoured Markdown (GFM) be a better aim? Its ...

1. compliant with CommonMark;
2. adds tables and other things Markdown does not do;
3. better services existing TW users who also use GitHub.

The specification comments ...

GFM is a strict superset of CommonMark. All the features which are supported in GitHub user content and that are not specified on the original CommonMark Spec are hence known as extensions, and highlighted as such.

See: GFM Specification

Early thoughts
Josiah 

[@TiddlyTweeter]

Ciao  PMario & others interested in TW markup fexibility ...

PMario wrote:

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

The name "CommonMark" is slightly misleading (in implying that all Markdown variants can be processed following its syntax. FWIW Markdown's maker, Gruber, objected to their use of the word "Standard" in early versions so they changed the name).

The initiative is a brilliant idea to create a superset of Markdown variants but its actually itself a variant (a good variant, very inclusive, less ambiguous, but NOT a total solution for all possible cases).

The main breakage with some versions of Markdown is it ASSERTS a few rules to reduce ambiguity in various implementations that are not fully backwards compatible with all Markdown variants (of which there are many).

But I think its attractive because its getting used (both GitHub & Dingus use it) in larger VOLUME sites.

In looking at a possible TW way forward with it I think that the volume usage cases are the most interesting & relevant? And the actual way people typically do mark-up in them, rather than the endless variants that CommonMark posits as possible, could be a viable way?

Just early thoughts.

Best wishes
Josiah

[Tobias Beer]

I would think the solution is rather simple / has been done before:

I once implemented inline support for TWC using this type of syntax:

§§§

your markdown here

§§§

based on @ShowDown, see docs at @MarkdownTutor

It could easily be generalized into this:

§§§commonmark

your commonmark-markdown here

§§§

or even

 

§§§md-cm

your commonmark-markdown here

§§§

if we wanted it thusly.

best -tb

On Tuesday, 12 December 2017 16:42:00 UTC+1, Jeremy Ruston wrote:

[PMario]

On Thursday, December 14, 2017 at 12:34:27 PM UTC+1, @TiddlyTweeter wrote:

Rather than CommonMark wouldn't Github Flavoured Markdown (GFM) be a better aim? Its ...

Nope. ... markdown is standardized now. (since ~ March 2016) ... https://tools.ietf.org/html/rfc7763

Variants spec: https://tools.ietf.org/html/rfc7764#page-10

 - GFM is one of several variants

 - CommonMark is an other variant ...

1. compliant with CommonMark;

that's ok, but It's different

 

1. adds tables and other things Markdown does not do;

TW has tables that imo are more powerful

 

1. better services existing TW users who also use GitHub.

IMO no problem, we can have several variants. eg: markdown-it library supports many variants already.

We just need to switch the mime type and we can deal with them. ...

 

I do have a finished plugin. ... I just have to prepare the edition.
... BUT ...
I want to get rid of external libraries, most of the time, they don't integrate well with TWs possibilities.

have fun!
mario

[@TiddlyTweeter]

Which part of Der Ring des Nibelungen do you want to start with...? OK, Das Rheingold.

PMario Wrote: 

I do have a finished plugin. ... I just have to prepare the edition.
... BUT ...
I want to get rid of external libraries, most of the time, they don't integrate well with TWs possibilities.

First chorus: Do you have a full faultless CommonMark?

  (expunges libraries)

Second chorus: No he doesn't.

  (expunges libraries)

First chorus: He might have.

  (expunges libraries)

Second chorus: Ask Him. Ask Him.

  (expunges libraries)

First chorus: Your plugin is a full faultless CommonMark?

Notte, Josiah 

[TonyM]

Mario,

It is interesting you say "TW has tables that imo are more powerful"  The reason I say this is because only recently I came to discover exactly how much of HTML is usable right in wikitext and tiddlywiki.  Here is a working example with a list widget in it.

<table style="width:100%;  align: left; ">

<caption>Monthly savings</caption>

  <tr>

    <th>Item</th>

    <th>Caption</th>

    <th style="align: right;">Icon</th>

  </tr>

<$list filter="[tags[Resources]]">

  <tr>

    <td>{{!!title}}</td>

    <td>{{!!caption}}</td>

    <td text-align: right;>{{!!icon}}</td>

  </tr>

</$list>

  <tr>

    <th>Item</th>

    <th>Caption</th>

    <th >Icon</th>

  </tr>

</table>

My Point is that html markup is often more powerful when dealing with more sophisticated objects such as tables and simplifies the creation process because elements are more easily duplicated and there is rudimentary error checking and structure, CSS insertion points etc..

To me HTML is an essential skill and I see no harm choosing it when it suits. With all the talk of alternate markup/down I hope we do not compromise TiddlyWikis direct relationship to the pervasive standards on the internet such as HTML/CSS especially since that is how the tiddlywiki is rendered.

Theoretically would most markdown standards have documented processes to markup up and down to/from HTML? If this was the case could we take any HTML and apply any markdown to it?

My idea is we could produce any content that you can see on the screen, Wiki Text, HTML, and any that works in Tiddlywiki, it is then rendered in HTML and then generate markdown texts that can then be transferred into their own tools. 

An Example is I would Create a chapter in TiddlyWiki using any method I choose, Display it (HTML) then choose to generate Wiki media markdown which I then save, copy or paste into Wiki Media. Perhaps we could harvest content in HTML from anywhere, Save it in Tiddlywiki and generate alternative markdown from it, even to its orignnal mark down.

The all we have to do is translate any markdown to HTML and back to any markdown. Which presumably is already done for most markdown formats.

Am I making sense?

Tony 

[@TiddlyTweeter]

Practically WYSIWYIG is part of this.

The point of this is whether we still value (or need) the idea of PLAIN TEXT representing something worth having. Markdown originally emerged for email as much as for the web. The idea was to be able to AUTHOR in a text style that was readable regardless of the final rendering.

I'm sceptical it applies now in the same way. Years have passed.

I DO think that SHORTCUT markup systems are still good. BUT. BUT. But like: you can ALSO use shortcuts towards HTML in live editing too.

Its an interesting issue.

Whether TW is somewhat OVER-valuing "fading systems of mark-up" or not. I'm personally divided on the issue but do see a tension.

The problem with HTML is only readability (the LOGIC is great, the visuals of the actual code are crap). I'd far rather have my originating stuff in TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true WYSIWYG in the editor. Once I have that I don't care.

Just thoughts
Josiah

TonyM wrote:

... To me HTML is an essential skill and I see no harm choosing it when it suits. With all the talk of alternate markup/down I hope we do not compromise TiddlyWikis direct relationship to the pervasive standards on the internet such as HTML/CSS especially since that is how the tiddlywiki is rendered.

[TonyM]

Josiah,

 

The problem with HTML is only readability (the LOGIC is great, the visuals of the actual code are crap). I'd far rather have my originating stuff in TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true WYSIWYG in the editor. Once I have that I don't care

I understand your point, but when it is a complex, nested structure with loops (lists) there is real value having open and closed elements, full control of line breaks and classes/styles.

Sometime this even beats WYSIWIG because the structure logic and tag names allow reference to the elements and logic there in.

I think there would be value identifying and documenting the use cases for HTML usage in tiddlywiki.

Regards

Tony

[@TiddlyTweeter]

Personally I think a SWOP of what we currently have would be ideal. By which I mean you edit in WYSIWYG and there is a pane that displays the underlying code (that ideally you could also edit).

Likely much later.  J, x

[TonyM]

I assume you have seen existing TiddlyWiki WISYWIG solutions?

[@TiddlyTweeter]

TonyM

I do use CKEDITOR in one TW. And I'm familiar with TinyMCE. Their expansion to allow live editing of whole pages (can't do that in TW) is great. Their growing model of "everything is editable" is a good one. A model where the rendered version and the editable version are co-terminus.

J.

[PMario]

On Friday, December 15, 2017 at 12:03:24 AM UTC+1, TonyM wrote:

My Point is that html markup is often more powerful when dealing with more sophisticated objects such as tables and simplifies the creation process because elements are more easily duplicated and there is rudimentary error checking and structure, CSS insertion points etc..

html is supported perfectly well. tables like your example is relatively straight forward. ... but try to dynamically nest them, and you will feel the pain.

With my - A script and todo manager workflow / UI experiment video series, I did explore dynamic table-like layouts, using CSS-grid. I think a concept like this is much easier to create and manage. .... With the experiment I did want to explore CSS-grid, the possibilities it provides ... and I wanted to see, how much "duplicated" code I produce to get it going. ... (Duplicated code let's you find out patterns, that can be optimized ... in a second step)

Video 4++: discusses dynamic table-like layouts

To me HTML is an essential skill and I see no harm choosing it when it suits.

The whole TW UI is  build with dynamically created HTML. ... There is no harm at all.

 

With all the talk of alternate markup/down I hope we do not compromise TiddlyWikis direct relationship to the pervasive standards on the internet such as HTML/CSS especially since that is how the tiddlywiki is rendered.

 

I don't intend to create the TW UI with markdown. ... TiddlyWiki as a chameleon. That makes it powerful on one side and rases the complexity on the other side.

Implementing MD as a first class citicen, for the basic writing syntax, would imo lower the entry-level for new users quite a bit.

Theoretically would most markdown standards have documented processes to markup up and down to/from HTML? If this was the case could we take any HTML and apply any markdown to it?

I think there is a misunderstanding here.

Original markdown is a "wiki-syntax" for writing text

TiddlyWiki syntax is a "wiki-syntax" for writing text

Both systems take "plain text" and convert it to HTML, using predefined rules, so browser can render it.

Those predefined rules define a: markup language

HTML means: .... HyperText Markup Language

eg: If we want to define something bold we use:

TW: ''bold text''

MD: **bold text**

HTML: <b>bold text</b>

The characters used to "mark" the enclosed text is 1 markup rule. ... "make it bold" ... many such rules define a markup-language or markpu-syntax. There are many markup-languages out there. TW, MD and HTML are only 3 of them.

TiddlyWiki and Markdown markup-syntax are designed to be human readable, even for big junks of prose text.

HTML was designed to display text on a computer display to make it human readable. ... Since it is text it is still readable by humans. But I don't want to read tables in HTML-text format. You need it to be rendered into something, that humans can read.

 

My idea is we could produce any content that you can see on the screen, Wiki Text, HTML, and any that works in Tiddlywiki, it is then rendered in HTML and then generate markdown texts that can then be transferred into their own tools. 

An Example is I would Create a chapter in TiddlyWiki using any method I choose, Display it (HTML) then choose to generate Wiki media markdown which I then save, copy or paste into Wiki Media. Perhaps we could harvest content in HTML from anywhere, Save it in Tiddlywiki and generate alternative markdown from it, even to its orignnal mark down.

So what you want here is: Convert one format to any other format. That's doable, but far away from simple.

With my example: **bold text** it's simple to convert from one language into the other: 

• Just replace MD ** with ''... and you have TW syntax .. super simple!
• HTML needs <b> for the beginning and </b> for the end. ...
  

So you can see, complexity goes up.

Now lets say we want red bold text. ... boom!! ... Markdown has NO rules to define this, because MD primary goal is human readability!

TW and HTML have rules do deal with that.

...BUT

We have a fundamental problem now!

We will lose information if we want to convert <b class="red">bold text</b> or ''@@.red bold text@@'' into Markdown. (I intentionally skip CSS here)

The all we have to do is translate any markdown to HTML and back to any markdown. Which presumably is already done for most markdown formats.

I hope you can see that it's not that easy. ... because your "roundtrip" from MD to HTML and back to MD will fail for red bold text. ...

There are solutions to deal with that, but I think there is already too much text :)

have fun!
mario

[PMario]

On Friday, December 15, 2017 at 12:46:33 AM UTC+1, @TiddlyTweeter wrote:

Practically WYSIWYIG is part of this.

Different topic. I want to discuss "What if " we include commonmark Markdown into TW.

No discussion about WYSIWYG .. I personally thing it's broken, but that's not the thread to discuss this.

-m

[PMario]

added video link to response.

Video 4++: discusses dynamic table-like layouts

-m

[PMario]

On Thursday, December 14, 2017 at 11:57:20 PM UTC+1, @TiddlyTweeter wrote:

First chorus: Your plugin is a full faultless CommonMark?

It uses the markdown-it library, which supports several MD variants. I did include config options for commonmark, gfm, and markdown-it native.

-m

[@TiddlyTweeter]

Ciao PMario

My apologies if I messed up your thread :-(. Hopefully I didn't. It was not intentional.

I will respond separately about why I think WYSIWYG is related in some ways to this thread.

Best wishes
Josiah

@TiddlyTweeter wrote:

Practically WYSIWYIG is part of this.

PMario wrote

[@TiddlyTweeter]

@TiddlyTweeter asked:

First chorus: Your plugin is a full faultless CommonMark?

PMario replied:

It uses the markdown-it library, which supports several MD variants. I did include config options for commonmark, gfm, and markdown-it native.

That sounds brilliant. Especially with config options for variants.

Part of my concern is coming to TW from MD that its using TW's huge flexibility to make migrants transfer to it easy.

As you wrote:

Implementing MD as a first class citicen, for the basic writing syntax, would imo lower the entry-level for new users quite a bit.

IMO, absolutely right. And enabling variants will make that aim much more achievable. Because "MD" in the wild is in many forms, some with serious extensions (GitHub & Dingus are the two I know of most).

I think practically that is the situation in the larger MD user bases--its MD plus already... Offering options for variants could really help, I think.

Best wishes
Josiah

[PMario]

On Saturday, December 16, 2017 at 4:49:50 PM UTC+1, @TiddlyTweeter wrote:

My apologies if I messed up your thread :-(. Hopefully I didn't. It was not intentional.

I will respond separately about why I think WYSIWYG is related in some ways to this thread.

WYSIWIYG is definitely related to any other text based markup system. ... But it is a completely different philosophy. ... and discussions, between those different point of views, can heat up, very quickly. ...

So I want to stay "text based" here. ... IMO prose mirror could be a middle ground ;)

have fun!
mario

[PMario]

On Friday, December 15, 2017 at 12:03:24 AM UTC+1, TonyM wrote:

The all we have to do is translate any markdown to HTML and back to any markdown. Which presumably is already done for most markdown formats.

 

Don't get me wrong. IMO every one of us would love to have this functionality, and it should be one of our goals!

-m

[Eric Shulman]

On Saturday, December 16, 2017 at 10:31:10 AM UTC-8, PMario wrote:

WYSIWIYG is definitely related to any other text based markup system. ... But it is a completely different philosophy. ... and discussions, between those different point of views, can heat up, very quickly. ...

Back in the early days of font rendering (i.e., 1984, when the original 128K Macintosh with 9" monochrome screen was shipped), we had two other acronyms:

WYSLPG = "Whistle Pig" (What You See Looks Pretty Good)

and the less friendly

WYGIWYD = "Wiggy Wid" (What You Get Is What You Deserve)

:)

-e

[TonyM]

Mario,

Thanks for your comprehensive response.  Sorry I took time to respond. I was cognisant of most of these issues, I will review dynamically nesting and see what the difficulty is.

Here is a response from me, such that after your effort, I do not stop the conversation prematurely.

I understand the loss of information, however if a given markup does not support CSS then the markdown from HTML will not include it (and must not). The way to not loose information is to keep it, eg; retain the first markdown text (in an additional field). The fact is with a nested list driven by tiddlywiki the resulting HTML theoretically looses all the logic applied to render it. I understand that.

I have used a "copy as HTML" Browser plugin in the past and pasted into the code (HTML) window  of wordpress, and the post looks as rendered in tiddlywiki, the HTML render does not contain the widgets used to create it. However you can see how I use tiddlywiki to generate sophisticated HTML and using references to data in my wiki. I used to do the reverse with <html>copy from browser selection</html> in TWC to some, but not total success, This was to capture nicely formatted material off the internet.

In this above case the universal code is HTML, but of course I can't past it into Wikipedia/WikiMedia (need to check that), but not as wikimedia standard.

It seemed to me that if we chose HTML as a Go between markup, Then we made tools to mark it down to one or more standards, for those that like a given standard, then this could be reversed to generate HTML to display it in tiddlywiki (as we must), but each new markdown need only have this To and From HTML translation (presumable one way already exists) and you could convert from any supported markdown to any other supported markdown, with the inevitable losses if that standard does not support everything HTML can render. HTML is the supper-set, the markdowns are a Subset. In a way this will clean the markdown of unsupported features, in effect creating a static and compliant version.

Of course TiddlyWIki may allow you to reintroduce css formats and even widgets to the second markdown in your wiki, and do a full render back to view it as html, but these will only work in tiddlywiki.

What would be interesting to see is if a reversible recode from HTML <> markdown standard,  was possible, just from the information needed to describe "markdownstandard > HTML, after all this must be documented for any markdown already.

It seemed like a more rapid and reusable way to handle alternative markdowns.

But I must of course, defer to your greater experience of these things.

You have fun as well, I am :)

Regards

Tony

[PMario]

Hi Tony,

Your Ideas are all right and sensible. ...

And we do want to have "roundtrips" to and from every possible content format. ... The pandoc project is a very good example / implementaiton for this workflow. .. With the exact same problems. (That's why they did their own MD extensions..)

But

If we really want something reliable, we can't allow data loss in the used "storage format"! ...

----------------  Some more internals and background

If you dig deeper into the topic you will inevitably end up at SGML Standard Generalized Markup Language. SGML roots goes back to the 1960s ...
Simplified: it is a language / standard, to define other languages. To achieve this, you need to increase complexity quite a bit. ..

SGML was originally designed to enable the sharing of machine-readable large-project documents in government, law, and industry. Many such documents must remain readable for several decades—a long time in the information technology field

...source WikiPedia

 

You may ask: Why, not just implement SGML as our storage format and all problems solved. .... IMO becasue our heads would explode. At least mine ;) SGML is really hard and absolutely not human friendly by design. ... see the quote from above ;)

That's why several "easier to handle" dialects have evolved: One of them is XML... (still clunky and human unfriendly)

Once there was an attempt to force XML on the web with XHTML. ..But the rules where so strict, that the browsers that implemented it broke the existing web. .. The project blew up in front of their face ...

So we ended up with HTML5 .... Which is a standard that is "just good enough", to be easy to use by machines and OK by humans.

----------------

As you found out HTML is a relatively good format to store "structured" text.  see: -> relatively. ... Because it was defined for a very specific use-case. ... "The Web"

TiddlyWiki actually mis-uses it, to store tiddlers in HTML format. ... If you open a tiddlywiki.hmtl file you can find elements like this. (scroll down to line ~10.000++)

<div author="JeremyRuston" core-version="&gt;=5.0.0" dependents="$:/themes/tiddlywiki/snowwhite" description="Centralises the story river" name="Centralised" plugin-type="theme" title="$:/themes/tiddlywiki/centralised" type="application/json" version="5.1.16-prerelease">
<pre>{
html-escaped content there < is encoded as &lt; and > is &gt; ....
</pre>
</div>

This is the system tiddler: $:/themes/tiddlywiki/centralised  were I replaced the content with a 1 liner.

At TW startup this conent is transfered into something much more useful for for javascript developers. -> JSON (JavaScript Object Notation)... It's easier to handle, because it is already part of the js-language itself. No bulky 3rd party libraries are needed. JS itselfe is just fine.

The cool thing with JSON is, it can also describe structues very well, in a programmer friendly / readable format.

http://www.jsonml.org/ has some examples, where an html table is described as JSON. ... Web programmers love JSON, because it is "just good enough" to be extremely useful. Programs can handle json descriptions very well.

The purpose of JsonML is to provide a compact format for transporting XML-based markup as JSON which allows it to be losslessly converted back to its original form.

source jsonml.org

IMO the page is outdated, and there are probably newer projects, but the idea behind it, is the right one.


TiddlyWiki uses JSON to convert

• tw-syntax -> into the parse-tree, the
  
• parse-tree is translated into -> the widget-tree
• widget-tree is translated into -> HTML output

If you open: https://tiddlywiki.com/prerelease/  and copy the following text into the tiddler.

* line 1
* element 2

Open the preview panel

and select parse-tree;

Mode:block

[
    {
        "type": "element",
        "tag": "ul",
        "children": [
            {
                "type": "element",
                "tag": "li",
                "children": [
                    {
                        "type": "text",
                        "text": "line 1"
                    }
                ]
            },
            {
                "type": "element",
                "tag": "li",
                "children": [
                    {
                        "type": "text",
                        "text": "element 2"
                    }
                ]
            }
        ]
    }
]

For this example the widget-tree looks similar, but if you enter {{HelloThere}}  you'll get a huge difference between parse and widget-tree. Those *-tree like formats are relatively easy to handle by algorithms. ...

The problem with the TW parse-tree at the moment is, that it "looses information". May be, because of performance reasons. Also implementation speed is faster, and complexity is much less, if you skip functions.

So we actually can't use it, as a storage format. ...

BUT

If we would solve that problem, we could transform TW-syntax <-> Markdown <-> HTML <-> TW-syntax <-> [you name it]

Such an itermediate storage fromat would be called an AST Abstract Syntax Tree. While ASTs are known for programming languages, the mechanism we need is the same. Our parse-tree and widget-tree basically are ASTs...

Some transformation paths exist and some _don't_. eg:

 - tw-syntax -> AST -> HTML ... exists

 - HTML -> AST -> tw-syntax ... doesn't exist

hope that helps

have fun!

mario

[PMario]

Hi,

I forgot to mention:

 - tw-syntax -> AST -> HTML ... exists

 - HTML -> AST -> tw-syntax ... doesn't exist

If we use the AST as the tw-storage format, as a JSON file, we could use this workflow

 - AST -> HTML
 - AST -> tw-syntax
 - AST -> any syntax
 - any-syntax -> AST

nice ... right?

-m

[@TiddlyTweeter]

PMario
... we could transform TW-syntax <-> Markdown <-> HTML <-> TW-syntax <-> [you name it]

No. Better.

Any Syntax -> Bits of that syntax TW supports/can translate already. -> Then convert Unsupported Syntax -> HTML

Josiah

[@TiddlyTweeter]

PMario

The pandoc project is a very good example / implementaiton for this workflow. .. With the exact same problems.

Right.

Yet encouraging PANDOC to support TiddlyWiki format would also be a decent thing.

Just saying.

Josiah

[PMario]

On Monday, December 18, 2017 at 2:19:39 PM UTC+1, @TiddlyTweeter wrote:

Yet encouraging PANDOC to support TiddlyWiki format would also be a decent thing.

The TW transclusion mechanism, is unique to TW. Most other system don't even have an idea about mechanisms, that we take for granted.

-m

[@TiddlyTweeter]

It would be good for them to grasp transclusion better. Though some of the Pandoc supported formats hold links to urls that are towards that.

In any case, I'm pointing at conversion FROM other formats TO TW to ease migration.

TW's flex as already a enough of a super-set markup that its likely very possible for Pandoc easy IMO. At least for the majority of  simpler Wiki markups it supports.

TW to other formats, I admit, may be much more difficult. But such limits never stopped Pandoc before did it?

- j