new parser rule proposed: dot-paragraph

[PMario]

We hijacked an other thread, which shouldn't happen. So I wanted to start over here:

On Tuesday, February 25, 2020 at 11:13:54 PM UTC+1, TonyM wrote:

Mario,

You are a wizard with this kind of thing, I hope you could assist me. I ask now because your above solution is slow close to what I have asked for for some time.

• Is there a possibility you could generalise this so others could modify it to produce their own parser/pragma?
• If you see in the core where * # ; : are defined it seems to me it should be trivial to add another special character and set html tags but it is not.
• could you make a standard plugin template we can modify to set the lead character and the html tags to apply?
• I really want the leading period "." to trigger a html paragraph around multiple sentences until we get to end of line (line break)
• However I see value if I could build others 

Thanks if this is possible

Tony

 

On Wednesday, February 26, 2020 at 6:37:01 PM UTC+1, PMario wrote:

On Tuesday, February 25, 2020 at 11:13:54 PM UTC+1, TonyM wrote:

..

You are a wizard with this kind of thing, I hope you could assist me. I ask now because your above solution is slow close to what I have asked for for some time.

...

• I really want the leading period "." to trigger a html paragraph around multiple sentences until we get to end of line (line break)

I have seen this request, but I don't understand it. I think the first time it came up with the "single linebreak" - "hard linebreak" in paragraphs discussion.

In my thinking having a dot at the start of the wikitext line is annoying. I'd rather go with some spaces, which you can sometimes see.

Also inserting a dot at the beginning of a line is the same manual work as hitting [Enter] or even 2 times Enter, to create a real wikitext paragraph.

The second problem I have is, that a "hard paragraph" will create HTML code like this:

<p>

<p>This text is wrapped because there was a "dot" at the beginning of the line</p>

</p>

Which is ugly, since TW already suffers from "divitis". So imo we shouldn't add "paragraphitis" if we can avoid it ;)

Mario

On Thursday, February 27, 2020 at 12:27:36 AM UTC+1, TonyM wrote:

Mario,

I would expect a dot at the beginning to simply do this.

<p>This text is wrapped because there was a "dot" at the beginning of the line</p>

To me this is a very valuable tool for use in my own wikis, not only for massaging imported content but helping me structure notes as I compose them. It would be easy to remove all leading periods if required and if we copy the resultant html for reuse it is good for sharing.

I want a bespoke setting here, not a change to the fundamental markup standards, and if possible offer the option for other bespoke leading charater markup. eg someone may like to indicate sections or chapters etc... with their own markup, basically allowing a markup character to be defined use standard html tags, tags not already catered for in wikitext or widgets.

Perhaps you are not interested but It would help me (and I believe others would like it) a lot. But the question is why is the addition of such bespoke markup not possible?, from what I can see programmatically it should be trivial, perhaps even entries in a data tiddler to define them. 

With an Editor toolbar  button I can turn all lines into paragraphs buy, select all, click, prepending the lines with a period.

One way to test the value is to copy a big block of multi paragraph lorum ipsum then use a editor toolbar item to select all lines and prepend with ";"

Empty `<P>` tags collapse to one blank line between paragraphs and indicating the "pre-wrapped" line starts helps and editor review the layout, especially for content sourced elsewhere. Which often contains multiple blank lines, using this they all collapse to a single blank line when rendered.

In rapid note taking I use ";" and ":" a lot, and this would complement these by providing automatically managed paragraphs. There is no non-bold equivalent to ";". 

If you have another character or markup pattern in mind happy to consider. But I realy, realy, want this, given my personal organiser, my data imports, note taking in courses etc.. Material I add tends to go nowhere but in my own personal wikis.

And yes, for some people this will satisfy there frustration with the line breaks.

Regards

Tony

On Thursday, February 27, 2020 at 2:52:36 AM UTC+1, PMario wrote:

On Thursday, February 27, 2020 at 12:27:36 AM UTC+1, TonyM wrote:

Perhaps you are not interested but It would help me (and I believe others would like it) a lot.

I wouldn't discuss it, if I wouldn't be interested. I just need to understand it in a broader context.

 

But the question is why is the addition of such bespoke markup not possible?

It is possible -- and it should create html markup that is in line with the html spec. The spec is important for eg: accessibility tools like screen readers. So our wikitext should produce output, that doesn't confuse those tools.

 

, from what I can see programmatically it should be trivial, perhaps even entries in a data tiddler to define them. 

Not really trivial. ... The elements you pointed out, are all part of the list-parser. So they create 2 elements.

eg:

* test ... creates an <ul> wrapper and <li> elements.

; test ... creates a <dl> wrapper and <dt> elements.

So if the leading-dot is added, in a naive way it creates nested paragraphs, which is outside the html specs.

I did test it with a <section> wrapper and <p> elements which seems to be OK.

eg:

.test 1

.test 2

will give us

<section>

<p>test 1</p>

<p>test 2</p>

</section>

There may be other possibilities, to add this behaviour, but I didn't think about that ... yet.

-mario

 

On Thursday, February 27, 2020 at 8:55:43 AM UTC+1, TonyM wrote:

Mario

Thanks. That explains a lot more. Would it be ok to nest it in a div, perhaps the div could have a class so styles could be applied e.g. the paragraph style, thus the transition to multiple paragraphs could be changed e.g. 1.5 line space between paragraphs.

Regards
Tony

 

On Thursday, February 27, 2020 at 9:14:50 AM UTC+1, TiddlyTweeter wrote:

Ciao TonyM

Here is my naive comment. I'm still not fully clear what the outcome you looking for is meant to "look like".

But I think CSS might be a workable solution. Either in a div container or a CSS rule that detects markup?

Best wishes

TT

[PMario]

On Thursday, February 27, 2020 at 8:55:43 AM UTC+1, TonyM wrote:

Mario

Thanks. That explains a lot more. Would it be ok to nest it in a div, perhaps the div could have a class so styles could be applied e.g. the paragraph style, thus the transition to multiple paragraphs could be changed e.g. 1.5 line space between paragraphs.

I did an other test, using the headings-parser as a starting point. The code is much much simpler than creating lists.

And there shouldn't be problems with the spec.

-m

[PMario]

Did some more experiments and found out, the "dot" as an indicator is problematic.

eg: I want to be able to create indented paragraphs, which is only possible at the moment with lists and special CSS

So, I used the tick ´ as an indicator:

´.test some text ... will create

<p class="test tc-p1">some text<p>

´´ some more text ... will create

<p class="tc-p2">some more text<p>

Which should allow us to create text like this:

some text

some more text

The line spacing can be done with CSS using .tc-p1, .tc-p2 or tc-p3 ... Maximum 3 levels should be possible.

If we use a dot, we loose the "user defined class" see ´.test at the very first example.

I'm not so happy with the ´ tick or the dot. ... What if we could use tabs for indentation?

What do you think?

-mario

[TonyM]

Mario,

Some progress here, thanks.

Codemirror allows indents/with tabs, but the render does not honor them, but here I want no indent, just a paragraph.

In my case I already use ":" to indent and make an effective paragraph, multiple indents as well, as you know ";" is bolded but is does what I want otherwise, but since I we are dealing with paragraphs it seems best to use the html P tag. At one point I removed the bold with ;.p where p was a class name that unbolded it. But ;.p is surprisingly distracting. 

I notice that using period as the second character to ;.classname and :.classname works, so theoretically ..classname could also work while rendering the text in an un-indented paragraph. But I am not wedded to period, however it is nicely easy to access on the keyboard (no shift) and it is often so unassuming if it was at the beginning of a paragraph (not wikifield) it is almost unnoticeable.

One idea is what if we used an invisible character? eg; non breaking space, one that is otherwise ignored?, the only question is how do we get a single key entry, and ideally display while editing raw text.

I also created .q and .a classes to highlight Questions and their Dnswers differently and this also allowed the Questions and answers in a long text to be listed in a footer. With matching I listed unanswered Questions. You can see here, this provides an ad hoc way for an author to add annotations, classification and formatting details to their content as they write, a bit like a home grown shorthand, which is what markup is all about. I may also add when trying to take notes in a class or watching a video the more easy tricks the better.

Regardless if such "customisations" were hackable more can be done. Inventions we can't imagine.

• One I can imagine - an indicator to later excise, and transclude or Link to that transclusion (needs beginning and end, or default to end of line/paragraph)

Regards

Tony

[PMario]

On Thursday, February 27, 2020 at 12:16:08 PM UTC+1, TonyM wrote:

Codemirror allows indents/with tabs, but the render does not honor them, but here I want no indent, just a paragraph.

I know. ... but I do want a generic solution, that may fit more usecases. 

We could create some (2) buttons that can create tabs without codemirror, just for indenting paragraphs.

In my case I already use ":" to indent and make an effective paragraph,

No you create a Description Details element ... that's a different thing. ... You are basically "misusing" it, because there is no visual difference.

But there is a semantic difference. eg: If you copy paste the html code into a different app, that tries to make sense out of the html structure, it will be confused.

  

multiple indents as well, as you know ";" is bolded but is does what I want otherwise, but since I we are dealing with paragraphs it seems best to use the html P tag. At one point I removed the bold with ;.p where p was a class name that unbolded it. But ;.p is surprisingly distracting. 

Yea distraction was the main point why I didn't like the . dot at the start of the line. ... But you are right. If there are more elements like ;.p it is even worse. ...

;.p looks like an emoji  ;p

I was thinking about the ´ tick, because it's also a small character. ... I don't know, what's better.

 

I notice that using period as the second character to ;.classname and :.classname works, so theoretically ..classname could also work while rendering the text in an un-indented paragraph.

You are right it should work for un-indented paragraphs. If ... means paragraph level 3, it won't work for ...test  paragraph level 2, since the parser can't recognize the difference between level 2-with-class and level 3

 

But I am not wedded to period, however it is nicely easy to access on the keyboard (no shift) and it is often so unassuming if it was at the beginning of a paragraph (not wikifield) it is almost unnoticeable.

... keyborad layout and character access is a thing. ... that's right.

 

One idea is what if we used an invisible character? eg; non breaking space, one that is otherwise ignored?, the only question is how do we get a single key entry, and ideally display while editing raw text.

I'm not a big fan of invisible "formatting", but for some things, we don't have a choice. ... The [tab] at the beginning of a line will be visible in edit mode. ... but it would look the same as if you did 8 spaces. ... But nobody makes 8 spaces at the start of a line. right ;)

 

I did also experiment with a new hard-linebreak inline-parsing-rule. using <space><space><enter> ... that will be translated to <br/>

It works nicely, but has very high potential for misuse, which will cause a lot of maintenance cost (for the users) in the long run.

I also created .q and .a classes to highlight Questions and their Dnswers differently and this also allowed the Questions and answers in a long text to be listed in a footer.

ok

 

With matching I listed unanswered Questions. You can see here, this provides an ad hoc way for an author to add annotations,

There is an html element for annotations. <aside>: The Aside element ,where the default ARIA role is complementary.

classification and formatting details to their content as they write, a bit like a home grown shorthand, which is what markup is all about.

I'm not sure what you mean with classification. ... but it could be <dt>: The Description Term element

 

I may also add when trying to take notes in a class or watching a video the more easy tricks the better.

That's right. .. But it would be nice, if the easy tricks also create valid html output.

 

Regardless if such "customisations" were hackable more can be done. Inventions we can't imagine.

• One I can imagine - an indicator to later excise, and transclude or Link to that transclusion (needs beginning and end, or default to end of line/paragraph)

That's right, but if it is hackable it may cause a lot of incompatibility between wikitext of user A and user B. I think, the wikitext markup should be globally interchangeable.

have fun!

mario

[TonyM]

Mario,

Whilst I agree with this, 

wikitext markup should be globally interchangeable

 

I do not think its enforcement should limit the flexibility of an editor in their own wiki. If content is not designed for sharing, or only shared to pre-configured wikis such as personal notes, then it is simply not sharable.

The plan should be to have sufficient richness in the markup to support interchangeability, but this comes with time, new facilities and features could be developed by editors for their own use and compelling solutions moved into global wiki markup in the fullness of time.

It makes me wonder if we need a kind of "escape character", behind which custom markup can be defined, but ignored if not handled by the recipient wiki. A field and its content could cause such an alternate markup or the type field. Perhaps copy to clipboard, and copy, export could include an option to ignore especially escaped markup.

Just some thoughts

Tony

[PMario]

On Thursday, February 27, 2020 at 11:16:15 PM UTC+1, TonyM wrote:

...

It makes me wonder if we need a kind of "escape character", behind which custom markup can be defined, but ignored if not handled by the recipient wiki. A field and its content could cause such an alternate markup or the type field. Perhaps copy to clipboard, and copy, export could include an option to ignore especially escaped markup.

hmmm, I don't think so. .. Prose wikitext is defined in a way, that it should make sense for most humans, even if the plain text isn't wikified at all.

eg: There is something //strange// going on in this text.

At the moment, if the parser doesn't understand any wikitext formatting, it displays it as text. So the user "can/should" make sense of it. I think this mechanism should stay.

"Eating" unknown markup, for me is like "hiding information". ... You probably know, I'm against hiding information.

Advanced markup like widgets will display an error message: eg: <$unknown/> ...Undefined widget 'unknown' ... IMO that's OK.

We already have the \rules only/except pragma at the beginning of tiddler texts.

We may be able to create a \rules use <regexp> dot-paragraph ... Which can do something sensible, if the "dot-paragraph" inline parser is missing.

eg: hide the leading dot, but show a "dot-paragraph parser is missing" message somewhere. ... A more intelligent solution would be an "import the missing stuff" dialogue ;)

-mario

[PMario]

On Tuesday, March 3, 2020 at 10:35:53 AM UTC+1, PMario wrote:

...

We may be able to create a \rules use <regexp> dot-paragraph ... Which can do something sensible, if the "dot-paragraph" inline parser is missing.

eg: hide the leading dot, but show a "dot-paragraph parser is missing" message somewhere. ... A more intelligent solution would be an "import the missing stuff" dialogue ;)

In an ideal world, TW would know, where to lookup the address to safely import the missing plugin.

-m

[TonyM]

Mario

I see what you mean by hidden.

Sounds good. What is the best way to proceed for a dot-paragraph?, and ideally so others can be added if desired.

Regards

Tony

[TiddlyTweeter]

TonyM wrote:

In my case I already use ":" to indent and make an effective paragraph,

 

PMario wrote: 

No you create a Description Details element ... that's a different thing. ... You are basically "misusing" it, because there is no visual difference.

But there is a semantic difference. eg: If you copy paste the html code into a different app, that tries to make sense out of the html structure, it will be confused. 

 

One of the reasons the semantics get misused quite often is that the lack of an alternative approach in markup.

It makes sense to be non-semantic because IF you can get the outcome to look like what you need then you not too bothered about the semantic level.

Its your wiki, not generalist compatability.

I DO agree that ideally its much better to stay semantic, but users will often be focused on visual outcome that serves their immediate aim.

So I think TonyM's interest in an extensible markup is very relevant both to enable his aim and keep semantic.

A comment

TT

[TiddlyTweeter]

PMario wrote:

We may be able to create a \rules use <regexp> dot-paragraph ... Which can do something sensible, if the "dot-paragraph" inline parser is missing.

Wondering if the idea here is extensible? For example ...

\rules use <regexp> three-lead-spaces

In other words, generic ability for the user to add BESPOKE RULES using regex?

IF so I think it would be excellent. It could, maybe, solve Tony's issue and simultaneously make the parsing process less obscure and practically extensible?

Very glad you are looking at it!

 

Best wishes

TT

[TiddlyTweeter]

PMario wrote:

I'm not a big fan of invisible "formatting", but for some things, we don't have a choice. ... The [tab] at the beginning of a line will be visible in edit mode. ... but it would look the same as if you did 8 spaces. ... But nobody makes 8 spaces at the start of a line. right ;)

They might :-).

The issue is the KIND of document. The point about light-weight markup is precisely to use as little markup as possible so that the text remains human readable.

Some kinds of document can have incredibly light markup because the actual written form already implies its own markup.

For instance, screenplays rely on spacing in the human readable version so parsers for them use spacing as a primary vector. An example is https://fountain.io/syntax which is an elegant space-centric variation on Markdown.

Regarding the OP, an interesting issue from my point of view, is some comments you made about concern for some kind of "rule check". I think you are right in that raw regex may do the job, but without an AST (a basic checking framework) I could foresee issues.

Anyway its very useful to open the parsing process to extension. Tony's case is a good one, I think, to test with.

Best wishes

TT

 

[PMario]

On Tuesday, March 3, 2020 at 12:43:12 PM UTC+1, TiddlyTweeter wrote:

PMario wrote:

We may be able to create a \rules use <regexp> dot-paragraph ... Which can do something sensible, if the "dot-paragraph" inline parser is missing.

Wondering if the idea here is extensible? For example ...

\rules use <regexp> three-lead-spaces

As I wrote in an earlier post, I want to be able to set classes too. like ´.test and also indent a paragraph like: ´´´ some text 

I was thinking about a possibility to use tabs to indent paragraphs. .. I would need to test <tab><tab>.test-class some text

In other words, generic ability for the user to add BESPOKE RULES using regex?

It may be possible. The biggest concern I have is, that we create a new "Tower of Babel", where TW-A wikitext isn't compatible with TW-B wikitext.

 

IF so I think it would be excellent. It could, maybe, solve Tony's issue and simultaneously make the parsing process less obscure and practically extensible?

I'm OK with extensibility. ... But it has to be in a way, that solves more problems, than it creates.

-mario

[PMario]

On Tuesday, March 3, 2020 at 1:10:14 PM UTC+1, TiddlyTweeter wrote:

...

The issue is the KIND of document. The point about light-weight markup is precisely to use as little markup as possible so that the text remains human readable.

Some kinds of document can have incredibly light markup because the actual written form already implies its own markup.

 

For instance, screenplays rely on spacing in the human readable version so parsers for them use spacing as a primary vector. An example is https://fountain.io/syntax

I have seen the discussion about fountain.io ... I think this is a completely new tiddler-type, which can be implemented as a plugin and type:   text/vnd.fountain   or something similar.

 

which is an elegant space-centric variation on Markdown.

see: https://fountain.io/faq  

>What exactly is the relationship between Fountain and Markdown?

>Fountain, formerly known as Screenplay Markdown, is inspired by John Gruber's super cool Markdown language,

>and uses some of its conventions, but Fountain is not Markdown.

>There's no use in converting Fountain to Markdown or the reverse.

>Markdown simply pointed the way to a rich but simple experience of creating beautiful,

>functional documents using only human-readable plain text.

-m

[TiddlyTweeter]

PMario

Right. The Fountain parser was strongly inspired by Markdown, but is actually much more minimalist.

The approach is only suited to highly structured documents where the "human readable" format is already "its own markup".

The only time in Fountain you use explicit markup is when a rule is broken. 

I could imagine extension of the approach to "Legal" documents that have a similar strict human readable format.

TT

[PMario]

On Tuesday, March 3, 2020 at 10:44:01 AM UTC+1, TonyM wrote:

...

Sounds good. What is the best way to proceed for a dot-paragraph?, and ideally so others can be added if desired.

I just found out, that we can't use a "global" dot-paragraph rule since it messes with the TW stylesheet parsing. eg:

A tiddler type: text/vnd.tiddlywiki, which is tagged: $:/tags/Stylesheet contains:

.test { css rules }

This will be translated into a paragraph, which will invalidate the CSS configuration.

The core CSS isn't affected, because it uses:

\rules only filteredtranscludeinline transcludeinline macrodef macrocallinline macrocallblock

So it will be very error prone for most users. I'm 100% sure, that we can count users, which would do it right on 1 hand.

-m

[TiddlyTweeter]

PMario wrote:

On Tuesday, March 3, 2020 at 10:44:01 AM UTC+1, TonyM wrote:

...

Sounds good. What is the best way to proceed for a dot-paragraph?, and ideally so others can be added if desired.

I just found out, that we can't use a "global" dot-paragraph rule since it messes with the TW stylesheet parsing. 

What simple, single, characters are left for Tony?

Also, would ".." work?

TT

[TiddlyTweeter]

PMario wrote:

I have seen the discussion about fountain.io ... I think this is a completely new tiddler-type, which can be implemented as a plugin and type:   text/vnd.fountain   or something similar. 

Right. Ideal. 

FYI for private use I use raw regex made via bj's content pre-parser to make a fully workable Fountain (crude) pre-parser. 

It works fine for me.

But I won't make it public because it would need an AST to capture basic errors to prevent confusion. Not a good idea to do it without one.

To be able to convert that into a TW parser I'd need understand JS, which, unfortunately, I don't.

TT

[PMario]

I did experiment with the ´ tick ... acute accent like so:

´´´ for level 3 paragraph _and_

´<tab><tab> also level 3

I personally like the second version, since it creates plain text, which shows a human readable indentation. .. but 2 tabs in a default text editor create a 16 character wide indent, which is ugly.

´<tab><tab>.test some text  ... will create a paragraph with class="test" ... Even if it is consistent, this construction looks strange ...

´´´.test some text ... feels better, since it is already known.

The tick, could be a problem with some keyboard layouts. ...

A .<space> would work but would probably loose the level _and_ class setting, or make the whole parser logic much much more complicated.

It's not ideal.

-m

[TonyM]

Mario,

I just found out, that we can't use a "global" dot-paragraph rule since it messes with the TW stylesheet parsing. eg:

 

Maybe this the point.  Such bespoke parsing could be designed to only operate for a given tiddler type or tiddlers with the pragma set to apply the "local" parsing rule.

As an example In the case of the dot paragraph The key places I would expect to use it would be 

• when pasting text from other sources and transforming to a more readable rendering
• when typing content in class or capturing ideas brain to text

In these examples I would be happy to introduce pragma to support this

\bespokemarkup notes
\bespokemarkup dotparagraphs

where notes defines standard markup + some rules defined in "notes".

• Although a way to apply it from the view template or another conditional method would be nice.
• The point being you would not do this on a tiddler containing CSS unless it provides some value.
• In tiddlywiki a local rule can become global if you apply it everywhere but I think bespoke markup could be constrained to nominated tiddlers.

The point here is I / We are free to use period and I do not need to go looking for other symbols.

Regards

Tony

[PMario]

On Tuesday, March 3, 2020 at 3:15:39 PM UTC+1, TiddlyTweeter wrote:

PMario wrote:

I have seen the discussion about fountain.io ... I think this is a completely new tiddler-type, which can be implemented as a plugin and type:   text/vnd.fountain   or something similar. 

Right. Ideal. 

FYI for private use I use raw regex made via bj's content pre-parser to make a fully workable Fountain (crude) pre-parser. 

@TiddlerTweeter

Do you have a link to the library?

-m

[PMario]

On Wednesday, March 4, 2020 at 12:01:57 AM UTC+1, TonyM wrote:

Mario,

I just found out, that we can't use a "global" dot-paragraph rule since it messes with the TW stylesheet parsing. eg:

 

Maybe this the point.  Such bespoke parsing could be designed to only operate for a given tiddler type

I don't really want to go that route. There is a pending issue by Jeremy from 2014 and a PR by BJ also from 2014.

http://bjtools.tiddlyspot.com/ contains some related stuff from 2016 which uses MIME-type parameters like: "text/vnd.twbase;flexibility=MDL ... The problem with such params is, that they need to be fully specified at the time, they are registered. ... Which will probably never happen with TW.

 

or tiddlers with the pragma set to apply the "local" parsing rule.

I think, that would be the faster way. ... I can't use \rules use xxxx since it would need to overwrite :/ the core wikiparser.js, which is the main wikitext parsing function.

As an example In the case of the dot paragraph The key places I would expect to use it would be 

• when pasting text from other sources and transforming to a more readable rendering
• when typing content in class or capturing ideas brain to text

In these examples I would be happy to introduce pragma to support this

\bespokemarkup notes
\bespokemarkup dotparagraphs

where notes defines standard markup + some rules defined in "notes".

yup...

Since I have to use a new pragma anyway eg: \parserule dotparagraph <setting> ... or something similar, there will be much more new code involved, as I wanted it to be :/

• Although a way to apply it from the view template or another conditional method would be nice.

I think, that's not possible since the ViewTemplate depends on the parser settings already. Parsing is done way ahead of rendering. Everything, which needs to be decided at viewing time needs to be a widget.

• The point being you would not do this on a tiddler containing CSS unless it provides some value.

That's right. I found out about the problem, while experimenting and suddenly some "custom CSS" ceased to work.

• In tiddlywiki a local rule can become global if you apply it everywhere but I think bespoke markup could be constrained to nominated tiddlers.

I "kind of like" this downside, since it will add some info eg: \parserule ...  at the top of the tiddler, that won't be "parsed" if copy pasted to a different TW. So users will have a clue, what to do. Users also have the choice to ignore and delete the pragma, if they want.

 

The point here is I / We are free to use period and I do not need to go looking for other symbols.

:) ... I personally didn't like the leading dot BUT I do see, that having it that way may be essential for other users. ... I'll try to make it configurable.

For the time being I'll attach an experimental zip file here with a global hack, that uses dot-space, which  shouldn't have side effects. So you can play with it.

Attachment will follow soon.

-mario

[PMario]

On Wednesday, March 4, 2020 at 12:01:57 AM UTC+1, TonyM wrote:

..

In these examples I would be happy to introduce pragma to support this

\bespokemarkup notes
\bespokemarkup dotparagraphs

where notes defines standard markup + some rules defined in "notes".

What does a notes do ?

-m

[PMario]

Here's the test wiki.

have fun!

mario

[TonyM]

Mario,

Thanks for you work I am having a look at the example now. 

Opening the attached wiki in firefox the .paragraph appear not work out of the box, I found the tiddler test-dot-paragraph and space-space-enter. 

So yes "dot space" triggers a paragraph.

• You have extended this idea further and it is interesting, including the indentation and classes 
• I tested it on a Lorum ipsum with no `<br><br>` and it works well to re/establish paragraphs. 
• It also works well to create a paragraph where there was not one <entre>.<space>
• Indented paragraphs are great. 
• So a complemetary editor toolbar to prefix lines with <dot><space> or <dot> will be good

In this example be is a simple line, or a full paragraph <dot><paragraph> is a wonderful complement to ";" ":" and the * and #

. Paragraph
. Para 2
. Para three

. First
.
.
.
. Second

If dot space is applied to all lines and they include blank lines I am getting the desirable result from the html paragraph tag, the blank lines collapse into one paragraph break.

This is helpful when pasting content from elsewhere because it is often full of multiple blank lines, eg copy off a webpage.

Would it be possible to still trigger this markup with <dot>any character? This was the original desire? Can your extended markup still accommodate this? Although I see that the space may allow to load it with more possibilities.

• I am yet to grasp the use of trailing "space space enter"
• Is there a reason why we could not issue a <br> for <dot><space><space>, or even turn the previous line into a paragraph as a result.

I have also started testing if and how the previous markup method work with and without the ".paragraph" to introduce css the ".classname" eg ";.classname "

You asked

What does a notes do ?

It was only an example of another bespoke markup, I imagined it boxing a div and providing dl dt tags to highlight a specific note.

Given your work so far and this particular case of providing markup to use `<p>` html tags makes me ask if it makes sense to also develop markup for `<section>`, `<div>` and `<span>` to which one can introduce css. I am thinking of what markup could be helpful for rapid authorship, note taking and very simple layout.

Thanks again,

I will give another example text once I have explored this more deeply.

eg: place this in a dot paragraph wiki and see how beautiful the result with little effort

. Class notes Feb 23 2020
;Subject: Analysis
. Analysis and Synthesis.
.. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Volutpat ac tincidunt vitae semper quis. Congue eu consequat ac felis donec et odio pellentesque diam. Dictumst vestibulum rhoncus est pellentesque elit ullamcorper. Orci nulla pellentesque dignissim enim sit. Feugiat scelerisque varius morbi enim. Malesuada fames ac turpis egestas maecenas pharetra. Dolor sed viverra ipsum nunc aliquet bibendum enim facilisis gravida. Gravida neque convallis a cras semper auctor neque vitae tempus. Lobortis scelerisque fermentum dui faucibus in. Faucibus scelerisque eleifend donec pretium vulputate sapien nec sagittis. Cursus in hac habitasse platea dictumst. Sollicitudin nibh sit amet commodo nulla. Elementum facilisis leo vel fringilla est ullamcorper eget nulla facilisi. Felis imperdiet proin fermentum leo. Diam maecenas sed enim ut sem viverra.
;Analysis
. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Volutpat ac tincidunt vitae semper quis. Congue eu consequat ac felis donec et odio pellentesque diam. Dictumst vestibulum rhoncus est pellentesque elit ullamcorper. Orci nulla pellentesque dignissim enim sit. Feugiat scelerisque varius morbi enim. Malesuada fames ac turpis egestas maecenas pharetra. Dolor sed viverra ipsum nunc aliquet bibendum enim facilisis gravida. Gravida neque convallis a cras semper auctor neque vitae tempus. Lobortis scelerisque fermentum dui faucibus in. Faucibus scelerisque eleifend donec pretium vulputate sapien nec sagittis. Cursus in hac habitasse platea dictumst. Sollicitudin nibh sit amet commodo nulla. Elementum facilisis leo vel fringilla est ullamcorper eget nulla facilisi. Felis imperdiet proin fermentum leo. Diam maecenas sed enim ut sem viverra.
;Synthesis
. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Volutpat ac tincidunt vitae semper quis. Congue eu consequat ac felis donec et odio pellentesque diam. Dictumst vestibulum rhoncus est pellentesque elit ullamcorper. Orci nulla pellentesque dignissim enim sit. Feugiat scelerisque varius morbi enim. Malesuada fames ac turpis egestas maecenas pharetra. Dolor sed viverra ipsum nunc aliquet bibendum enim facilisis gravida. Gravida neque convallis a cras semper auctor neque vitae tempus. Lobortis scelerisque fermentum dui faucibus in. Faucibus scelerisque eleifend donec pretium vulputate sapien nec sagittis. Cursus in hac habitasse platea dictumst. Sollicitudin nibh sit amet commodo nulla. Elementum facilisis leo vel fringilla est ullamcorper eget nulla facilisi. Felis imperdiet proin fermentum leo. Diam maecenas sed enim ut sem viverra.

Regards

Tony

[PMario]

On Wednesday, March 4, 2020 at 11:16:17 PM UTC+1, TonyM wrote:

Would it be possible to still trigger this markup with <dot>any character? This was the original desire?

This will follow once I did the \parserule dotparagraph pragma. ... It's a bit more complicated as it seems.

The \rules except/only pragma is only capable of removing wiki rules after the parser for a tiddler is instantiated. What we need for \parserule is: "_add_ a new rule and make sure it is higher priority as others" mechanism. Which doesn't exist atm. Most of the functions exist in the core, but I need to make it useable in a different way. Plus keep backwards compatibility, if possible.

 

Can your extended markup still accommodate this?

I think so.

 

Although I see that the space may allow to load it with more possibilities.

• I am yet to grasp the use of trailing "space space enter"

That's compatible to the new "CommonMark" markdown parser spec for "hard linebreaks". So if someone copies a markdown paragraph to TW, they should see the same result.

see: https://spec.commonmark.org/0.29/#hard-line-breaks .. (I just found out, that it's not 100% compatible, but good enough to start with)

This rule is also intended to work with TW syntax like:

* test  
next line should be part of the li-element.

If you copy the above code into tiddlywiki.com and your experimental wiki, you'll get a different result.

This is intended, to be a global modification and is intended to work for every list-like wikitext. eg: *#:;. and so on.

• Is there a reason why we could not issue a <br> for <dot><space><space>, or even turn the previous line into a paragraph as a result.

I think, it should be possible to "remove" empty lines or empty lines preceded by a dot from the generated DOM. At the moment empty p-tags are created. They are not visible, but they are there, which isn't nice.

 

I have also started testing if and how the previous markup method work with and without the ".paragraph" to introduce css the ".classname" eg ";.classname "

ok

You asked

What does a notes do ?

It was only an example of another bespoke markup, I imagined it boxing a div and providing dl dt tags to highlight a specific note.

Yea, I did think about that. ... But there will be a steeper learning curve, the more elements we add or change from the original behaviour.

 

Given your work so far and this particular case of providing markup to use `<p>` html tags makes me ask if it makes sense to also develop markup for `<section>`, `<div>` and `<span>` to which one can introduce css. I am thinking of what markup could be helpful for rapid authorship, note taking and very simple layout.

<section> and <div> should be no problem, because they allow "flow content" as "Permitted Content" see the link.

<span>'s are inline elements only. They only allow "phrasing content" (same as paragraphs). It will produce invalid html code if used at the start of the line like: * or # or .

I was thinking about an <aside> (note) using > at the beginning of the line.

 

eg: place this in a dot paragraph wiki and see how beautiful the result with little effort

.....

Yea, It looks nice. The html code is OK-ish, but there is potential for improvements. eg:

TW knows about a mechanism, to manipulate heading levels for transcluded tiddlers. see: https://tiddlywiki.com/#tv-adjust-heading-level%20Variable If it would be possible to set this veriable with a \pragma or \define and the rendering mechanism would accept it, it would be possible to start your notes like this:

\startheading 5

! Class notes Feb 23 2020
!! Subject: Analysis
. Analysis and Synthesis.


The html output could be

<h5>Class notes Feb 23 2020</h5>
<h6>Subject: Analysis<h6>

<p>Analysis and Synthesis.</p>


If there would be a proper CSS setting, it would be the same manual work, but produce much better html code, from a semantic point of view.

Having a closer look at your example code and the description:

If dot space is applied to all lines and they include blank lines I am getting the desirable result from the html paragraph tag, the blank lines collapse into one paragraph break.

This is helpful when pasting content from elsewhere because it is often full of multiple blank lines, eg copy off a webpage.

I also think, the manual workflow could be improved, using less time and getting improved html output. ... BUT I'll need to think about this. ... I think about new editor-shortcuts,

That can speed things up segnifficantly.

-m

PS: I'll need some time, since the last storm knocked over some of my trees. I have to remove them.

[PMario]

On Thursday, March 5, 2020 at 10:47:08 AM UTC+1, PMario wrote:

This rule is also intended to work with TW syntax like:

* test  
next line should be part of the li-element.

If you copy the above code into tiddlywiki.com and your experimental wiki, you'll get a different result.

needed to modify the code block in the post above.

-m

[TonyM]

Mario,

Thanks for your focus on this. I have read this response in detail and appreciate you knowledge and effort. 

A few Questions and comments

• With the use of pragma to introduce or exclude various parsing rules can we choose to apply it to a given tiddler through the import variables in a conditional list withing the view template?

Although I see that the space may allow to load it with more possibilities.

• I am yet to grasp the use of trailing "space space enter"

That's compatible to the new "CommonMark" markdown parser spec for "hard linebreaks". So if someone copies a markdown paragraph to TW, they should see the same result.

see: https://spec.commonmark.org/0.29/#hard-line-breaks .. (I just found out, that it's not 100% compatible, but good enough to start with)

Ahh, I appreciate your keeping these larger parsing issues in mind.

 

This rule is also intended to work with TW syntax like:

* test  
next line should be part of the li-element.

Are you suggesting this?;

• test  

next line should be part of the li-element.

should become 

• test  next line should be part of the li-element.

If there is a trailing "space space enter"?

Please see some sample wiki text below that illustrates some of my observations

* List
:notes
* list more
:more notes

# List
:notes
# list more
:more notes

**item 1 no matching"*" Eroniouse dot to left
**item 2

<ul><ul>
<li>item 1 no matching"*" no eroniouse dot to left</li>
<li>item 2</li>
</ul></ul>

:::*List items indented
:::*list

With the current Wikitext rules I observe the following issues with bullets

• I often follow one or more * with one or more : colons to provide additional lines under a bulleted list items however this screws up ordered lists, in many editors a <ctrl-enter> inserts a line break that does not stop the li element. Perhaps there is a better way
• We can use ":" as an indent but perhaps this is breaking some rule?
• To add additional line under a `<ul><li>`
• To indent before `<ul><li>`

I think, it should be possible to "remove" empty lines or empty lines preceded by a dot from the generated DOM. At the moment empty p-tags are created. They are not visible, but they are there, which isn't nice.

 

Yea, I did think about that. ... But there will be a steeper learning curve, the more elements we add or change from the original behaviour. 

I think my point here about notes etc.. rather than add elements, allow bespoke elements to be added and where possible provide an extensible element to keep it as simple as possible. There would be a fair argument that a high percentage of content in tiddlywiki's remain within the wiki in which it is written. So I hope that universalising markup so that the content is freely sharable, does not reduce the choices of the author to introduce bespoke methods. But of course such bespoke markup will appear but not evaluated in exported text. 

<section> and <div> should be no problem, because they allow "flow content" as "Permitted Content" see the link. 

<span>'s are inline elements only. They only allow "phrasing content" (same as paragraphs). It will produce invalid html code if used at the start of the line like: * or # or .

I was thinking about an <aside> (note) using > at the beginning of the line.

I have some more thoughts here, I will post a separate response.

\startheading 5

This is a nice idea, how will it effect transclusions, Actually applying this to subtiddlers transcluded at a different heading level could be helpful, again can we programmatically apply pragma? 

I also think, the manual workflow could be improved, using less time and getting improved html output. ... BUT I'll need to think about this. ... I think about new editor-shortcuts,

That can speed things up significantly. 

 

I am all for this and happy to contribute new editor toolbar buttons, because that is well within my skill level. The idea of a tool to add leading periods to every line, but also one to remove multiple empty lines as suggested by you. Actually I can see a set of optional editor toolbar buttons dedicated to "Cleaning" tiddlywiki content. Or on CSS to to minimise etc...

-m

PS: I'll need some time, since the last storm knocked over some of my trees. I have to remove them.

We have also had storms wind and heavy rain in sydney of the back off our recent fires. Nature will not let us rest.

Regards

Tony

 

[TonyM]

Mario,

On HTML elements and Wiki text,

Please bare with me here trying to express this, I believe this is an important concept, but I know I am still no expert here.

It seems to me there is a rich set of opportunities to use html tags to achieve much in tiddlywiki. As we were considering;

<section> and <div> should be no problem, because they allow "flow content" as "Permitted Content" see the link.
<span>'s are inline elements only. They only allow "phrasing content" (same as paragraphs). It will produce invalid html code if used at the start of the line like: * or # or .

Yes, provide but not at start of line 

I was thinking about an <aside> (note) using > at the beginning of the line.

• I raised this before because there seems to be a slow evolution within tiddlywiki of replacing such html tags with either markup or widgets. Changing markup is risky from a standards point of view, and widgets certainly do limit sharability to tiddlywiki without further conversion, but some bespoke markup could be available like the dot paragraph.
• adding to the wikitext markup globally as you are discussing is a worthy goal, however it is by nature is a slower process to maintain within standards.
• Of course we can feely add html to wiki text and have a lot of it honoured, but this defeats the value of a markup language because html makes the text much messier. An exception I find is in complex tables and lists, I prefer to use html tables containing list widgets.

A strategy I am thinking about

I wonder if we could introduce a way to open up the access to html elements without compromising the markup. Hence my suggestion for being able to introduce div and/or section elements (span like within a line) via wikitext that are themselves css addressable. Like your paragraphs with classes.

The three key reasons are;

• Provide a set of layout controls within tiddlers via markup
• Provide access to the richness of HTML elements via markup
• Reduce the need to progressively design wikitext/widget alternatives to introduce new features by providing the extendibility 

Some points

• We can already use the ".classname" against ";" and ":" markup ";.classname" 
• The key html tags for basic layout are here HTML inline and Block elements
• Any HTML element can actually be redefined with the display parameter 
• If we focused on one html block (eg section even paragraph) and one inline element that can be introduced with wikitext but then enable the ability to pass "style/class and display parameter settings" then we have a wikitext way to introduce html elements. I see here semantic elements which could be introduced to a block element via the display parameter. Also see the the  semantic elements that can be used to define different parts of a web page, it would be good if there was an avenue to make use of such layout within tiddlers via wikitext.
• We may be able to ensure a tiddler copied elsewhere will just fall back to the basic elements.

I hope I have being able to illustrate a concept that has being broiling in my mind for some time, and in quite relevant to this consideration of extending wikitext markup. The idea is a generic solution for bespoke extended markup, that resides safely within the markup standards. The idea is to make a leap rather than focus on gradual introduction of features to wiki text.

Speculation;

I would like to see some widgets that allows us to integrate html on-click and submit such that one could paste in a html form and have the input captured in tiddlers and fields. Even if we have to add a widget or two.

• I wonder if we could use a standard dom, with a way to connect to the TiddlyWiki dom, eg; allow html to populate a "sandbox" dom, with a submit or action that dumps it to a tiddlers fields.

The idea once again is to leverage the technologies and content available without compromising the markup.

Regards

Tony

[PMario]

On Thursday, March 5, 2020 at 11:53:51 PM UTC+1, TonyM wrote:

...

Are you suggesting this?;

• test  

next line should be part of the li-element.

should become 

• test  next line should be part of the li-element.

NO

It should become this:

• test
  next line should be part of the li-element.

I did test the zip wiki and it displays it as I intended it. ... Not sure how you did manage get a 1 liner. It should be 2

-m

[PMario]

On Thursday, March 5, 2020 at 11:53:51 PM UTC+1, TonyM wrote:

Please see some sample wiki text below that illustrates some of my observations

* List
:notes
* list more
:more notes

# List
:notes
# list more
:more notes

That's a nightmare. Not only the wikitext, which should be human readable looks ugly, with all the extra "style me" markup.

Also the rendered output is ugly. The spacing is a horror (for my taste).

And last but not least the html code is a mess.

Try the attached json tiddlers with the experimental wiki.

Both the wikitext and the output should be much nicer _and_ the numbering works.

-mario

[PMario]

On Thursday, March 5, 2020 at 11:53:51 PM UTC+1, TonyM wrote:

...

 

**item 1 no matching"*" Eroniouse dot to left

**item 2

<ul><ul>
<li>item 1 no matching"*" no eroniouse dot to left</li>
<li>item 2</li>
</ul></ul>

:::*List items indented
:::*list

hmmm,

I think there are several problems here. Some are technical and others are psychological.

a) tech) If you want to change the indentation level later, you'll need to touch every line of the text. 

b) psych) Readers may be confused, because there may be different indent levels, which otherwise look the same.

add a)

You use it wrong.

** Means "create a nested unordered list". see the docs.

There is no error. TW displays, what you told it to do. ... It may be different to what you expect, but that's it.

* You miss level 1 text here. <-- that's the error you made.

**item 1 no matching"*" Eroniouse dot to left
**item 2


The solution is simple: Create a little bit of CSS

.tc-tiddler-preview .in,

.tc-tiddler-body .in {

  margin-left: 2em;

}

and us

*.in You miss level 1 text here.

**item 1
**item 2


see attached file.

------------------------

Add b)

:::*List items indented
:::*list

Depending on the indentation level you use, your readers will see the difference. .. The problem is, that there is no other difference. eg:

• List items indented
• List

• List items indented
• List

• List items indented
• List

So the problem now is. Which list is more important. The one on the left or the one on the right. With a list as above it may be clear. BUT

• List items indented
• List

What happens if there is more text in between. So the question I have now is: "Why is the list below more indented as the list above"

Is there a problem with the formatting, or does it mean something.

• List items indented
• List

I personally wouldn't use indented lists that look "like level 1 lists". IMO they only confuse readers.

If we need indentation, it needs to look different AND the difference must have a meaning.

"It looks nicer" is not a valid argument, since "taste" is different for every user.

Possibilities can be seen at MDN CCS info page.

-mario

[PMario]

uups,

Forgot the attachment: test-list-indentation.json

-m

[PMario]

On Thursday, March 5, 2020 at 11:53:51 PM UTC+1, TonyM wrote:

\startheading 5

This is a nice idea, how will it effect transclusions, Actually applying this to subtiddlers transcluded at a different heading level could be helpful, again can we programmatically apply pragma? 

The main goal is consistency.

The tv-adjust-heading-level is already part of TW. So a tiddler using the pragma may look like this:

\startheading 5

! heading

If the tiddler is shown, it should create this html code.

<h5>heading</h5>

Even if the text uses h1-markup like: ! heading

If transcluded, it should also display a heading 5. If transcluded with tv-adjust-heading-level set to eg: 1, it will be shown as <h6>, setting 2 will show <h6> too, since it h6 is the biggest number. 

If the variable is set to -2 it will be shown as <h3>.

I think, that's consistent.

-mario

[TiddlyTweeter]

Ciao PMario & TonyM

IMO PMario is right for something that could be shared one needs to avoid ambiguity. Two key points, I think ...

1 - Stating the obvious: any extension of (PUBLIC shared) wikitext needs to be elegant and minimalist. Retention of human readability is paramount.

There is an ISSUE of scope limitation due simply to the fact the repertoire of available marker symbols is limited. 

Usage of them should be as restrained as possible.

2 - For PRIVATE developed wiki any schema, in theory, could be used. For my own use I use BJ's pre-parser to have lines starting ".r1", ".c5" etc for specialist layouts. I'd never make that public. Too confusing.

My point? I'm still unclear whether TonyM would be best served via (1) or (2).

What I would say is that ability for end-user to easier make a personal ad-hoc "rough-parser" is something Good, not bad.

Best wishes

TT