ReST mode: Italicisation and bold face markup in some special cases

[Friedrich Romstedt]

Hi Marshall,
Hi all,

During the last couple of years some more issues related to ReST
markup in vim have accumulated in my pipeline; this email covers the
first of two more observations.

My question is regarding *italicisation* and **bold face**. It is
probably described best by a set of Tests, together with their outcome
in vim-8.2.2845:

*hello*-*foobar*
*hello*_*foobar*
*hello*/*foobar*
(In the first line, the *foobar* is not displayed italicised;
everything else is.)

**hello**-**foobar**
**hello**_**foobar**
**hello**/**foobar**
(Same here: in the first line, **foobar** ins't printed bold-face;
everything else renders bold.)

-*foobar*
_*foobar*
/*foobar*
(Only the last line renders with italicisation.)

-**foobar**
_**foobar**
/**foobar**
(Same here: Only the last line is printed in bold face.)

I stumbled upon this observations by the very first example
(*hello*-*foobar*). Is there a reason for this behaviour or is this
an issue?

Best,
Friedrich

[Marshall Ward]

Hi Friedrich,

I believe that many of these cases have been resolved in the development branch, which may not have been sent to the main repository yet (https://github.com/marshallward/vim-restructuredtext)

I tested your inline markup examples and I am seeing some issues with dashes (`-`) but the underscore and slash delims appear to be correct.

There is a fixed list of characters which are allowed to follow the end of an inline markup.  It appears that slash `/` is on this list but dash `-` is missing and is not being handled correctly.  Underscore `_` is not allowed to follow and end-inline.

For example, in rst2html:  *hello*_*foobar* highlights as a single italic, <em>hello*_*foobar</em>.

There are similar requirements for the start of an inline highlight.  Again, dash and slash are permitted but underscore is not.

I will try to resolve this in the vim-restructuredtext repo and then send a patch over to Vim.

[Friedrich Romstedt]

Hi Marshall,

Am So., 16. Mai 2021 um 01:20 Uhr schrieb Marshall Ward
<marsha...@gmail.com>:

>
> I believe that many of these cases have been resolved in the development branch, which may not have been sent to the main repository yet (https://github.com/marshallward/vim-restructuredtext)
>
> I tested your inline markup examples and I am seeing some issues with dashes (`-`) but the underscore and slash delims appear to be correct.
>
> There is a fixed list of characters which are allowed to follow the end of an inline markup. It appears that slash `/` is on this list but dash `-` is missing and is not being handled correctly. Underscore `_` is not allowed to follow and end-inline.
>
> For example, in rst2html: *hello*_*foobar* highlights as a single italic, <em>hello*_*foobar</em>.
>
> There are similar requirements for the start of an inline highlight. Again, dash and slash are permitted but underscore is not.
>
> I will try to resolve this in the vim-restructuredtext repo and then send a patch over to Vim.

That's great.

Attached you find some small exploration of the behaviour of
``rst2html``. This behaviour might be seen as authoritative. After
having written that small document, I looked up the syntax description
upstream, and found
https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup,
which might be used to synchronise the rendering within vim with the
upstream specification?

From what you've written I understand that there already is some code
implementing that there are sets of characters allowed to
precede/follow inline markup strings like ``*`` or ``**``, so
hopefully the dash issue won't be too difficult to resolve?

Best,
Friedrich

[Marshall Ward]

On Mon, May 17, 2021 at 8:04 AM Friedrich Romstedt
<friedric...@gmail.com> wrote:

> Attached you find some small exploration of the behaviour of
> ``rst2html``. This behaviour might be seen as authoritative.

David Goodger is author of both the reStructuredText standard and
docutils, which includes rst2html. I suppose the standard is the true
authority, although in practice one is going to have trouble if
behavior doesn't match docutils.

In this case, luckily the behavior is clear and we need to add dash
(and all the rest) to the list of valid start/end tokens.

> From what you've written I understand that there already is some code
> implementing that there are sets of characters allowed to
> precede/follow inline markup strings like ``*`` or ``**``, so
> hopefully the dash issue won't be too difficult to resolve?

Yes, I have a fix which resolves the use of dashes.

< call s:DefineOneInlineMarkup(a:name, a:start, middle, a:end,
'\%(^\|\s\|\%ua0\|[/:]\)', '')
---
> call s:DefineOneInlineMarkup(a:name, a:start, middle, a:end,
> \ '\%(^\|\s\|\%ua0\|[-/:]\)', '')

At the moment, my only concern is that there seems to be some legacy
code blocks here, which seem like they were intended to catch dashes
but do not appear to be working. I'm not the original author of those
code blocks, so I don't know how they were supposed to work or what
the problem could be.

I will try to find a bit of time to overhaul the whole section and see
if it can be trimmed down. But if I cannot manage to clean it up,
then I'll just pass along this simple fix (after appending the other
missing symbols in the document you linked).

[Friedrich Romstedt]

Hi Marshall,

Am Mo., 17. Mai 2021 um 14:32 Uhr schrieb Marshall Ward
<marsha...@gmail.com>:
>

> On Mon, May 17, 2021 at 8:04 AM Friedrich Romstedt
> <friedric...@gmail.com> wrote:
>
> > Attached you find some small exploration of the behaviour of
> > ``rst2html``. This behaviour might be seen as authoritative.
>
> David Goodger is author of both the reStructuredText standard and
> docutils, which includes rst2html. I suppose the standard is the true
> authority, although in practice one is going to have trouble if
> behavior doesn't match docutils.

I certainly agree. To my impression David Goodger worked out the
grammar of ReST (particularly the section about inline markup I linked
to) very thoroughly.

> In this case, luckily the behavior is clear and we need to add dash
> (and all the rest) to the list of valid start/end tokens.

This sounds promising.

> > From what you've written I understand that there already is some code
> > implementing that there are sets of characters allowed to
> > precede/follow inline markup strings like ``*`` or ``**``, so
> > hopefully the dash issue won't be too difficult to resolve?
>
> Yes, I have a fix which resolves the use of dashes.

Great!

> < call s:DefineOneInlineMarkup(a:name, a:start, middle, a:end,
> '\%(^\|\s\|\%ua0\|[/:]\)', '')
> ---
> > call s:DefineOneInlineMarkup(a:name, a:start, middle, a:end,
> > \ '\%(^\|\s\|\%ua0\|[-/:]\)', '')

Please apologise that I do not truly understand this code snippet.

> At the moment, my only concern is that there seems to be some legacy
> code blocks here, which seem like they were intended to catch dashes
> but do not appear to be working. I'm not the original author of those
> code blocks, so I don't know how they were supposed to work or what
> the problem could be.

Might be ``git blame`` an option?

> I will try to find a bit of time to overhaul the whole section and see
> if it can be trimmed down. But if I cannot manage to clean it up,
> then I'll just pass along this simple fix (after appending the other
> missing symbols in the document you linked).

I am happy about this. Good luck!

It's great that you're closing in on a real fix. I am looking forward
to seeing it in action!

Let me know when I can do some testing (a short pointer on how to
install the resp. files user-locally would be appreciated).

Best wishes,
Friedrich