ReST Syntax

[Friedrich Romstedt]

Hi,

This is about a corner case of the ReST syntax highlighting. More
specifically, it is about an issue related to comments in ReST using
vim-8.2.0814 (which is the most recent version of vim packaged for
Arch Linux).

When starting a line with two dots while editing an ``*.rst`` file,
vim 8.2.0814 automatically indents the following line by 3 spaces.
This is useful in order to aid in providing *options* to directives.
It makes some sense also to continue a comment, when the text
following the two dots doesn't comprise a directive. Notice that
vim-8.2.0814 won't produce this indentation when opening a, say,
``*.txt`` file and issuing later ``:set syntax=rst``, while it *does*
appear when opening an ``*.rst`` file directly.

I am using a preprocessor for ReST files where comments like ``..
-de-`` or ``.. -de, en-`` etc. can be used to interlace content for
different *targets* of a file (here ``de`` and ``en``); there is often
interlaced content following *directly* to these comments (without a
blank line).

In this case, the ReST syntax mode of vim 8.2.0814 leads to results
impeding usability.

The abovementioned "sectioning comments" are never seen in the
produced "compounds", while the "interlaced" sources are compiled
never directly. Normally the unindented lines directly following these
abovementioned comments (e.g. ``.. -en-``) would be syntax errors,
but, in this case, these ReST "codes" make sense. Since the unindented
lines directly following a two-dots line are, strictly speaking, ReST
syntax errors, the behaviour of the vim ReST syntax highlighting mode
can be considered "unspecified" in this case.

The line following the two-leading-dots line can be dedented manually.
However, after the next line break, vim 8.2.0814 again indents the
next line.

Would it be possible, to "leave" the comment/directive mode which
leads to this indentation as soon as an unindented line is written?
Maybe there would be room for such a decision since the case does not
appear in "valid" ReST code. For the moment, I can work with vim
8.2.0510.

Best,
Friedrich

[Friedrich Romstedt]

Hi,

Am Fr., 31. Juli 2020 um 19:04 Uhr schrieb Friedrich Romstedt
<friedric...@gmail.com>:

>
> This is about a corner case of the ReST syntax highlighting. More

> specifically, it is about an issue related to comments in ReST ...

Is anyone interested in solving this problem (described in the
citation below)? Who is the maintainer of the ReST syntax mode?

Friedrich

> ... using

[Christian Brabandt]

On Fr, 07 Aug 2020, Friedrich Romstedt wrote:

> Hi,
>
> Am Fr., 31. Juli 2020 um 19:04 Uhr schrieb Friedrich Romstedt
> <friedric...@gmail.com>:
> >
> > This is about a corner case of the ReST syntax highlighting. More
> > specifically, it is about an issue related to comments in ReST ...
>
> Is anyone interested in solving this problem (described in the
> citation below)? Who is the maintainer of the ReST syntax mode?

I am not sure I understand the problem correctly. It sounds like you are
at one point just loading the syntax file, while when opening a proper
rst file, Vim will also set the filetype (e.g. set some options and also
sets indenting options).

Anyhow, it would actually help to see some specific examples of what
works and what you expect. Please also have a look at the existing rst
filetype as well as the indent plugin). Perhaps you can already disable
any options that you do not like.

Best,
Christian
--
Der sogannte Aberglaube beruht auf einer viel größeren Tiefe und
Delikatesse als der Unglaube.
-- Johann Wolfgang von Goethe (zu Riemer 12. Dez. 1806)

[Friedrich Romstedt]

Hi Christian,

Am Fr., 7. Aug. 2020 um 15:18 Uhr schrieb Christian Brabandt
<cbl...@256bit.org>:

>
> I am not sure I understand the problem correctly. It sounds like you are
> at one point just loading the syntax file, while when opening a proper
> rst file, Vim will also set the filetype (e.g. set some options and also
> sets indenting options).

Okay, I didn't know about the filetype option before. It indeed
explains that portion of my report.

> Anyhow, it would actually help to see some specific examples of what
> works and what you expect.

Attached you find a file where I am trying to explain my problem.
Actually I tried to record typing a copy of it by means of asciinema,
but somehow asciinema does not playback the recoding properly, at
least locally (it stops as soon as the recording enters ``vim``).
Having the cursor visible and how it gets indented and dedented would
help demonstrating what I'd like to show; however I hope the file
might be of some use also without this.

> Please also have a look at the existing rst
> filetype as well as the indent plugin). Perhaps you can already disable
> any options that you do not like.

I inspected ``:help ft-rst-{indent, syntax, plugin}``, and didn't find
any option applicable to my case.

Best,
Friedrich

[Friedrich Romstedt]

Hi,

Am Fr., 7. Aug. 2020 um 15:18 Uhr schrieb Christian Brabandt
<cbl...@256bit.org>:
>

> I am not sure I understand the problem correctly. It sounds like you are
> at one point just loading the syntax file, while when opening a proper
> rst file, Vim will also set the filetype (e.g. set some options and also
> sets indenting options).
>
> Anyhow, it would actually help to see some specific examples of what
> works and what you expect. Please also have a look at the existing rst
> filetype as well as the indent plugin). Perhaps you can already disable
> any options that you do not like.

Yesterday I uploaded an asciinema cast regarding my issue:
https://asciinema.org/a/406969. It illustrates the behaviour I am
referring to by examples in-line.

Best,
Friedrich

[Christian Brabandt]

I looked into your last uploaded file. It looks like the behaviour is
hard coded in the indent script, so CC'ing the indent script maintainer.

,----[ $VIMRUNTIME/indent/rst.vim ]-
| function GetRSTIndent()
| ...
| let psnum = s:get_paragraph_start()
| if psnum != 0
| if getline(psnum) =~ s:note_pattern
| let ind = 3
| endif
| endif
`----

So it is hard coded to be three. I think a better approach would be the
following patch:

diff --git a/runtime/indent/rst.vim b/runtime/indent/rst.vim
index a31ad8e08..64db170d8 100644
--- a/runtime/indent/rst.vim
+++ b/runtime/indent/rst.vim
@@ -38,9 +38,13 @@ function GetRSTIndent()

let psnum = s:get_paragraph_start()
if psnum != 0
- if getline(psnum) =~ s:note_pattern
- let ind = 3
+ if getline(psnum) =~ s:note_pattern
+ " return the indent after the first 2 leading '.'
+ let ind = match(getline(psnum), s:note_pattern .. '*\zs\S')
+ if ind < 3
+ let ind = 3
endif
+ endif
endif

if line =~ s:itemization_pattern

Best,
Christian
--
Die Katholiken Terrorisieren das Land mit einer Auffassung vom Wesen
der Ehe, die die ihre ist und die uns nichts angeht.
-- Kurt Tucholsky

[Marshall Ward]

The issue is that indentation is preserved in standard text but not in comments, is that right?

I think that I agree this indentation in comments should be preserved.  Christian's patch does not seem to be resolving the issue for me, but this did seem to get closer:

diff --git a/indent/rst.vim b/indent/rst.vim
index e1c7946..f3eb641 100644
--- a/indent/rst.vim
+++ b/indent/rst.vim
@@ -40,7 +40,7 @@ function GetRSTIndent()

   let psnum = s:get_paragraph_start()
   if psnum != 0

       if getline(psnum) =~ s:note_pattern
-          let ind = 3

+          let ind = max([3, ind])
       endif
   endif

That is, use the existing `ind` determined from the previous line.

I can submit this change after a bit of testing, assuming there's no obvious objections.

[Marshall Ward]

I've pushed this change as a PR to the vim-restructuredtext repository:

https://github.com/marshallward/vim-restructuredtext/tree/note_indent

If there's no objections then I will merge it and send an update to Vim.

[Friedrich Romstedt]

Hi Marshall,

Am Fr., 16. Apr. 2021 um 16:17 Uhr schrieb Marshall Ward
<marsha...@gmail.com>:

>
> The issue is that indentation is preserved in standard text but not in comments, is that right?
>
> I think that I agree this indentation in comments should be preserved. Christian's patch does not seem to be resolving the issue for me, but this did seem to get closer:
>
> diff --git a/indent/rst.vim b/indent/rst.vim
> index e1c7946..f3eb641 100644
> --- a/indent/rst.vim
> +++ b/indent/rst.vim
> @@ -40,7 +40,7 @@ function GetRSTIndent()
> let psnum = s:get_paragraph_start()
> if psnum != 0
> if getline(psnum) =~ s:note_pattern
> - let ind = 3
> + let ind = max([3, ind])
> endif
> endif
>
> That is, use the existing `ind` determined from the previous line.
>
> I can submit this change after a bit of testing, assuming there's no obvious objections.

Thanks for your work!

From what I understand, this would set the new line's indentation
level to the maximum of 3 and the previous indentation level, am I
right?

When I am correct, this would mean that your proposed change would
make it possible to indent by *more* than 3 spaces, and the
indentation would be preserved. However, when indenting with *less*
than 3 spaces, 3 spaces would be chosen?

My original proposition was probably a bit buried inside of the ReST
file I posted on 15 August 2020 and inside of the asciinema cast from
14 April this year: I proposed to remove the extra rule of indentation
for comments/directives *entirely*, keeping only the standard
propagation of indentation. This would mean, that the User (e.g., me)
would need to ident the second line of such a comment or directives
block *once*, and then the indentation would be propagated. This
functionality already exists, and would remain without any alteration.

For me, if I am right in my interpretation of the change you're about
to push, the new version wouln't solve my issue. I'd like to continue
writing directly in the next line following the comment/directive line
*without any indentation* (also for the following lines). In my case,
comment blocks can stop right the next line, because the respective
comment line is removed by the preprocessor I am using; marking them
up as a ReST comment just serves the purpose of using the syntax
highlighter to show them as "non"-content.

Hopefully I am not too late with this reply.

Thanks in advance,
Friedrich

[Marshall Ward]

On Wednesday, 12 May 2021 at 09:20:12 UTC-4 Friedrich Romstedt wrote:

From what I understand, this would set the new line's indentation
level to the maximum of 3 and the previous indentation level, am I
right?

When I am correct, this would mean that your proposed change would
make it possible to indent by *more* than 3 spaces, and the
indentation would be preserved. However, when indenting with *less*
than 3 spaces, 3 spaces would be chosen?

Yes, that's right.  I had thought that the suggestion was to preserve indentation within comment blocks, in the same way that indentation is preserved in text, which seems very reasonable to me.

In the current version, comments always indent to three spaces.

 

[...] I proposed to remove the extra rule of indentation

for comments/directives *entirely*, keeping only the standard
propagation of indentation. This would mean, that the User (e.g., me)
would need to ident the second line of such a comment or directives
block *once*, and then the indentation would be propagated. This
functionality already exists, and would remain without any alteration.

I see, you want to remove the indentation of comments entirely, and have it entirely depend on the previous line.

While I didn't introduce the comment indentation, I do personally find it useful to stay inside of the comment block, particularly when writing code snippets.  At the least, I think this would need to be something disabled by the user, as suggested by Christian.

There could also be some odd consequences here, e.g. autoindentation of the whole file, that would need to be explored.

Hopefully I am not too late with this reply.

The branch is still unmerged (and I am terrible about sending updates anyway) so please do add your comments here, and hopefully we can get additional feedback in the vim-restructuredtext repo:

https://github.com/marshallward/vim-restructuredtext/pull/64

 

[Friedrich Romstedt]

Hi Marshall,

Am Mi., 12. Mai 2021 um 16:11 Uhr schrieb Marshall Ward
<marsha...@gmail.com>:
>

> I see, you want to remove the indentation of comments entirely, and have it entirely depend on the previous line.
>
> While I didn't introduce the comment indentation, I do personally find it useful to stay inside of the comment block, particularly when writing code snippets. At the least, I think this would need to be something disabled by the user, as
suggested by Christian.

I do not entirely understand your reasoning here, about "staying

inside of the comment block, particularly when writing code snippets".

BTW, referring to Christian's message from 7 August 2020, to be
precise, Christian suggested checking for *existing* options to
disable?

As I see it, when writing a comment block, the indentation would be
preserved after having indented the first line following the "lead in"
line starting by ``..``, maybe this would satisfy your need I do not
understand completely?

> There could also be some odd consequences here, e.g. autoindentation of the whole file, that would need to be explored.

Wouldn't that be a bad idea, when the indentation in the ReST file has
been adapted also in other places manually?

However I think you are right about that my proposed change would
possibly affect the re-indentation behaviour. I do not know enough
about how indentation is deduced when re-indenting a paragraph or a
selection to assess the consequences.

In some places, the current indentation code produces odd results; you
can see them in the asciinema cast I posted on 14 April
(https://asciinema.org/a/406969). It is a bit longish (25 minutes),
probabaly it is a good idea to download it and play it back locally
with reduction of pauses and/or general speed-up. It should have been
recorded using a 80x25 terminal, if I am not mistaken.

> The branch is still unmerged (and I am terrible about sending updates anyway) so please do add your comments here, and hopefully we can get additional feedback in the vim-restructuredtext repo:
>
> https://github.com/marshallward/vim-restructuredtext/pull/64

I noticed that the URL given in your first post cannot be clicked to follow it.

For the moment, I am continuing replying by email because of its
existing context. Let me know if you want me to change that.
Possibly it would suffice to post the results in the PR conversation?

Best,
Friedrich

[Friedrich Romstedt]

Hi Marshall,

Today I stumbled again upon some behaviour related to comment blocks.
I am sending my observations attached.

I realised that it is probably important that I am usually working
*with* the ``autoindent`` option *set*. There might be conclusions of
mine which hold only when auto-indentation is in effect. This
complicates things a bit.

So far,
Friedrich