format=flowed and extra spaces

[martin f krafft]

Hello list,

I've set up Vim in tandem with Mutt to compose format=flowed emails,
i.e. using &fo+=aw in Vim.

I'm also in the habit of using numbered and bulletted lists in
emails a lot.

Unfortunately, the two don't seem to work together well, or I am
doing something wrong.

For instance, consider the following:

1. This is the first item, spanning two rows because the text is a
bit longer than 80 characters, or whatever &tw is set to.

2. This is the second item.

The way I have Vim configured means that the second line of the
first item is properly indented, i.e. I see:

| 1. This is … |
| bit long… |

At first, I thought those spaces at the start of the second line are
"local" in that they are only needed for presentation. However, when
Mutt creates a MIME message, it includes those spaces!

| 1. This is … text is a=20 |
| ···bit longer |

This means that recipients who don't use exactly the same font and
window size as I do might see the following instead:

| 1. This is … text |
| is a bit longer |

So there is no indent, but there are multiple subsequent spaces in
the middle of the line, which makes the whole thing harder to read.

I think all of this would be avoided if Vim didn't add those spaces
it needs for indenting (presentation) in format=flowed mode.

Is this possible? Or am I doing something fundamentally wrong?

Thank you!

--
@martinkrafft | https://riot.im/app/#/room/#madduck:madduck.net

in seattle, washington, it is illegal to carry a concealed weapon that
is over six feet in length.

spamtraps: madduc...@madduck.net

[Gary Johnson]

The idea of format=flowed is to allow email messages to be displayed
nicely by email clients that do not support format=flowed as well as
by those that do. Neither Vim nor mutt should do anything to
messages to corrupt their contents. In particular, neither should
automatically remove any leading spaces. See RFC 3676.

Vim takes care of wrapping lines at 78 columns and adds a single
trailing space to inter-paragraph line breaks. It can also handle
formatting quoted paragraphs. That's all it should do.

Mutt takes care of space-stuffing and does something with quoted
blocks, but I've forgotten what. That's all it should do.

It is the responsibility of the receiving agent to reformat
format=flowed text as it sees fit. How it does that is not
specified by the RFC.

If some receiving agent claims to support format=flowed, yet blindly
includes sequences of spaces in the middle of flowed lines, as in
your third example, I would say that agent is broken.

Also, your second example suggests that mutt is using
quoted-printable encoding. RFC 3676 says that quoted-printable
encoding "SHOULD NOT be used for Format=Flowed unless absolutely
necessary...."

Regards,
Gary

[Gary Johnson]

Oops. "inter-paragraph" should be "intra-paragraph".

[martin f krafft]

Thanks, Gary, for your response. It made me realise something,
namely that Mutt itself seems broken wrt format=flowed. I'll turn to
the mutt-users list for that.

However, I did have one comment relating to Vim:

>Neither Vim nor mutt should do anything to messages to corrupt
>their contents. In particular, neither should automatically remove
>any leading spaces. See RFC 3676.

I am 100% with you on that. However, with Vim being the composer of
a message, I'd argue that there is no corruption of contents going
on, if it differentiates between displaying spaces, and saving them
to the file:

Let me elaborate, using · for spaces and $ for EOL:

Take the following paragraph.

| 1. This is a multiline |
| indented paragraph |

If &formatoptions doesn't include "aw", then we need those leading
spaces explicitly, because the indenting is done in such a way as to
hard-code it into the file. Therefore, Vim should save:

| 1. This is … a multiline$ |
| ···indented paragraph.$ |

However, if &fo=aw as well as &fo=n is in place, and there is a
trailing space on the first line, then I would say Vim should really
not be saving those spaces. Instead, the buffer should really be

| 1. This is … a multiline$ |
| indented paragraph.$ |

But this then gets displayed as in the original example, i.e. Vim
displays leading spaces, recognizing the numbered list.

Or at least this should/could be configurable. Don't you agree?

>If some receiving agent claims to support format=flowed, yet blindly
>includes sequences of spaces in the middle of flowed lines, as in
>your third example, I would say that agent is broken.

Oh great. Yeah, I'll let the GMail admins know at nor...@gmail.com.

>Also, your second example suggests that mutt is using
>quoted-printable encoding. RFC 3676 says that quoted-printable
>encoding "SHOULD NOT be used for Format=Flowed unless absolutely
>necessary...."

Unfortunately, PGP/MIME needs quoted-printable (mutt's
pgp_strict_enc), and so one could argue it is "absolutely
necessary".

I don't quite understand the ramifications of this though.

Best, and thanks again!

--
@martinkrafft | https://riot.im/app/#/room/#madduck:madduck.net

"women love us for our defects.
if we have enough of them,
they will forgive us everything,
even our gigantic intellects."
-- oscar wilde

spamtraps: madduc...@madduck.net

[Gary Johnson]

I'm still thinking about this. You're asking for Vim to format its
display of a buffer according the formatting rules of
'formatoptions' without changing the actual contents of the buffer.
My first thought was, "Vim is an editor, not a word processor, and
it doesn't do that sort of formatting for display." But there are
exceptions. For example, I use 'linebreak', 'breakindent' and
conceal to make some text less cluttered more understandable.

However, as I wrote before, format=flowed is intended to help make
email text readable in mail readers that don't support it as well as
in those that do. So, in an email message sent to a general
audience, I would format it to look good in a mail reader that did
not understand format=flowed, then add the trailing spaces and the
Format=Flowed header to help a format=flowed-aware reader reflow the
text as necessary for its display.

So, if my message included a list item like this,

1. This is the first line
and this the second.

I would leave it like that for the benefit of
non-format=flowed-aware email readers, but add a trailing space to
the first line to show format=flowed-aware readers that the two
lines may be flowed together. I think it's up to the
format=flowed-aware reader to recognize that the leading spaces of
the second line are for indentation and should be collapsed to one
space, and that the leading "1. " indicates a list item and that
that line should be wrapped and indented according to its rules for
formatting list items.

> >If some receiving agent claims to support format=flowed, yet
> >blindly includes sequences of spaces in the middle of flowed lines,
> >as in your third example, I would say that agent is broken.
>
> Oh great. Yeah, I'll let the GMail admins know at nor...@gmail.com.

Yep, that's a problem.

> >Also, your second example suggests that mutt is using
> >quoted-printable encoding. RFC 3676 says that quoted-printable
> >encoding "SHOULD NOT be used for Format=Flowed unless absolutely
> >necessary...."
>
> Unfortunately, PGP/MIME needs quoted-printable (mutt's
> pgp_strict_enc), and so one could argue it is "absolutely necessary".
>
> I don't quite understand the ramifications of this though.

I don't either.

Regards,
Gary

[Cameron Simpson]

On 07Oct2019 09:31, martin f krafft <mad...@madduck.net> wrote:
>I've set up Vim in tandem with Mutt to compose format=flowed emails,
>i.e. using &fo+=aw in Vim.

For the record, I use:

set formatoptions=walqj

It also matters what your mail headers are set to. Your Content-Type
header says:

Content-Type: text/plain; charset="UTF-8"; format=flowed

Notably there is no delsp=yes; that means delsp=no.

When delsp is "yes", the trailing space on the line before the physical
line ending is logically removed when assembling the post-flowed line.

>I'm also in the habit of using numbered and bulletted lists in emails a
>lot.
>
>Unfortunately, the two don't seem to work together well, or I am doing
>something wrong.

RFC 3676 doesn't address lists in any way. It is deliberately designed
to turn plain text into flowable text. The RFC is here:

https://tools.ietf.org/rfcmarkup?doc=3676

>For instance, consider the following:
>
>1. This is the first item, spanning two rows because the text is a
>bit longer than 80 characters, or whatever &tw is set to.

The quoted text above seems to be 2 physical lines here (in vim), with 3
spaces at the end. The final space is the padding to indicate a
flowedline. I find that a little weird myself. because your original
(quoted-printable) message has:

1. This is the first item, spanning two rows because the text is a=20

bit longer than 80 characters, or whatever &tw is set to.

I've indented that 2 spaces, but what we've got there, post quoted
printable decoding, is:

- a line hard against the left margin starting with "1" and ending in a
single space.

- an indented line with 3 leading spaces and no trailing spaces.

So the former line is flowed (trailing space) and the latter line is
space stuffed (a leading space). A leading space is dropped from the
logical line.

Post handling format=flowed, that should become a single logical line
with 2 or 3 spaces between "a" and "bit" depending on delsp=yes or no
respectively.

The purpose of delsp is to make the SP-CR-LF phusical line break
entirely removed; in principle it can thus be planted anywhere in the
logical line, though it is meant to occur between words.

>2. This is the second item.
>
>The way I have Vim configured means that the second line of the first
>item is properly indented, i.e. I see:
>
> | 1. This is … |
> | bit long… |
>
>At first, I thought those spaces at the start of the second line are
>"local" in that they are only needed for presentation.

The first space is. The others are not, at least as far as format=flowed
is concerned.

>However, when Mutt creates a MIME message, it includes those spaces!
>
> | 1. This is … text is a=20 |
> | ···bit longer |
>
>This means that recipients who don't use exactly the same font and
>window size as I do might see the following instead:
>
> | 1. This is … text |
> | is a bit longer |

Yes, because those leading spaces are just... spaces.

>So there is no indent, but there are multiple subsequent spaces in the
>middle of the line, which makes the whole thing harder to read.
>
>I think all of this would be avoided if Vim didn't add those spaces it
>needs for indenting (presentation) in format=flowed mode.

I agree.

>Is this possible? Or am I doing something fundamentally wrong?

Well, vim is producing text which is unfriendly to a format-flowed mail
presenter.

What you may need to do is to post process your message before mutt
picks it up, eg by wrapping the invocation of vim in a little script to
post process nicely indents list items item format=flowed friendly
lines, eg by recognising a numbered or bulleted item and dropping the
leading spaces from the second and following lines.

All that would be unnecessary if someone pipes up with some nice vim
setting to help.

For the record, I compose format=flowed mutt emails in vim by setting
$editor to my script "vim-flowed", which presently just invokes vim like
this:

exec vim \
-c 'silent 1,/^$/s/ *$//' \
-c 'set filetype=mail' \
-c 'set formatoptions=walqj' \
${1+"$@"}

Cheers,
Cameron Simpson <c...@cskk.id.au>