message-kill-to-signature sometimes kills signature too

[Eduardo Mercovich]

Dear Mu4ers.

While this is nothing serious, it may be annoying since it pushes
me away from the flux... and working in the Emacsverse we strive
for nothing less than perfection. ;)

In the last stable version (0.9.18) I observed once in a while
that C-c C-z (message-kill-to-signature) kills up to the end of
message, deleting the signature too.

Since it's not a frequent issue, I still don't get the ocurrence
pattern, except that it never happened -to me- in a non touched
message (that is, as it appears as you "R"eply).

Did someone had this behaviour? Any idea about how can I found the
source of it?

As always, thank you very much for your dedication. :)

Best regards...


--
Eduardo Mercovich

Donde se cruzan tus talentos
con las necesidades del mundo,
ahí está tu vocación.
(Anónimo)

[J Irving]

Hey Eduardo

This is interesting - I have been observing a strangeness that I have
not yet understood, and I believe it may be related to what you're
seeing.

The correct way to separate a signature from the body of email is using
the string "-- \n". Note the extra space at the end of the line before
the newline.

I have found that when I reformat any text anywhere in the bugger using
`mu4e-fill-paragraph`, this results in that string being trimmed, so
that it becomes "--\n".

Just now I test my theory that this is what's causing your problem, and
though I cannot confirm that in your case it is the cause, the
difference between "-- \n" and "--\n" *definitely* has the same effect
that you're seeing.

As you say, nothing serious, but hey, some data ¯\_(ツ)_/¯

cheers, J

>>>>>> Eduardo Mercovich <eduardo....@gmail.com> at 2017-03-03T09:34:05-0500:

> --
> You received this message because you are subscribed to the Google Groups "mu-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--
Jonathan Irving
http://j0ni.ca

[Eduardo Mercovich]

Hi J.

>> [...] In the last stable version (0.9.18) I observed once in a

>> while that C-c C-z (message-kill-to-signature) kills up to the

>> end of message, deleting the [...]

> This is interesting - I have been observing a strangeness that I
> have not yet understood, and I believe it may be related to what
> you're seeing.
> The correct way to separate a signature from the body of email
> is using the string "-- \n". Note the extra space at the end of
> the line before the newline.

I didn't even knew so I never paid attention to that space...

> I have found that when I reformat any text anywhere in the
> bugger using `mu4e-fill-paragraph`, this results in that string
> being trimmed, so that it becomes "--\n".

Great observation!

> Just now I test my theory that this is what's causing your
> problem, and though I cannot confirm that in your case it is the
> cause, the difference between "-- \n" and "--\n" *definitely*
> has the same effect that you're seeing.

I checked what you said and yes, it is exactly what seems to
happen. Every time I used M-q the space dissapears. And after that
point, C-c C-z deletes everything until the end of the message
buffer.

> As you say, nothing serious, but hey, some data ¯\_(ツ)_/¯

More that that, a full and precise diagnosis. :D

Does this means that we should modify `mu4e-fill-paragraph` to
respect the signature separation? Or are there other solutions?

[J Irving]

>>>>>> Eduardo Mercovich <eduardo....@gmail.com> at 2017-03-03T11:16:37-0500:

> > I have found that when I reformat any text anywhere in the bugger using
> > `mu4e-fill-paragraph`, this results in that string being trimmed, so that it
> > becomes "--\n".

> Great observation!

Except I didn't mean to accuse mu4e of being or making use of buggers :P
Apologies for the typo, I meant "buffer".

> Does this means that we should modify `mu4e-fill-paragraph` to respect the
> signature separation? Or are there other solutions?

I think fixing `mu4e-fill-paragraph` is the right solution, if that is
indeed the culprit. I'm not entirely convinced that there isn't some
other interaction going on with my emacs whitespace settings; your
experience certainly seems to suggest otherwise though.

cheers, J

[Eduardo Mercovich]

Hi J.

[...]

>> Does this means that we should modify `mu4e-fill-paragraph` to
>> respect the signature separation? Or are there other solutions?

> I think fixing `mu4e-fill-paragraph` is the right solution, if
> that is indeed the culprit. I'm not entirely convinced that
> there isn't some other interaction going on with my emacs
> whitespace settings; your experience certainly seems to suggest
> otherwise though.

Ok, here is a proposal: I don't know elisp (or programming BTW)
but if someone tells me what to change, I'll do it here, try some
tests and report back.

If it works and after that Dirk Jan approves, it can go to the
main code.

What do you think? :)

[Joost Kremers]

On Fri, Mar 03 2017, Eduardo Mercovich wrote:
> Does this means that we should modify `mu4e-fill-paragraph` to
> respect the
> signature separation?

Well, `mu4e-fill-paragraph' basically just calls `fill-paragraph',
so you'd have to modify the latter.

If you want to go that way, you'd have to change
`mu4e-fill-paragraph' into an advice function around
`fill-paragraph', which first searches for the signature
separator, puts a marker there, then calls `fill-paragraph', and
after `fill-paragraph' returns, adds the space again. That's
doable, but I'm not entirely sure why you're running into this
problem in the first place, because...

> Or are there other solutions?

... not calling `mu4e-fill-paragraph' on the signature seems like
a possible solution. :-)

So, if I may ask, why do you refill your signature?

--
Joost Kremers
Life has its moments

[J Irving]

>>>>>> Joost Kremers <joostk...@fastmail.fm> at 2017-03-03T14:03:42-0500:

Well, you might have missed this detail, but as I mentioned it happens
when I refill *any* paragraph in the buffer. For example, I just hit M-q
without moving point from here, and sure enough the extra space was gone
afterwards.

Perhaps you could try it and report back? Then we'd have 3 consistent
examples, or 1 example demonstrating that it really is some other
problem.

[Eduardo Mercovich]

Hi Joost.

[...]

>> Or are there other solutions?

> ... not calling `mu4e-fill-paragraph' on the signature seems
> like
> a possible solution. :-)
> So, if I may ask, why do you refill your signature?

As J said before, we don't do that. It happens if you refill *any*
paragraph.

So if mu4e-fill-paragraph uses the fill-paragraph but defined in
the mu4e context, doesn't it make sense to add that detail here in
mu4e? (sorry if I'm being imprecise, I will learn to program at
some point in my life).

Something like "do your thing up to the point where you find "-- "
and stop right there".

[Dirk-Jan C. Binnema]

Yeah, mu4e-fill-paragraphs doesn't recognize the concept of a 'message
signature'. I think the solution would be (in `mu4e-fill-paragraph') to
narrow the buffer to just the body part before calling `fill-paragraph'.

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Joost Kremers]

On Fri, Mar 03 2017, Eduardo Mercovich wrote:

> Hi Joost.
>
> [...]
>>> Or are there other solutions?
>
>> ... not calling `mu4e-fill-paragraph' on the signature seems
>> like a possible
>> solution. :-)
>> So, if I may ask, why do you refill your signature?
>
> As J said before, we don't do that. It happens if you refill
> *any* paragraph.

Yes, I missed (or overlooked) that part. It could be argued that
that is a bug, because filling a paragraph shouldn't affect text
that is obviously *not* part of that paragraph.

`mu4e-fill-paragraph' basically just calls `fill-paragraph', which
in turn uses the value of `fill-paragraph-function'. In messages
buffers, this is `message-fill-paragraph', which calls
`message-newline-and-reformat' to do its work. This, then, is
probably the function that reduces "-- " to "--", but I'd need to
take a closer look to figure out where exactly that's happening.

> So if mu4e-fill-paragraph uses the fill-paragraph but defined in
> the mu4e
> context, doesn't it make sense to add that detail here in mu4e?
> (sorry if I'm
> being imprecise, I will learn to program at some point in my
> life).
>
> Something like "do your thing up to the point where you find "--
> " and stop
> right there".

Dirk-Jan's idea to narrow the buffer before calling
`fill-paragraph' will work, but shouldn't really be necessary,
given that there is a message-specific fill function already. But
still, it's the quickest way for mu4e to work around this bug. (If
it is indeed a bug, of course. I may be overlooking something.)

Joost

[Eduardo Mercovich]

Hi Joost.

[...]

> `mu4e-fill-paragraph' basically just calls `fill-paragraph',
> which in turn uses the value of `fill-paragraph-function'. In
> messages buffers, this is `message-fill-paragraph', which calls
> `message-newline-and-reformat' to do its work. This, then, is
> probably the function that reduces "-- " to "--", but I'd need
> to take a closer look to figure out where exactly that's
> happening.

[...]

> Dirk-Jan's idea to narrow the buffer before calling
> `fill-paragraph' will work, but shouldn't really be necessary,
> given that there is a message-specific fill function already.

Then, should we modifiy the `message-fill-paragraph' function
instead?

> But still, it's the quickest way for mu4e to work around this
> bug. (If it is indeed a bug, of course. I may be overlooking
> something.)

It seems now that you're right on spot... :)

So, what do you think would be better? Limit the scope of
`fill-paragraph' to the message body or modify
`message-fill-paragraph'?

[Eduardo Mercovich]

Hi everyone.

[...]

>> I think fixing `mu4e-fill-paragraph` is the right solution, if
>> that is
>> indeed the culprit. I'm not entirely convinced that there isn't
>> some
>> other interaction going on with my emacs whitespace settings;
>> your
>> experience certainly seems to suggest otherwise though.

> Yeah, mu4e-fill-paragraphs doesn't recognize the concept of a
> 'message
> signature'. I think the solution would be (in
> `mu4e-fill-paragraph') to
> narrow the buffer to just the body part before calling
> `fill-paragraph'.

Any idea how to do that?

I believe this should be the standard behavior: `fill-paragraph'
shouldn't mess with the signature.

Thanks... :)

[Joost Kremers]

On Tue, Mar 28 2017, Eduardo Mercovich wrote:
>> Yeah, mu4e-fill-paragraphs doesn't recognize the concept of a
>> 'message
>> signature'. I think the solution would be (in
>> `mu4e-fill-paragraph') to
>> narrow the buffer to just the body part before calling
>> `fill-paragraph'.

[...]

> I believe this should be the standard behavior: `fill-paragraph'
> shouldn't mess
> with the signature.

Yes. And I just took a quick look at the code again and noticed
that the actual culprit is actually, mu4e, not fill-paragraph (or
message-fill-paragraph, which is what actually does the filling).
The function that is called when you fill a paragraph in a mu4e
message buffer is `mu4e-fill-paragraph', which contains a call to
`delete-trailing-whitespace' before it calls `fill-paragraph'.

I don't really understand why that call is there, in fact I would
think it'd be better to just delete it. But I'm sure Dirk has a
reason for it, so I'll let him comment on it.

I don't know how one can reliably narrow the buffer to the body of
the message. What you can do until Dirk fixes the issue is to
redefine `mu4e-fill-paragraph' in your init file after loading
mu4e:

```
(with-eval-after-load 'mu4e
(defun mu4e-fill-paragraph (&optional region)
"If `use-hard-newlines', takes a multi-line paragraph and
makes
it into a single line of text. Assume paragraphs are separated
by blank lines. If `use-hard-newlines' is not enabled, this
simply executes `fill-paragraph'.

This is a redefinition of the original function that removes the
calls to `delete-trailing-whitespace'."
;; Inspired by https://www.emacswiki.org/emacs/UnfillParagraph
(interactive (progn (barf-if-buffer-read-only) '(t)))
(if mu4e-compose-format-flowed
(let ((fill-column (point-max))
(use-hard-newlines nil)) ; rfill "across" hard
newlines
(fill-paragraph nil region))
(fill-paragraph nil region))))
```

Or define this function under a different name (e.g.,
`my-mu4e-fill-paragraph') and bind that to M-q.

HTH

[Stig Brautaset]

Joost Kremers <joostk...@fastmail.fm> writes:

> On Tue, Mar 28 2017, Eduardo Mercovich wrote:
>>> Yeah, mu4e-fill-paragraphs doesn't recognize the concept of a
>>> 'message
>>> signature'. I think the solution would be (in
>>> `mu4e-fill-paragraph') to
>>> narrow the buffer to just the body part before calling
>>> `fill-paragraph'.
> [...]
>> I believe this should be the standard behavior: `fill-paragraph'
>> shouldn't mess
>> with the signature.
>
> Yes. And I just took a quick look at the code again and noticed that
> the actual culprit is actually, mu4e, not fill-paragraph (or
> message-fill-paragraph, which is what actually does the filling). The
> function that is called when you fill a paragraph in a mu4e message
> buffer is `mu4e-fill-paragraph', which contains a call to
> `delete-trailing-whitespace' before it calls `fill-paragraph'.

Yeah, that `mu4e-fill-paragraph' function is annoying. I have a local
branch that removes it. (It really frustrates the editing experience
when you want to send format=flowed mails.)

> I don't really understand why that call is there, in fact I would
> think it'd be better to just delete it. But I'm sure Dirk has a reason
> for it, so I'll let him comment on it.
>
> I don't know how one can reliably narrow the buffer to the body of the
> message.

Might be worth checking out what `message-kill-to-signature' `does.
(Bound to `C-c C-z' in message mode.)

Stig
--
; GNU Emacs 25.1.1, Mu4e 0.9.19

[Dirk-Jan C. Binnema]

I think the delete-trailing-whitespace is because of a patch I took that
fixes some cases where text was filled incorrectly when there was, well,
trailing whitespace. And I have seen that happening.

So simply back out the change is probably not the best solution -- but
I'd be happy to take a PR that narrows to the message body before
doing the delete-trailing-whitespace.

[Eduardo Mercovich]

Hi everyone.

[...]

Thanks Joost, Stig and Dirk.
I'll try Joost modified funcion (and report back if something
needs attention) in the meantime until Dirk defines what gets into
mu4e stable code. :)

Best...