File corruption after transmission over GMail using mutt + msmtp

[Marcelo Laia]

Nowadays, email recipients had give me feedback that attached files that
I sent was corrupted. I use Mutt + msmtp + offlineimap and OAuth GMail
app implementation.

Here is headers I got from ec...@xxxxxxx.xxxx.xx:

From: Marcelo <xxxx...@gmail.com>
To: echo <ec...@xxxxx.xxxxx.xx>
Date: Fri, 20 May 2022 12:35:49 -0300
Subject: Test 7

[-- Anexo No.1 --]
[-- Tipo: text/plain, Codificação: 7bit, Tamanho: 0K --]


[-- Anexo No.2: TCC-v3.docx --]
[-- Tipo: application/vnd.openxmlformats-officedocument.wordprocessingml.document, Codificação:
base64, Tamanho: 71K --]

[-- application/vnd.openxmlformats-officedocument.wordprocessingml.document não é aceito (use
'v' para ver esta parte) --]

And here is Attached View (after v in mutt page index)

I 1 <no description> [text/plain, 7bit, us-ascii, 0K]
A 2 TCC-v3.docx [applica/vnd.openxm, base64, 71K]

And here [1] is the Attached file that are stored in GMail sent box.

My muttrc have this:

set send_charset="us-ascii:iso-8859-1:iso-8859-15:windows-1252:utf-8"

I already tried this, also:

set use_8bitmime

or

set use_8bitmime = yes

$ uname -a
$ Linux marcelo 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux

$ mutt -v
Mutt 2.2.4 (2022-04-30)
System: Linux 5.17.0-1-amd64 (x86_64)
ncurses: ncurses 6.3.20220423 (compiled with 6.3)
libidn2: 2.3.2 (compiled with 2.3.2)
hcache backend: tokyocabinet 1.4.48

$ msmtp --version
msmtp versão 1.8.16

Have you some clue to point me out?

[1] https://www.dropbox.com/s/mnomo19xgiisoyq/TCC-v3.docx?dl=0

--
Marcelo

[Marcelo Laia]

On 21/05/22 at 04:09, Marcelo Laia wrote:
> Nowadays, email recipients had give me feedback that attached files that
> I sent was corrupted. I use Mutt + msmtp + offlineimap and OAuth GMail
> app implementation.
>

After a lot of tests, I found:

1. Files with image/figure was damaged

2. If images are remove from files, they are transferred by mutt intact.

Could you do a test for me? If so, please, download the next files and
send it to me, one per message, using GMail. Send direct to my private
e-mail.

pdf with image/figure
https://www.dropbox.com/s/y11l265wjt4dbbz/Mutt-test-estagio_assinado.pdf?dl=0

docx with image/figure
https://www.dropbox.com/s/ifu7qf1xnp2ev97/Mutt-test-figure.docx?dl=0

docx without image/figure
https://www.dropbox.com/s/q4c0t7vb8tu6ikp/Mutt-test.docx?dl=0

Please, before send, inspect it if they are opened very well in your
machine.

Tell me what system are you using. I use Debian testing (Debian 5.17.3-1
(2022-04-18) x86_64)

Thank you so much!

--
Marcelo

[David Wright]

It's relatively easy to check on what mutt is actually sending.
Run mutt with -d 5 in the command line.
Send an email with an attachment in the usual way. (For convenience,
pick a file to attach that isn't too big.) Then quit mutt.¹
There will be a file called .muttdebug0 in your home directory.

What the file contains is something like this:

[ … loads of stuff … ]
[2022-03-19 12:48:15] 5> --wRRV7LY7NUeQGEoC
[2022-03-19 12:48:15] 5> Content-Type: application/pdf
[2022-03-19 12:48:15] 5> Content-Disposition: attachment; filename="Quick+Setup+Guide.pdf"
[2022-03-19 12:48:15] 5> Content-Transfer-Encoding: base64
[2022-03-19 12:48:15] 5>
[2022-03-19 12:48:15] 5> JVBERi0xLjUNJeLjz9MNCjI0IDAgb2JqDTw8L0xpbmVhcml6ZWQgMS9MIDMyMTY0NS9PIDI3
[2022-03-19 12:48:15] 5> L0UgMjEyOTQ5L04gMi9UIDMyMTI5My9IIFsgNjA5IDIzNV0+Pg1lbmRvYmoNICAgICAgICAg
[2022-03-19 12:48:15] 5> ICAgICAgDQo2MCAwIG9iag08PC9EZWNvZGVQYXJtczw8L0NvbHVtbnMgNS9QcmVkaWN0b3Ig
[ … many of these … ]
[2022-03-19 12:48:15] 5> q1FHQx0wUOsreft/LOhs9jKG7U70d/Lrz7bn0OuvBm64luc2IxMrDz2H0Rr1EwYCSzoK2EY7
[2022-03-19 12:48:15] 5> ihr5dNgMkb3iKgKQk3XgOlBR84+3GTEqo5ujQn1a3PKQ5rQIgA481MAEyRPakpFi7GXxgCYO
[2022-03-19 12:48:15] updating progress: 390K
[2022-03-19 12:48:15] Sending message... 390K/425K (91%)
[2022-03-19 12:48:15] 5> /kZWjnK30PS4m5RWtGK1n19EA7PLCmVbpPwqafIjlkysU1t9/322csrZ5HWgNVRxEJ3Y4MrH
[2022-03-19 12:48:15] 5> OP6OHO/jiMmFJXEyECcDfBM5IpHiuYgrA1tAmoXLq4Fa8T9pcjRVwggGuIO+WArrDGCPhbOd
[ … many of these … ]
[2022-03-19 12:48:15] 5> cmVhbQ0KaN5iYgABJkZm+6sMTAwMLL5AkvH0QRBbYi+IVFoHIhWYQKSYIIiUFkfIKoJlGTeD
[2022-03-19 12:48:15] 5> SKFvYPZkMPkeaOYfyT8gNpBHmGT6BSIZGQACDAA9kA1ADQplbmRzdHJlYW0NZW5kb2JqDXN0
[2022-03-19 12:48:15] 5> YXJ0eHJlZg0KMTE2DQolJUVPRg0K
[2022-03-19 12:48:15] 5>
[2022-03-19 12:48:15] 5> --wRRV7LY7NUeQGEoC--
[2022-03-19 12:48:15] 5> .
[2022-03-19 12:48:16] 5< 250 VdBUnOhv57cXuVdBWncIHF mail accepted for delivery
[2022-03-19 12:48:16] 5> QUIT
[2022-03-19 12:48:16] Mail sent.
[ … loads more stuff … ]

The stuff at the beginning and end may vary according to how your
emails are sent (here it's direct to an ISP's smarthost, no exim).
If the file is large, there may be one or more of those interruptions
regarding progress.

Use whatever tools you prefer to clean up the block of base64.
Searching for " Content-" will get to the start of the attachment,
and "Mail sent." then end.

Grepping the "] 5> " strings will remove the interruptions; then cut
or an editor's global replace will remove the timestamps.
You need to end up with a file containing just the block:

JVBERi0xLjUNJeLjz9MNCjI0IDAgb2JqDTw8L0xpbmVhcml6ZWQgMS9MIDMyMTY0NS9PIDI3
L0UgMjEyOTQ5L04gMi9UIDMyMTI5My9IIFsgNjA5IDIzNV0+Pg1lbmRvYmoNICAgICAgICAg
ICAgICAgDQo2MCAwIG9iag08PC9EZWNvZGVQYXJtczw8L0NvbHVtbnMgNS9QcmVkaWN0b3Ig
[ … many of these … ]
q1FHQx0wUOsreft/LOhs9jKG7U70d/Lrz7bn0OuvBm64luc2IxMrDz2H0Rr1EwYCSzoK2EY7
ihr5dNgMkb3iKgKQk3XgOlBR84+3GTEqo5ujQn1a3PKQ5rQIgA481MAEyRPakpFi7GXxgCYO
/kZWjnK30PS4m5RWtGK1n19EA7PLCmVbpPwqafIjlkysU1t9/322csrZ5HWgNVRxEJ3Y4MrH
OP6OHO/jiMmFJXEyECcDfBM5IpHiuYgrA1tAmoXLq4Fa8T9pcjRVwggGuIO+WArrDGCPhbOd
[ … many of these … ]
cmVhbQ0KaN5iYgABJkZm+6sMTAwMLL5AkvH0QRBbYi+IVFoHIhWYQKSYIIiUFkfIKoJlGTeD
SKFvYPZkMPkeaOYfyT8gNpBHmGT6BSIZGQACDAA9kA1ADQplbmRzdHJlYW0NZW5kb2JqDXN0
YXJ0eHJlZg0KMTE2DQolJUVPRg0K

Now decode the base64, and check whether you get the original file contents:

$ base64 -d block-of-base64 > copy-of-original
$ cmp copy-of-original Quick+Setup+Guide.pdf
$

If it differs, vbindiff copy-of-original Quick+Setup+Guide.pdf
might give you an idea of just how different.

¹ If you start doing other things, like checking your mail, the
debug file, being at level 5, will grow at an astonishing rate.

Cheers,
David.

[Marcelo Laia]

On 21/05/22 at 04:09, Marcelo Laia wrote:
> Nowadays, email recipients had give me feedback that attached files that
> I sent was corrupted. I use Mutt + msmtp + offlineimap and OAuth GMail
> app implementation.

After more tests, I found that if I use the mutt native smtp, attach is
properly transmitted. I could open it very well!

So, I think there is a problem with msmtp or its configuration.


MSMTP config (msmtprc)

defaults
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log

account gmail
auth oauthbearer
host smtp.gmail.com
port 587
protocol smtp
tls on
from myl...@gmail.com
user myl...@gmail.com
passwordeval /path/to/scripts/OAuth2/get_teteraorg_token_gmail.sh


MUTT native smtp relevant config (muttrc)

set smtp_oauth_refresh_command="/path/to/scripts/OAuth2/get_teteraorg_token_gmail.sh"
set smtp_authenticators="oauthbearer"
set smtp_url = "smtp://myl...@gmail.com@smtp.gmail.com:587/"
set from = "myl...@gmail.com"
set use_from = yes
set realname = "My Real Name"


This is only changes I do to get all works very well!


--
Marcelo

[Marcelo Laia]

On 22/05/22 at 03:30, Marcelo Laia wrote:
> On 21/05/22 at 04:09, Marcelo Laia wrote:
> > Nowadays, email recipients had give me feedback that attached files that
> > I sent was corrupted. I use Mutt + msmtp + offlineimap and OAuth GMail
> > app implementation.
>
> After more tests, I found that if I use the mutt native smtp, attach is
> properly transmitted. I could open it very well!
>
> So, I think there is a problem with msmtp or its configuration.

No. Isn't msmtp problems.

> This is only changes I do to get all works very well!

I did more tests and found:

If I send a message using

tee >(lbdb-fetchaddr -a)|/usr/bin/msmtp -a gmail $@

the attached file with embedded image got corrupted

The same result is got with

tee >(lbdb-fetchaddr -a -c utf8)

If I send a message using

/usr/bin/msmtp -a gmail $@

the attached file with embedded image got loaded correctly.

Have you some clue, here?