Bug#1006830: elpa-mailscripts: notmuch-slurp-debbugs misses messages

[Vagrant Cascadian]

Package: elpa-mailscripts
Version: 0.24-1
Severity: normal
X-Debbugs-Cc: Vagrant Cascadian <vag...@debian.org>

I've noticed recently some messages do not get downloaded when I run
from emacs:

M-x notmuch-slurp-debbugs

I've noticed this in the recent past on other bugs, but didn't get a
chance to file a bug. This time I noticed it with:

https://bugs.debian.org/1004939

Where it simply doesn't download message #10 with message id
LTFE7R.CCN...@pappacoda.it

I manually downloaded the mbox for that specific message, and used
mbox2maildir to move it into one of my folders and the message appeared
just fine...



live well,
vagrant

-- System Information:
Debian Release: bookworm/sid
APT prefers testing
APT policy: (500, 'testing'), (1, 'experimental'), (1, 'unstable')
Architecture: arm64 (aarch64)

Kernel: Linux 5.16.0-3-arm64 (SMP w/6 CPU threads)
Kernel taint flags: TAINT_CRAP
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages elpa-mailscripts depends on:
ii dh-elpa-helper 2.0.10
ii elpa-notmuch 0.35-1
ii emacsen-common 3.0.4
ii mailscripts 0.24-1

Versions of packages elpa-mailscripts recommends:
ii emacs 1:27.1+1-3.1
ii emacs-nox [emacs] 1:27.1+1-3.1+b1

elpa-mailscripts suggests no packages.

-- no debconf information

[Sean Whitton]

Hello,

On Sat 05 Mar 2022 at 03:40pm -08, Vagrant Cascadian wrote:

> Package: elpa-mailscripts
> Version: 0.24-1
> Severity: normal
> X-Debbugs-Cc: Vagrant Cascadian <vag...@debian.org>
>
> I've noticed recently some messages do not get downloaded when I run
> from emacs:
>
> M-x notmuch-slurp-debbugs
>
> I've noticed this in the recent past on other bugs, but didn't get a
> chance to file a bug. This time I noticed it with:
>
> https://bugs.debian.org/1004939
>
> Where it simply doesn't download message #10 with message id
> LTFE7R.CCN...@pappacoda.it
>
> I manually downloaded the mbox for that specific message, and used
> mbox2maildir to move it into one of my folders and the message appeared
> just fine...

Could you determine whether downloading the mailbox like this contains
the message:

bts --bts-server https://bugs.debian.org --mbox --mailreader "true %s" show 1004939

--
Sean Whitton

[Vagrant Cascadian]

It didn't appear to visibly do anything (I guess true is a very low
interactive mailreader), although the .mbox did land in
.cache/devscripts/bts/ and contains the mentioned Message-Id:

$ grep LTFE7R.CCN...@pappacoda.it .cache/devscripts/bts/1004939.mbox
Message-Id: <LTFE7R.CCN...@pappacoda.it>
References: <LTFE7R.CCN...@pappacoda.it> <87czk3pluc.fsf@contorta>
In-Reply-To: <LTFE7R.CCN...@pappacoda.it>

The references/in-repy-to's are from my reply to the message after
having manually imported the message into my maildir...


FWIW, I only use emacs-nox instead of one of the GUI emacs frontends.


live well,
vagrant

[Sean Whitton]

control: tag -1 + unreproducible

Hello Vagrant,

Thanks. As it is included in the downloaded mbox, we can assume that
notmuch-slurp-debbug(1) is skipping over that message somehow, or
notmuch isn't processing it, or something like that.

Unfortunately I can't reproduce here, however -- I get all three
messages for #1004939. Let's keep this bug open as you've seen this
more than once. Let me know if you have other cases.

--
Sean Whitton

[Vagrant Cascadian]

Not sure if this is an entirely different or related problem, but also
having issues with 969697, this time getting none of the messages on the
bug.

So I tried the commandline interface instead, which had some curious
messages:

$ notmuch-slurp-debbug 969697
WARNING: Folder does not exist, failed opening maildir folder ~/Mail/debian/INBOX.debian-bugs/.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in -d at /usr/share/perl5/Mail/Box/MH.pm line 99.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in concatenation (.) or string at /usr/share/perl5/Mail/Box/Maildir.pm line 77.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in -d at /usr/share/perl5/Mail/Box/MH.pm line 99.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in concatenation (.) or string at /usr/share/perl5/Mail/Box/Maildir.pm line 77.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in -d at /usr/share/perl5/Mail/Box/MH.pm line 99.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in concatenation (.) or string at /usr/share/perl5/Mail/Box/Maildir.pm line 77.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in -d at /usr/share/perl5/Mail/Box/MH.pm line 99.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in concatenation (.) or string at /usr/share/perl5/Mail/Box/Maildir.pm line 77.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in string eq at /usr/share/perl5/Mail/Box/Manager.pm line 296.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in -d at /usr/share/perl5/Mail/Box/MH.pm line 99.
Use of uninitialized value $name in pattern match (m//) at /usr/share/perl5/Mail/Box/Dir.pm line 105.
Use of uninitialized value $dir in substitution (s///) at /usr/share/perl5/Mail/Box/Dir.pm line 106.
Use of uninitialized value $directory in concatenation (.) or string at /usr/share/perl5/Mail/Box/Maildir.pm line 77.
No new mail.
WARNING: Changes not written to read-only folder /home/vagrant/.cache/devscripts/bts/969697.mbox.
Suggestion: $folder->close(write => 'NEVER')

So, ~/Mail/debian/INBOX.debian-bugs/ definitely does exist, and seems to
be a valid Maildir. If it matters, ~/Mail/debian is a symlink.

My .config/mailscripts/notmuch-slurp-debbug contains:

maildir = ~/Mail/debian/INBOX.debian-bugs/

Also not sure why it thinks the .mbox file is a read-only folder ... it
appears writeable:

$ ls -latr .cache/devscripts/bts/969697.mbox
-rw-r--r-- 1 vagrant vagrant 11605 Mar 16 16:12 .cache/devscripts/bts/969697.mbox


And Now I Tried Changing .config/mailscripts/notmuch-slurp-debbug to:

maildir = /home/vagrant/debian/INBOX.debian-bugs/


Which at least works, with no errors or warnings from
notmuch-slurp-debbug.

The ~ for the homedir definitely *used* to be supported; been using it
since at least ~debconf18 that way. :)


live well,
vagrant

[Sean Whitton]

Hello Vagrant,

On Wed 16 Mar 2022 at 04:24PM -07, Vagrant Cascadian wrote:

> Not sure if this is an entirely different or related problem, but also
> having issues with 969697, this time getting none of the messages on the
> bug.

If the problem really is the lack of tilde expansion, it would be a
different bug, because tilde expansion not working would mean you get no
message, rather than some.

> So I tried the commandline interface instead, which had some curious
> messages:
>
> $ notmuch-slurp-debbug 969697
> WARNING: Folder does not exist, failed opening maildir folder ~/Mail/debian/INBOX.debian-bugs/.

Erk, why doesn't it die at that point?!

> So, ~/Mail/debian/INBOX.debian-bugs/ definitely does exist, and seems to
> be a valid Maildir. If it matters, ~/Mail/debian is a symlink.
>
> My .config/mailscripts/notmuch-slurp-debbug contains:
>
> maildir = ~/Mail/debian/INBOX.debian-bugs/
>
> Also not sure why it thinks the .mbox file is a read-only folder ... it
> appears writeable:
>
> $ ls -latr .cache/devscripts/bts/969697.mbox
> -rw-r--r-- 1 vagrant vagrant 11605 Mar 16 16:12 .cache/devscripts/bts/969697.mbox

It's just because it's opened read-only by the script.

> And Now I Tried Changing .config/mailscripts/notmuch-slurp-debbug to:
>
> maildir = /home/vagrant/debian/INBOX.debian-bugs/
>
>
> Which at least works, with no errors or warnings from
> notmuch-slurp-debbug.
>
> The ~ for the homedir definitely *used* to be supported; been using it
> since at least ~debconf18 that way. :)

Hmm... I don't see how it could have worked before. Does this fix it:

> diff --git a/notmuch-slurp-debbug b/notmuch-slurp-debbug
> index ad0db47..1568d9c 100755
> --- a/notmuch-slurp-debbug
> +++ b/notmuch-slurp-debbug
> @@ -50,7 +50,7 @@ if (-f $conf_f) {
> $maildir = catfile $database_path, "inbox";
> }
> $maildir = $mgr->open(
> - folder => $maildir,
> + folder => glob $maildir,
> access => "a",
> keep_dups => 1,
> type => "maildir"

--
Sean Whitton

[Sean Whitton]

Hello,

On Wed 16 Mar 2022 at 04:36PM -07, Sean Whitton wrote:

> Hmm... I don't see how it could have worked before. Does this fix it:
>
>> diff --git a/notmuch-slurp-debbug b/notmuch-slurp-debbug
>> index ad0db47..1568d9c 100755
>> --- a/notmuch-slurp-debbug
>> +++ b/notmuch-slurp-debbug
>> @@ -50,7 +50,7 @@ if (-f $conf_f) {
>> $maildir = catfile $database_path, "inbox";
>> }
>> $maildir = $mgr->open(
>> - folder => $maildir,
>> + folder => glob $maildir,
>> access => "a",
>> keep_dups => 1,
>> type => "maildir"

This patch won't work. I just pushed a commit which does.

--
Sean Whitton