Bug#1028356: procmail: Variable set with stdin pipe action fails leaving empty variable

[Bob Proulx]

Package: procmail
Version: 3.24-1
Severity: normal

I am very happy to see procmail upstream is now active again!
Excellent news! I am a long time user and have quite a few mail
filtering rules.

The recent upgrade to 3.24 broke using a pipe action to set a variable
using stdin. Reverting to 3.22 restores correct functioning.

For example due to UTF-8 subject encoding one might find a subject
with this contrived and constructed example.

Subject: =?utf-8?q?test-list_post_from_rwp=40proulx=2Ecom_require?=
=?utf-8?q?s_approval?=

I use the following idiomatic procmail expression to decode that
subject and place it into a SUBJECT variable. This following from my
.procmailrc file.

# If the subject is encoded and needs decoding then...
# extract the subject, decode it, store in the SUBJECT variable.
:0h
* ^Subject:.*=\?
SUBJECT=|formail -cXSubject: | perl -MEncode -pe 'Encode::from_to($_,"MIME-Header","UTF-8");'

# Store all remaining cases of Subject: into $SUBJECT.
# E - only execute this recipe if previous recipe was not executed.
:0hE
SUBJECT=|formail -cXSubject:

# Here forward SUBJECT contains the decoded text.
# Match it using the variable ?? syntax.
# * SUBJECT ?? ^Subject: blah blah blah

Then I use it like this following to handle Mailman messages.

:0
* ^X-BeenThere:
* SUBJECT ?? ^Subject: .*mailing list memberships reminder$
mailman-reminders/

:0
* ^X-List-Administrivia: yes
{
:0
* 1^0 SUBJECT ?? ^Subject: Auto-discard notification
* 1^0 SUBJECT ?? ^Subject: El envio a [-a-zA-Z0-9]+ de.*precisa.*de.*aprobacion
* 1^0 SUBJECT ?? ^Subject: Uncaught bounce notification
* 1^0 SUBJECT ?? ^Subject: [-a-zA-Z0-9]+ post from
* 1^0 SUBJECT ?? ^Subject: [-a-zA-Z0-9]+ subscription notification
* 1^0 SUBJECT ?? ^Subject: [-a-zA-Z0-9]+ unsubscribe notification
list-administrivia/
}

Unfortunately in 3.24 this functionality is broken. I see in the
Testing migration "excuses" this "Too young, only 4 of 5 days old".
Should severity be "important" in order to keep Testing from breaking?

Some experiments show to try to bound the condition. It's around the
use of stdin to the sub-process. This following works. This does not
use stdin.

SUBJECT=|echo hello
SUBJECT=|echo hello | tee /tmp/pt.rwp.debug
SUBJECT=|sh -c "echo hello | tee /tmp/pt.rwp.debug"

But this fails. (Using a cat| pipe to ensure $SHELLMETAS are seen.)
This fails to process stdin.

SUBJECT=|tee /tmp/pt.rwp.debug
SUBJECT=|cat | tee /tmp/pt.rwp.debug
SUBJECT=|sh -c "tee /tmp/pt.rwp.debug"
SUBJECT=|sh -c "cat | tee /tmp/pt.rwp.debug"

Thank you for maintaining procmail! :-)

Bob


-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-1-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages procmail depends on:
ii libc6 2.36-8

Versions of packages procmail recommends:
ii postfix [mail-transport-agent] 3.7.3-2+b1

procmail suggests no packages.

-- no debconf information

[Santiago Vila]

El 9/1/23 a las 23:13, Bob Proulx escribió:

> Unfortunately in 3.24 this functionality is broken. I see in the
> Testing migration "excuses" this "Too young, only 4 of 5 days old".
> Should severity be "important" in order to keep Testing from breaking?

I've just set it to serious, which is the minimal severity to prevent
a migration to testing.

I don't want regressions of any kind for this package. If this is
not possible, we'll have to stick with old 3.22 for bookworm.

I'll forward this to upstream and see what happens.

Thanks a lot.

[Santiago Vila]

Hello.

This is what I did today. Maybe someone can follow my steps
and arrive at something.

I'm using this $HOME/.procmailrc file:

---------------------------------
LOGFILE=$HOME/.cache/procmail.log
VERBOSE=yes

:0 h
SUBJECT=|formail -cXSubject:|cat

MYVAR=$SUBJECT

:0
/dev/null
---------------------------------

and this command to test it:

echo "Subject: thesubject" | formail | procmail

A non-buggy procmail (for example, the one in bullseye) will
produce a $HOME/.cache/procmail.log file like this:

procmail: [2236560] Tue Feb 14 20:27:41 2023
procmail: Assigning "SUBJECT="
procmail: [2236560] Tue Feb 14 20:27:41 2023
procmail: Executing "formail -cXSubject:|cat"
procmail: Assigning "MYVAR=Subject: thesubject"
procmail: Assigning "LASTFOLDER=/dev/null"
procmail: Opening "/dev/null"
procmail: Notified comsat: "bluser@0:/dev/null"
From foo@bar Tue Feb 14 20:27:41 2023
Subject: thesubject
Folder: /dev/null 60


However, a buggy procmail (for example, procmail_3.24-1) will
produce a $HOME/.cache/procmail.log file like this:

procmail: [2236823] Tue Feb 14 20:30:28 2023
procmail: Assigning "SUBJECT="
procmail: [2236823] Tue Feb 14 20:30:28 2023
procmail: Executing "formail -cXSubject:|cat"
procmail: Assigning "MYVAR="
procmail: Assigning "LASTFOLDER=/dev/null"
procmail: Opening "/dev/null"
procmail: Notified comsat: "bluser@0:/dev/null"
From foo@bar Tue Feb 14 20:30:28 2023
Subject: thesubject
Folder: /dev/null 60

The relevant thing here is that MYVAR is empty.


Now, to do a git-bisect (or something which resembles it),
I'm doing this:

git checkout some-git-id --force

To avoid putting the debian/* files on top of this,
I'm using this simple script "make-procmail" to compile:

#!/bin/sh
set -e
make CC=gcc CFLAGS0="" LDFLAGS0="" SEARCHLIBS="-lm" LOCKINGTEST=100
cat src/procmail > /usr/bin/procmail

where I've previously made /usr/bin/procmail mode 777 for that to work.
(This is naturally in a test machine).

It is possible that the above fails with "text file busy", then
I execute this other script as root:

#!/bin/sh
set -e
cd /usr/bin
rm -f procmail
touch procmail
chmod 777 procmail


If "some-git-id" is too old, the following error may arise:

formisc.h:20:2: error: conflicting types for ‘getline’
20 | getline P((void));
| ^~~~~~~
In file included from includes.h:41,
from formail.c:16:
/usr/include/stdio.h:616:18: note: previous declaration of ‘getline’ was here
616 | extern __ssize_t getline (char **__restrict __lineptr,

(This was Bug #549426 fixed in version 3.22-18).

In such case this is enough to get rid of such error:

git show 2df8bdb0d2557d6cdc7632e | patch -p1


Now the bad news:

According to my tests, this commit, tagged v3.23pre, is already buggy:

commit 98ca67898b3f38ea09f1a0d1119d0794c044f9b6 (HEAD, tag: v3.23pre)
Author: Philip Guenther <guen...@sendmail.com>
Date: Tue Nov 1 03:15:48 2005 +0000

Fixed properties

The previous commit is huge and it's already buggy:

commit 5281aac102d92966c5963ea8694d95e4f1103747
Author: Philip Guenther <guen...@sendmail.com>
Date: Tue Nov 1 01:44:56 2005 +0000

Various fixes.

[lots of stuff]


And finally, the commit before that makes this simple command

echo "Subject: thesubject" | formail | procmail

to take some time to return (locking problems?) and
the log output on procmail.log is like this:

procmail: [2243217] Tue Feb 14 21:04:28 2023
procmail: Assigning "SUBJECT="
procmail: [2243217] Tue Feb 14 21:04:28 2023
procmail: Executing "formail -cXSubject:|cat"
procmail: Out of memory
buffer 0: "formail -cXSubject:|cat"
buffer 1: "formail -cXSubject:|cat"
procmail: Notified comsat: "bluser@:**Bounced**"
From foo@bar Tue Feb 14 21:04:28 2023
Folder: **Bounced** 0

Because there is not a MYVAR= line, I'm not even able
to tell if this is also buggy or not.

So, to summarize: I still need help to fix this...

Thanks.