[ANN] filter for text-only base64-encoded msgs [LINUX][FOSS]

[Dario Niedermann]

Sometimes you'll get an email - without attachments, just text - and
that email is base64-encoded for no good reason. Spammers have been
known to do that in order to bypass spam filters. Mailing list soft-
ware has been unwisely configured to send out encoded messages. Or
clueless users tweaked their client the wrong way...

The end results:

1. any rule in your mail processing chain matching on message body
will fail;

2. ungreppable messages end up in your stored email.

`debbie' was written to address this specific problem. [...]

See <https://www.darioniedermann.it/sw/debbie.html> for more info.

--
Dario Niedermann. Also on the Internet at:

gopher://darioniedermann.it/ <> https://www.darioniedermann.it/

[Eli the Bearded]

X-ORIGINAL-NEWSGROUPS: comp.mail.misc,comp.mail.headers,comp.mail.mime

I don't subscribe to .headers, but .mime is deader than this group.

In comp.mail.misc, Dario Niedermann <da...@darioniedermann.it> wrote:
> Sometimes you'll get an email - without attachments, just text - and
> that email is base64-encoded for no good reason. Spammers have been
> known to do that in order to bypass spam filters. Mailing list soft-
> ware has been unwisely configured to send out encoded messages. Or
> clueless users tweaked their client the wrong way...
>
> The end results:
>
> 1. any rule in your mail processing chain matching on message body
> will fail;
>
> 2. ungreppable messages end up in your stored email.
>
> `debbie' was written to address this specific problem. [...]
>
> See <https://www.darioniedermann.it/sw/debbie.html> for more info.

"You don't have permission to access /sw/archives/debbie-1.0.tbz on this
server."

Here's how I've been doing it for years:

:r! cat ~/.procmail.base64
# inline decode non-multiparts that are base64

:0
* ^MIME-Version: 1.0
* ^Content-Type: *text/
* ^Content-Transfer-Encoding:[ ]*base64
{
# 'b' body, 'f' filter, 'w' wait to check exit code
:0bfw
| /usr/local/bin/mmencode -b -u;echo

# 'a' and also (but only if above exit code indicated success),
# 'h' headers, 'f' filter
:0ahf
| formail -R Content-Transfer-Encoding X-Orig-Content-Transfer-Encoding
}

The mmencode program has gotten pretty obscure these days. Modern Linux
users probably want to switch in "/usr/bin/base64 -d". mmencode has the
advantage of also being able to encode/decode quoted-printable.

> gopher://darioniedermann.it/ <> https://www.darioniedermann.it/

Your gopher software page clearly gets updated a lot less frequently
than your web one.

Elijah
------
has another recipe file for detaching attachments

[Dario Niedermann]

Eli the Bearded <*@eli.users.panix.com> wrote:

> "You don't have permission to access /sw/archives/debbie-1.0.tbz on this
> server."

I know, sorry about that. There's been a bit of a panic here about
an apparently missing message (within an hour from release and after
weeks of testing, mind you). So I blocked the archive. But further
investigation showed that debbie had done the right thing by passing the
message through, and the message wasn't even missing to begin with.

More confident than ever in debbie's safety, the tarball is back online.

[...]

> Your gopher software page clearly gets updated a lot less frequently
> than your web one.

It's not a mirror. If you are referring to my 'datekit' for 4.3BSD, it's
recent stuff.

[Eli the Bearded]

In comp.mail.misc, Dario Niedermann <da...@darioniedermann.it> wrote:

> More confident than ever in debbie's safety, the tarball is back online.

Your Makefile didn't make debbie for me. A plain "make" stops at
creating the debbie.c file. (/usr/bin/make on NetBSD 8.1). It looks like
you have gmake assumptions.

debbie.l.m4 line 87:
if (stReg & (B64_FOUND | IS_TEXT )

Shouldn't that be:
if (((stReg & (B64_FOUND | IS_TEXT )) ==
(B64_FOUND | IS_TEXT ))

Otherwise it will attempt to decode the body if it is text OR if it is
B64_FOUND. So "Content-Type: image/jpeg" gets decoded and CTE other
than base64 gets "decoded".

The suggested procmail recipe is saving your ass by doing the AND for
you.

Quoting me:
>> Your gopher software page clearly gets updated a lot less frequently
>> than your web one.
> It's not a mirror. If you are referring to my 'datekit' for 4.3BSD, it's
> recent stuff.

The web version says newest stuff is first. The .sig implies that gopher
is master and web is "also" -- secondary. There's no debbie in the
gopher space version.

> --
> Dario Niedermann. Also on the Internet at:
>
> gopher://darioniedermann.it/ <> https://www.darioniedermann.it/

Elijah
------
reading the dot sig as having two columns

[Dario Niedermann]

Eli the Bearded <*@eli.users.panix.com> wrote:

> Your Makefile didn't make debbie for me. A plain "make" stops at
> creating the debbie.c file. (/usr/bin/make on NetBSD 8.1). It looks
> like you have gmake assumptions.

Of course I do. It's made on Linux (see Subject) with the usual
GNU toolchain.

> debbie.l.m4 line 87:
> if (stReg & (B64_FOUND | IS_TEXT )
>
> Shouldn't that be:
> if (((stReg & (B64_FOUND | IS_TEXT )) ==
> (B64_FOUND | IS_TEXT ))
>
> Otherwise it will attempt to decode the body if it is text OR if it is
> B64_FOUND.

Yeah. Well spotted. I have oversimplified that expression. I've been
working on debbie-2.0, which among several other changes, doesn't use
that construct. It's due out very soon. In the mean time, using my
procmail recipe will hide the bug.

> The .sig implies that gopher is master and web is "also" -- secondary.

No, it doesn't... Just follow the link in the message body and you'll be
fine.

[Eli the Bearded]

In comp.mail.misc, Dario Niedermann <da...@darioniedermann.it> wrote:

> Eli the Bearded <*@eli.users.panix.com> wrote:
> > Your Makefile didn't make debbie for me. A plain "make" stops at
> > creating the debbie.c file. (/usr/bin/make on NetBSD 8.1). It looks
> > like you have gmake assumptions.
> Of course I do. It's made on Linux (see Subject) with the usual
> GNU toolchain.

1. NetBSD is also FOSS.
2. INSTALL does not say "needs gmake", it says "needs make". I mean, you
do call out flex instead of lex, not that it matters since it built and
ran fine for me with lex instead of flex.
3. And fixing your Makefile to work with non-gmake is very simple.

--- Makefile.orig 2019-09-29 12:10:43.000000000 -0400
+++ Makefile 2019-09-30 23:40:46.174357853 -0400
@@ -19,10 +19,10 @@
M4 = m4 $(M4FLAGS)
TBL =

-.INTERMEDIATE: debbie.c debbie.l debbie.1
-
all: debbie debbie.1.gz

+.INTERMEDIATE: debbie.c debbie.l debbie.1
+
debbie: debbie.c

debbie.c: debbie.l

> Yeah. Well spotted. I have oversimplified that expression. I've been
> working on debbie-2.0, which among several other changes, doesn't use
> that construct. It's due out very soon. In the mean time, using my
> procmail recipe will hide the bug.

If you are taking requests, I'd find this program more useful if it
could, possibly gated by command line options:

1. Also decode quoted-printable (the easiest and least exciting ask).
2. Correctly parse multipart/*, decoding all text/plain or text/* parts
(more complicated).
3. Accept command line arguments for which MIME types in multipart/* (or
whole non-multipart messages) to decode. (Most complicated.)

>> The .sig implies that gopher is master and web is "also" -- secondary.
> No, it doesn't... Just follow the link in the message body and you'll be
> fine.

Well, we have a difference of opinion.

Elijah
------
already has a working solution for text/plain message bodies

[Dario Niedermann]

> Sometimes you'll get an email - without attachments, just text - and

> that email is base64-encoded for no good reason. [...] The end results:

>
> 1. any rule in your mail processing chain matching on message body
> will fail;
>
> 2. ungreppable messages end up in your stored email.
>
> `debbie' was written to address this specific problem. [...]

Version 2.0 includes the following improvements:

* adds an "X-Original-Content-Transfer-Encoding:" header
to filtered messages

* bug fixes!