Mopher & Sendmail

[Chris Weber]

At the Moment i play arround with moper unter Debian7 + Sendmail.
It seems that mopher would like to have the 'v' Macro in the Milter CONNECT stage:

mopherd[4586]: milter_connect: smfi_getsymval for "v" failed


To provide it, i have to add it to the sendmail.mc and regenerate sendmail.cf:
---cut---
dnl -------------------------------------------------------------------
dnl The default for confMILTER_MACROS_CONNECT is typically
dnl `j, _, {daemon_name}, {if_name}, {if_addr}'
define(`confMILTER_MACROS_CONNECT', confMILTER_MACROS_CONNECT`, v')dnl
dnl
---cut---

[Manuel Badzong]

Thanks for sharing. Will be put into the docs asap.

Mopher is mainly tested with Postfix. If you encounter any problems using Sendmail or if you need assistance getting mopher up an running please let me know.

Cheers, Manuel

[Chris Weber]

Thank you for the offering. I only have a issue with spamassassin / spamd at the moment, but will look into that on my side first :-)

Some cosmetic things i did: prefering a more actual copy of the effective_tld_names.dat (which i fetched from https://publicsuffix.org/list/effective_tld_names.dat) and aquire the phyton script from a not so deep link (http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/prepare_tlds.py?raw=1). First steps for Debian packaging (primary my convenience because i dont like to install things on systems with the cp command). Looking into milter_sendmail_macros[] and see where to find solid information what should be present there. All in all not very importent stuff.

At the moment i run some other milters and in the evening i'll flip in mopher for some time to see how it behaves and how far i got with understanding whats happening :-)

Greetings, chris.

[Chris Weber]

Sendmail makes direct use of DNSBL through:

FEATURE(dnsbl, `ix.dnsbl.manitu.net', `NiX Spam Blocking List - see http://www.dnsbl.manitu.net/')dnl
FEATURE(dnsbl, `cbl.abuseat.org', `Composite Blocking List - see http://cbl.abuseat.org/')dnl

that leads to rejecting the connection and handing over a message to the calling party why the connection was refused.

sm-mta[1518]: s83D6uSN001518: ruleset=check_rcpt, arg1=<xxxx...@xxxxx.xx>, relay=[181.65.240.78], reject=553 5.3.0 <xxxx...@xxxxx.xx>... NiX Spam Blocking List - see http://www.dnsbl.manitu.net/

i tryed to move that over to mopher:

envrcpt rbl_NiX == "127.0.0.2" reject xcode 553 message "NiX Spam Blocking List - see http://www.dnsbl.manitu.net/"

but that returns a syntax error. Whatever i put behind the 'reject' triggers that. Even the Example from the mopher-0.4.1 (github) mail.acl.md

eom spamd_score > 15 reject xcode 550 message "Not accepting spam"

does throw a syntax error. Is this actually working?

[Manuel Badzong]

This line should do the trick:

envrcpt rbl_NiX == "127.0.0.2" reject reply 553 xcode "5.7.1" message "NiX Spam Blocking List - see http://www.dnsbl.manitu.net/"

To use custom smtp replies you need to add the reply keyword. The xcode keyword denotes the enhanced mail system status code (see RFC 3463). In your case 5.7.1 should be the correct code.

[Manuel Badzong]

Here's a slightly modified version of my running mail.acl:

- SNIP -------------------

connect  list_local                                    reject reply 550 xcode "5.7.1" message milter_addrstr + " is blacklisted"
connect  counter_relay > 5                             continue
connect                                                tarpit 15s

envrcpt  counter_penpal > 3                            continue
envrcpt                                                greylist reply 451 xcode "4.7.1" message "greylisting in action"
envrcpt  spf == SPF_FAIL                               jump spam
envrcpt  spf == SPF_SOFTFAIL                           jump spam
envrcpt  list_sorbs                                    jump spam
envrcpt  list_spamcop                                  jump spam
envrcpt  list_spamhaus                                 jump spam

eom      counter_penpal > 3                            jump stamp
eom      spamd_score > 8                               reject reply 550 xcode "5.7.1" message "message considered spam"
eom      spamd_score > 3                               jump spam
eom                                                    jump stamp

stamp    tarpit_delayed                                change header "X-Tarpit" value "message tarpitted for " + tarpit_delayed + " seconds"
stamp    greylist_delayed                              change header "X-Greylist" value "message greylisted for " + greylist_delayed + " seconds"
stamp    counter_relay                                 change header "X-Counter-Relay" value counter_relay
stamp    counter_penpal                                change header "X-Counter-Penpal" value counter_penpal

stamp    spamd_score                                   change header "X-Spam-Score" value spamd_score
stamp    spamd_symbols                                 change header "X-Spam-Symbols" value spamd_symbols
stamp    spamd_spam                                    change header "X-Spam-Flag" value "YES"

spam                                                   greylist delay 30m attempts 4 reply 451 xcode "4.7.1" message "greylisting in action"
spam     milter_stagename == "eom"                     jump stamp

- /SNIP -------------------

[Manuel Badzong]

For the record. Here's the ungarbeled mail.acl.

[Chris Weber]

Again, thank you!

With the Information that a 'reply' is needed to customize xcode and message things work. Would be worth mention in the documentation :) There are several other things that i would find usefull, like the complete defaults for mopher.conf (well, found them in the sources). It is mentioned that there is a new format for sockets i.e. inet:44...@127.0.0.1 while the 'old' one is told earlyer as a default (inet:127.0.0.1:44554). Any preference on which one to use? (that dosn't apply to my setup, i use the unix sockets to keep things local). Not noted but expected, spamd also could be addressed through unix:/var/run/milter/mopher.socket.

A Question out of interest: while i very my spamassassin installation with spamass-milter i noticed that it uses spamc to hand the message over. That one has a maximum size restriction for sending over mails to spamd. Does this also apply to the spamd module ?

At which point the spamd modul is invoked? Is it called by using a keyword (spamd_score) or ist it called 'because it was found in the list of loadable modules'.

And a final suggestion, in this Google Group it is noted that the versions available on GitHub, in the line below it is stated that the latest Version is mopher-0.1 while the release on GitHub reached 0.4.1. Wouldnt it be better to remove that link to Version 0.1 ?

[Manuel Badzong]

> Would be worth mention in the documentation

I know documentation is still scarce. Petar is working on the man pages
right now and your input is very appreciated!

> ... inet:44...@127.0.0.1 while the 'old' one is told earlyer as a

> default (inet:127.0.0.1:44554). Any preference on which one to use?

Mopher just passes the socket string to libmilter. Because libmilter
changed the format, the mopher configuration format also changed. All
other sockets in mopher still use the traditional format. In the future
all sockets will use the same format as libmilter (yet to be implemented).

> Not noted but expected, spamd also could be addressed through
> unix:/var/run/milter/mopher.socket.

The reason mopher uses inet sockets per default is, that different
systems have different filesystem layouts, users and permissions. If we
use inet sockets, everything should work out of the box without special
tweaking for every distribution.

> A Question out of interest: while i very my spamassassin installation
> with spamass-milter i noticed that it uses spamc to hand the message
> over. That one has a maximum size restriction for sending over mails
> to spamd. Does this also apply to the spamd module ?

Actually I don't know. If you find the time, feel free to test

> At which point the spamd modul is invoked?

Mopher evaluates all symbols as lazy as possible. If a message doesn't
need a symbol for evaluation there's no lookup.

Another thing worth mentioning is the empty greylist rule. In my
mail.acl there's an empty greylist rule at the beginning of the envrcpt
stage. This means mopher tries to load an existing greylist tuple. If it
exists greylisting is continued. There's no need to pass the same
message through spamassasin at every retry.

> Wouldnt it be better to remove that link to Version 0.1 ?

Done.

[Chris Weber]

On Thursday, September 4, 2014 4:30:18 PM UTC+2, Manuel Badzong wrote:

> Would be worth mention in the documentation
I know documentation is still scarce. Petar is working on the man pages
right now and your input is very appreciated!

Oh, i think he might curse me for that, but i also had a table in Mind what could be expected in which stage. The reject -> reply xcode ... worked well, so i put that into connect stage and sendmail logged 'rejecting commands'. After a second of thinking this is obvious, in that stage nothing is returned to the connecting party. So the 'reply' only makes sense at a stage where actually something could be transmitted (envrcpt).

> A Question out of interest: while i very my spamassassin installation
> with spamass-milter i noticed that it uses spamc to hand the message
> over. That one has a maximum size restriction for sending over mails
> to spamd. Does this also apply to the spamd module ?
Actually I don't know. If you find the time, feel free to test

I think i didnt explained the process which is used by spamass-milter very well :-/ That milter passes the mail through the commandline client of spamassasin to the daemon of spamassasin. The size restriction is in the commandline client configuration, not in the milter and neither in the daemon. But from your answer i read that there is no size limit in the spamd.so module of mopher.

> At which point the spamd modul is invoked?
Mopher evaluates all symbols as lazy as possible. If a message doesn't
need a symbol for evaluation there's no lookup.

 
Ok, in this case a 'avoid pipe half a gigabyte mail through spamassasin' could be constructed in the mail.acl.
---cut---
.
eom milter_body_size > 512K jump stamp
eom spamd_score >= 6          greylist delay 30m attempts 3
.
.
stamp isset spamd_score  add header "X-Mopher-Score" value hdr_score
---cut---

actually this line(s) are ripped from a post that Petar made 12/10/12 here (crippeling and mistypes by me). I'll take this that the 'isset' dosent trigger the evaluation through the spamd module. The reason why i'm up to this is not something obscure, it is that Lawyers send mails through my systems. They tend to put a package of paper (company contracts) on multifunctional printer / scanner and send the resulting 'pdf' sized from 10-100 MB to at least 5-10 other lawyers. It is not uncommon to see 100-500 MB of such mails in the queue.

Another thing worth mentioning is the empty greylist rule. In my
mail.acl there's an empty greylist rule at the beginning of the envrcpt
stage. This means mopher tries to load an existing greylist tuple. If it
exists greylisting is continued. There's no need to pass the same
message through spamassasin at every retry.

Thanks for clarification of that one.

Greetings

[Petar Bogdanovic]

Hi and welcome!

On Tue, Sep 02, 2014 at 04:01:23AM -0700, Chris Weber wrote:
>
> mopherd[4586]: milter_connect: smfi_getsymval for "v" failed

milter.c says:

mta_version = smfi_getsymval(ctx, "v");

If Postfix and Sendmail handle {v} differently, I'd say this is a bug
that deserves an own ticket on the GitHub issue-tracker.

Manu: Is there a reason for {v} not being part of
milter_{postfix,sendmail}_macros?

Petar Bogdanovic

[Petar Bogdanovic]

On Thu, Sep 04, 2014 at 01:42:14AM -0700, Chris Weber wrote:
>
> A Question out of interest: while i very my spamassassin installation with
> spamass-milter i noticed that it uses spamc to hand the message over. That one
> has a maximum size restriction for sending over mails to spamd. Does this also
> apply to the spamd module ?

The spamc protocol requires a `Content-length' header which mopher
provides but doesn't somehow limit (info based on a quick review of
spamd.c:spamd_query).

> At which point the spamd modul is invoked?

If by invoked you mean loaded.. I think mopher loads all available
modules on startup. You can verify that with `mopherd -d 7'.

[Petar Bogdanovic]

On Thu, Sep 04, 2014 at 10:03:27AM -0700, Chris Weber wrote:
>
> On Thursday, September 4, 2014 4:30:18 PM UTC+2, Manuel Badzong wrote:
>
> > Would be worth mention in the documentation
> I know documentation is still scarce. Petar is working on the man pages
> right now and your input is very appreciated!
>
>
> Oh, i think he might curse me for that, but i also had a table in Mind what
> could be expected in which stage.

No, I won't---thanks for the suggestion! :)

Mopher is desperate for some quality documentation (especially
man-pages) and I'll continue with it as soon as these two very
busy weeks end.

> Ok, in this case a 'avoid pipe half a gigabyte mail through spamassasin' could
> be constructed in the mail.acl.

It may be wrong, but I whitelist everything above 2MB.

> I'll take this that the 'isset' dosent trigger the evaluation through
> the spamd module.

I think you're right.. otherwise spamd_score would always be set.

[Petar Bogdanovic]

On Tue, Sep 02, 2014 at 08:32:06AM -0700, Chris Weber wrote:
>
> Some cosmetic things i did: prefering a more actual copy of the
> effective_tld_names.dat (which i fetched from https://publicsuffix.org/list/effective_tld_names.dat)

I remember that link being a pointer to whatever is in Mozilla's
repository. Mozilla uses this list for various purposes, one of them
being:

___ black _____
xxxx://www.paypal.com.willyhacker.net
^^^^^^^^^^ grey ^^^^^^

So while an up-to-date list is important for Firefox, mopher-users will
probably survive older versions. That was one of the reasons to make it
part of the source instead of something that lives in /etc (or /var) and
can be dynamically updated and reloaded.

> and aquire the phyton script from a not so deep link
> (http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/prepare_tlds.py?raw=1).

They used to change that script so at one point it started to generate
different code. I'll try to replace that with something simple in awk.

[Manuel Badzong]

 I'll take this that the 'isset' dosent trigger the evaluation through the spamd module.

Exactly. Isset can be used to check if a symbol is already set for a message without triggering a lookup. 

[Manuel Badzong]

If Postfix and Sendmail handle {v} differently, I'd say this is a bug
that deserves an own ticket on the GitHub issue-tracker.

I would't say they handle it differently, Sendmail just needs the macro to be set. I suggest testing for v and if unset, set mta_version to "unknown".

[Manuel Badzong]

> So while an up-to-date list is important for Firefox, mopher-users will
> probably survive older versions. That was one of the reasons to make it
> part of the source instead of something that lives in /etc (or /var) and
> can be dynamically updated and reloaded.

Anyhow keeping it up to date is advantageous. Storing it plain-text
somewhere in the filesystem would be nice..

[Petar Bogdanovic]

On Thu, Sep 04, 2014 at 11:33:03PM -0700, Manuel Badzong wrote:
>
> I suggest testing for v and if unset, set mta_version to "unknown".

There are 3 MTAs that implement the Milter interface:

* Sendmail
* Postfix
* Postoffice (last release Jan 2011)

So I counter-suggest that mopher tests for {v} and, if unset, sets
mta_version to Sendmail. Because that's what you'll assume anyway
when assigning macro_table in milter_macro_lookup().

We also emulate the Sendmail received header.

[Chris Weber]

This is what i found in milter.c, is this realy intendet? :)
---cut---

if (strncmp(version, POSTFIX, POSTFIX_LEN) == 0) {

macro_table = milter_postfix_macro_table;

}

else if (strncmp(version, SENDMAIL, SENDMAIL_LEN) == 0) {

macro_table = milter_postfix_macro_table;

} ---cut---

[Chris Weber]

On Thursday, September 4, 2014 8:21:07 PM UTC+2, Petar Bogdanovic wrote:

On Thu, Sep 04, 2014 at 10:03:27AM -0700, Chris Weber wrote:
>
> On Thursday, September 4, 2014 4:30:18 PM UTC+2, Manuel Badzong wrote:
>
>     > Would be worth mention in the documentation
>     I know documentation is still scarce. Petar is working on the man pages
>     right now and your input is very appreciated!
>
>
> Oh, i think he might curse me for that, but i also had a table in Mind what
> could be expected in which stage.

No, I won't---thanks for the suggestion! :)

Mopher is desperate for some quality documentation (especially
man-pages) and I'll continue with it as soon as these two very
busy weeks end.

 
:-)

btw, that 'table of what to expect' would even helpfull for guys that write this thing. This is what Manuel posted in his latest mail.acl:

---cut---

connect  list_local                 reject reply 550 xcode "5.7.1" message milter_addrstr + " is blacklisted"

---cut---

at that stage (connect) he wont get out anything as a reply to the calling party because the connection is simply rejected ;-)

[Chris Weber]

Is it possible to access result of the {auth_authen} Macro ? According to milter.org it is available at envfrom stage.

To explain, at the moment i'm still have milter-sender running, that one has the ability to skip handling if the user is authenticated by whatever mechs sendmail has at hand.
I have to service dialin connections from customers and tests could be left out if they are already authenticated.


Greetings

[Manuel Badzong]

> This is what i found in milter.c, is this realy intendet? :)

> ...
You're right, this is a bad solution. I think there's a FIXME line above
the sendmail_macro_table to indicate this.

I guess the proper solution is to use only one macro table for all MTAs.
I'll add it to the issues on GitHub. Feel free to fix ;)

[Manuel Badzong]

> Is it possible to access result of the {/auth_authen/} Macro ?

> According to milter.org it is available at envfrom stage.

Yes it is. The symbol name is milter_auth_authen. Have a look at
milter.c (milter_sendmail_macros).

[Manuel Badzong]

There was a typo in the macro table. The symbol name was written
"milter_auth_athen". Fixed in HEAD.

I also removed the {v} lookup at the connect stage and unified the macro
tables.

[Chris Weber]

There was a typo in the macro table. The symbol name was written
"milter_auth_athen". Fixed in HEAD.

I also removed the {v} lookup at the connect stage and unified the macro
tables.

unification and {v} change made it to GitHub while the correction of the typo seems to got lost on the way :-)

[Manuel Badzong]

On Mon, Sep 08, 2014 at 02:42:13AM -0700, Chris Weber wrote:
> unification and {v} change made it to GitHub while the correction of the typo
> seems to got lost on the way :-)

Now it is. Thanks for the review!