sa_userconf_maps and sa_username_maps syntax question

[Matt Goodman]

Hello all,

 

I am interested in two new features that got added to amavisd-new in version 2.7.0

 

Taken from the Release Notes (2.7.0)

 

- per-recipient (or per- policy bank) SpamAssassin configuration files or

  SQL configuration sets are supported (@sa_userconf_maps), and per-recipient

  SQL Bayes database usernames (@sa_username_maps);

 

However, I don’t see are any examples of how this configuration is enabled, or more specifically what syntax the configuration options are looking for. I gather that they are “maps” but I really don’t understand how maps are configured. Presently I have the @lookup_sql_dsn configured to check the database which Postfixadmin is adding users to. I’d like to have my users store per-user bayes information and per-user spamassasin information inside this database also.

 

Did these options merely get added and will become documented/utilized at a later point? Or is all the information already there, and I’m just not seeing it?

 

Update: I did find these two config options in the amavisd.conf-default.bz2 doc file

 

# @sa_userconf_maps = ();

# @sa_username_maps = ();

 

Alas, they are empty and I am not sure what to enter there to “tell” amavisd-new to look up in the SQL database.

 

I appreciate any replies.

 

Matt

[Matt Goodman]

Thank you very much for your reply. I understand a little bit better now. However, in example one below you listed a single domain name using the method ‘sql’. What if I have dozens of domains, with them being added/removed somewhat regularly by Postfix Admin (I can perceive adding at least 1-2 domains per month) to this system. Isn’t there a way to tell amavisd-new that all domains in the lookup_sql_dsn? The table is actually called ‘domain’ inside the ‘postfixadmin’ db.

 

For @sa_username_maps – I do not understand the syntax of that at all. Could you explain it in just a little more detail so I actually understand what’s being done, as opposed to just copying your example. Many thanks.

 

Thank you for the link to the patch, that will be helpful. And yes I have followed the SpamAssassin SQL README and have my db schema as-per those documents. A table called ‘userpref’ with one line per SA preference. I have even tested this using spamc and validated that the SQL configuration is good.

 

I am just a little lost as to the @sa_username_maps syntax (looks like regex to me), and I will read through the patch to try and determine what amavisd-new is looking for. In the meantime, if someone can help me understand how one might use all the domains in the postfixadmin table instead of one domain per line of @sa_userconf_maps, that would be helpful.

 

Matt

 

Sent: Thursday, September 01, 2011 11:21 AM
To: Matt Goodman
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

 

Hello Matt,

For sql user_prefs ( per user config )::

I had to add

1. @sa_userconf_maps = (
       {
          '.domain-t1.ro' => 'sql:',
       }
     );

@sa_username_maps = new_RE (
  [ qr'^([^@]+@.*)'i => '${1}' ]
);

2. this patch

http://lists.amavis.org/pipermail/amavis-users/attachments/20110812/756a66cf/attachment.obj 

  You have to read about SPAMMASSASSIN SQL userprf to know howto setup database and userpref table.

 

 

 

[Renato Botelho]

Hello Matt

On Fri, Sep 2, 2011 at 1:51 AM, Matt Goodman <ma...@dgisoda.com> wrote:
> Thank you very much for your reply. I understand a little bit better now.
> However, in example one below you listed a single domain name using the
> method ‘sql’. What if I have dozens of domains, with them being
> added/removed somewhat regularly by Postfix Admin (I can perceive adding at
> least 1-2 domains per month) to this system. Isn’t there a way to tell
> amavisd-new that all domains in the lookup_sql_dsn? The table is actually
> called ‘domain’ inside the ‘postfixadmin’ db.

If all domains are using sql, you can have:

@sa_userconf_maps = (
   {
      '.*' => 'sql:',
   }
);

> For @sa_username_maps – I do not understand the syntax of that at all. Could
> you explain it in just a little more detail so I actually understand what’s
> being done, as opposed to just copying your example. Many thanks.

In this case, since it is a single domain, i have personal userprefs
set just for uset, without domain, i mean, if email is us...@foo.com
prefs on SQL is set just for user, so, this map says exactly it, for
us...@foo.com, look just for user at SQL database.

@sa_username_maps = new_RE (
  [ qr'^([^@]+@.*)'i => '${1}' ]
);

> Thank you for the link to the patch, that will be helpful. And yes I have

> followed the SpamAssassin SQL README and have my db schema as-per those
> documents. A table called ‘userpref’ with one line per SA preference. I have
> even tested this using spamc and validated that the SQL configuration is
> good.

What that patch does is making amavis respect personal required_score
setting instead of use a global value for everyone, and the version sent
to this thread has a small issue if you use it with spamassassin < 3.0.

> I am just a little lost as to the @sa_username_maps syntax (looks like regex
> to me), and I will read through the patch to try and determine what
> amavisd-new is looking for. In the meantime, if someone can help me
> understand how one might use all the domains in the postfixadmin table
> instead of one domain per line of @sa_userconf_maps, that would be helpful.

--
Renato Botelho

[Matt Goodman]

OK. So is it correct in saying the following:

@sa_userconf_maps matches SA user preferences by domain
@sa_username_maps matches SA Bayes database by email address

As I understand it - in your regex example, amavisd-new will simply match
the recipient email address to the corresponding record in the bayes
database, correct? If this is not correct, any insight would be appreciated.
Again, I am not looking for a shortcut - only to understand the way the
subroutines were intended for use.

Ultimately I am trying to have a flexible virtual user email system with
per-user spamassassin preferences and per-user Bayesian filters, allowing
people to "train" their own Bayes tokens by "learning" messages which they
mark as Spam and "learning" those which are ham, while storing all that
information into a SQL database. The assumption here is that the Bayesian
classification will be more accurate to the individual user, rather than a
one-size-fits all for Bayes seen/tokens.

However, amavisd-new is my chosen mail filter system, simply because of how
flexible it is, and so far the help I have received here indicates that I
have made the right choice. I will share any findings that I learn back to
the community. Thank you again.

Matt

-----Original Message-----
From: Renato Botelho [mailto:rbg...@gmail.com]
Sent: Friday, September 02, 2011 12:07 AM
To: Matt Goodman
Cc: amavis...@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

[Matt Goodman]

Thank you for the insight. I examined the patch file that was presented and
I have to go a different direction. Since required_score is somewhat handled
already via the policy system (although users cannot specify their own
'score' they can indeed change policies, which in turn affects score), I
think I'm going to focus on per-user Bayes.

What part of amavisd-new goes and checks per-user bayes? I see nothing in
earlier examples nor in the patch which address having amavisd-new handle
per-user Bayes classification. Or is that solely done through SA's
configuration? If no changes are required in amavisd-new to get per-user
Bayes working, then I apologize. I can get SA's /etc/spamassassin/local.cf
to use the per-user Bayes. I'm just new to amavis and assume that amavis has
to handle checking against the per-user Bayes database. Is this correct?

So I will abandon sa_userconf_maps and sa_username_maps and go straight to
per-user Bayes. That should be enough, when combined with the flexibility of
the amavisd-new policies via SQL to accomplish what I need. I just have to
figure out whether anything is required on the amavis side of things.

Thanks,

[Mark Martinec]

Matt,

> I examined the patch file that was presented and
> I have to go a different direction. Since required_score is somewhat
> handled already via the policy system (although users cannot specify their
> own 'score' they can indeed change policies, which in turn affects score),
> I think I'm going to focus on per-user Bayes.

Makes sense, per-user Bayes is much more efficient than per-user
SpamAssassin preferences/config.

> What part of amavisd-new goes and checks per-user bayes? I see nothing in
> earlier examples nor in the patch which address having amavisd-new handle
> per-user Bayes classification.

There is a 80+ line description of @sa_username_maps and @sa_userconf_maps
with examples in the 2.7.0 release notes.

Search for sections starting with:
- per-recipient (or per- policy bank, or global) SpamAssassin configuration
and
- per-recipient (or per- policy bank) SpamAssassin SQL database usernames

> Or is that solely done through SA's configuration?

Before a call to SpamAssassin, amavisd tells SpamAssassin which username
to use. The rest (SQL lookup etc.) is done by SpamAssassin.

> If no changes are required in amavisd-new to get per-user
> Bayes working, then I apologize.

No changes should be necessary, apart from configuring @sa_username_maps
in amavisd.conf.

> I can get SA's /etc/spamassassin/local.cf
> to use the per-user Bayes. I'm just new to amavis and assume that amavis
> has to handle checking against the per-user Bayes database. Is this
> correct?

When per-recipient bayes is needed, amavisd calls SpamAssassin possibly
multiple times for multiple recipients of a message, once for each unique
username. Before the call the username is passed to SpamAssassin,
the rest is SpamAssassin's task.

Mark

[Matt Goodman]

Ah thanks, I am reviewing the release-notes now.

However, what I am reading is that the example listed for @sa_username_maps
statically maps usernames which is a smaller scale than I am trying to
implement. With a larger scale system, how do I tell @sa_username_maps to
pass the email address supplied from the RCPT TO as the SA username?

Or is that what the earlier reference to the following will do (provided by
Florin Portase on 09/01/2011):

@sa_username_maps = new_RE (
[ qr'^([^@]+@.*)'i => '${1}' ]
);

And finally, will I need to have @sa_userconf_maps set to "sql:" if I only
wish to do per-user bayes? Would this potentially cause any other problems
if we enable this setting but do not use any per-user preferences?

For reference:

All my users will have their own custom bayes without exception. We do not
wish to use per-domain or per-policy-bank bayes, per-user only. My SA/bayes
SQL configuration is sound and currently working - however, only one bayes
database is in the table, and it is owned by user 'amavis'.

-----Original Message-----
From: amavis-users-bounces+matt=dgiso...@amavis.org
[mailto:amavis-users-bounces+matt=dgiso...@amavis.org] On Behalf Of Mark
Martinec
Sent: Thursday, September 08, 2011 7:50 AM
To: amavis...@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question

[Renato Botelho]

On Thu, Sep 8, 2011 at 2:50 PM, Mark Martinec
<Mark.Marti...@ijs.si> wrote:
> Matt,
>
>> I examined the patch file that was presented and
>> I have to go a different direction. Since required_score is somewhat
>> handled already via the policy system (although users cannot specify their
>> own 'score' they can indeed change policies, which in turn affects score),
>> I think I'm going to focus on per-user Bayes.
>
> Makes sense, per-user Bayes is much more efficient than per-user
> SpamAssassin preferences/config.

Hello Mark,

I understand that it's more efficient, but, in some scenarios you *need* to
need user the powers to select their own required_score. That was the reason
i wrote that patch.

Do you see any real problem on the changed i made? I mean, do you think
it could break something?

And since amavisd-new is a one-file program, and this is a really big file, do
you consider applying this patch if it don't break anything? I'm asking because
maintain a custom patch for a so big file is a hard task sometimes.

Regards
--
Renato Botelho

[Mark Martinec]

Renato,

> I understand that it's more efficient, but, in some scenarios you *need* to
> need user the powers to select their own required_score. That was the
> reason i wrote that patch.

Makes sense, although I'd probably want to reverse the logic:
if a lookup on @spam_tag2_level_maps returns undef, then fall back
to a default 'required_hits' as provided by a SpamAssassin's config.

This way it will remain backwards compatible with existing configurations.

> Do you see any real problem on the changed i made? I mean, do you think
> it could break something?

Probably not.

> And since amavisd-new is a one-file program, and this is a really big file,
> do you consider applying this patch if it don't break anything? I'm asking
> because maintain a custom patch for a so big file is a hard task
> sometimes.

Sure, after reversing the logic.

Mark

[Matt Goodman]

For the @sa_username_maps I have the following provided by Florin:

@sa_username_maps = new_RE (
[ qr'^([^@]+@.*)'i => '${1}' ]
);

As he pointed out, he uses only a single domain on his mail server and this
expression maps the 'user' portion us...@domain.com and sends that as the SA
username. How would I modify this so that I pass 'us...@domain.com' as the SA
user? I've got the SQL portion up and running, I just don't have the
username maps set up properly. I've been trying for a few days to figure out
how to do this but I am unsure :(

Any insight would be appreciated.

Matt

-----Original Message-----
From: amavis-users-bounces+matt=dgiso...@amavis.org
[mailto:amavis-users-bounces+matt=dgiso...@amavis.org] On Behalf Of Mark
Martinec
Sent: Friday, September 09, 2011 2:36 PM
To: amavis...@amavis.org
Subject: Re: sa_userconf_maps and sa_username_maps syntax question