Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

RSA.M - Moderation Techniques & Software

6 views
Skip to first unread message

Sven Golly

unread,
Sep 25, 2002, 2:43:10 AM9/25/02
to
I've done a little online research into moderation software and
what it entails. Feel free to correct me if I'm wrong technically.
There are lots of FAQs out there but most are overly detailed for
what this group needs to know.

BASIC STEPS

1. Moderated group is approved and moderators are formalized. (No
comment on which comes first, the chicken or the egg or both.)

2. USENET servers are globally configured for new group, to accept
(unapproved) posts from users and forward them to a single e-mail
address (POP3 box) for review without posting them to the group
itself.

3. E-mails are retrieved from the POP3 box via client software for
review by the moderator.

4. If a post is approved, it is reposted by the moderator back to the
group with the addition of an "Approved:" header that contains an
authorized signature or e-mail address. News servers recognize the
appropriate signature and allow the post to propogate normally.

5. If a post is disapproved, it is normally returned to the poster
with a brief reason for the rejection.

That's the process at it's most basic level.

REVIEW PROCESS

Reviewing posts is obviously the most time consuming part of
moderation.

1. At the most basic level, a moderator reads each individual message
for content and approves or disapproves the post.

2. Filtering software can be used to assign scores to posts based on
the presence of keywords and/or blacklists & whitelists. Posts
falling below various thresholds can be flagged for individual
attention of a moderator and/or killed outright.

Keyword filtering can take place on the Subject, most headers and
message content.

Blacklists and whitelists can include individual posters and/or
domains, subdomains and individual IP addresses. Blacklists are
considered a last resort and applied only when a poster or posters
from a domain or subdomain have abused the moderation process and/or
continually post off-charter material. Whitelists are pre-approved
posters who's messages do not require filtering. Whitelist posters
are normally only added after the moderators have sufficient
experience to trust a pre-approved message from them.

It is up to the moderation team whether to disclose or not disclose
the operation of any filters or black/whitelists.

3. A combination of filtering and hand moderation is normally used as
a new moderated group gets started. As the filters become more
sophisticated, hand moderation can be reduced to a minimum IF the
proper software is used.

APPROVAL / DISAPPROVAL PROCESS

1. Approving a post requires converting the incoming e-mailed post
into a format acceptable for posting via NNTP to a news server. This
normally requires some header stripping/mungeing and adding an
"Approved:" header with the appropriate authorization info.

In groups where Approval forgeries are rampant, moderators may have
to resort to PGP-encrypted approval signatures.

2. Disapproving a post simply requires that the post be returned to
the sender via e-mail with an optional brief explanation for its
rejection.

3. IN NO CASE SHOULD A MODERATOR EVER EDIT A POST FOR CONTENT. It
should either be rejected and returned for editing by the originator
or posted as-is.

SOFTWARE TOOLS

There are many dedicated tools available to assist in the process of
moderation. A good starting list is available at:

http://www.landfield.com/moderators/

I have identified a few packages that can do the job depending on
what resources are available to the moderation team. All are free for
non-commercial use.

STUMP

If a UNIX server with an always-on connection to the net is
available, then the STUMP Robomoderation package is ideal. It will
handle POP3 collection, filtering, review, reposting, rejection and
most other tasks associated with moderating a USENET group. See:

http://www.algebra.com/~ichudov/stump/

It supports team moderation and allows moderation via web browser.

DMOD

DMOD is a basic Windows SMTP, POP3 & NNTP client that greatly
simplifies the process of collecting posts via e-mail, reviewing,
reposting and rejecting them. It can do some basic header filtering
but is not capable of filtering on message content or scoring. DMOD
may be appropriate to get the moderation process off the ground
without needing an always-connected UNIX box.

http://netwinsite.com/dfree/dmod.htm

MODTOOL 2.0

Modtool is a general purpose UNIX moderation tool written in Perl for
(non-binaries) newsgroups and moderated mailing lists. Features
include: posting, rejection, form rejection letters, forged article
cancellation, automatic digest creation, support for cross-posting to
multiple moderated groups, PGP Moose support (requires PGP Moose
programs), auto-replying to submissions, and support for moderation
of multiple groups.

http://www.landfield.com/moderators/modtool-2.0.tar.Z

Unfortunately, I can't open the embedded tarball inside the zip file
so I can't really look at this one.

OTHER TOOLS

See the http://www.landfield.com for a list of other resources. There
are lots of UNIX tools available but very few for Windows and none
for the Mac (that I can find).

ANOTHER POSSIBILITY FOR WINDOWS

It may be possible using a combination of the Hamster personal news
server and DMOD to effect almost the same results as STUMP. Hamster
has excellent filtering capabilities and could be used as a front end
for grabbing e-mails and allowing DMOD to repost them.

I'll leave it up to the experts to figure all this out.

--
Sven Golly

Remove the '_' to reply
X-No-Archive unset for this post

The Real Bev

unread,
Sep 25, 2002, 10:56:24 PM9/25/02
to
Sven Golly wrote:
>
> Sven Golly <sven_...@myrealbox.com> wrote in
> news:Xns9293F14804639sv...@216.168.3.40:

>
> > I've done a little online research into moderation software and
> > what it entails.
>
> Gee, no comments? Must have been perfect then! <G> (OK, back to your
> regularly scheduled flames.)

Jeez, Sven, some of us have a life! Or something fairly similar, at any
rate. I've looked at the STUMP stoff before, and it looked really neat --
for somebody with a full time connection and linux, of course. Didn't
somebody volunteer exactly that? Since the moderators are likely to be
distributed all over the planet, being able to moderate over the web would
be a real advantage -- people could take shifts depending on location.

Nice summary, for a Swede.

--
Cheers,
Bev
================================================================
"Some people say that when it rains it means that God is crying,
probably because of something that you did." --Jack Handey

David E. Bath

unread,
Sep 26, 2002, 1:01:23 PM9/26/02
to bath
In article <Xns9293F14804639sv...@216.168.3.40>,
Sven Golly <sven_...@myrealbox.com> writes:
> SOFTWARE TOOLS

> ANOTHER POSSIBILITY FOR WINDOWS
>
> It may be possible using a combination of the Hamster personal news
> server and DMOD to effect almost the same results as STUMP. Hamster
> has excellent filtering capabilities and could be used as a front end
> for grabbing e-mails and allowing DMOD to repost them.
>
> I'll leave it up to the experts to figure all this out.

Another solution for Windows is to look into one of the free GNU to
Windows ports such as Cygwin. You get a complete Unix environment that
runs on Windows, including an Xserver too. I do all of my moderation
in this way. It could even be a STUMP host if it has a full time
connection.

--
-------------------------------------------------------------------
David Bath mailto:dave...@excite.com

Kathy Morgan

unread,
Sep 27, 2002, 11:46:52 AM9/27/02
to
Sven Golly <sven_...@myrealbox.com> wrote:

[Good FAQ]

> Keyword filtering can take place on the Subject, most headers and
> message content.

As I understand it, depending on the software used, filtering can take
place on all headers and content.

--
Kathy
visit news:news.groups.reviews to read reviews of other newsgroups
help for new users of newsgroups at <http://www.aptalaska.net/~kmorgan/>
Good Net Keeping Seal of Approval at <http://www.gnksa.org/>
OE-quotefix can fix OE: <http://jump.to/oe-quotefix>

Denis McKeon

unread,
Oct 7, 2002, 3:34:52 PM10/7/02
to
In <Xns9293F14804639sv...@216.168.3.40> Sven Golly wrote:
>I've done a little online research into moderation software and
>what it entails. Feel free to correct me if I'm wrong technically.

You're 85-90% correct, but it sounds like you're looking from the point
of view of how a particular group might work, so in response I've done a
line-by-line review, based on my awareness about moderated groups in
general - my comments are not meant to be specific to any existing or
proposed moderated group in particular.

>There are lots of FAQs out there but most are overly detailed for
>what this group needs to know.
>
>BASIC STEPS
>
>1. Moderated group is approved and moderators are formalized. (No
>comment on which comes first, the chicken or the egg or both.)

If a CFV passes then a control message can be sent to create the group,
or to change the status of an existing group. The control message may
be delayed until the moderators are ready to handle a flow of messages.

>2. USENET servers are globally configured for new group, to accept
>(unapproved) posts from users and forward them to a single e-mail
>address (POP3 box) for review without posting them to the group
>itself.

Most news servers do not maintain their own copy of a set of forwarding
addresses, but forward instead to one of several forwarding hubs, which
then send to a single mailbox, which is not required to support POP.

>3. E-mails are retrieved from the POP3 box via client software for
>review by the moderator.

Once messages reach the moderation mailbox, the moderator(s) could use
whatever software they choose - POP3, IMAP, procmail, fetchmail, etc.

>4. If a post is approved, it is reposted by the moderator back to the
>group with the addition of an "Approved:" header that contains an
>authorized signature or e-mail address. News servers recognize the
>appropriate signature and allow the post to propogate normally.

Generally, servers just recognize the header.

>5. If a post is disapproved, it is normally returned to the poster
>with a brief reason for the rejection.

Like many things, that choice depends on the internal policy for the
group, either from the group charter, or an internal policy document, or
at the discretion of the moderator(s). Many moderators are unlikely to
go to the trouble of trying to return off-topic spam and then having to
deal with bounces due to forged From addresses. OTOH, many will return
messages that are on-topic, but rejected for whatever reason.

>That's the process at it's most basic level.
>
>REVIEW PROCESS
>
>Reviewing posts is obviously the most time consuming part of
>moderation.

It may be, depending on the methods used.

>1. At the most basic level, a moderator reads each individual message
>for content and approves or disapproves the post.
>
>2. Filtering software can be used to assign scores to posts based on
>the presence of keywords and/or blacklists & whitelists. Posts
>falling below various thresholds can be flagged for individual
>attention of a moderator and/or killed outright.
>
>Keyword filtering can take place on the Subject, most headers and
>message content.
>
>Blacklists and whitelists can include individual posters and/or
>domains, subdomains and individual IP addresses. Blacklists are
>considered a last resort and applied only when a poster or posters
>from a domain or subdomain have abused the moderation process and/or
>continually post off-charter material. Whitelists are pre-approved
>posters who's messages do not require filtering. Whitelist posters
>are normally only added after the moderators have sufficient
>experience to trust a pre-approved message from them.

Many people prefer a "judge the post, not the poster" approach, which
generally means that each message should be reviewed on the basis of the
message content rather than rejected out-of-hand on the basis of the
source of the message. Still, making it clear to someone who
persistently and deliberately trolls or flames that all messages from
them will be hand reviewed will usually have the effect of reducing the
number of such messages to a manageable amount.

Generally, a choice to use white or black lists should be more about
time, effort, and convenience than about individuals and personalities.
A moderated newsgroup where 95% of the posters are on a whitelist is
probably a newsgroup with fast message turnaround and low moderator
effort. However, that choice does represent a trade-off and some risk
of new voices and new ideas not being heard if the established community
of posters and moderators choose to exclude them.

>It is up to the moderation team whether to disclose or not disclose
>the operation of any filters or black/whitelists.
>
>3. A combination of filtering and hand moderation is normally used as
>a new moderated group gets started. As the filters become more
>sophisticated, hand moderation can be reduced to a minimum IF the
>proper software is used.
>
>APPROVAL / DISAPPROVAL PROCESS
>
>1. Approving a post requires converting the incoming e-mailed post
>into a format acceptable for posting via NNTP to a news server. This
>normally requires some header stripping/mungeing and adding an
>"Approved:" header with the appropriate authorization info.
>
>In groups where Approval forgeries are rampant, moderators may have
>to resort to PGP-encrypted approval signatures.
>
>2. Disapproving a post simply requires that the post be returned to
>the sender via e-mail with an optional brief explanation for its
>rejection.

Again, this is common, but not required.

>3. IN NO CASE SHOULD A MODERATOR EVER EDIT A POST FOR CONTENT. It
>should either be rejected and returned for editing by the originator
>or posted as-is.

A "no editing" policy is common, but by no means universal nor absolute.

Some group moderators will remove redundant multipart/alternative (HTML)
message parts, remove or replace some X- headers, replace or modify some
headers, add footers, headers, disclaimers, and in some cases append
editorial notes or comments. My opinion is that any automatic changes
should be documented in a policy document or FAQ, and that any editorial
notes should be obvious [perhaps set off in square brackets].

>SOFTWARE TOOLS

Good list deleted, no more comments.


--
Denis McKeon

0 new messages