Usenet Hierarchy Administration FAQ

[Russ Allbery]

Last-modified: 2018-07-16
Posted-by: postfaq 1.17 (Perl 5.28.1)
Archive-name: usenet/hierarchy-admin
URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
Posting-frequency: monthly

This FAQ attempts to provide help to Usenet hierarchy administrators, the
people who try to maintain the canonical lists of newsgroups in managed
hierarchies. It is aimed at the hierarchy administrators rather than at
news admins and tries to summarize the issues to consider in making it
easy for news admins to carry the hierarchy.

If you're reading this on Usenet, this FAQ is formatted as a minimal
digest, so if your news or mail reader has digest handling capabilities
you can use them to navigate between sections. In rn variants, you can
use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
each section into a separate article.

Please send any comments, suggestions, or updates to ea...@eyrie.org.
Bear in mind when sending me e-mail that I receive upwards of 800 mail
messages a day and sometimes have a large backlog of personal e-mail.

This FAQ is posted monthly to news.admin.hierarchies, and is available on
the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

------------------------------

Subject: Contents

1. Introduction and Terms
2. Basic Hierarchy Administration
3. PGP-Signing Control Messages
4. Maintaining Moderated Groups
5. About the ftp.isc.org Newsgroup Lists
6. Other Resources

------------------------------

Subject: 1. Introduction and Terms

This FAQ assumes a basic familiarity with Usenet (there are other
documents that explain the fundamentals better), but there are a few
additional concepts that are specifically important for Usenet hierarchy
administrators.

A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
that share a common naming prefix, such as all groups starting with
"comp." or all groups starting with "de.". The names of Usenet newsgroups
define a hierarchy of names, with "." used as the separator between the
levels of the hierarchy, like host names. Unlike host names, the most
significant part of the name is given first. The first component of the
name is special and more significant than the rest of the name, since it
defines the top-level Usenet hierarchy to which that group belongs.

Generally, every top-level hierarchy is completely independent of the
others (although there are a few exceptions where multiple hierarchies
share the same management procedures). How the list of newsgroups in that
hierarchy should be maintained varies very widely between hierarchies,
from the complete anarchy of alt.* to the highly formal system used by
comp.*, or simply by fiat of the organization running the hierarchy as
with microsoft.*. Which maintenance methods you should use for your
hierarchy is out of scope for this FAQ; this document is about how to
publish the results of those methods, assuming that you want to have a
single canonical list of newsgroups that everyone carrying that hierarchy
can agree on. If you don't want that (if your hierarchy is like alt.*,
for example), most of this document will not apply.

Usenet newsgroups are created and removed via specially formatted messages
called control messages that tell news servers to do something. A
hierarchy should have a hierarchy administrator who is responsible for
following whatever procedures were agreed on for changing the list of
groups in that hierarchy and then publishing the results using control
messages. The structure of control messages is explained in the Usenet
news standards and in many FAQs and web pages, so it won't be explained
here. (Some sites don't use control messages for various reasons, and
it's therefore best to also publish the results via other methods, as will
be explained.)

Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
control messages, there is also a control message called a checkgroups
which provides a complete list of newsgroups in a hierarchy complete with
short descriptions. A "checkgroups" is therefore also used to mean a
complete list of newsgroups in a hierarchy. The checkgroups format is a
list of groups, one per line, with the name of the group, a tab, and a
short description. If the newsgroup is moderated, the description must
end in the literal text " (Moderated)". (Sadly, you can't translate the
word to another language. It's an ugly wart on the protocol.)

Originally, control messages were authenticated only by the (easily
forged) address of the sender of the message, which worked when Usenet was
small but broke down badly as it got larger. As a result, most
hierarchies now sign their control messages using PGP, an open standard
for public key cryptography, allowing receiving sites to verify that the
control message was issued by the person it claims to be from.

------------------------------

Subject: 2. Basic Hierarchy Administration

A hierarchy administrator has three separate audiences who should be kept
in mind when publishing information about a Usenet hierarchy: users who
may be interested in reading or posting to the hierarchy, news
administrators who are not currently carrying the hierarchy on their
servers and want to, and news administrators who are already carrying the
hierarchy and want to keep current with changes to it.

The most important audience is probably the users, but that's also the
audience that's the hardest to make general statements about, since how
you communicate with potential users varies quite a bit by hierarchy. The
users will primarily be interested in a description of what the hierarchy
is for, a list of groups in the hierarchy, any hierarchy-wide policies,
the charters of the newsgroups, and the newsgroup creation procedure.
They'll generally speak the same language as the hierarchy (since if they
don't, they probably won't be interested in reading it).

News administrators who aren't currently carrying the hierarchy likely
won't be as interested in detail like particuliar policies or group
charters, but will also be interested in the overall purpose for the
hierarchy and any policies that specifically affect sites carrying the
groups. They will specifically need the list of newsgroups in the
hierarchy in checkgroups format, however, and preferrably as a plain-text
file that they can download and feed into their news software. They'll
also need instructions for processing control messages to pick up changes
to the hierarchy, generally as an INN control.ctl fragment. Finally, news
administrators may not speak the language of your hierarchy (they may be
running a news server for a large international ISP, for example), so you
may want to provide instructions specifically for news administrators in
English if the language of your hierarchy isn't English.

News administrators who are already carrying the hierarchy are mostly
interested in being notified of changes to it (usually via control
messages). They also want ways of checking their group list against the
current one so that they can get back into sync if they've missed some
updates.

So, given that, here are a few specific recommendations:

* Have a web site. Try to make sure that the URL for your hierarchy web
site is stable and doesn't change, since the URL makes it into various
FAQs and configuration files that live for years. Put all of the
hierarchy information on the web site, and make sure that the web site
stays up to date. <http://www.news-admin.org/> may be available as a
hosting site for your web site.

* Every time you create a group, remove a group, rename a group (which
can be done with a creation and removal), or change the moderation
status of a group, send the appropriate control message. By looking
for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
you can check that the control message has propagated. Control
messages should have, in the Newsgroups header, the group being created
or removed (even for newgroup messages; they'll propagate correctly
even though the group hasn't been created yet because newgroup messages
have special propagation rules). The exception is checkgroups
messages, which should normally be posted to the administrative group
of your hierarchy and possibly crossposted to news.admin.hierarchies.
Don't put a Distribution header in your control messages unless you
really intend to limit the availability of the groups to sites
configured to accept that distribution (which mostly doesn't work
anyway).

* Send a checkgroups control message periodically. If you do this, you
don't really need to send duplicate control messages for changes; if
someone misses a change, they'll catch it with the next checkgroups
message. In the checkgroups control message (and ideally in your other
control messages as well), include an X-* header pointing to your
hierarchy web site so that people can get more information about the
hierarchy. Do not put a Supersedes header on your checkgroups
messages; some sites will filter out any message that contains both a
Control header and a Supersedes header.

* Put the *current* checkgroups on your web site as a plain-text file
(ending the file name with .txt will give this hint to most web
servers) so that it can be easily downloaded by news administrators.
Make sure that this file is kept current. Having a separate pretty
HTMLified list of groups is often useful for users, but please be sure
to provide the plain text checkgroups message as well since the HTML
version is nearly useless for news administrators trying to add your
hierarchy.

* Write a guide for news administrators on how to carry the hierarchy,
including an overall description. Make sure that it clearly indicates
where to find the checkgroups for the hierarchy. Put this guide on
your web site, and it doesn't hurt to put a URL for it in the headers
of your control messages as well. If your hierarchy uses a language
other than English, write this guide both in that language and in
English, since English seems to be the de facto international language
for news administration right now. (At the least, try to use the
literal word "checkgroups" on the web site on the top page for a link
to the current checkgroups; that's the main thing that people will be
looking for, and that way news admins can find it even if they can't
read the rest of the site.)

* Write a control.ctl entry for your hierarchy and publish it on your web
site in the section for news admins and in the guide. For a hierarchy
that doesn't PGP-sign control messages, this will look like:

## EXAMPLE (Example Hierarchy, Olympus Mons)
# Contact: ne...@example.org
# URL: http://www.usenet.example.org/
checkgroups:ne...@example.com:example.*:doit
newgroup:ne...@example.com:example.*:doit
rmgroup:ne...@example.com:example.*:doit

The first line contains your hierarchy name in all caps and then in
parens a description of your hierarchy, including the applicable
location if the hierarchy is regional. Then comes information about
who to contact about the hierarchy and the URL for your web site.
Finally, the uncommented lines specify the type of control message, the
sender (which should match the From and Sender headers of your control
message), the pattern matching the groups in your hierarchy, and the
word "doit" saying to act on those control messages.

News administrators can then just cut and paste this entry into their
news server configuration files and start honoring your control
messages. If you're using PGP, see the section on that below.

* Use PGP to sign all your control messages. This isn't necessary for
small hierarchies, but if there end up being any fights over your
hierarchy or it draws hostile attention for some reason, people can
easily cause a lot of confusion if your control messages aren't signed.
There's more information later about this.

* Make sure the information on your web site stays up to date! In
particular, be *sure* that the checkgroups on the web site is accurate,
since that's the list most people will start with. It's also sometimes
more convenient for someone to process a checkgroups from a web site
than process a control message (one can use tools like actsync to do
that, for example), so if it's not accurate, a news administrator may
undo all your control messages by getting information from the web site
instead.

* Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
and make sure the information is accurate. The information here is
based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
check on recent changes to your hierarchy and ensure that nothing is
out of date.

Some hierarchies also have a publically available news server that carries
that hierarchy (this is particularly common for hierarchies intended for
support of the products of a particular company). If you do, make sure to
point both users and news administrators at it. Users can use that server
if their local server doesn't carry your hierarchy, and news
administrators can use that server to get an up-to-date list of newsgroups
in the hierarchy using tools like actsync. If you do have such a server,
add the line:

# Syncable server: news.example.org

pointing to that server to your control.ctl entry.

Be careful about running open servers like this, as they have frequently
been abused to damage the rest of Usenet. At the least, it's best to not
allow users to crosspost to groups not carried by the server or add
Supersedes or Control headers, and ideally you should have some sort of
spam filtering or posting rate limits in place.

------------------------------

Subject: 3. PGP-Signing Control Messages

Most hierarchies now PGP-sign all control messages. The PGP signature is
visible in the X-PGP-Sig header of the control message. Background
information and (now somewhat outdated) instructions are at:

<ftp://ftp.isc.org/pub/pgpcontrol/README.html>

and the exact format of the signature is at:

<ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

You don't need to understand the information in the last link, though, at
least if you're using Unix and can use the existing tools for generating
signed control messages.

In order to sign control messages, you'll need three things: A PGP key
pair that you'll use for signing and verifying the messages, a PGP
implementation you can use, and some software to generate the right type
of signatures for Usenet control messages.

If you're not already familiar with public key cryptography, here's a
brief primer: A PGP key pair consists of two keys, a public one and a
private one. You sign messages with the private key, which you have to
keep secret and protect since anyone with possession of it can pretend to
be the hierarchy administrator. You give out the public key to anyone and
everyone, and anyone with the public key can check a signature and verify
that it was signed by the private key (without being able to sign messages
themselves).

Most Usenet news sites that honor control messages are set up to verify
messages signed with an algorithm called RSA, which was the algorithm used
by the original PGP implementation. Unfortunately, this is now fairly
obsolete. Current PGP implementations use a newer, more secure algorithm
for generating signatures (although the additional security is probably
overkill for Usenet control messages, at least for right now). While this
doesn't pose a problem for signing messages (current PGP implementations
can still use old RSA keys to sign things), it does cause problems if
you're starting fresh, since the keys generated by current implementations
will not work with old versions of PGP.

What all this means is that you have a few hard choices when it comes to
choosing a PGP implementation and generating your initial key pair. You
can use GnuPG <http://www.gnupg.org/>, which is probably the best
available PGP implementation, and not bother with a RSA key at all. This
will mean, unfortunately, that only sites that are also using GnuPG or
another current PGP implementation will be able to verify your control
messages. Or you can go to <http://www.pgpi.org/>, download an older
version of PGP (something of the 2.6 vintage), and use it to sign your
control messages, which will work with all versions of PGP but may be more
of a pain. (You can also use an old version of PGP only to generate the
initial key, and then import it into GnuPG and use GnuPG to sign control
messages, but this is complex and not recommended for people who have
never touched PGP before. There are some instructions on the GnuPG web
site, though.)

Whatever choice you make, follow the documentation of your PGP
implementation to generate a key pair. Pay careful attention to the user
ID that you put on your key. In order to work with Usenet, that user ID
must not contain any spaces. The two most common key IDs used for signing
Usenet control messages are the name of the administrative group in the
hierarchy (like example.config) or the sender of the control message (like
ne...@example.org). The latter is better practice and is recommended for
new hierarchies, although make sure that e-mail address is stable and is
one that you will be able to use for decades to come.

If you're using GnuPG, in order to not get any spaces into the user ID,
you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
ID as the name, and then press Enter when asked for an e-mail address or
comment. The recommended user ID is the e-mail address of the sender, but
it has to be entered as the name or GnuPG will not generate the right user
ID. A later version of pgpverify will hopefully make this unnecessary,
but older versions will be around for quite some time.

Resist the temptation to put any additional user IDs on your key. Your
key should only have one user ID, the one that will be used in control.ctl
entries. If it has any additional user IDs, this can confuse the
pgpverify process with some PGP implementations and cause your control
messages to be ignored.

If you're using a modern PGP implementation, it will automatically sign
the public key (this is called a self-signature). If you're using an
older PGP implementation, make sure that you do this, following the
instructions for your software. Keys that aren't self-signed can be
tampered with in various ways, and modern PGP implementations will refuse
to import or honor them.

Now, you're ready to create and sign a control message. I'm aware of
several major implementations of the glue software to do the signing:

signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
This is the original implementation by David Lawrence of the signed
control message protocol. The signcontrol script is a Perl script
that works with pgp. It will require some editing to set information
about your hierarchy. There is also a shell implementation in the
same directory.

signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
A different implementation in Python that may be somewhat easier to
use. It similarly will require some editing to add information for
your particular hierarchy and requires that GnuPG be available.

News::Article <http://search.cpan.org/~agierth/>
A Perl module, and therefore mostly useful if you're writing your own
software for your hierarchy in Perl. It makes fewer checks than
signcontrol and in general does less (just the signing) which can be
more convenient. It requires the PGP::Sign module from CPAN; see your
Perl documentation for how to install it.

cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
to automatically generate control messages. It can be used to manage
multiple hierarchies.

Pointers to additional implementations, particularly instructions for
Windows users, are very welcome.

Once you have a signed control message, verify it to make sure that it
verifies properly. You can do that with the pgpverify program that comes
with INN or is available in the same directory as signcontrol above.

If you got all of that working, you should put the PGP public key into
your guide for news administrators. To see examples of PGP public keys
for various hierarchies, see:

<ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

You should also make the public key available for download on your web
site, preferrably as a plain text file (ending the file name in .txt will
give this hint to most web servers) that can be easily downloaded. It's
best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
Also change your control.ctl entry that you have on your web site to look
like:

## EXAMPLE (Example Hierarchy, Olympus Mons)
# Contact: ne...@example.org
# URL: http://www.usenet.example.org/
# Key URL: http://www.usenet.example.org/pgpkey.txt
# Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
# *PGP* See comment at top of file.
checkgroups:ne...@example.com:example.*:verif...@example.org
newgroup:ne...@example.com:example.*:verif...@example.org
rmgroup:ne...@example.com:example.*:verif...@example.org

The key URL field is the URL of the plain-text PGP public key available
from your web site. The fingerprint is the output of gpg --fingerprint or
pgp -kvc (depending on what version of PGP that you're using) and is used
as a check to be sure that the key downloaded is the right one. The *PGP*
comment makes sense in the context of the standard control.ctl file, which
has information about PGP-signed control messages at the top. And note
that the "doit" string to act on the control messages has been replaced by
"verify-" followed by the user ID of your PGP key (whatever that may be).

Now, just use your new signing procedure whenever you send a control
message for your hierarchy. Oh, and if you're changing to signing control
messages from not signing them, be sure to announce that in
news.admin.hierarchies.

------------------------------

Subject: 4. Maintaining Moderated Groups

If there are moderated newsgroups in your hierarchy, this involves a bit
of additional hassle over unmoderated groups. Rather than just checking
whether the newsgroup is there are not, news administrators also have to
make sure that the moderation status is set correctly so that users can
post, and news servers need to know to which e-mail address to send posts
so that they'll reach the moderator.

When managing moderated groups, be sure to always include the correct
moderation flag in your newgroup control messages, and make sure that your
checkgroups lines for moderated groups end in " (Moderated)". (Yes,
literally. It can't be translated; news software actually looks for
this.) It's sometimes useful to send a few duplicate control messages if
you ever convert an unmoderated group to a moderated group or vice versa,
since it's harder to get this change to take than a simple newsgroup
creation. Expect to have a few frustrated users who's news providers just
cannot get this right. You may want to put together a form message to
send to those providers explaining who you are and that they have groups
in your hierarchy configured incorrectly.

Finally, you have to deal with getting the posts to the moderator. There
are two main ways of handling this, which can be combined. One is to keep
the central moderators.isc.org relay systems up to date, and the other is
to set up your own relay system that accepts mail addressed to the
newsgroup name with all periods replaced by hyphens @ some system you
control and then forwards that mail along to the real moderator.

Nearly all news servers that try to handle moderated groups default to
forwarding any message posted to a moderated group to the name of the
group with periods replaced by hyphens @moderators.isc.org. The relay
systems behind that DNS record then forward those messages along to the
real moderator. To have one of those addresses created or to change where
it forwards to, mail moderator...@isc.org. These changes have to be
made manually and there's often a backlog; to save time, please identify
yourself as the hierarchy administrator for the hierarchy and preferrably
send the mail from your hierarchy contact address (and you may also want
to PGP-sign the message if you can do that easily). Even still, be
prepared for this to sometimes take a while.

It's possible to configure news servers to forward mail somewhere other
than to moderators.isc.org, and easiest to do that as wildcard entries for
hierarchies that forward all posts to moderated groups in that hierarchy
to a particular host instead of moderators.isc.org (again with the period
to hyphen change). Some hierarchies do this instead so that they don't
have to wait for moderators.isc.org changes, but it makes things harder
for news administrators who have to keep track of those special cases. If
you do do this, you have to tell all news administrators to add a line
like:

example.*:%s...@example.org

to their moderators file, where the first part is the pattern that matches
groups in your hierarchy and the second part is the e-mail address to send
postings to (%s will be replaced by the group name with periods changed to
hyphens). If that forwarding host ever goes away, or if you ever have to
change this, it will be a major hassle and it will take a very long time
to catch all the stragglers.

What many hierarchy administrators do is combine these two approaches.
They set up their own forwarding system that they have direct control
over, so that they can change the moderators for an existing group without
having to contact moderator...@isc.org, and they ask the
moderators.isc.org sites to just forward to the appropriate address at
that host. Then, the only things they have to let moderators-request know
about are new moderated groups or removal of moderated groups. This does
mean that posts to moderated groups go through two hops before reaching
the moderator instead of just one, though, which can make problems
slightly more difficult to diagnose.

Unfortunately, due to the way that they're distributed and used, the alias
lists for moderators.isc.org cannot contain wildcard entries, so you
cannot ask them to just forward all posts to any moderated example.* group
to example.org. Each group still has to be configured separately and each
new moderated group still has to be sent to moderator...@isc.org for
manual processing. Hopefully in the future there will be some more easily
automated system for handling this.

------------------------------

Subject: 5. About the ftp.isc.org Newsgroup Lists

Maintained at:

<ftp://ftp.isc.org/pub/usenet/CONFIG/>

is a copy of a control.ctl file with as many public hierarchies as
possible with active maintainers listed in it, along with lists of
newsgroups in active file and checkgroups format maintained by processing
control messages using those control.ctl rules. Also maintained from the
same information is a collection of hierarchy public keys for news
administrators to get from one source, in:

<ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

A unified view of this information, including logs of recent changes, is
at:

<http://usenet.trigofacile.com/hierarchies/>

These can be valuable resources for you as a hierarchy administrator in a
couple of different ways.

First, some users and some news sites use these lists of newsgroups to
determine what newsgroups to carry, either using the list verbatim,
checking the list to see if a newsgroup is listed before being willing to
add it, or synchronizing particular hierarchies against that list. Having
your hierarchy there may therefore increase the number of sites that carry
your hierarchy and may make it easier for interested users to get the
groups added at their local site.

Second, particularly if you're using PGP-signed control messages, this
list can serve as a check that your control messages are working properly.
The list updates once an hour, so if you send a control message and in a
few hours the result isn't reflected in the ftp.isc.org lists, something
may have gone wrong (maybe you forgot an Approved header, or the PGP
signature wasn't valid for some reason).

Finally, many news administrators, even if they don't use the newsgroup
lists, use the control.ctl file and PGPKEYS file to configure their own
news servers. This control.ctl file is also included in INN releases.

To get your hierarchy listed in these files, send mail to
usenet...@isc.org. Please include your control.ctl entry, as
described above, along with your PGP public key or a URL from which it can
be obtained (preferrably the latter). Also provide a pointer to a current
list of newsgroups in checkgroups format so that they can be added to the
list at the same time. usenet...@isc.org sometimes has a bit of a
backlog, so please allow a couple of weeks for a response.

------------------------------

Subject: 6. Other Resources

Here are some other resources to be aware of if you're maintaining a
hierarchy:

news.admin.hierarchies
Most discussion of new hierarchies and of hierarchy administration in
general takes place here. There isn't a lot of discussion normally,
just hierarchy FAQs, but a lot of people read the group and can
provide help and suggestions if you post.

news.software.nntp
The technical discussion group for news software and the news
protocols, the readers of this group may be able to help if you have
questions about news server configuration or about the details of the
PGP signature system for control messages.

Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
An archive of all newgroup and rmgroup control messages, used by some
sites to check to see whether a requested group had a valid control
message and useful as a check to be sure that your control messages
are getting out.

Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
A collected control.ctl file (which is also included in INN) and lists
of newsgroups maintained by those rules. Also includes a file
describing the checkgroups format, providing some guidelines for
newsgroup naming, and describing how to write newsgroup descriptions.

Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
A merged view of the information maintained at ftp.isc.org and the
easiest way to view the information together and check it for
accuracy.

pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
Mentioned many times above, this is the central site for the reference
implementations of the PGP control message signing protocol. It also
has a list of public keys of hierarchy administrators and information
about how to enable PGP verification of control messages.

news-admin.org <http://www.news-admin.org/>
A hosting service for hierarchy administrators and Usenet hierarchies
that you may want to consider using, as well as a good collection of
examples of web sites for various hierarchies.