control-archive 1.2.0 released
Russ Allbery
This software generates an INN control.ctl configuration file from
hierarchy configuration fragments, verifies control messages using GnuPG
where possible, processes new control messages to update a newsgroup list,
archives new control messages, and exports the list of newsgroups in a
format suitable for synchronizing the newsgroup list of a Netnews news
server. It is the software that maintains the control message and
newsgroup lists available from ftp.isc.org.
Changes from previous release:
When generating a control message report with exclusions, don't send
the report if all actions during the time period were excluded.
Add better debugging to control-summary for failure to load the
template and failure to connect to an NNTP server.
Install the control-report template on make install.
Hierarchy updates:
* Fix pubnet.* URL into the control archive
* Remove sdnet.* configuration, document in docs/hierarchies
* Add eternal-september.* from http://www.eternal-september.org/
* Add szaf.* (private)
* Update sender for opennews.*
* Update admin group for cl.*
* Update URL for hun.*
* Update URL and add syncable server for nlo.*
* Remove slac.* and bes.* configuration, document in docs/hierarchies
* Mention zipnews.* as obsolete in the hierarchy notes file
* Add admin group for ka.*
* Update ibm.* information in the hierarchy notes file
You can download it from:
<http://www.eyrie.org/~eagle/software/control-archive/>
This package is maintained using Git; see the instructions on the above
page to access the Git repository.
Please let me know of any problems or feature requests not already listed
in the TODO file.
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Julien ÉLIE
> * Add eternal-september.* from http://www.eternal-september.org/
It did not work on <12623007...@news.eternal-september.org>.
Still no newsgroup here:
http://usenet.trigofacile.com/hierarchies/index.py?see=ETERNAL-SEPTEMBER
In <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT> and
<ftp://ftp.isc.org/pub/usenet/CONFIG/README>:
[component] MUST not be longer than 14 characters
There is an additional criteria that is also taken from the current
USEFOR draft, but which is still a matter of some debate -- the limit on
the length of a component. The de facto limit was 14 octets for over a
decade and a half. It is currently proposed that this be lifted to a
"soft" limit of 30 octets (in other words, this limit is recommended but
not required). This limit is currently enforced. Entire newsgroup
names are limited to 80 characters, rather than the soft limit of 72
characters proposed by the standard, because two existing non-joke
groups have names longer than 72 characters.
Could the length of a component be increased to 30? (Unless it already is
and in that case I do not understand why the checkgroups is not processed.)
Also, grisbi.* checkgroups <checkgroups...@news.grisbi.org> was not
processed whereas there are changes in the descriptions of its newsgroups.
I think there is a problem in the Sender: line of this checkgroups. It should
be the same as From:, or not used at all...
I have just sent a mail to G�rald to make him aware of that.
> * Add szaf.* (private)
<hhk6hi$2nfj$1...@news.unit0.net> does not seem to have been received
(posted to szaf.admin).
--
Julien �LIE
� En v�rit�, le chemin importe peu, la volont� d'arriver suffit � tout. �
D. Stussy
news:hhkin9$7as$1...@news.trigofacile.com...
> ...
> Also, grisbi.* checkgroups <checkgroups...@news.grisbi.org> was
not
> processed whereas there are changes in the descriptions of its
newsgroups.
>
> I think there is a problem in the Sender: line of this checkgroups. It
should
> be the same as From:, or not used at all...
> I have just sent a mail to G�rald to make him aware of that.
??? Coming from the e-mail RFCs, sender is meant for when either there
are multiple authors specified in from (to indicate which one submitted the
message), or when it is DIFFERENT than from. Either way, it may be
included in from (i.e. a subset), but NOT the same as from; else omitted.
Adam H. Kerman
>"Julien ELIE" <iul...@nom-de-mon-site.com.invalid> wrote:
>>Also, grisbi.* checkgroups <checkgroups...@news.grisbi.org> was
>not
>>processed whereas there are changes in the descriptions of its
>newsgroups.
Are you ever going to use QuoteFix so you stop sending broken lines in
quotes? By default, Outlook Express is incapable of quoting in a
conventional manner. You've been doing this for years.
>>I think there is a problem in the Sender: line of this checkgroups. It
>should
>>be the same as From:, or not used at all...
>>I have just sent a mail to G�rald to make him aware of that.
>??? Coming from the e-mail RFCs, sender is meant for when either there
>are multiple authors specified in from (to indicate which one submitted the
>message), or when it is DIFFERENT than from. Either way, it may be
>included in from (i.e. a subset), but NOT the same as from; else omitted.
Sender identifies the author's agent that sent the message. It does not
suggest in any way that the agent was the author, and no, it is definitely
not a subset of authors listed on From. Generally, it's used to indicate
that the author sent a message through a host that he doesn't have an
account on using someone else's privileges.
If it's redundant of From, there is no standard requiring its omission,
even if it's pointless.
Julien ÉLIE
> If it's redundant of From, there is no standard requiring its omission,
> even if it's pointless.
It is what I changed, amongst other things, in my last release of
signcontrol.py
Only a SHOULD per RFC 5322:
If the originator of the message can be indicated
by a single mailbox and the author and transmitter are identical, the
"Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD
appear.
--
Julien �LIE
� En v�rit�, le chemin importe peu, la volont� d'arriver suffit � tout. �
Russ Allbery
>> * Add eternal-september.* from http://www.eternal-september.org/
> It did not work on <12623007...@news.eternal-september.org>.
> Still no newsgroup here:
> http://usenet.trigofacile.com/hierarchies/index.py?see=ETERNAL-SEPTEMBER
pgpverify fails for signatures for that hierarchy because the PGP key ID
doesn't follow the recommendations in:
http://www.eyrie.org/~eagle/faqs/usenet-hier.html
and is more than just the e-mail address. I really need to fix that bug
to extract the e-mail address instead, but the same bug will affect INN
and other servers, so it needs to be fixed everywhere.
> [component] MUST not be longer than 14 characters
> There is an additional criteria that is also taken from the current
> USEFOR draft, but which is still a matter of some debate -- the limit on
> the length of a component. The de facto limit was 14 octets for over a
> decade and a half. It is currently proposed that this be lifted to a
> "soft" limit of 30 octets (in other words, this limit is recommended but
> not required). This limit is currently enforced. Entire newsgroup
> names are limited to 80 characters, rather than the soft limit of 72
> characters proposed by the standard, because two existing non-joke
> groups have names longer than 72 characters.
> Could the length of a component be increased to 30? (Unless it already
> is and in that case I do not understand why the checkgroups is not
> processed.)
The software actually does not impose any limit on component length. I'll
update the documentation.
> Also, grisbi.* checkgroups <checkgroups...@news.grisbi.org> was
> not processed whereas there are changes in the descriptions of its
> newsgroups.
Same problem.
>> * Add szaf.* (private)
> <hhk6hi$2nfj$1...@news.unit0.net> does not seem to have been received
> (posted to szaf.admin).
This is an interesting problem. I don't really feel comfortable carrying
szaf.admin on Stanford's servers given that it's a private hierarchy, but
that means that checkgroups posted only there aren't received and archived
by the archive software.
Julien ÉLIE
>>> * Add eternal-september.* from http://www.eternal-september.org/
>>>
>> It did not work on <12623007...@news.eternal-september.org>.
>
> pgpverify fails for signatures for that hierarchy because the PGP key ID
> doesn't follow the recommendations in:
>
> http://www.eyrie.org/~eagle/faqs/usenet-hier.html
OK.
> The software actually does not impose any limit on component length. I'll
> update the documentation.
Thanks!
>> <hhk6hi$2nfj$1...@news.unit0.net> does not seem to have been received
>> (posted to szaf.admin).
>
> This is an interesting problem. I don't really feel comfortable carrying
> szaf.admin on Stanford's servers given that it's a private hierarchy, but
> that means that checkgroups posted only there aren't received and archived
> by the archive software.
Hmm... Or upgrade to INN 2.4.4 or later? :)
In 2.4.4, we fixed that issue. Unless there is another issue (if you
already have INN 2.4.4).
Checkgroups are now properly propagated even though the news server
does not carry the groups they are posted to.
Checkgroups were previously rejected and not propagated.
--
Julien ÉLIE
« Hey, I had to let awk be better at *something*... » (Larry Wall)
Russ Allbery
> Hmm... Or upgrade to INN 2.4.4 or later? :)
> In 2.4.4, we fixed that issue. Unless there is another issue (if you
> already have INN 2.4.4).
> Checkgroups are now properly propagated even though the news server
> does not carry the groups they are posted to.
> Checkgroups were previously rejected and not propagated.
I think they're using whatever came with CentOS, and I suspect that
getting it upgraded is going to be a bit of a hassle. (Budget cuts mean
that central IT doesn't run the news server any more, and while my group
has our own, we don't do the external feed stuff.)
Adam H. Kerman
>Hi Adam,
>>If it's redundant of From, there is no standard requiring its omission,
>>even if it's pointless.
>It is what I changed, amongst other things, in my last release of
>signcontrol.py
>Only a SHOULD per RFC 5322:
> If the originator of the message can be indicated
> by a single mailbox and the author and transmitter are identical, the
> "Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD
> appear.
Then D.Stussey was correct about that point, but not the other.
Julien ÉLIE
>> Hmm... Or upgrade to INN 2.4.4 or later? :)
>> In 2.4.4, we fixed that issue. Unless there is another issue (if you
>> already have INN 2.4.4).
>>
>> Checkgroups are now properly propagated even though the news server
>> does not carry the groups they are posted to.
>>
>> Checkgroups were previously rejected and not propagated.
>
> I think they're using whatever came with CentOS, and I suspect that
> getting it upgraded is going to be a bit of a hassle. (Budget cuts mean
> that central IT doesn't run the news server any more, and while my group
> has our own, we don't do the external feed stuff.)
Then another idea would be to pullnews (or suck) the control.checkgroups
newsgroup of for instance nntp.aioe.org (which is open).
Daily or twice a day because the retention is not high.
--
Julien ÉLIE
« Le carré est un triangle qui a réussi,
ou une circonférence qui a mal tourné. » (Pierre Dac)
Ray Banana
>>> * Add eternal-september.* from http://www.eternal-september.org/
>> It did not work on <12623007...@news.eternal-september.org>.
>> Still no newsgroup here:
>> http://usenet.trigofacile.com/hierarchies/index.py?see=ETERNAL-SEPTEMBER
> pgpverify fails for signatures for that hierarchy because the PGP key ID
> doesn't follow the recommendations in:
> http://www.eyrie.org/~eagle/faqs/usenet-hier.html
> and is more than just the e-mail address. I really need to fix that bug
> to extract the e-mail address instead, but the same bug will affect INN
> and other servers, so it needs to be fixed everywhere.
I created a new key, now available at
'http://www.eternal-september.org/control/pgpkey.txt
as indicated in the control messages.
Checked with pgpverify and everything looks OK:
./signcontrol < eternal-september.checkgroups \
| /usr/lib/news/bin/pgpverify -test
gpgv: Signature made Sat 02 Jan 2010 11:44:50 AM CET using RSA key ID E7AF34AD
[GNUPG:] SIG_ID ApaPoRvZrrQmHl6nI2yN5tm/440 2010-01-02 1262429090
[GNUPG:] GOODSIG A45E4690E7AF34AD ne...@eternal-september.org
gpgv: Good signature from "ne...@eternal-september.org"
[GNUPG:] VALIDSIG B79A2BC67D5AFF7918E2AC914BC125F1 2010-01-02 1262429090
0 3 0 1 1 01 B79A2BC67D5AFF7918E2AC914BC125F1
ne...@eternal-september.org
--
Time flies like an arrow, fruit flies like a Banana.
http://www.eternal-september.org
Thomas Hochstein
>> * Add szaf.* (private)
>
> <hhk6hi$2nfj$1...@news.unit0.net> does not seem to have been received
> (posted to szaf.admin).
*sigh*
Thanks for the heads-up. Neither szaf.* nor szaf.admin (nor any
control messages) should leak outside as they obviuosly do. I'll have
a look at it.
-thh
Russ Allbery
> *sigh*
> Thanks for the heads-up. Neither szaf.* nor szaf.admin (nor any
> control messages) should leak outside as they obviuosly do. I'll have
> a look at it.
News software really likes propagating control messages and controlling it
with the Newsgroups header is difficult. You probably want to use
Distribution, which is somewhat easier to keep out of outgoing feeds.
(You may already be doing this, but I thought I'd mention it.)
Russ Allbery
> I created a new key, now available at
> 'http://www.eternal-september.org/control/pgpkey.txt
> as indicated in the control messages.
> Checked with pgpverify and everything looks OK:
> ./signcontrol < eternal-september.checkgroups \
> | /usr/lib/news/bin/pgpverify -test
> gpgv: Signature made Sat 02 Jan 2010 11:44:50 AM CET using RSA key ID E7AF34AD
> [GNUPG:] SIG_ID ApaPoRvZrrQmHl6nI2yN5tm/440 2010-01-02 1262429090
> [GNUPG:] GOODSIG A45E4690E7AF34AD ne...@eternal-september.org
> gpgv: Good signature from "ne...@eternal-september.org"
> [GNUPG:] VALIDSIG B79A2BC67D5AFF7918E2AC914BC125F1 2010-01-02 1262429090
> 0 3 0 1 1 01 B79A2BC67D5AFF7918E2AC914BC125F1
> ne...@eternal-september.org
Thanks, I've updated everything to use that. It's a shame for you to have
to fall back to a 1024-bit RSA key, but ah well. Hopefully I'll get a
chance to fix pgpverify in the not-too-distant future.
Thomas Hochstein
> Thomas Hochstein <t...@inter.net> writes:
>> Thanks for the heads-up. Neither szaf.* nor szaf.admin (nor any
>> control messages) should leak outside as they obviuosly do. I'll have
>> a look at it.
>
> News software really likes propagating control messages and controlling it
> with the Newsgroups header is difficult.
It was a local configuration error (inadvertently feeding control.*),
as it always seems to be ... Personally, I prefer to never add "*",
but "!*" to any outgoing feed as an additional safeguard.
> You probably want to use
> Distribution, which is somewhat easier to keep out of outgoing feeds.
> (You may already be doing this, but I thought I'd mention it.)
I'll look into that (but I fear that is even more error prone).
Regards,
-thh
Thomas Hochstein
> Hierarchy updates:
[...]
Is there a suggested format for submitting additions and/or
corrections to usenet-config(at)isc.org?
Regards,
-thh
Russ Allbery
>> Hierarchy updates:
> [...]
Not really, although if you send a control.ctl entry, that includes all
the information that I need apart from the key itself. Note that nearly
all of a control.ctl entry is generated from a set of configuration files
in the control-archive package, and currently that machinery doesn't
support custom comments. It's on the TODO list (patches welcome, of
course).
Anyone who's really inspired can send me Git patches against the
repository at git://git.eyrie.org/usenet/control.git, of course. :)
Thomas Hochstein
> Thomas Hochstein <t...@inter.net> writes:
>> Is there a suggested format for submitting additions and/or
>> corrections to usenet-config(at)isc.org?
>
> Not really, although if you send a control.ctl entry, that includes all
> the information that I need apart from the key itself. [...]
>
> Anyone who's really inspired can send me Git patches against the
> repository at git://git.eyrie.org/usenet/control.git, of course. :)
Okay, I'll try. :)
Since December, I've done s survey on German (language) regional
hierarchies (see <nah.10011...@thorondor.akallabeth.de> which
will perhaps be more suitable for news.lists.misc in future). I'll
send Git patches to usenet-config for those hierarchies that already
are in the repository or are managed in some way.
However most "small" hierarchies are not "managed" (any longer) and
have no official syncable server (or administrative contact), but do
have a stable set of group which are sufficiently propagated (at least
in Germany) and are not (yet) defunct; on the contrary, some of them
have a surprisingly high traffic (and some already are in the active
and newsgroups files on <ftp://ftp.isc.org/pub/usenet/CONFIG/>, but
that list is not always up to date).
What would be the best way to get information on those hierarchies
(list of groups, contact information, ...) into a central repository?
They are registered with the "Master List of Newsgroup Hierarchies",
but the information therein mostly is not very helpful. Should I send
patches to the active and newsgroups files on ftp.isc.org? Would it be
sensible to add dummy entries to the control.ctl? What is your opinion
on that?
Regards,
-thh
--
Thomas Hochstein <t...@inter.net>
<http://th-h.de/>
Adam H. Kerman
>What would be the best way to get information on those hierarchies
>(list of groups, contact information, ...) into a central repository?
>They are registered with the "Master List of Newsgroup Hierarchies",
>but the information therein mostly is not very helpful. Should I send
>patches to the active and newsgroups files on ftp.isc.org? Would it be
>sensible to add dummy entries to the control.ctl? What is your opinion
>on that?
The placeholder should be nothing more than the URL of the list you
put together. I don't think the control message handling instructions
should be there (or changed if they existed previously), not until someone
has a valid reason to send a newgroup or rmgroup message.
Russ Allbery
> Okay, I'll try. :)
> Since December, I've done s survey on German (language) regional
> hierarchies (see <nah.10011...@thorondor.akallabeth.de> which
> will perhaps be more suitable for news.lists.misc in future). I'll
> send Git patches to usenet-config for those hierarchies that already
> are in the repository or are managed in some way.
Thanks! Applied and in the next release.
> However most "small" hierarchies are not "managed" (any longer) and
> have no official syncable server (or administrative contact), but do
> have a stable set of group which are sufficiently propagated (at least
> in Germany) and are not (yet) defunct; on the contrary, some of them
> have a surprisingly high traffic (and some already are in the active
> and newsgroups files on <ftp://ftp.isc.org/pub/usenet/CONFIG/>, but
> that list is not always up to date).
> What would be the best way to get information on those hierarchies
> (list of groups, contact information, ...) into a central repository?
If there is any sort of contact information or the other sort of metadata
that is tracked in control.ctl, let me know, and I'll add that to the
control.ctl config. I now have the capability in the code for generating
comment-only entries for hierarchies with no administrator that can
include the other metadata for the group.
If you have updated lists of newsgroups that you've verified as current,
please send them to me in checkgroups format and I'll apply the changes to
the active file distributed on ftp.isc.org.
> They are registered with the "Master List of Newsgroup Hierarchies",
> but the information therein mostly is not very helpful. Should I send
> patches to the active and newsgroups files on ftp.isc.org? Would it be
> sensible to add dummy entries to the control.ctl? What is your opinion
> on that?
I think dummy entries for active hierarchies is sensible. For
ftp.isc.org, the information is maintained in a database, so patches
aren't really helpful. Complete checkgroups for the hierarchy are trivial
for me to apply.