docheckgroups et al
John F. Morse
newsgroups and active files, and downloaded copies of the newsgroups
file from isc.org.
My notes from an old INN 2.4.3 server last year shows it did run:
------------------------------------------------------------------------
news@optima7:/var/lib/news$ date ; cat newsgroups |
/usr/lib/news/bin/docheckgroups ; date
Mon Apr 27 09:21:29 CDT 2009
Mon Apr 27 10:36:57 CDT 2009
news@optima7:/var/lib/news$
# Change these lines:
# comp.parallel Massively parallel hardware/software. (Moderated)
# comp.protocols.dns.bind Berkeley Internet Name Domain (BIND).
(Moderated)
# comp.unix.bsd.freebsd.announce Announcements pertaining to
FreeBSD. (Moderated)
# comp.unix.bsd.netbsd.announce Announcements pertaining to NetBSD.
(Moderated)
# comp.unix.bsd.openbsd.announce OpenBSD announcements. (Moderated)
# info.bind *The Berkeley BIND server (bi...@arpa.berkeley.edu).
(Moderated)
# linux.debian.announce debian-...@lists.debian.org (Moderated)
# linux.debian.announce.devel
debian-dev...@lists.debian.org (Moderated)
# linux.debian.announce.security
debian-secur...@lists.debian.org (Moderated)
# linux.debian.beowulf debian-...@lists.debian.org (Moderated)
# linux.debian.doc debia...@lists.debian.org (Moderated)
# linux.debian.isp debia...@lists.debian.org (Moderated)
# linux.debian.laptop debian...@lists.debian.org (Moderated)
# linux.debian.maint.boot debia...@lists.debian.org (Moderated)
# linux.debian.maint.dpkg debia...@lists.debian.org (Moderated)
# linux.debian.maint.firewall debian-...@lists.debian.org
(Moderated)
# linux.debian.news debia...@lists.debian.org (Moderated)
# linux.debian.policy debian...@lists.debian.org (Moderated)
# linux.debian.ports.68k debia...@lists.debian.org (Moderated)
# linux.debian.ports.powerpc debian-...@lists.debian.org
(Moderated)
# linux.debian.project debian-...@lists.debian.org (Moderated)
# linux.debian.security debian-...@lists.debian.org (Moderated)
# linux.debian.user debia...@lists.debian.org (Moderated)
# linux.debian.www debia...@lists.debian.org (Moderated)
# linux.samba sa...@lists.samba.org (Moderated)
# linux.samba.announce samba-a...@lists.samba.org (Moderated)
# rec.radio.broadcasting Discussion of global domestic broadcast
radio. (Moderated)
Mon Apr 27 10:36:57 CDT 2009
------------------------------------------------------------------------
However I cannot get this to work on INN 2.4.5 nor 2.5.2. Here's what
INN 2.4.5 shows:
news@optima11:/var/lib/news$ date ; cat newsgroups |
/usr/lib/news/bin/docheckgroups ; date
Tue Dec 14 10:07:18 CST 2010
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
Tue Dec 14 10:07:35 CST 2010
------------------------------------------------------------------------
When I run it against the downloaded newsgroup file from ISC, it does
identically:
news@optima11:/var/lib/news$ date ; cat 'newsgroups-ISC_20101214-45962'
| /usr/lib/news/bin/docheckgroups ; date
Tue Dec 14 10:08:52 CST 2010
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
Tue Dec 14 10:09:12 CST 2010
------------------------------------------------------------------------
Another way to run it also returns the same egrep errors:
news@optima11:/var/lib/news$ date ; docheckgroups <
newsgroups-ISC_20101214-45962 ; date
Tue Dec 14 10:10:29 CST 2010
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
Tue Dec 14 10:10:45 CST 2010
------------------------------------------------------------------------
Run either way on a INN 2.5.2 box it silently does nothing:
news@n102:/var/lib/news$ date ; cat newsgroups |
/usr/lib/news/bin/docheckgroups ; date
Tue Dec 14 10:12:00 CST 2010
Tue Dec 14 10:12:00 CST 2010
news@n102:/var/lib/news$
news@n102:/var/lib/news$ date ; docheckgroups <
newsgroups-ISC_20101214-45962 ; date
Tue Dec 14 10:13:29 CST 2010
Tue Dec 14 10:13:29 CST 2010
news@n102:/var/lib/news$
------------------------------------------------------------------------
The purpose of using docheckgroups locally is to see what I need to do
to update my active file with missing groups.
I realize actsync can do this, but it wastes far too much server time,
reporting every ctlinnd command, restarting all connections, etc., for
each and every newsgroup added. Dumping a list of newgroup commands into
a terminal also causes malformed newsgroup names when the ssh connection
is adding ctlinnd newgroup commands faster than the server can process them.
The docheckgroups command seems to just do the work when the output is
piped into mod-active, without leaving behind miles of logs, malformed
group names, etc.
If the docheckgroups groups output is what I really want, then I can run
it a second time with the -u option and with the pipe to mod-active
without worry of losing groups.
I say this because an earlier trial resulted in hundreds of faulty
newgroups in the active file. These were valid group names but had the
descriptions from the newsgroups file immediately appended to the group
name. I suspect the newsgroups file I used may have had multiple spaces
and no tab characters, but I could not verify that. All of them I have
do have tabs and not spaces. Perhaps mod-active is broken?
Is there a difference between docheckgroups for INN 2.4.3, 2.4.5 and
2.5.2? I see a docheckgroups change at 2.4.4, but don't believe it would
affect this.
The unmatched egrep errors may be in the existing newsgroups file (or
perhaps ISC's). The files are far too large to search manually, and I
don't know how to do it otherwise.
I would rather just ignore newsgroups and work with only the active
file. I can always simply wget a current newsgroups file, and add my
local group descriptions, which I've always done in the past.
Any suggestions on how to update the local active files (or in
hierarchical parts or stages, using a pattern like docheckgroups
"^c\..*$") with the current ISC active file?
I certainly do not want to throttle a server, copy and paste ranges of
the ISC's active file text into the working active file, reload, go,
etc., and on multiple remote servers. Not a good method at all. :-(
BTW, received control checkgroups messages are working just fine.
With that in mind, perhaps the local ISC copy needs to be formatted as a
"here-document" (something unusual like <<zRbJ inserted at beginning and
end), or can it be used as-is?
TIA
--
John
When a person has -- whether they knew it or not -- already rejected the Truth, by what means do they discern a lie?
Julien ÉLIE
> However I cannot get this to work on INN 2.4.5 nor 2.5.2. Here's what
> INN 2.4.5 shows:
>
> news@optima11:/var/lib/news$ date ; cat newsgroups |
> /usr/lib/news/bin/docheckgroups ; date
> Tue Dec 14 10:07:18 CST 2010
> egrep: Unmatched ) or \)
Is it linked to the issue you reported last year?
https://groups.google.com/group/news.software.nntp/browse_thread/thread/43bf5cc8836e49d
Wasn't the problem solved last time?!?
> I realize actsync can do this, but it wastes far too much server time,
> reporting every ctlinnd command, restarting all connections, etc., for
> each and every newsgroup added. Dumping a list of newgroup commands into
> a terminal also causes malformed newsgroup names when the ssh connection
> is adding ctlinnd newgroup commands faster than the server can process them.
I do not understand how the fastness of the SSH connection affects
the way ctlinnd newgroup commands are parsed.
Idea: why wouldn't you pipe the actsync result to mod-active?
It does not restart all connections after every command.
Another suggestion would be to use actsyncd (with a "d").
> If the docheckgroups groups output is what I really want, then I can run
> it a second time with the -u option and with the pipe to mod-active
> without worry of losing groups.
Yes.
> I say this because an earlier trial resulted in hundreds of faulty
> newgroups in the active file. These were valid group names but had the
> descriptions from the newsgroups file immediately appended to the group
> name. I suspect the newsgroups file I used may have had multiple spaces
> and no tab characters, but I could not verify that. All of them I have
> do have tabs and not spaces. Perhaps mod-active is broken?
It might. If you find out any example of wrong behaviour, please
give it and I will try to fix it.
> Is there a difference between docheckgroups for INN 2.4.3, 2.4.5 and
> 2.5.2?
Yes. For instance the new "-u" flag.
--
Julien ÉLIE
« Bibite, fratres, bibite, ne diauolus uos otiosos inueniat. »
John F. Morse
> Is it linked to the issue you reported last year?
>
> https://groups.google.com/group/news.software.nntp/browse_thread/thread/43bf5cc8836e49d
That does look familiar. I don't remember anything about it though.
Maybe I'm getting senile or Alzheimer's is setting in.
> Wasn't the problem solved last time?!?
I don't remember, Julien. It was nearly two years ago. I barely remember
last week. :-(
I do remember this last weekend when my two-year-old heat pump and
furnace decided it no longer had the fossil fuel heat kit, which comes
on using natural gas when the outside temperature drops below 40 F (4
C), and the heat pump doesn't have enough warm air to pump into the house.
Sunday and Monday the outdoor temperature was near zero F (-15C) with
20-30 MPH (32-48 k/h) wind, and the house dropped to 58 F (14 C) for the
two days.
The service company came Sunday morning but the young kid didn't know
anything about this type of system.
Monday a slightly older guy arrived, with a little more experience. He
called an "old-timer" who provided a few suggestions (guesstimates) for
reprogramming several options on the digital Honeywell thermostat, and
it's running fine now.
But will it fail again? I ask because why did it "forget" the settings
it had?
I have to get onto them to find out why the settings were changed on
their own before the warranty expires. If I don't, well.... $$$
Maybe my brain froze? ;-)
I think I just gave up trying to use docheckgroups and mod-active.
Instead I probably used either actsync, or just "dumped" many textfiles
of around 30 or so ctlinnd newgroup commands per time.
I hate using that method for more than a handful of groups. It is slow,
hogs the server, and can cause corrupted groups if I try using too many
on each "dump." It's quicker to just stop the server, paste in the new
text to the active file, and restart the server. However that opens up
the possibility of accidentally adding groups that already exist in the
active file.
So a lot of rcp/scp copying and diff checking is required, including awk
to remove the watermarks (awk '{print $1, $4}' active > active1-4.txt),
then sorting because the real active file is not in order, all over ssh,
plus any editing to eliminate duplicate groups. Quite a bit of work that
one of the utilities should be able to do quickly.
Actsync is difficult to impossible to use amongst the servers due to
ssl/tls and authentication. I would probably just scp the files to my
main workstation, work on them here, then scp them back to the server(s)
and run actsync against the local file instead of a host. Since I don't
remember using actsync before, I don't know if it pauses/throttles and
adds groups, whether it creates miles of logs and Daily Report entries,
or is "quiet."
>> I realize actsync can do this, but it wastes far too much server time,
>> reporting every ctlinnd command, restarting all connections, etc., for
>> each and every newsgroup added. Dumping a list of newgroup commands into
>> a terminal also causes malformed newsgroup names when the ssh connection
>> is adding ctlinnd newgroup commands faster than the server can
>> process them.
>
> I do not understand how the fastness of the SSH connection affects
> the way ctlinnd newgroup commands are parsed.
I can "dump" a ctlinnd newgroup textfile from the clip into a terminal
on a remote ssh session, and after about 20 newgroup commands, the
groups start getting their names truncated, with the truncated portion
just ignored as a bad BASH command. Or it can be added to another
newsgroup name.
Doesn't matter whether the server is throttled or not, or even if it is
to one of the new fast CPU servers. There is just too much overhead in
sequential ctlinnd newgroup commands for INN to keep up with.
It acts like old serial handshaking, where some kind of flow control is
needed (like XON/XOFF, RTS/CTS).
I was under the impression that one of the utilities (actsync,
docheckgroups, mod-active) could automate the process of expanding a
smaller active file, like Big-8 only plus a few local groups, to
something closer to the larger ISC active file, hopefully using a regex
to skip (not select) certain hierarchies (like alt.*).
I have the complete Big-8 and quite a few alt.comp.* and other computer
groups, plus several hundred local groups, and many foreign language
hierarchies. I've added all groups in the ^a.* and ^b.* hierarchies
except the alt.* hierarchy. From ^c.* to ^z.* are quite a few more.
There are presently the same 6723 groups on each of my servers
(transits, readers, test).
I'd like to add everything the ISC has in the active file except alt.*
and then later add in the "worthwhile" alt groups, skipping any binary
groups and the ones kiddies create. "Everything" here meaning groups and
hierarchies like houston.*, k12.*, law.*, etc.
What steps would you use if you wanted to add just the complete Big-8
hierarchies to your server?
> Idea: why wouldn't you pipe the actsync result to mod-active?
> It does not restart all connections after every command.
> Another suggestion would be to use actsyncd (with a "d").
I never thought about that method. I'll give it a try with a few
newsgroups and see how it does. Thanks.
But see above where I mention actsync is difficult between two hosts
when servers use SSL/TLS, password authentication (-A usually will not
work), and ports other than 119 (which could be overcome with the colon).
>> I say this because an earlier trial resulted in hundreds of faulty
>> newgroups in the active file. These were valid group names but had the
>> descriptions from the newsgroups file immediately appended to the group
>> name. I suspect the newsgroups file I used may have had multiple spaces
>> and no tab characters, but I could not verify that. All of them I have
>> do have tabs and not spaces. Perhaps mod-active is broken?
>
> It might. If you find out any example of wrong behaviour, please
> give it and I will try to fix it.
I will if it does this again. It did it on both of the new fast servers.
Several hundred identical corrupted newsgroups in the active file, which
took considerable time (several hours) to straighten out.
Somehow it also removed at least one group which I had to manually add
back in with ctlinnd newgroup. Amazingly none of the articles were lost.
It is a CNFS group if that matters (no real "file" was deleted, like
tradspool article files or the group directory would have been).
>> Is there a difference between docheckgroups for INN 2.4.3, 2.4.5 and
>> 2.5.2?
>
> Yes. For instance the new "-u" flag.
Of course, but I meant something like a patch, maybe it was
innshellvars? I noticed a reference to that somewhere, which appeared to
possibly be a prior fix you provided. But I can't find it now.
I gotta hit the bed. Been up far too many hours. Maybe you'll wake up
and give me a shove in the right direction. ;-)
TIA.
Julien ÉLIE
> So a lot of rcp/scp copying and diff checking is required, including awk
> to remove the watermarks (awk '{print $1, $4}' active > active1-4.txt),
> then sorting because the real active file is not in order, all over ssh,
> plus any editing to eliminate duplicate groups.
These are delicate operations. Do you also synchronize your overview
data to make it aware of the new newsgroups?
> What steps would you use if you wanted to add just the complete Big-8
> hierarchies to your server?
Take the last checkgroups of the hierarchy (in control.checkgroups)
and process it.
You could feed it to controlchan. I think "controlchan /path/to/the/file"
works. The file is the whole article (headers + body with checkgroups).
(Of course docheckgroups would work on the list of groups.)
> But see above where I mention actsync is difficult between two hosts
> when servers use SSL/TLS, password authentication (-A usually will not
> work), and ports other than 119 (which could be overcome with the colon).
Maybe you could try to wrap the call to actsync into stunnel?
(I have not tested.)
>>> Is there a difference between docheckgroups for INN 2.4.3, 2.4.5 and
>>> 2.5.2?
>>
>> Yes. For instance the new "-u" flag.
>
> Of course, but I meant something like a patch, maybe it was
> innshellvars? I noticed a reference to that somewhere, which appeared to
> possibly be a prior fix you provided. But I can't find it now.
Do you mean that patch?
http://inn.eyrie.org/trac/changeset/8320/trunk/scripts/innshellvars.in
Export a "C" value for LC_CTYPE so as to treat all input
as a stream of bytes in our scripts.
Problems are for instance triggered off by docheckgroups on
UTF-8 systems when it receives a ISO-8859-15 checkgroups
control message: sed regexps do not match what they should.
Take care, with the very cold weather you have at your side,
--
Julien ÉLIE
« En vérité, le chemin importe peu, la volonté d'arriver suffit à tout. »
John F. Morse
>> So a lot of rcp/scp copying and diff checking is required, including awk
>> to remove the watermarks (awk '{print $1, $4}' active > active1-4.txt),
>> then sorting because the real active file is not in order, all over ssh,
>> plus any editing to eliminate duplicate groups.
>
> These are delicate operations. Do you also synchronize your overview
> data to make it aware of the new newsgroups?
I cannot answer that, Julien. Off-hand I wouldn't know how to do it, and
therefore don't remember doing it in the past. It's been many months
since I used the paste-into-active method.
Perhaps the syncing is done at midnight, or with an rc.news start/stop
or ctlinnd reload or xexec innd? I just don't remember. Sorry.
All I can say is the servers have always seemed to be working just fine
after the edits of the active file. I see the new groups when checking
with a newsreader, and can access the articles without problems.
>> What steps would you use if you wanted to add just the complete Big-8
>> hierarchies to your server?
>
> Take the last checkgroups of the hierarchy (in control.checkgroups)
> and process it.
Where is this control.checkgroups file?
If it is the control.checkgroups group on my servers, I can't access it
because it is in CNFS. I don't have a Message-ID to use in grephistory
to obtain the token. Without that, I don't know how to read any article
in the group.
I can't find any archived checkgroups files at the ISC site. Are they
(still) there?
As a side note, I also can no longer locate the history (charter
messages) which created the groups. Did the ISC remove this?
I have a few checkgroups messages in my e-mail archive, but those are
very limited and for only a few small hierarchies.
Hierarchies vs. groups: Then how many hierarchies are there? I count
25,592 newsgroups from the 45,963 total ISC groups (without the alt.*
hierarchy of 20,371 groups) but I don't know just how many hierarchies
there are. I do know there are many and it would be very time consuming
to manually add groups by using each hierarchy.
The obvious solution would be to use a regex pattern like "^c\..*$"
(example for hierarchies beginning with the letter "c"), but so far,
that fails. Maybe the too many variables awk/grep/egrep/sed problem
mentioned at:
https://groups.google.com/group/news.software.nntp/browse_thread/thread/43bf5cc8836e49d
http://inn.eyrie.org/trac/export/8433/trunk/control/docheckgroups.in
(There may be that mention of innshellvars that I remembered seeing and
mentioned earlier.)
> You could feed it to controlchan. I think "controlchan
> /path/to/the/file"
> works. The file is the whole article (headers + body with checkgroups).
>
> (Of course docheckgroups would work on the list of groups.)
I wish it could. I can't seem to get it to work.
>> But see above where I mention actsync is difficult between two hosts
>> when servers use SSL/TLS, password authentication (-A usually will not
>> work), and ports other than 119 (which could be overcome with the
>> colon).
>
> Maybe you could try to wrap the call to actsync into stunnel?
> (I have not tested.)
Stunnel would be possible, but it would be simpler to just use ssh and
edit the active file directly on the servers, and I don't like doing that.
I do use Stunnel for accessing several various services from the WAN,
and for using Pan and other non-SSL/TLS equipped newsreaders on the LAN,
but not for inter-server tunneling.
Even worse, the -A option in actsync will not authenticate. I only get
one local group ("info") which doesn't require passwd authentication.
Perhaps my syntax is not correct, but I've tried the command various
ways. I'm not prompted for authentication, and supplying a
username:password combination somewhere in the command causes the
command to fail entirely.
Without authentication I cannot use actsync (to a different server). Now
I could use actsync on an active file and a copy of an active file, but
I could also use that active file to create a ctlinnd newgroup file by
using docheckgroups for use with mod-active.
Then there is another problem: the docheckgroups output cannot be piped
or directed into mod-active. Or more accurately, it doesn't work for me.
No errors, just active is not modified.
But I can run docheckgroups and redirect the output to a file. Then cut
all the pertinent lines of "\t/usr/lib/news/bin/ctlinnd newgroup
{group}" and save them in another textfile. Then I can feed that file
into mod-active. The server accepts those newgroups immediately, and
doesn't log each one.
This works great, is very fast, but I am performing many extra steps,
and having to use five individual files to keep docheckgroups from
gagging on too many egrep values.
Multiply this work on one server by the number of servers (5), and you
can see why the job took around eight hours yesterday!
If you are curious, here are the steps I had to use:
I started with the ISC Newsgroups file on a workstation computer. I
broke it down into eight separate "newsgroups_{alpha-range}" files to
eliminate the egrep overload.
I rsynced these eight files to one INN news server where I could use
docheckgroups on each one. This generated eight
"docheckgroups_{alpha-range}" files.
Then I copied these back to the workstation and removed all lines of
text other than the actual ctlinnd newgroup command lines.
Being smaller, they were then recombined into five
"combined_newgroup_{alpha-range}" files.
These five files were then rsynced to each of the five news servers into
the /var/lib/news directory.
At this point I used "mod-active < combined_newgroup_{alpha-range}" to
do the actual modification of the active file (on each of the five
files, and on each of the five servers).
Quite a bit of work, time-consuming, but faster and safer than direct
edits of the active files on five servers.
Here is what the file listing looks like on the workstation:
john@noc:~/Desktop/newsgroups$ ls -al
total 5788
drwxr-xr-x 2 john john 4096 2010-12-18 05:12 .
drwxrwxrwx 4 john john 4096 2010-12-18 05:12 ..
-rw-r--r-- 1 john john 52164 2010-12-18 05:02 combined_newgroup_a-b_no-alt
-rw-r--r-- 1 john john 106651 2010-12-17 08:53 combined_newgroup_c-e
-rw-r--r-- 1 john john 479566 2010-12-17 06:46 combined_newgroup_f
-rw-r--r-- 1 john john 414951 2010-12-17 07:59 combined_newgroup_g-q
-rw-r--r-- 1 john john 217420 2010-12-17 08:00 combined_newgroup_r-z
-rw-r--r-- 1 john john 39074 2010-12-18 04:54 docheckgroups_a-b_no-alt
-rw-r--r-- 1 john john 197272 2010-12-17 05:35 docheckgroups_c
-rw-r--r-- 1 john john 82431 2010-12-17 06:26 docheckgroups_d-e
-rw-r--r-- 1 john john 1169908 2010-12-17 06:45 docheckgroups_f
-rw-r--r-- 1 john john 382046 2010-12-17 06:57 docheckgroups_g-k
-rw-r--r-- 1 john john 640072 2010-12-17 07:15 docheckgroups_l-q
-rw-r--r-- 1 john john 516272 2010-12-17 07:29 docheckgroups_r-u
-rw-r--r-- 1 john john 32715 2010-12-17 07:38 docheckgroups_v-z
-rw-r--r-- 1 john john 51702 2010-12-17 05:20 newsgroups_a-b_no-alt
-rw-r--r-- 1 john john 127393 2010-12-17 05:28 newsgroups_c
-rw-r--r-- 1 john john 78114 2010-12-17 06:20 newsgroups_d-e
-rw-r--r-- 1 john john 426485 2010-12-17 06:34 newsgroups_f
-rw-r--r-- 1 john john 169026 2010-12-17 06:53 newsgroups_g-k
-rw-r--r-- 1 john john 305817 2010-12-17 07:04 newsgroups_l-q
-rw-r--r-- 1 john john 283551 2010-12-17 07:23 newsgroups_r-u
-rw-r--r-- 1 john john 15671 2010-12-17 07:36 newsgroups_v-z
Then I removed all those extra work files from each server.
Lastly I took a pristine newsgroups file from the ISC, added my local
groups information (which I archive in a text file), cleaned up all
multiple spaces into tabs, and rsynced that file to each server. Not
that newsgroups is any value on the transit servers, but it is available
for future docheckgroups if I can ever get that to function as designed
and documented.
The work here, for now, is nearly finished. The last job remaining is to
take the ISC's alt.* hierarchy and weed out the garbage, and binaries of
various spellings, then add it into the servers (the newsgroups files
will already be current).
I am also using the wanttrash and Aj features on the main transit
server, so if I don't have the groups in the active file, they will
still propagate to peers according to their requested newsfeeds
subscriptions. So far I haven't seen very many unwanted "j" entries in
the news log.
All servers are CNFS so the additional load will not fill up a hard
disk. Retention right now is 574 days on the main reader server, and 52
days on the main transit server.
>>>> Is there a difference between docheckgroups for INN 2.4.3, 2.4.5 and
>>>> 2.5.2?
>>>
>>> Yes. For instance the new "-u" flag.
>>
>> Of course, but I meant something like a patch, maybe it was
>> innshellvars? I noticed a reference to that somewhere, which appeared to
>> possibly be a prior fix you provided. But I can't find it now.
>
> Do you mean that patch?
> http://inn.eyrie.org/trac/changeset/8320/trunk/scripts/innshellvars.in
>
> Export a "C" value for LC_CTYPE so as to treat all input
> as a stream of bytes in our scripts.
> Problems are for instance triggered off by docheckgroups on
> UTF-8 systems when it receives a ISO-8859-15 checkgroups
> control message: sed regexps do not match what they should.
I don't think that is what I noticed. See the reference up above.
> Take care, with the very cold weather you have at your side.
Two days of HVAC servicemen (actually "boys") scratching their heads and
reprogramming, finally gave me heat.
But the question still remains: what actually caused the thermostat's
interface unit to forget the settings?
Reprogramming is treating the symptom, but there is still the disease to
locate and fix.
I'll wait until a warmer season and demand they return and find the
actual cause and fix it -- before the warranty runs out. ;-)
Julien ÉLIE
** And first of all, merry Christmas to all the readers of the
newsgroup. **
>>> What steps would you use if you wanted to add just the complete
>>> Big-8 hierarchies to your server?
>>
>> Take the last checkgroups of the hierarchy (in
>> control.checkgroups) and process it.
>
> Where is this control.checkgroups file?
> If it is the control.checkgroups group on my servers, I can't access
> it because it is in CNFS. I don't have a Message-ID to use in
> grephistory to obtain the token. Without that, I don't know how to
> read any article in the group.
You could open control.checkgroups in your newsreader and manually
search for the control article.
The message-ID of the last one (December, 15th) is:
<cmsg-20101215160002$03...@isc.org>
> I can't find any archived checkgroups files at the ISC site. Are they
> (still) there?
For checkgroups, have a look at:
ftp://ftp.isc.org/pub/usenet/control/other.ctl
> As a side note, I also can no longer locate the history (charter
> messages) which created the groups. Did the ISC remove this?
I do not know. Do you speak about newgroup control messages?
Or monthly charters?
> Even worse, the -A option in actsync will not authenticate. I only
> get one local group ("info") which doesn't require passwd
> authentication. Perhaps my syntax is not correct, but I've tried the
> command various ways. I'm not prompted for authentication, and
> supplying a username:password combination somewhere in the command
> causes the command to fail entirely.
Having looked at the source code, actsync uses the file named
passwd.nntp if you ask for authentication with the -A flag.
Be sure to keep this file not readable by everyone (but only
the news user for instance).
http://www.eyrie.org/~eagle/software/inn/docs/passwd.nntp.html
The documentation for actsync is not clear enough and does not
mention passwd.nntp; I will soon add a clarification. Thanks for
having pointed out the issue.
> Then there is another problem: the docheckgroups output cannot be
> piped or directed into mod-active. Or more accurately, it doesn't
> work for me. No errors, just active is not modified.
It is very weird. I do not understand why.
> Multiply this work on one server by the number of servers (5), and
> you can see why the job took around eight hours yesterday!
Just script it :-)
Yet, something is wrong somewhere if it does not work directly…
> I am also using the wanttrash and Aj features on the main transit
> server
That's great.
--
Julien ÉLIE
« Un seul être vous manque et tout est dépeuplé. » (Alphonse de
Lamartine)
John F. Morse
> ** And first of all, merry Christmas to all the readers of the
> newsgroup. **
>
I'll echo that, although a couple of days late!
>> Where is this control.checkgroups file?
>>
>
> http://usenet.trigofacile.com/hierarchies/index.py?see=COMP,HUMANITIES,MISC,NEWS,REC,SCI,SOC,TALK&only=checkgroups
>
That looks like what I call the "newsgroups" file, both in /var/lib/news
and at ftp://ftp.isc.org/pub/usenet/CONFIG/newsgroups
>> If it is the control.checkgroups group on my servers, I can't access
>> it because it is in CNFS. I don't have a Message-ID to use in
>> grephistory to obtain the token. Without that, I don't know how to
>> read any article in the group.
>>
>
> You could open control.checkgroups in your newsreader and manually
> search for the control article.
> The message-ID of the last one (December, 15th) is:
> <cmsg-20101215160002$03...@isc.org>
>
That would be impossible with readers.conf containing:
access "authenticated" {
users: "*"
newsgroups: "*,!junk,!control,!control.*"
}
However I can read that particular M-ID with "grephistory
'<cmsg-20101215160002$03...@isc.org>' | sm | less"
I see it is just the "newsgroups" listing for the Big-8 hierarchies
involved with the checkgroups.
>> I can't find any archived checkgroups files at the ISC site. Are they
>> (still) there?
>>
>
> For checkgroups, have a look at:
> ftp://ftp.isc.org/pub/usenet/control/other.ctl
>
That is handy to have for reference. Thanks.
I also use: ftp://ftp.isc.org/pub/usenet/CONFIG/LOGS/
>> As a side note, I also can no longer locate the history (charter
>> messages) which created the groups. Did the ISC remove this?
>>
>
> I do not know. Do you speak about newgroup control messages?
> Or monthly charters?
>
Well, I was trying to find what I could not name. ;-)
This might be it: ftp://ftp.isc.org/pub/usenet/control
N.B. for Russ (or whomever), the following has an XML Parsing Error:
ftp://ftp.isc.org/pub/usenet/control-old
>> Even worse, the -A option in actsync will not authenticate. I only
>> get one local group ("info") which doesn't require passwd
>> authentication. Perhaps my syntax is not correct, but I've tried the
>> command various ways. I'm not prompted for authentication, and
>> supplying a username:password combination somewhere in the command
>> causes the command to fail entirely.
>>
>
> Having looked at the source code, actsync uses the file named
> passwd.nntp if you ask for authentication with the -A flag.
> Be sure to keep this file not readable by everyone (but only
> the news user for instance).
>
> http://www.eyrie.org/~eagle/software/inn/docs/passwd.nntp.html
>
>
> The documentation for actsync is not clear enough and does not
> mention passwd.nntp; I will soon add a clarification. Thanks for
> having pointed out the issue.
>
I figured this out by attempting to use getlist, and then when it
failed, I RTFM for getlist(1)! Then I added the authentication to the
passwd.nntp file and it worked.
I believe you could use the getlist -A option description verbatim for
actsync's man page:
OPTIONS
-A Try to authenticate using the username and password
information in
passwd.nntp(5) before issuing the LIST command.
>> Then there is another problem: the docheckgroups output cannot be
>> piped or directed into mod-active. Or more accurately, it doesn't
>> work for me. No errors, just active is not modified.
>>
>
> It is very weird. I do not understand why.
>
Neither do I, but I will keep trying (on a new test server I'll create).
Christmas season has just been rather busy with eating, family from
out-of-town, eating, helping my wife with her new digital camera,
eating, etc. Later this week I may get a chance to take a look at it
again. (Each "eating" task mentioned above is, of course, followed by a
"napping" period!) ;-)
I managed to get all the Usenet groups added to all working servers in
less than an hour, but had to first create a file with only the "ctlinnd
newgroup {newsgroup}" lines, then use it for the mod-active input.
On one INN server (via ssh) I used "wget
ftp://ftp.isc.org/pub/usenet/CONFIG/newsgroups" to download the ISC
newsgroups file.
Then: "cat newsgroups.1 | docheckgroups > newsgroups_changes"
Then copied only the "ctlinnd newgroup {newsgroup}" lines and pasted
them into a new file "ctlinnd_newgroup_grouplist"
Then: "mod-active < ctlinnd_newgroup_grouplist"
That ran very fast, about five seconds for the complete alt* hierarchy
(minus alt.binaries and a few other garbage groups).
I copied the "ctlinnd_newgroup_grouplist" file to all the other servers
and used it for the mod-active input.
At least that is how I remember it all from a week ago, followed by too
much food. ;-)
> Yet, something is wrong somewhere if it does not work directly…
Well, if you want my "humble opinion," I personally would have preferred
to have a utility to work on the ISC's (or another server's) active
file, and not mess with the newsgroups file.
After all, it is the active file that contains the important controlling
information for a server. The newsgroups file is just an explanation of
some of the groups, and not very well authored nor maintained. Some
newsreaders don't support it, and some news servers don't even provide it.
If there were an easy way to modify the docheckgroup utility to use the
active file instead of the newsgroups file, perhaps by adding a new
option so it wouldn't be broken for those who do depend on checkgroups
control messages, then all someone wishing to maintain their newsgroups
file would need to do would be to occasionally download a fresh copy to
replace the old one.
The ISC's active file of 45,959 newsgroups, after something like "awk
'{print $1}' active > active_stripped" is 1 MB compared to 2.5 MB for
the ISC's newsgroups file. My ISP has 108,811 newsgroups.
Just my thoughts.... ;-)
Russ Allbery
> Well, if you want my "humble opinion," I personally would have preferred
> to have a utility to work on the ISC's (or another server's) active
> file, and not mess with the newsgroups file.
That's what actsync does. See its man page, particlarly the top of the
EXAMPLES section. You can configure it to do more complicated things,
like only synchronize certain groups and exclude some groups or
hierarchies. You can use it either with a downloaded active file or have
it connect to a remote NNTP server and use LIST.
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Please post questions rather than mailing me directly.
<http://www.eyrie.org/~eagle/faqs/questions.html> explains why.
John F. Morse
> "John F. Morse" <jo...@example.invalid> writes:
>
>> Well, if you want my "humble opinion," I personally would have preferred
>> to have a utility to work on the ISC's (or another server's) active
>> file, and not mess with the newsgroups file.
>>
>
> That's what actsync does. See its man page, particlarly the top of the
> EXAMPLES section. You can configure it to do more complicated things,
> like only synchronize certain groups and exclude some groups or
> hierarchies. You can use it either with a downloaded active file or have
> it connect to a remote NNTP server and use LIST.
>
Now that I found the passwd.nntp file was required, I might be able to
use actsync. ;-)
However (from my defective memory when I used actsync months/years ago),
doesn't actsync pound the server with sequential ctlinnd commands
instead of the mod-active batch with the server temporarily paused?
Mod-active is very smooth in its operation, but what about actsync?
Then there is the problem of intermediate editing to delete silly alt.*
groups which would not be possible with actsync alone. A file could be
created for editing, then injected with mod-active.
Jeffrey M. Vinocur
John F. Morse <dont.bother> wrote:
>
>However (from my defective memory when I used actsync months/years ago),
>doesn't actsync pound the server with sequential ctlinnd commands
>instead of the mod-active batch with the server temporarily paused?
>
>Mod-active is very smooth in its operation, but what about actsync?
>
>Then there is the problem of intermediate editing to delete silly alt.*
>groups which would not be possible with actsync alone. A file could be
>created for editing, then injected with mod-active.
Check the -o option (for actsync) to answer both of these
questions.
--
Jeffrey M. Vinocur
je...@litech.org
Julien ÉLIE
>> You could open control.checkgroups in your newsreader and manually
>> search for the control article.
>
> That would be impossible with readers.conf containing:
>
> access "authenticated" {
> users: "*"
> newsgroups: "*,!junk,!control,!control.*"
> }
Just add another access block for your own username “john” (or whatever
it is when you authenticate on your news server), with no restriction on
the newsgroups.
Or, more straight forward, allow everybody to read control.checkgroups.
No harm done!
> I believe you could use the getlist -A option description verbatim for
> actsync's man page:
Yes, thanks.
> The newsgroups file is just an explanation of
> some of the groups, and not very well authored nor maintained.
And also with different encodings…
--
Julien ÉLIE
« La perfection est atteinte non pas lorsqu'il n'y a plus rien à
ajouter, mais lorsqu'il n'y a plus rien à retirer. »
(Saint-Exupéry)
John F. Morse
Hi Jeff. Haven't seen you around in awhile. Hope you are in good health.
I have checked the actsync -o options, probably six times this month. I
didn't see the answer to my "two" questions.
None of the -o a... options are material because they set the output
format using the high and low watermarks. I'm only interested in the
first field (newsgroup name) and the fourth field (if it isn't "y").
Comparing my active with another, for instance using "diff -y
--suppress-common-lines ..." requires the watermarks to be stripped off
with "awk '{print $1, $4}' active > active1.txt" or nearly every line
will be different.
The statement about the "a" options: "allow one to format new active
file instead of producing ctlinnd commands" I read literally as a "new
active file." My task was to add to an "existing" active file. I
certainly don't want to overwrite the active file and cause the
watermarks to all be zeroed out (unless merge is used). But I still
wanted to examine the differences.
The -o x... options are dangerous because I don't get to see what will
happen beforehand. I suppose they could be used (after editing, testing,
etc.) to actsync multiple servers in one installation so they are all
identical, but not a good idea for syncing even one server against a
remote peer or file.
The -o c option prints the ctlinnd commands for examination. Then you
still need to do something with them. If you feed them into INN, then
there is the overrun problem I mentioned earlier, where some form of
handshaking is needed to keep the incoming data from getting garbled.
Using -o x pause (see the -z flag) can help in this situation of
overrunning the server's ability to handle the incoming ctlinnd commands
fast enough, but it introduces a very long run time when there are
hundreds or thousands of groups to add. Adding just 10,000 newgroups
with a four-second pause would take over 11 hours to run -- per server.
40,000 new groups would be two days -- per server. Or 12 hours with a
one-second delay.
Throttling the server would allow ctlinnd newgroup commands to process
faster, but then you lose a server for an extended period.
So reading actsync(8), the -o option as you suggest, does answer my
question(s) and verifies my memory that actsync will pound the server
with sequential ctlinnd newgroup commands, possibly garbling the
incoming newsgroup names if no -z is used, generating thousands of lines
of news.notice log, plus a huge Daily Report listing every added group.
I experienced this on one test server when I added the complete Big-8 in
2009, then decided it was better to just edit the active file by
"pasting in" the new groups. Certainly not elegant, but it was fast, and
it worked.
Even thought the missing actsync -A option description was solved by
reading getlist(1) -A description, I don't see how actsync could replace
the use of mod-active using the edited output from docheckgroups.
Mod-active is very slick the way it works. Fast and clean. It's too bad
mod-active will not work for me with the output of docheckgroups. This
may be a HUA problem, and I will dig into it again.
Adding a huge list of newgroups is no longer an issue since I now have
all the ISC groups added to all the servers. It may be an issue someday
in the future, should I ever need to add many groups, but I can refer to
my recent notes just like I did with those I made years ago (2005) on
actsync. Those old notes were of little help because of added
authentication (the -A problem), plus the servers are INN 2.5.2 on
Squeeze and INN 2.4.5 on Lenny, while the old notes were made on INN
2.4.3 on Sarge, without authentication and without SSL/TLS.
IOW, I built too many hoops to jump through. ;-)
Thanks for your suggestion anyway, Jeff. It made me go verify and reread
some things, and makes me want to set up another server just to see if I
can't get something working which others say does work for them.
I'm not a quitter, but I will sometimes get a "make-do" finished, then
move a problem to the back burner for another day due to other
more-pressing issues. Problem is, when that day arrives, I've forgotten
a lot, and sometimes software upgrades have broken something my notes
show as working in the past.
John F. Morse
>>> You could open control.checkgroups in your newsreader and manually
>>> search for the control article.
>>
>> That would be impossible with readers.conf containing:
>>
>> access "authenticated" {
>> users: "*"
>> newsgroups: "*,!junk,!control,!control.*"
>> }
>
> Just add another access block for your own username “john” (or
> whatever it is when you authenticate on your news server), with no
> restriction on the newsgroups.
>
> Or, more straight forward, allow everybody to read
> control.checkgroups. No harm done!
I had considered that, but probably would use something like
"administrator" or "newsmaster" so my regular "john" account would have
the same privileges as all authenticated users. This would allow me to
see what they see so I could take care of any possible abnormalities.
I do have a "localhost" but there has not been a keyboard nor monitor on
the servers since ~2005. I don't believe connecting via ssh would count
as "stdin" nor "localhost" -- but it might work if I installed a
text-only reader like tin or slrn (gnus for those who will surely
suggest such monstrosities). ;-)
auth "global" {
hosts: "*"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f
/var/lib/news/passlist"
default: "<anonymous>"
}
auth "localhost" {
hosts: "stdin,localhost"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f
/var/lib/news/passlist"
default: "<localhost>"
}
access "authenticated" {
users: "*"
newsgroups: "*,!junk,!control,!control.*"
}
access "special" {
users: "<localhost>"
newsgroups: "*"
access: "RPNAL"
}
access "public" {
users: "<anonymous>"
read: "local.info"
}
Perhaps I should create an auth group using a LAN IP netblock and/or the
domain? On second thought, what do you think about simply adding this
to the hosts in the auth "localhost" group, such as:
auth "localhost" {
--- hosts: "stdin,localhost"
+++ hosts: "stdin,localhost,*.my.net,192.168.*"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f
/var/lib/news/passlist"
default: "<localhost>"
}
Then I wouldn't need to install a newsreader on the server(s) and
connect via ssh.
Russ Allbery
> I have checked the actsync -o options, probably six times this month. I
> didn't see the answer to my "two" questions.
actsync -o c plus mod-active does exactly what I understood you to want to
do. It outputs only the changes based on the remote active file as a set
of ctlinnd commands that you can then review and modify as you choose.
Then you can pipe the results through mod-active to avoid hammering your
server.
> The -o c option prints the ctlinnd commands for examination. Then you
> still need to do something with them. If you feed them into INN, then
> there is the overrun problem I mentioned earlier, where some form of
> handshaking is needed to keep the incoming data from getting garbled.
That's what mod-active is for.
Julien ÉLIE
>> Just add another access block for your own username “john” (or
>> whatever it is when you authenticate on your news server), with no
>> restriction on the newsgroups.
>
> I had considered that, but probably would use something like
> "administrator" or "newsmaster" so my regular "john" account would have
> the same privileges as all authenticated users. This would allow me to
> see what they see so I could take care of any possible abnormalities.
You could also add such a block for your own user “john”, and comment
it. Only when you need it, just uncomment the block, do whatever you
want to do, and recomment the block afterwards.
> Perhaps I should create an auth group using a LAN IP netblock and/or the
> domain? On second thought, what do you think about simply adding this to
> the hosts in the auth "localhost" group, such as:
>
>
> auth "localhost" {
> --- hosts: "stdin,localhost"
> +++ hosts: "stdin,localhost,*.my.net,192.168.*"
> auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f /var/lib/news/passlist"
> default: "<localhost>"
> }
I think 192.168.* should be written 192.168.0.0/16, and that it is OK
for *.my.net.
Jeffrey M. Vinocur
Russ Allbery <r...@stanford.edu> wrote:
>"John F. Morse" <jo...@example.invalid> writes:
>
>> I have checked the actsync -o options, probably six times this month. I
>> didn't see the answer to my "two" questions.
>
>actsync -o c plus mod-active does exactly what I understood you to want to
>do. It outputs only the changes based on the remote active file as a set
>of ctlinnd commands that you can then review and modify as you choose.
>Then you can pipe the results through mod-active to avoid hammering your
>server.
Whew! (That's what I thought I was recommending, and then John's
reply made me worry that I was going loopy.)
So thanks for reassuring me, and happy 2011.
Jeffrey M. Vinocur
John F. Morse <dont.bother> wrote:
>
>the servers since ~2005. I don't believe connecting via ssh would count
>as "stdin" nor "localhost" -- but it might work if I installed a
>text-only reader like tin or slrn (gnus for those who will surely
>suggest such monstrosities). ;-)
Certainly a unix-based reader would do it. If you connect with
your remote client via an ssh tunnel, that should work as well,
but it's a little silly considering the other options as you say.
>Perhaps I should create an auth group using a LAN IP netblock and/or the
>domain? On second thought, what do you think about simply adding this
>to the hosts in the auth "localhost" group [...]
You could do that, if you are comfortable with whatever access
you are making available being open to everyone in the netblock
or domain.
Or else it's easy enough to just have two user/pass accounts for
yourself in your password file, and then special-case one of the
usernames to have different privileges.
John F. Morse
Perhaps my reply was encrypted?
How about a Happy MMXI to you? ;-)
What I was meaning to say was Russ didn't actually answer the two
questions that I had.
Well, he may be "answered" but in doing so the answer contained some
confusion.
I'm referring to Russ's use of "pipe the results" above. He mentions
doing what I have already done -- get the changes with actsync -o c, and
then edit them.
But you cannot "pipe" ("|") the result from actsync into mod-active if
you want to do the editing. You can pipe the output of the edited
actsync output into mod-active (or redirect the mod-active input from
the edited file).
I understand what he in implying, and guess I'm just nit-picking the
term, looking for a way to totally automate the news server so I can go
on vacation. ;-)
My concern was actsync could not be directly piped into mod-active with
any positive result. It did nothing.
I had to run actsync, redirecting the output to a file, and then remove
everything but the actual ctlinnd newgroup command lines before
mod-active would act on it. That is what I believe Russ is suggesting.
Since I already had this "answer" I didn't feel I had a question to be
answered, and improperly stated this, confusing everybody. Sorry for that.
I don't know if the latest discussion by Russ on the 1.5.0 release of
control-archive, specifically the "Fix broken checkgroups processing..."
has any connection to my problem with checkgroups or not. Perhaps Russ
found this bug while trying to recreate my reported difficulties?
Also possibly connected was considerable checkgroups, plus rmgroup and
newgroup control message activity in the same newsgroups in the past
couple of days. Were these possibly forged rmgroup messages?
If these removals were just errors by news administrators, they caused
the removal of several groups, including their watermarks. Then these
groups were replaced with zeroed watermarks when the groups were
recreated, and will result in newsreaders not seeing articles until the
new watermarks catch up to the various newsreaders (newsrc) databases.
John F. Morse
> "John F. Morse" <jo...@example.invalid> writes:
>
>> I have checked the actsync -o options, probably six times this month. I
>> didn't see the answer to my "two" questions.
>>
>
> actsync -o c plus mod-active does exactly what I understood you to want to
> do. It outputs only the changes based on the remote active file as a set
> of ctlinnd commands that you can then review and modify as you choose.
> Then you can pipe the results through mod-active to avoid hammering your
> server.
>
Exactly, and that is what I am having to do.
I may have earlier confused actsync with docheckgroups output, where the
docheckgroups man page states the docheckgroups output is usually piped
into mod-active.
Using docheckgroups is probably something I don't need since I can
occasionally rsync a fresh newsgroups file to all servers.
There are frequent checkgroups messages arriving though, and their
results vary between 2.4.5 and 2.5.2 servers, and one test server where
I don't have the pgp so I will receive Daily Report indications of
control group activity.
>> The -o c option prints the ctlinnd commands for examination. Then you
>> still need to do something with them. If you feed them into INN, then
>> there is the overrun problem I mentioned earlier, where some form of
>> handshaking is needed to keep the incoming data from getting garbled.
>>
>
> That's what mod-active is for.
>
Again, exactly, and thanks for verifying that I am using it in the
proper way. It is very efficient.
And forgive me for mixing up what you and Jeff stated. My mind is
starting to feel like Hal's ... slipping away. ;-)
John F. Morse
> In article <ifcqt0$ig8$1...@n102.xanadu-bbs.net>,
> John F. Morse <dont.bother> wrote:
>
>> I do have a "localhost" but there has not been a keyboard nor monitor on
>> the servers since ~2005. I don't believe connecting via ssh would count
>> as "stdin" nor "localhost" -- but it might work if I installed a
>> text-only reader like tin or slrn (gnus for those who will surely
>> suggest such monstrosities). ;-)
>>
>
> Certainly a unix-based reader would do it. If you connect with
> your remote client via an ssh tunnel, that should work as well,
> but it's a little silly considering the other options as you say.
>
That is my opinion too.
>> Perhaps I should create an auth group using a LAN IP netblock and/or the
>> domain? On second thought, what do you think about simply adding this
>> to the hosts in the auth "localhost" group [...]
>>
>
> You could do that, if you are comfortable with whatever access
> you are making available being open to everyone in the netblock
> or domain.
>
Just me (my wife wouldn't have a clue).
> Or else it's easy enough to just have two user/pass accounts for
> yourself in your password file, and then special-case one of the
> usernames to have different privileges.
>
Well, ... I find a user "newsmaster" in the passwd file, so I must have
done something similar in the old days. The password is of course
encrypted, so I'll need to look around for the plaintext version (it
might be saved on an old workstation I used years ago).
I may need to return here for further help with readers.conf, since I
remember years ago it was you who helped me finally get it set up for
passwords and separate access privileges.
John F. Morse
>>> Just add another access block for your own username “john” (or
>>> whatever it is when you authenticate on your news server), with no
>>> restriction on the newsgroups.
>>
>> I had considered that, but probably would use something like
>> "administrator" or "newsmaster" so my regular "john" account would have
>> the same privileges as all authenticated users. This would allow me to
>> see what they see so I could take care of any possible abnormalities.
>
> You could also add such a block for your own user “john”, and comment
> it. Only when you need it, just uncomment the block, do whatever you
> want to do, and recomment the block afterwards.
>
>> Perhaps I should create an auth group using a LAN IP netblock and/or the
>> domain? On second thought, what do you think about simply adding this to
>> the hosts in the auth "localhost" group, such as:
>>
>>
>> auth "localhost" {
>> --- hosts: "stdin,localhost"
>> +++ hosts: "stdin,localhost,*.my.net,192.168.*"
>> auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f /var/lib/news/passlist"
>> default: "<localhost>"
>> }
>
> I think 192.168.* should be written 192.168.0.0/16, and that it is OK
> for *.my.net.
Nothing I tried would work, including just the hostname, the FQDN, just
the domain, the *my.net, the full IP, etc.
Something isn't right in readers.conf therefore I'll need to keep
fiddling with it until it works. It's been a long time so perhaps after
several more trips through the man page will light my fire? ;-)
John F. Morse
John F. Morse wrote:
> Jeffrey M. Vinocur wrote:
>
>> Or else it's easy enough to just have two user/pass accounts for
>> yourself in your password file, and then special-case one of the
>> usernames to have different privileges.
>
> Well, ... I find a user "newsmaster" in the passwd file, so I must
> have done something similar in the old days. The password is of course
> encrypted, so I'll need to look around for the plaintext version (it
> might be saved on an old workstation I used years ago).
>
> I may need to return here for further help with readers.conf, since I
> remember years ago it was you who helped me finally get it set up for
> passwords and separate access privileges.
I located the "newsmaster" password, plus a couple of other user
accounts that I created for testing back around 2005, when I had certain
local groups restricted to limited users.
I'm missing something here.... Either I get no access to the control
groups as user newsmaster, or else any "user" can access everything when
I put a CIDR netblock IP in the "hosts" of the second auth block
following the "stdin,localhost" entry. Here, "any user" also includes a
casual user without any authinfo being supplied. Normally
unauthenticated users only get the local.info group.
I don't think I can use an IP nor domain in an auth block and still have
a different access for "newsmaster" and for "john" and for an
unauthenticated user, when they are all on the same LAN.
If I remove the authenticated access block's negation bangs from
!control and !control.* then everybody can see the control groups, which
would be expected but not desired.
I've moved the special access block up and down, added (then completely
replaced) users: "<localhost>" with users: "<newsmaster>" but nothing I
do seems to work. That block seems to be "untouchable."
Below is the actual readers.conf file that you provided around 2005,
minus a long string of usernames which were replaced by the asterisk in
the authenticated access block.
Could you please quote it back in its entirety, and then enter what I
would need to add or change to allow the user "newsmaster" to access all
groups with the RPNAL access privileges, and from only my CIDR netblock?
(Julien suggested 192.168.0.0/16 but I believe that CIDR should be
192.168.0.0/32.)
Using a "diff -U1" format (using "+++" and "---") would make the changes
easier to locate and see what was done to make it work correctly.
TIA, and I'll owe you a dinner. ;-)
auth "global" {
hosts: "*"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f
/var/lib/news/passlist"
default: "<anonymous>"
}
auth "localhost" {
hosts: "stdin,localhost"
auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f
/var/lib/news/passlist"
default: "<localhost>"
}
access "authenticated" {
users: "*"
newsgroups: "*,!junk,!control,!control.*"
}
access "special" {
users: "<localhost>"
newsgroups: "*"
access: "RPNAL"
}
access "public" {
users: "<anonymous>"
read: "local.info"
}
--
Russ Allbery
> I'm referring to Russ's use of "pipe the results" above. He mentions
> doing what I have already done -- get the changes with actsync -o c, and
> then edit them.
> But you cannot "pipe" ("|") the result from actsync into mod-active if
> you want to do the editing. You can pipe the output of the edited
> actsync output into mod-active (or redirect the mod-active input from
> the edited file).
> I understand what he in implying, and guess I'm just nit-picking the
> term, looking for a way to totally automate the news server so I can go
> on vacation. ;-)
Er, well, there's no way that you can totally automate the news server if
you want to manually edit what groups are added!
You'll need to write ignore rules for actsync to ignore the groups that
you don't want to add.
> My concern was actsync could not be directly piped into mod-active with
> any positive result. It did nothing.
That doesn't make any sense. Either it's a bug or you're doing something
wrong.
> I had to run actsync, redirecting the output to a file, and then remove
> everything but the actual ctlinnd newgroup command lines before
> mod-active would act on it.
What else was in the output of actsync -o c other than ctlinnd commands?
You're apparently having some sort of bizarre problem that I don't
understand.
> I don't know if the latest discussion by Russ on the 1.5.0 release of
> control-archive, specifically the "Fix broken checkgroups processing..."
> has any connection to my problem with checkgroups or not. Perhaps Russ
> found this bug while trying to recreate my reported difficulties?
No, this doesn't have anything to do with INN.
> Also possibly connected was considerable checkgroups, plus rmgroup and
> newgroup control message activity in the same newsgroups in the past
> couple of days. Were these possibly forged rmgroup messages?
I have no idea what you're talking about. Are you basing this on the
reports in news.lists.misc? If so, this didn't have anything to do with
rmgroup or newgroup control messages; it was a bug in checkgroups
processing.
> If these removals were just errors by news administrators, they caused
> the removal of several groups, including their watermarks. Then these
> groups were replaced with zeroed watermarks when the groups were
> recreated, and will result in newsreaders not seeing articles until the
> new watermarks catch up to the various newsreaders (newsrc) databases.
Yes, that always happens when a newsgroup is removed and then recreated in
INN.
Julien ÉLIE
> I had to run actsync, redirecting the output to a file, and then remove
> everything but the actual ctlinnd newgroup command lines before
> mod-active would act on it. That is what I believe Russ is suggesting.
> Since I already had this "answer" I didn't feel I had a question to be
> answered, and improperly stated this, confusing everybody. Sorry for that.
Could you please tell us what are the lines you are removing?
I for instance have:
19:36 news@trigo ~/work/inn/trunk% actsync news.lacave.net -p 1 -o c
actsync: line 607 <trigofacile.test.υτφ8> from localhost has a bad
newsgroup name
actsync: line 608 <dk.test.utf8-æøå> from localhost has a bad newsgroup name
ctlinnd rmgroup af.philo
ctlinnd rmgroup af.politique
ctlinnd rmgroup can.francais
[...]
but it still works with mod-active because unknown lines (with no
ctlinnd command) are silently ignored.
> Also possibly connected was considerable checkgroups, plus rmgroup and
> newgroup control message activity in the same newsgroups in the past
> couple of days. Were these possibly forged rmgroup messages?
It was an issue with control-active; and owing to your synchronizing
your active file with the one provided by ftp.isc.org, you were affected
with the removal and then the re-adding of these newsgroups.
> If these removals were just errors by news administrators, they caused
> the removal of several groups, including their watermarks. Then these
> groups were replaced with zeroed watermarks when the groups were
> recreated, and will result in newsreaders not seeing articles until the
> new watermarks catch up to the various newsreaders (newsrc) databases.
An argument in favour of the experimental LIST REMOVALS command along
with the “r” status flag :-)
--
Julien ÉLIE
« N'as-tu jamais fait ces rêves, Neo, qui ont l'air plus vrais que la
réalité ? Si tu étais incapable de sortir de l'un de ces rêves,
comment ferais-tu la différence entre le monde du rêve et le monde
réel ? » (Morpheus, _Matrix_)
Julien ÉLIE
> There are frequent checkgroups messages arriving though, and their
> results vary between 2.4.5 and 2.5.2 servers, and one test server where
> I don't have the pgp so I will receive Daily Report indications of
> control group activity.
That is to say? Do you have different ctlinnd commands between 2.4.5
and 2.5.2 servers?
(In case you are speaking about the commented lines in docheckgroups
output, it is normal — but actual ctlinnd commands should be the same.)
Russ Allbery
> I for instance have:
> 19:36 news@trigo ~/work/inn/trunk% actsync news.lacave.net -p 1 -o c
> actsync: line 607 <trigofacile.test.υτφ8> from localhost has a bad
> newsgroup name
> actsync: line 608 <dk.test.utf8-æøå> from localhost has a bad newsgroup name
> ctlinnd rmgroup af.philo
> ctlinnd rmgroup af.politique
> ctlinnd rmgroup can.francais
> [...]
> but it still works with mod-active because unknown lines (with no ctlinnd
> command) are silently ignored.
Oh, hm. Shouldn't the error messages be going to standard error and hence
not into a pipe to mod-active?
Julien ÉLIE
> Could you please quote it back in its entirety, and then enter what I
> would need to add or change to allow the user "newsmaster" to access all
> groups with the RPNAL access privileges, and from only my CIDR netblock?
> (Julien suggested 192.168.0.0/16 but I believe that CIDR should be
> 192.168.0.0/32.)
With /32, it would mean only one IP: 192.168.0.0.
However, this IP is the one of the route to your subnet…
> access "special" {
> users: "<localhost>"
> newsgroups: "*"
> access: "RPNAL"
> }
users: "<localhost>,john"
should do the trick here, I believe.
P.-S.: An interesting question would be how to have a username which
contains a comma. Impossible to put here I think (because of our
parsing expecting a comma-separated list).
--
Julien ÉLIE
« – Rendre compte ? Mais nous n'y sommes pas allés et nous n'avons
rien vu ! Et puis Jules César a dit…
– Je ne sais pas ce que Jules César a dit, mais ne pas y aller et
ne pas voir, c'est le meilleur moyen de ne pas être vaincus ! »
(Olibrius)
Russ Allbery
> P.-S.: An interesting question would be how to have a username which
> contains a comma. Impossible to put here I think (because of our parsing
> expecting a comma-separated list).
Doesn't the parsing use wildmat matching? If so, I think it could just
use uwildmat() (rather than splitting on comma and using uwildmat_simple()
if it currently does -- I didn't check), and then \, would do the right
thing.
Julien ÉLIE
>> 19:36 news@trigo ~/work/inn/trunk% actsync news.lacave.net -p 1 -o c
>
> Oh, hm. Shouldn't the error messages be going to standard error and hence
> not into a pipe to mod-active?
Error messages are already going to standard error. I have just
checked, and also seen that the -q flag can silent them.
% actsync news.lacave.net -p 1 -o c -q 12
It removes the warnings from both the local server and the remote server.
Only ctlinnd commands are now output.
John, did you try that command?
Julien ÉLIE
>> P.-S.: An interesting question would be how to have a username which
>> contains a comma. Impossible to put here I think (because of our parsing
>> expecting a comma-separated list).
>
> Doesn't the parsing use wildmat matching? If so, I think it could just
> use uwildmat() (rather than splitting on comma and using uwildmat_simple()
> if it currently does -- I didn't check), and then \, would do the right
> thing.
I did not check.
Now done, and :-)
First:
CompressList(curaccess->users);
Well, not good for usernames containing a space (this function removes
CR, LF and SP). Allowed by RFC 4643 if an implementation wishes to.
Then:
NGgetlist(&list, cp);
if (PERMmatch(list, user)) {
syslog(L_TRACE, "%s match_user %s %s", Client.host,
PERMuser, access_realms[i]->users);
with PERMmatch using uwildmat() but:
bool
NGgetlist(char ***argvp, char *list)
{
char *p;
for (p = list; *p; p++)
if (*p == ',')
*p = ' ';
return Argify(list, argvp) != 0;
}
I see that these functions are used at other places too for parsing
readers.conf; well, I believe it should be better to change a bit this
behaviour. Yet, I wonder how many people actually use commas or spaces
in usernames (maybe a space would be more frequent).
Russ Allbery
> I see that these functions are used at other places too for parsing
> readers.conf; well, I believe it should be better to change a bit this
> behaviour. Yet, I wonder how many people actually use commas or spaces
> in usernames (maybe a space would be more frequent).
It's probably fairly unlikely. But it would be nice to get rid of all
those duplicate implementations of uwildmat() anyway. I had a sneaking
suspicion there were several places that were doing their own splitting on
commas, since originally the wildmat functions didn't handle
comma-separated pattern lists.
John F. Morse
> "John F. Morse" <jo...@example.invalid> writes:
>
>
>> I understand what he in implying, and guess I'm just nit-picking the
>> term, looking for a way to totally automate the news server so I can go
>> on vacation. ;-)
>>
>
> Er, well, there's no way that you can totally automate the news server if
> you want to manually edit what groups are added!
>
> You'll need to write ignore rules for actsync to ignore the groups that
> you don't want to add.
>
"Automate" in this case meant to allow the control messages to keep the
server updated. That should (is) possible.
The editing was done when adding thousands of groups, mainly to
eliminate the alt.* garbage. That is finished, at least for the current
server farm.
The control.ctl has the proper flags to keep the alt.* hierarchy from
being destroyed:
newgroup:*:alt.*:drop
newgroup:group...@isc.org:alt.*:drop
newgroup:tale@*uu.net:alt.*:drop
rmgroup:*:alt.*:drop
>> My concern was actsync could not be directly piped into mod-active with
>> any positive result. It did nothing.
>>
>
> That doesn't make any sense. Either it's a bug or you're doing something
> wrong.
>
I guess I'll never find out then, since I was not able to get it to work
as I thought it should work.
However if I would have discovered earlier that any verbosity above -v0
would cause the unneeded text, we wouldn't be having this discussion.
IOW, this causes the output to contain more than mod-active would swallow:
actsync -o c -z 0 -m -v 4 -p 0 /var/lib/news/active ./active-new
>> I had to run actsync, redirecting the output to a file, and then remove
>> everything but the actual ctlinnd newgroup command lines before
>> mod-active would act on it.
>>
>
> What else was in the output of actsync -o c other than ctlinnd commands?
> You're apparently having some sort of bizarre problem that I don't
> understand.
>
Try actsync with -v larger than zero. ;-)
>> I don't know if the latest discussion by Russ on the 1.5.0 release of
>> control-archive, specifically the "Fix broken checkgroups processing..."
>> has any connection to my problem with checkgroups or not. Perhaps Russ
>> found this bug while trying to recreate my reported difficulties?
>>
>
> No, this doesn't have anything to do with INN.
>
>
>> Also possibly connected was considerable checkgroups, plus rmgroup and
>> newgroup control message activity in the same newsgroups in the past
>> couple of days. Were these possibly forged rmgroup messages?
>>
>
> I have no idea what you're talking about. Are you basing this on the
> reports in news.lists.misc? If so, this didn't have anything to do with
> rmgroup or newgroup control messages; it was a bug in checkgroups
> processing.
>
There was a lot of activity in the past 3-4 days. Several control
messages, including checkgroups, which removed a few groups, and were
followed later by control messages creating the same groups.
You can see the activity here:
ftp://ftp.isc.org/pub/usenet/CONFIG/LOGS/log.2011-01
The Daily Reports indicated this and made me aware of the activity, and
it was verified by running the "wc newsgroups active" command on each
server.
The result was two INN 2.5.2. servers had identical wc results, while
two INN 2.4.5 servers had identical results but were different than the
INN 2.5.2 servers. That seems odd.
A third INN 2.4.5 testing server does not have the same PGP capability
(no current keys) so it had yet another set of wc totals. This is not a
problem since it is only a testing server, and quite frankly, I
appreciate it reporting activity the other servers complete without any
report.
I simply was curious if your control-archive work in fixing broken
checkgroups might have had something to do with this activity over the
past few days, especially on January 2 per the log.2011-01. The common
denominator here is "checkgroups." ;-)
And if not, why the oddity of groups being removed and then added within
a few hours or the next day. This seems abnormal, possibly a rogue
attack, so I mentioned it.
>> If these removals were just errors by news administrators, they caused
>> the removal of several groups, including their watermarks. Then these
>> groups were replaced with zeroed watermarks when the groups were
>> recreated, and will result in newsreaders not seeing articles until the
>> new watermarks catch up to the various newsreaders (newsrc) databases.
>>
>
> Yes, that always happens when a newsgroup is removed and then recreated in
> INN.
>
Of course, but that was not a question. It was a statement of the extra
work involved and the upcoming problem newsreaders will have until their
newsrc catches up.
Russ Allbery
> "Automate" in this case meant to allow the control messages to keep the
> server updated. That should (is) possible.
Oh, okay. Yes, you can either do that with actsync running in a fully
automated mode, or by processing and authenticating control messages
yourself. For doing this sort of routine update, I would just use actsync
-o x. The pauses should keep it from doing weird things to the server.
(Although ideally we should figure out what's buggy about sending ctlinnd
commands as fast as the server can handle them and fix that problem.)
> However if I would have discovered earlier that any verbosity above -v0
> would cause the unneeded text, we wouldn't be having this discussion.
Ah!
Sorry, that didn't even occur to me. Yes, that would be the problem.
> There was a lot of activity in the past 3-4 days. Several control
> messages, including checkgroups, which removed a few groups, and were
> followed later by control messages creating the same groups.
> You can see the activity here:
> ftp://ftp.isc.org/pub/usenet/CONFIG/LOGS/log.2011-01
Ah, yes. This wasn't real activity; it was a bug in the software that
processes control messages to maintain the ISC active file. I'm sorry
about that. That's fixed now, but indeed, some groups would have
temporarily disappeared and then reappeared due to that bug.
> I simply was curious if your control-archive work in fixing broken
> checkgroups might have had something to do with this activity over the
> past few days, especially on January 2 per the log.2011-01. The common
> denominator here is "checkgroups." ;-)
Yes, indeed, you're right. :)
> And if not, why the oddity of groups being removed and then added within
> a few hours or the next day. This seems abnormal, possibly a rogue
> attack, so I mentioned it.
Yeah, several other people thought it was a rogue attack as well, and it
does sort of look like that. But it was just a software bug with
unfortunately consequences. Thankfully, it only affected one group per
hierarchy, and for the most part those were low-traffic announce groups,
so the impact on clients should hopefully not be too dire.
Julien ÉLIE
> However if I would have discovered earlier that any verbosity above
> -v0 would cause the unneeded text, we wouldn't be having this
> discussion.
>
> IOW, this causes the output to contain more than mod-active would
> swallow:
>
> actsync -o c -z 0 -m -v 4 -p 0 /var/lib/news/active ./active-new
I still do not understand.
The lines added by -v 4 do not contain the word “ctlinnd” and are
therefore ignored.
while (<>) {
if (/^\s*\S*ctlinnd newgroup (\S+) (\S+)/) {
$toadd{$1} = $2;
$changes++;
} elsif (/^\s*\S*ctlinnd rmgroup (\S+)/) {
$eval .= " next if \$group eq '$1';\n";
$changes++;
} elsif (/^\s*\S*ctlinnd changegroup (\S+) (\S+)/) {
$eval .= " s/ \\S+\$/ $2/ if \$group eq '$1';\n";
$changes++;
}
}
I see lines like:
actsync: STATUS: using default server: localhost
actsync: STATUS: obtaining active file from localhost
actsync: STATUS: read 608 groups, will merge 606 groups from localhost
actsync: STATUS: found 2 line errors from localhost
actsync: STATUS: obtaining active file from news.lacave.net
actsync: STATUS: read 1645 groups, will merge 1645 groups from news.
Nothing with “ctlinnd”.
Besides, just tested:
% actsync news.lacave.net -p 1 -o c -v 4 | grep comp.ai.shells | mod-active
reading list of groups to update
1 change(s) to do
ctlinnd pause batch active update, ok
rewriting active file
ctlinnd reload active 'updated from checkgroups'
--- /home/news/db/active.old 2011-01-03 21:29:53.000000000 +0100
+++ /home/news/db/active 2011-01-03 21:30:18.000000000 +0100
@@ -609,0 +610 @@
+comp.ai.shells 0000000000 0000000001 y
ctlinnd go batch active update, ok
updating /home/news/db/active.times
chmoding active files
Seems to work.
What do you have on your news server when you run mod-active?
--
Julien ÉLIE
« La libertad, Sancho, es uno de los más preciosos dones que a los
hombres dieron los cielos; con ella no pueden igualarse los tesoros
que encierran la tierra y el mar: por la libertad, así como por la
honra, se puede y debe aventurar la vida. » (Miguel de Cervantes
Saavedra)
Julien ÉLIE
And also with “grep ctlinnd” for the first pipe.
John F. Morse
>> I had to run actsync, redirecting the output to a file, and then remove
>> everything but the actual ctlinnd newgroup command lines before
>> mod-active would act on it. That is what I believe Russ is suggesting.
>> Since I already had this "answer" I didn't feel I had a question to be
>> answered, and improperly stated this, confusing everybody. Sorry for
>> that.
>
> Could you please tell us what are the lines you are removing?
I'll try.... ;-)
> I for instance have:
>
> 19:36 news@trigo ~/work/inn/trunk% actsync news.lacave.net -p 1 -o c
> actsync: line 607 <trigofacile.test.υτφ8> from localhost has a bad
> newsgroup name
> actsync: line 608 <dk.test.utf8-æøå> from localhost has a bad
> newsgroup name
> ctlinnd rmgroup af.philo
> ctlinnd rmgroup af.politique
> ctlinnd rmgroup can.francais
> [...]
>
> but it still works with mod-active because unknown lines (with no
> ctlinnd command) are silently ignored.
Well, if I would have used actsync, the lines might be the ones that -v1
through -v4 create. Such as:
actsync: STATUS: obtaining active file from /var/lib/news/active
actsync: STATUS: read 32968 groups, will merge 32968 groups from
/var/lib/news/active
actsync: STATUS: found 0 line errors from /var/lib/news/active
actsync: STATUS: obtaining active file from ./active-news2
actsync: STATUS: read 28102 groups, will merge 28102 groups from
./active-new
actsync: STATUS: found 0 line errors from ./active-new
actsync: STATUS: sorting groups
actsync: STATUS: host2 has 0 =type groups
actsync: STATUS: merging groups
actsync: STATUS: host1 has 0 =type groups
actsync: STATUS: sort-merge passed thru 28102 groups
actsync: STATUS: sort-merge marked 0 groups for removal
actsync: STATUS: marked 0 =type error groups from host1
actsync: STATUS: marked 0 =type error groups from host2
actsync: STATUS: marked 0 error groups for removal
actsync: STATUS: same=32968 add=7, change=0, remove=0
actsync: STATUS: ignore=0, work=7, err=0
actsync: STATUS: same+work+err=32975, host1_same=99.98%
I'm sure it would work with -v0 (or no -v), as I have used actsync in
the past.
Actsync failed because of the missing passwd.nntp data. Testing actsync
looks good now (with -v 0) on a local copy of an active file.
However I used docheckgroups, and that was a little fresher in my memory.
With docheckgroups, one major obstacle is the known egrep overflow:
-rw-rw-r-- 1 news news 2515382 2011-01-01 13:19 newsgroups
-rw-r--r-- 1 news news 2514617 2011-01-03 14:12 newsgroups-ISC
news@optima11:/var/lib/news$ cat newsgroups-ISC | docheckgroups
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
egrep: Unmatched ) or \)
Perhaps a future project would be to improve docheckgroups so egrep
doesn't choke?
To get around that egrep overflow problem, I'll shorten my
newsgroups-ISC list to make it smaller.
Here is docheckgroups with a small newsgroup file, which has a new group
added, and one group description modified:
news@optima11:/var/lib/news$ cat newsgroups-ISC_small | docheckgroups
# The following newsgroups are non-standard.
# abg.allgemein
# abg.amiga
# abg.gruesse
# abg.kultur
# abg.mampf
# abg.member
# abg.spiele
# abg.test04
# abg.veranstaltungen
# abg.witziges
# You can remove them by executing the commands:
/usr/lib/news/bin/ctlinnd rmgroup abg.allgemein
/usr/lib/news/bin/ctlinnd rmgroup abg.amiga
/usr/lib/news/bin/ctlinnd rmgroup abg.gruesse
/usr/lib/news/bin/ctlinnd rmgroup abg.kultur
/usr/lib/news/bin/ctlinnd rmgroup abg.mampf
/usr/lib/news/bin/ctlinnd rmgroup abg.member
/usr/lib/news/bin/ctlinnd rmgroup abg.spiele
/usr/lib/news/bin/ctlinnd rmgroup abg.test04
/usr/lib/news/bin/ctlinnd rmgroup abg.veranstaltungen
/usr/lib/news/bin/ctlinnd rmgroup abg.witziges
# The following newsgroups were missing and should be added.
# abg.fake
# You can do this by executing the command(s):
/usr/lib/news/bin/ctlinnd newgroup abg.fake y
# The following newsgroups descriptions are obsolete.
# abg.allgemein Augsburg: Verschiedenes
# abg.amiga Augsburg: fuer unsere spanischen Freunde
# abg.comp Augsburg: Diskussion ueber Computer
# abg.diskussion Augsburg: fuer alle moeglichen sinnlosen Diskussionen
# abg.english Augsburg: chatten auf englisch auch fuer
nichtenglaender...
# abg.flohmarkt Augsburg: Verkauf von an Privat
# abg.frauen Augsburg: frauenthemen, was immer das auch sein soll
# abg.gruesse Augsburg: was taeten wir nur ohne sie...
# abg.kultur Augsburg: Konzerttermine, Kritiken, Filme, Buecher
# abg.mampf Augsburg: Fuer Rezepte, Kochtips, Kneipentests u.ae.
# abg.member Augsburg: Mitglieder
# abg.spiele Augsburg: spieletips und fragen
# abg.test04 Augsburg: Testgruppe 2004, vorsicht Reflektoren
# abg.veranstaltungen Augsburg: Veranstaltungen
# abg.witziges Augsburg: muss auch sein....
# You can remove them by editing /var/lib/news/newsgroups.
# The following newsgroups descriptions were missing and should be added.
# abg.comp Augsburg: Diskussion ueber Computer
# abg.diskussion Augsburg: fuer alle moeglichen sinnlosen
Diskussionen
# abg.english Augsburg: chat in English even for
non-English people
# abg.fake A fake newsgroup for testing
# abg.flohmarkt Augsburg: Verkauf von
# abg.frauen Augsburg: frauenthemen, was immer das auch
sein soll
# You can add them by editing /var/lib/news/newsgroups.
exit # so you can feed this message into the shell
# And remember to update /var/lib/news/newsgroups.
# Remove these lines:
# abg.allgemein Augsburg: Verschiedenes
# abg.amiga Augsburg: fuer unsere spanischen Freunde
# abg.gruesse Augsburg: was taeten wir nur ohne sie...
# abg.kultur Augsburg: Konzerttermine, Kritiken, Filme, Buecher
# abg.mampf Augsburg: Fuer Rezepte, Kochtips, Kneipentests u.ae.
# abg.member Augsburg: Mitglieder
# abg.spiele Augsburg: spieletips und fragen
# abg.test04 Augsburg: Testgruppe 2004, vorsicht Reflektoren
# abg.veranstaltungen Augsburg: Veranstaltungen
# abg.witziges Augsburg: muss auch sein....
# Add these lines:
# abg.fake A fake newsgroup for testing
news@optima11:/var/lib/news$
------------------------------------------------------------------------
If my memory isn't completely bonkers, there are lines in the above
that, when used as input, cause mod-active to fail, and without any
error given. Perhaps it is this line?:
exit # so you can feed this message into the shell
Anyway, after spending some time trying and failing to get actsync to
supply the input to mod-active, my "fix" was to copy all of the
"/usr/lib/news/bin/ctlinnd ...group ..." lines into a new textfile, and
then use it as the input for mod-active. That works as expected.
Mod-active has no problem that I've encountered, and it is a fine program.
The docheckgroups complaint here is this method is not automated by
piping, and requires the creation of a new file, editing, etc. Not so
much a big deal if only one server were involved, but several remote
headless servers amplifies on the amount of work due to file copying to
each server. Plus there is the egrep problem, which is a non-issue to
someone who isn't wanting to use checkgroups with thousands of groups.
As for actsync, since I used getlist and discovered the -A option
requires a passwd.nntp file, and you verified this, I probably can use
actsync now.
Don't worry about me, Julien. I managed to meet my goal by doing a
little more work, and my servers are up-to-date (at the moment, of
course). ;-)
>> Also possibly connected was considerable checkgroups, plus rmgroup and
>> newgroup control message activity in the same newsgroups in the past
>> couple of days. Were these possibly forged rmgroup messages?
>
> It was an issue with control-active; and owing to your synchronizing
> your active file with the one provided by ftp.isc.org, you were
> affected with the removal and then the re-adding of these newsgroups.
No, it was due to yesterday's control messages and checkgroup messages.
My synchronizing my active (and newsgroups) was made last week, and they
were fine until yesterday.
They are fixed now. I used diff to provide the differences, created the
ctlinnd command prefix on each group, copied this file to each server,
then simply ran mod-active with the file for input.
Next time I'll use actsync on each server (via ssh), which will save me
copying files to each server for mod-active's use.
>> If these removals were just errors by news administrators, they caused
>> the removal of several groups, including their watermarks. Then these
>> groups were replaced with zeroed watermarks when the groups were
>> recreated, and will result in newsreaders not seeing articles until the
>> new watermarks catch up to the various newsreaders (newsrc) databases.
>
> An argument in favour of the experimental LIST REMOVALS command along
> with the “r” status flag :-)
I'm not familiar with this. Can you elaborate -- a link perhaps?
I do know that the newsrc file in some newsreaders (Mozilla products
especially) can create havoc for users who don't understand watermarks.
First they need to be aware something is wrong. That is often a challenge.
Their simple solution is to unsubscribe from certain groups, or in some
cases, completely remove a news server "account" form their newsreader.
Then start over after manually removing/deleting certain files that
Mozilla loves to keep forever. (It's probably not just Mozilla either.)
Of course they usually don't know how to manually edit the newsrc file
with an editor, or just delete it, and if they do, they don't realize
that they must quit the program before editing or it will simply
overwrite their work with what it has stored in RAM.
This is a problem with newsreading clients, but when a news server
experiences a renumber, it instigates the action.
Is there any easy solution? What about using a date and time instead of
an article Xref number? Doing this would mean totally changing the way
NNRP works.
And it doesn't take into consideration the many computers with incorrect
clock settings. ;-)
Don't stay up late thinking about that idea. Trop de vin peut vous tromper!
Mangez de la nourriture plus mexicaine, comme des enchiladas! ;-)
John F. Morse
> Hi Russ,
>
>>> 19:36 news@trigo ~/work/inn/trunk% actsync news.lacave.net -p 1 -o c
>>
>> Oh, hm. Shouldn't the error messages be going to standard error and
>> hence
>> not into a pipe to mod-active?
>
> Error messages are already going to standard error. I have just
> checked, and also seen that the -q flag can silent them.
>
> % actsync news.lacave.net -p 1 -o c -q 12
>
> It removes the warnings from both the local server and the remote server.
> Only ctlinnd commands are now output.
> John, did you try that command?
Not yet, but I have it marked for later.
Thanks.
John F. Morse
> Hi John,
>
>> There are frequent checkgroups messages arriving though, and their
>> results vary between 2.4.5 and 2.5.2 servers, and one test server where
>> I don't have the pgp so I will receive Daily Report indications of
>> control group activity.
>
> That is to say? Do you have different ctlinnd commands between 2.4.5
> and 2.5.2 servers?
No. I try and maintain all servers with the same active and newsgroups file.
The main differences amongst the servers is due to hard disk space
(different number of cycbuffs and sizes), incoming port assignments, no
PGP on one server, TLS on one server, etc.
> (In case you are speaking about the commented lines in docheckgroups
> output, it is normal � but actual ctlinnd commands should be the same.)
I believe I was thinking about what the Daily Report shows. There is a
slight difference between 2.4.5 and 2.5.2.
Additionally I found a six group difference when using "wc -l" to
compare the active files between the 2.4.5 and the 2.5.2 servers. That
was the total difference from several newgroup and rmgroup commands,
plus I suspect similar action from incoming checkgroups. All on January
2, and documented at ftp://ftp.isc.org/pub/usenet/CONFIG/LOGS/log.2011-01
I don't worry about the newsgroups files on the servers. They are easily
copied from ISC and replaced.
Julien ÉLIE
> With docheckgroups, one major obstacle is the known egrep overflow:
>
> news@optima11:/var/lib/news$ cat newsgroups-ISC | docheckgroups
> egrep: Unmatched ) or \)
> egrep: Unmatched ) or \)
> egrep: Unmatched ) or \)
> egrep: Unmatched ) or \)
> egrep: Unmatched ) or \)
> egrep: Unmatched ) or \)
>
> Perhaps a future project would be to improve docheckgroups so egrep
> doesn't choke?
egrep is no longer used in INN 2.5.2. It was in INN 2.4.x.
What are the remaining problems when you use docheckgroups on INN 2.5.2?
> To get around that egrep overflow problem, I'll shorten my
> newsgroups-ISC list to make it smaller.
> # You can remove them by editing /var/lib/news/newsgroups.
> # You can add them by editing /var/lib/news/newsgroups.
> # And remember to update /var/lib/news/newsgroups.
It is an old version of docheckgroups (INN 2.4.x).
> If my memory isn't completely bonkers, there are lines in the above
> that, when used as input, cause mod-active to fail, and without any
> error given. Perhaps it is this line?:
>
> exit # so you can feed this message into the shell
This line is ignored my mod-active.
>> An argument in favour of the experimental LIST REMOVALS command along
>> with the “r” status flag :-)
>
> I'm not familiar with this. Can you elaborate -- a link perhaps?
It is not implemented. And I do not see how additions and removals from
multiples programs can be handled smoothly (docheckgroups, mod-active,
manually…).
Basically, it would have been a means to mark a newsgroup as
to-be-removed without actually being removed. It would be readable, and
would not accept articles. It would be automatically removed after a
configurable number of days.
It would allow reinstatement of a newsgroup rmgroup'ed by error. On
receiving a checkgroups or a newgroup control article mentioning it is
still a valid newsgroup, its status would be restored to the one it had.
Therefore, no deletion.
> Don't stay up late thinking about that idea. Trop de vin peut vous tromper!
> Mangez de la nourriture plus mexicaine, comme des enchiladas! ;-)
Good point; it is such a long time since I have eaten Mexican food.
Time to change that.
--
Julien ÉLIE
« Il ne faut jamais parler sèchement à un Numide. » (Astérix)
John F. Morse
> Hi John,
>
>> Could you please quote it back in its entirety, and then enter what I
>> would need to add or change to allow the user "newsmaster" to access all
>> groups with the RPNAL access privileges, and from only my CIDR netblock?
>> (Julien suggested 192.168.0.0/16 but I believe that CIDR should be
>> 192.168.0.0/32.)
>
> With /32, it would mean only one IP: 192.168.0.0.
> However, this IP is the one of the route to your subnet…
Sorry, that was a typo (/32) when I meant /24. ;-)
I looked here: http://en.wikipedia.org/wiki/CIDR#Prefix_aggregation
On the a.b.c.0/24" line I saw:
IP/CIDR Δ to last Mask Hosts (*) Class
IP addr
a.b.c.0/24 +0.0.0.255 255.255.255.000 256 1 C
I took that mask to mean "exact.exact.exact.anything" which I thought
would allow me access from any client on my LAN.
The "Hosts" of 256 is misleading because only 254 IPs are really
available. The 0 and the 255 are nonassignable to a host. The "*"
footnote describes this.
Wouldn't the 0/16 be for a Class B address (= 256 Class C addresses)?
>> access "special" {
>> users: "<localhost>"
>> newsgroups: "*"
>> access: "RPNAL"
>> }
>
> users: "<localhost>,john"
>
> should do the trick here, I believe.
How would you like your steak?!
That did the trick, although I used the user "newsmaster" instead of
"john" so I could keep them separate.
Thank you very much.
Jeffrey M. Vinocur
John F. Morse <dont.bother> wrote:
>
>>> --- hosts: "stdin,localhost"
>>> +++ hosts: "stdin,localhost,*.my.net,192.168.*"
>>> auth: "/usr/lib/news/bin/auth/passwd/ckpasswd -f /var/lib/news/passlist"
>>> default: "<localhost>"
>>> }
>>
>> I think 192.168.* should be written 192.168.0.0/16, and that it is OK
>> for *.my.net.
>
>Nothing I tried would work, including just the hostname, the FQDN, just
>the domain, the *my.net, the full IP, etc.
For future reference, your problem here is one of understanding
the readers.conf paradigm (that is, step one assign an identity
using the auth blocks, step two assign access based on the
identity). Here you have made an auth group that will match your
IP, but then you authenticate against the usual password file and
therefore have the usual identity assigned (that is, you do not
get the identity "<localhost>" because ckpasswd gives back a more
specific identity, namely whatever you authenticated as, and then
that matches the same access block it usually does).
(If you really wanted to do something like this, you could use
the key: parameter.)
John F. Morse
"What" meaning what? ;-)
John F. Morse
>> % actsync news.lacave.net -p 1 -o c -v 4 | grep comp.ai.shells |
>> mod-active
>
> And also with “grep ctlinnd” for the first pipe.
>
Huh?
Why are you grepping a newsgroup name (comp.ai.shells) which you are not
yet aware of until you have run actsync?
Or were you just using this as a demonstration, to limit the changes to
one single group?
I suspect you meant to say that "grep ctlinnd" could be used to only
pipe those lines to mod-active?
John F. Morse
>> With docheckgroups, one major obstacle is the known egrep overflow:
>>
>> news@optima11:/var/lib/news$ cat newsgroups-ISC | docheckgroups
>> egrep: Unmatched ) or \)
>> egrep: Unmatched ) or \)
>> egrep: Unmatched ) or \)
>> egrep: Unmatched ) or \)
>> egrep: Unmatched ) or \)
>> egrep: Unmatched ) or \)
> >
> > Perhaps a future project would be to improve docheckgroups so egrep
> > doesn't choke?
>
> egrep is no longer used in INN 2.5.2. It was in INN 2.4.x.
> What are the remaining problems when you use docheckgroups on INN 2.5.2?
Here are the tests and results:
news@n102:/var/lib/news$ cat /etc/debian_version
squeeze/sid
news@n102:/var/lib/news$ innconfval -v
INN 2.5.2
news@n102:/var/lib/news$
For the test I copied a newsgroups file that I had downloaded from the
ISC on 20101223:
news@n102:/var/lib/news$ wc newsgroups-ISC
45959 236510 2514617 newsgroups-ISC
news@n102:/var/lib/news$
It contains 45959 newsgroup descriptions as shown above.
Then I ran docheckgroups twice, using two different input methods.
news@n102:/var/lib/news$ cat newsgroups-ISC | docheckgroups
news@n102:/var/lib/news$
news@n102:/var/lib/news$ docheckgroups < newsgroups-ISC
news@n102:/var/lib/news$
Using either form of input, docheckgroups does nothing but return the
prompt after about one second:
news@n102:/var/lib/news$ time cat newsgroups-ISC | docheckgroups
real 0m0.678s
user 0m0.592s
sys 0m0.084s
news@n102:/var/lib/news$
I made the input file smaller by cutting out all but 10 lines from the
newsgroups-ISC file and saved it as "newsgroups-ISC_small":
news@n102:/var/lib/news$ head newsgroups-ISC > newsgroups-ISC_small | wc
-l newsgroups-ISC_small
10 newsgroups-ISC_small
news@n102:/var/lib/news$
Then I ran docheckgroups with this smaller ten-line newsgroup file, and
it created over 34 thousand lines of corrections and needed changes,
basically listing the complete alt.* hierarchy, plus my local groups
which are listed in /etc/news/localgroups. Here's the tally:
news@n102:/var/lib/news$ cat newsgroups-ISC_small | docheckgroups | wc -l
34185
news@n102:/var/lib/news$
It listed 6876 groups for removal. This would have been a catastrophe
had I been piping this into mod-active! Sadly, this is what actually
happened to some 900 alt.* groups a couple of weeks ago, and the reason
for my posting.
news@n102:/var/lib/news$ cat newsgroups-ISC_small | docheckgroups | grep
"ctlinnd rmgroup" | wc -l
6876
news@n102:/var/lib/news$
So it appears docheckgroups still has a problem running when using a
large newsgroup comparison file.
Then it creates a big mess when the file is small enough to allow
docheckgroups to actually run.
It doesn't appear to recognize the /etc/news/localgroups list.
It wants to change two local moderated groups to "y" from "m" -- these
groups have "(no posting)" instead of "(Moderated)" in their
descriptions, which might be the problem. If I changed them to "n" I'd
probably still receive the docheckgroups attempt to change them to "y"
wouldn't I?
Hope this helps.
Russ Allbery
> For the test I copied a newsgroups file that I had downloaded from the ISC
> on 20101223:
> news@n102:/var/lib/news$ wc newsgroups-ISC
> 45959 236510 2514617 newsgroups-ISC
> news@n102:/var/lib/news$
> It contains 45959 newsgroup descriptions as shown above.
> Then I ran docheckgroups twice, using two different input methods.
> news@n102:/var/lib/news$ cat newsgroups-ISC | docheckgroups
> news@n102:/var/lib/news$
> news@n102:/var/lib/news$ docheckgroups < newsgroups-ISC
> news@n102:/var/lib/news$
> Using either form of input, docheckgroups does nothing but return the
> prompt after about one second:
So apparently you've already synchronized your active file with the ISC
one and there's nothing to change. Given that you've mentioned that
you're running actsync already, that doesn't surprise me. Did you not
expect this behavior?
> I made the input file smaller by cutting out all but 10 lines from the
> newsgroups-ISC file and saved it as "newsgroups-ISC_small":
> news@n102:/var/lib/news$ head newsgroups-ISC > newsgroups-ISC_small | wc
> -l newsgroups-ISC_small
> 10 newsgroups-ISC_small
> news@n102:/var/lib/news$
> Then I ran docheckgroups with this smaller ten-line newsgroup file, and it
> created over 34 thousand lines of corrections and needed changes,
> basically listing the complete alt.* hierarchy, plus my local groups which
> are listed in /etc/news/localgroups.
Well, yes. You created a checkgroups for the alt.* hierarchy that
contained only ten groups and then applied it. docheckgroups dutifully
wanted to remove every other group in the alt.* hierarchy. A checkgroups
is a complete list of the groups in the hierarchy, and applying a
checkgroups for a hierarchy will remove groups on the server in that
hierarchy that aren't listed in the checkgroups.
> It listed 6876 groups for removal. This would have been a catastrophe had
> I been piping this into mod-active!
Well, yes. Among other things that would be a catastrophe would be for
you to randomly rm -rf parts of your server. So don't do that! :)
> Sadly, this is what actually happened to some 900 alt.* groups a couple
> of weeks ago, and the reason for my posting.
You downloaded a newsgroups file, edited it to remove groups that are
already on your server, and ran it through docheckgroups, and then had
problems because the groups you deleted from the checkgroups were removed
from your server?
That's what you demonstrated in this message, so if what happened a couple
of weeks ago was something different, unfortunately this message didn't
help much in figuring out what happened.
> So it appears docheckgroups still has a problem running when using a large
> newsgroup comparison file.
> Then it creates a big mess when the file is small enough to allow
> docheckgroups to actually run.
I don't believe you've demonstrated either of those things. In both
cases, it seems to have done what I would have expected it to do. But
maybe you know there are groups in the large ISC newsgroups file that you
don't have on your server?
> It doesn't appear to recognize the /etc/news/localgroups list.
> It wants to change two local moderated groups to "y" from "m" -- these
> groups have "(no posting)" instead of "(Moderated)" in their descriptions,
> which might be the problem. If I changed them to "n" I'd probably still
> receive the docheckgroups attempt to change them to "y" wouldn't I?
Are these two paragraphs talking about the same issue, or are they two
different things?
Julien ÉLIE
>>> Could you please quote it back in its entirety, and then enter what I
>>> would need to add or change to allow the user "newsmaster" to access all
>>> groups with the RPNAL access privileges, and from only my CIDR netblock?
>>> (Julien suggested 192.168.0.0/16 but I believe that CIDR should be
>>> 192.168.0.0/32.)
>>
>> With /32, it would mean only one IP: 192.168.0.0.
>> However, this IP is the one of the route to your subnet…
>
> Sorry, that was a typo (/32) when I meant /24. ;-)
> I took that mask to mean "exact.exact.exact.anything" which I thought
> would allow me access from any client on my LAN.
>
> Wouldn't the 0/16 be for a Class B address (= 256 Class C addresses)?
Yes, /16 is for a Class B address.
I thought it was what you were looking for because of your exemple:
hosts: "stdin,localhost,*.my.net,192.168.*"
I understood 192.168.0.0/16 from it.
Your /24 would have been 192.168.0.* (supposing that it was an allowed
syntax in readers.conf — which is not).
--
Julien ÉLIE
« L'art de la médecine consiste à distraire le malade pendant que la
nature le guérit. » (Voltaire)
Julien ÉLIE
>>> IOW, this causes the output to contain more than mod-active would
>>> swallow:
>>>
>>> actsync -o c -z 0 -m -v 4 -p 0 /var/lib/news/active ./active-new
>>
>> Seems to work.
>> What do you have on your news server when you run mod-active?
>
> "What" meaning what? ;-)
What is the complete output of your actsync command?
I still do not understand why mod-active would do nothing with it.
Does it contain “ctlinnd” lines?
Julien ÉLIE
>>> % actsync news.lacave.net -p 1 -o c -v 4 | grep comp.ai.shells |
>>> mod-active
>>
>> And also with “grep ctlinnd” for the first pipe.
>
> Huh?
>
> Why are you grepping a newsgroup name (comp.ai.shells) which you are not
> yet aware of until you have run actsync?
For testing purpose, because I know it is a group that I do not carry.
> Or were you just using this as a demonstration, to limit the changes to
> one single group?
Yes; it is my own server and I do not want to create thousands of groups
just to test the command.
> I suspect you meant to say that "grep ctlinnd" could be used to only
> pipe those lines to mod-active?
Exactly. Just “ctlinnd” would be enough.
John F. Morse
> So apparently you've already synchronized your active file with the ISC
> one and there's nothing to change. Given that you've mentioned that
> you're running actsync already, that doesn't surprise me. Did you not
> expect this behavior?
>
My active file is not synchronized with the active file at ISC. I have
already reported that I do not carry most all of the alt.binaries
groups. That makes a huge difference.
I have not run actsync already. When I ran it it was many months ago on
an old server that is no longer online.
The issue is docheckgroups.
>> I made the input file smaller by cutting out all but 10 lines from the
>> newsgroups-ISC file and saved it as "newsgroups-ISC_small":
>>
>
>
>> news@n102:/var/lib/news$ head newsgroups-ISC > newsgroups-ISC_small | wc
>> -l newsgroups-ISC_small
>> 10 newsgroups-ISC_small
>> news@n102:/var/lib/news$
>>
>
>
>> Then I ran docheckgroups with this smaller ten-line newsgroup file, and it
>> created over 34 thousand lines of corrections and needed changes,
>> basically listing the complete alt.* hierarchy, plus my local groups which
>> are listed in /etc/news/localgroups.
>>
>
> Well, yes. You created a checkgroups for the alt.* hierarchy that
> contained only ten groups and then applied it. docheckgroups dutifully
> wanted to remove every other group in the alt.* hierarchy. A checkgroups
> is a complete list of the groups in the hierarchy, and applying a
> checkgroups for a hierarchy will remove groups on the server in that
> hierarchy that aren't listed in the checkgroups.
>
Ah, so it is based on hierarchies?
Then why did it want to remove my local groups?
>> It listed 6876 groups for removal. This would have been a catastrophe had
>> I been piping this into mod-active!
>>
>
> Well, yes. Among other things that would be a catastrophe would be for
> you to randomly rm -rf parts of your server. So don't do that! :)
>
Thanks for the help.
>> Sadly, this is what actually happened to some 900 alt.* groups a couple
>> of weeks ago, and the reason for my posting.
>>
>
> You downloaded a newsgroups file, edited it to remove groups that are
> already on your server, and ran it through docheckgroups, and then had
> problems because the groups you deleted from the checkgroups were removed
> from your server?
>
No, I edited the file to make it smaller so docheckgroups would not
croak on it and silently quit.
> That's what you demonstrated in this message, so if what happened a couple
> of weeks ago was something different, unfortunately this message didn't
> help much in figuring out what happened.
>
Guess not.
>> So it appears docheckgroups still has a problem running when using a large
>> newsgroup comparison file.
>>
>
>
>> Then it creates a big mess when the file is small enough to allow
>> docheckgroups to actually run.
>>
>
> I don't believe you've demonstrated either of those things. In both
> cases, it seems to have done what I would have expected it to do. But
> maybe you know there are groups in the large ISC newsgroups file that you
> don't have on your server?
>
I am very aware of that.
>> It doesn't appear to recognize the /etc/news/localgroups list.
>>
>
>
>> It wants to change two local moderated groups to "y" from "m" -- these
>> groups have "(no posting)" instead of "(Moderated)" in their descriptions,
>> which might be the problem. If I changed them to "n" I'd probably still
>> receive the docheckgroups attempt to change them to "y" wouldn't I?
>>
>
> Are these two paragraphs talking about the same issue, or are they two
> different things?
>
I don't know, Russ. I'm reporting what I did because Julien asked.
But let me state that the same issue is that docheckgroups is not doing
what I would expect it to do according to what I have read in the man page.
Let's just drop this issue. I can live with what I have figured out and
have used successfully.
John F. Morse
>>>> Could you please quote it back in its entirety, and then enter what I
>>>> would need to add or change to allow the user "newsmaster" to
>>>> access all
>>>> groups with the RPNAL access privileges, and from only my CIDR
>>>> netblock?
>>>> (Julien suggested 192.168.0.0/16 but I believe that CIDR should be
>>>> 192.168.0.0/32.)
>>>
>>> With /32, it would mean only one IP: 192.168.0.0.
>>> However, this IP is the one of the route to your subnet…
>>
>> Sorry, that was a typo (/32) when I meant /24. ;-)
>> I took that mask to mean "exact.exact.exact.anything" which I thought
>> would allow me access from any client on my LAN.
>>
>> Wouldn't the 0/16 be for a Class B address (= 256 Class C addresses)?
>
> Yes, /16 is for a Class B address.
> I thought it was what you were looking for because of your exemple:
>
> hosts: "stdin,localhost,*.my.net,192.168.*"
>
> I understood 192.168.0.0/16 from it.
> Your /24 would have been 192.168.0.* (supposing that it was an allowed
> syntax in readers.conf — which is not).
You are correct. Good catch.
What I said previously was a "semi-typo" along with that one real typo.
;-)
I actually run multiple Class C subnets, in a controllable bridged
arrangement, and was thinking about having newsmaster access from any of
them.
I probably should use a /22.
John F. Morse
>>>> IOW, this causes the output to contain more than mod-active would
>>>> swallow:
>>>>
>>>> actsync -o c -z 0 -m -v 4 -p 0 /var/lib/news/active ./active-new
>>>
>>> Seems to work.
>>> What do you have on your news server when you run mod-active?
>>
>> "What" meaning what? ;-)
>
> What is the complete output of your actsync command?
>
> I still do not understand why mod-active would do nothing with it.
> Does it contain “ctlinnd” lines?
>
I don't have the original command output. New groups have been added to
all servers since I had the problem.
This actsync command, run against an ISC download of 45959 groups on
20101223, produces 13534 lines of output, so I won't post it all here. ;-)
But I'll include a few lines from the beginning.
Also consider that this has the -o c format and the -v4 options, which
will result in something different.
news@optima11:/var/lib/news$ time actsync -o c -z 0 -m -v 4 -p 0
/var/lib/news/active ./active-ISC_20101223-45959 > actsync.out
actsync: STATUS: obtaining active file from /var/lib/news/active
actsync: STATUS: read 32968 groups, will merge 32968 groups from
/var/lib/news/active
actsync: STATUS: found 0 line errors from /var/lib/news/active
actsync: STATUS: obtaining active file from ./active-ISC_20101223-45959
actsync: STATUS: read 45959 groups, will merge 45959 groups from
./active-ISC_20101223-45959
actsync: STATUS: found 0 line errors from ./active-ISC_20101223-45959
actsync: STATUS: sorting groups
actsync: STATUS: host2 has 0 =type groups
actsync: STATUS: merging groups
actsync: STATUS: host1 has 0 =type groups
actsync: STATUS: sort-merge passed thru 45959 groups
actsync: STATUS: sort-merge marked 0 groups for removal
actsync: STATUS: marked 0 =type error groups from host1
actsync: STATUS: marked 0 =type error groups from host2
actsync: STATUS: marked 0 error groups for removal
actsync: STATUS: same=32962 add=13528, change=6, remove=0
actsync: STATUS: ignore=0, work=13534, err=0
actsync: STATUS: same+work+err=46496, host1_same=70.89%
...
ctlinnd newgroup alt.0a.fred-hall y actsync
ctlinnd newgroup alt.0a.fred-hall-is-a-bowl-of-piss y actsync
ctlinnd newgroup alt.0a.fred-hall.animals y actsync
ctlinnd newgroup alt.0a.fred-hall.beer y actsync
ctlinnd newgroup alt.0a.fred-hall.bites-the-dust y actsync
ctlinnd newgroup alt.0a.fred-hall.cats y actsync
ctlinnd newgroup alt.0a.fred-hall.dogs y actsync
ctlinnd newgroup alt.0a.fred-hall.e-cards y actsync
ctlinnd newgroup alt.0a.fred-hall.froth y actsync
ctlinnd newgroup alt.0a.fred-hall.god y actsync
...
ctlinnd newgroup alt.zx.dp y actsync
ctlinnd newgroup alt.zzyzx y actsync
ctlinnd newgroup alt.zzz.cypher y actsync
ctlinnd newgroup alt.zzzz.fred-hall-is-a-piece-of-shit y actsync
ctlinnd changegroup control y
ctlinnd changegroup control.cancel y
ctlinnd changegroup control.checkgroups y
ctlinnd changegroup control.newgroup y
ctlinnd changegroup control.rmgroup y
ctlinnd newgroup de.alt.etc.koerperpflege y actsync
ctlinnd newgroup de.alt.fan.prince y actsync
ctlinnd newgroup de.alt.rec.ascii-art y actsync
ctlinnd newgroup de.alt.rec.fantasy y actsync
ctlinnd newgroup de.alt.soc.aufmerksamkeitsdefizit y actsync
ctlinnd newgroup de.alt.tv.mash y actsync
ctlinnd newgroup ffm.admin m actsync
ctlinnd changegroup junk y
...
actsync: STATUS: 46496 group(s)
actsync: STATUS: 13528 group(s) to be added
actsync: STATUS: 0 group(s) to be removed
actsync: STATUS: 6 group(s) to be changed
actsync: STATUS: 32962 group(s) are the same
actsync: STATUS: 70.89% of lines unchanged
actsync: STATUS: 0 group(s) ignored
real 0m2.630s
user 0m2.232s
sys 0m0.328s
news@optima11:/var/lib/news$
news@optima11:/var/lib/news$ wc -l actsync.out
13534 actsync.out
news@optima11:/var/lib/news$
All of the newgroup commands seem to be valid, and are for only groups
which I did not want to add to the servers earlier.
This likely would work if the changegroup commands were not created, or
if that change is not important since these groups are already hidden in
readers.conf. They are all "y" in the ISC list, but on all of my servers
they are all "n" for some reason:
news@optima11:/var/lib/news$ cat active | grep n$
control 0000000214 0000000161 n
control.cancel 0000347431 0000343432 n
control.checkgroups 0000000178 0000000168 n
control.newgroup 0000000092 0000000092 n
control.rmgroup 0000000307 0000000307 n
junk 0000000006 0000000007 n
Were they "n" in the INN2 package from Debian, and in the source I
compiled for one server? I don't remember changing them, but I probably
simply copied an active file, modified with awk, from a working server
to each new one I created.
Julien ÉLIE
> ctlinnd changegroup control y
>
> This likely would work if the changegroup commands were not created, or
> if that change is not important since these groups are already hidden in
> readers.conf. They are all "y" in the ISC list, but on all of my servers
> they are all "n" for some reason:
You could use "actsync -i actsync.ign" with:
i control
i control.*
i junk
in your actsync.ign file. It would hide these changegroup commands.
> news@optima11:/var/lib/news$ cat active | grep n$
> control 0000000214 0000000161 n
> control.cancel 0000347431 0000343432 n
> control.checkgroups 0000000178 0000000168 n
> control.newgroup 0000000092 0000000092 n
> control.rmgroup 0000000307 0000000307 n
> junk 0000000006 0000000007 n
>
> Were they "n" in the INN2 package from Debian, and in the source I
> compiled for one server?
Yes. "n" is the default flag in the active file shipped with INN.
Not very important, though. It also works with "y".
Maybe they could be changed to "n" on ftp.isc.org? (I do not remember
whether the question was already asked or answered.)
--
Julien ÉLIE
« Dès que le silence se fait, les gens le meublent. »
(Raymond Devos)
Russ Allbery
>> news@optima11:/var/lib/news$ cat active | grep n$
>> control 0000000214 0000000161 n
>> control.cancel 0000347431 0000343432 n
>> control.checkgroups 0000000178 0000000168 n
>> control.newgroup 0000000092 0000000092 n
>> control.rmgroup 0000000307 0000000307 n
>> junk 0000000006 0000000007 n
>>
>> Were they "n" in the INN2 package from Debian, and in the source I
>> compiled for one server?
> Yes. "n" is the default flag in the active file shipped with INN.
> Not very important, though. It also works with "y".
> Maybe they could be changed to "n" on ftp.isc.org? (I do not remember
> whether the question was already asked or answered.)
I don't see any obvious reason not to, so done.
Julien ÉLIE
>>>
>>> Were they "n" in the INN2 package from Debian, and in the source I
>>> compiled for one server?
>>
>> Yes. "n" is the default flag in the active file shipped with INN.
>> Not very important, though. It also works with "y".
>>
>> Maybe they could be changed to "n" on ftp.isc.org? (I do not remember
>> whether the question was already asked or answered.)
>
> I don't see any obvious reason not to, so done.
Thanks, Russ.
--
Julien ÉLIE
« N'as-tu jamais fait ces rêves, Neo, qui ont l'air plus vrais que la
réalité ? Si tu étais incapable de sortir de l'un de ces rêves,
comment ferais-tu la différence entre le monde du rêve et le monde
réel ? » (Morpheus, _Matrix_)
D. Stussy
news:877hejy...@windlord.stanford.edu...
> Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:
> >> news@optima11:/var/lib/news$ cat active | grep n$
> >> control 0000000214 0000000161 n
> >> control.cancel 0000347431 0000343432 n
> >> control.checkgroups 0000000178 0000000168 n
> >> control.newgroup 0000000092 0000000092 n
> >> control.rmgroup 0000000307 0000000307 n
> >> junk 0000000006 0000000007 n
> >>
> >> Were they "n" in the INN2 package from Debian, and in the source I
> >> compiled for one server?
>
> > Yes. "n" is the default flag in the active file shipped with INN.
> > Not very important, though. It also works with "y".
>
> > Maybe they could be changed to "n" on ftp.isc.org? (I do not remember
> > whether the question was already asked or answered.)
>
> I don't see any obvious reason not to, so done.
I do. As "n", the pseudogroups will accept articles addressed specifically
to themselves, while marked as "x", they will not while still accepting
articles that fit their specific purposes. On my INN server, I have them
marked as "x" and they work just fine:
junk 0000745129 0000000001 x
control 0000000025 0000000012 x
control.cancel 0001989073 0001984088 x
control.checkgroups 0000000568 0000000128 x
control.newgroup 0000000614 0000000261 x
control.rmgroup 0000001533 0000000330 x
control.cmsg 0000024498 0000024477 x
The last group is the catch-all for those servers in Asia sending the
defective cancel messages (which I wish to keep out of "control").
Therefore, I think that ISC should show them marked with "x", not "n".
Julien ÉLIE
> The issue is docheckgroups.
OK, thanks to your help (access to your news server), I see that you do
not have gawk installed and therefore that mawk is used instead.
The behaviour seems to be different between these two programs.
Currently, we have lines written this way in docheckgroups:
${AWK} '{ if (($1 ~ /'"${1:-.}"'/) && ($1 ~ /'"${PATS}"'/)) \
{ print $1 } }' \
${T}/$$msg | ${SORT} >${T}/$$newsgrps
The problem comes from the quoted variables inside the patterns. I
manage to make them work with both mawk and gawk if I use this syntax:
${AWK} '{ if (($1 ~ progargument) && ($1 ~ filepattern)) \
{ print $1 } }' "progargument=${1:-.}" "filepattern=${PATS}" \
${T}/$$msg | ${SORT} >${T}/$$newsgrps
I must admit that I do not know why docheckgroups with mawk outputs
something with a small newsgroups file, and nothing with a big one.
(Pattern matchings should not change depending on the length of the
newsgroups file…)
Are there awk connoisseurs here who could tell if this is the right fix
to do throughout docheckgroups?
(It is the only affected file in INN.)
Thanks beforehand!
>> plus my local groups which are listed in /etc/news/localgroups.
>
> Ah, so it is based on hierarchies? Then why did it want to remove my
> local groups?
Because of the same problem; mawk does not do the expected job like gawk
to extract your local newsgroups.
Julien ÉLIE
> something with a small newsgroups file, and nothing with a big one.
> (Pattern matchings should not change depending on the length of the
> newsgroups file…)
Oh, interestingly, there are mawk segfaults in /var/log/messages:
Jan 5 23:03:07 news kernel: mawk[28435]: segfault at 202e5b7b ip
b771ceca sp bfe442ec error 4 in libc-2.3.6.so[b76b2000+127000]
So did we run into a mawk bug?!?
--
Julien ÉLIE
« Le Périssable et le Temporel ne sont que fictions et symboles ;
l'insuffisant est arrivé jusqu'ici ; l'inénarrable est accompli ;
l'Éternel féminin nous attire vers le Ciel. » (Goethe)
Julien ÉLIE
> control.cmsg 0000024498 0000024477 x
>
> The last group is the catch-all for those servers in Asia sending
> the defective cancel messages (which I wish to keep out of
> "control").
Thanks for pointing out this issue.
We're speaking about control articles badly formatted. For instance:
Newsgroups: nctu.talk
Subject: cmsg cancel <NCTU$A16I...@bbs.cs.nctu.edu.tw>
Control: cmsg cancel <NCTU$A16I...@bbs.cs.nctu.edu.tw.mirrorpost.nctu.talk>
Message-ID: <CNCTU$A16I...@bbs.cs.nctu.edu.tw.mirrorpost.nctu.talk>
I believe such invalid control articles MUST NOT (per RFC 5537)
have been injected into Usenet.
nnrpd rejects these control articles if a user tries to post one.
Wouldn't it be a useful check to do inside Cleanfeed? Rejecting
badly formatted control articles.
Cleanfeed can already reject "ihave", "sendsys" and a few other
obsolete control articles. It would be best if it could reject
everything that is not a standard control article.
Russ Allbery
> I do. As "n", the pseudogroups will accept articles addressed
> specifically to themselves, while marked as "x", they will not while
> still accepting articles that fit their specific purposes. On my INN
> server, I have them marked as "x" and they work just fine:
> junk 0000745129 0000000001 x
> control 0000000025 0000000012 x
> control.cancel 0001989073 0001984088 x
> control.checkgroups 0000000568 0000000128 x
> control.newgroup 0000000614 0000000261 x
> control.rmgroup 0000001533 0000000330 x
> control.cmsg 0000024498 0000024477 x
> The last group is the catch-all for those servers in Asia sending the
> defective cancel messages (which I wish to keep out of "control").
> Therefore, I think that ISC should show them marked with "x", not "n".
Well, bear in mind that they were originally y. Moving from y to n is at
least a step towards what you want.
"n" rejects local posts. "x" rejects local and remote posts. I'm more
willing to express a default policy about what should be sent than about
what should be received. Those groups do periodically get direct
postings, and while filtering out all such postings is a legitimate
decision to make, it's one that I'm a bit more reluctant to make by
default. This is the same reason why the defaults in INN are "n". junk
in particular used to get a fair bit of sporadic traffic.
Russ Allbery
> I must admit that I do not know why docheckgroups with mawk outputs
> something with a small newsgroups file, and nothing with a big
> one. (Pattern matchings should not change depending on the length of the
> newsgroups file…)
The mawk Debian package description includes:
Mawk is smaller and much faster than gawk. It has some compile-time
limits such as NF = 32767 and sprintf buffer = 1020.
I suspect one of those limits or some other compile-time limit has
something to do with it.
I unfortunately don't know enough awk to tell you whether that change
would work. At the point where I have to start writing awk like that, I
usually switch to Perl. :)
John F. Morse
> Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:
>
>> I must admit that I do not know why docheckgroups with mawk outputs
>> something with a small newsgroups file, and nothing with a big
>> one. (Pattern matchings should not change depending on the length of the
>> newsgroups file…)
>>
>
> The mawk Debian package description includes:
>
> Mawk is smaller and much faster than gawk. It has some compile-time
> limits such as NF = 32767 and sprintf buffer = 1020.
>
> I suspect one of those limits or some other compile-time limit has
> something to do with it.
It is good to verify that I'm not as crazy as I though I was yesterday. ;-)
I found these /var/log/messages concerning kernel segfaults with mawk
yesterday, around the time I was working with docheckgroups.
After my docheckgroups testing, the logging ended so I didn't pursue
them right away.
Then again today, while Julien was doing the same tests, I noticed them
again.
They don't happen when the newsgroups file is small. That explains why I
had docheckgroups failures "intermittently" a couple of weeks ago when
attempting to pipe the output into mod-active. I was using several
smaller portions of the complete newsgroups file.
Russ Allbery
> They don't happen when the newsgroups file is small. That explains why I
> had docheckgroups failures "intermittently" a couple of weeks ago when
> attempting to pipe the output into mod-active. I was using several
> smaller portions of the complete newsgroups file.
Yeah, that would explain a lot of things. mawk was probably failing in
such a way that the rest of docheckgroups thought there were no changes.
John F. Morse
> OK, thanks to your help (access to your news server), I see that you
> do not have gawk installed and therefore that mawk is used instead.
You are welcome, and thanks for finding some problems.
> The behaviour seems to be different between these two programs.
FYI there seems to be several implementations of awk:
http://en.wikipedia.org/wiki/Gawk_%28GNU_package%29#Versions_and_implementations
> Because of the same problem; mawk does not do the expected job like
> gawk to extract your local newsgroups.
OK. Good to know that.
Julien ÉLIE
> The mawk Debian package description includes:
>
> Mawk is smaller and much faster than gawk. It has some compile-time
> limits such as NF = 32767 and sprintf buffer = 1020.
>
> I suspect one of those limits or some other compile-time limit has
> something to do with it.
Oh, yes, it must be that. Thanks for having found that limitation.
> I unfortunately don't know enough awk to tell you whether that change
> would work. At the point where I have to start writing awk like that, I
> usually switch to Perl. :)
:-)
I have just made a new version (still not committed) which seems to
work fine.
Example of changes:
BEFORE:
cat | ${AWK} '{ if ($1 !~ /'"${2:-^#}"'/ && $1) { print $0 } }' >${T}/$$msg
NOW:
cat | ${PERL} -e 'while (<STDIN>) { my @fields = split();
print $_ if ((scalar(@fields) > 0) && ($fields[0] !~ /'${2:-^#}'/)
&& ($fields[0] !~ /^#/)); }' > ${T}/$$msg
BEFORE:
${AWK} '{ if (($1 ~ /'"${1:-.}"'/) && ($1 ~ /'"${PATS}"'/) && ($0 ~ / m$/)) { print $1 } }' ${ACTIVE} | ${SORT} >${T}/$$amod.all
NOW:
cat ${ACTIVE} | ${PERL} -e 'while (<STDIN>) { my @fields = split();
print $fields[0]."\n" if ((scalar(@fields) > 0) && ($fields[0] =~ /'${1:-.}'/)
&& ($fields[0] =~ /'${PATS}'/) && ($_ =~ / m$/)); }' | ${SORT} > ${T}/$$amod.all
BEFORE:
${AWK} -F'\t' '{if (length($1) < 8) {print $1"\t\t\t"$2} \
else {if (length($1) < 16) {print $1"\t\t"$2} \
else {print $1"\t"$2}}}' >${NEWSGROUPS}
NOW:
${PERL} -e 'while (<STDIN>) { my @fields = split("\t", $_, 2);
next if (scalar(@fields) == 0);
my $length = length("$fields[0]");
my $desc;
if (scalar(@fields) == 2) { $desc = "$fields[1]"; } else { $desc = ""; }
if ($length < 8) { print $fields[0]."\t\t\t".$desc; }
elsif ($length < 16) { print $fields[0]."\t\t".$desc; }
else { print $fields[0]."\t".$desc; } }' > ${NEWSGROUPS}
In case you see something suspicious, do not hesitate to tell.
(Maybe I could have used "perl -ne" to avoid the while() loop, or even
"perl -ane" to avoid the split(), but it looked clearer this way to
debug and better see what is going on in the script.)
--
Julien ÉLIE
« Medicina animi. »
Russ Allbery
> I have just made a new version (still not committed) which seems to
> work fine.
> Example of changes:
> BEFORE:
> cat | ${AWK} '{ if ($1 !~ /'"${2:-^#}"'/ && $1) { print $0 } }' >${T}/$$msg
> NOW:
> cat | ${PERL} -e 'while (<STDIN>) { my @fields = split();
> print $_ if ((scalar(@fields) > 0) && ($fields[0] !~ /'${2:-^#}'/)
> && ($fields[0] !~ /^#/)); }' > ${T}/$$msg
Hm, the mix of shell and Perl still makes this kind of hard to follow and
read. Have you considered rewriting the whole script in Perl?
(You do need the "" around the $2 parameter, in case it contains shell
metacharacters. There are a few other places like that as well.)
Julien ÉLIE
> Hm, the mix of shell and Perl still makes this kind of hard to
> follow and read.
Yes, you're totally right. This script becomes less and less
maintainable, patch after patch.
> (You do need the "" around the $2 parameter, in case it contains
> shell metacharacters. There are a few other places like that as
> well.)
OK, thanks!
> Have you considered rewriting the whole script in Perl?
I agree it would be the best thing to do.
Well, I will have a look at it.
http://inn.eyrie.org/trac/ticket/57
Urs Janßen has an updated version of docheckgroups written in Perl:
ftp://ftp.tin.org/pub/news/servers/utils/docheckgroups-inn-2.x.pl
This script also handles a complete checkgroups article in entry
(headers+body).
I will start from it. I have to add the new functionalities of the
docheckgroups shipped with INN 2.4 and INN 2.5.
Urs, in case you read us, do not hesitate to tell if this version on
ftp.tin.org is not the latest one for the script.
--
Julien ÉLIE
« – Par Thor !
– Par Odin !
– Par exemple ! » (Astérix)