INN config management
Matt McLeod
or somesuch, but we're not carrying that and the S:N here
is far more useful.
This weekend my project is rationalizing the Netizen news
setup. We've got about 25 or so peers (yup, more peers
than megs/day...). As part of this I'm thinking about
ways of making the configuration of INN a bit less
repetitive.
Before I go completely berserk and build something myself
(which I'll probably do anyway), I'm wondering if there's
anything already around which can take a description of
each peer, and generate appropriate entries for each of
the config files.
What I'm considering doing is something which would take
a file like this:
------
Path: newnews.netizen.com.au
Bofh: yes
2senet: yes
Incoming: news.netizen.com.au,news-in.netizen.com.au
Outgoing: news.netizen.com.au,news-out.netizen.com.au
Admin: ma...@netizen.com.au
Delayed: no
DEFAULT
------
That last bit in this case would snarf a default newsfeeds
entry from the appropriate place, but could instead simply
be the groups/dists pattern.
Then the appropriate entries would be generated, like:
newsfeeds:
netizen.bofh/newnews.netizen.com.au\
:@*,bofh.*/bofh\
:Tm:innfeed!
netizen.2senet/newnews.netizen.com.au\
:@*,net.*/4gh\
:Tm:innfeed!
netizen.unsound/newnews.netizen.com.au\
:@*,\
<whatever the default feed pattern is>
:Tm:innfeed!
(and so on for each config file -- you get the idea)
So there'd be a directory of files like the above,
and when you run the scripty-thing over it the INN
configs get rebuilt.
Are there any obvious hitches to this that I'm missing?
Better ways to do it? The one thing I can think of is
that it may help to allow stuff like:
incoming.conf!max-connections: 20
to explicitly over-ride defaults in configs in a way which
would not require any special knowledge of the allowed
keywords.
--
Service with a capital "Bugger Off".
Russ Allbery
> I suppose this might be better asked in news.software.nntp or somesuch,
> but we're not carrying that and the S:N here is far more useful.
The S/N in news.software.nntp is actually usually pretty good, but this
works too. :)
> This weekend my project is rationalizing the Netizen news setup. We've
> got about 25 or so peers (yup, more peers than megs/day...). As part of
> this I'm thinking about ways of making the configuration of INN a bit
> less repetitive.
Yup. Common desire.
There are a few people I know who have rolled their own scripts along the
lines that you're talking about, but I don't have any of them handy. It's
my intention to eventually replace the lot of newsfeeds, incoming.conf,
and innfeed.conf, along with probably nntpsend.ctl and maybe a few other
things, with a new feeds.conf that lets you specify everything in the same
place and uses the same block structure as the other newer INN
configuration files, but that will have to wait until after I rewrite the
config file parsing stuff and is probably an INN 2.5 project.
So down the road, this hopefully won't continue to be an issue, but in the
interim rolling your own solution is probably what you'll have to do,
unless you can hunt up someone else who's done the same thing.
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Matt McLeod
that Russ Allbery did write:
>Matt McLeod <use...@wibble.cx> writes:
>
>> I suppose this might be better asked in news.software.nntp or somesuch,
>> but we're not carrying that and the S:N here is far more useful.
>
>The S/N in news.software.nntp is actually usually pretty good, but this
>works too. :)
And I just realised we do actually carry it. Silly me.
>> This weekend my project is rationalizing the Netizen news setup. We've
>> got about 25 or so peers (yup, more peers than megs/day...). As part of
>> this I'm thinking about ways of making the configuration of INN a bit
>> less repetitive.
>
>Yup. Common desire.
And now I've got another question... :-)
I thought about it a bit, and concluded that the simplest possible
way to do it would be to just use the C preprocessor. One file per
peer with #ifdef's, one file per INN config with #define CONFIGFILE
at the top and then #include "peer" for each peer.
But...
The bastard thing eats newlines, and parses backslashes. I can
get over the backslashes (just gotta remember to use two instead
of one), but the newline-eating is annoying, especially as I'm
not sure how INN will cope with configs with thousand-character
lines...
Nothing in cpp's manpage (GNU cpp, FWIW) provides a clue on
how to turn this behaviour off -- does anyone have any ideas?
Maybe another preprocessor which would do the trick without
making me learn another language first?
(And no, I don't know m4. At least, not any more than I need
to use the m4 macros to build sendmail.cf)
--
God must have loved calories, she made so many of them.
Phil Homewood
use...@wibble.cx (Matt McLeod) writes:
> Nothing in cpp's manpage (GNU cpp, FWIW) provides a clue on
> how to turn this behaviour off -- does anyone have any ideas?
> Maybe another preprocessor which would do the trick without
> making me learn another language first?
>
> (And no, I don't know m4. At least, not any more than I need
> to use the m4 macros to build sendmail.cf)
m4 would be my recommendation. It's not hard, really, but it's
not much like cpp.
It's all based on divert()ing text onto different queues (or
whatever m4 calls them) and reassembling them in defined order
with undivert(). If you know enough m4 to do sendmail.cf, and
understand the purpose of the rulesets, you can learn a lot
from proto.m4.
The hard part would be planning what gets diverted where; you
have 10 divert queues to play with (and IIRC one or two of them
are reserved). You get to make decisions like
ifdef(something, do_this, else_do_this).
And it won't chew newlines. (If you need to do so, use the
dnl directive.)
--
Phil Homewood d...@atat.dotat.org ph...@rivendell.apana.org.au
Member, Australian Public Access Network Association
"If your operating system goes down more often than your girlfriend, then either
you need to get a Catholic girl, or you need to stop running NT." - Todd Markle.
Matt McLeod
>In article <slrn8g5ait...@megumi.wibble.cx>,
> use...@wibble.cx (Matt McLeod) writes:
>> Nothing in cpp's manpage (GNU cpp, FWIW) provides a clue on
>> how to turn this behaviour off -- does anyone have any ideas?
>> Maybe another preprocessor which would do the trick without
>> making me learn another language first?
>>
>> (And no, I don't know m4. At least, not any more than I need
>> to use the m4 macros to build sendmail.cf)
>
>m4 would be my recommendation. It's not hard, really, but it's
>not much like cpp.
Found a quickie tutorial at:
http://www.bit.net.au/~bhepple/using_m4/using_m4.html
It makes it rather look like all I really need to do is
replace #include's with m4_include(), #define with
m4_define(), and #ifdef with m4_ifdef().
On a first quick glance I'm not sure that the divert()
stuff would help very much, unless there's some way to
cause the content of each queue to go to a different file.
--
"I offer you a new vision of Hell: Watching an entire ISO
committee trying to agree on what wine to have with their meal."
-- Tanuki the Raccoon-dog, ASR
Matt McLeod
>Yea, it is written in the Book of Cyril
>that Phil Homewood did write:
>>m4 would be my recommendation. It's not hard, really, but it's
>>not much like cpp.
>It makes it rather look like all I really need to do is
>replace #include's with m4_include(), #define with
>m4_define(), and #ifdef with m4_ifdef().
Woohoo! Got something very basic up and running, took
all of ten minutes. What I've got is this:
newsfeeds.m4:
m4_define(`NEWSFEEDS',1)
# INN newsfeeds file, autogenerated from m4 macros
# DO NOT EDIT THIS FILE
# Usenet-2 net-monitor
net-monitor\
:!*,net.*,\
:Tp:/news/bin/net.monitor
# innfeed
innfeed!\
:!*,\
:Tc,Wnm*,S30000:/news/bin/startinnfeed
innfeed-delayed!\
:!*,\
:Tc,Wnm*,S16384:/news/bin/delayer 60 \
/news/bin/startinnfeed -c innfeed-delayed.conf
m4_include(`peers.m4')
then peers.m4:
m4_include(`peers/pikachu')
and finally the example peer, pikachu:
m4_ifdef(`NEWSFEEDS',`
pikachu.bofh/pikachu.ucs.ed.ac.uk\
:@*,bofh.*\
:Tm:innfeed!
pikachu/pikachu.ucs.ed.ac.uk\
:!*,@bofh.@net.*,alt.sysadmin.recovery\
:Tm:innfeed!
')
m4_ifdef(`INCOMING.CONF',`
peer pikachu {
hostname: "pikachu.ucs.ed.ac.uk"
}
')
m4_ifdef(`INNFEED.CONF',`
peer pikachu {
ip-name: "pikachu.ucs.ed.ac.uk"
}
peer pikachu.bofh {
ip-name: "pikachu.ucs.ed.ac.uk"
}
')
So now all I really need to do is copy the pikachu file for
each peer, add 'em all into peers.m4, and run newsfeeds.m4. As
the above implies, I've done the same for innfeed.conf and
incoming.conf.
Throw in a makefile, and it should all tootle along very
nicely indeed. But I'm sure there's something I'm missing
(what with having hardly touched m4 before), so feel free
to point out the blindingly obvious...
(No, don't point out the lack of a ME entry in newsfeeds.
Just spotted that myself...)
--
So little time, so little to do.
-- Oscar Levant
Kai Henningsen
> Yea, it is written in the Book of Cyril
> that Phil Homewood did write:
> >In article <slrn8g5ait...@megumi.wibble.cx>,
> > use...@wibble.cx (Matt McLeod) writes:
> >> (And no, I don't know m4. At least, not any more than I need
> >> to use the m4 macros to build sendmail.cf)
> >
> >m4 would be my recommendation. It's not hard, really, but it's
> >not much like cpp.
>
> Found a quickie tutorial at:
>
> http://www.bit.net.au/~bhepple/using_m4/using_m4.html
>
> It makes it rather look like all I really need to do is
> replace #include's with m4_include(), #define with
> m4_define(), and #ifdef with m4_ifdef().
>
> On a first quick glance I'm not sure that the divert()
> stuff would help very much, unless there's some way to
> cause the content of each queue to go to a different file.
Here's what I currently use. (Hey, Russ, didn't I send that to you?)
It's suboptimal in what it choses for feed naming - mainly historical
accident.
-- snip newsfeeds.m4 --
divert(-1)
define(`upcase',
`translit(`$*', `a-z', `A-Z')')
define(`downcase',
`translit(`$*', `A-Z', `a-z')')
define(`funnelname',
`regexp(`$1', `^\(\w\)\(\w*\)', `upcase(`\1')`'downcase(`\2')')')
define(`funnelfor',
`
funnelname($1):!*:Tf,Wfb,B4096/1024:$1
')
define(`u2feed',
`
$1-u2-feed/path_$1\
:@*,net.*/4gh\
:Ad,Tm:funnelname($1)
')
define(`bofhfeed',
`
$1-bofh-feed/path_$1\
:@*,bofh.*/bofh\
:Ad,Tm:funnelname($1)
')
define(`asrfeed',
`
$1-asr-feed/path_$1\
:!*,alt.sysadmin.recovery,@net.*,@westfalen.*,@bofh.*\
:Tm:funnelname($1)
')
define(`localfeed',
`
$1-regular-feed/path_$1`'excludes_$1\
:*,@net.*,@bofh.*,!t-online.*\
:Tm:funnelname($1)
')
define(`myfeed',
`
$1-regular-feed/path_$1`'excludes_$1\
:*,@net.*,@bofh.*\
:Tm:funnelname($1)
')
define(`regfeed',
`
$1-regular-feed/path_$1`'excludes_$1\
:*,@net.*,@westfalen.*,@bofh.*,!t-online.*\
:Tm:funnelname($1)
')
define(`feeds',
`ifelse($2, , `funnelfor($1)',
`$2feed($1)feeds($1, shift(shift($@)))')')
define(`paths',
`ifelse($1, , ,
`,path_$1`'paths(shift($@))')')
define(`path_crosspoint2', `khms.westfalen.de')
define(`path_goggle', `goggle.westfalen.de')
define(`path_jeffenstein', `opus.ies-energy.com,news.jeffenstein.org')
define(`path_muenster', `muenster.westfalen.de')
define(`path_publikom', `news.citykom.de')
define(`path_ruhr', `ruhr.de')
define(`path_tahina', `tahina.priv.at')
define(`path_zippo', `spln')
define(`path_supernews', `supernews.com,NNTP-from-news.supernews.com')
define(`path_btx', `NNTP-from-news.btx.dtag.de')
define(`excludes_crosspoint2',
`')
define(`excludes_jeffenstein',
`paths(zippo, supernews, btx, muenster, publikom, ruhr)')
#define(`excludes_muenster',
# `paths(zippo, supernews, publikom, jeffenstein, tahina)')
define(`excludes_muenster',
`paths(zippo, supernews, publikom)')
define(`excludes_publikom',
`paths(zippo, supernews, btx, muenster, ruhr, jeffenstein, tahina)')
define(`excludes_ruhr',
`paths(zippo, supernews, btx, muenster, publikom, jeffenstein, tahina)')
define(`excludes_tahina',
`paths(zippo, supernews, btx, muenster, publikom, ruhr)')
define(`excludes_zippo',
`paths(supernews, btx, muenster, ruhr, publikom, jeffenstein, tahina)')
divert`'dnl
ME:*,!junk::
overview:*:Tc,WO:/usr/lib/news/bin/overchan
# Muenster
feeds(muenster, u2, asr, local)
# Goggle
feeds(goggle, bofh)
# Ruhr
feeds(ruhr, u2, reg)
# Jeffenstein
feeds(jeffenstein, u2, bofh, asr, reg)
# Tahina
feeds(tahina, u2, bofh, asr, reg)
# Crosspoint
feeds(crosspoint2, u2, bofh, my)
# Zippo
feeds(zippo, u2, reg)
# Publikom
feeds(publikom, reg)
Net-Monitor\
:!*,net.*\
:Tp:/etc/news/net.monitor
#Net-Leaks\
# :!*,net.*/4gh\
# :Tx,Wf:/root/pathcheck.pl
Bofh-Leaks\
:!*,bofh.*\
:Tx,Wf:/var/spool/news/.bofhlist/siteupdate.pl
DbFeed\
:*\
:Tx,Wf,B40960/0:/root/dbfeed.pl
-- snap ---
Kai
--
http://www.westfalen.de/private/khms/
"... by God I *KNOW* what this network is for, and you can't have it."
- Russ Allbery (r...@stanford.edu)
Russ Allbery
> Here's what I currently use. (Hey, Russ, didn't I send that to you?)
That's right, you did. I probably still have it somewhere, in fact.
Wednesday
>So now all I really need to do is copy the pikachu
...Genki Decchu? Great; we can't seem to find a copy at the momen --
>file
Oh.
Rats.
--
that was not a sermon, that was a tirade. [bill mallonee] :::::::::: weds
Matt McLeod
>>I thought about it a bit, and concluded that the simplest possible
>>way to do it would be to just use the C preprocessor. One file per
>>peer with #ifdef's, one file per INN config with #define CONFIGFILE
>>at the top and then #include "peer" for each peer.
>
I know that now. I'm not much of a programmer, so have never
really played with cpp much. m4 it is.
I've taken things a little further than they were last I
posted about this. An example peer is:
m4_ifdef(`NEWSFEEDS',`
rivendell.bofh/rivendell.apana.org.au\
:_bofh_feed:innfeed!
rivendell.net/rivendell.apana.org.au\
:_net_feed:innfeed!
rivendell/rivendell.apana.org.au\
:_feed,\
alt.fan.pratchett,\
alt.fan.pratchett.announce,\
alt.fan.pratchett.bofh,\
alt.fan.pratchett.classic,\
alt.humor.best-of-usenet,\
alt.sysadmin.recovery,\
alt.talk.mended-drum,\
alt.tech-support.recovery\
:Tm:innfeed!
')
m4_ifdef(`INCOMING.CONF',`
peer rivendell {
hostname: "rivendell.apana.org.au"
}
')
m4_ifdef(`INNFEED.CONF',`
peer rivendell {
ip-name: "rivendell.apana.org.au"
}
peer rivendell.bofh {
ip-name: "rivendell.apana.org.au"
}
peer rivendell.net {
ip-name: "rivendell.apana.org.au"
}
')
Adding a new peer is as simple as running 'mkpeer.sh', which
sets the peer and host/ip-names correctly. For someone who
takes bofh, 2senet, and some other stuff all that needs changing
is the 'other' newsfeeds entry.
'regen.sh' does what a makefile should be doing instead but
I'm too lazy to do properly right now (I'm not a programmer,
haven't had a need for make... *hangs head in shame*), and
spits out newsfeeds, incoming.conf, innfeed.conf, and
innfeed-delayed.conf.
Now that I think about it, it probably makes more sense to
define the peer/path/hostnames at the top and just use those
instead. 'mkpeer.sh' just uses sed to clean things up, but
that would be handy for cases where those things change.
I'm sure there are other improvements to be made Guess
I'll figure those out after I put it in place and have
to change 25 peer files... :-(
Question time. How can I get rid of the excess newlines
m4 puts into the output? I can see that they could be
minimized by squashing things up a bit more, but I'd
prefer the peer files to be human-friendly.
The newlines don't hurt anything, they just look silly.
So it's hardly critical.
--
Errors have occurred.
We won't tell you where or why.
Lazy programmers.
Phil Homewood
use...@wibble.cx (Matt McLeod) writes:
> I've taken things a little further than they were last I
> posted about this. An example peer is:
<complicated, you want to just feed it host/path/feedpattern and
explode all that from within m4...>
> Question time. How can I get rid of the excess newlines
> m4 puts into the output? I can see that they could be
> minimized by squashing things up a bit more, but I'd
> prefer the peer files to be human-friendly.
dnl. Divert to newline. Shove a dnl on the end of lines that otherwise
generate spurious whitespace.
--
Phil Homewood d...@atat.dotat.org ph...@rivendell.apana.org.au
Member, Australian Public Access Network Association
"Oh Old MacDonald was dyslexic, e-o-i-o-e..." - Billy Connolly
Matt McLeod
>I'm sure there are other improvements to be made Guess
>I'll figure those out after I put it in place and have
>to change 25 peer files... :-(
OK. Anyone got any online m4 tutorials they feel like
recommending at this stage? I had another bright idea --
if _isbofh is defined, bung in appropriate entries for
a bofh feed. But m4 doesn't seem to like the way I'm
doing it:
m4_define(`_isbofh',1)
m4_define(`_bofh',`
_peer.bofh/_path\
:_bofh_feed:innfeed!
')
[...]
m4_ifdef(_isbofh,_bofh)
scores me:
peers/template:19: m4: Warning: Excess arguments to built-in `m4_ifdef' ignored
I had originally been trying to do:
m4_define(_isbofh)
m4_ifdef(NEWSFEEDS,`
[...]
m4_ifdef(_isbofh,`
stick bofh feed entry in here
')
')
but that's already inside an ifdef, and it doesn't seem to like
the nested quotes.
So -- general info on how to make this specific case work would
be nice, but a pointer to a good tutorial would be even better.
I did find one, but it's targetted at HTML stuff and doesn't
seem to cover this sort of situation.
--
"I only touch base with reality on an as-needed basis!"
-- Royal Floyd Mengot (Klaus)
Matt McLeod
>In article <slrn8g8hb3...@megumi.wibble.cx>,
> use...@wibble.cx (Matt McLeod) writes:
>> I've taken things a little further than they were last I
>> posted about this. An example peer is:
>
><complicated, you want to just feed it host/path/feedpattern and
>explode all that from within m4...>
Yeah, the most recent version does that. Thought of it while
writing what you replied to.
The whole thing is evolving very quickly I can see this
being a night of m4-geeking, and then going back to work
on Wednesday and m4-ing damned near everything. :-)
--
"I once successfully declined a departmental retreat,
saying that on that day I planned instead to advance."
-- Alan J. Rosenthal, ASR
Matt McLeod
> m4_ifdef(_isbofh,_bofh)
which, of course, should be;
m4_ifdef(_isbofh,`bofh')
and now it works!
Dunno why it was objecting to the nested quotes
before -- will have to experiment further.
So now I move those definitions into newsfeeds.m4 and
friends, and it all becomes even simpler...
Hell, I might even have a grip on this soon. :-)
--
"See? Mention 21 inches and he gets all excited!"
Phil Homewood
use...@wibble.cx (Matt McLeod) writes:
> m4_define(`_isbofh',1)
> peers/template:19: m4: Warning: Excess arguments to built-in `m4_ifdef' ignored
I have no idea what an m4_ifdef() is, but try using ifdef() instead.
I've never seen your m4_ variants used anywhere.
--
Phil Homewood d...@atat.dotat.org ph...@rivendell.apana.org.au
Member, Australian Public Access Network Association
"I'll trade you two [LINK]'s for an [IMAGE]" - jen@rivendell
Matt McLeod
>In article <slrn8g8jm1...@megumi.wibble.cx>,
> use...@wibble.cx (Matt McLeod) writes:
>> m4_define(`_isbofh',1)
>
>> peers/template:19: m4: Warning: Excess arguments to built-in `m4_ifdef' ignored
>
>I have no idea what an m4_ifdef() is, but try using ifdef() instead.
>I've never seen your m4_ variants used anywhere.
'm4 -P' uses it -- I figure it's safest to have the builtins
use the 'm4_' prefix...
Anyway, I now have something which seems to work reasonably well.
Example peer file:
m4_define(`_peer',`templatenode')
m4_define(`_path',`templatenode.domain')
m4_define(`_incoming',`templatenode.domain')
m4_define(`_outgoing',`templatenode.domain')
m4_define(`_isbofh',1)
m4_define(`_isnet',1)
m4_define(`_groupat',`alt.sysadmin.recovery,\
alt.tech-support.recovery')
m4_include(proto.m4)
Yeah, I could probably move the include of proto.m4 off
somewhere else (like peers.m4), but this seems sane enough.
This is pretty close to what I'd originally had in mind
(which was actually more like news/mail items, with the
grouplist as the 'body'), but doesn't require me to write
anything to parse it.
And as it generates all three configs fairly cleanly, I'm
not at all unhappy...
Peter da Silva
entry just to get it into one file? You could do better with a little shell
script that just ran awk or sed over the file half a dozen times.
--
This is The Reverend Peter da Silva's Boring Sig File - there are no references
to Wolves, Kibo, Discordianism, or The Church of the Subgenius in this document
Executive Vice President, Corporate Communications, Entropy Gradient Reversals.
"[Deep Impact]: the tragedy of Bad Science hitting the planet." -- David Jacoby
Matt McLeod
>entry just to get it into one file? You could do better with a little shell
>script that just ran awk or sed over the file half a dozen times.
Since I don't have to edit most of the text it doesn't make
a great deal of difference. I'm not entirely sure which
variant you're commenting on, though -- the early stuff
was fairly boneheaded.
What I have now takes about 8 lines plus the group list
for bofh/2senet/misc-stuff, seperates all the feeds out
nicely, and so on. For a simple "everything that's not
restricted" feed it's 6 lines, one of which is not at
all configurable.
Setting up a new peer is now a case of:
* ./mkpeer.sh peername pathname
* edit any bits that need editing in the editor it brings up
* ./regen.sh
* copy configs into place
the last step can be removed if I wasn't paranoid to want to
check the results by hand before letting it near a live news
server...
Yes, there are other ways to do it. I've done it this way
partly because I wanted to play with m4.
I've got it sitting on hiro (aka news.netizen.com.au) now,
and it seems to be working OK -- got a few wrinkles to iron
out though.
--
/earth is 98% full ... please delete anyone you can.
Russ Allbery
> For example, using an "extended newsfeeds" file format:
> peer news.lspace.org {
> hostname: "news.lspace.org, library.lspace.org"
> groups: "net.*,bofh.*,alt.sysadmin.recovery"
> }
You know, if someone would like to spec out such a file format that would
incorporate the current newsfeeds, incoming.conf, and innfeed.conf, I'd
love to see that and would try to get it implemented. I keep having
problems deciding just what it should look like.