info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
INN2 artcutoff (was: Re: Old articles)
8 views
Skip to first unread message
yamo'
Jan 21, 2024, 3:30:13 AMJan 21
to
Hi,
[FU2 news.software.nntp]
Jesse Rehmer a tapoté :
>
> The administrator of news.corradoroberto.it was using suck, pulling from my
> server and those articles made their way back out through other peers of
> news.corradoroberto.it who accepted the old articles. The injection-bsd1 node
> is only in a handful of articles on my server, so it likely won't do you much
> good to filter based on that.
Yes, nothing found in innreport today!
>
> News administrators should have a proper article cutoff set on the servers
> which they peer with others. If only the administrator of
> news.corradoroberto.it made the mistake originally it would not have leaked
> beyond.
Yes good advice but sometimes the newsmaster has to modify "artcutoff"
in inn.conf and forget changing it after...
Maybe reporting the value of "artcutoff" in log and INNReport
would be a nice thing.
--
Stéphane
Sorry for my bad English
[FU2 news.software.nntp]
Jesse Rehmer a tapoté :
>
> The administrator of news.corradoroberto.it was using suck, pulling from my
> server and those articles made their way back out through other peers of
> news.corradoroberto.it who accepted the old articles. The injection-bsd1 node
> is only in a handful of articles on my server, so it likely won't do you much
> good to filter based on that.
Yes, nothing found in innreport today!
>
> News administrators should have a proper article cutoff set on the servers
> which they peer with others. If only the administrator of
> news.corradoroberto.it made the mistake originally it would not have leaked
> beyond.
Yes good advice but sometimes the newsmaster has to modify "artcutoff"
in inn.conf and forget changing it after...
Maybe reporting the value of "artcutoff" in log and INNReport
would be a nice thing.
--
Stéphane
Sorry for my bad English
Julien ÉLIE
Jan 21, 2024, 3:36:39 AMJan 21
to
Salut Stéphane,
> Maybe reporting the value of "artcutoff" in log and INNReport
> would be a nice thing.
It already is at the beginning of the daily report, in a cryptic form :)
Server status:
Server running
Allowing remote connections
Parameters c 10 i 50 (18) l 600000 o 1010 t 300 H 2 T 60 X 0 normal
specified
It is "c 10", the value of the cutoff parameter.
--
Julien ÉLIE
« Audentes fortunat iuvat. » (Virgile)
> Maybe reporting the value of "artcutoff" in log and INNReport
> would be a nice thing.
Server status:
Server running
Allowing remote connections
Parameters c 10 i 50 (18) l 600000 o 1010 t 300 H 2 T 60 X 0 normal
specified
It is "c 10", the value of the cutoff parameter.
--
Julien ÉLIE
« Audentes fortunat iuvat. » (Virgile)
Nigel Reed
Jan 21, 2024, 4:56:12 AMJan 21
to
news admins would want to receive them. I guess I am relying on my
peers not to accept old articles if they dont want them should I get
them from other sources.
I still intend to backfill a number of my groups, I'm just nervous
about propagating old articles while still being able to receive and
send new ones.
--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23
Julien ÉLIE
Jan 21, 2024, 5:27:40 AMJan 21
to
Hi Nigel,
> I still intend to backfill a number of my groups, I'm just nervous
> about propagating old articles while still being able to receive and
> send new ones.
Did you try to add a fake hop when using pullnews? (-F flag)
https://www.eyrie.org/~eagle/software/inn/docs/pullnews.html
"""
In case you have running peers and don't want to propagate them the
articles you are pulling from upstream servers, you should add a fake
hop with the -F flag to all the pulled articles, and add that very fake
hop in the exclusion sub-field of all the sites configured in your
newsfeeds file. (For example, using "pullnews -F myserverimported",
change "sitename:*:Tm:innfeed!" to
"sitename/myserverimported:*:Tm:innfeed!" for every sitename in
newsfeeds you don't want to feed the pulled articles to.)
"""
This way, you will never be nervous again when running pullnews :-)
P.-S.: If someone has a better or quicker way to prevent pulled articles
from being sent to peers, please tell, and I'll update the pullnews
documentation accordingly.
I don't know whether suck has a similar facility.
> I still intend to backfill a number of my groups, I'm just nervous
> about propagating old articles while still being able to receive and
> send new ones.
https://www.eyrie.org/~eagle/software/inn/docs/pullnews.html
"""
In case you have running peers and don't want to propagate them the
articles you are pulling from upstream servers, you should add a fake
hop with the -F flag to all the pulled articles, and add that very fake
hop in the exclusion sub-field of all the sites configured in your
newsfeeds file. (For example, using "pullnews -F myserverimported",
change "sitename:*:Tm:innfeed!" to
"sitename/myserverimported:*:Tm:innfeed!" for every sitename in
newsfeeds you don't want to feed the pulled articles to.)
"""
This way, you will never be nervous again when running pullnews :-)
P.-S.: If someone has a better or quicker way to prevent pulled articles
from being sent to peers, please tell, and I'll update the pullnews
documentation accordingly.
I don't know whether suck has a similar facility.
Jesse Rehmer
Jan 21, 2024, 5:51:40 AMJan 21
to
On Jan 21, 2024 at 4:27:39 AM CST, "Julien ÉLIE"
With suck you need to use the post filter function. From the man page:
"POST FILTER
if the -y post_filter option is specified on the command line in
conjunction with any of the batch modes, then suck will call the post
filter specified, after downloading the articles, and before
batching/posting the articles. The filter is passed the directory
where the articles are stored (the -dm option). The filter program is
responsible for parsing the contents of the directory. See
sample/post_filter.pl for a sample post filter. This option was
designed to allow you to add your own host name to the Path: header,
but if you need to do anything else to the messages, you can."
The sample post_filter.pl:
#!/usr/local/bin/perl
# sample program for editting/changing the downloaded articles.
# this program is passed the directory where the articles are
# stored
# it modifies the Path: header, so you can add your own unique
# host to it, so you can tell INN not to upload these articles
opendir (DIRP, $ARGV[0]) or die "Can't open $ARGV[0]";
# get list of files, skipping hidden files
@files = grep ( !/^\./, readdir(DIRP));
foreach $file ( @files) {
# read the file in
$path = "${ARGV[0]}/${file}";
open FIP, "<${path}" or die "Can't read ${path}\n";@file = <FIP>;
close FIP;
# find the line and change it
foreach $line ( @file) {
if ( $line =~ /^Path: /) {
$line =~ s/^Path: /Path: myhost\!/
}
}
# save it back out
open FIP, ">${path}" or die "Can't write to ${path}\n";
print FIP @file;
close FIP;
}
"POST FILTER
if the -y post_filter option is specified on the command line in
conjunction with any of the batch modes, then suck will call the post
filter specified, after downloading the articles, and before
batching/posting the articles. The filter is passed the directory
where the articles are stored (the -dm option). The filter program is
responsible for parsing the contents of the directory. See
sample/post_filter.pl for a sample post filter. This option was
designed to allow you to add your own host name to the Path: header,
but if you need to do anything else to the messages, you can."
The sample post_filter.pl:
#!/usr/local/bin/perl
# sample program for editting/changing the downloaded articles.
# this program is passed the directory where the articles are
# stored
# it modifies the Path: header, so you can add your own unique
# host to it, so you can tell INN not to upload these articles
opendir (DIRP, $ARGV[0]) or die "Can't open $ARGV[0]";
# get list of files, skipping hidden files
@files = grep ( !/^\./, readdir(DIRP));
foreach $file ( @files) {
# read the file in
$path = "${ARGV[0]}/${file}";
open FIP, "<${path}" or die "Can't read ${path}\n";@file = <FIP>;
close FIP;
# find the line and change it
foreach $line ( @file) {
if ( $line =~ /^Path: /) {
$line =~ s/^Path: /Path: myhost\!/
}
}
# save it back out
open FIP, ">${path}" or die "Can't write to ${path}\n";
print FIP @file;
close FIP;
}
Julien ÉLIE
Jan 21, 2024, 8:13:09 AMJan 21
to
Hi Jesse,
>> I don't know whether suck has a similar facility.
>
> With suck you need to use the post filter function.
[...]
So the same tip of excluding articles with that fake hop from being sent
to outgoing peers can be used.
--
Julien ÉLIE
« Dans un mois, dans un an, comment souffrirons-nous ? » (Jean Racine)
>> I don't know whether suck has a similar facility.
>
> With suck you need to use the post filter function.
> # find the line and change it
> foreach $line ( @file) {
> if ( $line =~ /^Path: /) {
> $line =~ s/^Path: /Path: myhost\!/
> }
> }
OK, thanks for the confirmation.
> foreach $line ( @file) {
> if ( $line =~ /^Path: /) {
> $line =~ s/^Path: /Path: myhost\!/
> }
> }
So the same tip of excluding articles with that fake hop from being sent
to outgoing peers can be used.
--
Julien ÉLIE
« Dans un mois, dans un an, comment souffrirons-nous ? » (Jean Racine)
llp
Jan 21, 2024, 10:29:39 AMJan 21
to
Nigel Reed <sy...@endofthelinebbs.com> composa la prose suivante:
>I'm actually fine accepting old articles but understand that not all
>news admins would want to receive them. I guess I am relying on my
>peers not to accept old articles if they dont want them should I get
>them from other sources.
>
>I still intend to backfill a number of my groups, I'm just nervous
>about propagating old articles while still being able to receive and
>send new ones.
Wouldn't it be interesting to have an inn2 configuration option that
prevents the propagation of old articles to other servers by default ?
--
Liste de serveurs offrant un accès gratuit à la hiérarchie FR.*
http://usenet.ovh/?article=faq_serveur_gratuit
Recherche d'article Usenet
http://usenet.ovh/?article=ual
>I'm actually fine accepting old articles but understand that not all
>news admins would want to receive them. I guess I am relying on my
>peers not to accept old articles if they dont want them should I get
>them from other sources.
>
>I still intend to backfill a number of my groups, I'm just nervous
>about propagating old articles while still being able to receive and
>send new ones.
prevents the propagation of old articles to other servers by default ?
--
Liste de serveurs offrant un accès gratuit à la hiérarchie FR.*
http://usenet.ovh/?article=faq_serveur_gratuit
Recherche d'article Usenet
http://usenet.ovh/?article=ual
yamo'
Jan 22, 2024, 4:44:39 AMJan 22
to
Salut Julien,
Julien ÉLIE a tapoté :
> > Maybe reporting the value of "artcutoff" in log and INNReport
> > would be a nice thing.
>
> It already is at the beginning of the daily report, in a cryptic form :)
>
> Server status:
> Server running
> Allowing remote connections
> Parameters c 10 i 50 (18) l 600000 o 1010 t 300 H 2 T 60 X 0 normal
> specified
>
> It is "c 10", the value of the cutoff parameter.
Oh yes, the report in text mode but I prefer the html form.
I found in inn_status.html :
Cutoff: 10 days
--
Stéphane
Julien ÉLIE a tapoté :
> > Maybe reporting the value of "artcutoff" in log and INNReport
> > would be a nice thing.
>
> It already is at the beginning of the daily report, in a cryptic form :)
>
> Server status:
> Server running
> Allowing remote connections
> Parameters c 10 i 50 (18) l 600000 o 1010 t 300 H 2 T 60 X 0 normal
> specified
>
> It is "c 10", the value of the cutoff parameter.
I found in inn_status.html :
Cutoff: 10 days
--
Stéphane
Nigel Reed
Jan 22, 2024, 6:28:53 PMJan 22
to
On Sun, 21 Jan 2024 11:27:39 +0100
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Hi Nigel,
>
> > I still intend to backfill a number of my groups, I'm just nervous
> > about propagating old articles while still being able to receive and
> > send new ones.
>
> Did you try to add a fake hop when using pullnews? (-F flag)
> https://www.eyrie.org/~eagle/software/inn/docs/pullnews.html
>
> """
> In case you have running peers and don't want to propagate them the
> articles you are pulling from upstream servers, you should add a fake
> hop with the -F flag to all the pulled articles, and add that very
> fake hop in the exclusion sub-field of all the sites configured in
> your newsfeeds file. (For example, using "pullnews -F
> myserverimported", change "sitename:*:Tm:innfeed!" to
> "sitename/myserverimported:*:Tm:innfeed!" for every sitename in
> newsfeeds you don't want to feed the pulled articles to.)
> """
To be clear, I only need to do this with my peer connections right, not
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Hi Nigel,
>
> > I still intend to backfill a number of my groups, I'm just nervous
> > about propagating old articles while still being able to receive and
> > send new ones.
>
> Did you try to add a fake hop when using pullnews? (-F flag)
> https://www.eyrie.org/~eagle/software/inn/docs/pullnews.html
>
> """
> In case you have running peers and don't want to propagate them the
> articles you are pulling from upstream servers, you should add a fake
> hop with the -F flag to all the pulled articles, and add that very
> fake hop in the exclusion sub-field of all the sites configured in
> your newsfeeds file. (For example, using "pullnews -F
> myserverimported", change "sitename:*:Tm:innfeed!" to
> "sitename/myserverimported:*:Tm:innfeed!" for every sitename in
> newsfeeds you don't want to feed the pulled articles to.)
> """
ME or inpath or innfeed, controlchan etc.
news.bbs.nz/news.bbs.nz,nomilktodaythankyou\
:*,@*.bina*,@*.bain*,@*.dateien*,@*.pictures*,$LOCAL\
:Ap,Tm:innfeed!
I've added "nomilktodaythankyou" to all my peers, similar to this. and
will use -F nomilktodaythankyou when using pullnews so I guess this is
fair warning to my peers :)
Now, who has a 10 year backlog of the soc.* I can slurp for practice? :)
Jesse Rehmer
Jan 22, 2024, 7:56:35 PMJan 22
to
On Jan 22, 2024 at 5:28:52 PM CST, "Nigel Reed" <sy...@endofthelinebbs.com>
wrote:
Anyone is free to suck/pull from my server: news.blueworldhosting.com
Big8 hierarchies go back 20-30 years depending on the group.
Reading is open on 119 or 563 via TLS.
wrote:
Big8 hierarchies go back 20-30 years depending on the group.
Reading is open on 119 or 563 via TLS.
yamo'
Jan 23, 2024, 4:09:31 AMJan 23
to
Hi,
Jesse Rehmer a tapoté :
how simply sorting by date the articles?
The old posts sucked will be showned at first to NNRP client?
Or creating an old.* local hierarchie to show this olds posts?
--
Stéphane
Jesse Rehmer a tapoté :
>
> Anyone is free to suck/pull from my server: news.blueworldhosting.com
>
> Big8 hierarchies go back 20-30 years depending on the group.
>
> Reading is open on 119 or 563 via TLS.
Maybe it is in the faq or in the history of news.software.nntp but
> Anyone is free to suck/pull from my server: news.blueworldhosting.com
>
> Big8 hierarchies go back 20-30 years depending on the group.
>
> Reading is open on 119 or 563 via TLS.
how simply sorting by date the articles?
The old posts sucked will be showned at first to NNRP client?
Or creating an old.* local hierarchie to show this olds posts?
--
Stéphane
Nigel Reed
Jan 23, 2024, 1:12:46 PMJan 23
to
thanks,
Nigel
Julien ÉLIE
Jan 23, 2024, 1:48:27 PMJan 23
to
Hi Nigel,
> To be clear, I only need to do this with my peer connections right, not
> ME or inpath or innfeed, controlchan etc.
You should do that for your outgoing peers and I think the inpaths!
entry too so as not to send stats to the top1000 project about your
donwloaded articles.
Other entries do not need the exclusion.
--
Julien ÉLIE
« My wife and I were happy for twenty years. Then we met. » (Henny
Youngman)
> To be clear, I only need to do this with my peer connections right, not
> ME or inpath or innfeed, controlchan etc.
entry too so as not to send stats to the top1000 project about your
donwloaded articles.
Other entries do not need the exclusion.
--
Julien ÉLIE
« My wife and I were happy for twenty years. Then we met. » (Henny
Youngman)
Julien ÉLIE
Jan 23, 2024, 2:25:19 PMJan 23
to
Salut Stéphane,
> how simply sorting by date the articles?
> The old posts sucked will be showned at first to NNRP client?
That's how article numbers work, indeed. They increment when a new
article arrives. Old posts will then have a higher article number than
your current articles in the newsgroup.
You'll then have to renumber your newsgroups, which can be done by
re-feeding your spool to a fresh new instance. Naturally, the re-feed
is done by article posting date, so that they arrive in order.
You may find useful information in the "Renumber newsgroups by post
date" thread in news.admin.peering in April 2023
(<7Qb1M.3786652$nwq3.3...@fx13.ams4>).
> Or creating an old.* local hierarchie to show this olds posts?
A quicker solution, yes!
--
Julien ÉLIE
« Les amis de la vérité sont ceux qui la cherchent, et non ceux qui se
vantent de l'avoir trouvée. » (Condorcet)
> how simply sorting by date the articles?
> The old posts sucked will be showned at first to NNRP client?
article arrives. Old posts will then have a higher article number than
your current articles in the newsgroup.
You'll then have to renumber your newsgroups, which can be done by
re-feeding your spool to a fresh new instance. Naturally, the re-feed
is done by article posting date, so that they arrive in order.
You may find useful information in the "Renumber newsgroups by post
date" thread in news.admin.peering in April 2023
(<7Qb1M.3786652$nwq3.3...@fx13.ams4>).
> Or creating an old.* local hierarchie to show this olds posts?
--
Julien ÉLIE
« Les amis de la vérité sont ceux qui la cherchent, et non ceux qui se
vantent de l'avoir trouvée. » (Condorcet)
Nigel Reed
Jan 23, 2024, 2:40:53 PMJan 23
to
On Tue, 23 Jan 2024 20:25:18 +0100
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Salut Stéphane,
>
> > how simply sorting by date the articles?
> > The old posts sucked will be showned at first to NNRP client?
>
> That's how article numbers work, indeed. They increment when a new
> article arrives. Old posts will then have a higher article number
> than your current articles in the newsgroup.
>
> You'll then have to renumber your newsgroups, which can be done by
> re-feeding your spool to a fresh new instance. Naturally, the
> re-feed is done by article posting date, so that they arrive in order.
>
> You may find useful information in the "Renumber newsgroups by post
> date" thread in news.admin.peering in April 2023
> (<7Qb1M.3786652$nwq3.3...@fx13.ams4>).
Is thre no way to renumber a group without having to feed to another
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Salut Stéphane,
>
> > how simply sorting by date the articles?
> > The old posts sucked will be showned at first to NNRP client?
>
> That's how article numbers work, indeed. They increment when a new
> article arrives. Old posts will then have a higher article number
> than your current articles in the newsgroup.
>
> You'll then have to renumber your newsgroups, which can be done by
> re-feeding your spool to a fresh new instance. Naturally, the
> re-feed is done by article posting date, so that they arrive in order.
>
> You may find useful information in the "Renumber newsgroups by post
> date" thread in news.admin.peering in April 2023
> (<7Qb1M.3786652$nwq3.3...@fx13.ams4>).
news server? You'd think there'd be a script to do that or something.
If that's the case, would it not be easier to stop accepting from
peers, delete all the articles and then refeed?
It seems everytime I contemplate importing older articles, there's
another complication.
Could a script not just get all the posting dates and article numbers,
I'm sure there's more to it than that tho...
Russ Allbery
Jan 23, 2024, 2:50:03 PMJan 23
to
Nigel Reed <sy...@endofthelinebbs.com> writes:
> Is thre no way to renumber a group without having to feed to another
> news server? You'd think there'd be a script to do that or something.
There is not. Unfortunately, article numbers get very deeply encoded into
> Is thre no way to renumber a group without having to feed to another
> news server? You'd think there'd be a script to do that or something.
the data storage for INN in ways that are hard to change. In theory, one
could write such a script, but for example you have to change the Xref
header within the stored article itself and I think you may have to change
the storage token for the article and thus the history entry, in addition
to the more obvious things like the overview database.
--
Russ Allbery (ea...@eyrie.org) <https://www.eyrie.org/~eagle/>
Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.
Roberto CORRADO
Jan 23, 2024, 4:01:59 PMJan 23
to
Nigel Reed wrote:
> news.bbs.nz/news.bbs.nz,nomilktodaythankyou\
> :*,@*.bina*,@*.bain*,@*.dateien*,@*.pictures*,$LOCAL\
> :Ap,Tm:innfeed!
I would like to open the newsfeeds again, but I'm anxious to
> news.bbs.nz/news.bbs.nz,nomilktodaythankyou\
> :*,@*.bina*,@*.bain*,@*.dateien*,@*.pictures*,$LOCAL\
> :Ap,Tm:innfeed!
make a mistake again. Can you tell me if the following
instructions are correct?
"""
newsfeed.host.tld/newsfeed.host.tld,xyz\
:*,!suckgroup{n}.*,!alt.b,!*.bina*,!*.bain*,!*.dateien*,!*.pictures*,!junk/!local\
:Af,Ap,Tm:innfeed!
"""
replace !suckgroup{n}.* with a group pull with suck
I thank you all and at the moment I think I would have a lot to do.
-Roberto
Nigel Reed
Jan 24, 2024, 2:37:55 AMJan 24
to
On Tue, 23 Jan 2024 19:48:25 +0100
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Hi Nigel,
>
> > To be clear, I only need to do this with my peer connections right,
> > not ME or inpath or innfeed, controlchan etc.
>
> You should do that for your outgoing peers and I think the inpaths!
> entry too so as not to send stats to the top1000 project about your
> donwloaded articles.
> Other entries do not need the exclusion.
>
My plan was to:
Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> wrote:
> Hi Nigel,
>
> > To be clear, I only need to do this with my peer connections right,
> > not ME or inpath or innfeed, controlchan etc.
>
> You should do that for your outgoing peers and I think the inpaths!
> entry too so as not to send stats to the top1000 project about your
> donwloaded articles.
> Other entries do not need the exclusion.
>
1. Create a new inn instance on a new server
2. Suck in all the articles for all the groups the peer has
3. stop the current news server
4. re-pull to make sure I have all the latest articles
5. delete the spool on the existing server
6. stop inn in the new server and export the /spool directory (hence
the NFS question elsewhere)
7. bring up the server and everything is good in the world.
Now I'm thinking about it. This wont work because I have other groups
that are not carried by the peer I'll be pulling from.
I could just copy those directories and rebuild the index I guess but I
don't fancy that on a TB of data.
So after suggestions on how to get the two merged in the shortest
possible downtime and loss of articles.
The existing news server does not and will not have the disk space, and
as I mentioned before I'd really like to keep inn running there and
just store the data on the other server since it'll have less CPU and
memory.
All suggestions are good.
0 new messages