Question about INN expire

[Jesse Rehmer]

I'm trying to expire a bunch of crap that is crossposted to *.test groups. I
have the following in expire.ctl:

*:A:1:never:never
*.jobs*:AX:1:90:90
news.lists.filters:A:1:10:10
*.test:AX:1:100:100

When I run "expire -p" it does expire articles, but articles that have
Newsgroups headers like the one below that are crossposted to other groups
where alt.test is not the primary remain in the history and on the spool. I
thought adding the "X" flag to the expire.ctl line would take care of this,
but these articles still remain.

Date: Sat, 11 Apr 2009 00:22:55 GMT

I assume from the following portion of the expire.ctl man page that these
articles should be expiring, but perhaps I am misinterpreting this:

An expiration policy is applied to every article in a newsgroup it
matches. There is no way to set an expiration policy for articles
crossposted to groups you don't carry that's different than other
articles in the same group. Normally, articles are not completely
deleted until they expire out of every group to which they were posted,
but if an article is expired following a rule where <flag> contains
"X", it is deleted out of all newsgroups to which it was posted
immediately.

I in the example above the article is posted to rec.arts.disney.parks and
crossposted to alt.test, and based on "An expiration policy is applied to
every article in a newsgroup it matches", I assume since the article exists in
alt.test and I have the "X" flag that it be expired but this does not seem to
be the case.

Regards,

Jesse

[Jesse Rehmer]

Some of the headers that I pasted in my original message didn't make it
through to the posted message. I may be celebrating too hard while computing
this evening. ;)

Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
alt.test:8601

[Jesse Rehmer]

Hmmm or maybe my news client was stripping them, trying from slrn this
time:

Newsgroups: rec.arts.disney.parks,alt.test
Date: Sat, 11 Apr 2009 00:53:23 GMT

[Jesse Rehmer]

On Jul 3, 2022 at 9:08:17 PM CDT, "Jesse Rehmer" in

Also, shouldn't I be getting more output?

[news@spool1 ~]$ expire -p -v 5
Article lines processed 32971875
Articles retained 32966677
Entries expired 5198

[Julien ÉLIE]

Hi Jesse,

>> Newsgroups: rec.arts.disney.parks,alt.test
>> Date: Sat, 11 Apr 2009 00:53:23 GMT
>> Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
>> alt.test:8601

What's the posting date INN has for this article?
If you have its Message-ID, look at the last number in the result of
grephistory:

% grephistory -l
'<983023301...@freebsd-inject1.usenet.blueworldhosting.com>'
[A6E3D4975ED59BD167F5380850829D76] 1656898303~-~1656898302
@020162C242FF001200000000000000000000@

% convdate -dc 1656898302
Mon, 4 Jul 2022 01:31:42 -0000 (UTC)

> Also, shouldn't I be getting more output?
>
> [news@spool1 ~]$ expire -p -v 5
> Article lines processed 32971875
> Articles retained 32966677
> Entries expired 5198

"-v 5" should definitely be far more verbose, printing something for
each article line processed...

--
Julien ÉLIE

« C'est la goutte qui fait déborder l'amphore ! » (Assurancetourix)

[Jesse Rehmer]

On Jul 4, 2022 at 4:27:11 PM CDT, "Julien ÉLIE" in
<t9vlvf$17rri$1...@news.trigofacile.com> wrote:

> Hi Jesse,

>
> What's the posting date INN has for this article?
> If you have its Message-ID, look at the last number in the result of
> grephistory:
>
> % grephistory -l
> '<983023301...@freebsd-inject1.usenet.blueworldhosting.com>'
> [A6E3D4975ED59BD167F5380850829D76] 1656898303~-~1656898302
> @020162C242FF001200000000000000000000@
>
> % convdate -dc 1656898302
> Mon, 4 Jul 2022 01:31:42 -0000 (UTC)
>
>
>> Also, shouldn't I be getting more output?
>>
>> [news@spool1 ~]$ expire -p -v 5
>> Article lines processed 32971875
>> Articles retained 32966677
>> Entries expired 5198
>
> "-v 5" should definitely be far more verbose, printing something for
> each article line processed...

I ran expire several times with '-v 2' through '-v 5' and never get more
output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
large tradspool for most articles and a single CNFS buffer for binary
articles.

Seems it is ignoring the '-p' flag or is not acting on the right timestamp:

[news@spool1 ~]$ grephistory -l
'<Xns9BE9D482BF422lk...@207.115.33.102>'
[AA4B2314FF33940366F138FB4012C2BA] 1655680239~-~1239411203
@0500000091C500000000000032DA00000000@

[news@spool1 ~]$ convdate -dc 1239411203
Sat, 11 Apr 2009 00:53:23 -0000 (UTC)

[news@spool1 ~]$ convdate -dc 1655680239
Sun, 19 Jun 2022 23:10:39 -0000 (UTC)

I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
a test but same results.

Cheers,
Jesse

[Julien ÉLIE]

Hi Jesse,

> I ran expire several times with '-v 2' through '-v 5' and never get more
> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
> large tradspool for most articles and a single CNFS buffer for binary
> articles.
>

> I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
> to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
> a test but same results.

I've tested a bit.
In fact, the flags are correctly seen. The -v flag works fine but the
-p flag does not seem to do what is expected, and also maybe without -p
(the part of the code where expire removes articles does not look right
to me, using conditions related to groupbaseexpiry and whether the
storage method self expires).
I have to look at it deeper.

--
Julien ÉLIE

« Inside every large problem is a small problem struggling to get out. »

[Jesse Rehmer]

On Jul 6, 2022 at 6:25:44 PM CDT, "Julien ÉLIE" in

<ta55lp$1bii0$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>> I ran expire several times with '-v 2' through '-v 5' and never get more
>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>> large tradspool for most articles and a single CNFS buffer for binary
>> articles.
>>
>> I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
>> to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
>> a test but same results.
>
> I've tested a bit.
> In fact, the flags are correctly seen. The -v flag works fine but the
> -p flag does not seem to do what is expected, and also maybe without -p
> (the part of the code where expire removes articles does not look right
> to me, using conditions related to groupbaseexpiry and whether the
> storage method self expires).
> I have to look at it deeper.

If there is anything I can do to help assist, let me know. I do notice odd
behavior, such as running expire back to back results in a few thousand
articles being expired each time. I wouldn't expect so many articles to expire
in such a short time, but since I cannot get more output I don't know what it
is doing exactly.

What conditions did you run expire where the -v flag was honored? I've ran
expire the following ways and only get three lines of output:

expire -v2 -p
expire -v 2 -p
expire -p -v2
expire -p -v 2
expire -v5 -p
expire -v 5 -p
expire -p -v5
expire -p -v 5
news.daily noexpireover flags="-v5 -p"

[Julien ÉLIE]

Hi Jesse,

> If there is anything I can do to help assist, let me know. I do notice odd
> behavior, such as running expire back to back results in a few thousand
> articles being expired each time. I wouldn't expect so many articles to expire
> in such a short time, but since I cannot get more output I don't know what it
> is doing exactly.

Wouldn't it be articles contained in CNFS buffers that are wrapping?
Then each run of expire will mention the overwritten articles as expired.

> What conditions did you run expire where the -v flag was honored?

Simply use "-v 0" and you'll no longer see any output :-)


I've investigated more today, and I believe there's still a bit of
documentation to highlight more.
I discovered a few things (I still had not had a chance to have a closer
look at how expire really works).

I've played a bit with expire, adding a few lines of code and trying a
few things. I obtained errors of class definition missings (which are
the classes numbers in storage.conf).
And then I found out that expire does not do much work when
groupbaseexpiry is true. It does not even parse expire.ctl when using
"<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
is false).
And there's a note in the code saying:
/* don't process this line -- per-group expiry is done by expireover */


Ah ah, it becomes clearer now.
From inn.conf man page:

groupbaseexpiry
Whether to enable newsgroup-based expiry. If set to false, article
expiry is done based on storage class of storing method. If set to
true (and overview information is available), expiry is done by
newsgroup name.

Note the "and overview information is available".


So basically, if you want the behaviour you explained in the first
message of this thread:

* either you need to set groupbaseexpiry to false, rewrite expire.ctl
accordingly, and then "expire -p -v 3" will work as expected.
It would mean that you have storage classes definitions suiting your
needs of expiration (as it will be by class), so in fact the classes
should be done with a sorting by newsgroups: one class for "*.jobs*",
another class for "news.lists.filters", another for "*.test", etc.

* either you keep groupbaseexpiry to true, and then it means overview
data is needed.
In your previous article, you mentioned:

news.daily noexpireover flags="-v5 -p"

Does it mean you do not have overview data?

If no, then you need creating it or switching to groupbaseexpiry set to
false...

And if you have overview data, the Key is to run:

expireover -e -N -p

-e: Remove articles from the news spool and all overview databases as
soon as they expire out of any newsgroup to which they are posted

-N: Apply expire.ctl rules to expire articles even from storage methods
that have self-expire functionality

-p: Expiration decisions are based on the article posting date


and update your cron line:
news.daily expireover expireoverflags='-e -N -p' flags='-N -v1'







I agree that the expire man page is clearly misleading about that, and
should clearly explain that. I'll have a look at where to put such
information (several man pages are concerned).

If someone has a different understanding of how expire/expireover work
with regards to withgroupbaseexpiry, do not hesitate to react!
I may as well be misunderstanding something.

--
Julien ÉLIE

« Love is blind but marriage is an eye-opener. »

[Julien ÉLIE]

Hi Jesse,

>>> I ran expire several times with '-v 2' through '-v 5' and never get more
>>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>>> large tradspool for most articles and a single CNFS buffer for binary
>>> articles.
>

> If there is anything I can do to help assist, let me know.

"expireover -e -N -p" should do what you want (see my previous response).

I've a bit improved expire for logging and, above all, with a few comments.

I don't know whether you could try the following patch easily (you will have to rebuild INN and run "make update") but in case you want to test it, feel free :-)



--- a/expire/expire.c
+++ b/expire/expire.c
@@ -372,6 +372,7 @@ EXPremove(const TOKEN *token)

/*
** Do the work of expiring one line.
+** Returns true when the article should be kept for the time being.
*/
static bool
EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,
@@ -389,12 +390,16 @@ EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,
HasSelfexpire = true;
Selfexpired = true;
} else {
- /* the article is still alive */
+ /* The article is still alive, free it. */
SMfreearticle(article);
+ /* Per-group expiry is done by expireover, remember!
+ * That's why if groupbaseexpiry is set, expire won't verify
+ * the expiration rules set in expire.ctl. */
if (innconf->groupbaseexpiry || !Ignoreselfexpire)
HasSelfexpire = true;
}
}
+
if (EXPusepost && posted != 0)
when = posted;
else
@@ -403,19 +408,29 @@ EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,

if (HasSelfexpire) {
if (Selfexpired || token->type == TOKEN_EMPTY) {
+ if (EXPverbose > 3)
+ printf("%s (to remember or forget, not existing)\n",
+ TokenToText(*token));
EXPallgone++;
r = false;
} else {
+ if (EXPverbose > 3)
+ printf("%s (to keep, may self-expire)\n",
+ TokenToText(*token));
EXPstillhere++;
r = true;
}
} else {
kr = EXPkeepit(token, when, expires);
if (kr == Remove) {
+ if (EXPverbose > 3)
+ printf("%s (to remember or forget)\n", TokenToText(*token));
EXPremove(token);
EXPallgone++;
r = false;
} else {
+ if (EXPverbose > 3)
+ printf("%s (to keep)\n", TokenToText(*token));
EXPstillhere++;
r = true;
}
@@ -699,7 +714,7 @@ main(int ac, char *av[])
if (!Bad && NHistory != NULL) {
snprintf(buff, sizeof(buff), "%s.n.done", NHistory);
fclose(EXPfopen(false, buff, "w", true, Server, false));
- CleanupAndExit(Server, false, Bad ? 1 : 0);
+ CleanupAndExit(Server, true, 0);
}

CleanupAndExit(Server, !Bad, Bad ? 1 : 0);





You can use "-x -t" for a dry-run of expire.



expire -x -t -v4 > exp.log

will output in exp.log tons of lines like:

@030346523100000000000000002600000001@ (to keep, may self-expire)

and

grep -v will exp.log

will tell you the tokens expire will remove (or remember depending on the setting of /remember/ in expire.ctl):

@03034652330000000000000F2EBE00000001@ (to remember or forget, not existing)

These tokens no longer correspond to an existing stored article (either the CNFS buffer wrapped or the article was cancelled).

--
Julien ÉLIE

« – À la plage ? Mais il pleut !
– Pas du tout ! Dans le midi de la Gaule, il pleut. Ici, c'est tout
juste un peu humide. Vivifiant. Pas vrai, Astérix ?
– Ce matin, ça devient de plus en plus vivifiant ! » (Astérix)

[Julien ÉLIE]

Adding to my previous message:

> You can use "-x -t" for a dry-run of expire.
> expire -x -t -v4 > exp.log

If you run that without the patch I gave (which changes a boolean in a
call to CleanupAndExit), your news server will be paused after the run.
So you should add "-n" to the command (which prevents the pause).

Otherwise, run:

ctlinnd mode

and you'll see the reason to use in the first line "Expiring process
XXXX". Then:

ctlinnd go 'Expiring process XXXX'

and your news server will be running again.

[Jesse Rehmer]

On Jul 7, 2022 at 1:28:57 PM CDT, "Julien ÉLIE" in

<ta78l9$1d4m6$1...@news.trigofacile.com> wrote:

> Simply use "-v 0" and you'll no longer see any output :-)

I want MORE output, not less. With any -v flag above '1' I only get three
lines of output, the same three lines from '-v 1'.

I'm going to have to take more time later to digest this further. I do have
overview data, but I don't run expireover very often because on my spool it
takes more than a day and causes the locking problems I brought up recently,
but I was attempting to see how much space I could reclaim by getting rid of
stuff cross posted to *.test using expire.

[Jesse Rehmer]

On Jul 7, 2022 at 2:33:53 PM CDT, "Jesse Rehmer" in

<224323085...@freebsd-inject1.usenet.blueworldhosting.com> wrote:

> I want MORE output, not less. With any -v flag above '1' I only get three
> lines of output, the same three lines from '-v 1'.

Perhaps I am not articulating this well, but I expect more output from these
commands:

$ expire -v 3 -n -t
Expire tracing: history files not updated.
Article lines processed 206532
Articles retained 111477
Entries expired 95055

$ expire -v 5 -n -t
Expire tracing: history files not updated.
Article lines processed 206534
Articles retained 111479
Entries expired 95055


The above is from a different INN 2.7.0rc1 box than I've been having issues
with, just to test that it wasn't a single server issue.

[Julien ÉLIE]

Hi Jesse,

>> I want MORE output, not less. With any -v flag above '1' I only get three
>> lines of output, the same three lines from '-v 1'.
>
> Perhaps I am not articulating this well, but I expect more output from these
> commands:
>
> $ expire -v 3 -n -t
> Expire tracing: history files not updated.
> Article lines processed 206532
> Articles retained 111477
> Entries expired 95055

Yes I have well understood what you want with "-v 3".

When I said to try "-v 0", it was in response to your question of how I
was sure the arguments were read.

As for MORE verbose logs, unfortunately, as I wrote a few minutes ago,
they are available only when groupbaseexpiry is false.
I sent a patch to add more logs (which will now appear with
groupbaseexpiry set to true).

> The above is from a different INN 2.7.0rc1 box than I've been having issues
> with, just to test that it wasn't a single server issue.

No, that's not a single server issue.
I have the same behaviour on mine.

[Julien ÉLIE]

Hi Jesse,

> I'm going to have to take more time later to digest this further.

Yes, take your time!

> I do have
> overview data, but I don't run expireover very often because on my spool it
> takes more than a day and causes the locking problems I brought up recently

Ah yes, I remember.
Did you try to run the command I suggested to expire the overview of
only a subset of groups?

For your tests groups for instance:

grep -E '\.test ' <pathdb>/active | expireover -f - -Z <pathtmp>/lowmark

Or several groups:

grep -E 'news.lists.filters |\.jobs|\.test ' <pathdb>/active \
| expireover -f - -Z <pathtmp>/lowmark

And then:

ctlinnd lowmark <pathtmp>/lowmark
rm -f <pathtmp>/lowmark



How long does it take with only 1 populated group?
Do for instance:

echo "rec.arts.tv" | expireover -f - -Z <pathtmp>/lowmark

I'm still surprised that it takes so long for tradindexed.

> but I was attempting to see how much space I could reclaim by getting rid of
> stuff cross posted to *.test using expire.

Unfortunately, expire won't do that when groupbaseexpiry is set to true.

--
Julien ÉLIE

« Dites, ne vendez tout de même pas la peau du sanglier avant de l'avoir
tué ! » (Détritus)

[Jesse Rehmer]

On Jul 7, 2022 at 1:52:25 PM CDT, "Julien ÉLIE" in

<ta7a19$1d5ke$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>>> I ran expire several times with '-v 2' through '-v 5' and never get more
>>>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>>>> large tradspool for most articles and a single CNFS buffer for binary
>>>> articles.
>>
>> If there is anything I can do to help assist, let me know.
>
> "expireover -e -N -p" should do what you want (see my previous response).
>
> I've a bit improved expire for logging and, above all, with a few comments.
>
> I don't know whether you could try the following patch easily (you will have
> to rebuild INN and run "make update") but in case you want to test it, feel
> free :-)

That expireover command should do what I'm after without having to change
groupbaseexpiry to false? Or do I still need to set groupbaseexpiry to false,
setup the various storage classes, and change expire.ctl accordingly?

[Julien ÉLIE]

Hi Jesse,

>> "expireover -e -N -p" should do what you want (see my previous response).
>

> That expireover command should do what I'm after without having to change
> groupbaseexpiry to false?

Exactly!
And you can even try it on 1 newsgroup (or more) with the example of
echo or grep active piped to it.

> Or do I still need to set groupbaseexpiry to false,
> setup the various storage classes, and change expire.ctl accordingly?

No need to do change anything; groupbaseexpiry set to false was to make
expire work as you wanted (without expireover).

[Jesse Rehmer]

On Jul 7, 2022 at 3:26:03 PM CDT, "Julien ÉLIE" in

<ta7fgr$1d8cj$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> That expireover command should do what I'm after without having to change
>> groupbaseexpiry to false?
>
> Exactly!
> And you can even try it on 1 newsgroup (or more) with the example of
> echo or grep active piped to it.
>
>
>> Or do I still need to set groupbaseexpiry to false,
>> setup the various storage classes, and change expire.ctl accordingly?
>
> No need to do change anything; groupbaseexpiry set to false was to make
> expire work as you wanted (without expireover).

I'm going to do testing with a few groups as you outlined in a previous
response and report back my findings. The expireover issues I'm struggling
with on this box are perplexing me. I've used these tools on larger spools in
the past with faster results, so it feels like I am hitting a constraint, but
can't point to where. The IOPS and throughput during expireover don't even
come close to what the box does when I inject an entire spool's worth of
articles into it from another server so it doesn't feel like
hardware/throughput is the issue, but still trying to figure it out.

It's also possible my memory is absolutely trashed and I am not recalling my
previous experience or usage correctly. :)

[Jesse Rehmer]

On Jul 7, 2022 at 3:32:03 PM CDT, "Jesse Rehmer" in

<259231881...@freebsd-inject1.usenet.blueworldhosting.com> wrote:

> I'm going to do testing with a few groups as you outlined in a previous
> response and report back my findings.

This is interesting:

$ time echo "rec.arts.tv" | expireover -f - -Z /usr/local/news/tmp/lowmark
Article lines processed 1371021
Articles dropped 0
Overview index dropped 0

real 8m4.764s
user 0m22.424s
sys 1m3.622s

So for the largest group I currently have, it only took 8 minutes. I wonder
why it is taking so long on the entire spool when that group represents
something like 2-3% of total articles. It did cause the locking issue for a
few minutes during, but I think I can live with that if I can get things
staggered a bit.

[Jesse Rehmer]

On Jul 7, 2022 at 3:26:03 PM CDT, "Julien ÉLIE" in

<ta7fgr$1d8cj$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> That expireover command should do what I'm after without having to change
>> groupbaseexpiry to false?
>
> Exactly!
> And you can even try it on 1 newsgroup (or more) with the example of
> echo or grep active piped to it.

Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
expireover also needs to be called with '-e' to get the same behavior as the
"X" flag as described in the manpage for expire.ctl:

> An expiration policy is applied to every article in a newsgroup it
> matches. There is no way to set an expiration policy for articles
> crossposted to groups you don't carry that's different than other
> articles in the same group. Normally, articles are not completely
> deleted until they expire out of every group to which they were posted,
> but if an article is expired following a rule where <flag> contains
> "X", it is deleted out of all newsgroups to which it was posted
> immediately.

My assumption from reading the expire.ctl manpage was that my "AX" flag in
expire.ctl would accomplish the same thing as what 'expireover -e' does.

It would seem based on the observation you made (quoted below) that the X flag
in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
the '-e' flag. Is that correct, or am I still missing something?

[Jesse Rehmer]

On Jul 7, 2022 at 3:51:03 PM CDT, "Jesse Rehmer" in

Using these flags seems to significantly increases performance of expireover:

$ time expireover -e -N -p
Article lines processed 53656755
Articles dropped 8039
Overview index dropped 39271

real 349m6.183s
user 16m33.126s
sys 44m39.500s

The last time I ran expireover without any flags it ran for many more hours.

[Julien ÉLIE]

Hi Jesse,

>> "expireover -e -N -p" should do what you want (see my previous response).
>

> Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
> expireover also needs to be called with '-e' to get the same behavior as the
> "X" flag as described in the manpage for expire.ctl

Running "expireover -e" is tantamount to adding "X" to every expiry rule
(and in that case, you are not obliged to add these "X").

Oh, I see that you did not add "X" to news.lists.filters so in fact you
should not use the "-e" flag.

"expireover -N -p" is the command to run in your case.
I don't think you lost useful articles with your run with "-e",
according to your expire.ctl rules.

> My assumption from reading the expire.ctl manpage was that my "AX" flag in
> expire.ctl would accomplish the same thing as what 'expireover -e' does.

Yes, and "-e" will force "X" in all rules.

> It would seem based on the observation you made (quoted below) that the X flag
> in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
> the '-e' flag. Is that correct, or am I still missing something?

You're not obliged to add "-e" to expireover.

--
Julien ÉLIE

« Avez-vous remarqué qu'à table les mets que l'on vous sert vous mettent
les mots à la bouche ? » (Raymond Devos)

[Jesse Rehmer]

On Jul 8, 2022 at 10:32:16 AM CDT, "Julien ÉLIE" in

<ta9im0$1eoqq$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
>> expireover also needs to be called with '-e' to get the same behavior as the
>> "X" flag as described in the manpage for expire.ctl
>
> Running "expireover -e" is tantamount to adding "X" to every expiry rule
> (and in that case, you are not obliged to add these "X").
>
> Oh, I see that you did not add "X" to news.lists.filters so in fact you
> should not use the "-e" flag.
>
> "expireover -N -p" is the command to run in your case.
> I don't think you lost useful articles with your run with "-e",
> according to your expire.ctl rules.
>
>
>
>> My assumption from reading the expire.ctl manpage was that my "AX" flag in
>> expire.ctl would accomplish the same thing as what 'expireover -e' does.
>
> Yes, and "-e" will force "X" in all rules.
>
>
>> It would seem based on the observation you made (quoted below) that the X flag
>> in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
>> the '-e' flag. Is that correct, or am I still missing something?
>
> You're not obliged to add "-e" to expireover.

Got it, thank you very much for the clear explanation. I think I am still
mixing/confusing the operations of expireover and expire, where when running
expire along it wasn't doing what I expected (which now I also understand
thanks to your help).

Cheers,
Jesse

[Russ Allbery]

Jesse Rehmer <jesse....@blueworldhosting.com> writes:

> Got it, thank you very much for the clear explanation. I think I am
> still mixing/confusing the operations of expireover and expire, where
> when running expire along it wasn't doing what I expected (which now I
> also understand thanks to your help).

It doesn't help that this whole thing grew organically over many years and
involves multiple programs with multiple somewhat confusing modes of
operation and one really intrusive configuration option (groupbaseexpiry)
that basically changes how everything works. This thread makes it clear
that this is all more complicated than it probably should be, but of
course making it less complicated without breaking anything is also very
complicated!

--
Russ Allbery (ea...@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

[Julien ÉLIE]

Hi Jesse and Russ,

>> Got it, thank you very much for the clear explanation. I think I am
>> still mixing/confusing the operations of expireover and expire, where
>> when running expire along it wasn't doing what I expected (which now I
>> also understand thanks to your help).

You're welcome!
I also learnt a few things thanks to your questions.

> It doesn't help that this whole thing grew organically over many years and
> involves multiple programs with multiple somewhat confusing modes of
> operation and one really intrusive configuration option (groupbaseexpiry)
> that basically changes how everything works. This thread makes it clear
> that this is all more complicated than it probably should be, but of
> course making it less complicated without breaking anything is also very
> complicated!

Indeed, groupbaseexpiry complicates the whole thing...

Moreover, the expire(8) man page was cryptic and not totally right. Here is
a new wording for the description of expire(8). I hope it will now be clear
enough.


expire scans the history(5)-format text file pathdb/history and uses
the information recorded in it to purge itself of old news articles.
Its behaviour depends on the setting of groupbaseexpiry in inn.conf.

If groupbaseexpiry is false, expiry is done according storage class
numbers in storage.conf and related rules in expire.ctl. Articles that
should be expired are removed from the news spool and overview (if
any); their history records are purged if they are older than the
number of days specified in the "/remember/" line in expire.ctl.
Articles stored using a storage method that has self-expire
functionality like CNFS are by default not affected by expire's primary
behaviour (but see the -N flag to disable this).

For articles in self-expiring storage methods when groupbaseexpiry is
set to false in inn.conf and the -N flag is not given, or for *all*
articles when groupbaseexpiry is set to true, expire.ctl is ignored
except the "/remember/" line; expire only probes to see if the article
still exists, and purges the relevant history and overview entries if
appropriate.

Per-group expiry, that is to say when groupbaseexpiry is true, is
mostly done by expireover(8): news articles are removed from the news
spool by expireover, and then expire purges the history file.

Regardless the setting of groupbaseexpiry, expireover should be run
along with expire, usually via news.daily out of cron.

Note that expire never purges articles which do not match any entry in
expire.ctl.




Now that expire's documentation is fixed and "expire -p" is in fact working
as it does, let's release INN 2.7.0 :-)
I'll generate the release this week-end.

Thanks again Jesse for all your questions on your fresh INN installation which
permitted to make it better, and especially its documentation!

--
Julien ÉLIE

« – D'ailleurs je vais préparer de la potion magique qui ne servira bien
entendu que contre un retour éventuel des Romains !
– Mais, pour l'heure, les Romains ne sont pas encore revenus de leur
mésaventure. » (Astérix)

[Julien ÉLIE]

Hi Russ,

> It doesn't help that this whole thing grew organically over many years and
> involves multiple programs with multiple somewhat confusing modes of
> operation and one really intrusive configuration option (groupbaseexpiry)
> that basically changes how everything works. This thread makes it clear
> that this is all more complicated than it probably should be, but of
> course making it less complicated without breaking anything is also very
> complicated!

It would perhaps imply to make a new INN 3.0 product with all sort of
oddities removed and less options. Yet, it's still a hard work because
it would also need unifying all the configuration files (in YAML ^^) and
refactoring thigs...

[Jesse Rehmer]

On Jul 8, 2022 at 12:21:37 PM CDT, "Julien ÉLIE" in
<ta9p31$1f5nu$1...@news.trigofacile.com> wrote:

> Hi Jesse and Russ,

>
> Per-group expiry, that is to say when groupbaseexpiry is true, is
> mostly done by expireover(8): news articles are removed from the news
> spool by expireover, and then expire purges the history file.
>
> Regardless the setting of groupbaseexpiry, expireover should be run
> along with expire, usually via news.daily out of cron.
>
> Note that expire never purges articles which do not match any entry in
> expire.ctl.

Those three paragraphs really clear it up for me. My original issue stemmed
from my misunderstanding that expireover and expire functioned the same in
terms of using expire.ctl.

> Now that expire's documentation is fixed and "expire -p" is in fact working
> as it does, let's release INN 2.7.0 :-)
> I'll generate the release this week-end.
>
> Thanks again Jesse for all your questions on your fresh INN installation which
> permitted to make it better, and especially its documentation!

No rc2? I'm still pouring over everything, might find more... ;) (Joking - I
say go for it, I'll gladly update my boxes)

Glad I could contribute something useful.

Regards,

Jesse

[Julien ÉLIE]

Hi Jesse,

>> Now that expire's documentation is fixed and "expire -p" is in fact working
>> as it does, let's release INN 2.7.0 :-)
>> I'll generate the release this week-end.
>

> No rc2? I'm still pouring over everything, might find more... ;) (Joking - I
> say go for it, I'll gladly update my boxes)

I don't plan on releasing rc2 (changes since rc1 are essentially
documentation) and delay a bit more the final release.

And incidentally, if you're looking for what would be rc2, just try the
latest daily snapshot :-)
https://ftp.isc.org/isc/inn/snapshots/

You're of course welcome to go on pouring over everything. Fixes of
possible bugs, as well as other improvements in documentation, will be
in INN 2.7.1.

--
Julien ÉLIE

« I don't know if it's what you want, but it's what you get. » (Larry
Wall)

[Russ Allbery]

Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:

> Per-group expiry, that is to say when groupbaseexpiry is true, is
> mostly done by expireover(8): news articles are removed from the
> news spool by expireover, and then expire purges the history
> file.

This is getting at something that if I remember correctly is true, but
which I'm not sure we say anywhere. (But don't take my word for it since
it's been a very long time since I looked at this code!) Specifically, I
believe it's correct to say something like this:

When groupbaseexpiry is true, article expiration is primarily done by
expireover based on the expiration rules that match each newsgroup.
expire then does some additional cleanup to remove old history
database entries.

When groupbaseexpiry is false, article expiration is primarily done by
expire based on the expiration rules that match the storage class of
each article. expire removes the articles and the history entries,
and then expireover does the additional cleanup of removing the
overview database entry.

This is the thing that always confused me about groupbaseexpiry: depending
on what it's set to, which program does the actual work of deleting
articles changes.

[Jesse Rehmer]

On Jul 8, 2022 at 12:53:44 PM CDT, "Russ Allbery" in

<87bktz1...@hope.eyrie.org> wrote:

> This is getting at something that if I remember correctly is true, but
> which I'm not sure we say anywhere. (But don't take my word for it since
> it's been a very long time since I looked at this code!) Specifically, I
> believe it's correct to say something like this:
>
> When groupbaseexpiry is true, article expiration is primarily done by
> expireover based on the expiration rules that match each newsgroup.
> expire then does some additional cleanup to remove old history
> database entries.
>
> When groupbaseexpiry is false, article expiration is primarily done by
> expire based on the expiration rules that match the storage class of
> each article. expire removes the articles and the history entries,
> and then expireover does the additional cleanup of removing the
> overview database entry.
>
> This is the thing that always confused me about groupbaseexpiry: depending
> on what it's set to, which program does the actual work of deleting
> articles changes.

That adds additional clarification that is most helpful in understanding the
differences in their operation depending upon the groupbaseexpiry value.

From the content of this thread, I believe it is accurate.

[Julien ÉLIE]

Hi Russ,

> This is getting at something that if I remember correctly is true, but
> which I'm not sure we say anywhere.

I indeed did not see in the documentation the explanation you propose.
I believe it could be inserted in expire.ctl man page, when speaking
about groupbaseexpiry. And also in INSTALL.

> When groupbaseexpiry is true, article expiration is primarily done by
> expireover based on the expiration rules that match each newsgroup.
> expire then does some additional cleanup to remove old history
> database entries.

Yes, that's the behaviour I saw.

> When groupbaseexpiry is false, article expiration is primarily done by
> expire based on the expiration rules that match the storage class of
> each article. expire removes the articles and the history entries,
> and then expireover does the additional cleanup of removing the
> overview database entry.

Yes, agreed too.
With the subtlety that expire won't expire articles in CNFS unless -N is
given!

And you're right that expire doesn't do anything with overview (I'll
update my previous wording, which was wrong).

> This is the thing that always confused me about groupbaseexpiry: depending
> on what it's set to, which program does the actual work of deleting
> articles changes.

Sure! It is really confusing.
And news.daily does magic with the order of all the calls to
expire/expireover depending on groupbaseexpiry.


Hmm, if a news admin ever configures his server with groupbaseexpiry to
true and *without* overview, he won't be able to expire articles from
his spool, will he? (something maybe worth mentioning!)

--
Julien ÉLIE

« C'est une forêt vierge où la main de l'homme n'a jamais mis le pied. »

[Jesse Rehmer]

On Jul 8, 2022 at 1:22:21 PM CDT, "Julien ÉLIE" in

<ta9skt$1fehp$1...@news.trigofacile.com> wrote:

> Hmm, if a news admin ever configures his server with groupbaseexpiry to
> true and *without* overview, he won't be able to expire articles from
> his spool, will he? (something maybe worth mentioning!)

This may be helpful, my feeder has no overview and only a CNFS buffer:

$ grep groupbaseexpiry etc/inn.conf
groupbaseexpiry: true

$ expireover
expireover: enableoverview is not true
expireover: can't open overview database

$ expire -v 3
Article lines processed 124432
Articles retained 124428
Entries expired 4


My expire.ctl entries (for this test):

*:A:1:90:never
*.test:AX:1:1:1

[Russ Allbery]

Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:

> Hmm, if a news admin ever configures his server with groupbaseexpiry to
> true and *without* overview, he won't be able to expire articles from
> his spool, will he? (something maybe worth mentioning!)

I believe that's correct, although if you're only using self-expiring
storage backends like CNFS, you may not care. The history entries (the
only database outside of the article spool in that scenario) will still
get cleaned up by expire when it detects that the article no longer
exists.

[Julien ÉLIE]

Hi Jesse,

>> Hmm, if a news admin ever configures his server with groupbaseexpiry to
>> true and *without* overview, he won't be able to expire articles from
>> his spool, will he? (something maybe worth mentioning!)
>
> This may be helpful, my feeder has no overview and only a CNFS buffer:
>

> $ expire -v 3
> Article lines processed 124432
> Articles retained 124428
> Entries expired 4
>
> My expire.ctl entries (for this test):
>
> *:A:1:90:never
> *.test:AX:1:1:1

When groupbaseexpiry is set to true, expire just does a probe whether
the article is still in the spool.

I would tend to think the 4 entries expired are cancelled articles whose
tokens are removed from the history file. (More than 4 entries would
have expired for all test groups...)
Thanks for the test anyway!

If you want to be sure, you may want to run "grephistory <mid>" on a
Message-ID you know was posted to a test group 2 days ago, and see if
the command returns a token.

[Russ Allbery]

Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:

> When groupbaseexpiry is set to true, expire just does a probe whether the
> article is still in the spool.

> I would tend to think the 4 entries expired are cancelled articles whose
> tokens are removed from the history file. (More than 4 entries would
> have expired for all test groups...)

The other common case is articles with Expires headers, which is still
sometimes done for the few FAQs that are autoposted.

[Jesse Rehmer]

On Jul 8, 2022 at 2:59:44 PM CDT, "Russ Allbery" in

<87czefs...@hope.eyrie.org> wrote:

> Julien ÉLIE <iul...@nom-de-mon-site.com.invalid> writes:
>
>> When groupbaseexpiry is set to true, expire just does a probe whether the
>> article is still in the spool.
>
>> I would tend to think the 4 entries expired are cancelled articles whose
>> tokens are removed from the history file. (More than 4 entries would
>> have expired for all test groups...)
>
> The other common case is articles with Expires headers, which is still
> sometimes done for the few FAQs that are autoposted.

I ran it again a few minutes later and it expired a few hundred articles.
Though, as mentioned, I can't be certain these were removed from the spool by
expire or they self-expired due to be rolled over.

It only keeps a few days worth of articles, but will let it sit a day and do a
test with a known-existing Message-ID that should expire.

However, I just realized on this box the test with expire may be moot because
I have a minimal active file and everything is going to "junk" (I had
forgotten that earlier).