DNews deleting items it shouldn't
Ashley Barnes
Hi there.
I'm having some difficulty with dnews at the moment in that it keeps
deleting items from groups which it shouldn't.
This has happened to me once before. The symptom is that, all of a sudden,
items just vanish. Every single new item is deleted immediately, and it
takes a rebuild_index for it to start storing new items again.
Once the rebuild_index is complete, functionality returns.
However, the groups in question are then missing several
thousand earlier posts.
On closer examination of dnews.log, rebuild_index finishes, but
it is during resort_index that the problems arise.
29 08:58:32 :info: cmdq: Processing next background command (resort_index)
29 08:58:33 :info: Index Sort chunk size is 300000 items
29 08:58:33 :warn: mem: dbmmem malloced RESORT
29 08:58:33 :warn: Processing cancelled items, this could take some time
29 08:58:33 :info: db: cancel upto 0
29 08:58:33 :info: Cancel processing 0 buffered cancels, upto 0
29 08:58:33 :info: db_clear_cache(cancel_background)
29 08:58:33 :info: Group clear cache
29 08:58:33 :info: Group clear cache - done
29 08:58:33 :warn: Checking index for consistency (nightly check)
29 08:58:34 :info: group: Group Write S:\dnews\work/active.new
29 08:58:34 :info: group: Group write done
29 08:58:35 :info: Index check completed
29 08:58:35 :info: db_clear_cache(db_reindex)
29 08:58:35 :info: Group clear cache
29 08:58:35 :info: Group clear cache - done
29 08:58:35 :info: Sorting dbi.idx into 5 stacks
29 08:58:35 :info: db_reindex_new_scan 0
29 08:58:35 :info: Pruning large group demon.local:29349
29 08:58:36 :info: Pruning large group alt.alien.vampire.flonk.flonk.flonk:475386
29 08:58:36 :info: Pruning large group demon.local:2387
29 08:58:36 :info: Pruning large group uk.media.tv.misc:16
29 08:58:36 :info: Pruning large group demon.local:3686
29 08:58:36 :info: Pruning large group uk.media.tv.misc:4268
29 08:58:36 :info: Pruning large group demon.local:5340
29 08:58:36 :info: Pruning large group uk.media.tv.misc:10695
29 08:58:36 :info: Pruning large group demon.local:5361
29 08:58:36 :info: Pruning large group uk.media.tv.misc:10929
29 08:58:36 :info: Pruning large group demon.local:5388
29 08:58:36 :info: Pruning large group uk.media.tv.misc:10938
29 08:58:36 :info: Pruning large group demon.local:5389
29 08:58:36 :info: Pruning large group uk.media.tv.misc:10940
29 08:58:36 :info: Pruning large group demon.local:5419
29 08:58:36 :info: Pruning large group uk.media.tv.misc:11085
29 08:58:36 :info: Pruning large group demon.local:5432
The logs then fill up with thousands more lines stating that
items from groups have been "pruned". Yet, I do not want it
to "prune large groups" as I have my expire.conf deliberately
set on very large retentions. There is no problem with space...
-> tellnews showpiles
200
[...]
Disk space 35091MB - history*2=157MB,xover=255MB,dbi.idx*2=48MB = 34631MB for piles [...]
Pile 6, 2.4/5.0% 823MB, Sort uk.*
[...]
Pile 8, 2.0/20.0% 694MB, Sort alt.*,@demon.*
Pile 9, 2.6/(nolimit) 906MB, Sort demon.*
281 Completed
Any idea what's going on?
Thanks,
Ashley Barnes.
Netwin Support
I haven't come across that message before, it is in recently added code
(ChrisP). A brief look suggests it is related to the max_items setting
in dnews.conf. This specifies the maximum number of articles DNews can
store per group. Exceeding this value could cause problems, so the code
was most likely added to prevent the value being exceeded.
Set max_items in dnews.conf, I personally have set this to 1000000 on my
own server. You must restart DNews after making this change as it alters
internal buffer sizes.
- Roydon L.
Ashley Barnes
Netwin Support <sup...@netwinsite.com> wrote:
Hello, Roydon. Thank you for your reply.
>I haven't come across that message before, it is in recently added code
>(ChrisP). A brief look suggests it is related to the max_items setting
>in dnews.conf. This specifies the maximum number of articles DNews can
>store per group. Exceeding this value could cause problems, so the code
>was most likely added to prevent the value being exceeded.
Ahh. I've just checked the DNews manual based on what you've told me and
this does appear to be the problem. The manual says that the default
setting is 90,000, and I believe that the groups in question had reached
90,000, which would have triggered the "prune".
>Set max_items in dnews.conf, I personally have set this to 1000000 on my
>own server. You must restart DNews after making this change as it alters
>internal buffer sizes.
Given that you have your max_items set much higher than 90,000, I think it
is safe to assume that I shouldn't have problems if I use a similar value
to you!
Many thanks for your assistance.
Ashley Barnes.