kudos to ovdb + compression
Michael Grimm
JFTR: I finally upgraded my reader from 2.4.6 to 2.5.1 and rebuilt the
old ovdb overview database using the new compression functionality:
2.4.6 + ovdb (db47):
18,500,000 articles in spool
21 GB size overview database
5:45 hours for expireover
2.5.1 + ovdb (db47) + compression:
18,500,000 articles in spool
16,5 GB size overview database
2:20 hours for expireover
Well, many kudos to those involved! That is great, especially the
improvement in expireover runtime.
Heath Kehoe's findings were similar, a couple of years ago:
,----[ https://lists.isc.org/pipermail/inn-workers/2004-June/013106.html ]-
| On my server with about 21 million overview records, after
| rebuilding the databases with compression enabled, the overview
| directory went from around 20 GB to 12.6 GB, and the expireover
| run went from 6 hours to 2 hours.
`----
Regards and I'm very pleased,
Michael
--
to let
Julien ÉLIE
> 2.4.6 + ovdb (db47):
>
> 18,500,000 articles in spool
> 21 GB size overview database
> 5:45 hours for expireover
About 1.13 kb/article. It is what D. Stussy spoke about.
> 2.5.1 + ovdb (db47) + compression:
>
> 18,500,000 articles in spool
> 16,5 GB size overview database
> 2:20 hours for expireover
About 0.9 kb/article.
I would have expected less for a compressed ovdb (our documentation
mentions 0.7 kb/article -- maybe we should update it then).
It is great to hear that the time spent by expireover has been
reduced by a factor of 60%!!
It is amazing to see the change.
> Heath Kehoe's findings were similar, a couple of years ago:
> ,----[ https://lists.isc.org/pipermail/inn-workers/2004-June/013106.html ]-
> | On my server with about 21 million overview records, after
> | rebuilding the databases with compression enabled, the overview
> | directory went from around 20 GB to 12.6 GB, and the expireover
> | run went from 6 hours to 2 hours.
> `----
He had 0.6 kb/article!
Should we conclude that headers are currently larger than five years ago?
Do you have additional overview fields in LIST OVERVIEW.FMT?
Something that explains a rise (but not of 300 bytes) is that makehistory
now computes the :bytes metadata. It did not do that by default in INN 2.4.
Thanks again for your feedback!
--
Julien �LIE
� Rien ne serpe de courir ! � (Druides gaulois)
Michael Grimm
> Hi Michael,
>> 2.5.1 + ovdb (db47) + compression:
>>
>> 18,500,000 articles in spool
>> 16,5 GB size overview database
>> 2:20 hours for expireover
>
> About 0.9 kb/article.
> I would have expected less for a compressed ovdb (our documentation
> mentions 0.7 kb/article -- maybe we should update it then).
It's a regular feed regarding all newsgroups peered by 40 peers
excluding binaries.
> It is great to hear that the time spent by expireover has been
> reduced by a factor of 60%!!
> It is amazing to see the change.
Yeah, that really is more than expected! I haven't used those other
overview storage methods for years now. But I'm interested in learning
about comparable benchmarks. So, what are your expireover times out
there using which overview storage method?
>> Heath Kehoe's findings were similar, a couple of years ago:
>> ,----[ https://lists.isc.org/pipermail/inn-workers/2004-June/013106.html ]-
>> | On my server with about 21 million overview records, after
>> | rebuilding the databases with compression enabled, the overview
>> | directory went from around 20 GB to 12.6 GB, and the expireover
>> | run went from 6 hours to 2 hours.
>> `----
>
> He had 0.6 kb/article!
> Should we conclude that headers are currently larger than five years ago?
Yes, I would asume so.
> Do you have additional overview fields in LIST OVERVIEW.FMT?
No, just the default overview.fmt.
> Something that explains a rise (but not of 300 bytes) is that makehistory
> now computes the :bytes metadata. It did not do that by default in INN 2.4.
Sorry, but I do not understand that. -v, please.
> Thanks again for your feedback!
You're welcome and regards,
Michael
--
to let
Ray Banana
> Yeah, that really is more than expected! I haven't used those other
> overview storage methods for years now. But I'm interested in learning
> about comparable benchmarks. So, what are your expireover times out
> there using which overview storage method?
I changed from OVDB to tradindexed because of OVDB's erratic behaviour
under heavy load (600 concurrent nnrpds) 5 months ago and my expireover
elapsed time dropped from 6 hours to 45 minutes for ~20,000,000 articles.
--
Time flies like an arrow, fruit flies like a Banana.
http://www.eternal-september.org
Michael Grimm
> It is great to hear that the time spent by expireover has been
> reduced by a factor of 60%!!
> It is amazing to see the change.
What is your explanation about the cause of that reduction in time spent
in overexpire? I really do have some difficulties in understanding the
technical implications in that time reduction. Doesn't overexpire just
deletes overview entries of no longer found articles in a CNFS setup
(like mine)? If so, why should those deletions be faster in the case of
compressedly stored overview fields compared o uncompressedly stored
ones? Do ypu or anyone else have an explanation for me?
Regards,
Michael
--
to let
Michael Grimm
>> storage method?
>
> I changed from OVDB to tradindexed because of OVDB's erratic behaviour
> under heavy load (600 concurrent nnrpds) 5 months ago
Yes, I remember, but IIRC you used < db47 which seemed to be faulty at
my setup (with just myself as "concurrent" nnrpd user) as well.
> and my expireover elapsed time dropped from 6 hours to 45 minutes for
> ~20,000,000 articles.
Ok, that is by far better than anticipated ;-) Ok, then I do only have
stability left after hard reboots left, if at all? ;-)
Michael Grimm
>> storage method?
>
> I changed from OVDB to tradindexed because of OVDB's erratic behaviour
> under heavy load (600 concurrent nnrpds) 5 months ago
Yes, I remember, but IIRC you used < db47 which seemed to be faulty at
my setup (with just myself as "concurrent" nnrpd user) as well.
> and my expireover elapsed time dropped from 6 hours to 45 minutes for
> ~20,000,000 articles.
Ok, that is by far better than anticipated ;-) Ok, then I do only have
stability left after hard reboots, if at all? ;-)
Michael Grimm
> my expireover elapsed time dropped from 6 hours to 45 minutes for
> ~20,000,000 articles.
One additional question: which OS and FS? If you don't mind to make
public.
Ray Banana
>> under heavy load (600 concurrent nnrpds) 5 months ago
> Yes, I remember, but IIRC you used < db47 which seemed to be faulty at
> my setup (with just myself as "concurrent" nnrpd user) as well.
No, I tried all available versions up to and including 4.8, no joy.
At around 400 concurrent nnrpds I was getting timeouts and zombie OVDB
servers or I would have to set max_locks to absurdly high values
(240,000 IIRC) without OVDB servers.
>> and my expireover elapsed time dropped from 6 hours to 45 minutes for
>> ~20,000,000 articles.
> Ok, that is by far better than anticipated ;-) Ok, then I do only have
> stability left after hard reboots, if at all? ;-)
My experience was quite the opposite: After a hard reset I had to
inevitably run makehistory -O to get a consistent overview and that took
about 18 hours. I never had to run makehistory after changing to
tradindexed, tdx-util was able to fix all inconsistencies after a series
of hard crashes I had recently because of defective memory modules.
Julien ÉLIE
> Yeah, that really is more than expected! I haven't used those other
> overview storage methods for years now. But I'm interested in learning
> about comparable benchmarks. So, what are your expireover times out
> there using which overview storage method?
I use tradindexed; expireover takes 15 minutes for about 160,000 articles
(though I have 3 million articles in my history file). I do not have
a fast server (Celeron CPU 2GHz, 256MB RAM) so it is difficult to compare!
Articles stored in self-expiring methods like CNFS are not dealt with by
expireover.
expireover start samedi 28 novembre 2009, 04:15:28 (UTC+0100): ( -z/var/log/news/expire.rm -Z/var/log/news/expire.lowmark)
Article lines processed 158125
Articles dropped 1134
Overview index dropped 1135
expireover end samedi 28 novembre 2009, 04:29:42 (UTC+0100)
lowmarkrenumber begin samedi 28 novembre 2009, 04:29:42 (UTC+0100): (/var/log/news/expire.lowmark)
lowmarkrenumber end samedi 28 novembre 2009, 04:29:42 (UTC+0100)
expirerm start samedi 28 novembre 2009, 04:29:42 (UTC+0100)
expirerm end samedi 28 novembre 2009, 04:29:50 (UTC+0100)
expire begin samedi 28 novembre 2009, 04:30:20 (UTC+0100): (-v1)
Article lines processed 2853796
Articles retained 2849941
Entries expired 3855
expire end samedi 28 novembre 2009, 04:40:38 (UTC+0100)
all done samedi 28 novembre 2009, 04:40:38 (UTC+0100)
>> Something that explains a rise (but not of 300 bytes) is that makehistory
>> now computes the :bytes metadata. It did not do that by default in INN 2.4.
>
> Sorry, but I do not understand that. -v, please.
LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
Bytes:
Lines:
Xref:full
.
(for a default overview.fmt list)
When you run makehistory, each article is opened and parsed so as to generate
its overview data. With INN 2.4, the fields named "Bytes:" and "Lines:" are
taken from the headers of the article. They do not always have one, so
the overview data can contain empty fields for them (maybe the number of lines
was computed -- I do not remember). The number of bytes may have been forced
by "makehistory -e", which was unusual.
With INN 2.5, there is no "makehistory -e". The generated overview data
always contains values for the number of bytes and lines. It is an accurate
value (named as metadata ":bytes" and ":lines" per RFC 3977).
I hope my explanation is clear...
--
Julien ï¿œLIE
ￜ The hardest thing is to go to sleep at night, when there are
so many urgent things needing to be done. A huge gap
exists between what we know is possible with today's machines
and what we have so far been able to finish. ￜ (Donald Knuth)
Michael Grimm
> Thus spake Michael Grimm <tras...@odo.in-berlin.de>
>>> I changed from OVDB to tradindexed because of OVDB's erratic
>>> behaviour under heavy load (600 concurrent nnrpds) 5 months ago
>>
>> Yes, I remember, but IIRC you used < db47 which seemed to be faulty
>> at my setup (with just myself as "concurrent" nnrpd user) as well.
>
> No, I tried all available versions up to and including 4.8, no joy.
Sorry, then I didn't remember correctly.
> At around 400 concurrent nnrpds I was getting timeouts and zombie OVDB
> servers or I would have to set max_locks to absurdly high values
> (240,000 IIRC) without OVDB servers.
Hmm. I'm off here, I do not have other readers than myself. I do not
judge your experiences, but do you have an explantion why a somehow
mature database system would be inferior to a flat filesystem overview?
>> Ok, then I do only have stability left after hard reboots, if at all?
>> ;-)
>
> My experience was quite the opposite: After a hard reset I had to
> inevitably run makehistory -O to get a consistent overview and that
> took about 18 hours.
Hmm. That are really opposit experiences. Honestly, I did have a lot of
difficulties after hard resets (power failures) with tradindex, and
never again with ovdb. Why, do we and others have those different
experiences at all?
> I never had to run makehistory after changing to tradindexed, tdx-util
> was able to fix all inconsistencies after a series of hard crashes I
> had recently because of defective memory modules.
Ok, I switched to ovdb before tdx-util became available at all.
Ray Banana
>> ~20,000,000 articles.
> One additional question: which OS and FS? If you don't mind to make
> public.
Debian (Woody - Lenny) with ext2, ext3, and JFS, Ubuntu 8.10 LTS with
ext3 and JFS, Ubuntu 9.10 with JFS and ext4, FreeBSD 5-8 with ufs,
Berkeley DB 4.2 - 4.8, IIRC. I'm currently running Ubuntu 9.10 and
Debian Lenny with JFS on my production systems.
Michael Grimm
>> public.
>
> Debian (Woody - Lenny) with ext2, ext3, and JFS, Ubuntu 8.10 LTS with
> ext3 and JFS, Ubuntu 9.10 with JFS and ext4, FreeBSD 5-8 with ufs,
> Berkeley DB 4.2 - 4.8, IIRC.
Holy cow?! How many servers at hand? ;-)
> I'm currently running Ubuntu 9.10 and Debian Lenny with JFS on my
> production systems.
Ok, I used to have some debian (ham, rex or bo, I really can't remember)
and ext2 without journaling running at those times. Today I'm at FBSD 7.2
with ufs.
Michael Grimm
>> Yeah, that really is more than expected! I haven't used those other
>> overview storage methods for years now. But I'm interested in
>> learning about comparable benchmarks. So, what are your expireover
>> times out there using which overview storage method?
>
> I use tradindexed; expireover takes 15 minutes for about 160,000
> articles (though I have 3 million articles in my history file).
Hmm, how do you do that? expire.clt magic? or?
> I do not have a fast server (Celeron CPU 2GHz, 256MB RAM) so it is
> difficult to compare! Articles stored in self-expiring methods like
> CNFS are not dealt with by expireover.
Again, how do you configure that? Did I oversee an approach regarding
CNFS and unnessary expireover?
>>> Something that explains a rise (but not of 300 bytes) is that
>>> makehistory now computes the :bytes metadata. It did not do that by
>>> default in INN 2.4.
>>
>> Sorry, but I do not understand that. -v, please.
>
> LIST OVERVIEW.FMT 215 Order of fields in overview database Subject:
> From: Date: Message-ID: References: Bytes: Lines: Xref:full .
>
> (for a default overview.fmt list)
>
> When you run makehistory, each article is opened and parsed so as to
> generate its overview data. With INN 2.4, the fields named "Bytes:"
> and "Lines:" are taken from the headers of the article. They do not
> always have one, so the overview data can contain empty fields for
> them (maybe the number of lines was computed -- I do not remember).
> The number of bytes may have been forced by "makehistory -e", which
> was unusual.
>
> With INN 2.5, there is no "makehistory -e". The generated overview
> data always contains values for the number of bytes and lines. It is
> an accurate value (named as metadata ":bytes" and ":lines" per RFC
> 3977).
>
> I hope my explanation is clear...
I'm awefully sorry, but I didn't understand your explanations at all.
But I'm sure that is originated in my inferior coding skills ;-) No,
offense intended, really.
Julien ÉLIE
>> I use tradindexed; expireover takes 15 minutes for about 160,000
>> articles (though I have 3 million articles in my history file).
>
> Hmm, how do you do that? expire.clt magic? or?
>
>> I do not have a fast server (Celeron CPU 2GHz, 256MB RAM) so it is
>> difficult to compare! Articles stored in self-expiring methods like
>> CNFS are not dealt with by expireover.
>
> Again, how do you configure that? Did I oversee an approach regarding
> CNFS and unnessary expireover?
Ah ah, mystery... I thought it was because of CNFS.
Looking at my expire.ctl, I see that I have:
*:A:never:never:never
control.cancel:A:7:7:7
and a few rules for hierarchies stored in timehash, timecaf and tradspool.
So maybe expireover does not look at newsgroups marked as "never:never:never"...
which is what I want for CNFS (just letting the buffer wrap and expire handle
that).
Nonetheless, such groups are processed because the overview data is removed.
I have just checked.
And my overview data contains 3 million lines
22:38 news@trigo ~/spool/overview% find . | grep DAT | xargs wc -l
[...]
3276162 total
Maybe a bug in the count of expireover (?)
I will have a look.
> I'm awefully sorry, but I didn't understand your explanations at all.
Basically, the overview data of an article contains 8 fields.
When you run makehistory, one of these fields remains empty with INN 2.4
whereas it contains a number with INN 2.5. Therefore, the generated
overview data is a bit larger with makehistory of INN 2.5.
--
Julien ï¿œLIE
ￜ J'aurais pu faire forￜt comble. ￜ (Astￜrix)
D. Stussy
news:herptk$49q$1...@news.trigofacile.com...
> > 2.4.6 + ovdb (db47):
> >
> > 18,500,000 articles in spool
> > 21 GB size overview database
> > 5:45 hours for expireover
>
> About 1.13 kb/article. It is what D. Stussy spoke about.
Which comes from the ovdb manual page....
> > 2.5.1 + ovdb (db47) + compression:
> >
> > 18,500,000 articles in spool
> > 16,5 GB size overview database
> > 2:20 hours for expireover
>
> About 0.9 kb/article.
> I would have expected less for a compressed ovdb (our documentation
> mentions 0.7 kb/article -- maybe we should update it then).
The PROBLEM is that the ov data size in the traditional overview format for
a non-reply post is on the order of 200 bytes. I don't have a way of
factoring in the overhead for the .IDX file or the directory structure of
the overview data, but note that for largely populated groups (many posts),
this starts to approach a low number (near zero, but I suspect it might be
4 bytes).
That's why I considered what was really needed for an overview file: An
article number and a message pointer (the storage manager token became the
best value for the latter). If the overview manager were able to have an
LRU cache of articles it has expanded, it might be of comparable speed.
Only writing and testing it would we find out.
Michael Grimm
>>> Articles stored in self-expiring methods like CNFS are not dealt
>>> with by expireover.
>>
>> Again, how do you configure that? Did I oversee an approach regarding
>> CNFS and unnessary expireover?
>
> Ah ah, mystery... I thought it was because of CNFS.
> Looking at my expire.ctl, I see that I have:
>
> *:A:never:never:never
> control.cancel:A:7:7:7
>
> and a few rules for hierarchies stored in timehash, timecaf and tradspool.
Ok. I erroneously considered all your articles stored in CNFS.
> So maybe expireover does not look at newsgroups marked as "never:never:never"...
> which is what I want for CNFS (just letting the buffer wrap and expire handle
> that).
>
> Nonetheless, such groups are processed because the overview data is removed.
> I have just checked.
That is what I consider regular behaviour of expireover, cleanup of history
and removal of those no longer needed entries in the overview database.
When I used to run my newsserver on a VIA C7 CPU, I ran the fast expire
daily, and the very slow expireover (w.r.t. ovdb) every fortnight.
(I should re-think my current strategy of daily expireovers; it doesn't
make musch sense in removing every night articles that are sitting in my
spool for three or one year, respectively. Disk space isn't an issue, so
why not cleaning overviews on a monthly basis? At least with ovdb I
never saw a difference between daily or bi-weekly expireover runtimes,
IIRC.)
>> I'm awefully sorry, but I didn't understand your explanations at all.
>
> Basically, the overview data of an article contains 8 fields.
> When you run makehistory, one of these fields remains empty with INN 2.4
> whereas it contains a number with INN 2.5. Therefore, the generated
> overview data is a bit larger with makehistory of INN 2.5.
Thanks, now I understood ;-)
Julien ÉLIE
>>>> Articles stored in self-expiring methods like CNFS are not dealt
>>>> with by expireover.
I have just checked, and it explains why the number of article lines
processed by expireover is that low. A line is not reported to
have been processed when the article is stored in a self-expiring backend.
if (SMprobe(SELFEXPIRE, &token, NULL)) {
if (!OVignoreselfexpire)
/* this article should be kept */
return false;
}
EXPprocessed++;
> (I should re-think my current strategy of daily expireovers; it doesn't
> make musch sense in removing every night articles that are sitting in my
> spool for three or one year, respectively. Disk space isn't an issue, so
> why not cleaning overviews on a monthly basis? At least with ovdb I
> never saw a difference between daily or bi-weekly expireover runtimes,
> IIRC.)
You can use different calls to news.daily in your crontab, depending
on the day. Use "noexpire" and "noexpireover" if you want.
--
Julien ï¿œLIE
ￜ Prolonger les adieux ne vaut jamais grand-chose ;
ce n'est pas la prￜsence que l'on prolonge, mais le dￜpart. ￜ (Bibesco)
Michael Grimm
>>>>> Articles stored in self-expiring methods like CNFS are not dealt
>>>>> with by expireover.
>
> I have just checked, and it explains why the number of article lines
> processed by expireover is that low. A line is not reported to have
> been processed when the article is stored in a self-expiring backend.
Ok, understood. But that line must have processed in order to decide if
stored in a self-expiring backend ;-)
>> (I should re-think my current strategy of daily expireovers; it
>> doesn't make musch sense in removing every night articles that are
>> sitting in my spool for three or one year, respectively. Disk space
>> isn't an issue, so why not cleaning overviews on a monthly basis? At
>> least with ovdb I never saw a difference between daily or bi-weekly
>> expireover runtimes, IIRC.)
>
> You can use different calls to news.daily in your crontab, depending
> on the day. Use "noexpire" and "noexpireover" if you want.
Yeah, I had to split news.daily calls in one day for expireover, next
day without news.daily, and remaining days of the month for simple
expire runs, only. Reason: my slow 386 PC needed 29 hours for a complete
expireover ;-)
Julien ÉLIE
> That's why I considered what was really needed for an overview file: An
> article number and a message pointer (the storage manager token became the
> best value for the latter). If the overview manager were able to have an
> LRU cache of articles it has expanded, it might be of comparable speed.
> Only writing and testing it would we find out.
In a newsgroup that contains for instance 10 000 articles,
OVER 1-
is *far* faster than
HDR Content-Type 1-
No need to open 10 000 articles and read their headers.
--
Julien �LIE
� Pour d�fendre une cause, un avocat met sa robe. Une femme... l'enl�ve. �