Freenix rankings?
Brad Knowles
Anyone taken a close look at the rankings for September (see
<http://www.top1000.org/1999/top9909.html>)? I love jumping thirty spots
to be in the Top 65, but I have to suspect that I don't really deserve
quite such a high ranking.
I guess I'm just wondering if anyone has done any in-depth analysis of
the fairly major changes I've seen are actually reflected in reality, or
if something screwy has happened.
--
Brad Knowles <br...@shub-internet.org> <http://www.shub-internet.org/brad/>
Are you a USENET news server administrator outside of North America or
Western Europe? Would you like a newsfeed from a site in the Freenix
Top 65? If so, please contact me via private e-mail for details.
Terry Kennedy
> However, I wasn't expecting that I could jump as much as thirty
> spots. And certainly not anywhere as high as I am now.
I (news.new-york.net) went from 141 to 129, a respectable jump considering
I had major server problems last month (but I did replace lots of hardware
and my average outbound bits/sec jumped from about 21Mbit/sec to 30Mbit/sec.
It should be interesting to see how I rank next month.
However, I want to make sure I have a stable server again before I answer
the flurry of "want to peer with us" email messages that I'll get by posting
here 8-)
> Hmm. You must be going the "efficiency" route, whereas I'm taking the
> "mucking big sledgehammer" approach -- there's not much that will beat
> having your history database on RAM disk. ;-)
Well, I've had my Diablo history file on a solid state SCSI drive (1GB,
2-hour battery backup, data retention hard disk) for about a year now, and
while it helps, it didn't help as much as I'd hoped.
Terry Kennedy Operations Manager, Academic Computing
te...@spcvxa.spc.edu St. Peter's College, Jersey City, NJ USA
+1 201 915 9381 (voice) +1 201 435-3662 (FAX)
Brad Knowles
<r...@stanford.edu> wrote:
> Well, your rating is consistent with mine and with the quantity of posts
> you're accepting from me.
Yeah, but look how far demon (7 -> 16) and cwix (4 -> 11) fell this
month. I'd say that's quite precipitous. There are others that have also
significantly changed their rankings this month, but those are the two big
drops that I spotted at first glance.
> I was expecting to jump up a bit, though, based on some changes I made, so
> I wasn't that overly surprised.
I was hoping to jump up a bit, what with our E-3 to LINX going online
last month (and thus giving me much lower latency to several highly rated
news servers), plus the news servers that I added peerings for (still
missing GTEI and tli.de -- anyone got any contacts they'd be willing to
share? ;-), and the speed improvements I hope that I'm getting from Lars'
"pipe" patch for Diablo.
However, I wasn't expecting that I could jump as much as thirty
spots. And certainly not anywhere as high as I am now.
Maybe the Freenix folks have finally implemented compensation for how
much spam is being routed by a particular site?
> (Still highly amused at how high
> newsfeed.stanford.edu is given my configuration, though.)
Hmm. You must be going the "efficiency" route, whereas I'm taking the
"mucking big sledgehammer" approach -- there's not much that will beat
having your history database on RAM disk. ;-)
--
Brad Knowles <br...@shub-internet.org> <http://www.shub-internet.org/brad/>
<http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=0xE38CCEF1>
Kai 'wusel' Siering
% Anyone taken a close look at the rankings for September (see
% <http://www.top1000.org/1999/top9909.html>)? I love jumping thirty spots
% to be in the Top 65, but I have to suspect that I don't really deserve
% quite such a high ranking.
%
% I guess I'm just wondering if anyone has done any in-depth analysis of
% the fairly major changes I've seen are actually reflected in reality, or
% if something screwy has happened.
Looking at the last six months for the current TOP 200, I don't see
really major changes; I've put the table I'm referring to online at
http://gaspode.uu.org/newsadmin/freenix-TOP200-sites.199909 -- might
be slow, it's an rather old machine ;)
Most changes seem to be either related (supernews, remarq, rqdq)
or consistent with the trend of the last month(s), e. g.
System 9909 9908 9907 9906 9905 9904
----------------------------------------------------------------------
supernews.com 8 11 15 15 20 19
newsfeed.icl.net 9 9 31 78 78 119
remarq.com 10 12 17 17 21 22
dispose.news.demon.net 16 7 5 4 3 8
hermes.visi.com 17 25 32 50 245
nntp.gctr.net 19 15 12 441
news-peer.gip.net 37 28 19 9 7 7
bignews.mediaways.net 42 76 118 121 228 634
fu-berlin.de 57 60 63 49 41 38
rqdq 58 59 170 320 276 251
easynet.net 63 57 60 104 128 174
skynet.be 64 94 268 250 188 131
newsfeed.stanford.edu 67 90 108 123 141 151
corp.supernews.com 103 106 691
Regards,
kai
--
Kai 'wusel' Siering eMail @ home: wu...@uu.org
Traveller on the Information Highway doing full-time administration.
The views expressed here are not neccessarily those of any employer.
- Looking for peers for several non-european national TLHs, please -
- contact me for details so we can arrange a peering. -
Brad Knowles
(Mike Horwath) wrote:
> Now, that Brad guy, he is swamped with them and his phat server...
You're a good one to talk, Mr. "I just went from #25 to #17"! ;-)
--
Brad Knowles <br...@shub-internet.org> <http://www.shub-internet.org/brad/>
Are you a USENET news server administrator outside of North America or
Brad Knowles
<te...@spcunb.spc.edu> wrote:
> Well, I've had my Diablo history file on a solid state SCSI drive (1GB,
> 2-hour battery backup, data retention hard disk) for about a year now, and
> while it helps, it didn't help as much as I'd hoped.
My guess is that this is because Diablo mmap()s the index to the
history database, meaning that most of it (at least, the most commonly
accessed parts) is usually kept in memory anyway. It then uses lseek() to
position directly within the file to perform direct reads and writes,
which avoid the buffer cache.
Thus, putting the whole thing in memory doesn't buy you all that much
more performance. In fact, I'm thinking about dropping the mfs filesystem
I use today, so that I can see how much my performance drops when I can
reclaim that 384MB of virtual memory back -- would dnewslinks and the
diablo daemon (as well as child processes) be able to make better use of
that memory?
Brad Knowles
> I (news.new-york.net) went from 141 to 129, a respectable jump considering
> I had major server problems last month (but I did replace lots of hardware
> and my average outbound bits/sec jumped from about 21Mbit/sec to 30Mbit/sec.
> It should be interesting to see how I rank next month.
Jeez friggin' Louise!!! I just realized that this means you went from
doing about 210GB/day to 300GB/day! I'm only doing 45-50GB/day outbound,
and that's only within the last few days since I added a binary peering
with ICL.
On a related topic, I'm wondering what it means when two of my peers
offer me on the order of 1.3-1.5 million articles per day
(newsfeed.icl.net & newsfeed1.funet.fi), whereas no one else offers me
much more than 800k articles per day (including sites I have existing
binary peerings with, such as remarQ and ebone-newscore.univie.ac.at)?
Brad Knowles
(Kai 'wusel' Siering) wrote:
> Looking at the last six months for the current TOP 200, I don't see
> really major changes; I've put the table I'm referring to online at
> http://gaspode.uu.org/newsadmin/freenix-TOP200-sites.199909 -- might
> be slow, it's an rather old machine ;)
Out of curiosity, are you planning on updating this page monthly, as
Iain Lea was supposed to be doing with his version of this information?
Thanks!
Andrew Gierth
Brad> On a related topic, I'm wondering what it means when two of
Brad> my peers offer me on the order of 1.3-1.5 million articles per
Brad> day (newsfeed.icl.net & newsfeed1.funet.fi), whereas no one
Brad> else offers me much more than 800k articles per day (including
Brad> sites I have existing binary peerings with, such as remarQ and
Brad> ebone-newscore.univie.ac.at)?
Those peers are offering you the articles more than once.
There are some obvious reasons why this might happen (e.g. the
presence of a delayed backup feed), and some slightly less obvious
reasons (e.g. they may have too short a history for their age cutoff,
and so may be recycling old articles). (On INN sites there are also
defers to be taken into account, but that doesn't apply to you.)
--
Andrew.
Terry Kennedy
> So, yah, a lot of bandwidth that is expensive being used for an
> absolutely worthless cause - a larger penis^H^H^H^H^Hfreenix rating.
Not really. Since Freenix considers all articles identical, sending
binaries doesn't help a site's Freenix rating much, and can actually
hurt it. The only way it will help is if a site you send binaries to
reports to Freenix and has mostly non-binary peers. If you're the only
binary peer of a Freenix-reporting site, then it could help, but I
think that's a rather unlikely situation. And while I'm sending them
that 100MB monster (which they're likely to reject for some reason),
their non-binary peers have sent them a couple hundred articles that
I also have queued for them, but behind the monster in the queue. So
the other sites' rankings go up.
I don't split my outgoing feeds by size (except for one peer that
requires it) - if I wanted to chase a higher Freenix spot I suppose I
should do that.
Back to Brad's "Jeez" comment - netstat -s shows that I send out a bit
over 2 terabytes/week. Those 64-bit counters in netstat sure come in
handy 8-)
Bill Gunshannon
|>
|> I don't split my outgoing feeds by size (except for one peer that
|> requires it) - if I wanted to chase a higher Freenix spot I suppose I
|> should do that.
|>
Being the one who suggested this idea in the first place, I would have to
say that even if chasing Freenix numbers isn't your cup of tea, the reason
your numbers would go up is a lofty enough goal. It has to effect the overall
propogation of news and isn't that what we all do this for in the first place??
I have been pretty busy, but I have decided to start playing with those
different size queues for a bit. Hopefully to find the optimum breakpoint
based on size. I might even see if maybe three breaks might be the way to.
Freenix, on the other hand, has me completely confused. When I first started
this I jumped over 200 places (into the top 1000) and then one month later
there I was, back in my usual spot. The amount of news I handled didn't
change at all over this period. Baffles me. But, I hope to shuffle some
hardware in the back room before too long and maybe end out with a more
efficient spool for diablo.
bill
--
Bill Gunshannon | de-moc-ra-cy (di mok' ra see) n. Three wolves
bi...@cs.uofs.edu | and a sheep voting on what's for dinner.
University of Scranton |
Scranton, Pennsylvania | #include <std.disclaimer.h>
Kai 'wusel' Siering
% : Jeez friggin' Louise!!! I just realized that this means you went from
% : doing about 210GB/day to 300GB/day!
That's sad but true, if you really peer and distribute binaries (cf.
http://www.mediaways.net/spezial/news.shtml as an example).
% Uh, sir, my news feeder box peaks at over 55Mbps outbound and averages
% (according to MRTG and weekly stats) 26.2Mbps. (It hit 61Mbps in
% September once...)
I'm still looking for a way to have innfeed deliver rrd-ready data --
anyone else trying this already?
% So, yah, a lot of bandwidth that is expensive being used for an
% absolutely worthless cause - a larger penis^H^H^H^H^Hfreenix rating.
Binaries do burn bandwidth and thus slow down the transmission of other
articles (depends of your setup, of course); while they can help you
with the penis^H^H^H^H^Hfreenix ratings, a dozen of non-binary feeds
to lesser connected sites (doing inpaths) would be much cheaper and far
more effective.
Brad Knowles
(Mike Horwath) wrote:
> Uh, sir, my news feeder box peaks at over 55Mbps outbound and averages
> (according to MRTG and weekly stats) 26.2Mbps. (It hit 61Mbps in
> September once...)
If I did that, our Network Manager would personally hunt me down and
kill me. We only have two major routes out of this network, and they're
both E-3s. One is a low fixed-cost (no bandwidth utilization surcharge),
the other is much more expensive.
Trying to cram 61Mbps down 68Mbps total bandwidth that we have for the
entire network would be a sure ticket to a slow and painful death. ;-)
Brad Knowles
(Mike Horwath) wrote:
> No one wants to peer with me anymore...well, they already do and
> everyone else over the last few months really just want a feed, not a
> peering arrangement.
At this point, I look at the Top 30, and I realize that I have
peerings with almost all of them (or their counterparts, such as sol.net
vs. execpc.com). I need to start expanding my peerings in other parts of
the globe, as well as with primary sources of news (even if those places
aren't major routers of news).
As for feeds, I'm willing to consider giving those too. I am getting
a little more selective in who I'll give a feed to, although almost all of
the feed requests I've gotten so far have come from countries where I'd
like to expand our peerings, so I've pretty much granted them without
question.
Brad Knowles
Gunshannon) wrote:
> I have been pretty busy, but I have decided to start playing with those
> different size queues for a bit. Hopefully to find the optimum breakpoint
> based on size. I might even see if maybe three breaks might be the way to.
I based my break points on the stats I found at
<http://www.news-feed.inet.tele.dk/innreport/news-notice.1999.10.11-05_00_49.html#inn_dist>.
16KB covers about 75% of all articles they saw.
Kai 'wusel' Siering
% > http://gaspode.uu.org/newsadmin/freenix-TOP200-sites.199909 -- might
% > be slow, it's an rather old machine ;)
%
% Out of curiosity, are you planning on updating this page monthly, as
% Iain Lea was supposed to be doing with his version of this information?
Not unless you asked ;) I'm updating this list monthly with a number
of servers in .at. .ch and .de for a local mailinglist, but it's not
that difficult ("head -${n} list >target; ./script.pl target") to do
it for the first "n" sites also. If there's a second voice in favor,
I'll tell cron about it. Ciao,
kai
--
Kai 'wusel' Siering eMail @ home: wu...@uu.org
Traveller on the Information Highway doing full-time administration.
The views expressed here are not neccessarily those of any employer.
- Looking for peers for several non-european national TLHs, please -
Terry Kennedy
> I have been pretty busy, but I have decided to start playing with those
> different size queues for a bit. Hopefully to find the optimum breakpoint
> based on size. I might even see if maybe three breaks might be the way to.
I'd be very interested in seeing what sizes you think are good breakpoints.
Brad Knowles
(Mike Horwath) wrote:
> Yah, I have done a couple for the same reasons, but just offering a
> feed, full blown at 55GB/day, well, that costs real money, like 9Mbps
> worth or so.
If you do the math, 55GB/day should work out to about 5Mbps
utilization, on average. Are you talking about peak data rates?
--
Brad Knowles <br...@shub-internet.org> <http://www.shub-internet.org/brad/>
<http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=0xE38CCEF1>
Brad Knowles
> I have been pretty busy, but I have decided to start playing with those
> different size queues for a bit. Hopefully to find the optimum breakpoint
> based on size. I might even see if maybe three breaks might be the way to.
Looking again at
<http://www.news-feed.inet.tele.dk/innreport/news-notice.1999.10.12-05_00_54.html#inn_dist>,
16KB gives you around 75% of the articles, and rather less than 5% of the
volume. There are huge peaks of volume at 487,099 and 688,862
bytes/article, and it looks to me like a 400KB break point (in addition to
the break over at 16KB) would get around 95% of the articles and 30% of
the volume (32.6%, actually).
If you wanted to get a larger volume in this middle category, going up
to 487,099 bytes would add another 25.5%, for a total of 58.1% total
volume, although this doesn't seem to be a significant additional
percentage of total number of articles. A break over of 256KB would
appear to get you around 13% of the volume, and something like 87% of the
articles.
From this quick scan, I'd be willing to bet that a 400KB break over
would be pretty good, although I don't know how close to "optimal" it
would be. Of course, a site that had a 256KB or 512KB maximum article
size would kinda screw up this distribution. ;-)
And if we're setting a high breakover, so that we can get three
categories, I'm wondering if maybe we don't want to think about moving the
lower breakover, to something closer to 4-8KB, where we'd still get up to
70% of the articles, but they'd take less time to transmit than if we had
a 16KB break over.
Brad Knowles
(Mike Horwath) wrote:
> I would guess the answer will be YES.
I would have thought that too. However, I have since removed the
memory-based filesystem I was using, and while the average amount of idle
time hasn't significantly decreased, instead of spending most of my
non-idle time in user mode, it's now mostly spent in system.
From looking at the system, I'd say my disk accesses are now ten times
as frequent, which greatly increases my probability of hitting significant
waits due to rotational delays, etc....
I haven't really waited long enough to be sure, but it looks like my
hourly stats have taken a real nose-dive, too. I sure wish I'd been smart
and looked at iostat and vmstat on the system before I unmounted that
memory-based filesystem, so that I could compare with what I've got
now....
--
Brad Knowles <br...@shub-internet.org> <http://www.shub-internet.org/brad/>
Are you a USENET news server administrator outside of North America or
Brad Knowles
(Mike Horwath) wrote:
> How big is your history at this time?
Well, that's part of what I meant when I said that I hadn't watched it
for long enough to be sure, but now the dhistory file is currently
32,525,328 bytes. I can tell you that I'm not sending nearly as many
articles as I'm used to. The whole system just seems quite a bit slower
to transit news articles.
Brad Knowles
br...@shub-internet.org (Brad Knowles) wrote:
> Well, that's part of what I meant when I said that I hadn't watched it
> for long enough to be sure, but now the dhistory file is currently
> 32,525,328 bytes. I can tell you that I'm not sending nearly as many
> articles as I'm used to. The whole system just seems quite a bit slower
> to transit news articles.
Note that previously, I had never seen load average peaks on this
machine over 4. I just saw a load average of 12, and I'm not sure if that
was the peak. I think I'm going to re-enable that mfs....
Brad Knowles
(Mike Horwath) wrote:
> @ 32MB in size...there is no reason for it not to be cached except if
> you don't have enough 'buffer' allocated.
Only the index gets mmap()ed, and that's the only part that would
normally be cached. Everything else is done with lseek() and direct reads
and writes, so as to avoid thrashing the buffer cache.
Still, this is only after less than a day. With a 384MB mfs, I've
seen it get 60% full once it's been running for several days.
> Show me output of the top of 'top'.
I've just re-enabled the mfs, so it's going to be a few days before
things settle down far enough that I can show you anything meaningful.
Jeremy
Russ Allbery <r...@stanford.edu> wrote:
>> I don't split my outgoing feeds by size (except for one peer that
>> requires it) - if I wanted to chase a higher Freenix spot I suppose I
>> should do that.
>
> If you just want to utilize your machine better and don't care about
> Freenix at all, you should do that.
I don't think you need not to care about Freenix to split the feeds. I
do it routinely, and I think it actually helps. It certainly helps to
reduce backlogs, which should in turn help Freenix ratings. And, I'm at
#8 now, so I must be doing something right. :)
> And yeah, if you really want to climb Freenix, not propagating binaries
> at all will help. The length of time it takes to accept, store, and
> propagate a binary is badly disproportionate to how much it counts
> towards your ranking.
It's also kind of cheating, though, isn't it? You end up comparing full
feed servers with partial feed servers.
--
Jeremy | jer...@exit109.com
"There is nothing worse than having a spare couple of hours and you
can't find an open server to abuse." --Tim Thorne
Jeremy
> Looking at the last six months for the current TOP 200, I don't see
> really major changes; I've put the table I'm referring to online at
> http://gaspode.uu.org/newsadmin/freenix-TOP200-sites.199909 -- might
> be slow, it's an rather old machine ;)
>
> Most changes seem to be either related (supernews, remarq, rqdq)
> or consistent with the trend of the last month(s), e. g.
What I'm trying to figure out is why there is still a difference in
the numbers between "supernews.com" and "remarq.com". I started using
both for transitional reasons, but at this point, and for some time now,
anything I stamp with one, I stamp with the other, period. They go into
the server, get stamped with *both* entries, and then leave. It seems
to me that the numbers for the two path entries should be identical, and
yet they aren't. I'd like to drop one of them (if only to keep from
hogging two slots near the top of the list), but I'd like to figure out
why they're different first...
--
Jeremy | jer...@exit109.com
"Do you think a princess, and a guy like me..." --Han Solo
Joe Greco
Two words: "path preloading".
... JG
bill davidsen
Jeremy <jer...@exit109.com> wrote:
| > And yeah, if you really want to climb Freenix, not propagating binaries
| > at all will help. The length of time it takes to accept, store, and
| > propagate a binary is badly disproportionate to how much it counts
| > towards your ranking.
|
| It's also kind of cheating, though, isn't it? You end up comparing full
| feed servers with partial feed servers.
Anyone who doesn't take 60GB/day is a partial feed server. I know some
of the sites in the top 100 have a limit of 1-2MB on article size. It's
a matter of degree, and if you have an article limit of 500k or larger
you probably transit 95% or better of articles.
I did a breakdown of size on two transit servers which have a 1MB
cutoff, and I got something like this for an arbitrary time period:
Server 1:
size<k arts MB %size %arts %incr-k %cuml-k
16 461752 814.4 3.4 79.8 0.0 3.4
100 51607 2652.5 11.2 8.9 325.7 14.6
256 31348 4989.1 21.0 5.4 143.9 35.6
512 24579 9605.9 40.5 4.3 113.6 76.1
1024 9041 5672.8 23.9 1.6 31.4 100.0
Server 2:
size<k arts MB %size %arts %incr-k %cuml-k
16 484487 799.6 4.1 85.2 0.0 4.1
100 38778 1789.5 9.2 6.8 223.8 13.3
256 14238 2308.6 11.8 2.5 89.2 25.1
512 22236 8953.2 46.0 3.9 182.8 71.1
1024 8971 5631.2 28.9 1.6 40.7 100.0
The "%incr-k field is the percentage increase caused by raising your
article size from the previous line of the table to this value. So going
from 256k to 512k would require something between 120-180% increase in
bandwidth and storage.
Obviously to be useful you need to run this on a week of logs on your
own transit machine, as it depends on the characteristics of the feeds.
A lot of the peers of these machines only feed 500k articles due to
their bandwith limits or costs.
From artsize.pl, soon to be artsize.c since it is slower than a snail
in molasses as is.
--
bill davidsen <davi...@tmr.com> CTO, TMR Associates, Inc
When taking small children to a carnival, always have them go potty
*before* you let them go on the rides, and let them eat all the junk
food and candy *after*.