Traffic estimates?
Matt McLeod
involved with a *,!binaries* feed or six?
I'm doing usage estimates for my employer. I'm guessing
that it's at least 200Mb/day, but something better than
my guesstimate would be nice...
--
"I offer you a new vision of Hell: Watching an entire ISO
committee trying to agree on what wine to have with their meal."
-- Tanuki the Raccoon-dog, ASR
Russ Allbery
> Does anyone have any estimates on the amount of stuff
> involved with a *,!binaries* feed or six?
If you don't filter misplaced binaries, I think it's in the 5GB range per
day, possibly higher, if my faulty memory is reliable on this point. With
misplaced binaries filtered out, probably something around 1GB or 2GB?
I should get the code in place to get better numbers on this....
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Russ Allbery
> My very limited (some Big 8, minimal alt.*, bofh.* and net.*) feed
> generally keeps me at about 2GB on spool on about a 90 day expire. And
> that's pre-cleaned and stripped of as much luse as possible... haven't
> bothered to look at actual daily traffic estimates in a while.
> So I'm totally curious what volume-wise percentage of a full feed I'm
> actually carrying. Heh. If that makes any sense.
I think Big Eight (and minimal alt.*) is probably only 25% or so of the
total text volume of Usenet these days, if that. There are a lot of other
rather large hierarchies now, tw.* being one of the largest and larger
than quite a few of the Big Eight hierarchies, plus a full alt.* feed is a
lot of posts.
Abby Franquemont
Russ Allbery <r...@stanford.edu> wrote:
>Matt McLeod <use...@wibble.cx> writes:
>
>> Does anyone have any estimates on the amount of stuff
>> involved with a *,!binaries* feed or six?
>
>If you don't filter misplaced binaries, I think it's in the 5GB range per
>day, possibly higher, if my faulty memory is reliable on this point. With
>misplaced binaries filtered out, probably something around 1GB or 2GB?
>
>I should get the code in place to get better numbers on this....
My very limited (some Big 8, minimal alt.*, bofh.* and net.*) feed
generally keeps me at about 2GB on spool on about a 90 day expire.
And that's pre-cleaned and stripped of as much luse as possible...
haven't bothered to look at actual daily traffic estimates in a while.
So I'm totally curious what volume-wise percentage of a full feed I'm
actually carrying. Heh. If that makes any sense.
--
Abby Franquemont "I might have amnesia -- but I'm not stupid!"
J. Random BOFH --Jackie Chan
Matt McLeod
that Russ Allbery did write:
>Abby Franquemont <ab...@ucan.foad.org> writes:
>
>> My very limited (some Big 8, minimal alt.*, bofh.* and net.*) feed
>> generally keeps me at about 2GB on spool on about a 90 day expire. And
>> that's pre-cleaned and stripped of as much luse as possible... haven't
>> bothered to look at actual daily traffic estimates in a while.
>
>> So I'm totally curious what volume-wise percentage of a full feed I'm
>> actually carrying. Heh. If that makes any sense.
>
>total text volume of Usenet these days, if that. There are a lot of other
>rather large hierarchies now, tw.* being one of the largest and larger
>than quite a few of the Big Eight hierarchies, plus a full alt.* feed is a
>lot of posts.
Ah. I think that at 2-5G/day, we'd opt for something a bit more
limited. Big Eight plus limited alt.* is about all I could really
justify anyway -- I mean, it *is* a business, even if that isn't
always obvious. :-)
Any guesstimates on big-8+minimal-alt? What matters here is the
amount of stuff which goes through the router, not the amount
which ends up in the spool -- at 19c/Mb you've gotta be faily
careful.
--
"Don't tell my momma I'm a sysadmin, she thinks I play piano in a whorehouse."
-- Alan J Rosenthal, ASR
Russ Allbery
> Ah. I think that at 2-5G/day, we'd opt for something a bit more
> limited. Big Eight plus limited alt.* is about all I could really
> justify anyway -- I mean, it *is* a business, even if that isn't always
> obvious. :-)
> Any guesstimates on big-8+minimal-alt?
I'd guess somewhere in the 400MB to 500MB range, provided that your
upstream didn't feed you binaries. If you throw in misplaced binaries,
I'm not sure.
Matt McLeod
that Russ Allbery did write:
>
>> Ah. I think that at 2-5G/day, we'd opt for something a bit more
>> limited. Big Eight plus limited alt.* is about all I could really
>> justify anyway -- I mean, it *is* a business, even if that isn't always
>> obvious. :-)
>
>> Any guesstimates on big-8+minimal-alt?
>
>I'd guess somewhere in the 400MB to 500MB range, provided that your
>upstream didn't feed you binaries. If you throw in misplaced binaries,
>I'm not sure.
Lots of upstreams, because I can't trust the average Australian
ISP to provide a useful newsfeed.
I had guesstimated 200Mb/day. Clearly things have grown somewhat
since I last ran that kind of feed. :-)
Ah well. I guess it'd make an Ausbone connection a *lot*
easier to justify...
--
"Yup, men are from Macs,
women are from VMS."
-- Erwin, User Friendly by Illiad
Abby Franquemont
Russ Allbery <r...@stanford.edu> wrote:
>Matt McLeod <use...@wibble.cx> writes:
>
>> Ah. I think that at 2-5G/day, we'd opt for something a bit more
>> limited. Big Eight plus limited alt.* is about all I could really
>> justify anyway -- I mean, it *is* a business, even if that isn't always
>> obvious. :-)
>
>> Any guesstimates on big-8+minimal-alt?
>
>I'd guess somewhere in the 400MB to 500MB range, provided that your
>upstream didn't feed you binaries. If you throw in misplaced binaries,
>I'm not sure.
I'm guesstimating my traffic on partial Big 8 and minimal alt.* roughly
averages 300-400MB/day. I just spent about 10 minutes watching server
connections with trafshow and generally, on my feed from Stanford, I'm
seeing between 1000-5000 CPS. Streaming mp3s off a remote web server
uses more immediate pipe than my newfeed, which I find amusing.
Again the key here is the pre-filtering of misplaced binaries and spam
and articles over a certain size, and the newsfeeds file entry seventeen
miles long so I don't have to have all of alt.fan.* just to carry a few
selected alt.fan.* froups. If it hadn't been for being able to get a
really clean feed of just what I want, I probably would have had to give
up having a news server.
Brian Edmonds
>> Does anyone have any estimates on the amount of stuff involved with a
>> *,!binaries* feed or six?
> If you don't filter misplaced binaries, I think it's in the 5GB range
> per day, possibly higher, if my faulty memory is reliable on this
> point. With misplaced binaries filtered out, probably something
> around 1GB or 2GB?
I carry a pretty full, non-binaries feed, and filter misplaced binaries.
My server reports from 500-600MB/day typically. Stats available at
http://newsxfer.cs.ubc.ca/~news/
Brian.
Matt McLeod
>Matt McLeod <use...@wibble.cx> staggered into the Black Sun and said:
>>Any guesstimates on big-8+minimal-alt? What matters here is the
>>amount of stuff which goes through the router, not the amount
>>which ends up in the spool -- at 19c/Mb you've gotta be faily
>>careful.
>
>do you have to pay that much domestically too?
Cost varies with provider. I'm doing the full 'net connectivity
proposal thing for work, covering pretty much everything.
There are four providers who seem happy to provide the small
stuff we want to start, and who are able to scale up to bigger
stuff without any great trouble. Of those, 19c/Mb is about the
cheapest for international traffic. Although you can get it
down to 15c/Mb by accepting the extra latency of sat (which is
OK for news, but not so great when you've got PHBs wanting to
know why voyeur.com or whatever is slower to load).
The most expensive is about 23c/Mb.
The hitch is that the 19c/Mb places are that price for *all*
incoming data, regardless of source. Even if the data originates
in their network, or even at the same POP, it's still 19c/Mb.
The pricier international option comes with differential pricing,
so what they're calling 'domestic' is 10c/Mb. But that's within
their network, not within .au. And their network, quite frankly,
sucks.
With the sorts of volumes we're talking for a basic newsfeed,
I'd say I'll be putting forward a case for hooking up with Ausbone.
That's $750/month plus the cost of the tail (call it another $1000
for something vaguely reasonable), which is a bit over half the
cost of bringing in 500Mb of news/day, includes a newsfeed,
HTTP cache-sharing with local ISPs, free data between local
peers, and 6c/Mb nationally through their network. The extra
cost is that you're expected to contribute something back --
would probably be a Sun patch mirror or something, since we
already keep that around.
So yeah, this stuff is obscenely expensive in .au. We can
get discounts -- UUNet are offering something like 20% simply
because of who we are, and one of the others is willing to
go down to around 17c/Mb if we're pulling in 50Gb/month --
we won't be moving that much stuff I hope, but even at 10Gb
there's a small discount.
But of course I'd rather not do business with UUNet. The best
option right now seems to be Davnet (Lionel's mob) who can do
us 2Mbps HDSL for about the same price as everyone else does
128kbps using frame relay for the bearer. But they don't seem
to grok ideas like "why the hell should I pay 19c/Mb for
data coming from other people on your network?" and "why
can't you be reasonable about newsfeeds? you want to charge
me 19c/Mb for your cruddy feed, plus 19c/Mb for all the top-ups
I'll have to do, and you're not willing to negotiate even
though you're gonna derive a lot of value from this?".
Part of the problem there is that I'm dealing with a salesguy
who doesn't seem to get the idea that there's a benefit to them
too if I'm running a decent news server and passing stuff back.
Not really sure how to get it through to him, because he seems
to defy all attempts at explanation...
>A basic assumption to the above is that one of your first moves in
>setting up a newsswerver at work would be to buy a nice bottle of wine
>for the bloke wot runs Netizen's swerver and persuade him to give you a
>feed. :-)
I might suggest that to the boss. That Netizen guy is fairly
generous, and he might even share the bottle. :-)
(But in all seriousness, it wouldn't help a lot. Netizen takes
a very small feed -- between 10 and 20 Mb per day, on average.
And of course it'd still cost 19c/Mb...)
--
"Personally, I take it as a compliment if someone quotes me,
whether or not they ask permission."
-- Lionel Lauer, ASR
Matt McLeod
that Anthony DeBoer did write:
>more-expensive connection will see virtually only international traffic.
>We have a bit of that here, too, with a local Internet exchange downtown
>giving us really fast cheap access to our near neighbours; our paid
>backbone connections suddenly aren't seeing any traffic that they could
>push straight out across the same POP.
That's true to some extent, but unfortunately Ausbone/VIX don't
have any of the larger providers as members. Ausbone are working
on it, but I won't hold my breath as Telstra/CCA/Optus/etc seem
unlikely (to me) to want to play nice.
But at least with Ausbone we'd save most of the news traffic
(probably still need topups, and as I understand it they're just
getting a batched feed via sat, running every night), the cache
sharing should save a bit too, and the mirrors of popular stuff
should also help.
I'll be suggesting that we establish direct connections to our
major customers too. That should save a boatload on all those
bloody Word and PowerPoint attachments that the manager types
like to mail around...
Tim Skirvin
>I'd guess somewhere in the 400MB to 500MB range, provided that your
>upstream didn't feed you binaries. If you throw in misplaced binaries,
>I'm not sure.
I'm getting ~450-500M/day, mostly from you, Russ. That's Big-8/
alt.* and a few regionals.
- Tim Skirvin (tski...@killfile.org)
--
<URL:http://www.killfile.org/~tskirvin/> Skirv's Homepage <FISH><
<URL:http://www.killfile.org/dungeon/> The Killfile Dungeon <*>
Brian Edmonds
> I'm getting ~450-500M/day, mostly from you, Russ. That's Big-8/ alt.*
> and a few regionals.
This is on deepthot? I'm hurt. I just checked the stats for yesterday,
and you accepted 114k articles from me, vs 96k from Russ. *pout*
Granted, he's usually ahead of me, but not by a "most" margin.
Brian.
Brian Edmonds
> This is on deepthot?
Bah, ok, so it's not deepthot. I don't know why I keep getting you and
Jay confused in my head. Terribly sorry. Would you like a feed? :)
Brian.
Tim Skirvin
>> This is on deepthot?
>Bah, ok, so it's not deepthot. I don't know why I keep getting you and
>Jay confused in my head. Terribly sorry. Would you like a feed? :)
Interesting question. I'd like to say yes, but considering how
precarious my news position is right now, I'll have to say no. At least,
not to news.crhc.
This reminds me - once I get my DNS going again on killfile.org, I
ought to start using feeds besides just using my work account for everything.
Who wants to feed net.*/bofh.* to a DSL system, and/or take feeds of all
the moderated groups that I'm in charge of?
Abby Franquemont
Tim Skirvin <tski...@killfile.org> wrote:
>
> This reminds me - once I get my DNS going again on killfile.org, I
>ought to start using feeds besides just using my work account for everything.
>Who wants to feed net.*/bofh.* to a DSL system, and/or take feeds of all
>the moderated groups that I'm in charge of?
I can definitely do ya bofh.* and net.*, so while I'm at it I suppose
there'd be no problem with me passing your moderated froup stuff along
to Russ. Of course, you might just wanna ask Russ. ;-) I've got several
bofh.* and net.* peers but for alt.* and Big 8 stuff I only really get
a feed from Russ and trade a short list with pir.net.
Russ Allbery
> This reminds me - once I get my DNS going again on killfile.org, I ought
> to start using feeds besides just using my work account for everything.
> Who wants to feed net.*/bofh.* to a DSL system, and/or take feeds of all
> the moderated groups that I'm in charge of?
I actively seek out peering with moderators. :) I'd be happy to spin up
another feed for you.