Why are binaries posted in parts?
Jonathan Grobe
administrators who filter out large posts.] Are there limits in
news servers or newsreaders or NNTP which require this?
--
Jonathan Grobe
Curt Welch
All news servers have hard-coded limits on the maximum size articles
they can deal with. It's not directly an NNTP issue (NNTP has no
technical limits on article size). But indirectly it's an NNTP problem
because there is no way to restart the transfer of a large article
with NNTP if you have to abort the transfer mid-stream.
The design of servers is greatly simplified if you limit the maximum size
of articles. For example, with a 4MB max article size and 1GB memory sizes
common, it's safe to assume that the entire article will always be able to
"fit" in memory at once. So you can write your entire server based on the
assumption that articles can be held in memory at once (even that many
articles will fit in memory at once). If there was no article size limit,
and people wanted to send 10GB articles, you would have to write the server
code very differently. Or just make it a lot more complex so it can deal
with small articles one way (holding them in-core) and large articles
another way (writing them out to disk as you receive them).
It's also an historic issue. The first servers had size limits just to
make them easier to write, and once you have a large base of servers with
article size limits, there's not much point in trying to increase the limit
because the large articles will simply not be transmitted to the old
servers. And faced with that fact, people writting new servers have no
motivation to write the far more complex code required to support both
small and huge articles.
Mostly this all has happened because Usenet was designed to transport text
messages, not files, and putting a 1MB limit on the size of a single "text"
message seemed very resonable.
Though, there has been some discussion with the ideas of servers knowing
more about "multipart articles" and treating them more like a single post
even if they are sent as multiple articles. For example, with new
protocols, users could transfer the file to the server as one large
article, and the server would generate all the multipart articles. And
multipart articles could be better labeled in some well defined standard
fashion so that news servers could easilly match them up, decode them, and
present them to the user as a single file - and for servers which wanted to
limit the size of their feed, they would be easier to filter out as a
"whole" instead of the problems we see today where some parts are dropped
and the rest transmitted.
--
Curt Welch http://CurtWelch.Com/
cu...@kcwc.com Webmaster for http://NewsReader.Com/
Jonathan Grobe
>gr...@netins.net (Jonathan Grobe) wrote:
>> Why are binaries posted in parts? [Other than to get around news
>> administrators who filter out large posts.] Are there limits in
>> news servers or newsreaders or NNTP which require this?
>
>All news servers have hard-coded limits on the maximum size articles
>they can deal with.
>Mostly this all has happened because Usenet was designed to transport text
>messages, not files, and putting a 1MB limit on the size of a single "text"
>message seemed very resonable.
>
So are the usually limits in news servers 1MB? While I haven't looked
at any binaries for a long time, my recollection was that the multi-part
binaries were split into much smaller posts than that. What are the
usual sizes produced by Agent et al? As you increase the number of
parts the probability that a site has all of them goes does very
rapidly. So it would seem that the problem of missing parts could be
improved substantially if news readers were to adopt a 1MB posting
size.
--
Jonathan Grobe
Curt Welch
> So are the usually limits in news servers 1MB?
The default for INN was always 1MB. (or maybe 1,000,000 bytes - something a
saw recently made me wonder about that). I think it still is but I'm not
sure. Anyone can change it when it's installed but few do.
The default for Typhoon has always been 4MB.
> While I haven't looked
> at any binaries for a long time, my recollection was that the multi-part
> binaries were split into much smaller posts than that.
It varies, but 200K to 400K sizes are very common.
> What are the
> usual sizes produced by Agent et al? As you increase the number of
> parts the probability that a site has all of them goes does very
> rapidly.
You would think so, but that's not how it works. The probability that an
article will be lost goes up very rapidly with it's size. Most news feeds
are set up to give smaller articles priority, and many news feeds are size
limited so that everything over a fixed size is blocked. The larger the
article, the fewer paths it has to move though the net, and the more likely
it is to be dropped due to a backlog. A news server that doesn't have the
bandwidth to get a full feed would rather drop one 500K binary post than
drop 250 2K text posts anyday, so most servers that drop articles have been
configured to drop the larger articles first.
That effect will generally overwhelm the effect you speak off. A file
posted as 1000 2K parts will generally show up complete on servers that are
unable to get any other multipart post complete. But posting 1000 articles
instead of posting 10x200K is very disruptive to the newsgroup (everyone
has to download 1000 headers when they should have only been forced to
download 10). So that's why it's not done. But that is why multipart post
sizes keep creaping downward - people learn that the smaller the parts, the
more likely the post is to show up complete. This however is a bad habit -
if servers are dropping large binary posts, then you should not cheat their
wishes by making your post look like 1000 text posts. It's like cutting in
front of a line. When someone posts small articles, it doesn't "fix" the
problem, it just causes somebody else's post to be dropped instead of
theirs.
> So it would seem that the problem of missing parts could be
> improved substantially if news readers were to adopt a 1MB posting
> size.
No, posting large parts just increases the odds that a part will be
dropped.
The missing part problem is caused because thousands of ISPs and other
sites are running news servers that aren't able to keep up with a full feed
(or they are not willing to pay for the bandwidth required to get the full
feed). The owners don't upgrade them because it costs too much. They
simply don't have enough users willing to pay enough to justify the cost.
If users want complete multipart posts, all they have to do is pay for it.
There are many servers in the world where you can pay $10 per month or less
and get complete multipart posts. If the users want to fix the "missing
part" problem, all they have to do is pay for it. If they don't pay for
it, it won't be fixed.
Ivar Madsen
>The default for INN was always 1MB. (or maybe 1,000,000 bytes - something a
>saw recently made me wonder about that). I think it still is but I'm not
>sure. Anyone can change it when it's installed but few do.
>The default for Typhoon has always been 4MB.
what about Diablo?
--
Ivar Madsen Der med dette indlæg, repræsentere mig selv,
og dem der er enig med mig,,,
http://news.milli.dk med info om news.milli.dk opdateret 1/1
WEB serveren køre nu VirtualHost, og vil blive kraftigt ændret snarest
Curt Welch
> On 03 Jan 2002 20:52:05 GMT, cu...@kcwc.com (Curt Welch) wrote:
>
> >The default for INN was always 1MB. (or maybe 1,000,000 bytes -
> >something a saw recently made me wonder about that). I think it still
> >is but I'm not sure. Anyone can change it when it's installed but few
> >do. The default for Typhoon has always been 4MB.
>
> what about Diablo?
I don't remeber what it used for an article size limit. Maybe someone else
here can fill us in?