Better encoding for attachments
Martin Schlatter
8bit characters. So it should be possible to use a 8bit encoding (I
call it JMS encoding) for binary mail attachments. 7bit encodings like
base64 waste about 30-40% space.
In this 8bit JMS encoding for mail attachments, it should only be
necessary to escape some codes like 0d,0a,3d and 2e.
Is there anything else which is important?
Jens Martin Schlatter
Jem Berkes
Mark Crispin
> I think modern mail transfer agents do not have problems transporting
> 8bit characters. So it should be possible to use a 8bit encoding (I
> call it JMS encoding) for binary mail attachments. 7bit encodings like
> base64 waste about 30-40% space.
This is not a new idea. It is a very old idea.
First, there is the question of what you purpose to save by doing this.
Network bandwidth? If these binaries are large enough that the size
differential becomes significant to network traffic (remember that SMTP is
a very RTT-intense protocol), perhaps you should be using another means to
transport them.
Disk space? I'll be happy to give you a US cent which will allow you to
afford the additional disk space.
Second, there is the question of how binary is to be encoded. SMTP has
line length limitations, and treats 0x00, 0x0a, 0x0d specially. You may
also have to worry about 0xff in some buggy implementation. There's also
the special case of 0x0a0d2e0a0d and the question about possible variants
such as 0x0a2e0a. Remember that binary is much more sensitive to
corruption than text.
It turns out that there is a *lot* of software that has to be fixed. To
make things worse, since SMTP is store-and-forward, you have to have some
means of negotiating the fact that you are fixed along the entire path in
order to guarantee safe arrival, and you have to make this negotiation
mandatory. But in practice people don't do this. Consider the SMTP 8BIT
extension -- there are known implementations out there that advertise it
but do not comply with the downgrade-to-7BIT implementation requirements.
For all of these reasons and more, most people consider that the effort to
do SMTP binary was more trouble than the relatively modest gains were
worth. There are some people who do SMTP-binary, but it's a very small
number and typically only within an enterprise.
-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.
Alan Connor
<snip>
> Science does not emerge from voting, party politics, or public debate.
Anyone who would say the above knows nothing about the real world of so-called
'science'.
The 'truths' that scientists 'discover' had better conform to the wishes of
whoever is paying their salary and for all their equipment and energy and the
buildings that house their laboratories, or they will find themselves working
at a fastfood establishment so fast it will make their heads spin.
Which is why, of course, that the 'scientific truth' changes so regularly.
Formerly suppressed information suddenly comes into vogue.
Or a new corporation provides funding for a scientist, and it has different products
to sell.
Most high-profile trials feature scientists with reams of 'proof' debating
diametretrically-opposed viewpoints
On and on.
I have rarely seen such a naive proclamation on the Usenet, especially from
an obviously intelligent person.
But then, you refuse to use a proper sig delimiter, forcing people to see your
sig whether they want to or not, (after having been asked to correct it several
times) so I guess we are just being treated to another one of your neuroses.
AC
Martin Schlatter
> First, there is the question of what you purpose to save by doing this.
>
> Network bandwidth?
Yes, and transmission time. If you send or get a 2MB avi, 30% gain is
much.
JMS
Martin Schlatter
I have no idea if there are still 7bit MTAs. Are there?
Jens Martin
Mark Crispin
about meeting Bigfoot, had killfiled me for 90 days. Could 90 days have
passed already?
Why doesn't he do me a favor and make it permanent?
-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Mark Crispin
Yes, there are.
Mark Crispin
> > First, there is the question of what you purpose to save by doing this.
> > Network bandwidth?
> Yes, and transmission time. If you send or get a 2MB avi, 30% gain is
> much.
Maybe, maybe not. It depends upon the underlying speed and other
characteristics (e.g. compression) of your transmission media.
A typical SMTP transaction involves 6 or more RTTs. This is a substantial
amount of overhead, and is the bulk of the overall cost of an SMTP
session, although the relative cost decreases as the size of the data
being transmitted goes up.
You also overlooked the other point, which was that maybe you should look
into using another means to transmit a 2MB AVI than SMTP. That's likely
to be quite a bit more successful than any attempt to replace BASE64 as
the means to transport binary over SMTP.
The biggest thing working against you is deployment. BASE64 was
successful because it did not require SMTP servers to cooperate in any
way; in fact, a critical portion of its design was the assumption of
recalcitrant SMTP servers (and worse). For your project, not only would
you need to update the endpoints, but also the intermediaries (the SMTP
servers).
It's been well over a decade. 8-bit MIME is still not universal; the
reports of the death of QUOTED-PRINTABLE remain exaggerated. There are
still many 7-bit MTAs out there. What you will require for your effort is
much more ambitious than 8-bit MIME.
To make matters worse, binary is more sensitive to corruption than text.
Typically, text can tolerate some modest amount of corruption without
becoming unusable; certainly more than an AVI file can tolerate.
I don't oppose you undertaking such an effort; indeed it would be nice if
such a thing were to happen. I just think that you'll burn out long
before completion, and that your energies would be more usefully spent
doing something else.
Jem Berkes
> much.
Please don't email a 2MB file ;)
Alan Connor
>
>
> I thought that Alan Connor, the nutcase who posts on misc.survivalism
> about meeting Bigfoot, had killfiled me for 90 days. Could 90 days have
> passed already?
>
> Why doesn't he do me a favor and make it permanent?
>
You seem to think that you are exempt from the Usenet guidelines we all
follow, which are called "Netiquette", and that you have the right to
force everyone to see your sig whether they want to or not.
Myself and others have informed you on several occassions that it is
necessary to use a proper sig delimiter "-- " so that people can choose
whether to see your sig or not.
Here, once again, is where you can find all you need to know about the
subject:
If I see your illegal sig again, I will be contacting your school and
posting the results of that contact on these groups.
I'm sure they will be interested in learning that someone forcing people
to read the name of their fine institution is also engaged in libelous
behavior on the globe-spanning Usenet.
And refusing to follow the long-established rules therof.
Maybe they will very sensibly take a closer look at your department's budgets.
AC
Mark Crispin
> > I thought that Alan Connor, the nutcase who posts on misc.survivalism
> > about meeting Bigfoot, had killfiled me for 90 days.
> to read the name of their fine institution is also engaged in libelous
> behavior on the globe-spanning Usenet.
The truth is not libel. The evidence is easy to find on Google.
On July 3, 2003, Alan Connor posted misc.survivalism, and (among other
things) stated:
I have met several "Bigfoot".
The full text of that post can be found at:
On December 7, 2003, Alan Connor posted on comp.mail.misc that he had put
me in his kill file for 90 days.
The full text of that post can be found at:
Today is December 30, 2003, a mere 23 days later.
Mike Lawson
[...]
I'd be mighty careful about f*king with an email expert.
Bloke's liable to send 200 complaints from 200 different addresses at
50 ISPs. Or it would *look* like that to the folks at yer college.
He's right ya know. Yer sig is a baddy.
Another stuck up academic, are ya?
Think yer shit don't stink like that of us regular folk?
--
Mike Lawson
inetfactory.net
Seattle, WA, USA
Sam
> If I see your illegal sig again, I will be contacting your school and
> posting the results of that contact on these groups.
I couldn't agree more. You should DEFINITELY contact his school, and
absolutely post every little detail of what happened to this newsgroup
afterwards. Don't leave anything out.
> I'm sure they will be interested in learning that someone forcing people
> to read the name of their fine institution is also engaged in libelous
> behavior on the globe-spanning Usenet.
Beavis: remember what happened during your last encounter with the issue of
Internet access' Terms Of Service/Acceptable Usage Policy?
Sam
You're much better off compressing the entire SMTP session, or just the DATA
transaction. As long as you want to attempt to extend SMTP, this sounds
like a more palatable solution. Using nothing more than a generic Huffman
coding algorithm you'll end up with the exact same 30% savings, since the
overwhelming majority of binary messages will contain the 6-bit base64
alphabet, which will be neatly collapsed into a six-bit code. You might
even get adventurous and go for a fixed compression tree, saving an initial
pass over the message to compute the frequency distribution.
And for extra credit, use a traditional run-length encoded mechanism (gzip
will do nicely) for messages without binary content. Plain text should
compress nicely with that.
Martin Schlatter
> such a thing were to happen. I just think that you'll burn out long
> before completion, and that your energies would be more usefully spent
> doing something else.
Hmmm...ok...I see...thanks mark!
Jens
Mark Crispin
> I'd be mighty careful about f*king with an email expert.
Thank you very much for the funniest thing that I read in many many years!
I, of all people, am advised to be careful of an individual because that
person is "an email expert"!
Rather pathetic, actually.
> Bloke's liable to send 200 complaints from 200 different addresses at
> 50 ISPs. Or it would *look* like that to the folks at yer college.
This also is rather funny, and 1980s retro to boot; a threat to launch a
mailbomb!
> Mike Lawson
> inetfactory.net
> Seattle, WA, USA
What a pretty domain name! Unassigned too. Maybe I should buy it.
Nah...I don't need it...
But thank you for the very funny sock puppet show. I look forward to your
future entertainments.
Jonathan de Boyne Pollard
MS> transporting 8bit characters. So it should be possible
MS> to use a 8bit encoding (I call it JMS encoding) for binary
MS> mail attachments. 7bit encodings like base64 waste about
MS> 30-40% space. [...]
Those who don't learn from the mistake that is yEnc are doomed to repeat it.