Base64 encoded email

[Jeff Layman]

I received an email from my ISP about a service upgrade, but it was
unreadable. Example "text":

"SU1QT1JUQU5UIENIQU5HRVMgUkVHQVJESU5HIFlPVVIgRVVST1BBQ09NIEVNQUlMIFNFUlZJQ0U6
IDNyZCBPQ1RPQkVSCjIwMTIKCkFzIHBhcnQgb2Ygb3VyIG9uLWdvaW5nIHNlcnZpY2VzIG1haW50
ZW5hbmNlIGFuZCB1cGdyYWRlIHByb2dyYW1tZSwgd2UKYXJlIHVwZ3JhZGluZyBFdXJvcGFjb20n..."

Downloading the same message with Opera Mail it appears correctly:

"IMPORTANT CHANGES REGARDING YOUR EUROPACOM EMAIL SERVICE: 3rd OCTOBER
2012

As part of our on-going services maintenance and upgrade programme, we..."

I checked the message source (Ctrl-U) and it shows:
Content-Transfer-Encoding: base64

How do I get TB to display this as normal text in the same way that
Opera Mail does? I don't see Base64 in any of the TB encoding options.

--

Jeff

[Mike Easter]

Jeff Layman wrote:
> I received an email from my ISP about a service upgrade, but it was
> unreadable.

> I checked the message source (Ctrl-U) and it shows:
> Content-Transfer-Encoding: base64
>
> How do I get TB to display this as normal text in the same way that
> Opera Mail does? I don't see Base64 in any of the TB encoding options.

As an experiment, some time ago I sent myself a message encoded in b64;
OE has a configuration^1 that allows one to do that.

Tb displayed the message just fine.

I suspect that something is wrong with the way your provider's message
is constructed.

Here is an example of part of the message headers for the message that
Tb displays properly:

MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: base64

Here is what the message source shows of the body:

VGhpcyBpcyBiNjQuDQoNCg0KTWlrZSBFYXN0ZXI=

Here is what Tb displays of that body:

This is b64.

(and my sig)


^1 OE's configuration: OE/ Tools/ Options/ Send tab/ Mail sending format
- plain text - Plain Text Settings button/ Message format - select MIME
- Encode text using (choose one) None, Quoted-Printable, Base 64 ---
Alternative to MIME one can choose Uuencode.

--
Mike Easter

[Mike Easter]

Mike Easter wrote:
> Jeff Layman wrote:
>> I received an email from my ISP about a service upgrade, but it was
>> unreadable.
>
>> I checked the message source (Ctrl-U) and it shows:
>> Content-Transfer-Encoding: base64

> I suspect that something is wrong with the way your provider's message
> is constructed.

> MIME-Version: 1.0
> Content-Type: text/plain;
> charset="iso-8859-1"
> Content-Transfer-Encoding: base64

This wikipedia article discusses the use of Content-Transfer-Encoding
and b64 as a 'subset' of MIME

http://en.wikipedia.org/wiki/MIME#Content-Transfer-Encoding

If a mail tries to use the content-transfer-encoding header to 'do
something' and doesn't properly announce itself as MIME, how is the
agent supposed to 'know' that it is MIME?

I have run into discussions elsewhere about problems which arise when
content-transfer-encoding header is used without MIME announcement.



--
Mike Easter

[Mike Easter]

Mike Easter wrote:
> Mike Easter wrote:
>> Jeff Layman wrote:
>>> I received an email from my ISP about a service upgrade, but it was
>>> unreadable.

>> I suspect that something is wrong with the way your provider's message
>> is constructed.

> I have run into discussions elsewhere about problems which arise when
> content-transfer-encoding header is used without MIME announcement.

Although I can't find exactly the discussion I was looking for, I found
some other discussions about how people are sending email with their
'phones which email is not constructed compliantly.

I suspect that if the headers of this problematic message are examined,
they may not only show a missing MIME header, but also that the agent
was a 'phone's app.

But, I'm getting ahead of the available information; I may be on the
wrong pony.



--
Mike Easter

[Jeff Layman]

Thanks for the input. But if there is a problem in the way that the
message was constructed, why did Opera Mail have no problem with it?

FWIW, the message displays without problem in FF if I use my ISP's
webmail facility.

--

Jeff

[Mike Easter]

Jeff Layman wrote:
> Mike Easter wrote:
>> Mike Easter wrote:
>>> Mike Easter wrote:
>>>> Jeff Layman wrote:
>>>>> I received an email from my ISP about a service upgrade, but it was
>>>>> unreadable.
>>
>>>> I suspect that something is wrong with the way your provider's message
>>>> is constructed.

>> I suspect that if the headers of this problematic message are examined,
>> they may not only show a missing MIME header, but also that the agent
>> was a 'phone's app.

> Thanks for the input. But if there is a problem in the way that the
> message was constructed, why did Opera Mail have no problem with it?

How about a theory that Opera (and ohers) is/are more tolerant of the
compliance problem I'm describing.

> FWIW, the message displays without problem in FF if I use my ISP's
> webmail facility.

You didn't say whether or not the view of the message source showed the
header:

MIME-Version: 1.0

If that is the missing piece, you might experiment with saving the
message, editing the headers with a text editor and re-saving, then
opening the modified message with Tb.


--
Mike Easter

[Michael A. Puls II]

On Tue, 02 Oct 2012 16:22:41 -0400, Jeff Layman <jmla...@invalid.invalid>
wrote:

Hard to say without the full source of the message as an eml file or mbox
file (via import/export tools extension or exported in Opera with ctrl +
s).

--
Michael

[Jeff Layman]

On 03/10/2012 09:12, Mike Easter wrote:
> Jeff Layman wrote:
>> Mike Easter wrote:
>>> Mike Easter wrote:
>>>> Mike Easter wrote:
>>>>> Jeff Layman wrote:
>>>>>> I received an email from my ISP about a service upgrade, but it was
>>>>>> unreadable.
>>>
>>>>> I suspect that something is wrong with the way your provider's message
>>>>> is constructed.
>
>>> I suspect that if the headers of this problematic message are examined,
>>> they may not only show a missing MIME header, but also that the agent
>>> was a 'phone's app.
>
>> Thanks for the input. But if there is a problem in the way that the
>> message was constructed, why did Opera Mail have no problem with it?
>
> How about a theory that Opera (and ohers) is/are more tolerant of the
> compliance problem I'm describing.
>
>> FWIW, the message displays without problem in FF if I use my ISP's
>> webmail facility.
>
> You didn't say whether or not the view of the message source showed the
> header:
>
> MIME-Version: 1.0

From the header:

X-MailCleaner-SPF: (invalid)
Content-Disposition: inline
Content-Transfer-Encoding: base64
Content-Type: text
MIME-Version: 1.0
X-Mailer: MIME::Lite 3.029 (F2.77; T1.35; A2.11; B3.08; Q3.08)

--

Jeff

[Michael A. Puls II]

On Wed, 03 Oct 2012 07:35:46 -0400, Jeff Layman <jmla...@invalid.invalid>
wrote:

> Content-Type: text

It needs to be "text/plain", not just "text" for Thunderbird.

Even though "text" is incorrect, Opera's built-in mail client for example
still displays the base64-decoded version.

Thunderbird must consider "text" not a text type and therefore doesn't
decode it as it wouldn't make sense to show the decoded version of a
non-text type inline.

Opera might just look for a mime type that starts with "text" to consider
it a text type. Didn't really check.

--
Michael

[Jeff Layman]

Thanks - that did it. Changing "Content-Type: text" to "Content-Type:
text/plain", saving as a *.eml and opening in TB allowed the message to
be read without problem.

--

Jeff

[Mike Easter]

Jeff Layman wrote:

> From the header:
>
> X-MailCleaner-SPF: (invalid)
> Content-Disposition: inline
> Content-Transfer-Encoding: base64
> Content-Type: text
> MIME-Version: 1.0
> X-Mailer: MIME::Lite 3.029 (F2.77; T1.35; A2.11; B3.08; Q3.08)

The docs for the MIME::Lite address the fact that text should be labeled
with the MIME term such as text/plain (as opposed to text).

See: A MIME Primer about 80% of the way down this page for MIME::Lite

http://search.cpan.org/~eryq/MIME-Lite-2.117/lib/MIME/Lite.pm#Content_types

text
Textual data, meant for humans to read. text/plain, text/html...


The same page, but at the top notes the newest v. is the one stamped in
the X-Mailer:

Latest Release: MIME-Lite-3.029


--
Mike Easter

[Arivald]

W dniu 2012-10-03 14:31, Michael A. Puls II pisze:

> On Wed, 03 Oct 2012 07:35:46 -0400, Jeff Layman
> <jmla...@invalid.invalid> wrote:
>
>> Content-Type: text
>
> It needs to be "text/plain", not just "text" for Thunderbird.
>
> Even though "text" is incorrect, Opera's built-in mail client for
> example still displays the base64-decoded version.
>
> Thunderbird must consider "text" not a text type and therefore doesn't
> decode it as it wouldn't make sense to show the decoded version of a
> non-text type inline.

TB do it wrong, imho. It should decode body according to
Content-Transfer-Encoding before even checking Content-Type.
Content-Type have any meaning on decoded bony only.

> Opera might just look for a mime type that starts with "text" to
> consider it a text type. Didn't really check.
>

Opera do it right. First decode (known method) then use Content-Type.
"text" is invalid, so just display message as stream of characters.


--
Arivald

[Mike Easter]

Jeff Layman wrote:
> Michael A. Puls II wrote:

>> It needs to be "text/plain", not just "text" for Thunderbird.

> Thanks - that did it. Changing "Content-Type: text" to "Content-Type:
> text/plain", saving as a *.eml and opening in TB allowed the message to
> be read without problem.

Thanks for following up experimentally with the edit of the header.


--
Mike Easter

[Michael A. Puls II]

Might be totally wrong, but this might be the relevant code:

<http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#81>
<http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#380>
<http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#221>


m_partIsText defaults to true.

But, if the content-type line doesn't have "text/" in it, m_partIsText is
set to false.

Then, somehow, I think each line of the base64 data gets to this part
<http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#300>
where the base64 data is intentionally copied as-is (as is in, not
base64-decoded).

So, although:

Content-Type: text

doesn't get Thunderbird to base64-decode the data,

Content-Type: text/

does.

So, as a workaround for this specific 'Content-Type: text' case,
<http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#380>
could be modified to check if the content-type is "text" or "text/" or
starts with "text/". That's what Opera seems to do ("footextbar/" and
"footextbar" don't count as text).

Also for Opera, for non-text types, Opera forces content-disposition:
attachment, doesn't render the file and just shows it as an attachment to
download.

--
Michael

[Jeff Layman]

No problem - I'm just pleased to have got it sorted out. But I'm still
not clear as to whether it's TB or Opera Mail which isn't following the
rules.

--

Jeff

[Mike Easter]

I think the configurer of the MIME-Lite might have done something wrong.

I haven't downloaded and looked at all of the MIME-Lite docs, but what
I've seen indicates that the MIME-Lite developer knows the plaintext
mime header is/contains text/plain -- but 'internally' the brief
overview docs refer 'loosely' to plain text as 'text'.

Perhaps the person configuring the software let that loose reference
influence hir somehow to use the reference to text (which should 'mean'

Content-Type: text/plain

... to cause hir to just stick text in there.

There is a specific syntax for every single punctuation mark and space
in a header line. Content-Type isn't Content type or something else;
colon single space isn't colon space space and text/plain isn't text.

Think of all of the browser struggles and conflicts and 'competitions'
for which browser could or should be able to 'handle' webpage errors.
There isn't nearly as much leeway in disturbing the syntax of message
headers, altho' in some pieces and parts there is a little latitude such
as the exact syntax of a Received: from traceline's from content and by
content and datestamp content.


--
Mike Easter

[Ken Whiton]

*-* On Wed, 03 Oct 2012, at 14:57:25 +0100,
*-* In Article <RJidnW0VA6Xb2fHN...@mozilla.org>,
*-* Jeff Layman wrote
*-* About Re: Base64 encoded email

> On 03/10/2012 13:31, Michael A. Puls II wrote:
>> On Wed, 03 Oct 2012 07:35:46 -0400, Jeff Layman
>> <jmla...@invalid.invalid>
>> wrote:
>
>>> Content-Type: text
>
>> It needs to be "text/plain", not just "text" for Thunderbird.
>

> Thanks - that did it. Changing "Content-Type: text" to
>"Content-Type: text/plain", saving as a *.eml and opening in TB
> allowed the message to be read without problem.

In case this happens again (and considering that the problem
message came from your ISP that would seem likely) you might want to
check out the Header Tools Lite extension. It enables you to
edit/modify the headers and/or bodies of e-mail messages within TB's
message base, without having to resort to .eml or other types of
external files.

<https://addons.mozilla.org/en-US/thunderbird/addon/header-tools-lite/>
or the developer's web site
<http://nic-nac-project.de/~kaosmos/headertoolslite-en.html>

Ken Whiton
--
FIDO: 1:132/152
InterNet: kenw...@surfglobal.net.INVAL (remove the obvious to reply)

[Jeff Layman]

Looks useful. Thanks.

--

Jeff

[»Q«]

Assuming that change were made, what would Thunderbird do after decoding
the base64 post with "Content-Type: text"? Try to display it as
text/plain or text/html or some other text/* mimetype?

[John H Meyers]

On 10/3/2012 3:11 PM, Jeff Layman wrote:

>>>> It needs to be "text/plain", not just "text" for Thunderbird.

> But I'm still not clear as to whether
> it's TB or Opera Mail which isn't following the rules.

Suppose you see that a car is turning into a street
that you are crossing. Now a "rule" of driving says
that the driver should be signaling the turn being made,
using the appropriate turn signals,
but no signals are actually visible as the car turns toward you.

So, what should you then do?

Continue crossing the street right into the car's path,
because there's no signal, or take the fact
that you can see where the car is going anyway
as being an overriding indicator that you'd
better stop or get out of the way?

The sender of your message apparently didn't make the right signal;
Opera saw it coming anyway, and adjusted itself accordingly,
while TB decided to let you get hit with the oncoming pile of garbage :)

The best "internet driving rule" that I've heard expressed
is rather informal, yet very obviously wise:

"Be strictly conservative in what you send,
but liberal in what you'll receive."

If everyone followed this extra "meta-rule,"
many things would work out better.


BTW, if some non-decoded "Base64" content does slip through a crack,
various on-line decoders are available, e.g.:

<http://ostermiller.org/calc/encode.html> [for simple plain text]

<http://www.motobit.com/util/base64-decoder-encoder.asp>
[for large files and/or binary data]

As you also noted, TB itself will decode it, if you insert some
headers in front of any "base64" encoded data,
separated from the encoded data by one empty line,
and save the result in a file named anyname.eml
which TB will then open as an email message:

MIME-Version: 1.0
Subject: Decoding some base64-encoded unknown data into a file
Content-Type: application/octet-stream; name="Decoded.data"
Content-Transfer-Encoding: base64

SU1QT1JUQU5UIENIQU5HRVMgUkVHQVJESU5HIFlPVVIgRVVST1BBQ09NIEVNQUlMIFNFUlZJQ0U6
IDNyZCBPQ1RPQkVSCjIwMTIKCkFzIHBhcnQgb2Ygb3VyIG9uLWdvaW5nIHNlcnZpY2VzIG1haW50
ZW5hbmNlIGFuZCB1cGdyYWRlIHByb2dyYW1tZSwgd2UKYXJlIHVwZ3JhZGluZyBFdXJvcGFjb20n

--

[Michael A. Puls II]

text/plain. If the sending client uses just "text" to mean anything else,
that client is way too out of line.

I suppose if there are a bunch of messages out there that use just "text"
but mean "text/html" for example, it might be slightly ok to assume
text/html. Then, for the messages that use "text" and mean text, the
content can be interpreted as HTML and will render the same as text most
of the time unless there are some markup characters in that text that get
interpreted as elements etc.

Or, the data could be analyzed to see if it looks like HTML (has <body>
for example) where text/html is used instead of text/plain.

But those two things are goofy and going overboard. I think assuming
text/plain is the only way to go.

With all that said though, this is the first I've heard of this problem
("text" in the content-type header instead of "text/plain"). If it's not
very common, it's probably too soon to change any code to work around it.

--
Michael

[Robert Miles]

Does it also work on newsgroups posts? If so, is it able to combine
multiple posts in binary newsgroups to handle situations where a long
post was split into several actual posts?

Robert Miles

[Robert Miles]

On 10/4/2012 4:31 AM, Ken Whiton wrote:

[Robert Miles]

On 10/6/2012 10:35 AM, »Q« wrote:
> On Wed, 03 Oct 2012 11:26:27 -0400
> "Michael A. Puls II" <shado...@gmail.com> wrote:
>
>> On Wed, 03 Oct 2012 10:08:41 -0400, Arivald
>> <arivald_@at_interia_dot_pl.mozilla.org> wrote:
>>
>>> W dniu 2012-10-03 14:31, Michael A. Puls II pisze:
>>>> On Wed, 03 Oct 2012 07:35:46 -0400, Jeff Layman
>>>> <jmla...@invalid.invalid> wrote:

[snip]

>> So, as a workaround for this specific 'Content-Type: text' case,
>> <http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgBodyHandler.cpp#380>
>> could be modified to check if the content-type is "text" or "text/"
>> or starts with "text/". That's what Opera seems to do ("footextbar/"
>> and "footextbar" don't count as text).
>
> Assuming that change were made, what would Thunderbird do after decoding
> the base64 post with "Content-Type: text"? Try to display it as
> text/plain or text/html or some other text/* mimetype?

Could it then list all the text/* types it knows how to decode and ask
the user which one to use?

Or is there some other header line that indicates which mimetype?

[»Q«]

It could, but it doesn't seem to have any capability for dealing with
such broken MIME'd messages. This is a bug in whatever software your
ISP is using to generate the e-mails, and while it would be nice if
Thunderbird could deal with it, the fix really needs to be on the other
end, IMO.

> Or is there some other header line that indicates which mimetype?

No, the Content-Type header is the only one that gives the type.
The MIME standards require both type and subtype, and you see in this
case why it's important for sending clients not to break standards.