mail plain text message wrong charset

725 views
Skip to first unread message

szimszon

unread,
Feb 17, 2010, 3:16:23 AM2/17/10
to web2py-users
Hello!

I like to send email with:
context=dict(...)
message=response.render( 'template.txt', context )
mail.send( to = ['some email'],
subject = 'proba',
message = message,
)
There is some non-ASCII chars in the template, but I got

Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

And not charset='utf-8' :(

Can I change it somehow?

Tnx.

...

Date: Wed, 17 Feb 2010 09:10:53 +0100 (CET)
From: ....@.......hu

--===============1207362275==
Content-Type: multipart/alternative;
boundary="===============2040283988=="
MIME-Version: 1.0

--===============2040283988==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Sziasztok!

Új számla lett felvéve a rendszerbe az alábbi adatokkal:

========================================================
Eladó : New-Network
Számlaszám: ffgg45

....

szimszon

unread,
Feb 22, 2010, 2:30:42 PM2/22/10
to web2py-users
I wonder if somebody could turn me in the right direction?

mdipierro

unread,
Feb 23, 2010, 1:47:25 AM2/23/10
to web2py-users
You either encode the message in utf8 (default) or you specify a
different encoding= as argument of Mail

szimszon

unread,
Feb 23, 2010, 3:23:08 AM2/23/10
to web2py-users
I got:

File "/home/szimszon/fejlesztes/sajat/web2py/applications/raktar/
models/db.py", line 43, in <module>
mail=Mail( encoding = "UTF-8" ) #
mailer
TypeError: __init__() got an unexpected keyword argument 'encoding'

And I think my message is in utf8.

mdipierro

unread,
Feb 23, 2010, 3:48:22 AM2/23/10
to web2py-users
You must have an old version of web2py. Mail takes an encoding
argument

mdipierro

unread,
Feb 23, 2010, 3:30:06 AM2/23/10
to web2py-users
You may have an old version

class Mail:
def __init__(
self,
payload,
filename=None,
content_id=None,
content_type=None,
encoding='utf-8')


On Feb 23, 2:23 am, szimszon <szims...@gmail.com> wrote:

szimszon

unread,
Feb 23, 2010, 4:05:01 AM2/23/10
to web2py-users
tools.py │ 113589│febr 18 21.57

class Mail(object):
"""
Class for configuring and sending emails with alternative text /
html
body, multiple attachments and encryption support

Works with SMTP and Google App Engine.
"""

class Attachment(MIMEBase.MIMEBase):
"""
Email attachment

...
def __init__(self, server=None, sender=None, login=None,
tls=True):
"""
Main Mail object

Arguments::

server: SMTP server address in address:port notation
sender: sender email address
login: sender login name and password in login:password
notation
or None if no authentication is required
tls: enables/disables encryption (True by default)

...

So it's a most recent version of tools.py fresh from
http://www.web2py.com/examples/static/web2py_src.zip :) and there
isn't any encoding just
server=None, sender=None, login=None, tls=True


On febr. 23, 09:30, mdipierro <mdipie...@cs.depaul.edu> wrote:
> You may have an old version
>
> class Mail:
>         def __init__(
>             self,
>             payload,
>             filename=None,
>             content_id=None,
>             content_type=None,
>             encoding='utf-8')

this init is for class Attachment

I don't want to include attachments I'll only send plaintext messages
with Content-Type: text/plain; charset="utf-8" :)

Thadeus Burgess

unread,
Feb 23, 2010, 11:32:28 AM2/23/10
to web...@googlegroups.com
Mail has an encoding variable, unless you are running an older version
of web2py.

The code you posted is for the attachment class.

-Thadeus

> --
> You received this message because you are subscribed to the Google Groups "web2py-users" group.
> To post to this group, send email to web...@googlegroups.com.
> To unsubscribe from this group, send email to web2py+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
>
>

szimszon

unread,
Feb 23, 2010, 1:34:37 PM2/23/10
to web2py-users
This is the sniplet of the most recent (1.75.4) web2py's tools.py's
class Mail:

def __init__(self, server=None, sender=None, login=None,
tls=True):
"""
Main Mail object
Arguments::
server: SMTP server address in address:port notation
sender: sender email address
login: sender login name and password in login:password
notation
or None if no authentication is required
tls: enables/disables encryption (True by default)

I can't do another, the file is dated:


tools.py │ 113589│febr 18 21.57

Is it old?

Thadeus Burgess

unread,
Feb 23, 2010, 2:06:34 PM2/23/10
to web...@googlegroups.com
Your looking in the wrong place.

lines 172-182

On line 181 is the encoding.

http://code.google.com/p/web2py/source/browse/gluon/tools.py#181

def send(
self,
to,
subject='None',
message='None',
attachments=None,
cc=None,
bcc=None,
reply_to=None,
encoding='utf-8'
):


-Thadeus

szimszon

unread,
Feb 23, 2010, 2:33:51 PM2/23/10
to web2py-users
Okay. I see...

The default is utf-8 and if I set it to utf-8 manually there is no
difference. This is the generated mail content :( :

--===============1981490134==
Content-Type: multipart/alternative;
boundary="===============0000859693=="
MIME-Version: 1.0

--===============0000859693==


Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Thadeus Burgess

unread,
Feb 23, 2010, 2:47:21 PM2/23/10
to web...@googlegroups.com
Have you tried

mail.send(encoding='us-ascii') ?

-Thadeus

szimszon

unread,
Feb 23, 2010, 2:59:01 PM2/23/10
to web2py-users
No change:

--===============2046819983==
Content-Type: multipart/alternative;
boundary="===============1304182396=="
MIME-Version: 1.0

--===============1304182396==


Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

Thadeus Burgess

unread,
Feb 23, 2010, 3:01:23 PM2/23/10
to web...@googlegroups.com
I am unable to replicate this.

-Thadeus

szimszon

unread,
Feb 25, 2010, 5:32:34 AM2/25/10
to web2py-users
I tested it again,

I created a new scaffolding app with the admin interface then I
changed:
db.py:
mail=Mail() # mailer
mail.settings.server='localhost:25' # your SMTP server
mail.settings.sender='proba@localhost' # your email

default.py (controller)
def index():
"""
example action using the internationalization operator T and flash
rendered by views/default/index.html or views/generic.html
"""
response.flash = T('mailtest')
context=dict(name="Gipsz Jakab",
addr="Szt. Erzsébet körút")
message=response.render('emailbody.txt', context)
mail.send(to="szimszon",
subject="test email",
message=message)

return dict(message=T('Mail sent'))

made an emailbody.txt in views:
emailbody.txt
Üdvözlöm {{=name}}!

Címed: {{=addr}}

Ez egy teszt elektronikus levél.

Árvíztűrő tükörfúrógép!
------------------

Then I navigate to the site to send myself a letter:

----- cut -----
From proba@localhost Thu Feb 25 11:24:05 2010
Return-Path: <proba@localhost>
X-Original-To: szimszon
Delivered-To: szimszon@xxxxxxxxxxxxx
Received: from xxxxxxxxxxxxxxx (localhost [127.0.0.1])
by xxxxxxxxxxx (Postfix) with ESMTP id C542A31CBF
for <szimszon>; Thu, 25 Feb 2010 11:24:05 +0100 (CET)
Content-Type: multipart/related;
boundary="===============1518445122=="
MIME-Version: 1.0
To: szimszon@xxxxxxxxxxxxxxxx
Subject: test email
Message-Id: <20100225102405.C542A31CBF@xxxxxxxxxxxxxxx>
Date: Thu, 25 Feb 2010 11:24:05 +0100 (CET)
From: proba@localhost

--===============1518445122==
Content-Type: multipart/alternative;
boundary="===============1974471883=="
MIME-Version: 1.0

--===============1974471883==


Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Üdvözlöm Gipsz Jakab!

Címed: Szt. Erzsébet körút

Ez egy teszt elektronikus levél.

Árvíztűrő tükörfúrógép!

--===============1974471883==--
--===============1518445122==--

----- cut -----


charset: us-ascii


If I changed the
mail.send(to="szimszon",
subject="test email",
message=message)

to
mail.send(to="szimszon",
subject="test email",
message=message,
encoding="utf-8")
made no difference :(


On febr. 23, 21:01, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I am unable to replicate this.
>
> -Thadeus
>
>
>
> On Tue, Feb 23, 2010 at 1:59 PM, szimszon <szims...@gmail.com> wrote:
> > No change:
>
> > --===============2046819983==
> > Content-Type: multipart/alternative;
> > boundary="===============1304182396=="
> > MIME-Version: 1.0
>
> > --===============1304182396==
> > Content-Type: text/plain;charset="us-ascii"
> > MIME-Version: 1.0
> > Content-Transfer-Encoding: 7bit
>
> > On febr. 23, 20:47, Thadeus Burgess <thade...@thadeusb.com> wrote:
> >> Have you tried
>
> >>mail.send(encoding='us-ascii') ?
>
> >> -Thadeus
>
> >> On Tue, Feb 23, 2010 at 1:33 PM, szimszon <szims...@gmail.com> wrote:
> >> > Okay. I see...
>
> >> > The default is utf-8 and if I set it to utf-8 manually there is no

> >> > difference. This is the generatedmailcontent :(  :

> >> >> >>Mailhas an encoding variable, unless you are running an older version

> >> >> >> >> > > > > And notcharset='utf-8' :(

mdipierro

unread,
Feb 25, 2010, 6:20:02 AM2/25/10
to web2py-users
Because encoding="utf-8" is default. You have to figure
out what is the encoding you need.

> ...
>
> read more »

szimszon

unread,
Feb 25, 2010, 8:12:58 AM2/25/10
to web2py-users
I need utf-8 :)

On febr. 25, 12:20, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Because               encoding="utf-8" is default. You have to figure
> out what is the encoding you need.
>
> On Feb 25, 4:32 am, szimszon <szims...@gmail.com> wrote:
>
>
>
> > I tested it again,
>
> > I created a new scaffolding app with the admin interface then I
> > changed:
> > db.py:
> >mail=Mail()                                  # mailer
> >mail.settings.server='localhost:25'    # your SMTP server
> >mail.settings.sender='proba@localhost'         # your email
>
> > default.py (controller)
> > def index():
> >     """
> >     example action using the internationalization operator T and flash
> >     rendered by views/default/index.html or views/generic.html
> >     """
> >     response.flash = T('mailtest')
> >     context=dict(name="Gipsz Jakab",
> >                  addr="Szt. Erzsébet körút")
> >     message=response.render('emailbody.txt', context)
> >    mail.send(to="szimszon",
> >               subject="test email",
> >               message=message)
>

> >     return dict(message=T('Mailsent'))

> > > >> >> >> >> > > > > Can I change it...
>
> tovább »

Thadeus Burgess

unread,
Feb 25, 2010, 11:18:22 AM2/25/10
to web...@googlegroups.com
There might be an issue that web2py is encoding the email correctly,
however your SMTP server is re-coding this into its own encoding.

I use Google Apps to send all of my emails with my web2py instances. I
just checked some of the messages received and they are all encoded
"us-ascii".

Further investigation using a SMTP server that we manage ourselves
with (mt), all emails sent from this server are encoded utf-8.

This must not be a web2py issue but SMTP issue. Google and the like
decide to recode all of their messages, and there might not be
anything that can be done about it besides using your own SMTP server.

-Thadeus

szimszon

unread,
Feb 25, 2010, 5:20:14 PM2/25/10
to web2py-users
Hmmm... I don't use gmail or google to send email...

You talk about the character real representation like ÁÖ...
or the email part header


--===============1974471883==
Content-Type: text/plain;charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

if you say us-ascii?

I think the email message part header


--===============1974471883==
Content-Type: text/plain;charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

is part of the mail body and generated by python. I don't think it
gets rewritten by mail transport agent. So my primary concern is the
generated instruction about how to read the rest of the message. Is it
us-ascii or utf-8. I don't mind how is the text actually encoded...

On febr. 25, 17:18, Thadeus Burgess <thade...@thadeusb.com> wrote:
> There might be an issue that web2py is encoding the email correctly,
> however your SMTP server is re-coding this into its own encoding.
>
> I use Google Apps to send all of my emails with my web2py instances. I
> just checked some of the messages received and they are all encoded
> "us-ascii".
>
> Further investigation using a SMTP server that we manage ourselves
> with (mt), all emails sent from this server are encoded utf-8.
>
> This must not be a web2py issue but SMTP issue.  Google and the like
> decide to recode all of their messages, and there might not be
> anything that can be done about it besides using your own SMTP server.
>

Thadeus Burgess

unread,
Feb 25, 2010, 5:43:28 PM2/25/10
to web...@googlegroups.com
I did some more testing.

Server A -> Google SMTP -> receive Content-Type us-ascii
Server B -> Custom SMTP -> receive Content-Type utf-8

If I take Server B's credentials for the SMTP server, and stick them
in Server A this is the result..

ServerA -> Custom SMTP -> receive Content-Type utf-8
ServerB -> Custom SMTP -> receive Content-Type utf-8

And I also did the reverse

ServerA -> Google SMTP -> receive Content-Type us-ascii
ServerB -> Google SMTP -> receive Content-Type us-ascii

The only change is the credentials and SMTP server I am connecting to,
no code changes whatsoever.

-Thadeus

szimszon

unread,
Feb 26, 2010, 12:55:29 AM2/26/10
to web2py-users
hm. That's interesting :-o

I don't use credentials to send mail because I use my local mail
transport agent so I don't need it and I got Content-Type us-ascii
too.

So the error is possible in the code to send mail without
credentials...?

> >> >> > > >> >> >> >      ...
>
> tovább »

szimszon

unread,
Mar 2, 2010, 3:41:57 PM3/2/10
to web2py-users
It isn't good in 1.76.1 too :(

On febr. 25, 23:43, Thadeus Burgess <thade...@thadeusb.com> wrote:
> I did some more testing.
>
> Server A -> Google SMTP -> receive Content-Type us-ascii
> Server B -> Custom SMTP -> receive Content-Type utf-8
>
> If I take Server B's credentials for the SMTP server, and stick them
> in Server A this is the result..
>
> ServerA -> Custom SMTP -> receive Content-Type utf-8
> ServerB -> Custom SMTP -> receive Content-Type utf-8
>
> And I also did the reverse
>
> ServerA -> Google SMTP -> receive Content-Type us-ascii
> ServerB -> Google SMTP -> receive Content-Type us-ascii
>
> The only change is the credentials and SMTP server I am connecting to,
> no code changes whatsoever.
>
> -Thadeus
>
>
>
> On Thu, Feb 25, 2010 at 4:20 PM, szimszon <szims...@gmail.com> wrote:
> > Hmmm... I don't use gmail or google to sendemail...
>
> > You talk about the character real representation like ÁÖ...

> > or theemailpart header


> > --===============1974471883==
> > Content-Type: text/plain;charset="us-ascii"
> > MIME-Version: 1.0
> > Content-Transfer-Encoding: 8bit
>
> > if you say us-ascii?
>

> > I think theemailmessage part header

> >> >> >               subject="testemail",

> >> >> >               subject="testemail",


> >> >> >               message=message)
>
> >> >> > to
> >> >> >    mail.send(to="szimszon",

> >> >> >               subject="testemail",

> >> >> > > >> >> >> >      ...
>
> tovább »

szimszon

unread,
Mar 21, 2010, 8:04:51 AM3/21/10
to web2py-users
Okay I found the
----------- cut --------------
else:
text =
text.read().decode(encoding).encode('utf-8')
attachment.attach(MIMEText.MIMEText(text))
if html != None:
if isinstance(html, str):
html = html.decode(encoding).encode('utf-8')
----------- cut --------------
at line 285 in gluon/tools.py.

I changed the
attachment.attach(MIMEText.MIMEText(text))
line to

attachment.attach(MIMEText.MIMEText(text,_charset='utf-8'))

"_charset is the character set of the text and is passed as a
parameter to the
MIMENonMultipart constructor; it defaults to us-ascii. No guessing or
encoding is performed
^^^^^^^^
on the text data."

because of http://docs.python.org/library/email.mime.html:
----------------------- cut ------------------------
class email.mime.text.MIMEText(_text[, _subtype[, _charset]])
Module: email.mime.text

A subclass of MIMENonMultipart, the MIMEText class is used to create
MIME objects of major type text. _text is the string for the payload.
_subtype is the minor type and defaults to plain. _charset is the
character set of the text and is passed as a parameter to the
MIMENonMultipart constructor; it defaults to us-ascii. No guessing or
encoding is performed on the text data.

Changed in version 2.4: The previously deprecated _encoding argument
has been removed. Encoding happens implicitly based on the _charset
argument.
----------------------- cut ------------------------

But the Subject line isn't good because it has no encoding information
in it...

> > > is part of themailbody and generated by python. I don't think it
> > > gets rewritten bymailtransport agent. So my primary concern is the

> > >> >> > > >> >> >> >    Works...
>
> tovább »

mdipierro

unread,
Mar 21, 2010, 10:41:17 AM3/21/10
to web2py-users
Thank you! Fix is in trunk.

On Mar 21, 7:04 am, szimszon <szims...@gmail.com> wrote:
> Okay I found the
> ----------- cut --------------
>                 else:
>                     text =
> text.read().decode(encoding).encode('utf-8')
>                 attachment.attach(MIMEText.MIMEText(text))
>             if html != None:
>                 if isinstance(html, str):
>                     html = html.decode(encoding).encode('utf-8')
> ----------- cut --------------
> at line 285 in gluon/tools.py.
>
> I changed the
>                 attachment.attach(MIMEText.MIMEText(text))
> line to
>
> attachment.attach(MIMEText.MIMEText(text,_charset='utf-8'))
>
> "_charset is the character set of the text and is passed as a
> parameter to the
> MIMENonMultipart constructor; it defaults to us-ascii. No guessing or
> encoding is performed
>                                              ^^^^^^^^
> on the text data."
>

> because ofhttp://docs.python.org/library/email.mime.html:

> ...
>
> read more »

szimszon

unread,
Mar 21, 2010, 3:52:48 PM3/21/10
to web2py-users
Some more patch for headers:

gluon/tools.py from line 259:

payload['To'] = header.Header(',
'.join(to).decode(encoding).encode('utf-8'),'utf-8')
...
payload['Reply-To'] =
header.Header(reply_to.decode(encoding).encode('utf-8'),'utf-8')
...
payload['Subject'] =
header.Header(subject.decode(encoding).encode('utf-8'),'utf-8')
...
payload['Cc'] = header.Header(',
'.join(cc).decode(encoding).encode('utf-8'),'utf-8')
...
payload['Bcc'] = header.Header(',
'.join(bcc).decode(encoding).encode('utf-8'),'utf-8')

> > > > >> >> > > >> > On febr. 23, 20:06, Thadeus Burgess...
>
> tovább »

Reply all
Reply to author
Forward
0 new messages