SMTPHandler and Unicode

[norbert]

Hello,

I want to send error messages with SMTPHandler logging. But
SMTPHandler does not seem to be unicode aware. Is there something
doable without playing with sys.setdefaultencoding ?

import logging,logging.handlers
smtpHandler =
logging.handlers.SMTPHandler(mailhost=("smtp.example.com",25),
fromaddr="to...@example.com", toaddrs="to...@example.com",
subject=u"error message")

LOG = logging.getLogger()
LOG.addHandler(smtpHandler)

LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

gives : UnicodeEncodeError: 'ascii' codec can't encode character
u'\xe9' in position 117: ordinal not in range(128)

Thank you !

[Chris Withers]

norbert wrote:
> I want to send error messages with SMTPHandler logging. But
> SMTPHandler does not seem to be unicode aware. Is there something
> doable without playing with sys.setdefaultencoding ?

try MailingLogger:

http://www.simplistix.co.uk/software/python/mailinglogger

If you have unicode problems with that, I'd be interested in fixing them!

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[norbert]

On 5 juil, 13:17, Chris Withers <ch...@simplistix.co.uk> wrote:
> try MailingLogger:

>
> If you have unicode problems with that, I'd be interested in fixing them!

Your package has the same unicode problem :
import logging,logging.handlers
from mailinglogger.MailingLogger import MailingLogger
mailingLogger = MailingLogger(mailhost=('smtp.example.com',
25),fromaddr='to...@example.com',toaddrs=('to...@example.com',))
LOG = logging.getLogger()
LOG.addHandler(mailingLogger)

LOG.error(u"sans accent")
LOG.error(u"accentu\u00E9")

--> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
in position 7: ordinal not in range(128)

[Chris Withers]

norbert wrote:
> Your package has the same unicode problem :
> import logging,logging.handlers
> from mailinglogger.MailingLogger import MailingLogger
> mailingLogger = MailingLogger(mailhost=('smtp.example.com',
> 25),fromaddr='to...@example.com',toaddrs=('to...@example.com',))
> LOG = logging.getLogger()
> LOG.addHandler(mailingLogger)
> LOG.error(u"sans accent")
> LOG.error(u"accentu\u00E9")
>
> --> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
> in position 7: ordinal not in range(128)

Interesting, I don't know what the logging framework's position is on
unicode...

What happens when you try the same logging with just a FileHandler
registered? What encoding does the log file use?

[norbert]

On 5 juil, 14:32, Chris Withers <ch...@simplistix.co.uk> wrote:
> norbert wrote:
> > Your package has the same unicode problem :
> > import logging,logging.handlers
> > from mailinglogger.MailingLogger import MailingLogger
> > mailingLogger = MailingLogger(mailhost=('smtp.example.com',

> > 25),fromaddr='t...@example.com',toaddrs=('t...@example.com',))

> > LOG = logging.getLogger()
> > LOG.addHandler(mailingLogger)
> > LOG.error(u"sans accent")
> > LOG.error(u"accentu\u00E9")
>
> > --> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
> > in position 7: ordinal not in range(128)
>
> Interesting, I don't know what the logging framework's position is on
> unicode...
>
> What happens when you try the same logging with just a FileHandler
> registered? What encoding does the log file use?
>

a FileHandler works as expected, the log file being UTF-8 encoded. The
SMTPHandler is the only logger I know with this problem, maybe
connected to SMTPLib implementation ?

[Antoine Pitrou]

On Mon, 5 Jul 2010 06:17:38 -0700 (PDT)
norbert <ncau...@gmail.com> wrote:
>
> a FileHandler works as expected, the log file being UTF-8 encoded.

Ouch. Implicit encoding sounds like a bad behaviour.

> The
> SMTPHandler is the only logger I know with this problem, maybe
> connected to SMTPLib implementation ?

I suggest you report an issue on http://bugs.python.org

[Chris Withers]

Antoine Pitrou wrote:
> On Mon, 5 Jul 2010 06:17:38 -0700 (PDT)
> norbert <ncau...@gmail.com> wrote:

>> a FileHandler works as expected, the log file being UTF-8 encoded.
>

> Ouch. Implicit encoding sounds like a bad behaviour.

Yes indeed, hence my question on python-dev...

[norbert]

> > Ouch. Implicit encoding sounds like a bad behaviour.

Looking at the FileHandler source (
http://svn.python.org/view/python/trunk/Lib/logging/__init__.py?view=markup
) : the utf-8 encoding is a fallback. But *FileHandler family let you
specify the encoding you want, so that's OK I think.

But SMTPHandler does not have such a thing it sends its email with :
msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
self.fromaddr,
",".join(self.toaddrs),
self.getSubject(record),
formatdate(), msg)
...
smtp.sendmail(from,to,msg)

And there is no encoding in all this.

It seems pretty dangerous to me (so my first post) because your
application will work without any problem with a FileHandler and the
day you'll decide to send email in case of serious problem, it will
crash with a UnicodeError. I can't see any workaround, except by
subclassing SMTPHandler's emit method to be unicode-aware or at least
URF-8 aware.

[Vinay Sajip]

On Jul 5, 2:35 pm, Antoine Pitrou <solip...@pitrou.net> wrote:
> > a FileHandler works as expected, the log file being UTF-8 encoded.
>
> Ouch. Implicit encoding sounds like a bad behaviour.

UTF-8 is only used as a fallback in an exception handler, you can use
any supported encoding using the encoding= keyword argument to a
FileHandler.

>
> I suggest you report an issue onhttp://bugs.python.org

Yes, please do that and I will investigate.

Regards,

Vinay Sajip

[Vinay Sajip]

norbert <ncauderan <at> gmail.com> writes:

>
> crash with a UnicodeError. I can't see any workaround, except by
> subclassing SMTPHandler's emit method to be unicode-aware or at least
> URF-8 aware.
>

Well, you could use an approach like the one suggested here:

http://plumberjack.blogspot.com/2010/07/using-custom-formatter-to-deal-with.html

[norbert]

> Well, you could use an approach like the one suggested here:
>

> http://plumberjack.blogspot.com/2010/07/using-custom-formatter-to-dea...

That's nice, thanks. I'll use something like this. Just a thought : I
will use "errors=replace" in the call to the encode method to be sure
that the logger does not raise any exception.

[norbert]

On Jul 5, 3:35 pm, Antoine Pitrou <solip...@pitrou.net> wrote:
> I suggest you report an issue onhttp://bugs.python.org

done : http://bugs.python.org/issue9208

[Chris Withers]

Hi Norbert,

It's taken a ridiculously long amount of time (sadly due to no-one else
complaining) but this is now fixed in the source control for
mailinglogger, and will be in the next release:

https://github.com/Simplistix/testfixtures

[Chris Withers]

On 13/01/2012 20:17, Chris Withers wrote:
>> Your package has the same unicode problem :
>> import logging,logging.handlers
>> from mailinglogger.MailingLogger import MailingLogger
>> mailingLogger = MailingLogger(mailhost=('smtp.example.com',
>> 25),fromaddr='to...@example.com',toaddrs=('to...@example.com',))
>> LOG = logging.getLogger()
>> LOG.addHandler(mailingLogger)
>> LOG.error(u"sans accent")
>> LOG.error(u"accentu\u00E9")
>>
>> --> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9'
>> in position 7: ordinal not in range(128)
>

> It's taken a ridiculously long amount of time (sadly due to no-one else
> complaining) but this is now fixed in the source control for
> mailinglogger, and will be in the next release:
>
> https://github.com/Simplistix/testfixtures

Sorry, I meant this:

https://github.com/Simplistix/mailinglogger/commit/1c95f532c65ab18e1dd8513e1aa1ae328a19d249