Using Scheduler to send email unicode error. Possible bug.

[Leonel Câmara]

I am getting a strange bug where if I use this to send email:

def send_mail(*args, **kwargs):
    """
    Task for the scheduler to send mail.
    """
    mail.send(*args, **kwargs)


scheduler.queue_task('send_mail', pargs=[email, subject, html_message])

I get the usual unicode error in mail.send: 

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 21: ordinal not in range(128)

Whereas if I send the exact same mail using mail.send directly there are no problems. I am guessing it has to do with the JSON serialization going on in the scheduler which will convert my message/subject to unicode, then when it reaches mail for some reason it tries to encode it in 'ascii'. Any idea on how to fix this?

Right now my solution was to change send_mail to this:

def send_mail(*args, **kwargs):
    """
    Task for the scheduler to send mail.
    """
    utf8args = [arg.encode('utf-8') if isinstance(arg, unicode) else arg for arg in args]  
    utf8kwargs = {k:v.encode('utf-8') if isinstance(v, unicode) else v for k,v in kwargs.iteritems()}
    mail.send(*utf8args, **utf8kwargs)

This works, but it feels dirty. I don't see the average web2py user being able to solve this problem easily.

[Niphlod]

uhm. this feels to be due to https://github.com/web2py/web2py/blob/master/gluon/scheduler.py#L239 .
However, it should only be called to process vars (dict-based) and not args (list-based).
If you can't investigate further, I'll take a watch ASAP, in a few days though.

[Leonel Câmara]

Found it.

https://github.com/web2py/web2py/blob/master/gluon/scheduler.py#L313

It loads the args without using the object_hook like it does for the vars.

[Niphlod]

should it ?

[Leonel Câmara]

Well not exactly, that would also be a bug, line 313 should be:

            args = _decode_list(loads(task.args))

Since object_hook is only called for dicts and not lists, you need to do it after you load the list.

[edit] https://github.com/web2py/web2py/pull/818

[Niphlod]

If this patch is needed for the corner-case and doesn't impact who uses the scheduler without "corner-cases", I'm good with that.

The whole idea behind is that the function gets back what you dump. 

If not, maybe it's not the best solution.

[Leonel Câmara]

There's no way to serialize using json and getting the exact same thing. Because if you serialize an utf-8 string with characters like 'á' you won't get an utf-8 string back when you json.load it, you will get an unicode string.  

  

With this patch you get that utf-8 string back, however, you will also have unicode strings that you have placed in args converted to utf-8.

Note that this behaviour is already what happens with kwargs.  

  

I don't think it's a corner case for anyone using anything other than ascii.

The only solution I can think off to keep the exact argument types would be to serialize the args and kwargs with pickle instead of json or having a sort of custom json serializer.

[Niphlod]

damn. this would have been easier if we could chat ^_^

in my simple mind, I didn't get the discrepancy...the hook was there for dicts (kwargs) because
test = {'aè': 1, 'b' : 2}
test2 = simplejson.loads(simplejson.dumps(ciao))
test2['aè'] <-- keyerror
test2 = simplejson.loads(simplejson.dumps(ciao), object_hook=_decode_dict)
test2['aè'] <-- 1

however,
test = ['aè', 'b']
test2 = simplejson.loads(simplejson.dumps(test))
test[0] == test2[0] <-- False
test2 = _decode_list(simplejson.loads(simplejson.dumps(test)))
test[0] == test2[0] <-- False

so, the patch is fine (and I'll add another test to the scheduler's suite). Thanks for spotting it

At this point, I wonder if it shouldn't be better to just code an object_hook that comprehends both cases (lists and dicts) just for the sake of having a more straightforward piece of code.

[Leonel Câmara]

> damn. this would have been easier if we could chat ^_^

Eheheh it probably would, I'm usually on google hangouts if you ever want to chat.