SAGE chokes on French character

[dean moore]

When I was writing some other code this came out; finally decided to report it.  Do the following
in an online SAGE notebook:

1+1

We get two.  Now run the following:

# Limaçon
1+1

Get:

Exception (click to the left for traceback):
...
SyntaxError: Non-ASCII character '\xe7' in file /home/server2/sage_notebook/worksheets/dino/9/code/3.py on line 4, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/server2/sage_notebook/worksheets/dino/9/code/3.py", line 4
SyntaxError: Non-ASCII character '\xe7' in file /home/server2/sage_notebook/worksheets/dino/9/code/3.py on line 4, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

---

Sacre bleu!  It's in a comment!  Looking at the error & the web site it may be a Python thing & untouchable.
But can we get around this?  Shouldn't our French friends (and countless others) be able to use SAGE?

Dean

[Jason Grout]

dean moore wrote:
> When I was writing some other code this came out; finally decided to
> report it. Do the following
> in an online SAGE notebook:
>

> /1+1/

>
> We get two. Now run the following:
>

> /# Limaçon
> 1+1
> /
> Get:
>
> /Exception (click to the left for traceback):

> ...
> SyntaxError: Non-ASCII character '\xe7' in file
> /home/server2/sage_notebook/worksheets/dino/9/code/3.py on line 4, but
> no encoding declared; see http://www.python.org/peps/pep-0263.html for
> details
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/home/server2/sage_notebook/worksheets/dino/9/code/3.py", line 4
> SyntaxError: Non-ASCII character '\xe7' in file
> /home/server2/sage_notebook/worksheets/dino/9/code/3.py on line 4, but
> no encoding declared; see http://www.python.org/peps/pep-0263.html for

> details/
>
> ---
>
> /Sacre bleu!/ It's in a comment! Looking at the error & the web site

> it may be a Python thing & untouchable.
> But can we get around this? Shouldn't our French friends (and countless
> others) be able to use SAGE?

Apparently we can set the encoding of a python source file with a
comment like:

# -*- coding: utf_8 -*-

at the very top of the file (It seems like it has to be pretty much the
first line).

Are there any comments about automatically adding this (or utf_16?) to
the top of all of the notebook files?

If there are no comments, I can post a patch for the notebook right away.

Your example works with the above magic comment as the first line of a
python file, but does not work if that comment is moved down one line.

Jason

[William Stein]

Please definitely add this. Make a trac ticket asap for it. Thanks!!

[didier deshommes]

You can also use the codecs module to handle this:
codecs.open('stuff.txt','wb','utf-8')

opens the file for writing with this encoding in mind.

didier

[Jason Grout]

Done!

The patch is up at http://trac.sagemath.org/sage_trac/ticket/2399

Jason

[Jason Grout]

Is this better? The patch at #2399 does the magic comment trick. That
way, the encoding is part of the file, so the file is valid python in
and of itself.

Jason

[William Stein]

I think your solution at #2399 is better.

William

[Jason Grout]

Does someone want to review the patch positively? Maybe someone can
even convince mabshoff to apply it to 2.10.3 :)

Now we can make a demo with a notebook that has some very weird utf8
characters in it to show how Sage reaches out to the world via the web
interface (well, now=after patch is applied).

Something with very different-looking characters would be a nice visual
demonstration, like thai or cambodian or chinese or japanese characters.

Jason

[didier deshommes]

Frankly I don't know (and didn't realize you had already posted the
patch). Unless python3.0's internals change for reading files, yours
should work.

didier

>
>
>
> Jason
>
>
> >
>