Gallery problems: UnicodeEncodeError: 'ascii' codec can't encode character

[Fredrik Blomqvist]

As soon as I try to upload an image, or an archive of images, with names consisting of special characters (eg. é or country specific åäö) it throws an error. I googled it and read that it may be the LANG setting that is wrong. I checked that however and it is set to en_US.UTF-8.

I got 2 different tracebacks (very similar though):

http://pastebin.com/q7bLvn9e

http://pastebin.com/DqFtFHcN

While I could just simply rename the images before I upload them, it would be awesome if I could skip that and everything just worked as it should.

Any help is appreciated!

Regards, Fredrik

[Josh Cartmell]

Generally this is caused by an incorrect locale, if you search the group (I don't remember exactly where the threads are) for locale you should come across a few threads about this, with some instructions on how to fix them.

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken Bolton]

Hi Fredrik,

Use the Fabric script at https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/fabfile.py#L346-L350 and https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/fabfile.py#L389-L391 as a starting points for ensuring your locales are correct.

ken

[Fredrik Blomqvist]

I'm sorry Kenneth but I'm at a loss about what to do with those scripts (or the specific lines you linked?). Something I forgot to mention is that my app is deployed at OpenShift, but not using Fabric (when I SSH'd in and tried to run update-locale manually it told me that the command doesn't exist).

I checked however and found that no LC_ envars exist (only the LANG one does). I found a thread on a forum where a guy had some problems with locales and he solved it by setting "LC_ALL" to his locale (which is what is happening in the first script I believe).

Could that info be to any help?

[Ken Bolton]

The Fabric scripts are intended to run on Debian-based OS, not ones based on RedHat. The command you seek is, I believe:

localedef -c -f UTF-8 -i en_US en_US.UTF-8

export LC_ALL=en_US.UTF-8

[Fredrik Blomqvist]

Oh. Well I ran that line (localedef, the export works fine but didn't improve the situation) and got this:

cannot open locale archive "/usr/lib/locale/locale-archive": Permission denied

Aaaaargh! I'm so sick and tired of OpenShift right now...

Anyway I did discover something interesting:

If I SSH and then jump into a Python prompt and then do a os.stat('filewithunicodename.jpg') it returns successfully. Doesn't that mean that something is incorrectly configured in Mezzanine/Django?

[Ken Bolton]

sudo it?

[Fredrik Blomqvist]

Oh you…if only I had permissions to sudo.

--
You received this message because you are subscribed to a topic in the Google Groups "Mezzanine Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mezzanine-users/RFdNTA4dr28/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mezzanine-use...@googlegroups.com.

[Ken Bolton]

On Wed, Jul 2, 2014 at 1:52 PM, Fredrik Blomqvist <fredrik.bl...@gmail.com> wrote:
>
> Oh you…if only I had permissions to sudo.

Pardon my ignorance of OpenShift.

How about:
$ rhc env set LC_ALL=en_US.UTF-8

[Fredrik Blomqvist]

It's okay, I'm so glad you care to help me at all.

I ran that (successfully) and then restarted everything. No success.

This is what "locale" outputs:

LANG=en_US.UTF-8

LC_CTYPE="en_US.UTF-8"

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_PAPER="en_US.UTF-8"

LC_NAME="en_US.UTF-8"

LC_ADDRESS="en_US.UTF-8"

LC_TELEPHONE="en_US.UTF-8"

LC_MEASUREMENT="en_US.UTF-8"

LC_IDENTIFICATION="en_US.UTF-8"

LC_ALL=en_US.UTF-8

This further strengthens my theory that the system is correctly configured, but Django/Mezzanine is not. Though with my level of knowledge in this particular area my theory is probably worth zero. Isn't there any way to force Django/Mezzanine to use UTF-8?

[Fredrik Blomqvist]

Something else that may or may not be totally irrelevant is that I don't have a language selection box on the admin interface. When I played around in Mezzanine (locally) with another project I had one. But on this project I don't (everything is configured the same I believe).

Does this setting matter:

LANGUAGE_CODE = "sv"

??

[Ken Bolton]

During my research earlier, I found a thread where someone was having similar problems. Funny thing was that his locale was fine when ssh'd in, but borked in his application. https://bugzilla.redhat.com/show_bug.cgi?id=904077

I think the next likely culprit is your database's encoding. Can you check that?

Django (and Mezz, which is just a Django application) will use the LANG, as far as I know. 

--

[Fredrik Blomqvist]

I looked up that bug thread and did the check (by putting locale in the pre_build hook) but it gave the same info as when I SSH'd in (en_US.UTF-8 everywhere).

DB:

show server_encoding;

server_encoding 

-----------------

 UTF8

(1 row)

show all;     // only lc_*

 lc_collate                      | en_US.UTF-8                                                                 | Shows the collation order locale.

 lc_ctype                        | en_US.UTF-8                                                                 | Shows the character classification and case conversion locale.

 lc_messages                     | en_US.utf8                                                                  | Sets the language in which messages are displayed.

 lc_monetary                     | en_US.utf8                                                                  | Sets the locale for formatting monetary amounts.

 lc_numeric                      | en_US.utf8                                                                  | Sets the locale for formatting numbers.

 lc_time                         | en_US.utf8                                                                  | Sets the locale for formatting date and time values.

Any difference between UTF-8 and utf8?

Is that all?

[Fredrik Blomqvist]

I just want to close this case by saying that this "fixed itself" after a couple of days of no work (one which was my birthday; this must have been my gift from OpenShift lol).

I can now upload images with unicode characters without problems!

Thanks everyone who helped me, one of you most likely produced the solution (just took some time for OpenShift to get it I guess)!

Regards, Fredrik

[Ken Bolton]

On Mon, Jul 7, 2014 at 8:46 AM, Fredrik Blomqvist <fredrik.bl...@gmail.com> wrote:

a couple of days of no work (one which was my birthday; this must have been my gift from OpenShift lol).

 

Many happy returns! (Which may or may not be (PEP 20)[19].)

[Adam Simon]

Not sure if this helps, but the problem in my case was solved by deleting images with non ascii characters in the media files.  Maybe it might be wise to reconsider including them in the demo.  Not that they are not beautiful images.


Adam

[Adam Simon]

Sorry, posted to soon - the problem disappears only after deleting:
<img class="img-responsive" src="{{ MEDIA_URL }}{% thumbnail image.file 131 75 %}">

On Monday, July 7, 2014 at 5:58:06 AM UTC-7, Kenneth Bolton wrote: