FileSystemEncodingChanged exceptions

[Doug Evenhouse]

I'm spinning my wheels a bit with an FileSystemEncodingChanged exception that is thrown whenever I try to browse media files. I was happily uploading images and all was well for awhile. Now I cannot open the file browser at all without this exception being thrown. I don't believe I have created any folders nor uploaded any files with unusual encoding or names. As far as I can nothing would have changed with regard to locale settings. I removed the entire project and recreated it to no avail. The exceptions occur on both a staging server and a production server running Ubuntu 12.04, and deployed using vanilla fabric settings with Mezzanine 1.2.4. LC_ALL appears to be set correctly to "en_US.UTF-8".

Anyone have advice as to possible root causes and/or troubleshooting steps to take?

Traceback and request settings from the exception email - this is running on our internal staging server):

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 105, in get_response
    response = middleware_method(request, callback, callback_args, callback_kwargs)

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/mezzanine/pages/middleware.py", line 45, in process_view
    return view_func(request, *view_args, **view_kwargs)

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/django/contrib/admin/views/decorators.py", line 16, in _checklogin
    return view_func(request, *args, **kwargs)

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/filebrowser_safe/views.py", line 116, in browse
    elif fileobject.filetype == 'Folder' and fileobject.is_empty:

  File "/home/doug/virtualenvs/hccs_website/local/lib/python2.7/site-packages/filebrowser_safe/base.py", line 134, in _is_empty
    raise FileSystemEncodingChanged()

FileSystemEncodingChanged: Access was attempted on a file that contains unicode characters in its path, but somehow the current locale does not support utf-8. You may need to set 'LC_ALL' to a correct value, eg: 'en_US.UTF-8'.


<WSGIRequest
path:/admin/media-library/browse/,

GET:<QueryDict: {u'type': [u'Image'], u'pop': [u'1']}>,

POST:<QueryDict: {}>,
COOKIES:{'__utma': '74311563.372776308.1351892605.1351898182.1351966440.4',
 '__utmz': '74311563.1351892605.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
 'csrftoken': 'l754gpeXgo6WrMKmH5gQ0UpsG96acn41',

 'messages': '

81375d9cdb262e44aff2f71e2a66e74c9742ea2a$[["__json_message",40,"The requested Folder does not exist."]]',

 'mezzanine-admin-toolbar': '',
 'sessionid': '4fbd7d0d1511a2c157fe2bdc60aed3cf'},
META:{'CSRF_COOKIE': 'l754gpeXgo6WrMKmH5gQ0UpsG96acn41',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8',
 'HTTP_CONNECTION': 'close',

 'HTTP_COOKIE': '__utma=74311563.372776308.1351892605.1351898182.1351966440.4; __utmz=74311563.1351892605.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); sessionid=4fbd7d0d1511a2c157fe2bdc60aed3cf; mezzanine-admin-toolbar=; csrftoken=l754gpeXgo6WrMKmH5gQ0UpsG96acn41; messages="81375d9cdb262e44aff2f71e2a66e74c9742ea2a$[[\\"__json_message\\"\\05440\\054\\"The requested Folder does not exist.\\"]]"',
 'HTTP_HOST': 'hccs-website.evenhouseconsulting.local',
 'HTTP_REFERER': 'http://hccs-website.evenhouseconsulting.local/admin/people/person/add/',

 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
 'HTTP_X_FORWARDED_FOR': '192.168.30.53',
 'HTTP_X_FORWARDED_PROTOCOL': 'http',
 'HTTP_X_REAL_IP': '192.168.30.53',
 'PATH_INFO': u'/admin/media-library/browse/',

 'QUERY_STRING': 'type=Image&pop=1',
 'RAW_URI': '/admin/media-library/browse/?type=Image&pop=1',

 'REMOTE_ADDR': '192.168.30.53',
 'REMOTE_PORT': '80',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': 'hccs-website.evenhouseconsulting.local',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gunicorn/0.16.1',

 'gunicorn.socket': <socket._socketobject object at 0x304a8a0>,

 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7fcdddc80270>,
 'wsgi.file_wrapper': <class gunicorn.http.wsgi.FileWrapper at 0x1df10b8>,

 'wsgi.input': <gunicorn.http.body.Body object at 0x2fa16d0>,

'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}>

[Doug Evenhouse]

Thank you Pat for pointing me in (what appears to be so far) the right direction! Much appreciated!

I confirmed that starting the gunicorn processes directly from the command line corrected the issue. I don't have a gunicorn.sh script but could run the gunicorn_django script after stopping the supervisor service. I attempted to add LC_ALL=en_US.UTF-8 to my supervisor startup script as you did but for some reason that did not change anything for me. What I ended up doing that worked for me was to add the following line to the supervisor.conf template in Mezzanine's deploy folder:

environment=LANG="en_US.UTF-8",LC_ALL="en_US.UTF-8",LC_LANG="en_US.UTF-8"

Then after a "fab deploy" and then manually stopping and starting the supervisor service I was able to browse media files again.

By the way this post was particularly helpful to me:  http://albertoconnor.ca/blog/2012/Jul/21/unicodeencodeerror-when-uploading-files-django-usi

Thanks again Pat.

On Sunday, November 25, 2012 5:51:29 PM UTC-6, Pat James wrote:

I had the same problem today.  What a headache! 

Are you using supervisor?  I was able to narrow the problem down to supervisord not getting the locale environment variable correctly.  Running my gunicorn.sh startup script directly from the command line resulted in everything working correctly (so I think it is a general unix environment problem, not a Django/Mezzanine problem), it was only when supervisor was launching my gunicorn workers (via the same gunicorn.sh script) that the locale setting failed to pass through.  I believe I finally fixed it by hacking in a LC_ALL=en_US.UTF-8 line on my /etc/init.d/supervisor startup script.  I run supervisord as root and gunicorn as a less-privileged user.

-Pat

[Stephen McDonald]

Doug would you like to send a pull request with this change? Sounds like it's quite necessary.

--
Stephen McDonald
http://jupo.org

[Doug Evenhouse]

Steve, I'd like to have a little more time to be sure I'm out of the woods on this but, yes, I can send a pull request with this change. This will be my first pull request ever so be prepared to be patient :)

[Ken Bolton]

like!

[Stephen McDonald]

No rush at all Doug, better to take the time to get it right.

Thank you!

[Maciek Gryka]

Just wanted to say thanks as well - this solved a long-annoying problem for me! Would be good to see the change deployed :)

[Stephen McDonald]

Not sure if you're referring to this fix being added to Mezzanine proper, but Doug contributed this and it was merged in 4 months ago:

https://github.com/stephenmcd/mezzanine/commit/bbfd1382e8666340058cb9d2be5989ee39cb9d44

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bartosz Wiklak]

I had similar problem, but I wasn't using supervisor. A file name in /media/uploads/gallery has strange character encoding (the folder contains jspeg Austin, Athens, Amsterdam ...), I wasn't uploading anything there, suppose this files are uploaded by mezzanine. My locale is pl_PL.UTF8. 

The exception description is a little bit misleading.  

[Terry Brown]

+10 on this comment - spent most of a day setting LC_* to en_US.UTF-8 all over the place, didn't help.  Deleted

./static/media/uploads/gallery/Ávila, Spain.jpg

./static/media/uploads/gallery/Düsseldorf, Germany.jpg

./static/media/uploads/gallery/.thumbnails/Ávila, Spain-60x60.jpg

./static/media/uploads/gallery/.thumbnails/Düsseldorf, Germany-60x60.jpg

and all worked again.  Note that the system always worked on the Python 3.3 dev. environment, the problem with the above was on the Python 2.7 deploy environment.

Thanks Bartosz!

[Ken Bolton]

Hi Terry,

If you look at the fabfile.py, https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/fabfile.py#L339-L354, you can see how the system should be configured to handle the demonstrative gallery images included in Mezzanine.

hth,

ken

[Radomir Wojcik]

I ran into this problem today, and I don't use supervisor, nor do I use gunicorn. Today I switched to gunicorn though cuz I couldn't figure out what was going on. There were no issues in dev mode when I ran the server using runserver. The issues weren't obvious even when running the uwsgi server from command line, it "seemed" to work, but there was a complete issue when trying to run it as a service using upstart...  I should probably use the same stack everyone else is using and learn the fab file that comes with.. But anyways, I switched to gunicorn and it worked as a service so I thought ok problem solved. But then I got the Local Changed error.  What fixed it for me was exporting all these locale variables as described above using export.  Now even uwsgi is working great now.. I'm still sticking to sockets but I'll use gunicorn just to be safe.  I thought uWSGI was supposed to be faster thats why I was using it.  I don't know much about supervisor so I use upstart, and I use sockets because its slightly faster (no TCP/IP overhead).

 

export LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, LC_LANG=en_US.UTF-8

exec gunicorn -k eventlet ${DJANGO_WSGI_MODULE}:application \

  --name $NAME \

  --workers $NUM_WORKERS \

  --log-level=debug \

  --bind=unix:$SOCKFILE