I experience problem with special chars like æøå in filename when using models.ImageField. and models.FileField

[MIL]

I experience problem with special chars like æøå in filename when
using models.ImageField.

What am I doing wrong?

models.py
picture = models.ImageField(upload_to='pics', blank=True,
verbose_name='Picture of you')

Using model forms

Let say I want to upload picture named "æøå.jpg"

If I do that, I get the following error msg:
'ascii' codec can't encode characters in position 59-61: ordinal not
in range(128)

Django ver 1.2 Alpha

Please help me thanks

[Bjørn Høj Jakobsen]

Hi

Try and put "# -*- encoding: utf-8 -*-#" in the top of the py script.

These danish characters really are a pain the ass.

Regards

Bjørn

[MichaleHjulskov]

Hi Bjørn

I have allready # -*- encoding: utf-8 -*-# in top of my settings.py
and models.py

Should I put it in all my .py files?

Thanks :o)

[Alexander Jeliuc]

did you tried so:

import sys

reload(sys)

sys.setdefaultencoding('utf-8')

or something similar?

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Karen Tracey]

The full traceback instead of just the error message would help people help you. Perhaps try this: http://docs.djangoproject.com/en/1.2/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror. Though that is in with the mod_python doc, it is not a mod_python specific problem. In general, no matter what deployment environment, your environment must be set to allow unicode to be passed to file functions, if you want to be able to manipulate files that contain non-ASCII characters.

Also, you should really update to a 1.2 release level. Why are you running alpha?

Karen

--
http://tracey.org/kmt/

[MichaleHjulskov]

Hi Karen, I did not know there was a new release, sorry.

So if I just install the new release, it will solve the problem just
like that?
Or do I still need to do something, in order to make it work with
special chars in the filenames?

Thanks :o)

Michael



On 13 Jun., 01:47, Karen Tracey <kmtra...@gmail.com> wrote:

> On Sat, Jun 12, 2010 at 8:37 AM, MIL <needb...@gmail.com> wrote:
> > I experience problem with special chars like æøå in filename when
> > using models.ImageField.
>
> > What am I doing wrong?
>
> > models.py
> >   picture = models.ImageField(upload_to='pics', blank=True,
> > verbose_name='Picture of you')
>
> > Using model forms
>
> > Let say I want to upload picture named "æøå.jpg"
>
> > If I do that, I get the following error msg:
> > 'ascii' codec can't encode characters in position 59-61: ordinal not
> > in range(128)
>
> > Django ver 1.2 Alpha
>
> The full traceback instead of just the error message would help people help

> you. Perhaps try this:http://docs.djangoproject.com/en/1.2/howto/deployment/modpython/#if-y....

[MichaleHjulskov]

Hi Alexander

No I havent tried that.
What does it do?
Im a little newbie in python, so its a little tricky for me to
understand. Sorry

Michael

On 13 Jun., 01:39, Alexander Jeliuc <jeliucalexa...@gmail.com> wrote:
> did you tried so:
>
> import sys
> reload(sys)
> sys.setdefaultencoding('utf-8')
>
> or something similar?
>

> > django-users...@googlegroups.com<django-users%2Bunsu...@googlegroups.com>
> > .

[Alexander Jeliuc]

It also depends on your filesystem i think

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[Alexander Jeliuc]

reload(sys) etc you can put it in start of your settings.py it will reload python utf-8 support so you will able to use different i had such problem executing spain/portugal project

import os, sys

reload(sys)

sys.setdefaultencoding( "utf-8" )

[Karen Tracey]

On Sun, Jun 13, 2010 at 6:18 PM, MichaleHjulskov <need...@gmail.com> wrote:

Hi Karen, I did not know there was a new release, sorry.

So if I just install the new release, it will solve the problem just
like that?
Or do I still need to do something, in order to make it work with
special chars in the filenames?

No, with the current release you still need to ensure your environment is set properly to allow Django to pass unicode to file system functions. The behavior I am guessing you are seeing (absent the full traceback to be sure) is not a bug in Django, it's an error in your environment setup. I pointed to the place in the doc where this is mentioned, and where some details of properly setting things up for Apache is covered, in the first paragraph of my first response.

I just mentioned upgrading from the alpha level of code because there are plenty of bugs that were fixed between alpha and release, and your life will likely be easier if you use release level code instead of alpha.

Karen

[Sam]

Hello,

Are you using MySQL and what is the collation of your database. I am
running 1.2.1 and encountering similar problems.

My issue involves utf8_bin collation on mysql 5.1.41-3ubuntu12.3. The
issue is thus: I create a model with an ImageField and save a new
instance to my utf8_bin collated mysql backend. This instance has a
file associated with non-ascii characters in it, e.g.
"Sinéad_OConnor.jpg". Next I attempt delete the file (via the admin)
and I get a:

'ascii' codec can't decode byte 0xc3 in position 18: ordinal not in
range(128)

FYI, MySQLdb does not return unicode strings with a utf8_bin collation
set.
for a brief description of that issue see:
http://code.djangoproject.com/ticket/8340#comment:4

The traceback from my exception reveals the exception being thrown in
"django/db/models/fields/files.py" in get_prep_value (line 248).
FileField is a subclass of Field, but implements the same backend
MySQL type (varchar) as a CharField. However it seems that FileField
and CharField have completely different implementations of
get_prep_db.

Here is CharField's implementation:
def to_python(self, value):
if isinstance(value, basestring) or value is None:
return value
return smart_unicode(value)

def get_prep_value(self, value):
return self.to_python(value)

Here is Filefield's:
def get_prep_value(self, value):
"Returns field's value prepared for saving into a database."
# Need to convert File objects provided via a form to unicode
for database insertion
if value is None:
return None
return unicode(value)

My experimentations revealed that if I replace the FileField
implementation of get_prep_value with the CharField, the exception
goes away. The issue is that the default encoding is ascii and so
unicode() called on a utf8 byte str blows up. The CharField
implementation simply checks if the value is an instance of basestring
and just passes it through. This latter approach seems better to me.
As it stands, I'm inclined to think this issue is a bug.

Thanks much,
-Sam





On Jun 13, 6:22 pm, Karen Tracey <kmtra...@gmail.com> wrote:

[Karen Tracey]

On Sun, Jun 13, 2010 at 8:22 PM, Sam <sam.v...@gmail.com> wrote:

As it stands, I'm inclined to think this issue is a bug.

It sounds like it. Could you open a ticket for it?

Karen
--
http://tracey.org/kmt/

[Sam]

Here it is!

http://code.djangoproject.com/ticket/13758

Any feedback on that ticket would be appreciated.

Regards,
Sam

On Jun 13, 11:20 pm, Karen Tracey <kmtra...@gmail.com> wrote:

[MichaleHjulskov]

Hi again Karen

Im sorry to say that I only understand some of these technical terms.
I really try to, but its very geeeky from my perspective.
(My Django inviroment is hosted at djangohosting.ch)

So I hope that you (or someone else) will explain a 1 2 3 procedure
for me to fix it, so that our users can upload their weird files :o)

In case I need to put this code somewhere:
export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'
Then please tell me where I should put it. in my settings.py or?

I have read Alexanders entry with interest, and Im wondering if it
would be the solution to change some things in "django/db/models/
fields/files.py" ??

Thanks for your help and patience :o)

Michael



On 14 Jun., 01:22, Karen Tracey <kmtra...@gmail.com> wrote:

[MichaleHjulskov]

By the way I have updatet to the latest release :o)

On 14 Jun., 01:22, Karen Tracey <kmtra...@gmail.com> wrote:

[MichaelHjulskov]

Hi Karen

My good friend just told me to take a look here and define a function
on upload_to that filters special chars from filenames.

http://docs.djangoproject.com/en/1.2/ref/models/fields/#django.db.models.FileField.upload_to

Do you aggree? Would that be the an appropriate solution to solve this
problem?

Thanks

On 13 Jun., 01:47, Karen Tracey <kmtra...@gmail.com> wrote:

> On Sat, Jun 12, 2010 at 8:37 AM, MIL <needb...@gmail.com> wrote:
> > I experience problem with special chars like æøå in filename when
> > using models.ImageField.
>
> > What am I doing wrong?
>
> > models.py
> >   picture = models.ImageField(upload_to='pics', blank=True,
> > verbose_name='Picture of you')
>
> > Using model forms
>
> > Let say I want to upload picture named "æøå.jpg"
>
> > If I do that, I get the following error msg:
> > 'ascii' codec can't encode characters in position 59-61: ordinal not
> > in range(128)
>
> > Django ver 1.2 Alpha
>
> The full traceback instead of just the error message would help people help

> you. Perhaps try this:http://docs.djangoproject.com/en/1.2/howto/deployment/modpython/#if-y....

[Karen Tracey]

On Tue, Jun 15, 2010 at 4:56 AM, MichaleHjulskov <need...@gmail.com> wrote:

Im sorry to say that I only understand some of these technical terms.
I really try to, but its very geeeky from my perspective.
(My Django inviroment is hosted at djangohosting.ch)

So I hope that you (or someone else) will explain a 1 2 3 procedure
for me to fix it, so that our users can upload their weird files :o)

In case I need to put this code somewhere:
export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'
Then please tell me where I should put it. in my settings.py or?

These things need to be set properly in the environment of the web server. If you don't have direct control over the web server configuration given your hosting set up, then you should contact your hosting provider and ask them to set things up properly.

Karen
--
http://tracey.org/kmt/

[Karen Tracey]

On Tue, Jun 15, 2010 at 8:18 AM, MichaelHjulskov <need...@gmail.com> wrote:

My good friend just told me to take a look here and define a function
on upload_to that filters special chars from filenames.

http://docs.djangoproject.com/en/1.2/ref/models/fields/#django.db.models.FileField.upload_to

Do you aggree? Would that be the an appropriate solution to solve this
problem?

You could use that as a workaround, but I would not call that the right answer. The right answer is for the environment of the web server to be set up so that unicode strings containing non-ASCII data can be successfully passed to Python file system functions.

Karen
--
http://tracey.org/kmt/

[MichaelHjulskov]

Okay I think I will do both though, since I prefer not to have special
wierd chars in my url's and filenames.

So I think I will
1. take contact to my hosting provider for correct setup.
2. use a filter function that removes special chars from the uploaded
file's filenames

I attempted to write that filter function for use in my models.py and
here it is:

def upload_to(path, attribute):
#removes special chars from filenames being uploaded, and can also
be used for seo purposes etc. to order images into special dirs
def upload_callback(instance, filename):
if attribute == '':
#e.g.: image = models.ImageField(upload_to =
upload_to('uploads/', ''))
return '%s/%s' % (path, unicode(slugify(filename)))
else:
#e.g.:
#title = models.CharField(max_length=25)
#image = models.ImageField(upload_to =
upload_to('uploads/', 'title'))
return '%s%s/%s' % (path,
unicode(slugify(getattr(instance, attribute))),
unicode(slugify(filename)))
return upload_callback


Please give me feedback :o)
Will that work?

Michael

[MichaelHjulskov]

You are totally right Karen
This solved the problem, just as you said it would:

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

Thanks alot Karen :o)

So now I only need to:
Fix the security issue to make sure that people dont upload php files
etc. (I just read a warning about that)
And if its possible, create some kind of resize function that can
resize huge picture files before they get saved.
And maybe create the rename function i menchined earlier in here, that
removes special chars.

On 15 Jun., 14:22, Karen Tracey <kmtra...@gmail.com> wrote:

> On Tue, Jun 15, 2010 at 8:18 AM, MichaelHjulskov <needb...@gmail.com> wrote:
>
> > My good friend just told me to take a look here and define a function
> > on upload_to that filters special chars from filenames.
>

> >http://docs.djangoproject.com/en/1.2/ref/models/fields/#django.db.mod...