Re: [Django] #8391: slugify template filter poorly encodes non-English strings
Django
-------------------------------------+-------------------------------------
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: reopened
Milestone: | Component: Template system
Version: SVN | Severity: Normal
Resolution: | Keywords:
Triage Stage: Design | Has patch: 0
decision needed | Needs tests: 0
Needs documentation: 0 | Easy pickings: 0
Patch needs improvement: 0 |
UI/UX: 0 |
-------------------------------------+-------------------------------------
Changes (by mitar):
* cc: mmitar@… (added)
* ui_ux: => 0
* easy: => 0
Comment:
I have added made [https://bitbucket.org/mitar/django-
missing/src/ebd814ed834b/missing/templatetags/url_tags.py slugify2
function] which first downcodes and then translates to slug. It behaves
exactly the same as its [https://bitbucket.org/mitar/django-
missing/src/ebd814ed834b/missing/static/missing/urlify2.js JavaScript
counterpart]. So now it is possible to have both in Python and JavaScript
same behavior.
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:31>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Type: Bug | Status: reopened
Milestone: | Component: Template system
Version: SVN | Severity: Normal
Resolution: | Keywords:
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------+---------------------------------
Changes (by ptone):
* stage: Design decision needed => Accepted
Comment:
see #16853 for a Turkish case
Seems that there have been no objections to the downcode then slugify
approach.
This seems ready for someone to take a shot at implementing that approach
in a patch.
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:32>
Django
------------------------------------+---------------------------------
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: reopened
Milestone: | Component: Template system
Version: SVN | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------+---------------------------------
Comment (by mitar):
You can take the above slugify2 function.
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:33>
Django
------------------------------------+---------------------------------
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: reopened
Milestone: | Component: Template system
Version: SVN | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------+---------------------------------
Comment (by yasar11732@…):
Above slugify2 function won't fix #16853.
{{{
# -*- coding: utf-8 -*-
import sys
import re
from django.utils import encoding
TURKISH_MAP = {
u'ş':'s', u'Ş':'S', u'ı':'i', u'İ':'I', u'ç':'c', u'Ç':'C', u'ü':'u',
u'Ü':'U',
u'ö':'o', u'Ö':'O', u'ğ':'g', u'Ğ':'G'
}
ALL_DOWNCODE_MAPS = [
TURKISH_MAP,
]
class Downcoder(object):
map = {}
regex = None
def __init__(self):
self.map = {}
chars = u''
for lookup in ALL_DOWNCODE_MAPS:
for c, l in lookup.items():
self.map[c] = l
chars += encoding.force_unicode(c)
self.regex = re.compile(ur'[' + chars + ']|[^' + chars + ']+',
re.U)
downcoder = Downcoder()
def downcode(value):
downcoded = u''
pieces = downcoder.regex.findall(value)
if pieces:
for p in pieces:
mapped = downcoder.map.get(p)
if mapped:
downcoded += mapped
else:
downcoded += p
else:
downcoded = value
return value
def slugify2(value):
"""
Normalizes string, converts to lowercase, removes non-alpha
characters,
and converts spaces to hyphens.
"""
import unicodedata
value = downcode(value)
value = unicodedata.normalize('NFD', value).encode('ascii', 'ignore')
value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
return re.sub('[-\s]+', '-', value)
print(slugify2(u"Işık ılık süt iç"))
}}}
This prints "isk-lk-sut-ic", but expected value is, "isik-ilik-sut-ic".
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:34>
Django
------------------------------------+---------------------------------
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: reopened
Milestone: | Component: Template system
Version: SVN | Severity: Normal
Resolution: | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------+---------------------------------
Comment (by mitar):
Ups. That was a bug. [https://bitbucket.org/mitar/django-
missing/src/d17bac5f8a5a/missing/templatetags/url_tags.py Fixed version of
slugify2].
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:35>
Django
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: new
Component: Template system | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
Note that slugify2 is now here: https://github.com/mitar/django-
missing/blob/master/missing/templatetags/url_tags.py
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:37>
Django
---------------------------------+------------------------------------
Reporter: bjornkri | Owner: nobody
Component: Template system | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
* status: new => closed
* resolution: => wontfix
Comment:
There's obviously more than one way to achieve slugification, depending on
your tastes and constraints.
If we try to be smart, we'll get dozens and dozens of tickets from people
who want to be smarter -- see the urlize filter for an example.
Django's implementation has the advantage of being simple and relying only
on the stdlib. Pretty good solutions are available externally.
The drawbacks of implementing something more complicated outweigh the
advantages at this stage.
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:38>
Django
---------------------------------+------------------------------------
Reporter: bjornkri | Owner: nobody
Type: Bug | Status: closed
Component: Template system | Version: master
Severity: Normal | Resolution: wontfix
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+------------------------------------
Comment (by kmike):
See also: https://code.djangoproject.com/ticket/23558
--
Ticket URL: <https://code.djangoproject.com/ticket/8391#comment:39>