I'd like to know if it's possible to do something about it? What are previous conclusions and facts since the last discussion? I'm new in Django and I may help in Python but not in js so I need your help ;).
My current problem is for french accents so it's not really difficult (I've pasted a js from a french blog app on my ticket) but I'm conscious there are more problems with other languages. Concerning utf-8 URLs, I don't know if it's really a good idea because this is actually associated to phishing...
On Thu, 2006-07-06 at 10:57 +0200, David Larlet wrote: > Hi all,
> I've recently added an enhancement (ticket #2282) about urlify without > checking for duplicate and there is already a proposal (my mistake) > and a discussion on this mailing-list which were unfortunatly closed > now: http://groups.google.com/group/django-developers/browse_thread/thread...
> I'd like to know if it's possible to do something about it? What are > previous conclusions and facts since the last discussion? I'm new in > Django and I may help in Python but not in js so I need your help ;).
There was reasonable consensus in one of the threads about doing something similar (but a bit smaller) than what Wordpress does. Now it's a case of "patches gratefully accepted". A lot of people say this is a big issue for them, so it's something that will be fixed one day, but nobody has put in a reasonable patch yet. When that happens, we can progress.
Malcolm Tredinnick wrote: > There was reasonable consensus in one of the threads about doing > something similar (but a bit smaller) than what Wordpress does. Now it's > a case of "patches gratefully accepted". A lot of people say this is a > big issue for them, so it's something that will be fixed one day, but > nobody has put in a reasonable patch yet. When that happens, we can > progress.
What's the expected scope of the downcoding? Would it be throwing a few dicts together in the admin js, or a callback to unicodedata.normalize?
On Fri, 2006-07-07 at 10:06 +0100, Bill de hÓra wrote: > Malcolm Tredinnick wrote:
> > There was reasonable consensus in one of the threads about doing > > something similar (but a bit smaller) than what Wordpress does. Now it's > > a case of "patches gratefully accepted". A lot of people say this is a > > big issue for them, so it's something that will be fixed one day, but > > nobody has put in a reasonable patch yet. When that happens, we can > > progress.
> What's the expected scope of the downcoding? Would it be throwing a few > dicts together in the admin js, or a callback to unicodedata.normalize?
I thought there was some sort of consensus; I didn't claim all the details had been settled. Personally, I was kind of hoping whoever wrote the patch might think this sort of thing through and give us a concrete target to throw ideas at. :-)
My own misguided thoughts (I *really* don't want to have write this patch): I thought the original design wish was "something that read sensibly" here, since slugifying is already a lossy process. If I had to write it today, I would do the "dictionary mapping on the client side" version. But you're more of an expert here: what does normalization gain us without having to move to fully internationalised URLs, which still seem to be a phishing vector: if we allow fully international URLs, then doing everything properly would make sense. However, is it universally supported as "not a security risk" in all common browsers yet?
> What's the expected scope of the downcoding? Would it be throwing a > few > dicts together in the admin js, or a callback to > unicodedata.normalize?
I’m not sure unicodedata.normalize is enough. It kind of works, if you do something like:
def slugify_utf8_slug(slug): normalized = [] for c in slug.decode('utf-8'): normalized.append(unicodedata.normalize('NFD', c)[0]) return ''.join(normalized)
But this is because “ü” and “é” can be decomposed as “u” and “e” plus accent or diacritic. But then you couldn’t have language-specific decompositions like the “Ä = Ae” mentioned here:
Antonio Cavedoni wrote: > So this would be no good.
> Perhaps I’m missing something but unicodedata won’t cut it.
This is my point. Cut what exactly? "No good" for what exactly? We could file patches to see what sticks, but it might be better to figure what's wanted first, instead of playing fetch me a rock.
A slug function can range from a regex replace to a complete text normalization/decomposition/lookup service that will never be enough because even unicode+mappings aren't a complete solution.
If it's the full unicode+mappings case, I'm doubtful that processing should be done on the client, not only because the unicode database is large, but also because the server will have a well tested setup via unicodedata.
If there's a need to keep the slug current behaviour, fill out as you write as opposed to fill out on the server, that suggests an ajax callback to the server to get at unicodedata.
If a latin1 hack is enough, that can be sent down to the client in the admin js. RT editors like fck do this all the time with entity replacements. No need to use Python if we're dealing with a small subset.
Mappings: yes, ord/text mappings are grand (Greek, Russian, Turkish of the top of my head would be good inbuilts, as would latin if a unicode db isn't used).
If there's a need for mapping extension, there needs to be a place for people to put dictionaries.
If the code falls into an else because it has no lookup, does it insert a stringified hexcode or blank the character out.
> On Fri, 2006-07-07 at 10:06 +0100, Bill de hÓra wrote: >> Malcolm Tredinnick wrote:
>>> There was reasonable consensus in one of the threads about doing >>> something similar (but a bit smaller) than what Wordpress does. Now it's >>> a case of "patches gratefully accepted". A lot of people say this is a >>> big issue for them, so it's something that will be fixed one day, but >>> nobody has put in a reasonable patch yet. When that happens, we can >>> progress. >> What's the expected scope of the downcoding? Would it be throwing a few >> dicts together in the admin js, or a callback to unicodedata.normalize?
> I thought there was some sort of consensus; I didn't claim all the > details had been settled. Personally, I was kind of hoping whoever wrote > the patch might think this sort of thing through and give us a concrete > target to throw ideas at. :-)
> My own misguided thoughts (I *really* don't want to have write this > patch): I thought the original design wish was "something that read > sensibly" here, since slugifying is already a lossy process. If I had to > write it today, I would do the "dictionary mapping on the client side" > version. But you're more of an expert here: what does normalization gain > us without having to move to fully internationalised URLs, which still > seem to be a phishing vector: if we allow fully international URLs, then > doing everything properly would make sense. However, is it universally > supported as "not a security risk" in all common browsers yet?
Normalisation/decomposition gains you greater assuredness you'll throw away what you think you're throwing away, before you try a mapping. Unicode provides mappings down to ascii but it's not complete; mapping decisions tend to be localized/controversial.
The phishing problem with Internationalised URLs (IRIs) is in the internationalized domain name (IDN) where you can get redirected, and not so much the path segment where the slug lives. I work on atom protocol and IRIs are official IETF/W3C goodness these days (funny, we just went through slugging on the protocol list yesterday). IRIs are designed to to be treated as encoded Unicode (utf8 most likely) so they pass through systems without losing information. Slugging as I tend to understand it is really about dropping down to ascii and throwing character information away. I'm thinking that for slugs people want to have a character replaced with an ascii equivalent and not /preserve/ character data via encoding.
It really does depend on what people want from this feature. A full full full downcoding solution needs to go back to the server I think, do the whole unicode bit, and use whatever custom mappings onto ascii. Whereas a good enough approach would be set of js dicts sent to the client; that keeps the nice js autofill feature in the admin, and will probably cover 95% of use cases.
Malcolm Tredinnick wrote: > Personally, I was kind of hoping whoever wrote > the patch might think this sort of thing through and give us a concrete > target to throw ideas at. :-)
var ALL_DOWNCODE_MAPS=new Array() ALL_DOWNCODE_MAPS[0]=LATIN_MAP ALL_DOWNCODE_MAPS[1]=LATIN_SYMBOLS_MAP ALL_DOWNCODE_MAPS[2]=GREEK_MAP ALL_DOWNCODE_MAPS[3]=TURKISH_MAP ALL_DOWNCODE_MAPS[4]=RUSSIAN_MAP
var Downcoder = new Object() ;
Downcoder.Initialize = function() { if (Downcoder.map) // already made return ; Downcoder.map ={} Downcoder.chars = '' ; for(var i in ALL_DOWNCODE_MAPS) { var lookup = ALL_DOWNCODE_MAPS[i] for (var c in lookup) { Downcoder.map[c] = lookup[c] ; Downcoder.chars += c ; } } Downcoder.regex = new RegExp('[' + Downcoder.chars + ']|[^' + Downcoder.chars + ']+','g') ;
}
downcode= function( slug ) { Downcoder.Initialize() ; var downcoded ="" var pieces = str.match(Downcoder.regex); if(pieces) { for (var i = 0 ; i < pieces.length ; i++) { if (pieces[i].length == 1) { var mapped = Downcoder.map[pieces[i]] ; if (mapped != null) { downcoded+=mapped; continue ; } } else { downcoded+=pieces[i]; } } } else { downcoded = slug; } return downcoded;
}
function URLify(s, num_chars) { s = downcode(s); removelist = ["a", "an", "as", "at", "before", "but", "by", "for", "from", "is", "in", "into", "like", "of", "off", "on", "onto", "per", "since", "than", "the", "this", "that", "to", "up", "via", "with"]; r = new RegExp('\\b(' + removelist.join('|') + ')\\b', 'gi'); s = s.replace(r, ''); // if downcode fails, the char will be stripped here s = s.replace(/[^-A-Z0-9\s]/gi, ''); s = s.replace(/^\s+|\s+$/g, ''); // trim leading/trailing spaces s = s.replace(/[-\s]+/g, '-'); // convert spaces to hyphens s = s.toLowerCase(); // convert to lowercase return s.substring(0, num_chars);// trim to first num_chars chars return s.substring(0, num_chars);// trim to first num_chars chars
}
]]]
I need to test this properly and fill in the mappings, but the gist of the approach should be clear. When that's done, unless someone has an objection, I'll file a patch against
On Wed, 2006-07-12 at 02:54 +0100, Bill de hÓra wrote: > Malcolm Tredinnick wrote:
> > Personally, I was kind of hoping whoever wrote > > the patch might think this sort of thing through and give us a concrete > > target to throw ideas at. :-)
> function URLify(s, num_chars) { > s = downcode(s); > removelist = ["a", "an", "as", "at", "before", "but", "by", "for", > "from", > "is", "in", "into", "like", "of", "off", "on", > "onto", "per", > "since", "than", "the", "this", "that", "to", "up", > "via", > "with"]; > r = new RegExp('\\b(' + removelist.join('|') + ')\\b', 'gi'); > s = s.replace(r, ''); > // if downcode fails, the char will be stripped here > s = s.replace(/[^-A-Z0-9\s]/gi, ''); > s = s.replace(/^\s+|\s+$/g, ''); // trim leading/trailing spaces > s = s.replace(/[-\s]+/g, '-'); // convert spaces to hyphens > s = s.toLowerCase(); // convert to lowercase > return s.substring(0, num_chars);// trim to first num_chars chars > return s.substring(0, num_chars);// trim to first num_chars chars > } > ]]]
Probably only one of the last two lines is necessary. :-)
I am the about the worst guy in the world to review Javascript code, but I agree it seems to behave logically. I'm just not familiar enough with the ins and outs enough to spot any sneaky problems that might creep into code like this. Fortunately, the audience is full of people with real clues.
> I need to test this properly and fill in the mappings, but the gist of > the approach should be clear. When that's done, unless someone has an > objection, I'll file a patch against
In the interests of economy I just closed #1602 as a dupe of #2282 (the latter has more links to mailing list threads). It's too hard to track all the dupes and semi-related bugs.
Anyway, thanks for doing the work, Bill. This looks like something that is a great start and that we can tweak as people find problems in their own locales.
> I need to test this properly and fill in the mappings, but the gist of > the approach should be clear. When that's done, unless someone has an > objection, I'll file a patch against
The structure and logic look good; my only worry is that this will eventually get pretty unwieldy; just looking over the languages we already support through the i18n system, we'd be carrying around a pretty huge mapping from the get-go, and it would only grow over time. And God help us if we seriously decide to support slugifying CJK ;)
-- "May the forces of evil become confused on the way to your house." -- George Carlin
Malcolm Tredinnick wrote: >> return s.substring(0, num_chars);// trim to first num_chars chars >> return s.substring(0, num_chars);// trim to first num_chars chars >> } >> ]]]
> Probably only one of the last two lines is necessary. :-)
DRY!
> I am the about the worst guy in the world to review Javascript code, but > I agree it seems to behave logically. I'm just not familiar enough with > the ins and outs enough to spot any sneaky problems that might creep > into code like this. Fortunately, the audience is full of people with > real clues.
Fair enough; it's out there so people can push back on the approach, before I crack on.
> Anyway, thanks for doing the work, Bill. This looks like something that > is a great start and that we can tweak as people find problems in their > own locales.
Ok, I'll start testing this and filling in some mappings. I'll send in a patch as soon as I can.
James Bennett wrote: > On 7/11/06, Bill de hÓra <b...@dehora.net> wrote: >> I need to test this properly and fill in the mappings, but the gist of >> the approach should be clear. When that's done, unless someone has an >> objection, I'll file a patch against
> The structure and logic look good; my only worry is that this will > eventually get pretty unwieldy; just looking over the languages we > already support through the i18n system, we'd be carrying around a > pretty huge mapping from the get-go, and it would only grow over time. > And God help us if we seriously decide to support slugifying CJK ;)
Thanks James. I think if it got to that stage, it's in the class of good problems to have (Django having achieved world domination). I guess we'd need to look at going back to python to run against unicodedata and/or server sided mappings as Antonio pointed out. I think using a Downcoder object gives room to callback to the server to the do the work if needed (the logic would be refactored into python). I'm not sure how people would fold in their locale/custom mappings at that stage, but it might be a nice to have feature.
> > Personally, I was kind of hoping whoever wrote > > the patch might think this sort of thing through and give us a concrete > > target to throw ideas at. :-)
> Hi Malcolm,
> Here we go:
> [snip]
> I need to test this properly and fill in the mappings, but the gist of > the approach should be clear. When that's done, unless someone has an > objection, I'll file a patch against
Thanks for your job, seems good to me but what about the size of the final urlify.js? I'm afraid that it will increase a lot and maybe the current one is good for most of the cases, is it possible to easily switch between the two for english writers?
Another question is about customisation of removelist. What's your opinion about that?
> This is my point. Cut what exactly? "No good" for what exactly? We > could file patches to see what sticks, but it might be better to > figure > what's wanted first, instead of playing fetch me a rock.
This is handled by Unicode standard and is called transliteration. The problem is that it's locale dependent. AFAIK Python's codecs don't implement it (but ICU4R does). If you go for tables it's going to be _many_.
URLs can be Unicode-aware, just encoded - so why not replacing whitespace with dashes and doing a Unicode downcase, and be done with it? Some browsers (Safari) even show you the request string verbatim, so it's very readable.
On 7/12/06, Julian 'Julik' Tarkhanov <julian.tarkha...@gmail.com> wrote:
> This is handled by Unicode standard and is called transliteration.
And also not quite true. Arabic, for example, but also Hindi, have no real standardized transliteration schemes. Also, for Japanese, are you going to follow kunrei-shiki or rather the more widely used hepburn transliteration? Or perhaps even nippon-shiki if you feel like sticking to strictness. Chinese, want to settle for hanyu pinyin or rather wade-giles?
Be careful because you are treading on very unstable ground now.
> On 7/12/06, Julian 'Julik' Tarkhanov <julian.tarkha...@gmail.com> wrote: >> This is handled by Unicode standard and is called transliteration.
> Also, for Japanese, are you going to follow kunrei-shiki or rather the > more widely used hepburn transliteration? Or perhaps even nippon-shiki > if you feel like sticking to strictness.
i think we do not need to discuss japanese at all. after all, there's no transliteration for kanji. so it's imho pointless to argue about kana-transliteration, when you cannot transliterate kanji.
> Jeroen Ruigrok van der Werven wrote: > > On 7/12/06, Julian 'Julik' Tarkhanov <julian.tarkha...@gmail.com> wrote: > >> This is handled by Unicode standard and is called transliteration.
> > Also, for Japanese, are you going to follow kunrei-shiki or rather the > > more widely used hepburn transliteration? Or perhaps even nippon-shiki > > if you feel like sticking to strictness.
> i think we do not need to discuss japanese at all. after all, there's no > transliteration for kanji. so it's imho pointless to argue about > kana-transliteration, when you cannot transliterate kanji.
We Japanese know that we can't transarate Japanese to ASCII. So I want to do it as follows at least. A letter does not disappear and is restored. #FileField and ImageField have same letters disappear problem.
> We Japanese know that we can't transarate Japanese to ASCII. > So I want to do it as follows at least. > A letter does not disappear and is restored. > #FileField and ImageField have same letters disappear problem.
I’m not convinced by this approach, but I would suggest using the “punycode” instead of the “idna” encoder anyway. The results don’t include the initial “xn--” marks which are only useful in a domain name, not in a URI path. Also, the “from encodings […]” line appears to be unnecessary on my Python 2.3.5 and 2.4.1 on OSX.
[[[ >>> p = u"perché" >>> from encodings import idna >>> p.encode('idna') 'xn--perch-fsa' >>> p.encode('punycode') 'perch-fsa' >>> puny = 'perch-fsa' >>> puny.decode('punycode') u'perch\xe9' >>> print puny.decode('punycode') perché >>> pu = puny.decode('punycode') # it's reversible >>> print pu perché ]]]
Antonio Cavedoni wrote: > On 17 Jul 2006, at 8:25, tsuyuki makoto wrote: >> We Japanese know that we can't transarate Japanese to ASCII. >> So I want to do it as follows at least. >> A letter does not disappear and is restored. >> #FileField and ImageField have same letters disappear problem.
> I’m not convinced by this approach, but I would suggest using the > “punycode” instead of the “idna” encoder anyway. The results don’t > include the initial “xn--” marks which are only useful in a domain > name, not in a URI path. Also, the “from encodings […]” line appears > to be unnecessary on my Python 2.3.5 and 2.4.1 on OSX.
> [[[ > >>> p = u"perché" > >>> from encodings import idna > >>> p.encode('idna') > 'xn--perch-fsa' > >>> p.encode('punycode') > 'perch-fsa' > >>> puny = 'perch-fsa' > >>> puny.decode('punycode') > u'perch\xe9' > >>> print puny.decode('punycode') > perché > >>> pu = puny.decode('punycode') # it's reversible > >>> print pu > perché > ]]]
i somehow have the feeling that we lost the original idea here a little.
(as far as i understand, by urlify.js we are talking about slug auto-generation, please correct me if i'm wrong).
we are auto-generating slugs when it "makes sense". for example, for english it makes sense to remove all the non-word stuff, because what remains can still be read, be understood, and generally looks fine when being a part of the URL.
also, for many languages (hungarian or slavic ones), it also "makes sense" to simply drop all the diacritical marks, because the rest can still be read, be understood, and looks fine as part of an URL.
but with punycode or whatever-code encoding japanese, what's the point? what you get will be completely unreadable.. if you only need to preserve the submitted data, you don't need to do anything. simply take your unicode text, encode it to utf8, url-escape it and use it as a part of the url. it will be ok. and on the other side you can url-unescape and utf8-decode it and you're back. you will even be able to have ascii stuff readably-preserved.
form my point of view, with the current slug-approach, you either can convert your text into ascii that "makes sense" or not. if the former, then enhancing urlify.js makes sense. if the latter, then it makes no sense. imho.
Gábor Farkas wrote: > i somehow have the feeling that we lost the original idea here a little.
> (as far as i understand, by urlify.js we are talking about slug > auto-generation, please correct me if i'm wrong).
> we are auto-generating slugs when it "makes sense". for example, for > english it makes sense to remove all the non-word stuff, because what > remains can still be read, be understood, and generally looks fine when > being a part of the URL.
> also, for many languages (hungarian or slavic ones), it also "makes > sense" to simply drop all the diacritical marks, because the rest can > still be read, be understood, and looks fine as part of an URL.
> but with punycode or whatever-code encoding japanese, what's the point? > what you get will be completely unreadable.. if you only need to > preserve the submitted data, you don't need to do anything. simply take > your unicode text, encode it to utf8, url-escape it and use it as a part > of the url. it will be ok. and on the other side you can url-unescape > and utf8-decode it and you're back. you will even be able to have ascii > stuff readably-preserved.
I agree; this has gone *way* past the original idea.
Transcription of characters onto ascii, (aka "slugging") is not the same problem as passing around encoded unicode IRI segments between clients and servers. There's a standards track IETF document for the latter purpose - RFC3987. If you want to do this, do it to spec.
I think the js mapping approach is good enough for the admin interface. Once I can get the greek table to be picked up (argh), a patch will land...
> i think we do not need to discuss japanese at all. after all, there's no > transliteration for kanji. so it's imho pointless to argue about > kana-transliteration, when you cannot transliterate kanji.
If you mean that you cannot easily deduce whether the kanji for moon 月 should be transliterated according to the reading 'tsuki' or 'getsu', then yes, you are correct. But you *can* transliterate them according to their on or kun reading.
|
