JFS default behavior
Nicolas Mailhot
> You write:
> | So what ?
> | Do you think an app that expects utf-8 filenames won't crash today when
> | served a byte sequence that's invalid UTF-8 ? (or an app that expects
> | ascii when served utf-8 oddities)
>
> Such apps are buggy and need to be fixed.
Well, this means every single java app right now at least.
> This is not Unix's problem,
The w2k problem was at the app level mostly.
It would not have been OS responsibility to fix it.
*However* since the unix time conventions were a bit more sane than
other os, the damage was less.
> any more than it is Unix's problem if an application frees memory twice,
> writes over unallocated memory, or destroys its stack.
The core os responsability is to share sanely ressources between apps.
Filenames are a shared ressource.
When encodings starts to be incompatible, resulting in applications
crashes it's the OS job to define and enforce sane conventions so apps
can coexist together.
Past oversights should not mean the problem should not be fixed
(especially if solutions exist, even if they are not totally painless).
There is no more justification to keep encoding undefined as there is to
keep time zone undefined. Last I've seen we're all pretty happy system
time actually means something on unix (unlike other systems where it can
be anything depending on the location where the initial installation was
performed).
> If all you care about is the future, you need no kernel support.
> Declare that all filesystem names are written in UTF-8, and make your
> tools deal with it. (Most will not care. A few will have to be fixed a
> bit.)
Tools won't change unless they're forced to. That's a plain fact.
As you wrote there shouldn't be a lot of fixups to do, since apps that
can't deal with utf-8 now use ascii-only filenames anyway, but the few
fixups that are needed won't happen without a little OS prodding.
(and without OS enforcement illegal utf-8 filename injection will remain
a security risk)
And I write utf8 here, but any unicode form is fine with me as long as
it's clearly defined and enforced by the FSs.
Cheers,
--
Nicolas Mailhot
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
vi...@parcelfarce.linux.theplanet.co.uk
> There is no more justification to keep encoding undefined as there is to
> keep time zone undefined. Last I've seen we're all pretty happy system
> time actually means something on unix (unlike other systems where it can
> be anything depending on the location where the initial installation was
> performed).
"System time" is amount of time elapsed since the epoch. Period. What does
it have to any timezone?
The only place where timezone enters the picture is conversion of time to
year:month:day:hours:minutes:seconds and that's
a) process-dependent and
b) done outside of kernel
The same goes for file names. Filename is a sequence of bytes, no more and
no less. Anything beyond that belongs to applications.
Nicolas Mailhot
> On Sat, Feb 14, 2004 at 03:27:50PM +0100, Nicolas Mailhot wrote:
>
>>There is no more justification to keep encoding undefined as there is to
>>keep time zone undefined. Last I've seen we're all pretty happy system
>>time actually means something on unix (unlike other systems where it can
>>be anything depending on the location where the initial installation was
>>performed).
>
>
> "System time" is amount of time elapsed since the epoch. Period. What does
> it have to any timezone?
And everyone agrees on the epoch and that's why it works.
(just like sensors output is not just any numerical value but has a
well-defined unit)
With filenames we have a value but what it means exactly is a matter of
conjecture. That's the problem.
(it wouldn't be if filenames were just magic cookies that never needed
to be interpreted but there's a lot of actors, be it apps or humans that
need to agree on what the byte string)
Cheers,
--
Nicolas Mailhot
Nicolas Mailhot
> to be interpreted but there's a lot of actors, be it apps or humans that
> need to agree on what the byte string)
... actually means
(bad proofreading, sorry)
Robin Rosenberg
> The same goes for file names. Filename is a sequence of bytes, no more and
> no less. Anything beyond that belongs to applications.
Should be a sequence of characters since humans are supposed to use them and
it should be the same characters wheneve possible regardless of user's locale.
The "sequence of bytes" idea is a legacy from prehistoric times when byte == character
was true. That is no longer the case and actually hasn't been for quite a while in
some parts of the world. Interchange is important. The application cannot handle
this since it cannot know what characters a byte string represents. Fixing it in the
kernel is the simple solution since it knows the locale. Its also a small change I
believe. Having an iocharset options for all file systems make it backward compatible
and creates a migration path to UTF-8 as system default locale.
-- robin
vi...@parcelfarce.linux.theplanet.co.uk
> On Saturday 14 February 2004 16.40, you wrote:
> > The same goes for file names. Filename is a sequence of bytes, no more and
> > no less. Anything beyond that belongs to applications.
>
> Should be a sequence of characters since humans are supposed to use them and
> it should be the same characters wheneve possible regardless of user's locale.
> The "sequence of bytes" idea is a legacy from prehistoric times when byte == character
> was true.
Bullshit. It has _nothing_ to characters, wide or not. For system filenames
are opaque. The only things that have special meanings are:
octet 0x2f ('/') splits the pathname into components
"." as a component has a special meaning
".." as a component has a special meaning.
That's it. The rest is never interpreted by the kernel.
> Having an iocharset options for all file systems make it backward compatible
> and creates a migration path to UTF-8 as system default locale.
Try to realize that different users CAN HAVE DIFFERENT LOCALES. On the same
system. And have files on the same fs. Moreover, homedirs that used to be
on different filesystems can end up one the same fs. What iocharset would
you use, then? Sigh...
Again, there is no such thing as iocharset of filesystem - it varies between
users and users can and do share filesystems. Think of /home; think of /tmp.
It isn't feasible. At all. Just as timezone doesn't belong in kernel, locales
have no place there.
Robin Rosenberg
> On Sun, Feb 15, 2004 at 12:06:23AM +0100, Robin Rosenberg wrote:
> > The "sequence of bytes" idea is a legacy from prehistoric times when byte == character
> > was true.
>
> Bullshit. It has _nothing_ to characters, wide or not. For system filenames
> are opaque. The only things that have special meanings are:
> octet 0x2f ('/') splits the pathname into components
> "." as a component has a special meaning
> ".." as a component has a special meaning.
> That's it. The rest is never interpreted by the kernel.
and sees a string of characters, not a byte sequence.
> Try to realize that different users CAN HAVE DIFFERENT LOCALES. On the same
> system. And have files on the same fs. Moreover, homedirs that used to be
> on different filesystems can end up one the same fs. What iocharset would
> you use, then? Sigh...
Ok, I've got the iocharset option wrong, god knows why. The problem
however remains.
It seems you simply don't want to understand the problem, which is that users
CAN HAVE DIFFERENT LOCALES on the same system and on different system.
Sigh...
I less concerned with which solution than that a solution should be found. So it
seems no file system has a solution today. Still an iocharset option would relieve
the problem for removable media and muli-boot systems. Most linux machines
are essentially single user and have either the same locale for all users or all
users are using UTF-8 with their locale. It's not the locale, but the charset used
for encoding the locale. The rest cannot be helped.
-- robin
Linus Torvalds
On Sun, 15 Feb 2004, Robin Rosenberg wrote:
> >
> > Bullshit. It has _nothing_ to characters, wide or not. For system filenames
> > are opaque. The only things that have special meanings are:
> > octet 0x2f ('/') splits the pathname into components
> > "." as a component has a special meaning
> > ".." as a component has a special meaning.
> > That's it. The rest is never interpreted by the kernel.
>
> I know how it is (to some degree), and its wrong. The user sees inside the filename
> and sees a string of characters, not a byte sequence.
Yes, the user sees a string of characters, but the octet 0x2f ('/') and
the terminating NUL character '\0' are still perfectly normal characters
and there is no confusion.
The reason: UTF-8. It's the only sane encoding (apart from a pure extended
ASCII setup, which is also sane, but is obviously unacceptable for a large
portion of the world).
If some misguided person has told you about UCS-2 and horrors like UTF-9,
just ignore them. They are crazy and deluded, and - perhaps more
importantly - stupid.
In short: the kernel talks bytestreams, and that implies that if you want
to talk to the kernel, you HAVE TO USE UTF-8.
At which point there are no locale issues any more. The only locale issue
you can have is user space mistaking a stream of bytes as extended ASCII,
which will cause all your pretty UTF-8 characters to be shown as strange
latin1 (or other) squiggles.
> It seems you simply don't want to understand the problem, which is that users
> CAN HAVE DIFFERENT LOCALES on the same system and on different system.
> Sigh...
People understand the problem. And UTF-8 is the solution.
It's getting there. I think even Microsoft has seen the light, and is
phasing out their crapola (UCS-2LE? Whatever).
> I less concerned with which solution than that a solution should be found. So it
> seems no file system has a solution today. Still an iocharset option would relieve
> the problem for removable media and muli-boot systems.
No. Things like "iocharset" are not the solution. They are literally the
_problem_. The solution is to use something that not only acts as ASCII,
but also has a wide enough range to cover the whole required space (UCS-2
fails _both_ of these fundamental tests). At which point "iocharset" makes
no sense any more, and only exists as a way to translate legacy crap into
the one true format.
And that one true format is UTF-8. End of story. If you try to talk to the
kernel in UCS-2 or anything else, you _will_ fail.
Linus
Matthias Urlichs
> In short: the kernel talks bytestreams, and that implies that if you =
want
> to talk to the kernel, you HAVE TO USE UTF-8.
>
> At which point there are no locale issues any more.
Not locale, but normalization problems and identical-glyph problems.
Which is actually worse, because you don't have filenames which look
like crap -- instead you have filenames which look perfectly sane, but
they still do not work. Example: is an á one character, or is it a=
n a
followed by a composing ´?
Mac OSX, just as an example, only uses decomposed filenames. I don't kn=
ow
the current situation, but 10.2 has major problems when you try to acce=
ss
files with composite characters in their name (across NFS for instance)=
.
I wonder if Linux, i.e. Linus ;-) should decree one single standard
normalization. (I am NOT saying that enforcing this would be the kernel=
's
job!)
--
Matthias Urlichs
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
vi...@parcelfarce.linux.theplanet.co.uk
> Mac OSX, just as an example, only uses decomposed filenames.
So how long does it take for a filename to decompose? ;-)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Robin Rosenberg
> On Sun, Feb 15, 2004 at 04:33:48AM +0100, Matthias Urlichs wrote:
>
> > Mac OSX, just as an example, only uses decomposed filenames.
>
> So how long does it take for a filename to decompose?
As long as it takes to switch locale to UTF-8 :) or vice verse.
-- robin
Pascal Schmidt
>> Then I did unicode_stop. Guess what: it put the display back in
>> iso-8859-1 for that virtual terminal, but the keyboard remained
>> stuck in UTF-8 for _all_ virtual terminals.
> kbd_mode -a to reset to ASCII mode.
And as I just figured out, loadkeys has to be invoked again, also.
I can go to utf-8 with:
setfont lat0-16
kbd_mode -u
loadkeys de-latin1-nodeadkeys
and return to latin-1 with:
setfont lat1-16
kbd_mode -a
loadkeys de-latin1-nodeadkeys
Without the loadkeys after returning to latin-1 mode, I can no longer
input umlauts and other special characters correctly.
--
Ciao,
Pascal
yoda...@fsmlabs.com
> On Sun, Feb 15, 2004 at 04:33:48AM +0100, Matthias Urlichs wrote:
>
> > Mac OSX, just as an example, only uses decomposed filenames.
>
> So how long does it take for a filename to decompose? ;-)
Depends on whether it is junk or not.
Nicolas Mailhot
| In short: the kernel talks bytestreams, and that implies that if you
| want to talk to the kernel, you HAVE TO USE UTF-8.
In that case :
- should the kernel allow apps to write filenames that are invalid
UTF-8 and will crash UTF-8 apps ?
- should this UTF-8 rule be noted somewhere (in a FAQ/man page/LSB spec/
whatever) so apps authors know they are supposed to read and write UTF-8
filenames and not apply locale rules to kernel objects ?
- what happens to already existing invalid UTF-8 filenames ? Should the
kernel forcibly rewrite them (in 2.7.0...) to remove legacy mess ? What
should happen if someone plug an unconverted FS in such a system
afterwards ?
These are the questions people have been asking.
jw schultz
> | Linus Torvalds pointed the way of Tux :
>
> | In short: the kernel talks bytestreams, and that implies that if you
> | want to talk to the kernel, you HAVE TO USE UTF-8.
>
> In that case :
> - should the kernel allow apps to write filenames that are invalid
> UTF-8 and will crash UTF-8 apps ?
Yes. The kernel interface specifies it as a bytesteam with
0x00 and 0x2f having special meaning. That is a constraint,
not a policy. It is user space that determines the policy
of UTF-8.
> UTF-8 and will crash UTF-8 apps ?
Fix the broken apps. Crashing because of "invalid" UTF-8 is
no more excusable than crashing because of a string longer
than expected (buffer overrun). Filenames as read from the
filesystem should be treated just like any other untrusted
input.
> - should this UTF-8 rule be noted somewhere (in a FAQ/man page/LSB spec/
> whatever) so apps authors know they are supposed to read and write UTF-8
> filenames and not apply locale rules to kernel objects ?
Since the LSB spec describes user space it might be a
suitable place.
> - what happens to already existing invalid UTF-8 filenames ? Should the
> kernel forcibly rewrite them (in 2.7.0...) to remove legacy mess ? What
If you have a filesystem with filenames that don't conform
to your policy write userspace tools to detect and/or fix
them. If you have programs creating non-conforming
filenames, fix or rm those programs.
> kernel forcibly rewrite them (in 2.7.0...) to remove legacy mess ? What
> should happen if someone plug an unconverted FS in such a system
> afterwards ?
The kernel won't care. Any user space code that treats the
filenames as something other than bytestreams should be able
to cope with any sequence of bytes.
> These are the questions people have been asking.
OK. The questions have been asked and answered.
Asking again and again and again won't change the answer.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: j...@pegasys.ws
Remember Cernan and Schmitt
Jan Knutar
> the kernel forcibly rewrite them (in 2.7.0...) to remove legacy mess
> ? What should happen if someone plug an unconverted FS in such a
> system afterwards ?
What I would like would be a userspace tool, that would recurse and
convert filename encodings from specified locale to UTF-8. Something
like "any2utf8 -from iso8859-1 -recurse /mnt/myoldmp3disk".
Does anyone know if such a tool exists already?
Nicolas Mailhot
> > - what happens to already existing invalid UTF-8 filenames ? Should
> > the kernel forcibly rewrite them (in 2.7.0...) to remove legacy mess
> > ? What should happen if someone plug an unconverted FS in such a
> > system afterwards ?
>
> What I would like would be a userspace tool, that would recurse and
> convert filename encodings from specified locale to UTF-8. Something
> like "any2utf8 -from iso8859-1 -recurse /mnt/myoldmp3disk".
> Does anyone know if such a tool exists already?
One can do find+ recode magic now
The question is :
- can this be automated ?
- how can one recognise and unconverted fs ?
- how can on guess the encodings(s) that have been used before on such
an fs ?
You're assuming the situation is merely a iso8859-1 to utf-8 migration.
Far from it. The core problem is everyone damn wrote what it pleased him
without considering future readers.
Cheers,
--
Nicolas Mailhot
Valdis.K...@vt.edu
> You're assuming the situation is merely a iso8859-1 to utf-8 migration.
> Far from it. The core problem is everyone damn wrote what it pleased him
> without considering future readers.
Given the fact that there isn't in general any way for the kernel to know what
was intended, I don't see how any kernel policy other than "NUL and / are
special, but if you use anything other than UTF-8 it will eventually come back
to haunt you" can possibly be made to work.
For that matter, I have seen actual production code that intentionally created
fairly deep directory trees and terminal file names that were basically hashes
written in radix-254 and blatted out in binary. Lots of them. The original
problem report I got was along the lines of "We installed XYZ, and the file
system appears corrupted - ls -R weird the screen out, and 'find | wc -l' is
127,000 different than what 'df -i' reports".
I was ready to strangle the guilty party - radix-64 wouldn't have been a big
efficiency hit and at least the uuencode/base-64 charset doesn't weird your
terminal out. :)
So it's not even always possible to make the assumption that the filename is
supposed to make sense in *any* charset. This one requires fixing in some
combination of userspace and meatspace....
Eduard Bloch
Pascal Schmidt schrieb am Sunday, den 15. February 2004:
> >> iso-8859-1 for that virtual terminal, but the keyboard remained
> >> stuck in UTF-8 for _all_ virtual terminals.
> > kbd_mode -a to reset to ASCII mode.
>
> And as I just figured out, loadkeys has to be invoked again, also.
>
> I can go to utf-8 with:
>
> setfont lat0-16
> kbd_mode -u
> loadkeys de-latin1-nodeadkeys
When I do this, I still cannot enter unicode chars "as usual". I see
them, mutt (for example) displays everything correct with a UTF-8
locale. However, I cannot insert them correctly. When I use vim, I have
to press another key (eg. Space) 2..4 times after an umlaut was pressed,
only then the char appears.
Needless to say that the same applications work fine in X with the same
UTF-8 locale.
Regards,
Eduard.
--
Lob ist eine gewaltige Antriebskraft, dessen Zauber seine Wirkung nie
verfehlt.
-- Andor Foldes
Jamie Lokier
> People understand the problem. And UTF-8 is the solution.
Linus, I agree 100%.
My own filesystems have UTF-8 file names, of course.
There are still practical problems, two of which stand out:
1. Just because you hope a filesystem is UTF-8, does not preclude
readdir() from returning non-UTF-8 names. (These are far too
easy to create by accident).
Because of that, programs which interpret the result of
readdir() as text, yet are expected to handle any name without
silently rejecting them or aborting, are forced into strange
compromises which break basic expectations.
Spot the bug in this perl script:
perl -e 'for (glob "*") { rename $_, "ņi-".$_ or die "rename:=
$!\n"; }'
(NB: The prefix string is N WITH CEDILLA followed by "i-").
(Hint: it mangles perfectly fine non-ASCII file names).
Perl has no perfect behaviour to offer, because what should that
behaviour be if readdir() might return a non-UTF-8 byte sequence
as a name?
2. Terminals are not all UTF-8, and some never will be.
So when someone types something like this on a non-UTF-8
terminal, they get non-UTF-8 filename:
vi el-niño.txt
It isn't just a problem of display. Now you have created a
filename which isn't valid UTF-8, and GUI programs may complain,
perhaps refusing to let you select the file.
Furthermore, how exactly do you expect a user to use UTF-8 on
the filesystem when their terminal is not (or sometimes is not)
using UTF-8?
==> This problem would be very nicely solved with an additional
terminal flag. We have "stty ocrnl", "onlcr", "igncr" etc. to
translate between terminal line endings and the unix convention of
LF at the end of each line. Why not create "stty utf8" so that
non-UTF-8 terminals and UTF-8 terminals alike can work with a
Linux convention that all programs enter and display UTF-8? It
would simplify a lot of things.
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
Pascal Schmidt
> When I do this, I still cannot enter unicode chars "as usual". I see
> them, mutt (for example) displays everything correct with a UTF-8
> locale. However, I cannot insert them correctly. When I use vim, I have
> to press another key (eg. Space) 2..4 times after an umlaut was pressed,
> only then the char appears.
You're right, inputing UTF-8 (in joe) doesn't work, but that's an
application problem, I think, because it works just fine on a shell
prompt.
--
Ciao,
Pascal
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> If you have a filesystem with filenames that don't conform
> to your policy write userspace tools to detect and/or fix
> them. If you have programs creating non-conforming
> filenames, fix or rm those programs.
You do understand that GNU coreutils, bash etc. are among those
programs, right? As in "touch zöe.txt" creates a non-conforming
filename...
> OK. The questions have been asked and answered.
> Asking again and again and again won't change the answer.
The question of what a program like this should do has not been
answered:
perl -e 'for (glob "*") { rename $_, "ņi-".$_ or die "rename: $=
!\n"; }'
(NB: The prefix string is N WITH CEDILLA followed by "i-").
Hint: it mangles perfectly fine non-ASCII file names, instead of just
prefixing the prefix string. If you change the program to correctly
prepend the prefix string, then it mangles non-UTF-8 names, which is
arguably correct, but can result in you losing some files.
This _is_ a userspace problem, but it is a genuine problem for which
no good answer is yet apparent.
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
Gerd Knorr
> 2. Terminals are not all UTF-8, and some never will be.
> ==> This problem would be very nicely solved with an additional
> terminal flag. We have "stty ocrnl", "onlcr", "igncr" etc. to
> translate between terminal line endings and the unix convention o=
f
> LF at the end of each line. Why not create "stty utf8" so that
> non-UTF-8 terminals and UTF-8 terminals alike can work with a
> Linux convention that all programs enter and display UTF-8? It
> would simplify a lot of things.
It's probably possible to extend luit doing that too. luit comes with
recent xfree86 releases and does utf-8 <=> locale conversion. Right
now it does just the opposite: let people use non-utf8 locales in a
utf-8 xterm.
Gerd
--
Es geht darum, daß ein Haufen Scriptkiddies gerade dabei sind, USE=
NET in
Bunt neu zu erfinden, und sie derzeit einen Haufen Fehler neu machen,
die schon seit 20 Jahren nicht mehr Gegenstand der Forschung sind.
-- Kristian Köhntopp über blogs und blogger
Marc Lehmann
been mentioned]
On Sat, Feb 14, 2004 at 06:41:20PM -0800, Linus Torvalds <torv...@osdl.org> wrote:
[discussion on why UTF-8 is the only sane encoding, which I absolutely
agree with, removed]
> In short: the kernel talks bytestreams, and that implies that if you want
> to talk to the kernel, you HAVE TO USE UTF-8.
This is not the problem at all. It's perfectly easy to write
applications that talk UTF-8 and just UTF-8 with the kernel.
The problem is that the kernel does not use UTF-8, i.e. applications in
the current linux model have to deal with the fact that the kernel
happily breaks the assumed protocol of using UTF-8 by delivering illegal
byte sequences to applications.
There is no way for applications to handle UTF-8 and illegal-utf8 in
a sane way, so most apps will either eat the illegal bytes, skip the
filename, or crash (the latter case is clearly a bug in the app, thr
former cases aren't).
Fixing the VFS to actually enforce what linus claims (2filenames are
utf-8") is a very good idea, imho.
As I understand it, the reason linux currently doesn't, is that this utf-8
rule was obviously non-enforcable in practise in recent years, since
UTF-8 simply wasn't widespread (even today, applications such as bash or
grep are clearly not UTF-8 ready, as they start to crawl in UTF-8 locales
without special patches, and even with special patches).
So the only sane way to implement this enforcement is usign an
additional moutn-flag, e.g. "force-utf8".
An encoding=xyz mount flag OTOH would be total overkill, as the plan
must be to switch to UTF-8 in the long run, while allowing deviating
behaviour in the short run.
Conversely, filesystems such as NTFS, VFAT etc. need to convert from the
fs encoding to UTF-8 and vice versa automatically, at least when this
flag is specified.
It should become the default in some future linux version.
> People understand the problem. And UTF-8 is the solution.
The kernel needs to fully implement it. Just as a kernel accepting:
open ("directory", O_WRONLY); write (dirfd, ...)...
open ("/some/file", ...)
mkdir ("../some/file", ...)
is considered rather broken behaviour from unix kernels (although these
might have been allowed in some dialects or versions of unix) today, this:
mkdir ("</ encoded using illegal multibyte sequence>", ...)
will be considered broken behaviour in the future. The RFC defining UTF-8
clearly considers this a bug in UTF-8 implementations, the the kernel
in fact does NOT implement UTF-8 right now, although some people claim
that the kernel accepting UTF-8 (and more) is correct behaviour, it isn't
according to the RFC.
> It's getting there. I think even Microsoft has seen the light, and is
> phasing out their crapola (UCS-2LE? Whatever).
Microsoft and Java officially use UTF-16 nowadays. The funny thing is
that "next character" iterators in both languages skip to the next word
in UCS-2, so the claim of both parties of UTF-16 support is basically a
marketing lie.
> No. Things like "iocharset" are not the solution. They are literally the
> _problem_. The solution is to use something that not only acts as ASCII,
[full agreement]
> And that one true format is UTF-8. End of story. If you try to talk to the
> kernel in UCS-2 or anything else, you _will_ fail.
Just that the kernel does not support UTF-8. It delivers and accepts
non-UTF-8 strings such as \xc0\x80. The kernel clearly should not deliver
broken characters when the official stanza is that the linux VFS API is
UTF-8 only (see 3.2, Chapater 3, C12, conformance, ony why it currently
isn't UTF-8).
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
Linus Torvalds
On Mon, 16 Feb 2004, Marc Lehmann wrote:
>
> > In short: the kernel talks bytestreams, and that implies that if you want
> > to talk to the kernel, you HAVE TO USE UTF-8.
>
> This is not the problem at all. It's perfectly easy to write
> applications that talk UTF-8 and just UTF-8 with the kernel.
>
> The problem is that the kernel does not use UTF-8, i.e. applications in
> the current linux model have to deal with the fact that the kernel
> happily breaks the assumed protocol of using UTF-8 by delivering illegal
> byte sequences to applications.
You didn't read what I said.
READ MY POSTING. You even quoted it, but you didn't understand it.
I'm saying that "the kernel talks bytestreams".
I have never claimed that the kernel really talk s UTF-8, and indeed, I
would say that such a kernel would be terminally and horribly broken.
The kernel is _agnostic_ in what it does. As it should be. It doesn't
really care AT ALL what you feed it, as long as it is a byte-stream.
Now, that implies that if you want to have extended characters, then YOU
HAVE TO USE UTF-8.
That's what I'm saying. I am _not_ saying that the kernel uses UTF-8. The
kernel doesn't care one way or the other. As far as the kernel is
concerened, you could uuencode all the stuff, and the kernel wouldn't
think you're crazy. The kernel _only_ cares about byte streams. And that
is as it should be.
> There is no way for applications to handle UTF-8 and illegal-utf8 in
> a sane way, so most apps will either eat the illegal bytes, skip the
> filename, or crash (the latter case is clearly a bug in the app, thr
> former cases aren't).
What you're complaining about are bad user applications. It has _zero_ to
do with the kernel.
> Fixing the VFS to actually enforce what linus claims (2filenames are
> utf-8") is a very good idea, imho.
No. Read my claim again. You obviously do not understand it AT ALL.
What you suggest would be a horribly idiotic and bad idea. The kernel
doesn't set policy. The kernel says "this is what I can do, you set
policy".
And UTF-8 just happens to be the only sane policy for encoding complex
characters into a byte stream. But it is not the only policy.
Another sane policy is to say "byte streams are latin1". It's not an
acceptable policy for encoding _complex_ characters, but it is a policy.
And it's a perfectly sane one.
In short: filenames are byte streams. Nothing more. They don't even have a
"character set". They literally are just a series of bytes.
And when I say that you have to talk to the kernel using UTF-8, I'm only
claiming that it is the only sane way to encode extended characters in a
byte stream. Nothing more.
Linus
Eduard Bloch
Pascal Schmidt schrieb am Monday, den 16. February 2004:
> > When I do this, I still cannot enter unicode chars "as usual". I see
> > them, mutt (for example) displays everything correct with a UTF-8
> > locale. However, I cannot insert them correctly. When I use vim, I have
> > to press another key (eg. Space) 2..4 times after an umlaut was pressed,
> > only then the char appears.
>
> You're right, inputing UTF-8 (in joe) doesn't work, but that's an
> application problem, I think, because it works just fine on a shell
> prompt.
No, does not work for me either (up-to-date bash). General multibyte
support in joe is a different problem.
Regards,
Eduard.
--
Wie man sein Kind nicht nennen sollte:
Mario Hanna
Jeff Garzik
> In short: filenames are byte streams. Nothing more. They don't even have a
> "character set". They literally are just a series of bytes.
>
> And when I say that you have to talk to the kernel using UTF-8, I'm only
> claiming that it is the only sane way to encode extended characters in a
> byte stream. Nothing more.
Nod. Maybe it helps Marc to point out the key difference between
characters and bytes, in UTF8.
In UTF8, the number of characters in a string is less-than-or-equal-to
the number of bytes in the string.
And the kernel just cares about bytes.
This is the whole benefit to UTF8, right here in this thread. UTF8 was
designed such that ten-year-old C code using standard C strings would
function just fine. No need to rip up large swaths of your code just to
call multi-byte versions of the standard string functions. Most code
that doesn't deal with locale-specific details like uppercase/lowercase
Just Works(tm).
Jeff
John Bradford
> Linus Torvalds wrote:
> > In short: filenames are byte streams. Nothing more. They don't even have a
> > "character set". They literally are just a series of bytes.
> >
> > And when I say that you have to talk to the kernel using UTF-8, I'm only
> > claiming that it is the only sane way to encode extended characters in a
> > byte stream. Nothing more.
>
>
> Nod. Maybe it helps Marc to point out the key difference between
> characters and bytes, in UTF8.
>
> In UTF8, the number of characters in a string is less-than-or-equal-to
> the number of bytes in the string.
>
> And the kernel just cares about bytes.
>
> This is the whole benefit to UTF8, right here in this thread. UTF8 was
> designed such that ten-year-old C code using standard C strings would
> function just fine. No need to rip up large swaths of your code just to
> call multi-byte versions of the standard string functions. Most code
> that doesn't deal with locale-specific details like uppercase/lowercase
> Just Works(tm).
The real problem is with mis-configured userspaces, where buggy UTF-8
decoders are trying to make sense of data in legacy encodings
containing essentially random bytes > 127, which are not part of valid
UTF-8 sequences.
None of this is a real problem, if everything is set up correctly and
bug free. Unfortunately the Just Works thing falls apart in the,
(frequent), instances that it's not :-(.
John.
Linus Torvalds
On Mon, 16 Feb 2004, John Bradford wrote:
>
> The real problem is with mis-configured userspaces, where buggy UTF-8
> decoders are trying to make sense of data in legacy encodings
> containing essentially random bytes > 127, which are not part of valid
> UTF-8 sequences.
>
> None of this is a real problem, if everything is set up correctly and
> bug free. Unfortunately the Just Works thing falls apart in the,
> (frequent), instances that it's not :-(.
The way to handle that is to aim to never _ever_ decode utf-8 unless you
really have to. Always leave the string in utf-8 "raw bytestring" mode as
long as possible, and convert to charater sets only when actually
printing.
If you do that, then at worst you'll show the user a strange name (extra
points for marking it as being errenous), but everything still works. You
can still lookup/delete/whatever the file (internally the program still
works on the raw byte sequence and isn't confused). Basically accept the
fact that UTF-8 strings can contain "garbage", and don't try to fix it up.
And no, I'm not claiming that it's wonderfully clean and that we should
all love it. But it's _practical_, and the ugliness is certainly a lot
less than in the alternatives.
And it largely works today.
Linus
Marc Lehmann
> > The problem is that the kernel does not use UTF-8, i.e. applications in
> > the current linux model have to deal with the fact that the kernel
> > happily breaks the assumed protocol of using UTF-8 by delivering illegal
> > byte sequences to applications.
>
> You didn't read what I said.
I read it.
> READ MY POSTING. You even quoted it, but you didn't understand it.
You were able to explain it clearly enough for me, I think.
> I'm saying that "the kernel talks bytestreams".
And I am saying that this is not good, which is my sole point.
> I have never claimed that the kernel really talk s UTF-8, and indeed, I
> would say that such a kernel would be terminally and horribly broken.
And I'd say such a kernel would be highly useful, as it would standardize
the encoding of filenames, just as unix standardizes on "mostly ascii"
(i.e. the SuS).
However, just as POSIX is a nice but very limited base, (mostly) ASCII is a nice
and very limited base. UTF-8 would also be a good base.
8-bit bytes as filenames is not a good base, however, since they enforce
a difefrent layer of interrpetation between the user and the kernel, and
this interpretation cannot be based on the locale nor the filesystem
itself, as there is no way to find out what encoding the filename is in.
8-bit bytes is convinient, but not useful for i18n environments. in the
past, it was also convinient and nobody cared, since everything was
either 8-bit or double-byte, and nobody exchanged files.
This, however, is going to change, and the current methodology of "just
guess, you might be right" is a hindrance to this.
> The kernel is _agnostic_ in what it does.
No, it's not. If at all, the kernel specifies a specially-interpreted
(ascii sans / and \0) byte-stream, as you say yourself.
However, just as with URLs (which are byte-streams, too), byte-streams are
useless to store text. You need bytestreams + known encoding.
If filenames were not names, but just binary id's I would agree, but
this is not at all how filenames are used not how their use is applied.
Filenames are composed of text, but the kernel gives no indication
on how to interpret this text, and as a matter of fact, nothing else
gives this indication. glib etc. uses G_BROKEN_FILENAMES to force
locale-encoding. But as others have said, one mans locale is unlike other
mens locale.
> really care AT ALL what you feed it, as long as it is a byte-stream.
>
> Now, that implies that if you want to have extended characters, then YOU
> HAVE TO USE UTF-8.
You say so, but there is no logical connection between these two
statements. I can store latin1 easily in a bytestream, as I can store
iso-2022-jp or euc-jp. But they are incompatible to UTF-8.
You are yelling at me for no good reason. "YOU HAVE TO USE UTF-8". Why
should this be? The kernel certainly enforces this. Even you claim that
I don't have to, as the kernel doesn't care.
However, if you think so violently that it has to be UTF-8 that you even
yell it, then why doesn't the kernel comply to this rule? Why should an
applicaiton "HAVE TO" use utf-8 for input when the kernel doesn't even
try to comply and hands out illegal output?
This is just like mmap sometimes returning a page number and sometimes a
byte address... this would also not be useful unless you know the unit
that mmap returns (addreesses in multiples of 1).
> That's what I'm saying. I am _not_ saying that the kernel uses UTF-8.
But you are saying that you have to feed UTF-8 into the kernel, which is
not the case either. I certainly don't have to..., and, what's worse,
you haven't given any indication of why one has to. Just because you say
so? Or is there actually a reason? If there is a reason, why doesn't the
kernel, in return, also follow this reasoning?
> kernel doesn't care one way or the other. As far as the kernel is
It doesn't. But the point is that it should. If the kernel would do
everything we want it to do there would be no point in enhancing it.
> concerened, you could uuencode all the stuff, and the kernel wouldn't
(Yes, because uuencode has the peculiar property of neither generting \0
nor /. It doesn't work in general with byte-streams).
> think you're crazy. The kernel _only_ cares about byte streams.
The kernel _interprets_ the byte-stream already. And some byte-streams are
no valid filenames _already_.
> > There is no way for applications to handle UTF-8 and illegal-utf8 in
> > a sane way, so most apps will either eat the illegal bytes, skip the
> > filename, or crash (the latter case is clearly a bug in the app, thr
> > former cases aren't).
>
> What you're complaining about are bad user applications. It has _zero_ to
> do with the kernel.
Could you elaborate on why these apps are bad? What I am interested in
is to know how to fix them? since there is simply no way to interpret
the names returned by the kernel as the corresponding meta-information
is missing.
Consider an OS that allows different characters for path-seperators (unix
only allows '/'). Without the knowledge of the path seperator it would be
impossible to interpret paths. And without knowledge about encoding it's
impossible (but slightly less dramatic) to correctly interpret filenames.
> > Fixing the VFS to actually enforce what linus claims (2filenames are
> > utf-8") is a very good idea, imho.
>
> No. Read my claim again. You obviously do not understand it AT ALL.
...
> What you suggest would be a horribly idiotic and bad idea.
Why?
> The kernel doesn't set policy. The kernel says "this is what I can do,
> you set policy".
Exactly. The kernel could specify the API to use UTF-8. This is not more
policy than it currently enforces.
Or do you suggest that the ability to change the source and replace all
occurences of '/' by '\\' means that '/' is not enforced as policy on
path seperators?
We basically seem to disagree on what, exactly, policy is. Policy (to
me) is something that differentiates between several incompatible
alternatives. Chosing policy means to rule out other (useful)
alternatives.
One could argue that '/' is a policy because it precludes the '/'
character from being used in filenames, sth. some filenames or operating
systems support.
I'd say (probably as much as you) that this policy is not a real policy,
it's just idiotic.
But enforcing other restrictions on filenames should magically be real
policy? This is obviously bot idiotic at all, and should be carefully
explored.
> And UTF-8 just happens to be the only sane policy for encoding complex
> characters into a byte stream. But it is not the only policy.
Just as '/' is not the only possible path seperator. If that is your
point, you should explain why enforcing this is ok while supporting utf-8
(not enforcing, just supporting, meaning having the ability to rule out
non-utf-8 sequences when the admin wants this) is not.
> Another sane policy is to say "byte streams are latin1". It's not an
> acceptable policy for encoding _complex_ characters, but it is a policy.
> And it's a perfectly sane one.
I agree that it is sane. But it is not very useful for the future, as
people who want russian filenames are plainly unable to use the other
filenames in a sensible way. There is no way to know the encoding.
> In short: filenames are byte streams. Nothing more.
Right now, they aren't. Not all sequences of bytes are valid filenames
already, and I think this is perfectly o.k.
> And when I say that you have to talk to the kernel using UTF-8, I'm only
> claiming that it is the only sane way to encode extended characters in a
> byte stream. Nothing more.
And i fully agree.
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
Marc Lehmann
> Quote from Jeff Garzik <jga...@pobox.com>:
> bug free. Unfortunately the Just Works thing falls apart in the,
> (frequent), instances that it's not :-(.
BTW, to people trying to explain some properties of UTF-8 to me. I don't
think ad-hominem attacks like assuming that I don't understand UTF-8
(without any indication that this is so) are useful.
The point here is that the kernel does, in a very narrow interpretation,
not support the use of UTF-8, because proper support of UTF-8 means that
no illegal byte sequences will be produced.
Of course, I can feed the kernel UTF-8, and if everybody does that, it
will generally work quite fine. However, Windows surely works fine if
every program only feeds allowed values into system calls. And even unix
dialects without memory protection work, as long as everybody plays
fair.
The point is, however, that this is highly undesirable, and it would be
nice to have a kernel that would (optionally) fully support a UTF-8
environment in where applications can feed UTF-8 and _expect_ UTF-8 in
return, which _is_ a security issue.
It's very desirable to have a kernel that actively supports this. IT is
clearly not _required_, of course. But then again, process abstraction
is also not required...
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
Marc Lehmann
> works on the raw byte sequence and isn't confused). Basically accept the
> fact that UTF-8 strings can contain "garbage", and don't try to fix it up.
But you are wrong, UTF-8 strings never contain garbage. UTF-8 is
well-defined and is always proper UTF-8. It's a tautology.
The evry idea of "UTF-8 with garbage in it" doesn't make sense.
> And no, I'm not claiming that it's wonderfully clean and that we should
> all love it.
It's also a totally useless idiom...
> And it largely works today.
> Linus
On ascii-only-systems, it works fine. My system is largely ascii-only,
with only very few filenames (japanese and german ones mostly) in
UTF-8. Sometimes in EUC-JP, but that's a bug in rar.
It also works fine in single-user environments where the user just forces
everything to be in her locale. It does fail miserably on multi-user
systems. It does fail miserably in ISO-C's locale model. It does fail
miserably with gnu shellutils, fileutils and most other apps.
It fails, because it's not at all well supported by the kernel.
Claiming that it largely works today is simply not true for most
non-ascii-users (which increasingly includes the US).
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
Jeff Garzik
> The point here is that the kernel does, in a very narrow interpretation,
> not support the use of UTF-8, because proper support of UTF-8 means that
> no illegal byte sequences will be produced.
Incorrect. Byte stream transports need not care about their contents.
The only places that need to care about illegal UTF8 byte sequences are
things like CONFIG_NLS_UTF8.
Jeff
bert hubert
> The way to handle that is to aim to never _ever_ decode utf-8 unless you
> really have to. Always leave the string in utf-8 "raw bytestring" mode as
> long as possible, and convert to charater sets only when actually
> printing.
Additional good news is that following octets in a utf-8 character sequence
always have the highest order bit set, precluding / or \x0 from appearing,
confusing the kernel.
The remaining zit is that all these represent '..':
2E 2E
C0 AE C0 AE
E0 80 AE E0 80 AE
F0 80 80 AE F0 80 80 AE
F8 80 80 80 AE F8 80 80 80 AE
FC 80 80 80 80 AE FC 80 80 80 80 AE
This in itself is not a problem, the kernel will only recognize 2E 2E as the
real .., but it does show that 'document.doc' might be encoded in a myriad
ways.
So some guidance about using only the simplest possible encoding might be
sensible, if we don't want the kernel to know about utf-8.
> And it largely works today.
Indeed.
--
http://www.PowerDNS.com Open source, database driven DNS Software
http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Linus Torvalds
On Mon, 16 Feb 2004, Marc Lehmann wrote:
>
> > I'm saying that "the kernel talks bytestreams".
>
> And I am saying that this is not good, which is my sole point.
Fair enough.
However, that's where the unix philosophy comes in. The unix philosophy
has always been to not try to understand the data that the user passes
around - and that "everything is a bytestream" is very much encoded in the
basic principles of how unix should work.
That agnosticism has a lot of advantages. It literally means that the
basic operating system doesn't set arbitrary limitations, which means that
you can do things that you couldn't necessarily otherwise easily do.
It does mean that you can do "strange" things too, and it does mean that
user space basically has a lot of choice in how to interpret those byte
streams.
And yes, it can cause confusion. You don't like the confusion, so you
argue that it shouldn't be allowed. It's a valid argument, but it's an
argument that assumes that choice is bad.
If you want to _force_ everybody to use UTF-8, then yes, the kernel could
enforce that readdir() would never pass through a broken UTF-8 string, and
all the path lookup functions also would never accept a broken string. It'
snot technically impossible to to, although it would add a certain amount
of pain and overhead.
But the thing is, not everyone uses UTF-8. The big distributions have only
recently started moving to UTF-8, and it will take _years_ before UTF-8 is
ubiquotous. And even then it might be the wrong thing to disallow clever
people from doing clever things. Encoding other information in filenames
might be proper for a number of applications.
> And I'd say such a kernel would be highly useful, as it would standardize
> the encoding of filenames, just as unix standardizes on "mostly ascii"
> (i.e. the SuS).
It would also be very painful, since it would mean that when you mount an
old disk, you may be totally unable to read the files, because they have
filenames that such a kernel would never accept.
> > The kernel is _agnostic_ in what it does.
>
> No, it's not. If at all, the kernel specifies a specially-interpreted
> (ascii sans / and \0) byte-stream, as you say yourself.
>
> However, just as with URLs (which are byte-streams, too), byte-streams are
> useless to store text. You need bytestreams + known encoding.
You don't "need" a known encoding. The kernel clearly doesn't need one.
It's a container, and the encoding comes from the outside.
And that's what I mean by agnostic - you can make your own encoding.
Most of the time (but not always) these days UTF-8 is the only sane
encoding to use. But let people do what they want to do.
Choice is _inherently_ good. Trying to force a world-view is bad. You
should be able to tell people what they should do to avoid confusion ("use
UTF-8"), but you should not _force_ them to that if they have good reasons
not to (and "backwards compatibility" is a better reason than just about
anything else).
> But you are saying that you have to feed UTF-8 into the kernel, which is
> not the case either.
No. I'm saying that
(a) "if you want to use complex character sets"
then
(b) "you really have to use UTF-8"
to talk to the kernel.
Note the two parts. You're hung up on (b), while I have tried to make it
clear that (a) is a prerequisite for (b).
Not everybody cares about (a). There are still people who use extended
ASCII, simply because they DO NOT CARE about complex character sets. And
if they don't care, and (a) isn't true, then (b) has no meaning any more.
(In all fairness, some people will disagree with (b) even when (a) is true
and like things like UCS-2. Those people are crazy, but I guess I'd just
mention that possibility anyway).
And this is why I say that the kernel only cares about byte streams, and
having it filter to only accept proper UTF-8 sequences would be a horribly
bad idea. Because it _assumes_ (a). That's what "making policy" is all
about. The kernel should not assume that everybody cares about complex
character sets.
This may change, btw. I'm nothing if not pragmatic. In another twenty
years, maybe everybody _literally_ uses complex character sets, and this
whole discussion is totally silly, and the kernel may enforce UTF-8 or
Klingon or whatever. At some point assumptions become _so_ ingrained that
they are no longer policy any more, they are just "fact".
Linus
Linus Torvalds
On Mon, 16 Feb 2004, Marc Lehmann wrote:
>
> On Mon, Feb 16, 2004 at 11:48:35AM -0800, Linus Torvalds <torv...@osdl.org> wrote:
> > works on the raw byte sequence and isn't confused). Basically accept the
> > fact that UTF-8 strings can contain "garbage", and don't try to fix it up.
>
> But you are wrong, UTF-8 strings never contain garbage. UTF-8 is
> well-defined and is always proper UTF-8. It's a tautology.
>
> The evry idea of "UTF-8 with garbage in it" doesn't make sense.
Sure it does.
You live in a theoretical world where
(a) there is only one standard
(b) people read it
(c) people actually follow it and never have bugs
I've got news for you: none of the above is true.
Which means that IN PRACTICE you will find strings that you think are
UTF-8-encoded, but that don't end up being proper UTF-8.
That's the difference between real world and theory.
And you can either write your programs to be "theoretically correct", or
you can write them to "work".
It's your choice. I know which program I'd prefer to use.
Linus
Marc Lehmann
> The remaining zit is that all these represent '..':
No, they don't. Read the UTF-8 definition...
> This in itself is not a problem, the kernel will only recognize 2E 2E as the
> real .., but it does show that 'document.doc' might be encoded in a myriad
> ways.
No, it can only be encoded in exactly one way *in UTF-8*. It can of course
be encoded differently in other encodings, but in UTF-8, there is only a
single representation. There are no ambiguities.
> So some guidance about using only the simplest possible encoding might be
> sensible, if we don't want the kernel to know about utf-8.
Fortunately, this has all already been taken care of, and is not a problem.
I mean, the _definition_ of UTF-8 works. Wether specific applications
(wether in the kernel or apps) work is a different question. But at
least the specification is rather clear.
Compare this to the URL definition, which only hints that you don't know
the encoding, and therefore, the interpretation as text, of a URL unless
you have an extra channel that communicates it.
While possible, this channel does not exist in practise, creating big
problems for people writing i18n-ized web applications.
The thing is that the kernel certainly _works_ on a very basic level, but
I think the situaiton can be improved by making it clear how to interpret
filenames, which currently is not the case.
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
Marc Lehmann
> > And I am saying that this is not good, which is my sole point.
>
> Fair enough.
Thank you (honestly), for acknowledging it.
> However, that's where the unix philosophy comes in. The unix philosophy
> has always been to not try to understand the data that the user passes
> around - and that "everything is a bytestream" is very much encoded in the
> basic principles of how unix should work.
This never really applied to paths or filenames, although I admit that
this was the intention indeed, hindred by the need of having _some_
out-of-band data like '/'.
So yes, that's the principle.
> That agnosticism has a lot of advantages. It literally means that the
> basic operating system doesn't set arbitrary limitations, which means that
> you can do things that you couldn't necessarily otherwise easily do.
I think '/' is arbitrary, but of course there is a difference between
ruling out 2 bytes out of 256 as opposed to ruling out hundreds of
thousands of combinations.
However, UTF-8 was invented to transform unicode _text_ to filenames by
the inventors of unix.
Yes, this does not mean that they intended your typical kernel to
enforce this, but still this is what plan9 does, and it's a very useful
"limitation" because it removes the need for difficult-to-get-by-with
out-of-band data stating the interpretation of filenames.
Unlike file contents, filenames are always meant to represent text, not
binary data, which I hope you agree with.
> It does mean that you can do "strange" things too, and it does mean that
> user space basically has a lot of choice in how to interpret those byte
> streams.
Windows, for example, allows a lot more strange things like "x.y." to
mean the same as "x.y", or even worse things playing with "illegal
UTF-8".
Most, if not all, of these strange exceptions in windows resulted in
exploitable security flaws.
While this certainly is workable (in the sense that these all were
applicaiton bugs), I think it would be highly useful to be able to enforce
this by saying: "you can put whatever you like into filenames, as long as
it is encoded using UTF-8".
Just like the kernel currently enforces "it must be encoded using
octects", which is trivial, of course, but consider that UCS-2, while also
being a bytestream, is NOT a valid encoding for filenames in linux.
So the kernel is certainly not agnostic versus encodings.
It is, however, agnostic to multibyte encodings as used in e.g. ISO-C.
It does not support wide character encodings.
This is not bad at all, but I claim that the kernel simply isn't as
agnostic as we all might wish, and if there is a specification on
well-formed encodings, it could just as well be "UTF-8".
(Right now it is "most 8-bit encodings, all multibyte encodings but no
multiword encodings with embedded nulls or '/'", which is simply too
unsharp for me to be acceptable. For me, this is clearly a subjective
opinion).
> And yes, it can cause confusion. You don't like the confusion, so you
> argue that it shouldn't be allowed. It's a valid argument, but it's an
> argument that assumes that choice is bad.
Actually, I called for an optional way of enforcing a very promising
encoding over a lot of other encodings.
The kernel simply supports only a subset of possible encodings, and I said
that being able to limit this choice to the only encoding that even _you_
call the only sane one, as an option.
> If you want to _force_ everybody to use UTF-8, then yes, the kernel could
I don't want to do that. I sitll happen to find the rare iso-8859-1
filename on my disk, and almost puke everytime because it breaks bash
etc. I even modified my terminal emulator (rxvt) to support the
occasional german umlaut because it's so convinient.
But all in all, it would be very much easier if this simply couldn't
happen. Especially because it's easy to get from "illegal utf-8 allowed"
to "security issue in formerly secure code".
> enforce that readdir() would never pass through a broken UTF-8 string, and
> all the path lookup functions also would never accept a broken string. It'
> snot technically impossible to to, although it would add a certain amount
> of pain and overhead.
I can certainly yield to arguments like "if you want it, send a patch good
enough to be accepted" :)
> But the thing is, not everyone uses UTF-8.
I don't want to enforce it, exactly because, as you say, it'll
take years. I do think that a way to enforce this as local (or
distribution-wise) policy, would be very helpful indeed.
I don't mind if some embedded system doesn't want the overhead of having a
UTF-8 environment that simply isn't useful for it.
> people from doing clever things. Encoding other information in filenames
> might be proper for a number of applications.
But it will simplify a lot of applications that otherwise have no idea on
filename encoding (it's a pain in the ass in perl, which has generally
good (unicode-)text support, but still filenames, which are text, simply
come out garbled most of the time).
It's rather similar to the discussion about extended attributes (ok, it's
hornets nest, or at leats was one :). Having a clear API definition would
certainly help.
It's no policy either, as applicaitons wanting to encode e.g. binary
information can certainly do so using UTF-8, just as they currently can do
so by other means, e.g. escaping \0 and /.
> It would also be very painful, since it would mean that when you mount an
> old disk, you may be totally unable to read the files, because they have
> filenames that such a kernel would never accept.
That's why I asked for a mount option. I might be too extreme for your
taste in my opinions, but I am still living on earth. I, too, have these
disks.
However, I also have some windows disks, and umlauts or japanese
characters on these often result in chaos, which is not really a bug in
linux either, but highly annoying to people trying to use linux.
> You don't "need" a known encoding. The kernel clearly doesn't need one.
> It's a container, and the encoding comes from the outside.
Right now the kernel does need encoding for _some_ byte values. I can't
use UCS-2, it simply isn't agnostic.
Please, I don't want to use UCS2, UTF-16, or other such atrocities :) I
am just trying to make the point that the kernel already enforces some
encodings.
> And that's what I mean by agnostic - you can make your own encoding.
Given the limitations of the kernel in interpreting byte streams, I can.
However, only under the existing constraints.
I'd like to have a more forcing constraint - UTF-8. It might never be
implemented (especially not against your will), but I think it isn't more
idiotic than what the kernel currently does.
Again, I find the "restrictions" unix has with respect to filenames
very sane and useful, and if at all, it's usually a problem that
filenames can contain unusual characters like \n.
> Choice is _inherently_ good. Trying to force a world-view is bad. You
Yes! I want the choice of having a kernel supporting UTF-8 (again, meaning
active support, i.e. not accepting malformed utf-8) :)
> UTF-8"), but you should not _force_ them to that if they have good reasons
> not to (and "backwards compatibility" is a better reason than just about
> anything else).
Well, being able (having the choice) to force is good. Just like having
nosuid mount options is good, because it allows admins the choice.
> No. I'm saying that
> (a) "if you want to use complex character sets"
> then
> (b) "you really have to use UTF-8"
> to talk to the kernel.
>
> Note the two parts. You're hung up on (b), while I have tried to make it
> clear that (a) is a prerequisite for (b).
Ok, this is true.
> (In all fairness, some people will disagree with (b) even when (a) is true
> and like things like UCS-2. Those people are crazy, but I guess I'd just
> mention that possibility anyway).
You have to admit, however, that apart from UCS-2 being obvious insanity
as opposed to UTF-16 or some 32 encoding, that some people have a point
in asking for these.
It IMHO not useful to support this simply because the posix API cannot be
made to deal with it. But I wouldn't rule out encodings like UTF-32 as
simply being crazy.
> This may change, btw. I'm nothing if not pragmatic. In another twenty
> years, maybe everybody _literally_ uses complex character sets, and this
> whole discussion is totally silly, and the kernel may enforce UTF-8 or
> Klingon or whatever. At some point assumptions become _so_ ingrained that
> they are no longer policy any more, they are just "fact".
True. Thanks a lot for explaining your arguments in this detail. In
fact, I can accept most if not all of your arguments, but I sitll think
it would be nice to have this extra functionality.
Arguments like "it's a pain to implement" (which I don't think it is, but
you are clearly better in judging that!), weigh even more to me.
So even if I think it's a good idea, it might never be implemneted for
purely practical reasons.
(end of discussion, I think, for me at last)
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
vi...@parcelfarce.linux.theplanet.co.uk
> The point is, however, that this is highly undesirable, and it would be
> nice to have a kernel that would (optionally) fully support a UTF-8
> environment in where applications can feed UTF-8 and _expect_ UTF-8 in
> return, which _is_ a security issue.
>
> It's very desirable to have a kernel that actively supports this. IT is
> clearly not _required_, of course. But then again, process abstraction
> is also not required...
Mind taking the demagogy elsewhere? Note that the same handwaving applies
to e.g. file contents. Care to explain what makes read() and write()
different in that respect?
Jamie Lokier
> Why not create "stty utf8" so that non-UTF-8 terminals and UTF-8
> terminals alike can work with a Linux convention that all programs
> enter and display UTF-8? It would simplify a lot of things.
I little thought and an experiment later, and I discovered:
When you edit a line with the kernel's terminal line editor, when you
press the Delete key, it writes backspace-space-backspace and removes
one byte from the input. That fails to do the right thing on UTF-8
terminals.
For example, in a UTF-8 xterm or Gnome terminal, or even on the Linux
console after running "unicode_start", run the command "cat" by
itself, then type "ייי", then hit DEL twice - it will show one
accented letter(*). Press enter, and cat will echo the line
containing _two_ letters.
There is no fancy environment setting which corrects this problem.
The kernel needs to know it's dealing with a UTF-8 terminal for basic
line editing to work.
(*) The text in quotes is three E WITH ACUTE letters, in case that
doesn't show properly in your mailer.
Jamie Lokier
> perl -e 'for (glob "*") { rename $_, "ņi-".$_ or die "renam=
e: $!\n"; }'
>
> (NB: The prefix string is N WITH CEDILLA followed by "i-").
> (Hint: it mangles perfectly fine non-ASCII file names).
>
> Perl has no perfect behaviour to offer, because what should that
> behaviour be if readdir() might return a non-UTF-8 byte sequence
> as a name?
I've had someone point out that the perl script mangles non-UTF-8
filenames, and there is no correct behaviour for that case.
In fact the _real_ bug is that it mangles perfectly fine UTF-8 filename=
s.
It's a Perl quirk, but the behaviour is like that for compatibility
with non-UTF-8 filesystems. I wanted to show how just using UTF-8 for
filenames isn't _yet_ as straightforward as it should be.
Linus Torvalds
On Mon, 16 Feb 2004, Jamie Lokier wrote:
>
> I little thought and an experiment later, and I discovered:
>
> When you edit a line with the kernel's terminal line editor, when you
> press the Delete key, it writes backspace-space-backspace and removes
> one byte from the input. That fails to do the right thing on UTF-8
> terminals.
Yes. I looked at that a year ago, and it should be pretty easy to make the
backspace code look more like the "delete word" code - except the "word"
is just a utf character.
(Btw, that's one of the things I like about UTF-8, and shows how _well_
designed it is - it's trivial to find the beginning of a UTF-8 character,
even when just doing a stupid scan backwards).
I didn't care enough to really bother fixing it - the fact is, that people
who care about UTF-8 tend to have to be in graphics mode anyway, and there
is something to be said for keeping the text console simple even if it
means it lacks functionality.
But if somebody cares more than I do (hint, hint ;), I do think it should
be fixed.
> There is no fancy environment setting which corrects this problem.
> The kernel needs to know it's dealing with a UTF-8 terminal for basic
> line editing to work.
Yes. And I'd happily take patches for it.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> It would also be very painful, since it would mean that when you mount an
> old disk, you may be totally unable to read the files, because they have
> filenames that such a kernel would never accept.
Alas, once userspace has migrated to doing everything in UTF-8, you
won't be able to read those files because userspace will barf on them.
Then you'll be glad to have a mount option which converts iso-8859-1
to UTF-* :) (Even if the old disk as actually not iso-8859-1, at least
you'll be able to read it's mangled filenames, rather than userspace
tripping over them).
-- Jamie
Linus Torvalds
On Mon, 16 Feb 2004, Jamie Lokier wrote:
>
> Alas, once userspace has migrated to doing everything in UTF-8, you
> won't be able to read those files because userspace will barf on them=
.
Nope. Read my other email. Done right, user space will _not_ barf on th=
em,
because it won't try to "normalize" any UTF-8 strings. If the string ha=
s
garbage in it, user space should just pass the garbage through.
We've had this _exact_ issue before. Long before people worried about
UTF-8, people worried about the fact that programs like "ls" shouldn't
print out the extended ASCII characters as-is, because that would cause
bad problems on a terminal as they'd be seen as terminal control
characters.
Does that mean that unix tools like "rm" cannot remove those files? Hel=
l
no! It just means that when you do "rm -i *", the filename that is prin=
ted
may not have special characters in it that you don't see.
Same goes for UTF-8. A "broken" UTF-8 string (ie something that isn't
really UTF-8 at all, but just extended ASCII) won't _print_ right, but
that doesn't mean that the tools won't work. You'll still be able to ed=
it
the file.
Try it with a regular C locale. Do a simple
echo > åäö
(that's latin1), and do a "rm -i åäö", and see what it says.
Right: it does the _right_ thing, and it prints out:
torvalds@home:~> rm -i åäö
rm: remove regular file `\345\344\366'?
In other words, you have a program that doesn't understand a couple of =
the
characters (because they don't make sense in its "locale"), but it stil=
l
_works_. It just can't print them.
Which, if you think about is, is 100% EXACTLY equivalent to what a UTF-=
8
program should do when it sees broken UTF-8. It can still access the fi=
le,
it can still do everything else with it, but it can't print out the
filename, and it should use some kind of escape sequence to show that
fact.
The two cases are 100% equivalent. We've gone through this before. Ther=
e
is a bit of pain involved, but it's not something new, or something
fundamentally impossible. It's very straightforward indeed.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
Linus Torvalds
On Mon, 16 Feb 2004, Linus Torvalds wrote:
>
> Which, if you think about is, is 100% EXACTLY equivalent to what a UTF-8
> program should do when it sees broken UTF-8. It can still access the file,
> it can still do everything else with it, but it can't print out the
> filename, and it should use some kind of escape sequence to show that
> fact.
Side note: a UTF-8 program needs to do escape handling _anyway_, because
even if the filename is 100% UTF-8 compliant, you still can't print out
all the characters as such. In particular, charcters like '\n' etc are
obviously perfectly fine UTF-8, yet they need to be escaped when printing
out filenames in a file selector.
So I claim (and yes, people are free to disagree with me) that a
well-written UTF-8 program won't even have any real extra code to handle
the "broken UTF-8" code. It's just another set of bytes that needs
escaping, and they need escaping for _exactly_ the same reason some
regular utf-8 characters need escaping: because they can't be printed.
So it's all the same thing - it's just the reasons for "unprintability"
that are slightly different.
Now, I'll agree that getting the escaping right (whether for things like
'\n' or for byte sequences that are invalid UTF-8) can be painful. I just
don't think that the pain is in any way specific for "invalid UTF-8". It's
just _hard_ to think of all the special cases, and most programs have bugs
because somebody forgot something.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Alex Belits
> > I have never claimed that the kernel really talk s UTF-8, and indee=
d, I
> > would say that such a kernel would be terminally and horribly broke=
n.
>
> And I'd say such a kernel would be highly useful, as it would standar=
dize
> the encoding of filenames, just as unix standardizes on "mostly ascii=
"
> (i.e. the SuS).
>
> However, just as POSIX is a nice but very limited base, (mostly) ASCI=
I
> is a nice and very limited base. UTF-8 would also be a good base.
UTF-8 is dependent on Unicode, that is cumbersome, not user-expandabl=
e,
not includes an ability to reliably implement subsets of it, poorly
supports language identification/language-dependent processing, and is
controlled by a single organization with extremely poor ability to
incorporate changes into the standard when it becomes necessary. This
means, it's quite possible that this standard will be replaced by
something better in the future when multilingual documents will become
widely used -- right now they certainly are not, and this is why poor
design of Unicode is tolerated by users, and this is also why many peop=
le
use non-Unicode-based charsets. Enforcing UTF-8 will burn the bridges t=
o
any other language support infrastructure or encoding, right at the tim=
e
when such infrastructure is likely to be created.
> 8-bit bytes as filenames is not a good base, however, since they enfo=
rce
> a difefrent layer of interrpetation between the user and the kernel, =
and
> this interpretation cannot be based on the locale nor the filesystem
> itself, as there is no way to find out what encoding the filename is =
in.
This is a matter of GUI implementation. If someone cared about this, =
he
would store language metadata with filename, too, however this is clear=
ly
contrary to the Unix filesystem design.
> 8-bit bytes is convinient, but not useful for i18n environments. in t=
he
> past, it was also convinient and nobody cared, since everything was
> either 8-bit or double-byte, and nobody exchanged files.
I did, and it worked _fine_. Everyone who is willing to use UTF-8 is
free to do this right now, and everything will already work great for
them. Writing software to deliberately enforce UTF-8 is something
completely different from using UTF-8 for yourself.
> This, however, is going to change, and the current methodology of "ju=
st
> guess, you might be right" is a hindrance to this.
This was "going to change" for more than a decade already, or,
alternatively, already happened if you listen to someone like Martin
Duerst. The reality is, everything can pass UTF-8 already, yet people u=
se
other encodings for everything, too, and as long as they don't break,
things work. Breaking byte-value transparency in any place in the syste=
m
is counterproductive until the moment when everyone uses UTF-8. But I h=
ope
before that day comes, UTF-8 and Unicode will be replaced with somethin=
g
more sane.
> > The kernel is _agnostic_ in what it does.
>
> No, it's not. If at all, the kernel specifies a specially-interpreted
> (ascii sans / and \0) byte-stream, as you say yourself.
>
> However, just as with URLs (which are byte-streams, too), byte-stream=
s are
> useless to store text. You need bytestreams + known encoding.
MIME has a perfectly usable standard for declaring encodings, and hug=
e
amounts of text (that may include filenames) are distributed by
MIME-compliant or MIME-like protocols (mail and HTTP, to name two). Thi=
s
keeps the metadata in the userspace, and it also happens that everythin=
g
that displays multilingual text is entirely in userspace. Why kernel is
supposed to mess with this, is beyond me.
> If filenames were not names, but just binary id's I would agree, but
> this is not at all how filenames are used not how their use is applie=
d.
>
> Filenames are composed of text, but the kernel gives no indication
> on how to interpret this text, and as a matter of fact, nothing else
> gives this indication. glib etc. uses G_BROKEN_FILENAMES to force
> locale-encoding. But as others have said, one mans locale is unlike o=
ther
> mens locale.
And this is perfectly fine. Displaying and editing multilingual text =
is
a user interface issue, that kernel should not be involved in. Dealing
with locales is not limited to displaying but includes input methods, a=
nd
in many cases text processing, hyphenation rules, phonetic matching, et=
c.
that are specific to locale. Nothing in UTF-8 helps with those things,
they still have to be done based on metadata that is entirely in
userspace.
I admit that some of my time would be better spent if I did a design =
of
an expandable pluggable language support modules library that would
understand "strings with metadata" and support some kind of stateful
language/encoding serialization for this purpose. It doesn't look like =
the
multilingual documents are common enough to justify designing such a th=
ing
yet, however if such system was developed, it would be a good example o=
f
how expandability is at best ortogonal, and at worst poorly compatible
with enforcing Unicode. Obviously, all this will have to be done entire=
ly
in userspace, and all that it will require from kernel is byte-value
transparency. Singling out filenames as something that has to use a
particular encoding, just because current king of the i18n mess is
Unicode, is at least nearsighted, at most inconsistent.
> > really care AT ALL what you feed it, as long as it is a byte-stream=
.
> >
> > Now, that implies that if you want to have extended characters, the=
n YOU
> > HAVE TO USE UTF-8.
>
> You say so, but there is no logical connection between these two
> statements. I can store latin1 easily in a bytestream, as I can store
> iso-2022-jp or euc-jp. But they are incompatible to UTF-8.
Latin1 is most likely to be stored in iso8859-1, with a single curren=
tly
widely used exception of MS Word and similar programs. I hope, it is
pretty clear that what those programs operate on, is not plain text, an=
d
can not replace plain text any soon.
> You are yelling at me for no good reason. "YOU HAVE TO USE UTF-8". Wh=
y
> should this be? The kernel certainly enforces this. Even you claim th=
at
> I don't have to, as the kernel doesn't care.
>
> However, if you think so violently that it has to be UTF-8 that you e=
ven
> yell it, then why doesn't the kernel comply to this rule? Why should =
an
> applicaiton "HAVE TO" use utf-8 for input when the kernel doesn't eve=
n
> try to comply and hands out illegal output?
If the program breaks on the malformed input, it's a bad program in t=
he
first place. And if it makes assumptions about what is "malformed" base=
d
on something that other users may disagree with, it's a problem of eith=
er
program's users or authors. OS kernel may be in a position where it can
step in and resolve this dispute in favor of either group, however it w=
ill
be highly inappropriate to do so based on some little discussion betwee=
n
UTF-8 supporters and opponents that we can have here.
I can point at the example of this "solution" that happened years ago
when UCS-2 was all the rage, and it got hardcoded and enforced by NTFS
and everything that handles it. Who is laughing about that decision now=
?
> This is just like mmap sometimes returning a page number and sometime=
s a
> byte address... this would also not be useful unless you know the uni=
t
> that mmap returns (addreesses in multiples of 1).
Huh? From the user's point of view mmap always returns a pointer.
Implementation enforces page boundaries, and all applications should be
aware that the address given to it can not be assumed to be used for
anything meaningful, however every application relies on the fact that =
it
gets a pointer to a mapped address. If someone jumped in and demanded t=
hat
given address must be used, and mmap() should assume MAP_FIXED to be
always set, it would be similar to the demand that kernel should only
allow UTF-8 -- just in that case the brokenness would be more obvious.
> > That's what I'm saying. I am _not_ saying that the kernel uses UTF-=
8.
>
> But you are saying that you have to feed UTF-8 into the kernel, which=
is
> not the case either. I certainly don't have to..., and, what's worse,
> you haven't given any indication of why one has to. Just because you =
say
> so? Or is there actually a reason? If there is a reason, why doesn't =
the
> kernel, in return, also follow this reasoning?
>
> > kernel doesn't care one way or the other. As far as the kernel is
>
> It doesn't. But the point is that it should. If the kernel would do
> everything we want it to do there would be no point in enhancing it.
Kernel certainly CAN enforce an encoding. It also can disallow filena=
mes
that contain "Pu-239" substring, and there may be people who would argu=
e
that it should be a good thing.
> > concerened, you could uuencode all the stuff, and the kernel wouldn=
't
>
> (Yes, because uuencode has the peculiar property of neither generting=
\0
> nor /. It doesn't work in general with byte-streams).
>
> > think you're crazy. The kernel _only_ cares about byte streams.
>
> The kernel _interprets_ the byte-stream already. And some byte-stream=
s are
> no valid filenames _already_.
This only affects two byte values, that happen to be universally
understood, and arenever required by any charset to represent anything =
but
what they are in ASCII. UTF-8 is hardly unique in the way of handling
them, however it imposes huge number of restrictions that may conflict
with what people do or will likely do.
> > > There is no way for applications to handle UTF-8 and illegal-utf8=
in
> > > a sane way, so most apps will either eat the illegal bytes, skip =
the
> > > filename, or crash (the latter case is clearly a bug in the app, =
thr
> > > former cases aren't).
> >
> > What you're complaining about are bad user applications. It has _ze=
ro_ to
> > do with the kernel.
>
> Could you elaborate on why these apps are bad? What I am interested i=
n
> is to know how to fix them? since there is simply no way to interpret
> the names returned by the kernel as the corresponding meta-informatio=
n
> is missing.
"Be liberal in what you accept and strict in what you produce". Most =
of
applications don't even have user interface, and therefore should not b=
e
concerned with how characters look like in the first place.
> Consider an OS that allows different characters for path-seperators (=
unix
> only allows '/'). Without the knowledge of the path seperator it woul=
d be
> impossible to interpret paths.
Show me the person who uses file names and doesn't know how the path
separator looks like.
> And without knowledge about encoding it's
> impossible (but slightly less dramatic) to correctly interpret filena=
mes.
This is a userspace issue. Some kernel MAY support this metadata,
however Unix filesystem is not designed that way. If someone is eager t=
o
embed the encoding information into a filename, MIME allows that alread=
y,
it's just impractical.
> > The kernel doesn't set policy. The kernel says "this is what I can =
do,
> > you set policy".
>
> Exactly. The kernel could specify the API to use UTF-8. This is not m=
ore
> policy than it currently enforces.
Says who? How would this policy, if enforced, improve anything, other
than supporting UTF-8 proponents that were unsuccessful in their attemp=
ts
to eradicate all other charsets and encodings despite a decade-old
religious war? They may want to recruit Linux kernel to support their
efforts, however no matter how anyone feels about this issue, why would
kernel design mess with this? What's next, preventing C compilers from
reading goto from the source files? Correcting "teh" and "yuo", and
refusing to send SMTP when the message contais "Me, too"? Have a flag t=
hat
prevents running vi or emacs depending on which of those editors the
sysadmin, or distro maker, hates with a passion? Aren't all those thing=
s
way too far in the area of "activism"?
>
> Or do you suggest that the ability to change the source and replace a=
ll
> occurences of '/' by '\\' means that '/' is not enforced as policy on
> path seperators?
>
> We basically seem to disagree on what, exactly, policy is. Policy (to
> me) is something that differentiates between several incompatible
> alternatives. Chosing policy means to rule out other (useful)
> alternatives.
But again, who should be able to choose the rules? UTF-8 is not
universally accepted. It is a fact. One may wish that it was accepted,
someone else may wish that it was less ugly so it could be accepted, an=
d
vast majority of people don't care, use whatever they have, and will be
pissed if it will get broken just because someone is on a crusade to te=
ll
everyone, how they are supposed to write text.
> One could argue that '/' is a policy because it precludes the '/'
> character from being used in filenames, sth. some filenames or operat=
ing
> systems support.
This is a universally accepted policy. All Unix users know that, it i=
s
written in standards (real ones, like POSIX), and no one uses path
separator for anything else. UTF-8 is a standard that some people use, =
and
some other people want everyone to use, however it's not universally
accepted, and deliberately breaking others' stuff to force them into us=
ing
UTF-8 is, again, unwarranted activism beyond the scope of what kernel i=
s
supposed to do.
> I'd say (probably as much as you) that this policy is not a real poli=
cy,
> it's just idiotic.
It is in a standard, that no one who uses Linux (or any other unixlik=
e
system) ever broken because it was there from the very beginning. UTF-8=
is
something that was designed by a bunch of people who wanted to make it =
a
great gift to the world (or to profit from making more convoluted softw=
are
-- hard to tell at this point), and it _competes_ with the status quo.
Linux is not a new system that defines═the rules, plan9 is. Plan9 is
developed as something that was supposed to "cure" Unix inconsistency a=
nd
lack of rules, so it got stuffed with personal idiosyncrasies of its
author. And the result is hardly a confirmation that those idiosyncrasi=
es
resulted in a superior, or even just more practical system.
> But enforcing other restrictions on filenames should magically be rea=
l
> policy? This is obviously bot idiotic at all, and should be carefully
> explored.
>
> > And UTF-8 just happens to be the only sane policy for encoding comp=
lex
> > characters into a byte stream. But it is not the only policy.
>
> Just as '/' is not the only possible path seperator. If that is your
> point, you should explain why enforcing this is ok while supporting u=
tf-8
> (not enforcing, just supporting, meaning having the ability to rule o=
ut
> non-utf-8 sequences when the admin wants this) is not.
'/' is not the only possible separator, however it's pretty clear th=
at
no other separator is superior to it. Only two other characters were ev=
er
used for this purpose, and both are abandoned by pretty much everyone, =
so
it's unlikely that many people are going to suffer from the choice of a
slash. Ever.
UTF-8 is one of the infinite number of possible solutions, and judgi=
ng
by the rate of acceptance and by the rate at which Unicode representati=
ons
and charsets are being invented, it is very likely that not the last
encoding that will be used by humans.
> > Another sane policy is to say "byte streams are latin1". It's not a=
n
> > acceptable policy for encoding _complex_ characters, but it is a po=
licy.
> > And it's a perfectly sane one.
>
> I agree that it is sane. But it is not very useful for the future, as
> people who want russian filenames are plainly unable to use the other
> filenames in a sensible way. There is no way to know the encoding.
Russians sometimes use filenames in koi8-r, sometimes even in
windows-1251, and mostly in ASCII. Never heard any Russian complaining
about this -- and certainly never in my life I have seen a Russian
filename in UTF-8, nor do I look forward to it. And I am Russian.
> > In short: filenames are byte streams. Nothing more.
>
> Right now, they aren't. Not all sequences of bytes are valid filename=
s
> already, and I think this is perfectly o.k.
Slippery slope.
> > And when I say that you have to talk to the kernel using UTF-8, I'm=
only
> > claiming that it is the only sane way to encode extended characters=
in a
> > byte stream. Nothing more.
>
> And i fully agree.
>
>
I disagree -- not with the point that UTF-8 is kinda sane but with th=
e
idea that UTF-8 should be enforced to make it the last encoding that
people will be ever able to use. It may be "sanest" now for a limited
purpose of handling truly multilingual text, but certainly only a
practical use of multilingual documents (what is extremely rare now) ma=
y
answer the question, what encoding(s) should be used for this purpose. =
I
don't think, someone ten years from now would enjoy writing a wrapper t=
hat
will convert filenames from some "Expandable Multilingual Metacharset
2014" so they look like valid UTF-8 when they are written into some arc=
haic
filesystem that doesn't even support Elvish and Klingon properly.
--
Alex
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
jw schultz
> jw schultz wrote:
> > If you have a filesystem with filenames that don't conform
> > to your policy write userspace tools to detect and/or fix
> > them. If you have programs creating non-conforming
> > filenames, fix or rm those programs.
>
> You do understand that GNU coreutils, bash etc. are among those
Doesn't matter where they come from.
> programs, right? As in "touch zöe.txt" creates a non-conforming
> filename...
Your concrete example is a good one. Where did that
filename come from? It would seem to have come from the
keyboard via a tty (or simulator) which also had to display
it. I'd say this is an argument for the terminal to display
UTF-8 and convert intput into UTF-8. That is something that
seems to be not consistantly done as yet. Ultimately it
seems to be a responsiblity of the user interface, whether
tty or GUI. Until that happens the shells might be able to
fill the gap, however poorly.
Perhaps the utilities that don't attempt to interpret
filenames should treat filenames exactly like the kernel
does.
> > OK. The questions have been asked and answered.
> > Asking again and again and again won't change the answer.
>
> The question of what a program like this should do has not been
> answered:
>
> perl -e 'for (glob "*") { rename $_, "??i-".$_ or die "rename: $!\=
n"; }'
>
> (NB: The prefix string is N WITH CEDILLA followed by "i-").
>
> Hint: it mangles perfectly fine non-ASCII file names, instead of just
> prefixing the prefix string. If you change the program to correctly
> prepend the prefix string, then it mangles non-UTF-8 names, which is
> arguably correct, but can result in you losing some files.
Then if there is incorrect behavior is it the shell, tty or perl that i=
s
getting things wrong here.
> This _is_ a userspace problem, but it is a genuine problem for which
> no good answer is yet apparent.
I'll buy that. Then the first question to ask is "what is
the correct forum for resolving this".
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: j...@pegasys.ws
Remember Cernan and Schmitt
Marc A. Lehmann
.org> wrote:
> Try it with a regular C locale. Do a simple
>
Just for your info, though. You can't even input these characters in a =
C
locale, since your libc (and/or xlib) is unable to handle them (lots of=
SO
C functions will barf on this one). C is 7 bit only.
> Which, if you think about is, is 100% EXACTLY equivalent to what a UT=
F-8
> program should do when it sees broken UTF-8.
The problem is that the very common C language makes it a pain to use
this in i18n programs. multibyte functions or iconv will no accept
these, so programs wanting to do what you are expecting to do need to
re-implement most if not all of the character handling of your typical
libc.
Yes, it's possible....
> The two cases are 100% equivalent. We've gone through this before. Th=
ere
> is a bit of pain involved, but it's not something new, or something
> fundamentally impossible. It's very straightforward indeed.
The "bit" is enourmous, as you can't use your libc for text processing
anymore.
Yes, it works in non-i18n programms, but right now most programs get
i18n support, which means they will all fail to properly handle
non-locale characters.
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
jw schultz
> > Quote from Jeff Garzik <jga...@pobox.com>:
> > None of this is a real problem, if everything is set up correctly and
> > bug free. Unfortunately the Just Works thing falls apart in the,
> > (frequent), instances that it's not :-(.
>
> And this is the whole point.
>
> BTW, to people trying to explain some properties of UTF-8 to me. I don't
> think ad-hominem attacks like assuming that I don't understand UTF-8
> (without any indication that this is so) are useful.
>
> The point here is that the kernel does, in a very narrow interpretation,
> not support the use of UTF-8, because proper support of UTF-8 means that
> no illegal byte sequences will be produced.
That "interpretation" is so narrow as to be unrealistic.
The kernel supports UTF-8 the same way a stage supports
rock musicians. You confuse support with enforce, rather
like confusing tolerance with endorsement.
And it should be noted that the kernel doesn't produce file
names. It only passes them along.
> Of course, I can feed the kernel UTF-8, and if everybody does that, it
> will generally work quite fine. However, Windows surely works fine if
> every program only feeds allowed values into system calls. And even unix
> dialects without memory protection work, as long as everybody plays
> fair.
>
> The point is, however, that this is highly undesirable, and it would be
> nice to have a kernel that would (optionally) fully support a UTF-8
You mean enforce again. That enhancement request has been
rejected repeatedly because such a thing would be highly
undesirable. What might be a convenient but unnecessary
restriction today is too likely to become an unbearable
restriction tomorrow. I don't want the kernel to have to
care about what is or isn't valid UTF-8. I certainly don't
want to have the kernel loaded with outdated character
tables.
> environment in where applications can feed UTF-8 and _expect_ UTF-8 in
> return, which _is_ a security issue.
I want an environment where applications can feed bytestreams
and expect the same bytestream in return. I see enough
problems as a result of filesystems that don't do that.
> It's very desirable to have a kernel that actively supports this. IT is
You mean enforces again. Kernel as police, next thing you
will want is a kernel that prevents undesirable character
sequences.
> clearly not _required_, of course. But then again, process abstraction
> is also not required...
I'll tell you what. Patch libc. You can add UTF-8 filename
enforcement to libc. There are only a few system calls that
would need to have their wrappers enlarged. I'm sure the
libc people will direct you to someplace very warm if you
ask them for this enhancement.
--
________________________________________________________________
J.W. Schultz Pegasystems Technologies
email address: j...@pegasys.ws
Remember Cernan and Schmitt
Nick Piggin
Marc Lehmann wrote:
>On Mon, Feb 16, 2004 at 07:48:19PM +0000, John Bradford <jo...@grabjohn.com> wrote:
>
>>Quote from Jeff Garzik <jga...@pobox.com>:
>>None of this is a real problem, if everything is set up correctly and
>>bug free. Unfortunately the Just Works thing falls apart in the,
>>(frequent), instances that it's not :-(.
>>
>
>And this is the whole point.
>
>BTW, to people trying to explain some properties of UTF-8 to me. I don't
>think ad-hominem attacks like assuming that I don't understand UTF-8
>(without any indication that this is so) are useful.
>
>The point here is that the kernel does, in a very narrow interpretation,
>not support the use of UTF-8, because proper support of UTF-8 means that
>no illegal byte sequences will be produced.
>
>
So does the kernel support the English language? Does your
email client?
Helge Hafting
> On Mon, Feb 16, 2004 at 02:40:25PM -0800, Linus Torvalds <torvalds@os=
dl.org> wrote:
>
>>Try it with a regular C locale. Do a simple
>>
>> echo > едц
>
>
a C
> locale, since your libc (and/or xlib) is unable to handle them (lots =
of SO
> C functions will barf on this one). C is 7 bit only.
>
>
>>Which, if you think about is, is 100% EXACTLY equivalent to what a UT=
F-8
>>program should do when it sees broken UTF-8.
>
>
> The problem is that the very common C language makes it a pain to use
> this in i18n programs. multibyte functions or iconv will no accept
> these, so programs wanting to do what you are expecting to do need to
l
> libc.
>
> Yes, it's possible....
All you need is a possible_garbage_to_properly_escaped_utf8(char *strin=
g)
in libc. Any program that wants to display filenames it got
straight from readdir (or any binary file contents) will simple feed
the string through that and get back a string with
escapes for anything that isn't utf8. It is a write-once, use
everywhere thing.
Once up on a time, there were serious problems when someone created
filenames like "; rm -fr *" Today we use tab completion
and get bash to present the filename with proper escapes. It is then h=
armless.
Bad utf8 can be handled the same way.
> The "bit" is enourmous, as you can't use your libc for text processin=
g
> anymore.
Not the current libc, but libc can be improved upon. The same happened =
to
silly code that weren't 8-bit clean.
Helge Hafting
Jamie Lokier
> So I claim (and yes, people are free to disagree with me) that a
dle
> the "broken UTF-8" code. It's just another set of bytes that needs
> escaping, and they need escaping for _exactly_ the same reason some
.
Even XML suffers from these sorts of problems: some Unicode characters
aren't allowed in XML, even as numeric references, so in theory XML
applications have to reject or escape some strings.
> So it's all the same thing - it's just the reasons for "unprintabilit=
y"
> that are slightly different.
My difficulty with directories containing non-UTF-8 filenames shows up
with web pages in Perl, and not the printability part. Please excuse
the Perl-oriented examples; Perl has good support for UTF-8 while also
working with arbitrary byte strings, so it's a fine language to
illustrate current problems.
What do you put in a URL composed from filenames in a directory
listing page? The obvious thing is to %-escape each byte of the
names, in fact that's what everybody does.
In a language like Perl, where strings are labelled according to their
encoding, that means when you unscape the URL you get a string
labelled as "byte string". You shouldn't tell Perl it's a "UTF-8
string" because some of them won't be (they are strings from
directories).
That's fine if you don't do anything except use those strings
unchanged, but as soon as you want to do something else like prepend a
character with code >= 256 or apply a regex where the pattern has
Unicode characters, Perl transcodes "byte string" to "UTF-8 string"
assuming it was latin1. That, of course, mangles the string when it's
come from a source which is "nominally UTF-8 but might not be".
Your recommendation to simply pass around bytes all the time doesn't
work well, because to maintain basic properties of strings such as
length(a) + length(b) = length(a+b), that implies you either (1)
always do indexing, lengths, splitting etc. on strings as bytes not
characters, or (2) every operation that operates on a string must be
able to accept non-UTF-8 bytes and treat them the same way. (2) is
particularly nasty because then your program's logic can't depend on
the nice properties of UTF-8 strings.
That's why this line of Perl fails:
for (glob "*") { rename $_, "ņi-".$_ or die "rename: $!\n"; }
(The source file, by the way, is assumed to be UTF-8-encoded text).
Perl reads each file name, and declares it to be of type "byte
string". Then "ņi-" is prepended, which contains a character code=
>=
256, so the result must be UTF-8 encoded according to Perl. The
original file name is transcoded from what was assumed to be
iso-8859-1 to UTF-8, "ņi-" is prepended, and that becomes the targ=
et
file name for rename().
This mangles the names; both UTF-8 and non-UTF-8 filenames are mangled
equally badly.
Your suggestion means that Perl should do bytewise concatenation of
the the "ņi-" (in UTF-8) and the filename (no encoding assumed).
It's a good one; it's exactly the right thing to do, and it works.
To do that in Perl, when you take a random byte string (such as from
readdir()) you must tell Perl it's a UTF-8 string, so shouldn't be
transcoded when it's combined with another UTF-8 string. You can do
it, breaking documented rules of course (which say only do this when
you know it's valid UTF-8), with Encode::_utf8_on().
Guess what? That actually works. It does the filename operations
properly given any arbitrary filenames.
But remember I said "every operation that operates on a string must be
able to accept non-UTF-8 bytes and treat them the same way" earlier,
and how this is bad because it's nice to depend on UTF-8 properties?
You've just told Perl to treat an arbitrary byte sequence as UTF-8,
when sometimes it isn't. Among other things, simple operators like
length() and substr() don't work as expected on those weird strings.
When I say don't work as expected, I mean if you had a file named
"müeller" in latin1, Perl will think it's length() is 2. If you h=
ave
a file named "müller", Perl will not only report a length() of 1,
it'll spew a horrible error message when it calculates it.
These aren't Perl problems. These are problems that any program will
have if it follows your suggestion of "keep everything in bytes" but
wants to combine filenames with other text or do pattern matching on
filenames.
It's not a problem if you can pass around a flag with each byte
sequence, carefully keeping readdir() results separate from text until
the point where your prepared to have a policy saying what to do with
non-UTF-8 readdir() results.
But it is a problem when you want to stuff readdir() results in a
general purpose "string" which is also used for text.
That's technically the wrong thing to do, in all programs. In
practice, that's what programs do anyway because it's a lot easier
than having different string types for different data sources.
Most times it works out ok, but for the corners:
> It's just _hard_ to think of all the special cases, and most
> programs have bugs because somebody forgot something.
Exactly.
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
Dave Kleikamp
> True. Thanks a lot for explaining your arguments in this detail. In
> fact, I can accept most if not all of your arguments, but I sitll think
> it would be nice to have this extra functionality.
>
> Arguments like "it's a pain to implement" (which I don't think it is, but
> you are clearly better in judging that!), weigh even more to me.
>
> So even if I think it's a good idea, it might never be implemneted for
> purely practical reasons.
Use jfs with the mount option iocharset=utf8 and you'll get exactly what
you are asking for.
--
David Kleikamp
IBM Linux Technology Center
Linus Torvalds
On Tue, 17 Feb 2004, Marc wrote:
> On Mon, Feb 16, 2004 at 02:40:25PM -0800, Linus Torvalds <torvalds@os=
dl.org> wrote:
> > Try it with a regular C locale. Do a simple
> >
> > echo > едц
>
> Just for your info, though. You can't even input these characters in =
a C
> locale, since your libc (and/or xlib) is unable to handle them (lots =
of SO
> C functions will barf on this one). C is 7 bit only.
Ehh.. It's pointless to tell me that I can't do it. I just did.
The C locale is _not_ 7-bit only. The C locale is the traditional "byte=
locale" for UNIX. It will happily collate 8-bit-characters in their
(numerical) order. Anything else would be seriously broken.
> > Which, if you think about is, is 100% EXACTLY equivalent to what a =
UTF-8
> > program should do when it sees broken UTF-8.
>
> The problem is that the very common C language makes it a pain to use
> this in i18n programs. multibyte functions or iconv will no accept
> these, so programs wanting to do what you are expecting to do need to
> re-implement most if not all of the character handling of your typica=
l
> libc.
These are all teething problems. The thing is, true multi-locale progra=
ms
haven't been around long enough that people take the problems for grant=
ed.
A lot of them work today, but "work" is different from "always does the
right thing". These things take a _long_ time for people to sort out th=
e
full implications of.
(Analogy time: how many people _still_ use "find ... | xargs xxx", even
though that can lead to problems and is thus wrong? You should really =
use
"find ... -print0 | xargs -0 xxx" to get it _right_, but most people
ignore that, because the common form works for most cases.)
The process is complicated by the fact that most of the people who real=
ly
care about UTF-8 and locales are very strict about it: they have been
hitting their heads against latin1 users for a logn time, and they are
frustrated and _tired_ of it, and so they often hate single-byte usage
with a passion, and consider it not only wrong but EVIL. Which is
obviously silly, but hey, I understand why they can feel a bit put off =
by
the problem.
So the multi-byte people often stare at the standards, and then _refuse=
_
to touch anything that isn't standards-compliant. When they see somethi=
ng
incorrect, they'd rather dump core (or just truncate it) than try to
handle it gracefully, becuase they want the whole world to see how
incorrect it is.
Which flies in the face of "Be strict in what you generate, be liberal =
in
what you accept". A lot of the functions are _not_ willing to be libera=
l
in what they accept. Which sometimes just makes the problem worse, for =
no
good reason.
The fact is, you shouldn't use "iconv()" unless you controlled the inpu=
t.
It's a bit like "gets()" - unsafe to use unless you generated the damn
thing yourself and you _know_ it fits in the buffer. But we just don't
have the functions (yet) to do it _right_, and to escape the input some=
way (yeah, yeah, I know you can do it with iconv() and a lot of cruft
around it - the point is that nobody does it, because it's too painful)=
.
Linus
Linus Torvalds
On Tue, 17 Feb 2004, Marc Lehmann wrote:
>
> Because there is a fundamental difference between file contents and
> filenames. Filenames are supposed to be text.
I think this is actually the fundamental point where we disagree.
You think of filenames as something the user types in, and that is
"readable text". And I don't.
I think the filenames are just ways for a _program_ to look up stuff, and
the human readability is a secondary thing (it's "polite", but not a
fundamental part of their meaning).
So the same way I think text is good in config files and I dislike binary
blobs (hey, look at /proc), I think readable filenames are good. But that
doesn't mean that they have to be readable. I can well imagine encoding
meta-data in the filename for some database that uses the filesystem as
its backing store and generates files for large blobs. And then there
would be little if any "goodness" to keeping the filenames readable.
That's also a situation where case-insensitivity can _really_ screw you
(just one of the many).
It may be rare, but unlike you, I don't think there is anything "wrong"
with considering path components to be just "data".
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> Which flies in the face of "Be strict in what you generate, be liberal in
> what you accept". A lot of the functions are _not_ willing to be liberal
> in what they accept. Which sometimes just makes the problem worse, for no
> good reason.
Unicode specifies that a program claiming to read UTF-8 _must_ reject
malformed UTF-8.
Ok, we can just ignore Unicode. :)
But the reason they cite is security: when applications allow
malformed UTF-8 through, there's plenty of scope for security holes
due to multiple encodings of "/" and "." and "\0".
This is a real problem: plenty of those Windows worms that attack web
servers get in by using multiple-escaped funny characters and
malformed UTF-8 to get past security checks for ".." and such.
In theory these are not problems; all programs should be liberal in
what they accept, and robust in handling data from the outside world.
In practice, programs quickly lose track of which text is from the
outside world and which is from a trusted source or checked source.
These worms are quite successful at exploiting things the programmers
didn't think of. Being _conservative_ at all places which scan UTF-8
does seem like it might help a little.
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> I think the filenames are just ways for a _program_ to look up stuff,=
and
> the human readability is a secondary thing (it's "polite", but not a
> fundamental part of their meaning).
Politeness is nice. I'm sure there's a pragmatic reason most
filenames are meaningful text in some human language :)
I'd like a way to type something like "touch zöe.txt" on an ordinary
latin1 terminal and get a UTF-8 filename in my filesystem. Thanks :)
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
Marc Lehmann
> >
> > Because there is a fundamental difference between file contents and
> > filenames. Filenames are supposed to be text.
>
> I think this is actually the fundamental point where we disagree.
I guess that probably explains it. And I know of no striking arguments to
convince you of changing your fundamental opinion.
*sigh*. Ok, we agree to disagree :)
> It may be rare, but unlike you, I don't think there is anything "wrong"
> with considering path components to be just "data".
Yeah, there are three things - text, binary, and data (and probably more).
Filenames are then "mostly text", "no binary", and still suitable for
"data".
I have read the example somebody posted of some application encoding
"near-binary" data into filenames (e.g. uglies like "\n" or worse).
However, I think that these cases are extremely rare and not really worth
supporting. Not supporting this is not a problem for applications - after
all, base64 or escaping (that is needed even for "near-binary") works fine
for these apps, too, ignoring the problem of backwards compatibility.
I think that everyone having had the experience of dealing with filenames
containing \n etc., despite your shell/GUI helping in quoting, will easily
share this opinion about usefulness.
That's why it should be a mount option, i.e. an enforcable standard.
And since it seems that JFS already supports this (to some degree unknown
to me), I don't think it should be such a pain to implement.
But yes, I am most probably not going to implement it, especially not if
it will simply never be accepted.
So I guess it simply won't be done. I think it's omissing some highly
useful feature, but I will survive it.
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / p...@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
Stefan Smietanowski
>>Because there is a fundamental difference between file contents and
>>filenames. Filenames are supposed to be text.
>
> I think this is actually the fundamental point where we disagree.
>
> You think of filenames as something the user types in, and that is
> "readable text". And I don't.
>
> I think the filenames are just ways for a _program_ to look up stuff, and
> the human readability is a secondary thing (it's "polite", but not a
> fundamental part of their meaning).
>
> So the same way I think text is good in config files and I dislike binary
> blobs (hey, look at /proc), I think readable filenames are good. But that
> doesn't mean that they have to be readable. I can well imagine encoding
> meta-data in the filename for some database that uses the filesystem as
> its backing store and generates files for large blobs. And then there
> would be little if any "goodness" to keeping the filenames readable.
Just look at Mozilla's cache... They may have turned the blob into
ascii but it's still a blob.
// Stefan
vi...@parcelfarce.linux.theplanet.co.uk
> But the reason they cite is security: when applications allow
> malformed UTF-8 through, there's plenty of scope for security holes
> due to multiple encodings of "/" and "." and "\0".
>
> This is a real problem: plenty of those Windows worms that attack web
> servers get in by using multiple-escaped funny characters and
> malformed UTF-8 to get past security checks for ".." and such.
Pardon? For that kernel would have to <drumrolls> interpret the bytestream
as UTF-8. We do not. So your malformed UTF-8 for .. won't be treated as
.. by the kernel.
BTW, speaking of Plan 9, they do *NOT* reject malformed UTF-8 in pathnames.
Filtering they do is against ASCII controls - i.e. \1--\37 and \177.
All differences between our generic checks and Plan 9 generic checks (aside
of whatever checks particular fs might do) is:
1) they allow longer pathnames (64K vs our 4K, from my reading of
9/port/chan.c)
2) they do not allow pathnames containing any octet in range 1--31
3) they do not allow pathnames containing DEL (octet 127)
The rest is identical:
* Pathname is split into components by instances of octet 47 (/).
* Component is special if it's {octet 46} or {octet 46, octet 46}
(. and .. resp.).
* Name is terminated by octet 0 (NUL).
* Name components are fed to filesystem drivers without any conversions
- they go as arrays of char, with no concern for encoding.
So could we please put that strawman to rest?
Måns Rullgård
> Linus Torvalds wrote:
>> I think the filenames are just ways for a _program_ to look up stuff=
, and
>> the human readability is a secondary thing (it's "polite", but not a
>> fundamental part of their meaning).
>
> Politeness is nice. I'm sure there's a pragmatic reason most
> filenames are meaningful text in some human language :)
>
> I'd like a way to type something like "touch zöe.txt" on an ordinar=
y
> latin1 terminal and get a UTF-8 filename in my filesystem. Thanks :)
Then hack either bash (or whatever shell you use) or touch to do just t=
hat.
--
Måns Rullgård
m...@kth.se
Jamie Lokier
> On Tue, Feb 17, 2004 at 04:36:13PM +0000, Jamie Lokier wrote:
> > But the reason they cite is security: when applications allow
> > malformed UTF-8 through, there's plenty of scope for security holes
> > due to multiple encodings of "/" and "." and "\0".
> >
> > This is a real problem: plenty of those Windows worms that attack web
> > servers get in by using multiple-escaped funny characters and
> > malformed UTF-8 to get past security checks for ".." and such.
>
> Pardon? For that kernel would have to <drumrolls> interpret the bytestream
> as UTF-8. We do not. So your malformed UTF-8 for .. won't be treated as
> .. by the kernel.
Well, the security checks on ".." which worms get past aren't in the
kernel either. This time _you_ made the strawman :)
What happens is that one program or library checks an incoming path
for ".." components - that code knows nothing about UTF-8 of course.
Then it passes the string to another program which assumes the path
has been subject to appropriate security checks, munges it in UTF-8,
and eventually does a file operation with it. The munging generates
".." components from non-minimal UTF-8 forms - if it's not obeying the
Unicode rejection requirement (which wasn't in earlier versions), that is.
A realistic example is where the second program reads files whose
paths are mentioned in a text file which is parsed as UTF-8, after the
first program has done a security check by grepping for ".."
components.
Unicode says the second program shouldn't accept malformed UTF-8,
precisely because in real scenarios (like this one) there's a mix of
programs and libraries, some aware of UTF-8, some not, and the latter
are involved in security decisions.
Here on linux-kernel we're saying that if the second program accepts
any old byte sequence in a filename, it should preserve the byte
sequence exactly. But any program whose parser-tokeniser is scanning
UTF-8 is very unlikely to do that - it's just too complicated to say
some bits of a text stream must be remembered as literal bytes, and
others must be scanned as multibyte characters.
We can't blame the second program for allowing those dodgy paths,
because it's the _first_ program which is setting policy. We can't
blame the first program, because it doesn't care about UTF-8. The
second program is just obeying orders, and the first program is just
applying POSIX rules.
These type of security holes are quite real, among software which
handles UTF-8 and also deals with paths. At the current time, that
especially means XML, HTML, URIs, web servers and things behind them.
The holes only arise because software which is interpreting UTF-8 is
mixed with software which isn't. That's one of the most useful
features of UTF-8, after all - that's why we use it for filenames.
Understand, this isn't a kernel problems; it is simply a good reason
to reject malformed UTF-8 by programs which parse UTF-8.
-- Jamie
Linus Torvalds
On Tue, 17 Feb 2004, Jamie Lokier wrote:
>
> Well, the security checks on ".." which worms get past aren't in the
> kernel either. This time _you_ made the strawman :)
Note that this is something that the kernel _can_ fix easily.
In particular, we already have flags like LOOKUP_FOLLOW and
LOOKUP_DIRECTORY that we use internally in the kernel to specify how to do
certain operations. We export _part_ of that to user space with the
O_DIRECTORY flag that says "allow open of directories".
And yes, we have security-related ones too (LOOKUP_NOALT disables the
alternamte mount-point lookup).
And it would be _trivial_ to add a LOOKUP_NODOTDOT and allow user space to
use it through a O_NODOTDOT thing. But the people who need it really need
to do it and test it, and they need to be committed enough that they say
"yes, we'd use this, even though it's not portable". Because I don't want
to add features to the kernel that people don't use, and a lot of the
users don't want to use Linux-only things..
Same goes for O_NOFOLLOW or O_NOMOUNT, to tell the kernel that it
shouldn't follow symbolic links or cross mount-points - another thing that
some software might want to use in order to check that you can't "escape"
your subtree.
So these things would be literally trivial to add, and the only issue is
whether people would really use them.
> What happens is that one program or library checks an incoming path
> for ".." components - that code knows nothing about UTF-8 of course.
>
> Then it passes the string to another program which assumes the path
> has been subject to appropriate security checks, munges it in UTF-8,
> and eventually does a file operation with it. The munging generates
> ".." components from non-minimal UTF-8 forms - if it's not obeying the
> Unicode rejection requirement (which wasn't in earlier versions), that is.
But note how my point was that YOU SHOULD NEVER EVER MUNGE A PATHNAME!
It is fundamentally _wrong_ to convert pathnames. You _cannot_ do it
correctly.
The rule should be:
- convert user-input to UTF-8 early (do _nothing_ to it before the
conversion). Allow escape sequences here.
- never ever convert readdir/getcwd/etc system-specified paths AT ALL.
They are already in "extended UTF-8" format (where the "extended" part
is the 'broken UTF-8' thing. I can be like MS and call my breakage
"extended" too ;)
- always _always_ work on the "extended UTF-8" format, and never EVER
convert that to anything else (except when you need to actually print
it, but then you encode it properly with escape sequences, the way you
have to _anyway_).
If you follow the above simple rules, you can't get it wrong. And in those
rules, ".." is the BYTE SEQUENCE in the "extended UTF-8". Nothing more.
Linus
Jamie Lokier
> Understand, this isn't a kernel problems; it is simply a good reason
> to reject malformed UTF-8 by programs which parse UTF-8.
I should make clear: since the kernel _doesn't_ parse UTF-8, the
kernel _isn't_ an appropriate place to reject it.
Any userspace program which treats the result of readdir() as UTF-8
characters for any purpose should reject malformed names. The tough
design decisions are: where in the program to do it, and how to ensure
it will always be done.
You have to reject or escape malformed names at _some_ stage when they
are going to appear in a text context. The trouble is doing it too
soon (where the program calls readdir()) prevents operating on some
files, and doing it later (where the program is going to use it in a
text context) is easy to forget because by the time a string from
readdir() has travelled through many layers of abstraction between
libraries, it's easy to forget its byteish properties.
vi...@parcelfarce.linux.theplanet.co.uk
> What happens is that one program or library checks an incoming path
> for ".." components - that code knows nothing about UTF-8 of course.
>
> Then it passes the string to another program which assumes the path
> has been subject to appropriate security checks, munges it in UTF-8,
> and eventually does a file operation with it. The munging generates
> ".." components from non-minimal UTF-8 forms - if it's not obeying the
> Unicode rejection requirement (which wasn't in earlier versions), that is.
> A realistic example is where the second program reads files whose
> paths are mentioned in a text file which is parsed as UTF-8, after the
> first program has done a security check by grepping for ".."
> components.
>
> Unicode says the second program shouldn't accept malformed UTF-8,
> precisely because in real scenarios (like this one) there's a mix of
> programs and libraries, some aware of UTF-8, some not, and the latter
> are involved in security decisions.
>
> Here on linux-kernel we're saying that if the second program accepts
> any old byte sequence in a filename, it should preserve the byte
> sequence exactly. But any program whose parser-tokeniser is scanning
> UTF-8 is very unlikely to do that - it's just too complicated to say
> some bits of a text stream must be remembered as literal bytes, and
> others must be scanned as multibyte characters.
So what you are saying is that conversion of invalid multibyte sequences
into non-error wide chars followed by conversion back into UTF-8 can
lead to trouble? *DUH*
> The holes only arise because software which is interpreting UTF-8 is
> mixed with software which isn't. That's one of the most useful
> features of UTF-8, after all - that's why we use it for filenames.
The holes only arise because software which is interpreting UTF-8 doesn't
care to do it properly. Software that doesn't interpret it (including the
kernel) doesn't enter the picture at all.
Jamie Lokier
> And it would be _trivial_ to add a LOOKUP_NODOTDOT and allow user space to
> use it through a O_NODOTDOT thing.
Nope. That wouldn't help for a bundle of libraries that goes:
1. Eliminate "." and ".." components, leaving only leading ".."s.
2. Reject path if it has a leading "..".
3. Shove it in a string with some other text and pass to other library.
Next program does:
4. Extract path from string.
5. open ("/var/public/files/$PATH", ...)
O_NODOTDOT won't protect against that.
> Same goes for O_NOFOLLOW or O_NOMOUNT, to tell the kernel that it
> shouldn't follow symbolic links or cross mount-points - another thing that
> some software might want to use in order to check that you can't "escape"
> your subtree.
( O_NOMOUNT is a good idea. I like O_NOFOLLOW - already use it to
avoid lstat() calls. )
> But note how my point was that YOU SHOULD NEVER EVER MUNGE A PATHNAME!
>
> It is fundamentally _wrong_ to convert pathnames. You _cannot_ do it
> correctly.
I know. You know. I think everyone else got it the first time too ;)
Have you ever written a script which takes a pathname and puts it in a
text file, and passes to another to operate on, and just skipped over
the details of what a poorly placed control character would do?
Real applications do exactly that sort of thing. Mostly it works,
occasionally security holes are found. Welcome to the land of dodgy
CGI scripts and program generated Makefiles.
This is the same.
> - always _always_ work on the "extended UTF-8" format, and never EVER
> convert that to anything else (except when you need to actually print
> it, but then you encode it properly with escape sequences, the way you
> have to _anyway_).
>
> If you follow the above simple rules, you can't get it wrong. And in those
> rules, ".." is the BYTE SEQUENCE in the "extended UTF-8". Nothing more.
Yup. It works right up until you pass your string to a library which
doesn't follow that rule, and which munges malformed UTF-8 because
it's _expecting_ well formed UTF-8. E.g. you pass a path in an XML
document; the XML parser at the other end will either munge your path
(causing a security hole), or reject it (which is good).
The right thing to do on these occasions is check and/or escape
"extended UTF-8" prior to putting it into a text context.
Practically, that means a UTF-8 aware program has to keep track of
which text is "extended UTF-8" (i.e. bytes), and which text is real UTF-8.
Practically, it means every interface where a path may be passed in a
UTF-8 string has to define whether that's an escaped path, which will
be unescaped before being used for a system call, or an unescaped path.
Then you get into what kind of escaping.
In theory all those checks and escapings will be in the right places.
In theory C programs don't have buffer overflows either.
It is exasperated because UTF-8 is often passed through middle-layer
programs and libraries that don't know anything about it, so when
assembling a whole system it's all too easy to lose track of where to
put the checks and escapings - and where not to.
Yes there _is_ a perfectly fine solution: the one you gave.
In practice it is difficult to ensure a whole system where paths are
mixed with text is consistent about that. And that's where we get a
good selection of our Windows worms from.
-- Jamie
John Bradford
> > any old byte sequence in a filename, it should preserve the byte
> > sequence exactly. But any program whose parser-tokeniser is scanning
> > UTF-8 is very unlikely to do that - it's just too complicated to say
> > some bits of a text stream must be remembered as literal bytes, and
> > others must be scanned as multibyte characters.
>
> So what you are saying is that conversion of invalid multibyte sequences
> into non-error wide chars followed by conversion back into UTF-8 can
> lead to trouble? *DUH*
>
> > The holes only arise because software which is interpreting UTF-8 is
> > mixed with software which isn't. That's one of the most useful
> > features of UTF-8, after all - that's why we use it for filenames.
>
> The holes only arise because software which is interpreting UTF-8 doesn't
> care to do it properly. Software that doesn't interpret it (including the
> kernel) doesn't enter the picture at all.
So is your approach to this problem that because the security issue
isn't specifically in the kernel, we shouldn't discuss it here?
Dispite the fact that there are perfectly good ways of not only
working around the issue, but preventing it from even exising, that
could be implemented in the kernel?
Filenames _are_ arbitrary strings of bytes, I.E. binary data, and that
is how they should be. I totally agree with that, (except that
obviously \0 and / have to be treated specially).
However, I don't see why it is any more logical to make the suggestion
that filenames generally be treated as UTF-8, IFF they are text at
all, than it is to suggest that filename should be arbitrary strings
of 32-bit words.
Why not:
* State that filenames are strings of 32-bit words. UCS-4 should be
the prefered format for storing text in them, but storing legacy
encodings in the low 8 bits is acceptable, (but a Bad Thing for new
installations).
* Let legacy applications store 8-bit values in those 32-bit words if
they want to, but strongly recommend only 7-bit ASCII values are
stored, not values 128-255.
* Create a divide - filesystems that support strings of 32-bit words
in their on-disk format, and those that don't. Those that don't can
simulate 32-bit words for the new functions that require them, by
padding with \0 high bytes.
* Hide all filenames with any values > 255 in them from legacy
applications, by not returning data about them in existing legacy
kernel functions.
* Introduce new routines to deal with 32-bit filenames, which Unicode
applications can use when they need to store non-ASCII, (I.E. non
7-bit).
* Note that UTF-8 stored in the low bytes is still acceptable, (but
depreciated in favour of UCS-4).
* Note that this preserves the philosophies of:
* No policy in the kernel
* Filenames are arbitrary bytestreams, (but now optionally 32-bit
ones, not just 8-bit ones!)
* Note that this is very different to my last suggestion, which was
fundamentally broken in many ways.
John.
Robin Rosenberg
>
> On Tue, 17 Feb 2004, Marc Lehmann wrote:
> >
> > Because there is a fundamental difference between file contents and
> > filenames. Filenames are supposed to be text.
>
> I think this is actually the fundamental point where we disagree.
>
> You think of filenames as something the user types in, and that is
> "readable text". And I don't.
>
> I think the filenames are just ways for a _program_ to look up stuff, and
> the human readability is a secondary thing (it's "polite", but not a
> fundamental part of their meaning).
So why don't we use an int as "filename" and why are users to "type" in
filenames? How foolish...
-- robin
Jamie Lokier
> So what you are saying is that conversion of invalid multibyte sequences
> into non-error wide chars followed by conversion back into UTF-8 can
> lead to trouble? *DUH*
Yes. (The point being that it's a common bug, just like buffer
overflows are common. Rejecting malformed UTF-8 is a defensive
strategy against it).
> > The holes only arise because software which is interpreting UTF-8 is
> > mixed with software which isn't. That's one of the most useful
> > features of UTF-8, after all - that's why we use it for filenames.
>
> The holes only arise because software which is interpreting UTF-8 doesn't
> care to do it properly.
That's right. Software which does it properly rejects malformed UTF-8.
That is the entire point of my post.
> Software that doesn't interpret it (including the kernel) doesn't
> enter the picture at all.
Yes.
Your posting merely repeated what I said, so I assume we're in agreement :)
-- Jamie
Jamie Lokier
> However, I don't see why it is any more logical to make the suggestion
> that filenames generally be treated as UTF-8, IFF they are text at
> all, than it is to suggest that filename should be arbitrary strings
> of 32-bit words.
Ok, but... why? What does 32-bit words get you that UTF-8 does not?
I can't think of a single advantage, just lots of disadvantages.
-- Jamie
vi...@parcelfarce.linux.theplanet.co.uk
> Why not:
[snip a massive pile of idiocy]
> * Note that this is very different to my last suggestion, which was
> fundamentally broken in many ways.
So is that one. To the level of Hohensee and RBJ.
*plonk*
Linus Torvalds
On Tue, 17 Feb 2004, Jamie Lokier wrote:
>
> Nope. That wouldn't help for a bundle of libraries that goes:
>
> 1. Eliminate "." and ".." components, leaving only leading ".."s.
Who does this anyway? It's wrong. It gives the wrong answer if there was a
symlink somewhere.
I remember this exact bug in gcc (I think), at some point - trying to
"optimize" the path "aa/bb/../cc" into "aa/cc" is WRONG WRONG WRONG. They
are not the same thing at all.
Any library that does the above is broken.
> 2. Reject path if it has a leading "..".
> 3. Shove it in a string with some other text and pass to other library.
>
> Next program does:
>
> 4. Extract path from string.
> 5. open ("/var/public/files/$PATH", ...)
>
> O_NODOTDOT won't protect against that.
Ok, so explain why? O_NODOTDOT will certainly guarantee that it stays
inside "/var/public/files", since it has no way to escape (modulo
symlinks/mounts, of course).
The point being that with O_NODOTDOT | O_NOMOUNT | O_NOFOLLOW, you can
just do a simple "prepend my beginning pathname" operation, and do the
open that way without having to be careful.
Then, if the thing fails, you now need to be really careful, and perhaps
do a user-space "walk one component at a time" thing to see where it
failed. But what the O_NODOTDOT | O_NOMOUNT | O_NOFOLLOW thing gave you is
that you get a fast-path for the common case (ie you don't _always_ have
to do the "walk one component at a time" crud - only if you hit a case you
might be worried about).
(Now, O_NOMOUNT isn't actually useful if you use an absolute path like the
above example - it kind of assumes that you start from pwd which woul dbe
your "safe point", and that you expect all "safe" files to be under that
one filesystem. With an absolute path, you'll clearly often end up having
to cross mount-points unless your whole thing is on the root filesystem,
which kind of makes O_NOMOUNT useless in the first place).
Btw, right now O_NOMOUNT isn't a big issue, since only root can mount
things anyway. But if we start allowing user mounts (likely with
restrictions like you can only mount if the mount-point is owned by you
and writable), O_NOMOUNT may actually become a good idea.
Linus
John Bradford
> John Bradford wrote:
> > However, I don't see why it is any more logical to make the suggestion
> > that filenames generally be treated as UTF-8, IFF they are text at
> > all, than it is to suggest that filename should be arbitrary strings
> > of 32-bit words.
>
> Ok, but... why? What does 32-bit words get you that UTF-8 does not?
> I can't think of a single advantage, just lots of disadvantages.
The advantage is that you can use them to store UCS-4.
Now, for file _contents_ this would be a compatibility disaster, which
is why UTF-8 is so convenient, but for file_names_ UCS-4 lets you
unambiguously represent any string of Unicode characters. Basically -
no more multiple representations of the same thing. No more funny
corner cases where several different strings of bytes eventually
resolve to the same name being presented to the user.
Of course, there is still the issue where the same glyphs could be
displayed on the screen for two files, for example, one called
"vertical bar", and the other one called "pipe", which is confusing,
but that's a _completely_ different issue.
As far as I can see, storing all filenames as either 7-bit ASCII or
flat UCS-4 simply eliminates the whole lot of security issues you're
thinking of. If you never use non 7-bit ASCII, legacy applications
continue to work exactly as before.
John.
John Bradford
> On Tue, Feb 17, 2004 at 08:35:22PM +0000, John Bradford wrote:
> > Why not:
>
> [snip a massive pile of idiocy]
>
> > * Note that this is very different to my last suggestion, which was
> > fundamentally broken in many ways.
>
> So is that one. To the level of Hohensee and RBJ.
Correct me if I'm wrong, but your suggestions seems to be along the
lines of:
"Sit and watch a load of buggy userspace applications get written,
content in the knowledge that it wasn't my fault, because the security
vulnerabilities aren't actually in the kernel".
John.
Jamie Lokier
> > I'd like a way to type something like "touch zöe.txt" on an ordin=
ary
> > latin1 terminal and get a UTF-8 filename in my filesystem. Thanks =
:)
>
> Then hack either bash (or whatever shell you use) or touch to do just=
that.
Hacking touch is obviously useless - I'd need to hack all the other
2000 shell utilities to get any useful behaviour.
Hacking bash -- actually readline -- is a much better idea. Then you
can enter names and they'll be created right. The only flaw in this
is that "ls" won't be useful, so that'll need to be hacked as well. etc=
.
No, I think hacking the terminal I/O is the best bet here. Then _all_
programs which currently work with UTF-8 terminals, which is rapidly
becoming most of them, will work the same with both kinds of terminal,
and the illusion of perfection will be complete and beautiful.
-- Jamie
Linus Torvalds
On Tue, 17 Feb 2004, John Bradford wrote:
>
> Why not:
I'll start with the first one. That already kills the rest.
> * State that filenames are strings of 32-bit words. UCS-4 should be
> the prefered format for storing text in them, but storing legacy
> encodings in the low 8 bits is acceptable, (but a Bad Thing for new
> installations).
UCS-4 is as braindamaged as UCS-2 was, and for all the same reasons.
It's bloated, non-expandable, and not backwards compatible.
In contrast, UTF-8 doesn't measurably expand any normal text that didn't
need it, is backwards compatible in the major ways that matter, and can be
extended arbitrarily.
UCS-4 has _zero_ advantages over UTF-8.
Please. Give it up. Anybody who thinks that _any_ other encoding format
than UTF-8 is valid is just _wrong_.
(Now, I'll give that a lot of people don't like Unicode, so I'll allow
that maybe you'd want to use the UTF-8 _encoding_scheme_ for some other
mapping, but I don't see that that is worth the pain any more. Unicode may
be a horrible enumeration, but in the end all font encodings are arbitrary
anyway, so the unicode haters might as well start giving up).
In short: even if you hate Unicode with a passion, and refuse to touch it
and think standards are worthless, you should still use the same
transformation that UTF-8 does to your idiotic character set of the day.
Because the _transform_ makes sense regardless of character set encoding.
Linus
Linus Torvalds
On Tue, 17 Feb 2004, John Bradford wrote:
> > Ok, but... why? What does 32-bit words get you that UTF-8 does not?
> > I can't think of a single advantage, just lots of disadvantages.
>
> The advantage is that you can use them to store UCS-4.
Wrong. UTF-8 can store UCS-4 characters just fine.
Admittedly you might need up to six octets for the worst case, but hey,
since you only need one for the most common case (by _far_), who cares?
And with the same UTF-8 encoding, you could some day encode UCS-8 too if
the idiotic standards bodies some day decide that 4 billion characters
isn't enough because of all the in-fighting.
> Now, for file _contents_ this would be a compatibility disaster, which
> is why UTF-8 is so convenient, but for file_names_ UCS-4 lets you
> unambiguously represent any string of Unicode characters.
Why do you think UTF-8 can't do this? Did you read some middle-aged text
written by monks in a monestary that said that UTF-8 encodes a 16-bit
character set?
> Basically - no more multiple representations of the same thing. No more
> funny corner cases where several different strings of bytes eventually
> resolve to the same name being presented to the user.
Welcome to normalized UTF-8. And realize that the "non-normalized" broken
stuff is what allows us backwards compatibility.
Of course, since you like UCS-4, you don't care about backwards
compatibility.
Linus
John Bradford
>
>
> On Tue, 17 Feb 2004, John Bradford wrote:
> >
> > Why not:
>
> I'll start with the first one. That already kills the rest.
>
> > * State that filenames are strings of 32-bit words. UCS-4 should be
> > the prefered format for storing text in them, but storing legacy
> > encodings in the low 8 bits is acceptable, (but a Bad Thing for new
> > installations).
>
> UCS-4 is as braindamaged as UCS-2 was, and for all the same reasons.
>
> It's bloated, non-expandable, and not backwards compatible.
Which I hardly see as real pain for filenames, especially as I covered
the backward compatibility bit anyway, and wanting to expand beyond
2^31 characters isn't really on my to-do list at the moment, which
just leaves filename bloat, which is laughably trivial in at least
99.9% of cases, and probably just a minor inconvenience the other
0.1%.
But, I don't think I care anymore, anyway, clearly we are going to end
up with UTF-8 filenames everywhere, and security vulnerabilities to go
with them, and as long as I'm aware of that fact, I should be OK.
John.
Alex Belits
> No, I think hacking the terminal I/O is the best bet here. Then _all_
> programs which currently work with UTF-8 terminals, which is rapidly
> becoming most of them, will work the same with both kinds of terminal,
> and the illusion of perfection will be complete and beautiful.
UTF-8 terminals (and variable-encoding terminals) alreay exist,
gnome-terminal is one of them. They are, of course, bloated pigs, but I
would rather have the bloat and idiosyncrasy in the user interface where
it belongs.
--
Alex
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> UTF-8 is dependent on Unicode, that is cumbersome, not user-expandable,
Ah, Alex, welcome back. :)
> This means, it's quite possible that this standard will be replaced
> by something better in the future
You mean like Unicode 4 will be replaced by Unicode 5 or something? :)
Seriously, if there was another standard encompassing all languages
and characters, why would they call it something different?
> and this is why poor design of Unicode is tolerated by users, and
> this is also why many people use non-Unicode-based charsets.
You've said this many times before, without explanation.
As far as I know, Unicode is a superset of all pre-existing computer
charsets used anywhere - but do feel free to correct me.
Unicode does have its problems - but what possible advantage does
_any_ known non-Unicode charset have over Unicode, apart from space saving?
You mention that Unicode doesn't well support language identification.
This is true - but the non-Unicode charsets (koi8-r etc.) don't
support that either! Or do they?
> And this is perfectly fine. Displaying and editing multilingual text is
> a user interface issue, that kernel should not be involved in.
Actually the kernel does have a line editor which needs to know a little.
> I can point at the example of this "solution" that happened years ago
> when UCS-2 was all the rage, and it got hardcoded and enforced by NTFS
> and everything that handles it. Who is laughing about that decision now?
We are all laughing ;)
-- Jamie
Jamie Lokier
> > 1. Eliminate "." and ".." components, leaving only leading ".."s.
>
> Who does this anyway? It's wrong. It gives the wrong answer if there was a
> symlink somewhere.
It's wrong for GCC, but correct for HTML/XLink relative path
resolution.
> > O_NODOTDOT won't protect against that.
>
> Ok, so explain why? O_NODOTDOT will certainly guarantee that it stays
> inside "/var/public/files", since it has no way to escape (modulo
> symlinks/mounts, of course).
Oh, I meant to say "will" but my mailer must have used the wrong
character encoding. :)
> O_NOMOUNT may actually become a good idea.
Right now, you can avoid crossing filesystems by calling lstat()
_except_ that it doesn't detect a bind mount. Among other things,
this makes it difficult to know for sure when there is only one path
to a file, because n_link==1 doesn't mean that any more. O_NOMOUNT
might be useful as a way to detect when you're crossing a bind mount.
-- Jamie
Linus Torvalds
On Tue, 17 Feb 2004, John Bradford wrote:
> >
> > Wrong. UTF-8 can store UCS-4 characters just fine.
>
> Does just fine include unambiguously?
If you don't care about backwards compatibility, then yes. You just have
to use "strict" UTF-8.
> Sure, standards-conforming
> UTF-8 is unambiguous, but you've already said time and again that that
> doesn't happen in the real world. I just don't agree on the UTF-8 can
> store UCS-4 characters just fine thing _at all_.
You get to choose between "throw the baby out with the bathwater" or "be
compatible".
Sane people choose compatibility. But it's your choice. You can always
normalize thing if you want to - but don't complain to me if it breaks
things. It will still break _fewer_ things than UCS-4 would, so even if
you always normalize you'd still be _better_ off with UTF-8 than you would
be with UCS-4.
Linus
John Bradford
>
> > > Ok, but... why? What does 32-bit words get you that UTF-8 does not?
> > > I can't think of a single advantage, just lots of disadvantages.
> >
> > The advantage is that you can use them to store UCS-4.
>
> Wrong. UTF-8 can store UCS-4 characters just fine.
Does just fine include unambiguously? Sure, standards-conforming
UTF-8 is unambiguous, but you've already said time and again that that
doesn't happen in the real world. I just don't agree on the UTF-8 can
store UCS-4 characters just fine thing _at all_.
> Admittedly you might need up to six octets for the worst case, but hey,
> since you only need one for the most common case (by _far_), who cares?
>
> And with the same UTF-8 encoding, you could some day encode UCS-8 too if
> the idiotic standards bodies some day decide that 4 billion characters
> isn't enough because of all the in-fighting.
>
> > Now, for file _contents_ this would be a compatibility disaster, which
> > is why UTF-8 is so convenient, but for file_names_ UCS-4 lets you
> > unambiguously represent any string of Unicode characters.
>
> Why do you think UTF-8 can't do this? Did you read some middle-aged text
> written by monks in a monestary that said that UTF-8 encodes a 16-bit
> character set?
At the end of the day, I just don't see how your suggestion of leaving
UTF-8 undecoded unless you're presenting it to the user is ever going
to be practical, which brings us back to my first point, that UTF-8
can't, in the real world, represent UCS-4 characters acceptably,
(I.E. unambiguously).
> > Basically - no more multiple representations of the same thing. No more
> > funny corner cases where several different strings of bytes eventually
> > resolve to the same name being presented to the user.
>
> Welcome to normalized UTF-8. And realize that the "non-normalized" broken
> stuff is what allows us backwards compatibility.
>
> Of course, since you like UCS-4, you don't care about backwards
> compatibility.
I don't particularly like UCS-4, I do care about backwards
compatibility, and addressed it right from the begining.
..and I totally don't get the bit about "non-normalised" UTF-8 being
what allows backwards compatibility. Compatibility with what!?
Existing broken implementations? Real, standards compliant UTF-8 is
fully backwards compatible with 7-bit ASCII, which is really just
about all any standard which wants to get accepted as a universal
standard can hope to be compatible with.
John.
Matthew Kirkwood
> No, I think hacking the terminal I/O is the best bet here. Then _all_
> programs which currently work with UTF-8 terminals, which is rapidly
> becoming most of them, will work the same with both kinds of terminal,
> and the illusion of perfection will be complete and beautiful.
Yep. A charset-translating tty proxy, a little like screen
or detachtty is what you want. I wonder if there's an SSH
client or server which can do that.
Matthew.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Jamie Lokier
> Your concrete example is a good one. Where did that
> filename come from? It would seem to have come from the
> keyboard via a tty (or simulator) which also had to display
> it. I'd say this is an argument for the terminal to display
> UTF-8 and convert intput into UTF-8. That is something that
> seems to be not consistantly done as yet. Ultimately it
> seems to be a responsiblity of the user interface, whether
> tty or GUI. Until that happens the shells might be able to
> fill the gap, however poorly.
Many terminals will not ever display UTF-8. Think: all the serial terminals.
This is why I think "stty utf8" or something along those lines would
be useful. The terminal itself doesn't have to talk UTF-8; however,
the applications talking with /dev/tty would always see UTF-8.
That seems to solve most of the practical user interface problems of
the command line, in one single clean place.
-- Jamie
Alex Belits
> In short: even if you hate Unicode with a passion, and refuse to touch it
> and think standards are worthless, you should still use the same
> transformation that UTF-8 does to your idiotic character set of the day.
> Because the _transform_ makes sense regardless of character set encoding.
Pretty much every charset other than Unicode does not NEED encoding
because it was already designed to work with existing system. The decision
to make the basic representation of charset full of zero bytes was the
reason that created the need for UTF-8. People who use other charsets may
not have planned for multilingual environments like they should've done,
but they aren't stupid enough to require someone to "bless" them with a
variable-length encoding.
--
Alex
Jamie Lokier
> > No, I think hacking the terminal I/O is the best bet here. Then _all_
> > programs which currently work with UTF-8 terminals, which is rapidly
> > becoming most of them, will work the same with both kinds of terminal,
> > and the illusion of perfection will be complete and beautiful.
>
> UTF-8 terminals (and variable-encoding terminals) alreay exist,
> gnome-terminal is one of them. They are, of course, bloated pigs, but I
> would rather have the bloat and idiosyncrasy in the user interface where
> it belongs.
Yes, I am using it right now. The fancy characters work well in it.
Problem is, sometimes I have to use a non-UTF-8 terminal, and I would
naturally like to access my files in the same way.
-- Jamie
Linus Torvalds
On Tue, 17 Feb 2004, Jamie Lokier wrote:
>
> > I can point at the example of this "solution" that happened years ago
> > when UCS-2 was all the rage, and it got hardcoded and enforced by NTFS
> > and everything that handles it. Who is laughing about that decision now?
>
> We are all laughing ;)
Crying. Sadly, when MS makes a whopper of a mistake (and they do it all
too often), we're left having to work with the resulting breakage.
I suspect most samba developers are already technically insane (*).
Linus
(*) Of course, since many of them are Australians, you can't tell.
Alex Belits
> > This means, it's quite possible that this standard will be replaced
> > by something better in the future
>
> You mean like Unicode 4 will be replaced by Unicode 5 or something? :)
>
> Seriously, if there was another standard encompassing all languages
> and characters, why would they call it something different?
Because the only way to do it right is to make it automatically
expandable. Unicode by its very design is monolithic with a single address
space that has to be controlled by one standard body, with no generalized
procedure for adding anything other than going through their "standard
process".
> > and this is why poor design of Unicode is tolerated by users, and
> > this is also why many people use non-Unicode-based charsets.
>
> You've said this many times before, without explanation.
How am I supposed to explain it if most of people on the list live many
thousand of kilometers from the nearest place where non-iso8859-1 charsets
are used? I have seen what people use in Russia, and it's not Unicode. Am
I supposed to bring all of you there, so you can confirm that, or should I
just point at the percentage of web pages, messages in mailing list
archives and other readily available evidence? Or do you think that this
is all irrelevant because Martin Duerst says so?
> As far as I know, Unicode is a superset of all pre-existing computer
> charsets used anywhere - but do feel free to correct me.
Unicode is not exactly a superset of all of them, but the issue is
deeper than that. There are two kinds of standards. Ones define rules of
what can be compliant, and allow different levels of implementation that
still can interoperate and be reasonably small in the minimal
implementation. Others are "this is it -- you have to implement all of
that, or you aren't compliant". Charsets are by definition the latter. But
every charset is something reasonably small, can be even _remembered_
by people who use it, and some additional things like matching input
methods, keyboard layouts, even some language-dependent processing is
within what can be written in a small piece of paper. Not so with Unicode.
No single person can even draw a Unicode font, leave alone remember what
is there. A terminal that implements Unicode has to support everything.
There is no editing/input procedure that can be applied based on the
knowledge that some text is in Unicode. It's a display-only thing, and
ridiculously bloated yet not even expandable. An example of a design made
not just by committee but by a committee of people who would never use
most of it.
> Unicode does have its problems - but what possible advantage does
> _any_ known non-Unicode charset have over Unicode, apart from space saving?
Not known, one that will be developed in the future. At this point the
demand for multilingual processing is so miniscule compared to documents
that use a single charset, the decisions that are made for this small area
of application may be completely arbitrary, and no one would notice. Same
as the situation before non-ASCII symbols appeared -- an overwhelming
majority of computer-processed text was in English, so ASCII and its
trivial extensions were tolerated despite ther inadequacies. When the real
demand will appear, Unicode will quickly show its inadequacies, and the
current "Unicode crusade" is nothing but an attempt to freeze the standard
before this will happen. Then "unicoders" will claim that "it sucks, but
it's everywhere, so live with it. It worked for Windows, right?
> You mention that Unicode doesn't well support language identification.
> This is true - but the non-Unicode charsets (koi8-r etc.) don't
> support that either! Or do they?
Other charsets don't have language identification, this is true. However
if language identification IS done, the need for unified charset
immediately disappears -- what can identify a language, can identify a
charset, it's just metadata that can be passed in-band or out of band, in
a general case. If this will be done (and it has to be done for anything
meaningful), the use of Unicode is the answer to the question that never
was asked.
Unicode is great for representation of multilingual fonts, or as an
intermediate format in conversions, but declaring it the actual format for
all data to be processed is far beyond its area of applicability. It's a
creeping functionality (closely related to creeping featurism) where
every problem looks like a nail for a standard body that standardizes on
hammers.
> > And this is perfectly fine. Displaying and editing multilingual text is
> > a user interface issue, that kernel should not be involved in.
>
> Actually the kernel does have a line editor which needs to know a little.
Kernel does not edit multilingual text unless a terminal discipline
supports it. Some limited support is possible on the text console, however
even if Unicode allows printing all characters, it's not multilingual
editing yet. Complex character-handling routines, bidirectional support
and input methods that know multiple languages still belong in userspace,
so it's at most a good reason to make a nice userspace-based support of
multilingual processing.
> > I can point at the example of this "solution" that happened years ago
> > when UCS-2 was all the rage, and it got hardcoded and enforced by NTFS
> > and everything that handles it. Who is laughing about that decision now?
>
> We are all laughing ;)
And this is why I think, it is very arrogant to claim now that UTF-8 is
not going the way of UCS-2 in the near future. Again, I am not against
letting people who want to use UCS-2, UTF-8 or UCS-4 Unicode with Klingon
in a private area, to use what they want. I am against hardcoding things
just for the purpose of making it difficult to use anything else. XML, for
example, is a great example of a standard where the model with
single charset/documenr + multiple languages/document artificially created
an unsolvable problem for non-Unicode charset users who wanted to remain
within the standard, and I see this decision as not technically justified
but arbitrary and ideologically based.
--
Alex
Linus Torvalds
On Tue, 17 Feb 2004, Jamie Lokier wrote:
>
> Many terminals will not ever display UTF-8. Think: all the serial terminals.
>
> This is why I think "stty utf8" or something along those lines would
> be useful. The terminal itself doesn't have to talk UTF-8; however,
> the applications talking with /dev/tty would always see UTF-8.
>
> That seems to solve most of the practical user interface problems of
> the command line, in one single clean place.
Doesn't "screen" already do this? I don't think you want to have the
locale handling in the kernel, along with translation of multi-key
characters (and from things like CJK terminals? I don't know what format
they send). Sounds like you should use a user-mode thing that knows about
locales...
Linus
Robin Rosenberg
> Quote from Linus Torvalds <torv...@osdl.org>:
> > On Tue, 17 Feb 2004, John Bradford wrote:
> > > > Ok, but... why? What does 32-bit words get you that UTF-8 does not?
> > > > I can't think of a single advantage, just lots of disadvantages.
> > > The advantage is that you can use them to store UCS-4.
> > Wrong. UTF-8 can store UCS-4 characters just fine.
for Unicode characters. </nitpick>
> At the end of the day, I just don't see how your suggestion of leaving
> UTF-8 undecoded unless you're presenting it to the user is ever going
> to be practical, which brings us back to my first point, that UTF-8
> can't, in the real world, represent UCS-4 characters acceptably,
> (I.E. unambiguously).
The standard say a decode should not accept invalid UTF-8 characters. Not
decoding them and just pass the garbage on is one way of not "accepting" them; i.e.
"i'm not decoding this trash". In UTF-8 seen as a byte stream this is trivial. Those that
recode to a 16-bit encoding like QT (recode them to invalid UTF-16 so it can be encoded
back to the original invalid UTF-8). That's at least what the kopete people told me. (Haven't
read the code yet). With UCS-4 or UCS-2 the decoder must reject the data or make a very
good decision. Normalizing is a very bad one, since, as we know, an invalid UTF-8 sequence
simply does not represent a unicode character.
> > > Basically - no more multiple representations of the same thing. No more
> > > funny corner cases where several different strings of bytes eventually
> > > resolve to the same name being presented to the user.
> >
> > Welcome to normalized UTF-8. And realize that the "non-normalized" broken
> > stuff is what allows us backwards compatibility.
And then (above) there is no normalized UTF-8. There are strings of valid characters
and invalid characters. Any app that tries to make sense of any garbage is a security
risk. This apples not only to UTF-8. but function like atof that decodes anything
to a double at best NaN).
-- robin
-- robin
Andries...@cwi.nl
> press the Delete key, it writes backspace-space-backspace and removes
> one byte from the input. That fails to do the right thing on UTF-8
> terminals.
Yes. I looked at that a year ago, and it should be pretty easy to make the
backspace code look more like the "delete word" code - except the "word"
is just a utf character.
I didn't care enough to really bother fixing it - the fact is, that people
who care about UTF-8 tend to have to be in graphics mode anyway, and there
is something to be said for keeping the text console simple even if it
means it lacks functionality.
But if somebody cares more than I do (hint, hint ;), I do think it should
be fixed.
> There is no fancy environment setting which corrects this problem.
> The kernel needs to know it's dealing with a UTF-8 terminal for basic
> line editing to work.
Yes. And I'd happily take patches for it.
OK - this sounds like a good moment to come with such patches.
The below is essentially a five-year-old patch by Bruno Haible.
It introduces utf8 mode, and the effect it has on erasing input characters.
(For the console more is needed, there are a few more patches there.)
Andries
P.S. Is there a reason for the different definition of
IMAXBEL in asm-parisc/termbits.h, or was that a typo?
If there is a reason, I think a comment is in order.
-----
diff -uprN -X /linux/dontdiff a/drivers/char/n_tty.c b/drivers/char/n_tty.c
--- a/drivers/char/n_tty.c 2003-12-18 03:58:04.000000000 +0100
+++ b/drivers/char/n_tty.c 2004-02-17 23:25:36.000000000 +0100
@@ -172,6 +172,16 @@ ssize_t n_tty_chars_in_buffer(struct tty
return n;
}
+static inline int is_utf8_continuation(unsigned char c)
+{
+ return (c & 0xc0) == 0x80;
+}
+
+static inline int is_continuation(unsigned char c, struct tty_struct *tty)
+{
+ return I_IUTF8(tty) && is_utf8_continuation(c);
+}
+
/*
* Perform OPOST processing. Returns -1 when the output device is
* full and the character must be retried.
@@ -226,7 +236,7 @@ static int opost(unsigned char c, struct
default:
if (O_OLCUC(tty))
c = toupper(c);
- if (!iscntrl(c))
+ if (!iscntrl(c) && !is_continuation(c, tty))
tty->column++;
break;
}
@@ -330,7 +340,7 @@ static inline void finish_erasing(struct
static void eraser(unsigned char c, struct tty_struct *tty)
{
enum { ERASE, WERASE, KILL } kill_type;
- int head, seen_alnums;
+ int head, seen_alnums, cnt;
unsigned long flags;
if (tty->read_head == tty->canon_head) {
@@ -368,8 +378,18 @@ static void eraser(unsigned char c, stru
seen_alnums = 0;
while (tty->read_head != tty->canon_head) {
- head = (tty->read_head - 1) & (N_TTY_BUF_SIZE-1);
- c = tty->read_buf[head];
+ head = tty->read_head;
+
+ /* erase a single possibly multibyte character */
+ do {
+ head = (head - 1) & (N_TTY_BUF_SIZE-1);
+ c = tty->read_buf[head];
+ } while (is_continuation(c, tty) && head != tty->canon_head);
+
+ /* do not partially erase */
+ if (is_continuation(c, tty))
+ break;
+
if (kill_type == WERASE) {
/* Equivalent to BSD's ALTWERASE. */
if (isalnum(c) || c == '_')
@@ -377,9 +397,10 @@ static void eraser(unsigned char c, stru
else if (seen_alnums)
break;
}
+ cnt = (tty->read_head - head) & (N_TTY_BUF_SIZE-1);
spin_lock_irqsave(&tty->read_lock, flags);
tty->read_head = head;
- tty->read_cnt--;
+ tty->read_cnt -= cnt;
spin_unlock_irqrestore(&tty->read_lock, flags);
if (L_ECHO(tty)) {
if (L_ECHOPRT(tty)) {
@@ -388,7 +409,12 @@ static void eraser(unsigned char c, stru
tty->column++;
tty->erasing = 1;
}
+ /* if cnt > 1, output a multi-byte character */
echo_char(c, tty);
+ while (--cnt > 0) {
+ head = (head+1) & (N_TTY_BUF_SIZE-1);
+ put_char(tty->read_buf[head], tty);
+ }
} else if (kill_type == ERASE && !L_ECHOE(tty)) {
echo_char(ERASE_CHAR(tty), tty);
} else if (c == '\t') {
@@ -403,7 +429,7 @@ static void eraser(unsigned char c, stru
else if (iscntrl(c)) {
if (L_ECHOCTL(tty))
col += 2;
- } else
+ } else if (!is_continuation(c, tty))
col++;
tail = (tail+1) & (N_TTY_BUF_SIZE-1);
}
diff -uprN -X /linux/dontdiff a/include/asm-alpha/termbits.h b/include/asm-alpha/termbits.h
--- a/include/asm-alpha/termbits.h 2003-12-18 03:59:05.000000000 +0100
+++ b/include/asm-alpha/termbits.h 2004-02-17 23:18:21.000000000 +0100
@@ -56,12 +56,10 @@ struct termios {
#define ICRNL 0000400
#define IXON 0001000
#define IXOFF 0002000
-#if !defined(KERNEL) || defined(__USE_BSD)
- /* POSIX.1 doesn't want these... */
-# define IXANY 0004000
-# define IUCLC 0010000
-# define IMAXBEL 0020000
-#endif
+#define IXANY 0004000
+#define IUCLC 0010000
+#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-arm/termbits.h b/include/asm-arm/termbits.h
--- a/include/asm-arm/termbits.h 2003-12-18 04:00:00.000000000 +0100
+++ b/include/asm-arm/termbits.h 2004-02-17 23:12:29.000000000 +0100
@@ -49,6 +49,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-arm26/termbits.h b/include/asm-arm26/termbits.h
--- a/include/asm-arm26/termbits.h 2003-12-18 03:59:04.000000000 +0100
+++ b/include/asm-arm26/termbits.h 2004-02-17 23:22:31.000000000 +0100
@@ -49,6 +49,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-cris/termbits.h b/include/asm-cris/termbits.h
--- a/include/asm-cris/termbits.h 2003-12-18 03:58:16.000000000 +0100
+++ b/include/asm-cris/termbits.h 2004-02-17 23:23:05.000000000 +0100
@@ -53,6 +53,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-h8300/termbits.h b/include/asm-h8300/termbits.h
--- a/include/asm-h8300/termbits.h 2003-12-18 03:59:58.000000000 +0100
+++ b/include/asm-h8300/termbits.h 2004-02-17 23:21:34.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-i386/termbits.h b/include/asm-i386/termbits.h
--- a/include/asm-i386/termbits.h 2003-12-18 03:58:40.000000000 +0100
+++ b/include/asm-i386/termbits.h 2004-02-17 23:20:59.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-ia64/termbits.h b/include/asm-ia64/termbits.h
--- a/include/asm-ia64/termbits.h 2004-02-05 19:55:21.000000000 +0100
+++ b/include/asm-ia64/termbits.h 2004-02-17 23:10:43.000000000 +0100
@@ -60,6 +60,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-m68k/termbits.h b/include/asm-m68k/termbits.h
--- a/include/asm-m68k/termbits.h 2003-12-18 03:59:46.000000000 +0100
+++ b/include/asm-m68k/termbits.h 2004-02-17 23:11:54.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-mips/termbits.h b/include/asm-mips/termbits.h
--- a/include/asm-mips/termbits.h 2003-12-18 03:58:08.000000000 +0100
+++ b/include/asm-mips/termbits.h 2004-02-17 23:10:02.000000000 +0100
@@ -77,6 +77,7 @@ struct termios {
#define IXANY 0004000 /* Any character will restart after stop. */
#define IXOFF 0010000 /* Enable start/stop input control. */
#define IMAXBEL 0020000 /* Ring bell when input queue is full. */
+#define IUTF8 0040000 /* Input is UTF8 */
/* c_oflag bits */
#define OPOST 0000001 /* Perform output processing. */
diff -uprN -X /linux/dontdiff a/include/asm-parisc/termbits.h b/include/asm-parisc/termbits.h
--- a/include/asm-parisc/termbits.h 2003-12-18 03:58:56.000000000 +0100
+++ b/include/asm-parisc/termbits.h 2004-02-17 23:07:35.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0040000
+#define IUTF8 0100000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-ppc/termbits.h b/include/asm-ppc/termbits.h
--- a/include/asm-ppc/termbits.h 2003-12-18 03:58:57.000000000 +0100
+++ b/include/asm-ppc/termbits.h 2004-02-17 23:11:14.000000000 +0100
@@ -58,6 +58,7 @@ struct termios {
#define IXANY 0004000
#define IUCLC 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-ppc64/termbits.h b/include/asm-ppc64/termbits.h
--- a/include/asm-ppc64/termbits.h 2003-12-18 03:59:04.000000000 +0100
+++ b/include/asm-ppc64/termbits.h 2004-02-17 23:09:02.000000000 +0100
@@ -66,6 +66,7 @@ struct termios {
#define IXANY 0004000
#define IUCLC 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-s390/termbits.h b/include/asm-s390/termbits.h
--- a/include/asm-s390/termbits.h 2003-12-18 03:59:29.000000000 +0100
+++ b/include/asm-s390/termbits.h 2004-02-17 23:20:26.000000000 +0100
@@ -59,6 +59,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-sh/termbits.h b/include/asm-sh/termbits.h
--- a/include/asm-sh/termbits.h 2003-12-18 03:58:18.000000000 +0100
+++ b/include/asm-sh/termbits.h 2004-02-17 23:23:40.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-sparc/termbits.h b/include/asm-sparc/termbits.h
--- a/include/asm-sparc/termbits.h 2003-12-18 03:59:20.000000000 +0100
+++ b/include/asm-sparc/termbits.h 2004-02-17 23:13:13.000000000 +0100
@@ -78,6 +78,7 @@ struct termios {
#define IXANY 0x00000800
#define IXOFF 0x00001000
#define IMAXBEL 0x00002000
+#define IUTF8 0x00004000
/* c_oflag bits */
#define OPOST 0x00000001
diff -uprN -X /linux/dontdiff a/include/asm-sparc64/termbits.h b/include/asm-sparc64/termbits.h
--- a/include/asm-sparc64/termbits.h 2003-12-18 03:58:18.000000000 +0100
+++ b/include/asm-sparc64/termbits.h 2004-02-17 23:22:06.000000000 +0100
@@ -80,6 +80,7 @@ struct termios {
#define IXANY 0x00000800
#define IXOFF 0x00001000
#define IMAXBEL 0x00002000
+#define IUTF8 0x00004000
/* c_oflag bits */
#define OPOST 0x00000001
diff -uprN -X /linux/dontdiff a/include/asm-v850/termbits.h b/include/asm-v850/termbits.h
--- a/include/asm-v850/termbits.h 2003-12-18 03:58:04.000000000 +0100
+++ b/include/asm-v850/termbits.h 2004-02-17 23:23:25.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/asm-x86_64/termbits.h b/include/asm-x86_64/termbits.h
--- a/include/asm-x86_64/termbits.h 2003-12-18 03:58:08.000000000 +0100
+++ b/include/asm-x86_64/termbits.h 2004-02-17 23:06:05.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
#define IXANY 0004000
#define IXOFF 0010000
#define IMAXBEL 0020000
+#define IUTF8 0040000
/* c_oflag bits */
#define OPOST 0000001
diff -uprN -X /linux/dontdiff a/include/linux/tty.h b/include/linux/tty.h
--- a/include/linux/tty.h 2003-12-18 03:58:49.000000000 +0100
+++ b/include/linux/tty.h 2004-02-17 23:04:31.000000000 +0100
@@ -200,6 +200,7 @@ struct tty_flip_buffer {
#define I_IXANY(tty) _I_FLAG((tty),IXANY)
#define I_IXOFF(tty) _I_FLAG((tty),IXOFF)
#define I_IMAXBEL(tty) _I_FLAG((tty),IMAXBEL)
+#define I_IUTF8(tty) _I_FLAG((tty),IUTF8)
#define O_OPOST(tty) _O_FLAG((tty),OPOST)
#define O_OLCUC(tty) _O_FLAG((tty),OLCUC)
Linus Torvalds
On Tue, 17 Feb 2004 Andries...@cwi.nl wrote:
>
> OK - this sounds like a good moment to come with such patches.
> The below is essentially a five-year-old patch by Bruno Haible.
> It introduces utf8 mode, and the effect it has on erasing input characters.
> (For the console more is needed, there are a few more patches there.)
Heh. Looks fine and "obviously correct" by me. Can people verify that it
works for them? (I'd rather not put it in now, but I can put in the first
2.6.4-rc kernel).
> P.S. Is there a reason for the different definition of
> IMAXBEL in asm-parisc/termbits.h, or was that a typo?
> If there is a reason, I think a comment is in order.
I think it's a typo. But I've never used HP-UX, maybe it tries to be
binary compatible and there's some other HP-UX bit in between those
values?
Linus
John Bradford
> normalize thing if you want to - but don't complain to me if it breaks
> things. It will still break _fewer_ things than UCS-4 would, so even if
> you always normalize you'd still be _better_ off with UTF-8 than you would
> be with UCS-4.
Well, if all the UTF-8 diddling is eventually done by glibc, or some
other library, it might just be made to work.
The keep-it-in-UTF-8-all-the-time thing will still break down when a
user inputs a filename by copying the display of a badly encoded
filename using GPM, or in X, but that isn't a kernel issue.
I still don't really get what enforcing strictly standards compliant
UTF-8 has to do with backwards compatibility, though.
_But_ at least I'm about 5% more confident that filenames won't
suddenly blow up in my face, so I can sleep soundly tonight :-).
John.
Hans Reiser
and found by those associations. Some of those files will consist of
ascii keywords, some will be icon images, etc..... Human readability
should not be considered fundamental to a name component, especially
since programs with no interest in readability may be the only direct
users of the name.
Hans
Robin Rosenberg
> ReiserFS 6 plans to allow files to be associated with arbitrary files
> and found by those associations. Some of those files will consist of
> ascii keywords, some will be icon images, etc..... Human readability
> should not be considered fundamental to a name component, especially
> since programs with no interest in readability may be the only direct
> users of the name.
If the user never sees a name, it doesn't matter. However the user actually sees
and reads the filenames in /home, portable media, networks devices and lots of
places. However, when a user has named a component those characters are those
that are important to the user because those form an "image" (since you introduced
the term) or "sound" that the user remembers and associates with the content. A
character is the simplest form of image so it should always look the same.
-- robin
H. Peter Anvin
By author: Matthias Urlichs <sm...@smurf.noris.de>
In newsgroup: linux.dev.kernel
>
> Not locale, but normalization problems and identical-glyph problems.
>
> Which is actually worse, because you don't have filenames which look
> like crap -- instead you have filenames which look perfectly sane, bu=
t
> they still do not work. Example: is an á one character, or is it=
an a
> followed by a composing ÂŽ?
>
> Mac OSX, just as an example, only uses decomposed filenames. I don't =
know
> the current situation, but 10.2 has major problems when you try to ac=
cess
> files with composite characters in their name (across NFS for instanc=
e).
>
> I wonder if Linux, i.e. Linus ;-) should decree one single standard
> normalization. (I am NOT saying that enforcing this would be the kern=
el's
> job!)
>
I believe that for most applications, normalization form C should be
used.
However, I suspect there are some applications for which this would
not apply.
-hpa
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
H. Peter Anvin
By author: bert hubert <a...@ds9a.nl>
In newsgroup: linux.dev.kernel
>
> Additional good news is that following octets in a utf-8 character sequence
> always have the highest order bit set, precluding / or \x0 from appearing,
> confusing the kernel.
>
Indeed. The original name for the encoding was, in fact, "FSS-UTF",
for "filesystem safe Unicode transformation format."
> The remaining zit is that all these represent '..':
> 2E 2E
> C0 AE C0 AE
> E0 80 AE E0 80 AE
> F0 80 80 AE F0 80 80 AE
> F8 80 80 80 AE F8 80 80 80 AE
> FC 80 80 80 80 AE FC 80 80 80 80 AE
No, they don't.
The first represent "..", the remaining two are illegal encodings and
do not decode to anything.
Those of us who have been involved with the issue have fought
*extremely* hard against DWIM decoders which try to decode the latter
sequences into ".." -- it's incorrect, and a security hazard. The
only acceptable decodings is to throw an error, or use an out-of-band
encoding mechanism to denote "bad bytecode."
> This in itself is not a problem, the kernel will only recognize 2E 2E as the
> real .., but it does show that 'document.doc' might be encoded in a myriad
> ways.
No, it doesn't.
> So some guidance about using only the simplest possible encoding might be
> sensible, if we don't want the kernel to know about utf-8.
UTF-8 requires the use of the shortest possible encoding. An
application which doesn't obey that and tries to be "smart" is a
security hazard.
It is a bit unfortunate that the encoding don't exclude these by
design as opposed by error checking; it makes it a little too easy for
clueless programmers to skip :(
-hpa
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
H. Peter Anvin
By author: Jamie Lokier <ja...@shareable.org>
In newsgroup: linux.dev.kernel
>
> Linus Torvalds wrote:
> > Which flies in the face of "Be strict in what you generate, be libe=
ral in
> > what you accept". A lot of the functions are _not_ willing to be li=
beral
> > in what they accept. Which sometimes just makes the problem worse, =
for no
> > good reason.
>
> Unicode specifies that a program claiming to read UTF-8 _must_ reject
> malformed UTF-8.
>
> Ok, we can just ignore Unicode. :)
>
> But the reason they cite is security: when applications allow
> malformed UTF-8 through, there's plenty of scope for security holes
> due to multiple encodings of "/" and "." and "\0".
>
> This is a real problem: plenty of those Windows worms that attack web
> servers get in by using multiple-escaped funny characters and
> malformed UTF-8 to get past security checks for ".." and such.
>
Actually, the kernel is 100% compliant in that respect.
The only byte sequences the kernel interpret:
00
2E
2E 2E
2F
.. and it correctly rejects (in the sense that it doesn't alias) any
other possible byte stream that could be interpreted as the same
sequences by a naïvely incorrect UTF-8 encoder.
-hpa
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"=
H. Peter Anvin
By author: Linus Torvalds <torv...@osdl.org>
In newsgroup: linux.dev.kernel
>
> UCS-4 is as braindamaged as UCS-2 was, and for all the same reasons.
>
> It's bloated, non-expandable, and not backwards compatible.
>
UCS-4 is actually a very nice format for *internal processing*. For
data interchange, it sucks eggs.
UCS-2 is historic. It's successor, UTF-16, is one of the worst
horrors ever inflicted on mankind by Microsoft.
-hpa