Ruby UTF-8
Peter C
*.rb files with notepad using UTF-8 but I can't run them with Ruby.
This is what happens when I try to run it.
c:\> ruby -Ku myFile.rb
jpn.rb:1: undefined method `' for main:Object
(NoMethodError)
Am I doing something wrong?
My goal is the read/write strings (containing Japanese characters)
from a web browser. Is there a recommend way of doing this?
Peter
Wolfgang Nádasi-Donner
Wolfgang Nádasi-Donner
won...@donnerweb.de
"Peter C" <pkc...@gmail.com> schrieb im Newsbeitrag
news:2c9220c5.05031...@posting.google.com...
The Windows-Editor writes always a "Byte Order Mark" (BOM) at the beginning
of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded file begins with
"EF BB BF" (hex). These non-characters should usually be ignored (for more
information see http://www.unicode.org/).
One possibility is to remove the first three bytes of the UTF-8 encoded file
using some filter program or a hex editor. You should do this into a copy,
because the Windows editor cannot work correctly on this changed data :-((
Florian Gross
> "Peter C" <pkc...@gmail.com> schrieb im Newsbeitrag
> news:2c9220c5.05031...@posting.google.com...
>
>>I'm working with Japanese character sets in Windows. I can save my
>>*.rb files with notepad using UTF-8 but I can't run them with Ruby.
>>This is what happens when I try to run it.
>>
>>c:\> ruby -Ku myFile.rb
>>jpn.rb:1: undefined method `' for main:Object
>>(NoMethodError)
>>
> The Windows-Editor writes always a "Byte Order Mark" (BOM) at the beginning
> of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded file begins with
> "EF BB BF" (hex). These non-characters should usually be ignored (for more
> information see http://www.unicode.org/).
>
> One possibility is to remove the first three bytes of the UTF-8 encoded file
> using some filter program or a hex editor. You should do this into a copy,
> because the Windows editor cannot work correctly on this changed data :-((
Another one is to have an assignment to a scratch variable at the
beginning of the script. Ruby will parse the BOM as the part of the
variable name and thus not complain about it.
I've posted this to ruby-core a few months ago and would like to see it
fixed...
Nikolai Weibull
> > I'm working with Japanese character sets in Windows. I can save my
> > *.rb files with notepad using UTF-8 but I can't run them with Ruby.
> The Windows-Editor writes always a "Byte Order Mark" (BOM) at the
> beginning of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded
> file begins with "EF BB BF" (hex). These non-characters should usually
> be ignored (for more information see http://www.unicode.org/).
Why does it write a BOM for UTF-8 encoded files? It's utterly
meaningless to discuss byte order for UTF-8 encoded text,
nikolai
--
::: name: Nikolai Weibull :: aliases: pcp / lone-star / aka :::
::: born: Chicago, IL USA :: loc atm: Gothenburg, Sweden :::
::: page: minimalistic.org :: fun atm: gf,lps,ruby,lisp,war3 :::
main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}
Wolfgang Nádasi-Donner
Wolfgang Nádasi-Donner
won...@donnerweb.de
"Nikolai Weibull" <mailing-lis...@rawuncut.elitemail.org> schrieb
im Newsbeitrag news:2005032014...@puritan.pcp.ath.cx...
> * Wolfgang N?dasi-Donner (Mar 15, 2005 19:10):
Simply said, because it is allowed by the Unicode Standard.
I assume that Microsoft uses it because the Notepad can decide which
Encoding is used in existing data. This means, that one cannot edit UTF-8
encoded Data using Notepad, if there is no appropriate BOM.
Florian Gross
>>>I'm working with Japanese character sets in Windows. I can save my
>>>*.rb files with notepad using UTF-8 but I can't run them with Ruby.
>
>>The Windows-Editor writes always a "Byte Order Mark" (BOM) at the
>>beginning of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded
>>file begins with "EF BB BF" (hex). These non-characters should usually
>>be ignored (for more information see http://www.unicode.org/).
>
> Why does it write a BOM for UTF-8 encoded files? It's utterly
> meaningless to discuss byte order for UTF-8 encoded text,
So that it can identify the file as UTF-8 encoded in the future without
having to guess based on byte count, I assume.
I think that that behavior makes sense and would like to see it
supported in Ruby.
Lothar Scholz
FG> Nikolai Weibull wrote:
>>>>I'm working with Japanese character sets in Windows. I can save my
>>>>*.rb files with notepad using UTF-8 but I can't run them with Ruby.
>>
>>>The Windows-Editor writes always a "Byte Order Mark" (BOM) at the
>>>beginning of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded
>>>file begins with "EF BB BF" (hex). These non-characters should usually
>>>be ignored (for more information see http://www.unicode.org/).
>>
>> Why does it write a BOM for UTF-8 encoded files? It's utterly
>> meaningless to discuss byte order for UTF-8 encoded text,
FG> So that it can identify the file as UTF-8 encoded in the future without
FG> having to guess based on byte count, I assume.
FG> I think that that behavior makes sense and would like to see it
FG> supported in Ruby.
Doesn't ruby CVS already do this ?
I thought i read something about this on the ruby core list.
If there is no BOM i would really recommend the way python is using
for specifying font encodings, it's simple and excellent.
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's
Florian Gross
> FG> So that it can identify the file as UTF-8 encoded in the future without
> FG> having to guess based on byte count, I assume.
>
> FG> I think that that behavior makes sense and would like to see it
> FG> supported in Ruby.
>
> Doesn't ruby CVS already do this ?
> I thought i read something about this on the ruby core list.
No idea, I'm on 1.8.2.
Sam Roberts
> * Wolfgang N???dasi-Donner (Mar 15, 2005 19:10):
> > > I'm working with Japanese character sets in Windows. I can save my
> > > *.rb files with notepad using UTF-8 but I can't run them with Ruby.
>
> > The Windows-Editor writes always a "Byte Order Mark" (BOM) at the
> > beginning of UTF-8/16LE/16BE coded files. In this case a UTF-8 coded
> > file begins with "EF BB BF" (hex). These non-characters should usually
> > be ignored (for more information see http://www.unicode.org/).
>
> Why does it write a BOM for UTF-8 encoded files? It's utterly
> meaningless to discuss byte order for UTF-8 encoded text,
It's a tag to indicate the data is UTF-8. If it wasn't there it could be
anything, iso-8859-*, koi8, ...
Apple does this in NSString when I convert UCS-2 data to UTF-8, too. It
confused me. I still don't like it, but I have a suspicion that its
partly because these OSes have a legacy "assumed encoding" for 8-bit
text, and that it isn't UTF-8.
In a way it means that there are two "dialects" of utf-8. This is
unfortunate.
Sam
Christian Neukirchen
To what extent do BOMs interfere with shebang-lines?
--
Christian Neukirchen <chneuk...@gmail.com> http://chneukirchen.org
Nikolai Weibull
> To what extent do BOMs interfere with shebang-lines?
They can't coexist, unless the operating-system deals with a BOM
appropriately. Linux doesn't, for one,
Sam Roberts
They are the first characters in the file, so the first two characters
are not # and !.
Cheers,
Sam
Florian Gross
> Quoting chneuk...@gmail.com, on Mon, Mar 21, 2005 at 01:45:57AM +0900:
>
>>Florian Gross <fl...@ccan.de> writes:
>>To what extent do BOMs interfere with shebang-lines?
Are you really sure I wrote that? ;)