Error in Handling Unicode(UTF16-LE) File & String
iaminsik
But, I want to handle utf-16le files directly.
My first source is "read a line from utf-16le file and write it in
utf-16le encoding".
It works well.
==========================================================
use utf8;
use Encode;
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
print $outfile $line;
}
close($infile);
close($outfile);
==========================================================
Second source is "read one line, split it into array, and print array
by line in utf-16le encoding".
It seemed to work well, but some characters were broken. It didn't
work well.
After a long web searching, I recognized Unicode::String could solve
this problem.
==========================================================
use utf8;
use Encode;
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
chomp($line);
@words = split(/[ ]+/, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
==========================================================
Using Unicode::String, I made the third source, but still it doesn't
work.
It means "reading" is OK, but split function isn't.
Is there any solution?
==========================================================
use utf8;
use Encode;
use Unicode::String;
Unicode::String->stringify_as('utf16');
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $infile;
open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
binmode $outfile;
while ($line = <$infile>)
{
chomp($line);
$sep = new Unicode::String ("[ ]+");
@words = split($sep, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
==========================================================
Best Regards.
Remi
Ben Bullock
> In most cases, I converted utf-16le files into utf-8 encoding. But, I
> want to handle utf-16le files directly.
>
> My first source is "read a line from utf-16le file and write it in
> utf-16le encoding".
> It works well.
No it doesn't. Your problems are all in the first file.
> open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
> binmode $infile;
> open ($outfile, ">:raw:encoding(UTF-16LE):crlf", "unicodefile.out");
> binmode $outfile;
Do you know what binmode does? You'd better have another look at the
manual (perldoc -f binmode). The binmode statements here switch OFF all
the :raw:encoding(UTF... stuff you'd put in the previous lines, which
explains all the other problems you had.
To demonstrate, try this:
#!/usr/local/bin/perl
use warnings;
use strict;
use utf8;
use Encode;
binmode STDOUT, "utf8";
my $utf8 = "モンスター 自惚れ";
for (qw/file1 file2/) {
open (my $outfile, ">:raw:encoding(UTF-16LE):crlf", "$_.dat") or die
$!;
binmode $outfile if /1/; # do what you did for file1 only
print $outfile $utf8;
close $outfile or die $!;
open (my $infile, "<:encoding(UTF-16LE):crlf", "$_.dat");
while (my $line = <$infile>)
{
print "$_: $line\n";
}
close($infile) or die $!;
}
The reason your code appeared to work is because you never did anything
with the data. It was actually just reading and writing it as bytes
without any knowledge of the encoding. As soon as you tried to manipulate
the data, the problem which had been there all along became visible.
P.S. use warnings; use strict; & check the values of open and close as
above.
Ben Bullock
> open (my $infile, "<:encoding(UTF-16LE):crlf", "$_.dat");
> P.S. use warnings; use strict; & check the values of open and close as
> above.
Oops!
iaminsik
Reading your reply, I could afford to understand 'binmode' slightly.
I modified my source in two ways.
1. First Source
========================================================================
no warnings;
use utf8;
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
binmode $outfile;
{
chomp($line);
@words = split(/[ ]+/, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
The first source generates 'wide character warnings',
and saves outfile in utf8 format, weirdly.
2. Second Source
========================================================================
no warnings;
use utf8;
$\ = "\n";
open ($infile, "<:encoding(UTF-16LE):crlf", "unicodefile.dat");
while ($line = <$infile>)
{
chomp($line);
@words = split(/[ ]+/, $line);
foreach $word (@words)
{
print $outfile $word;
}
}
close($infile);
close($outfile);
I made 'binmode $outfile;' as a comment line,
and it saves outfile in UTF-16LE format I wanted.
3. Several Questions
I, a newbie in Perl programming language, couldn't understand two
parts in your codes.
========================================================================
for (qw/file1 file2/) { <===== what it means? it's a short expression
for loop?
binmode $outfile if /1/; <===== what /l/ means?
========================================================================
Best Regards for your help.
Remi.
Ben Bullock
> The first source generates 'wide character warnings',
> and saves outfile in utf8 format, weirdly.
It's not weird; you have "use utf8;" there, so it reads in using the
encoding you specified, then the binmode switches off the output
formatting, then it prints it out in the default format, which
generates wide character warnings because you haven't explicitly set
the mode of the output to anything. Use
binmode $outfile,"utf8";
to switch those wide character warnings off.
> I made 'binmode $outfile;' as a comment line,
> and it saves outfile in UTF-16LE format I wanted.
Good news.
> 3. Several Questions
> I, a newbie in Perl programming language, couldn't understand two
> parts in your codes.
> ========================================================================
> for (qw/file1 file2/) { <===== what it means? it's a short expression
> for loop?
This sets $_ to "file1" then "file2". qw/a b/ equals ('a', 'b').
> binmode $outfile if /1/; <===== what /l/ means?
It's not an l it's a 1. "if /1/" has the effect of saying 'if $_ is
"file1"'. The /1/ detects the character '1' in the name. Try
experimenting with the code to understand what it does.
iaminsik
Your comment helped me a lot.
Thanks, Ben!
Best Regards,
Remi.