[perl #69678] require of UTF-16LE is sensitive to newlines
Nicholas Clark
# Please include the string: [perl #69678]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=69678 >
This is a bug report for perl from ni...@ccl4.org,
generated with the help of perlbug 1.39 running under perl 5.11.0.
-----------------------------------------------------------------
[Please describe your issue here]
It seems that do/require of a UTF-16 LE file is sensitive to it ending with
a newline. UTF-16 BE isn't. This is strange.
Consider this test script:
$ cat utf.pl
#!perl -w
use strict;
my ($enc, $tag, $bom, $nl) = @ARGV;
open UTF_PL, ">:raw:encoding($enc)", "utf$$.pl"
or die "utf.pl($enc,$tag,$bom): $!";
print UTF_PL chr 0xFEFF if $bom;
print UTF_PL $tag;
print UTF_PL "\n" if $nl;
close UTF_PL;
$! = 0;
my $got = do "./utf$$.pl";
print defined $got ? "Got '$got'\n" : "Got undef: $!\n";
__END__
With LE, we need the newline, else do fails:
$ ./perl -Ilib utf.pl utf16le 123 bom nl
Got '123'
$ ./perl -Ilib utf.pl utf16le 123 bom
Got undef:
$ ./perl -Ilib utf.pl utf16le 123 '' nl
Got '123'
$ ./perl -Ilib utf.pl utf16le 123 ''
Got undef:
With BE, the newline is irrelevant.
$ ./perl -Ilib utf.pl utf16be 123 bom nl
Got '123'
$ ./perl -Ilib utf.pl utf16be 123 bom
Got '123'
$ ./perl -Ilib utf.pl utf16be 123 '' nl
Got '123'
$ ./perl -Ilib utf.pl utf16be 123 ''
Got '123'
(I found this whilst refactoring t/comp/utf.t
It's trivial to augment that to make regression tests for this)
Nicholas Clark
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
Site configuration information for perl 5.11.0:
Configured by nick at Fri Oct 9 04:07:23 BST 2009.
Summary of my perl5 (revision 5 version 11 subversion 0) configuration:
Commit id: 0d013e67f5a10da66fa68e1c542628a0749435b4
Platform:
osname=linux, osvers=2.6.18-xenu, archname=x86_64-linux-thread-multi
uname='linux zazen 2.6.18-xenu #1 smp thu oct 4 12:23:41 bst 2007 x86_64 gnulinux '
config_args='-Dusedevel=y -Dcc=ccache g++ -Dld=g++ -Ubincompat5005 -Uinstallusrbinperl -Dcf_email=ni...@ccl4.org -Dperladmin=ni...@ccl4.org -Dinc_version_list= -Dinc_version_list_init=0 -Doptimize=-Os -Dusethreads -Duse64bitall -Uusemymalloc -Duseperlio -Dprefix=~/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6 -Uusevendorprefix -Uvendorprefix=~/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6 -Dinstallman1dir=none -Dinstallman3dir=none -Uuserelocatableinc -de'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='ccache g++', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-Os',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.3.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc.so.6, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.7'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -Os -L/usr/local/lib -fstack-protector'
Locally applied patches:
---
@INC for perl 5.11.0:
lib
/home/nick/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6/lib/perl5/site_perl/5.11.0/x86_64-linux-thread-multi
/home/nick/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6/lib/perl5/site_perl/5.11.0
/home/nick/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6/lib/perl5/5.11.0/x86_64-linux-thread-multi
/home/nick/Sandpit/snap5.9.x-v5.11.0-75-g0d013e6/lib/perl5/5.11.0
.
---
Environment for perl 5.11.0:
HOME=/home/nick
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/home/nick/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/local/sbin:/sbin:/usr/sbin
PERL_BADLANG (unset)
SHELL=/bin/bash
Nicholas Clark
I should be looking at $@
$ ./perl -Ilib utf.pl utf16le 123 bom
Got undef: panic: utf16_to_utf8: odd bytelen 5.
What is going wrong is that S_swallow_bom() simply passes in whatever it
first gets to utf16_to_utf8_reversed(), without considering the possibility
that it read an number of bytes. That -1 is wrong - it's a kludge that
represents the fact that usually the line is read with an octet of 10 at the
end, which sv_gets() assumes is a NL character in its own right, but is
actually the first half of "\n\0" in UTF16-LE:
filter_add(utf16rev_textfilter, NULL);
Newx(news, (PL_bufend - (char*)s) * 3 / 2 + 1, U8);
utf16_to_utf8_reversed(s, news,
PL_bufend - (char*)s - 1,
&newlen);
Fixing this properly involves making utf16rev_textfilter() correctly carry
over partial reads between invocations.
I infer currently (not tested) that the UTF16-LE filter is broken for any
input with character above 255. (It will be running with input off by one,
and hence its byte transposing relies on the high byte of every character
being the same - zero)
Nicholas Clark
Vincent Pit via RT
http://perl5.git.perl.org/perl.git/commit/c28d61051c446453c532f387d478df78d6f95c55.