[perl #37366] Space character in -F switch
Steve Peters via RT
>
> This is a bug report for perl from kra...@telus.net,
> generated with the help of perlbug 1.35 running under perl v5.8.6.
>
>
> -----------------------------------------------------------------
> [Please enter your report here]
>
>
> A space character in a character class with the -F switch 'eats' the
> following
> characters in the class.
>
> $ perl -F'[ ]' -lane'print for @F'
> Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
> $ perl -F'[;#!%^&* ]' -lane'print for @F'
> Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE ;#!%^&*/.
> $ perl -F'[ ;#!%^&*]' -lane'print for @F'
> Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
>
Looking at the code in perl.c, a space cannot be used in a -F pattern.
case 'F':
PL_minus_F = TRUE;
PL_splitstr = ++s;
while (*s && !isSPACE(*s)) ++s;
*s = '\0';
PL_splitstr = savepv(PL_splitstr);
return s;
This is a significant limitation of the -F switch and should be
documented, if not changed to all a space within a "" or //.
Rafael Garcia-Suarez
> Looking at the code in perl.c, a space cannot be used in a -F pattern.
>
> case 'F':
> PL_minus_F = TRUE;
> PL_splitstr = ++s;
> while (*s && !isSPACE(*s)) ++s;
> *s = '\0';
> PL_splitstr = savepv(PL_splitstr);
> return s;
>
> This is a significant limitation of the -F switch and should be
> documented, if not changed to all a space within a "" or //.
Yes. That's a Feature, to make -F work on #! lines.
Change 25745 by rgs@bloom on 2005/10/13 08:14:32
Document that -F doesn't accept whitespace in patterns.
(fixes bug #37366)
Affected files ...
... //depot/perl/pod/perlrun.pod#139 edit
Differences ...
==== //depot/perl/pod/perlrun.pod#139 (text) ====
@@ -451,7 +451,7 @@
specifies the pattern to split on if B<-a> is also in effect. The
pattern may be surrounded by C<//>, C<"">, or C<''>, otherwise it will be
-put in single quotes.
+put in single quotes. You can't use literal whitespace in the pattern.
=item B<-h>
John W. Krahn
>>[kra...@telus.net - Thu Oct 06 10:59:48 2005]:
>>
>>This is a bug report for perl from kra...@telus.net,
>>generated with the help of perlbug 1.35 running under perl v5.8.6.
>>
>>
>>-----------------------------------------------------------------
>>[Please enter your report here]
>>
>>
>>A space character in a character class with the -F switch 'eats' the
>>following
>>characters in the class.
>>
>>$ perl -F'[ ]' -lane'print for @F'
>>Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
>>$ perl -F'[;#!%^&* ]' -lane'print for @F'
>>Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE ;#!%^&*/.
>>$ perl -F'[ ;#!%^&*]' -lane'print for @F'
>>Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
>>
>
>
> case 'F':
> PL_minus_F = TRUE;
> PL_splitstr = ++s;
> while (*s && !isSPACE(*s)) ++s;
> *s = '\0';
> PL_splitstr = savepv(PL_splitstr);
> return s;
>
> This is a significant limitation of the -F switch and should be
> documented, if not changed to all a space within a "" or //.
Ah, but a space CAN be used with the -F switch:
$ echo "one two" | perl -F' ' -lane'print "-$_-" for @F'
-one-
-two-
$ echo "one two" | perl -F' +' -lane'print "-$_-" for @F'
-one-
-two-
Although that just ignores the -F switch contents and uses the default split
for @F, and it should also be noted that -F'' counter-intuitively uses the
default split as well.
$ echo "one two" | perl -F'' -lane'print "-$_-" for @F'
-one-
-two-
$ echo "one two" | perl -F// -lane'print "-$_-" for @F'
-o-
-n-
-e-
- -
- -
- -
-t-
-w-
-o-
$ echo "one two" | perl -lne'print "-$_-" for split q//'
-o-
-n-
-e-
- -
- -
- -
-t-
-w-
-o-
$ echo "one two" | perl -lne'print "-$_-" for split //'
-o-
-n-
-e-
- -
- -
- -
-t-
-w-
-o-
Anyway, it *looks* like a bug. :-)
John
--
use Perl;
program
fulfillment
Rafael Garcia-Suarez
> Ah, but a space CAN be used with the -F switch:
Yes but anything beginning with the space is ignored :
$ echo "one two" | bleadperl -F'o ' -lane 'print "-$_-" for @F'
--
-ne tw-
Just like what happens on a shebang line.
--
He favored the verse form in the theater because it prevents the spectators
from forgetting unreality, which is the necessary condition of art.
-- Borges
Rick Delaney
> "Steve Peters via RT" <perlbug-...@perl.org> wrote:
> > Looking at the code in perl.c, a space cannot be used in a -F pattern.
> >
> > case 'F':
> > PL_minus_F = TRUE;
> > PL_splitstr = ++s;
> > while (*s && !isSPACE(*s)) ++s;
> > *s = '\0';
> > PL_splitstr = savepv(PL_splitstr);
> > return s;
> >
> > This is a significant limitation of the -F switch and should be
> > documented, if not changed to all a space within a "" or //.
>
> Yes. That's a Feature, to make -F work on #! lines.
OK, but we could still implement Steve's suggestion and allow spaces
when there are explicit delimiters. Something like (not meant to be
robust):
diff -pruN perl-current/perl.c perl-current-dev/perl.c
--- perl-current/perl.c 2005-10-13 12:21:14.000000000 -0400
+++ perl-current-dev/perl.c 2005-10-13 14:05:43.000000000 -0400
@@ -2950,7 +2950,17 @@ Perl_moreswitches(pTHX_ char *s)
case 'F':
PL_minus_F = TRUE;
PL_splitstr = ++s;
- while (*s && !isSPACE(*s)) ++s;
+ if (*s == '/' || *s == '\'' || *s == '"') {
+ char *c;
+ if (c = strrchr(s + 1, *s)) {
+ s = c + 1;
+ }
+ else
+ die("Can't find terminator %c in -F switch\n", *s);
+ }
+ else {
+ while (*s && !isSPACE(*s)) ++s;
+ }
*s = '\0';
PL_splitstr = savepv(PL_splitstr);
return s;
Then you could split on spaces either on the command line
perl -lanF'/[ ]/' -e 'print $F[2]'
or on the shebang line
#!/usr/bin/perl -lanF/[ ]/
print $F[2];
It seems a shame to let awk do something else that perl can't.
> Change 25745 by rgs@bloom on 2005/10/13 08:14:32
>
> Document that -F doesn't accept whitespace in patterns.
> (fixes bug #37366)
>
> Affected files ...
>
> ... //depot/perl/pod/perlrun.pod#139 edit
>
> Differences ...
>
> ==== //depot/perl/pod/perlrun.pod#139 (text) ====
>
> @@ -451,7 +451,7 @@
>
> specifies the pattern to split on if B<-a> is also in effect. The
> pattern may be surrounded by C<//>, C<"">, or C<''>, otherwise it will be
> -put in single quotes.
> +put in single quotes. You can't use literal whitespace in the pattern.
without explicit delimiters.
--
Rick Delaney
ri...@bort.ca
Rick Delaney
> On 10/13/05, Rick Delaney <ri...@bort.ca> wrote:
> > Then you could split on spaces either on the command line
> >
> > perl -lanF'/[ ]/' -e 'print $F[2]'
>
> $ perl -lanF'/[ ]/' -e 'print $F[2]'
> Unmatched [ in regex; marked by <-- HERE in m//[ <-- HERE /.
Works for me, with patch.
> > or on the shebang line
> >
> > #!/usr/bin/perl -lanF/[ ]/
> > print $F[2];
> >
> > It seems a shame to let awk do something else that perl can't.
>
> Be more specific.
% echo "From: somebody; To: somebody else" \
> | awk -F'[:;] +' '{OFS="\n"; print $2,$4}'
somebody
somebody else
Equivalent use of perl would be:
% echo "From: somebody; To: somebody else" \
> | perl -lanF'/[:;] +/' -e '$,="\n";print @F[1,3]'
The "something else" was just a reference to perlvar:
Remember: the value of C<$/> is a string, not a regex. B<awk> has
to be better for something. :-)
> I remember problems in the past with that; might be worthwhile to dig
> the archives. In short it seems to me that you can't rely on doing the
> argument splitting yourself. Helpful kernels might do it for you.
Hmm, didn't think of that. So what happens in a case like
#!/usr/bin/perl -lanF"
?
On Linux, this splits on " before the patch and dies with a "missing
terminator" message after the patch (which is probably reason enough to
leave things alone). Still, what would such a helpful kernel
(FreeBSD, right?) do? Does it just split args on whitespace, or does it
do its own mini-shell parsing and look for quotes and escapes?
--
Rick Delaney
ri...@bort.ca
Yitzchak Scott-Thoennes
> > A space character in a character class with the -F switch 'eats' the
> > following
> > characters in the class.
> >
> > $ perl -F'[ ]' -lane'print for @F'
> > Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
> > $ perl -F'[;#!%^&* ]' -lane'print for @F'
> > Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE ;#!%^&*/.
> > $ perl -F'[ ;#!%^&*]' -lane'print for @F'
> > Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE /.
> >
>
> Looking at the code in perl.c, a space cannot be used in a -F pattern.
>
> case 'F':
> PL_minus_F = TRUE;
> PL_splitstr = ++s;
> while (*s && !isSPACE(*s)) ++s;
> *s = '\0';
> PL_splitstr = savepv(PL_splitstr);
> return s;
>
> This is a significant limitation of the -F switch and should be
> documented, if not changed to all a space within a "" or //.
Note:
$ echo foo bar baz|perl -F'[\x20]' -lane'print for @F'
foo
bar
baz
Whoever mucks with the code, make sure it works when you do
$ perl foo.pl
and foo.pl has an -F on the shebang line, with other options after it.
David Landgren
> On Thu, Oct 13, 2005 at 08:28:29PM +0200, Rafael Garcia-Suarez wrote:
>
[...]
> Hmm, didn't think of that. So what happens in a case like
>
> #!/usr/bin/perl -lanF"
>
> ?
>
> On Linux, this splits on " before the patch and dies with a "missing
> terminator" message after the patch (which is probably reason enough to
> leave things alone). Still, what would such a helpful kernel
> (FreeBSD, right?) do? Does it just split args on whitespace, or does it
> do its own mini-shell parsing and look for quotes and escapes?
david@profane:~/perl% uname
FreeBSD
david@profane:~/perl% cat a
From: somebody; To: somebody else
david@profane:~/perl% perl -lanF'/[:;]/' -e '$,="\n";print @F' a
From
somebody
To
somebody else
david@profane:~/perl% perl -lanF'/[:;] +/' -e '$,="\n";print @F' a
From: somebody; To: somebody else
david@profane:~/perl% perl -lanF'/[:;]\x20+/' -e '$,="\n";print @F' a
From
somebody
To
somebody else
__END__
David
--
"It's overkill of course, but you can never have too much overkill."
Rick Delaney
>
> david@profane:~/perl% uname
> FreeBSD
> david@profane:~/perl% cat a
> From: somebody; To: somebody else
> david@profane:~/perl% perl -lanF'/[:;]/' -e '$,="\n";print @F' a
> From
> somebody
> To
> somebody else
> david@profane:~/perl% perl -lanF'/[:;] +/' -e '$,="\n";print @F' a
> From: somebody; To: somebody else
> david@profane:~/perl% perl -lanF'/[:;]\x20+/' -e '$,="\n";print @F' a
> From
> somebody
> To
> somebody else
>
> __END__
I'm not sure what any of that is supposed to show since there are no
shebang lines here. The question is how shebang lines are parsed. For
instance:
rick@biff:~/perl[66]% uname -sr
Linux 2.6.12-1-amd64-generic
rick@biff:~/perl[67]% cat testargs.pl
#!/usr/bin/perl
$" = "][";
print "[@ARGV]\n";
rick@biff:~/perl[68]% cat shebang.pl
#!./testargs.pl a "b" c " " ' / /
rick@biff:~/perl[69]% ./shebang.pl
[a "b" c " " ' / /][./shebang.pl]
So on this kernel, the args are all in one string.
--
Rick Delaney
ri...@bort.ca
David Landgren
Oh right. People were talking about -e. So...
% head sb?
==> sb1 <==
#! /usr/local/bin/perl -lanF'/[:;]/'
$,="\n";
print @F;
==> sb2 <==
#! /usr/local/bin/perl -lanF'/[:;] +/'
$,="\n";
print @F;
==> sb3 <==
#! /usr/local/bin/perl -lanF'/[:;]\x20+/'
$,="\n";
print @F;
david@profane:~/perl/shebang% for n in 1 2 3; do echo $n; ./sb$n a; done
1
From: somebody; To: somebody else
2
Can't open perl script "+/'": No such file or directory
3
From: somebody; To: somebody else
__END__
IOW, the -lanF'/[:;] +/' gets completely messed on fbsd.
David
Rick Delaney
>
> ==> sb2 <==
> #! /usr/local/bin/perl -lanF'/[:;] +/'
> $,="\n";
> print @F;
> Can't open perl script "+/'": No such file or directory
So the args are just split on whitespace with no shell-like
interpretation of quotes. Getting this to work does seem quite
hopeless so I guess that's that (at least for me).
--
Rick Delaney
ri...@bort.ca