Lines written to a file are not contiguous

[Sherman Willden]

Lines written to a file are not contiguous.

First I will describe what I want the script to perform and then I will include the script at the end of this message. I downloaded the Oreilly Programming Perl Version 4 examples. There are 30 directories with 2211 files.  I want each chapter to have a consolidated file with all the examples for that chapter. I created the consolidate_examples.pl script which goes into each directory, reads each file, and then writes the files lines to the consolidated_perl4_examples/<directory name>. An example is consolidated_perl4_examples/ch00_preface with contains six lines. ch27_perl_functions is the largest and has 435 files which results in a single file with 870 lines. What I expected is as follows. If the first file I read from contains six lines then I expected the resulting consolidated file to have the first six lines align with read-from file. But they don't. The consolidated file's lines are mixed with the first six lines comming from file 1 or file 8 or whatever. The lines read in are not contiguous. For example in ch12 the first consolidated line comes from file 004 and the second consolidated line comes from 055. 

------------ SCRIPT --------------

#!/usr/bin/perl

use v5.16.3;
# use v5.14.2;
use File::Basename;
use File::Path;

my $base_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/programming_perl_examples/program_listings";
my $basename_directory = basename($base_directory) || die "Unable to determine the basename: $!\n";
my $write_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/consolidated_perl4_examples";
if ( not -e "$write_directory" ) {
    mkdir("$write_directory") || die "Unable to create $write_directory: $!\n";
}

chdir($base_directory) || die "Unable to change directory to $basename_directory: $!\n";

opendir(PERLDIR, ".") or die "Unable to open $basename_directory: $!\n";
my @docdirs = grep { $_ ne '.' and $_ ne '..' } readdir PERLDIR;
closedir(PERLDIR) || die "Unable to close $basename_directory:$!\n";
foreach my $directory ( @docdirs ) {
    chdir("$base_directory/$directory") || die "Unable to change directory to $directory: $!\n";
    my $write_file = "$write_directory/$directory";
    opendir(EXAMPLES, ".") or die "Unable to open $directory: $!\n";
    my @docfiles = grep {$_ ne '.' and $_ ne '..'} readdir EXAMPLES;
    closedir(EXAMPLES) || die "Unable to close $directory: $!\n";
    foreach my $file ( @docfiles ) {
        my ( $write, $read );
        open(WRITE, ">>$write_file") || die "Unable to open $write_file: $!\n";
        open(READ, $file) || die "Unable to open $file: $!\n";
        my $line = readline(READ) || die "Unable to read $file: $!\n";
        print WRITE "$line\n";
        close(WRITE) || die "Unable to close $write_file: $!\n";
        close(READ) || die "Unable to close $file: $!\n";
    }        
}

1;

[Brandon McCaig]

On Tue, May 07, 2013 at 12:00:10PM -0600, Sherman Willden wrote:
> Lines written to a file are not contiguous.

OK. Let's take a look then...

(Note that your mail client seems tos have wrapped the code...
I'm going to fix that in my quotes so it's readable)

> #!/usr/bin/perl
>
> use v5.16.3;
> # use v5.14.2;
> use File::Basename;
> use File::Path;
>
> my $base_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/programming_perl_examples/program_listings";
> my $basename_directory = basename($base_directory) || die "Unable to determine the basename: $!\n";

You should be cautious using the || operator for this purpose.
What you're really saying is:

my $basename_directory = (
basename($base_directory) ||
die("Unable to determine the basename: $!")
);

The 'or' operator exists for this purpose. It has extremely low
precedence so that the assignment operator ('=') is evaluated
first:

my $basename_directory = basename($base_directory) or
die "Unable to determine the basename: $!";

Which means:

(my $basename_directory = basename($base_directory)) or
die("Unable to determine the basename: $!");

Since 'die' raises an exception it doesn't really matter to Perl
because the assignment will never be evaluated, and even if it
were the value would never be used. It might matter though if the
right-hand expression isn't die so keep it in mind. The 'or'
operator expresses your intent more accurately in either case.

One last thing: When your program prints the exception it will
add context information and add a newline automatically. So the
newline that you're adding is probably redundant. :)

> my $write_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/consolidated_perl4_examples";
> if ( not -e "$write_directory" ) {

The use of 'not' here is similar to '||' above, except reversed.
It has very low precedence, whereas '!' has higher precedence.
Again, it doesn't matter here, but it's something to keep in
mind. They aren't exactly synonymous. Optionally you could use
'unless' instead of 'if', and eliminate the explicit
not-operation entirely, but it's just semantics: some like it and
some don't.

> mkdir("$write_directory") || die "Unable to create $write_directory: $!\n";
> }
>
> chdir($base_directory) || die "Unable to change directory to $basename_directory: $!\n";
>
> opendir(PERLDIR, ".") or die "Unable to open $basename_directory: $!\n";

You generally should prefer lexical file and dir handles instead
of barewords:

opendir(my $dh, '.') or
die "Unable to open $basename_directory: $!";

> my @docdirs = grep { $_ ne '.' and $_ ne '..' } readdir PERLDIR;
> closedir(PERLDIR) || die "Unable to close $basename_directory:$!\n";
> foreach my $directory ( @docdirs ) {
> chdir("$base_directory/$directory") || die "Unable to change directory to $directory: $!\n";

Changing directories is fine, but in the case where you want to
write to many files in various places it probably makes better
sense to remain where ever you start (or move somewhere specific
for precautionary reasons, depending on the program). I would
argue that the changing of directories adds needless
complexities.

> my $write_file = "$write_directory/$directory";
> opendir(EXAMPLES, ".") or die "Unable to open $directory: $!\n";
> my @docfiles = grep {$_ ne '.' and $_ ne '..'} readdir EXAMPLES;
> closedir(EXAMPLES) || die "Unable to close $directory: $!\n";
> foreach my $file ( @docfiles ) {
> my ( $write, $read );

$read and $write do not appear to be used. Even if they were,
it's generally preferred to declare them where you first
initialize/use them.

> open(WRITE, ">>$write_file") || die "Unable to open $write_file: $!\n";

Again, you should prefer lexical file handles. You should also
prefer the 3-argument (or more) open(). It is safer because the
'mode' is a separate parameter.

open my $write_fh, '>>', $write_file) or
die "Unable to open $write_file: $!";

> open(READ, $file) || die "Unable to open $file: $!\n";

Again, 3-argument open is preferred. If $file has a specially
crafted name then this might do Something Evil(tm). For a hint,
see what your shell says about:

bash$ touch 'echo rm -i ~'

So make sure to always use the 3-argument open. Even if the
arguments are guaranteed to be safe ("untainted") right now, that
might change.

open my $read_fh, '<', $file or die "Unable to open $file: $!";

> my $line = readline(READ) || die "Unable to read $file: $!\n";
> print WRITE "$line\n";

This is where you lose me. Unless I'm missing something you seem
to be only reading a single line from each file? Then you append
that into the "$write_directory/$directory" file. IIRC, you said
that you believe the output is actually more sporadic than that
though. Which I'm not following...

One thing to note is that <> and readline don't strip the newline
character from the line, so the additional one that you're
writing there is likely causing duplicates in the output. And for
future reference, non-ancient versions of Perl have a say()
alternative to print() that appends a newline for you.

> close(WRITE) || die "Unable to close $write_file: $!\n";
> close(READ) || die "Unable to close $file: $!\n";
> }
> }
>
> 1;

If that isn't enough to figure out your problem then I think
you're going to have to take another shot at explaining what it
is that you're trying to achieve, and what you're getting
instead. Many of us will not have the aforementioned book so you
should try to explain it to us assuming no external knowledge.

Regards,


--
Brandon McCaig <bamc...@gmail.com> <bamc...@castopulence.org>
Castopulence Software <https://www.castopulence.org/>
Blog <http://www.bamccaig.com/>
perl -E '$_=q{V zrna gur orfg jvgu jung V fnl. }.
q{Vg qbrfa'\''g nyjnlf fbhaq gung jnl.};
tr/A-Ma-mN-Zn-z/N-Zn-zA-Ma-m/;say'

[Jim Gibson]

On May 7, 2013, at 11:00 AM, Sherman Willden wrote:

> Lines written to a file are not contiguous.
>
> First I will describe what I want the script to perform and then I will include the script at the end of this message. I downloaded the Oreilly Programming Perl Version 4 examples. There are 30 directories with 2211 files. I want each chapter to have a consolidated file with all the examples for that chapter. I created the
> consolidate_examples.pl
> script which goes into each directory, reads each file, and then writes the files lines to the consolidated_perl4_examples/<directory name>. An example is consolidated_perl4_examples/ch00_preface with contains six lines. ch27_perl_functions is the largest and has 435 files which results in a single file with 870 lines. What I expected is as follows. If the first file I read from contains six lines then I expected the resulting consolidated file to have the first six lines align with read-from file. But they don't. The consolidated file's lines are mixed with the first six lines comming from file 1 or file 8 or whatever. The lines read in are not contiguous. For example in ch12 the first consolidated line comes from file 004 and the second consolidated line comes from 055.

I believe your problem is that you are only reading one line from each input file. These two lines:

my $line = readline(READ);
print WRITE "$line\n";

should be replaced by

my @lines = readline(READ);
print WRITE @lines;

Note that you do not need to add a newline at the end of each output line, as the original newline character[s] will be read in with the lines from the input file.

>
> ------------ SCRIPT --------------
>
> #!/usr/bin/perl
>
> use v5.16.3;
> # use v5.14.2;
> use File::Basename;
> use File::Path;
>
> my $base_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/programming_perl_examples/program_listings";
> my $basename_directory = basename($base_directory) || die "Unable to determine the basename: $!\n";
> my $write_directory = "/home/sherman/perl_documentation/programming_perl_4_examples/consolidated_perl4_examples";

You should have something like this:

my $homedir = '/home/sherman/perl_documentation/programming_perl_4_examples';

and use that for generating the other variables:

my $base_directory = "$homedir/program_listings";
my $write_directory = "$homedir/consolidated_perl4_examples";

which is easier to write, easier to read, and less error-prone.

> if ( not -e "$write_directory" ) {
> mkdir("$write_directory") || die "Unable to create $write_directory: $!\n";
> }

You do not have to enclose $write_directory in double-quotes. Doing so can cause problems (although probably not here.)

If you leave off the newline at the end of your die message, Perl will tell you the line number of the error, which can be very useful sometimes.

>
> chdir($base_directory) || die "Unable to change directory to $basename_directory: $!\n";
>
> opendir(PERLDIR, ".") or die "Unable to open $basename_directory: $!\n";

It is generally better to use undefined lexical variables for file handles:

opendir( my $perldir, '.') or die(...);

> my @docdirs = grep { $_ ne '.' and $_ ne '..' } readdir PERLDIR;
> closedir(PERLDIR) || die "Unable to close $basename_directory:$!\n";
> foreach my $directory ( @docdirs ) {
> chdir("$base_directory/$directory") || die "Unable to change directory to $directory: $!\n";
> my $write_file = "$write_directory/$directory";
> opendir(EXAMPLES, ".") or die "Unable to open $directory: $!\n";
> my @docfiles = grep {$_ ne '.' and $_ ne '..'} readdir EXAMPLES;
> closedir(EXAMPLES) || die "Unable to close $directory: $!\n";
> foreach my $file ( @docfiles ) {
> my ( $write, $read );

These two variables don't seem to be used anywhere.

> open(WRITE, ">>$write_file") || die "Unable to open $write_file: $!\n";
> open(READ, $file) || die "Unable to open $file: $!\n";
> my $line = readline(READ) || die "Unable to read $file: $!\n";
> print WRITE "$line\n";
> close(WRITE) || die "Unable to close $write_file: $!\n";
> close(READ) || die "Unable to close $file: $!\n";
> }
> }
>
> 1;
>

Good luck!

[Sherman Willden]

Thank you all. It will take a day or two to get through all of this

Sherman

--
To unsubscribe, e-mail: beginners-...@perl.org
For additional commands, e-mail: beginne...@perl.org
http://learn.perl.org/

[David Christensen]

On 05/07/13 11:00, Sherman Willden wrote:
> ... I downloaded the

> Oreilly Programming Perl Version 4 examples. There are 30 directories
> with 2211 files. I want each chapter to have a consolidated file with
> all the examples for that chapter. I created the
> consolidate_examples.pl script which goes into each directory, reads
> each file, and then writes the files lines to the

> consolidated_perl4_examples/<directory name>. ...

1. I hope you don't expect the "consolidated file" to be a valid Perl
program (?)...

2. Rather than reading/ writing the files programmatically with Perl,
why not use Perl to invoke "cat" via the backticks operator (e.g. shell
script):

$| = 1; # [1] $OUTPUT_AUTOFLUSH
print `cat * >>foo.out`; # [2] qx//
if ($? != 0) { # [3] $CHILD_ERROR
# error handling --
}

HTH,

David



References:

[1] http://perldoc.perl.org/perlvar.html#Variables-related-to-filehandles

[2] http://perldoc.perl.org/perlop.html#Quote-Like-Operators

[3] http://perldoc.perl.org/perlvar.html#Error-Variables

[Shlomi Fish]

Hello David.

On Tue, 07 May 2013 20:26:23 -0700
David Christensen <dpch...@holgerdanske.com> wrote:

> On 05/07/13 11:00, Sherman Willden wrote:
> > ... I downloaded the
> > Oreilly Programming Perl Version 4 examples. There are 30 directories
> > with 2211 files. I want each chapter to have a consolidated file with
> > all the examples for that chapter. I created the
> > consolidate_examples.pl script which goes into each directory, reads
> > each file, and then writes the files lines to the
> > consolidated_perl4_examples/<directory name>. ...
>
> 1. I hope you don't expect the "consolidated file" to be a valid Perl
> program (?)...
>
> 2. Rather than reading/ writing the files programmatically with Perl,
> why not use Perl to invoke "cat" via the backticks operator (e.g. shell
> script):
>
> $| = 1; # [1] $OUTPUT_AUTOFLUSH

This is better:

http://perl-begin.org/tutorials/bad-elements/#properly_autoflushing

> print `cat * >>foo.out`; # [2] qx//

This is the same as «system('cat * >> foo.out')» (only more costly), and the
command in this case should not emit any output (because it is redirected to a
file). Furthermore, many non-UNIX-like systems don't contain a cat command. See:

http://perl-begin.org/tutorials/bad-elements/

Regards,

Shlomi Fish

--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/
Optimising Code for Speed - http://shlom.in/optimise

The X in XSLT stands for eXtermination.
— http://www.shlomifish.org/humour/bits/facts/XSLT/

Please reply to list if it's a mailing list post - http://shlom.in/reply .

[Rob Dixon]

On 08/05/2013 10:24, Shlomi Fish wrote:
>
> This is better:
>
> http://perl-begin.org/tutorials/bad-elements/#properly_autoflushing
>
>> print `cat * >>foo.out`; # [2] qx//
>
> This is the same as «system('cat * >> foo.out')» (only more costly), and the
> command in this case should not emit any output (because it is redirected to a
> file). Furthermore, many non-UNIX-like systems don't contain a cat command. See:
>
> http://perl-begin.org/tutorials/bad-elements/
>
> Regards,
>
> Shlomi Fish
>

It would make me feel a lot better if you would declare your interest in
your own web site when you link to it. It's hardly an endorsement if you
link to something else you have written yourself.

Rob

[Shlomi Fish]

Hello, Rob,

I'm giving that page on Perl-Begin, as a resource where people can find
more information about my advice, not as an act of endorsement.

Regards,

Shlomi Fish

--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/

List of Networking Clients - http://shlom.in/net-clients

Chuck Norris doesn’t make mistakes. (Su‐Shee) He corrects God. (Shlomi Fish)
— http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/

[David Christensen]

I wrote:
>> $| = 1; # [1] $OUTPUT_AUTOFLUSH

On 05/08/13 02:24, Shlomi Fish wrote:
> http://perl-begin.org/tutorials/bad-elements/#properly_autoflushing

Which recommends:

use IO::Handle;
STDOUT->autoflush(1);

1. For larger, longer-lived, shared, portable, "serious", "Modern
Perl", etc., programs, that way is better.

2. I saw the OP's requirements as being met by a short "throw-away"
script that wouldn't be kept or distributed. $| is fast and cheap, and
it works.

3. Part of gaining proficiency in Perl (and other Unix tools) is
learning the more commonly used special variables, such as $|, $?, $@, etc..

4. Autoflush might not be needed in this case. I put it in there as a
reminder for scripts that do need it.

>> print `cat *>>foo.out`; # [2] qx//
> This is the same as «system('cat *>> foo.out')» (only more costly), and the
> command in this case should not emit any output (because it is redirected to a
> file).

1. I thought about:

if (my $e = system('cat * >> foo.out')) {
# error handling
}

But, a decent system() example should use the multi-argument list form.
That means I would have to redirect STDOUT first and glob the file
names in the argument list. Too much complexity for a "beginner" posting.

2. Yes, cat's STDOUT ends up in foo.out and STDERR ends up on the
console. I put the "print" in there as much to make people say "WTF?",
look it up, and ponder it, thus reinforcing their knowledge of backticks.

3. The append redirect ">>" is a bug/ mis-feature -- foo.out will grow
every time the script is run, The user is required to delete foo.out
manually before running the script. The alternative, ">", destroys
data. Pick your poison.

> Furthermore, many non-UNIX-like systems don't contain a cat command.

Thank God for electronics recycling. ;-)


David

[Rob Dixon]

On 08/05/2013 14:16, Shlomi Fish wrote:
>>> See:
>>>
>>> http://perl-begin.org/tutorials/bad-elements/
>>>
>>> Regards,
>>>
>>> Shlomi Fish
>>>
>>
>> It would make me feel a lot better if you would declare your interest in
>> your own web site when you link to it. It's hardly an endorsement if you
>> link to something else you have written yourself.
>
> I'm giving that page on Perl-Begin, as a resource where people can find
> more information about my advice, not as an act of endorsement.

Whatever your reasons, I wish you would make it clear that it was just
more of your own words.

Rob

[Shlomi Fish]

Hi David,

On Wed, 08 May 2013 19:37:30 -0700
David Christensen <dpch...@holgerdanske.com> wrote:

> I wrote:
> >> $| = 1; # [1] $OUTPUT_AUTOFLUSH
>
> On 05/08/13 02:24, Shlomi Fish wrote:
> > http://perl-begin.org/tutorials/bad-elements/#properly_autoflushing
>
> Which recommends:
>
> use IO::Handle;
> STDOUT->autoflush(1);
>
> 1. For larger, longer-lived, shared, portable, "serious", "Modern
> Perl", etc., programs, that way is better.
>
> 2. I saw the OP's requirements as being met by a short "throw-away"
> script that wouldn't be kept or distributed. $| is fast and cheap, and
> it works.
>

Yes, and obscure.

> 3. Part of gaining proficiency in Perl (and other Unix tools) is
> learning the more commonly used special variables, such as $|, $?, $@, etc..

And an even further part of gaining proficiency is to know to make your code
less obscure and less clever and more
http://en.wikipedia.org/wiki/KISS_principle (not my own link!).

>
> 4. Autoflush might not be needed in this case. I put it in there as a
> reminder for scripts that do need it.
>

OK.

>
> >> print `cat *>>foo.out`; # [2] qx//
> > This is the same as «system('cat *>> foo.out')» (only more costly), and the
> > command in this case should not emit any output (because it is redirected
> > to a file).
>
> 1. I thought about:
>
> if (my $e = system('cat * >> foo.out')) {
> # error handling
> }
>
> But, a decent system() example should use the multi-argument list form.
> That means I would have to redirect STDOUT first and glob the file
> names in the argument list. Too much complexity for a "beginner" posting.

Nothing wrong with a non-list call to system() if that is what you want and need
(as long as you say why you intend to do that and use precautions such as
https://metacpan.org/release/String-ShellQuote ). Doing «print `...`» is silly.

>
> 2. Yes, cat's STDOUT ends up in foo.out and STDERR ends up on the
> console. I put the "print" in there as much to make people say "WTF?",
> look it up, and ponder it, thus reinforcing their knowledge of backticks.

Please don't give people "WTF?" moments like that. I am an experienced
programmer and your code gave me that.

>
> 3. The append redirect ">>" is a bug/ mis-feature -- foo.out will grow
> every time the script is run, The user is required to delete foo.out
> manually before running the script. The alternative, ">", destroys
> data. Pick your poison.

Yes.

> > Furthermore, many non-UNIX-like systems don't contain a cat command.
>
> Thank God for electronics recycling. ;-)
>

What do you mean?

Regards,

Shlomi Fish

--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/

Original Riddles - http://www.shlomifish.org/puzzles/

George: I guess you can enter now. I’m just following orders.
SGlau: The Nuremberg defense!
George: This place gives you no other option.
— http://www.shlomifish.org/humour/Summerschool-at-the-NSA/

[Sherman Willden]

Thank you, David. I thought about doing cat but I thought it would have to be a system call.

Sherman

[David Christensen]

Shlomi Fish wrote:
>>> Furthermore, many non-UNIX-like systems don't contain a cat command.

I wrote:
>> Thank God for electronics recycling. ;-)

On 05/08/13 23:50, Shlomi Fish wrote:
> What do you mean?

I meant that systems that are non-UNIX-like (or cannot be made Unix-like
with Linux, *BSD, etc.) should be decommissioned and recycled. Please
note the wink smiley.


David

[David Christensen]

On 05/09/13 07:56, Sherman Willden wrote:
> Thank you, David. I thought about doing cat but I thought it would have to
> be a system call.

YW. TIMTOWTDI -- system calls or otherwise. More importantly, there is
more than one style of Perl programming. I find Perl especially useful
for "quick and dirty" shell scripts (e.g. instant gratification). And,
I keep the more useful ones around and improve them over time (e.g.
upgradability).


David

[Dr.Ruud]

On 07/05/2013 20:00, Sherman Willden wrote:

> foreach my $file ( @docfiles ) {
> my ( $write, $read );

What were they meant for?

--
Ruud

[Sherman Willden]

Thank you all. I am not sure what $write and $read are for since I am just now getting back to this. They may have been something I used then discarded. I am surprised that I didn't get a notice that they are only used once.

Sherman