File IO Question: opening for appending without flock?
Colin Meyer
I was asked to make some maintenance changes to a cgi script that
logs the values of certain cookies to a file for user tracking.
My first examination of the script revealed that the original author
did not lock the log file after opening for appending.
[ open(OF,">>$fname") || return "$fname error: $!\n"; ]
My intuition and the example in perldoc -f flock say that when
opening a file for appending, one should lock & seek to the end before
actually writing.
I ran some tests to try and verify that not locking will clobber some
data, but was unsuccessful in any clobbering.
Test 1:
Two copies of the following script running in the background:
####################################################################
#!/usr/local/bin/perl
my $file = '/home/cmeyer/work/mediaone/testlog';
# the second copy used "test2\n"
my $line = "test1\n";
for (1..100_000)
{
open FILE, ">>$file";
print FILE $line;
close FILE;
}
####################################################################
> logtest1.pl &
> logtest2.pl &
[wait for processes to finish]
> wc -l testlog
200000 testlog
It seems that no clobbering took place.
Test 2:
This time I tried to more closely simulate the web server process by
running each append as a separate perl process. Also, I tried writing
more data to the log.
####################################################################
#!/usr/local/bin/perl
# log1.pl
for (1..10_000)
# log2.pl would show logtest2.pl
{ `./logtest1.pl` }
####################################################################
####################################################################
#!/usr/local/bin/perl
# logtest1.pl
my $file = '/home/cmeyer/work/mediaone/testlog';
my $line;
for (1..15)
# logtest2 would show test2:...
{ $line .= "test1:$_:" . "x" x 100 . "\n" }
open FILE, ">>$file";
print FILE $line;
close FILE;
####################################################################
> log1.pl &
> log2.pl &
This test writes much more data to testlog and fires up a new instance
of perl each time the log is appended to, much as the web-server does.
Still, no clobbering. Also interesting is that each instance of
logtest1.pl or logtest2.pl appended all 15 lines before the other
would append.
Test 3:
uses log1.pl and logtest1.pl as in test2 along with:
####################################################################
#!/usr/local/bin/perl
# stdin2test.pl
my $file = '/home/cmeyer/work/mediaone/testlog';
open FILE, ">>$file";
while (<STDIN>)
{
print FILE $_;
}
close FILE;
####################################################################
> log1.pl &
> stdin2test.pl
line one
line two
line three
^d
> wc -l testlog
150003 testlog
From testlog:
test1:15:[100 x's snipped]
line one
line two
line three
test1:1:[100 x's snipped]
test1:2:[100 x's snipped]
It seems that no data was clobbered and the stdin append happened in
between two callings of logtest1.pl. This behaviour leads me to
believe that either perl or the os is locking this file auto-magically
when I open for appending.
I have tested this on linux 2.0.33, irix 6.2 and sun4-solaris, all
using perl 5.004_04 with the same results.
If none of these tests fail, under what circumstances will data get
clobbered?
Thanks for clearing up this issue,
-Colin.
Another Perl Journeyman.
Larry Rosler
It is not necessary to lock a file being appended to. The flock
documentation is too conservative on this point; a better example would
show a file being opened for read/write (+<...).
The following description comes from my HP-UX manual fopen(3S), but the
underlying behavior is guaranteed by the open(2) function in all ANSI-C
compliant implementations:
When a file is opened for append ... it is impossible to overwrite
information already in the file. All output is written at the end of
the file, regardless of intervening calls to fseek(). If two separate
processes open the same file for append, each process can write freely
to the file without fear of destroying output being written by the
other. Output from the two processes will be intermixed in the file in
the order in which it is written.
Larry Rosler
Hewlett-Packard Laboratories
l...@hpl.hp.com
Colin Meyer wrote:
>
> I was asked to make some maintenance changes to a cgi script that
> logs the values of certain cookies to a file for user tracking.
>
> My first examination of the script revealed that the original author
> did not lock the log file after opening for appending.
> [ open(OF,">>$fname") || return "$fname error: $!\n"; ]
> My intuition and the example in perldoc -f flock say that when
> opening a file for appending, one should lock & seek to the end before
> actually writing.
... long snip ...
M.J.T. Guy
In article <351C58FE...@hpl.hp.com>, Larry Rosler <l...@hpl.hp.com> wrote:
>It is not necessary to lock a file being appended to. The flock
>documentation is too conservative on this point; a better example would
>show a file being opened for read/write (+<...).
>
>The following description comes from my HP-UX manual fopen(3S), but the
>underlying behavior is guaranteed by the open(2) function in all ANSI-C
>compliant implementations:
>
>When a file is opened for append ... it is impossible to overwrite
>information already in the file. All output is written at the end of
>the file, regardless of intervening calls to fseek(). If two separate
>processes open the same file for append, each process can write freely
>to the file without fear of destroying output being written by the
>other. Output from the two processes will be intermixed in the file in
>the order in which it is written.
In practice, you'll usually get away with this. But note that the
interleaving is not defined - you might get a record from one process
in the middle of a record from another process. So strictly speaking
locking _is_ required.
Mike Guy