Why do users need FileHandles?
Dave Whipp
capabilities of "string"s.
Given that we'll have things like $str.bytes, etc. It doesn't seem a stretch
to suggest that we could also have $str.lines. Once we have that, and also a
level of pervasive laziness (lazy evaluation), it seems to me that we don't
really need user-visible file handles for most of the common operations on
files.
Imagine:
my $text is TextFile("/tmp/foo.txt");
for $text.lines {
...
}
The for loop is equivalent to the old "while(<FH>)" construct, but is more
general in the sense that it applies to any test-like thing, not just
file-handles. It also makes it easy to see how a use would apply a grammar
to the contents of a file.
Not all files are TextFiles, of course. We might have XML files, RawBinary
files, etc. If we can think in terms of tying, then we can use all this
things without ever touching a file handle.
Writing a file is similarly encapsulated:
my $text is TextFile("/tmp/bar");
$text = "hello"; # writes, truncates
$text ~= ", world\n"; # appends
$text.print "again\n"; # for old-times sake
Another use of filehandles for interactive communication over (e.g.) a
socket. Interactive communication can be thought of as message passing --
which perhaps should be unified under a general mechanism for sending
messages between threads/process/etc. A "producer" sends messages to a
"consumer", and thus has an object (proxy/handle) that represents that
consumer. The fact that the implementation is a "File Handle" is not
something that a user needs to worry about (usually).
I guess what I'm saying is that if we can make tying the standard idiom,
then we can relax you huffmanization worries for things like the "open"
function.
Dave.
Andrew Shitov
DW> $text = "hello"; # writes, truncates
DW> $text ~= ", world\n"; # appends
DW> $text.print "again\n"; # for old-times sake
Anyhow we still need $text.flush() or $text.close() methods.
--
____________________________________________________________________________
Андрей, an...@shitov.ru
____________________________________________________________________________
Dave Whipp
news:1616135014.2...@shitov.ru...
> DW> my $text is TextFile("/tmp/bar");
> DW> $text = "hello"; # writes, truncates
> DW> $text ~= ", world\n"; # appends
>
> DW> $text.print "again\n"; # for old-times sake
>
> Anyhow we still need $text.flush() or $text.close() methods.
I'd argue that these are only rarely needed. Even with Perl5 filehandles, I
rarely find myself needing to close a file. Garbage collection does the job
adequately.
The case of "flush" is more interesting. It is really used for two purposes:
forcing the ram-contents to be stored on disk, so it can be recovered even
id the process dies violently, is one use; ensuring a message is sent so it
can be received by a consumer is the other. Even if both these roles have
the same underlying implementation, we might want to abstract them
differently. C<flush> is as good a name as any for the sync-to-disc usage;
but the second might be better named C<send> -- and probably wouldn't be on
a "text" object, anyway. I'm not sure that the details of inter-thread
messaging have been decided yet, but I'd think that a common messaging
paradigm would apply.
Dave.
Rod Adams
>I was thinking about the discussions about the "open" function, and of the
>capabilities of "string"s.
>
>Given that we'll have things like $str.bytes, etc. It doesn't seem a stretch
>to suggest that we could also have $str.lines. Once we have that, and also a
>level of pervasive laziness (lazy evaluation), it seems to me that we don't
>really need user-visible file handles for most of the common operations on
>files.
>
>Imagine:
>
> my $text is TextFile("/tmp/foo.txt");
> for $text.lines {
> ...
> }
>
><snip>
>
>I guess what I'm saying is that if we can make tying the standard idiom,
>then we can relax you huffmanization worries for things like the "open"
>function.
>
>
Uhm, my impression was that most of the "huffmanization" discussion was
centered around declaring a file handle to be read only, write only,
read-write, exclusive, etc. Masking the file handle with what basically
amounts to a file handle subclass like you describe will still need to
allow the user to specify all those attributes.
So you would still need to allow:
my $text is TextFile("/tmp/foo.txt" :rw );
my $text is TextFile("/tmp/foo.txt" :excl );
Not that having wrapper classes for file handles is a bad idea, it just
doesn't relate to what I saw being discussed.
Oh, and "TextFile" should be spelled "IO::File::Text", IMHO.
-- Rod Adams
Dave Whipp
news:40FB0D91...@rodadams.net...
> Uhm, my impression was that most of the "huffmanization" discussion was
> centered around declaring a file handle to be read only, write only,
> read-write, exclusive, etc. Masking the file handle with what basically
> amounts to a file handle subclass like you describe will still need to
> allow the user to specify all those attributes.
>
> So you would still need to allow:
> my $text is TextFile("/tmp/foo.txt" :rw );
> my $text is TextFile("/tmp/foo.txt" :excl );
my $text is TextFile("/tmp/foo") is rw;
my $text is TextFile("/tmp/foo") is const;
truncate Vs append would be infered from usage (assign => truncate). One
might be able to infer read Vs write in a similar way -- open the file based
on the first access; re-open it (behind the scenes) if we write it after
reading it.
:excl would probably need to be an option, but is not sufficiently common to
be agressively huffmanised:
my $text is TextFile("foo.txt", :no_overwrite);
my $text is TextFile("foo.txt") does no_overwrite;
> Not that having wrapper classes for file handles is a bad idea, it just
> doesn't relate to what I saw being discussed.
> Oh, and "TextFile" should be spelled "IO::File::Text", IMHO.
Possibly, but would need a hufmanized alias for common use. Possible just
"file":
my Str $text is file("foo.txt") does no_follow_symlink does no_create;
Do we have an antonym for C<does>?
Dave.
Rod Adams
>"Rod Adams" <r...@rodadams.net> wrote in message
>news:40FB0D91...@rodadams.net...
>
>
>>Uhm, my impression was that most of the "huffmanization" discussion was
>>centered around declaring a file handle to be read only, write only,
>>read-write, exclusive, etc. Masking the file handle with what basically
>>amounts to a file handle subclass like you describe will still need to
>>allow the user to specify all those attributes.
>>
>>So you would still need to allow:
>>my $text is TextFile("/tmp/foo.txt" :rw );
>>my $text is TextFile("/tmp/foo.txt" :excl );
>>
>>
>
>my $text is TextFile("/tmp/foo") is rw;
>my $text is TextFile("/tmp/foo") is const;
>
>truncate Vs append would be infered from usage (assign => truncate). One
>might be able to infer read Vs write in a similar way -- open the file based
>on the first access; re-open it (behind the scenes) if we write it after
>reading it.
>
>
Case 1:
So I wanted to do a read/write scan, so I create my TextFile, start
reading in data, so the file is opened for reading. Then, I come to the
part where I want to update something, so I do a write command. Suddenly
the file has to be closed, and then re-opened for read and write. And
all my buffers, file pointers and the like are reset, (though curable
with very careful planning), leaving me in a bad spot. Better if I could
just declare the file open for read and write at open time.
Case 2:
I meant to use some critical data file in read-only mode, and accidently
use a write command somewhere I didn't mean to, and silently just
clobbered /etc/passwd. Better if I could have just opened the file read
only, and trigger an error on the write command.
What design philosophy would you envision TextFile taking to handle both
these cases in a coherent fashion?
>:excl would probably need to be an option, but is not sufficiently common to
>be agressively huffmanised:
>my $text is TextFile("foo.txt", :no_overwrite);
>my $text is TextFile("foo.txt") does no_overwrite;
>
>>ot that having wrapper classes for file handles is a bad idea, it just
>>doesn't relate to what I saw being discussed.
>>Oh, and "TextFile" should be spelled "IO::File::Text", IMHO.
>>
>>
>Possibly, but would need a hufmanized alias for common use. Possible just
>"file":
>
>
s/file/open/ and we're back where we started.
>my Str $text is file("foo.txt") does no_follow_symlink does no_create;
>
my $text = open("foo.txt" :no_follow_symlink :no_create);
I don't think anyone (read: Larry) has declared exactly what the
capabilities of the default file handle object are yet. It seems me that
you could very well get what you want.
-- Rod Adams
Dave Whipp
> Case 1:
> So I wanted to do a read/write scan, so I create my TextFile, start
> reading in data, so the file is opened for reading. Then, I come to the
> part where I want to update something, so I do a write command. Suddenly
> the file has to be closed, and then re-opened for read and write. And
> all my buffers, file pointers and the like are reset, (though curable
> with very careful planning), leaving me in a bad spot. Better if I could
> just declare the file open for read and write at open time.
>
> Case 2:
> I meant to use some critical data file in read-only mode, and accidently
> use a write command somewhere I didn't mean to, and silently just
> clobbered /etc/passwd. Better if I could have just opened the file read
> only, and trigger an error on the write command.
>
> What design philosophy would you envision TextFile taking to handle both
> these cases in a coherent fashion?
Firstly, I must say that if you phase a problems in terms of a solution,
then its not surprising that the solution looks a bit your problem statement
(E.g. "start reading in data" -- the important thing is to process it, not
to read it) From a users point of view, you're just doing string
manipulations:
for $text.lines { /:w ^myfield (\d+)/ and $1 = "X" x $1.chars }
Under the hood, I don't really care, as long as it works (and is
sufficiently efficient). There are obvious implementations that might be a
bit inefficient on the first iteration (e.g. close and reopen). Quite
frankly, the number of times I open unstructured files in rd/wr mode in a
typical program can be measured on the fingers of one foot! If I want to do
a R-M-W operation then I do like -i, and use a tmp file. Maybe the
hypothetical TextFile object would do the same (by default). If I want to
micromanage the actual access to the file object, then I'd be happy to
c<use> a module that lets me manipulate file handles directly. I just don't
see that as the common case.
Your case 2 is easy: "my Str $passwds is File("/etc/passwd") is const". With
that, we might even catch your error at compile time.
> s/file/open/ and we're back where we started.
Except that we've lost a layer of abstraction: the programmer manipulates a
file's contents, not its accessor. Text files would be just an
implementation of strings. No need to learn/use a different set of
operators. Want to read bytes: use $str.bytes. Graphemes: $str.graphs. Also,
we use the existing access control mechanisms ("is rw", "is const", instead
of inventing new ones to pass to the C<open> function as named-args).
Alan Cooper, in his book on human-computer interface design, makes the case
that files are an obsolete abstraction for users. I guess I'm making the
same argument for programmers.
Dave.
Rod Adams
>Under the hood, I don't really care, as long as it works (and is
>sufficiently efficient). There are obvious implementations that might be a
>bit inefficient on the first iteration (e.g. close and reopen). Quite
>frankly, the number of times I open unstructured files in rd/wr mode in a
>typical program can be measured on the fingers of one foot! If I want to do
>a R-M-W operation then I do like -i, and use a tmp file. Maybe the
>hypothetical TextFile object would do the same (by default). If I want to
>micromanage the actual access to the file object, then I'd be happy to
>c<use> a module that lets me manipulate file handles directly. I just don't
>see that as the common case.
>
>Your case 2 is easy: "my Str $passwds is File("/etc/passwd") is const". With
>that, we might even catch your error at compile time.
>
>>/file/open/ and we're back where we started.
>>
>>
>
>Except that we've lost a layer of abstraction: the programmer manipulates a
>file's contents, not its accessor. Text files would be just an
>implementation of strings. No need to learn/use a different set of
>operators. Want to read bytes: use $str.bytes. Graphemes: $str.graphs. Also,
>we use the existing access control mechanisms ("is rw", "is const", instead
>of inventing new ones to pass to the C<open> function as named-args).
>
>
I think part of the "mental jam" (at least with me), is that the
read/write, exclusive, etc, are very critical to the act of opening the
file, not only an after the fact restriction on what I can do later. If
I cannot open a file for writing (permissions, out of space, write
locked, etc), I want to know the instant I attempt to open it as such,
_not_ when I later attempt to write to it. Having all these features
available to open as arguements seems a much better idea to me. It's
"Open a file with these specifications", not "Open a file, and then
apply these specifications to it".
I do admit there is merit to your abstraction system, but IMO, it
belongs in a library.
-- Rod Adams
Austin Hastings
> Dave Whipp wrote:
>
> >
> > Your case 2 is easy: "my Str $passwds is File("/etc/passwd") is
> > const". With that, we might even catch your error at compile
> > time.
> >
> >>/file/open/ and we're back where we started.
> >
> > Except that we've lost a layer of abstraction: the programmer
> > manipulates a file's contents, not its accessor. Text files
> > would be just an implementation of strings. No need to learn/use
> > a different set of operators. Want to read bytes: use $str.bytes. >
> Graphemes: $str.graphs. Also, we use the existing access control
> > mechanisms ("is rw", "is const", instead of inventing new ones to
> > pass to the C<open> function as named-args).
>
> I think part of the "mental jam" (at least with me), is that the
> read/write, exclusive, etc, are very critical to the act of opening
> the file, not only an after the fact restriction on what I can do
> later.
But why? I'd argue that this ties in to the verbose/exception
discussion of a few weeks back: if the operation fails, let it pass an
exception up the chain that can be caught and resolved (once) at a high
level.
Given that file processing is so common in Perl, it deserves a high
huffman scoring. The best way to do that is to abstract the operations
away and replace them with a single declaration of intent. That
declaration, of course, becomes a front-end for C or heavily optimized
parrot.
In a heavily OO paradigm, there would be a swarm of subclasses of type
stream -- istream, ostream, iostream, exclusive_iostream, whatever.
The suggestion is that we can derive an equally expressive vocabulary
using barewords and the occasional adverbial modifier.
> If I cannot open a file for writing (permissions, out of space,
> write locked, etc), I want to know the instant I attempt to open it
> as such, _not_ when I later attempt to write to it. Having all
> these features available to open as arguements seems a much better
> idea to me. It's "Open a file with these specifications", not "Open
> a file, and then apply these specifications to it".
But why? Do you really open files and then perform an hour of work
before attempting to use them? I'll argue that's not the normal case;
rather, the normal case is something like
open .... or die ...
other_stuff()
while (...) {
print ...
}
close
and the intervening delay (other_stuff) is negligible in wall-clock
terms: when a failure occurs, the user hears about it immediately.
> I do admit there is merit to your abstraction system, but IMO, it
> belongs in a library.
I think rather that the abstraction should be the default, and the
individual "I don't trust Perl" functions should be available as
separate entry points if the user explicitly requires them.
=Austin
Rod Adams
>--- Rod Adams <r...@rodadams.net> wrote:
>
>
>>I think part of the "mental jam" (at least with me), is that the
>>read/write, exclusive, etc, are very critical to the act of opening
>>the file, not only an after the fact restriction on what I can do
>>later.
>>
>>
>
>But why? I'd argue that this ties in to the verbose/exception
>discussion of a few weeks back: if the operation fails, let it pass an
>exception up the chain that can be caught and resolved (once) at a high
>level.
>
>
Guess I'm still in the
open "foo" or die;
mentality.
>Given that file processing is so common in Perl, it deserves a high
>huffman scoring. The best way to do that is to abstract the operations
>away and replace them with a single declaration of intent. That
>declaration, of course, becomes a front-end for C or heavily optimized
>parrot.
>
>In a heavily OO paradigm, there would be a swarm of subclasses of type
>stream -- istream, ostream, iostream, exclusive_iostream, whatever.
>
>
Is
my $file = append_text_stream "foo.txt";
really so much better than
my $file = open ">>foo.txt";
I'd strongly prefer having a few, extremely flexible, orthoganal, and
complete tools, that DWIM the common case for me, than to have to sort
through a long list of classes to get what I'm looking for.
Now, there's nothing stopping open from returning an object of class
FileHandle::OStream::Text::Exclusive or whatever. And that object can
have lots of useful methods to play with. But let me describe to open
what type of file I want, and let it sort it out.
Another part of me that resists this is that I don't necessarily agree
with a heavy OO paradigm. I've written several large projects in both a
OO environment, and non-OO. I have almost always found the OO paradigms
force me to convert what I wanted to do into something much more painful
than the non-OO methods. It typically breaks down into the moment you
want to do something to an object that the class designer did not take
into account, you basically have to either rebuild parts of the object
heirarchy from scratch, or get into really ugly things like declaring
everything a friend of each other, or having to many accessor method
calls you can't help but slow the whole program down.
Also, my experience is that when following a heavy OO paradigm, you
often fall into the trap of "There is only one way to do it. Why would
you ever want another?"
Is all OO bad? of course not. I use for several things, on a frequent
basis. Is something that's OO necessarily better than something that's
not? Despite the rumors from the Java crowd, no.
So while I embrace Perl6 having extremely strong OO capabilities, I will
argue strongly against it taking over the language.
>>If I cannot open a file for writing (permissions, out of space,
>>write locked, etc), I want to know the instant I attempt to open it
>>as such, _not_ when I later attempt to write to it. Having all
>>these features available to open as arguements seems a much better
>>idea to me. It's "Open a file with these specifications", not "Open
>>a file, and then apply these specifications to it".
>>
>>
>
>But why? Do you really open files and then perform an hour of work
>before attempting to use them? I'll argue that's not the normal case;
>rather, the normal case is something like
>
> open .... or die ...
> other_stuff()
> while (...) {
> print ...
> }
> close
>
>and the intervening delay (other_stuff) is negligible in wall-clock
>terms: when a failure occurs, the user hears about it immediately.
>
>
It's often that I'll open a file to make sure I can save my results,
then begin some process that is better measured in hours than seconds,
and then begin outputting. It's not infrequent for me to have a list of
five or six open statements at the start of a long process, and then
close them all at the end.
>>I do admit there is merit to your abstraction system, but IMO, it
>>belongs in a library.
>>
>>
>I think rather that the abstraction should be the default, and the
>individual "I don't trust Perl" functions should be available as
>separate entry points if the user explicitly requires them.
>
>
TMTOWTDI can apply here, I believe. You give me my way, I'll give you
yours. Leave me open with all my parameters, and you can have your list
of file abstraction classes. I could see having those classes part of
core, if there's enough support for them, and then something simple like
"use Files;" to turn them on.
Just my 2ข.
-- Rod
Larry Wall
: TMTOWTDI can apply here, I believe. You give me my way, I'll give you
: of file abstraction classes. I could see having those classes part of
: core, if there's enough support for them, and then something simple like
: "use Files;" to turn them on.
The reality will be somewhere in the middle--or maybe it's off to one
side or the other. I don't think anyone in this discussion (including
me) has a complete grasp of how roles will change the way OO is done in
Perl 6. When you open a filehandle, or create a string, or something
in between, you're just making an object with various capabilities.
With roles, you don't even necessarily have to specify a particular
class to construct. You can view an "open" statement as a request
for an object that fulfills certain roles, and let Perl search for
and/or compose an appropriate class. You don't have to care about some
artificial, predefined OO hierarchy because roles do an end run around
the use of subclassing for a task that subclassing isn't good for.
File handles are a classic case of "the god object" syndrome. With roles,
I think we can have our OO cake, and maybe even bear to eat it too.
Larry
Michele Dondi
> DW> my $text is TextFile("/tmp/bar");
> DW> $text = "hello"; # writes, truncates
> DW> $text ~= ", world\n"; # appends
>
> DW> $text.print "again\n"; # for old-times sake
>
> Anyhow we still need $text.flush() or $text.close() methods.
^^^^^^^^^^
Not necessarily if lexically scoped, just as in (recent enough) perl5...
Michele
--
>A question out of curiousity: who is this Green of Green's functions?
>Is he the same person of Green's theorem? :)
Yes. He was also an early environmentalist; hence the current
phrases "green" this and "green" that...
- David C. Ullrich on sci.math, thread "Who is Green?"
David Storrs
> truncate Vs append would be infered from usage (assign => truncate). One
> might be able to infer read Vs write in a similar way -- open the file based
> on the first access; re-open it (behind the scenes) if we write it after
> reading it.
You might run into issues if the user starts doing seeks before
writing...although maybe not, since that just means that we need to
(behind the scenes) remember the current location of the seek pointer
when re-opening.
Exclusivity issue: when it re-opens, should it lock before opening?
Race condition: what if something deletes the file between the moment
that perl closes the file and the moment that it re-opens it? Is
there a cross-platform way to do an atomic reopen?
FWIW, although I'm not sure it will work in practice, I really like
this idea of eliminating FileHandles as a user-level object.
> my Str $text is file("foo.txt") does no_follow_symlink does no_create;
>
> Do we have an antonym for C<does>?
How about 'no'?
my Str $text is file("foo.txt") no follow_symlink no create;
Other options (not all good):
without
not
dont
doesnt
--Dks
David Storrs
> Case 1:
> So I wanted to do a read/write scan, so I create my TextFile, start
> reading in data, so the file is opened for reading. Then, I come to the
> part where I want to update something, so I do a write command. Suddenly
> the file has to be closed, and then re-opened for read and write. And
> all my buffers, file pointers and the like are reset, (though curable
> with very careful planning), leaving me in a bad spot. Better if I could
> just declare the file open for read and write at open time.
>
> Case 2:
> I meant to use some critical data file in read-only mode, and accidently
> use a write command somewhere I didn't mean to, and silently just
> clobbered /etc/passwd. Better if I could have just opened the file read
> only, and trigger an error on the write command.
>
> What design philosophy would you envision TextFile taking to handle both
> these cases in a coherent fashion?
my $default is TextFile("/tmp/foo");
my $rw is TextFile("/tmp/foo") is rw;
my $ro is TextFile("/etc/passwd") is const;
$default will have the semantics that Dave has been
describing--initially opened read-only, then re-opened as r/w if you
write to it.
$rw is explicitly r/w. Attempts to write to it succeed, and do not
require an implicit re-open.
$ro is explicitly ro. Attempts to write to it fail, and do not
perform an implicit re-open.
Given the above code, here is some usage:
print $ro.readline(); # Prints first line
$ro = "boom!"; # THROWS AN ERROR
(assume error was trapped somehow)
# Print file, inefficiently
print $default.readline for 1..$default.lines;
# Append a line
$rw .= "an additional line\n";
# New line is visible through $rw
print $rw.lines[-1]; # (*)
# last line not yet visible through $default because it was
# added by external handle--just like in a tail -f, we
# need to move file pointer manually
$default.seek(File.current_pos);
# Can't re-open for write mode, since another handle
# already has the write-lock. Throw error to that effect.
$default = "oops, this doesn't work";
(*) The .lines[-1] semantic is feasible if we are willing to tolerate
very slow performance (at least the first time; it could cache the
locations of $/ after scanning and dump the cache if $/ changes), the
fact that it wouldn't work on all files (/dev/null, /dev/zero, etc),
and a few other issues.
> I don't think anyone (read: Larry) has declared exactly what the
> capabilities of the default file handle object are yet. It seems me that
> you could very well get what you want.
True on both counts.
--Dks
David Storrs
> I think part of the "mental jam" (at least with me), is that the
> read/write, exclusive, etc, are very critical to the act of opening the
> file, not only an after the fact restriction on what I can do later. If
> I cannot open a file for writing (permissions, out of space, write
> locked, etc), I want to know the instant I attempt to open it as such,
> _not_ when I later attempt to write to it. Having all these features
> available to open as arguements seems a much better idea to me. It's
> "Open a file with these specifications", not "Open a file, and then
> apply these specifications to it".
>
> I do admit there is merit to your abstraction system, but IMO, it
> belongs in a library.
>
> -- Rod Adams
First, why are they incompatible? Offer both, let TIMTOWTDI sort it
out.
Second, I would suggest that it NOT go in a library...this is
reasonably serious under-the-hood magic and should be integrated into
the core for efficiency.
--Dks
Deborah Pickett
> --- Rod Adams <r...@rodadams.net> wrote:
> > If I cannot open a file for writing (permissions, out of space,
> > write locked, etc), I want to know the instant I attempt to open it
> > as such, _not_ when I later attempt to write to it. Having all
> > these features available to open as arguements seems a much better
> > idea to me. It's "Open a file with these specifications", not "Open
> > a file, and then apply these specifications to it".
>
> But why? Do you really open files and then perform an hour of work
> before attempting to use them? I'll argue that's not the normal case;
> rather, the normal case is something like
>
> open .... or die ...
> other_stuff()
> while (...) {
> print ...
> }
> close
>
> and the intervening delay (other_stuff) is negligible in wall-clock
> terms: when a failure occurs, the user hears about it immediately.
The issue isn't about how long it takes other_stuff() to run. The issue is
whether other_stuff does something irrevocable that shouldn't have been done
if the file couldn't be opened.
Imagine a double-entry bookkeeping system that needs to atomically update two
files at once. You don't want to discover that one file is inaccessible
after you've already written the transaction to the other file. You'd end up
circumventing the DWIMmery by writing empty strings to the files just to make
sure they exist:
$handle1 = open "file1";
$handle2 = open "file2";
print $handle1: ""; # Did it really open?
print $handle2: "transaction 2" or die;
print $handle1: "transaction 1" or die;
I contend that however many examples you could come up with that make
"open-on-write" look neat, it's possible to contrive just as many examples
that make "open-on-write" look awkward. Which, IMHO, is a perfect argument
for putting this functionality into a library where those who want to use it,
will.
It's probably time to leave all of this up to @Larry to concoct something that
everyone's happy with.
--
Debbie Pickett
http://www.csse.monash.edu.au/~debbiep
deb...@csse.monash.edu.au
Dave Whipp
> Race condition: what if something deletes the file between the moment
> that perl closes the file and the moment that it re-opens it? Is
> there a cross-platform way to do an atomic reopen?
I'm not sure if you need to close it before you reopen it. You can usually
open the file a second time before closing it. (the only issue would be if
you were using mandatory locks, in which case you're probably a power-user
using the FileHandle module, anyway).
I don't know about the cross-platform aspect, but a similar scenario is that
the file changes on disk while we've using it. In most modern editors, the
user is asked: "file changed on disk: reload file? (Y/N)" when this happens.
I'd like to think that we could arrange for an exception to be thrown
(resumable, of course) if this happens when we've mapped a file into an
object.
Dave.
Dave Whipp
> # Print file, inefficiently
> print $default.readline for 1..$default.lines;
print it efficiently:
print $default;
> # Append a line
> $rw .= "an additional line\n";
$rw ~= "\n" unless $rw.chars[-1] eq "\n";
$rw ~= "an additional line\n";
> # New line is visible through $rw
> print $rw.lines[-1]; # (*)
>
> # last line not yet visible through $default because it was
> # added by external handle--just like in a tail -f, we
> # need to move file pointer manually
> $default.seek(File.current_pos);
I don't think the manual sync is really needed: name the method something a
bit more neutral:
$default.refresh;
> (*) The .lines[-1] semantic is feasible if we are willing to tolerate
> very slow performance (at least the first time; it could cache the
> locations of $/ after scanning and dump the cache if $/ changes), the
> fact that it wouldn't work on all files (/dev/null, /dev/zero, etc),
> and a few other issues.
The performance here could depend on the encoding. If the file is ASCII,
then we don't need to worry about multi-byte characters, so the
under-the-hood implementation could follow a heuristic such as "seek to 1000
bytes from end of file: scan forward for "\n". If none found, then go back
further. Otherwise continue to scan to find last "\n" in the file".
Dave.
Chromatic
> Second, I would suggest that it NOT go in a library...this is
> reasonably serious under-the-hood magic and should be integrated into
> the core for efficiency.
You must have amazingly fast hard drives.
-- c
Austin Hastings
I mount /tmp on swap. My "hard drive" is bitchin fast.
=Austin
Joseph Ryan
From: David Storrs <dst...@dstorrs.com>
Date: Monday, July 19, 2004 5:04 pm
Subject: Re: Why do users need FileHandles?
> Second, I would suggest that it NOT go in a library...this is
> reasonably serious under-the-hood magic and should be integrated into
> the core for efficiency.
How would integrating this in the core make it more efficient? Core or not, I'd see something like this being implemented with a custom pmc. I tend to think of inclusion in the core being more of a philosophical decision...
- Joe
Luke Palmer
Yes, it is. Whether or not a PMC is in the core, it should be equally
fast. That is, unless Parrot is allowed intimate knowledge of the PMC's
internals. And there are very few (any?) PMCs that possess this
property even now.
Even more philosophical is "what is core?" I get the impression that
Larry is trying to blur the distinction between core and non-core as
much as possible. So making it "go in the core" may just mean that it's
on the list of recommended modules to install.
Luke
Joseph Ryan
From: Luke Palmer <lu...@luqui.org>
Date: Thursday, July 22, 2004 2:48 pm
Subject: Re: Why do users need FileHandles?
>> JOSEPH RYAN writes:
> >
> > How would integrating this in the core make it more efficient? Core
> > or not, I'd see something like this being implemented with a custom
> > pmc. I tend to think of inclusion in the core being more of a
> > philosophical decision...
>
> Yes, it is. Whether or not a PMC is in the core, it should be equally
> fast. That is, unless Parrot is allowed intimate knowledge of the
> PMC'sinternals. And there are very few (any?) PMCs that possess this
> property even now.
But why would Parrot need to see the PMC's internals? I was thinking more along the lines of a class just looked just like PerlString, except with different, uh, stuff in the pmc class methods.
> Even more philosophical is "what is core?" I get the impression that
> Larry is trying to blur the distinction between core and non-core as
> much as possible. So making it "go in the core" may just mean
> that it's
> on the list of recommended modules to install.
You're probably right. Every time I think about the idea of "there will be no core modules", a conservative part of me becomes scared out of its mind, and that conservative side wants to riot. But, once I think about it a little more, that riot quickly gets beaten down with thought batons like: "Sure, there won't be any core modules, but most of what Perl5's core will be seamlessly built into the language anyways, and not having other core modules will free everyone from having to continue to use crusty old interfaces like Exporter and File::Find".
But, going back to the original topic, something like this kind of I/O abstraction should still be some sort of *module* (core or not), not a core *behaivor*.
Dan Hursh
How about we say "in core" means it packaged with the perl6 source and
covered in the coresponding camel & lama books. :)
Dan
Joseph Ryan
Well, that's what all of the ruckus is about.
There is a strong leaning towards including *no*
builtin modules with the core. So, that leaves only
the builtin functions and classes as "the core", and
so what is "in core" becomes a pretty big deal.
- Joe
David Green
>Even more philosophical is "what is core?"
I believe the standard definition is "Anything I want to use goes in
the core; anything everyone else wants goes wherever there's room
left over." ...
>So making it "go in the core" may just mean that it's
>on the list of recommended modules to install.
Does that mean having to "use Some::Module" to use it? If so, that
partially defeats the point of having a magical DWIMy shorthand in
the first place. (And if not -- then I guess it's irrelevant to most
end users whether it's "really" a module or not.)
I think stringified filehandles should be "core" -- i.e. always
available without having to install or include anything -- because
they're exactly the sort of easy thing that Perl is supposed to make
trivial. And of course, since there are plenty of cases where
pretending a file is just a string won't work, Perl should also make
the harder cases easy by including plain old 'open' in The Core as
well. [Not] using one doesn't in any way get in the way of using the
other, and they'd both be commonly used, so I think they both need to
be readily available.
In fact, my general attitude towards random feature $foo is: useful
to lots of people who want to use it + not harmful to people who
don't = put it all in the core.
-David "at least until starting perl begins
to take longer than your coffee break" Green
Luke Palmer
> >So making it "go in the core" may just mean that it's
> >on the list of recommended modules to install.
>
> Does that mean having to "use Some::Module" to use it?
Not necessarily. Glop, on which I'm doing a presentation at OSCON (have
to plug it sometime ;-), makes use of an idiom where you don't have to
'use' everything you want. That's because the small module quanta used
in the design would make the first 50 lines of your program 'use'
statements.
I should expect a similar thing in Perl 6 if traits are going to be
packaged up in their own modules.
> If so, that partially defeats the point of having a magical DWIMy
> shorthand in the first place. (And if not -- then I guess it's
> irrelevant to most end users whether it's "really" a module or not.)
Here's one defining feature of core:
if $feature.has_shorthand {
$feature.in_core = !$feature.requires_use;
}
Sorry. Got tired of English.
> I think stringified filehandles should be "core" -- i.e. always
> available without having to install ...
There's a lot of stuff like that. Way too much to include in a
distribution. That's why we're going to distribute close to nothing
with Perl 6 in order to force vendors to install a decent module suite.
It's likely that CPAN will have a Bundle::EYEWTIBWATA. [1]
> In fact, my general attitude towards random feature $foo is: useful
> to lots of people who want to use it + not harmful to people who
> don't = put it all in the core.
And there's our second system syndrome. Fortunately, Perl is embracing
its second system syndrome.
I think with a sensible auto-inclusion system (really, they're quite
nice--if it can be done at compile time), we'll be able to keep things
out of core and still make them feel like they're in core. There's
something to be said for modules like that.
Luke
[1] Everything You Ever Wanted To Install But Were Afraid To Ask
Aaron Sherman
> Well, that's what all of the ruckus is about.
> There is a strong leaning towards including *no*
> builtin modules with the core. So, that leaves only
> the builtin functions and classes as "the core", and
> so what is "in core" becomes a pretty big deal.
I don't think so.
Do any C++ programmers consider the STL to be anything other than "in
core" even though it's not part of the compiler, and with at least GCC,
it's distributed as a separate component?
Do any C programmers consider strlen or sprintf to be outside of the
core? It's part of the ANSI C *STANDARD LIBRARY*, not the ANSI C spec.
Define "outside of the core", please.
If I was going to make a recommendation (which I guess I am), I would
suggest having 4 layers of library:
1. Really-really core (the stuff you can't write the other stuff
without), and is probably all built-ins in Parrot.
2. Really core. This is the sort of "standard library". Just the most
essential bits that are required for general Perl usability. You'd
probably include most of these, even in a "trimmed down" release, such
as an OS installer
3. The "extended core". The modules from some CPAN-alike that are
considered to be "supported Perl extensions" In Perl 5, I think of
libwww-perl and libnet this way, but making an official distinction lets
users decide what modules to rely on in code that will have to be
"updatable" for 5 years.
4. The CPAN-like grab-bag of goodies.
--
Aaron Sherman <a...@ajs.com>
Senior Systems Engineer and Perl Toolsmith
http://www.ajs.com/~ajs/resume.html
Joseph Ryan
packaged with Perl itself". Things you'd define as
"part of the language." I/O stuff, threading stuff,
standard types, builtin functions, etc. And yeah,
most of that stuff will be written natively in C,
PIR, or be part of parrot itself.
I think there will also be a much larger "standard
library" that will be bundled separately, which would
contain your libwww, libnet, and other
widely/commonly used modules. The way I think of it
is kinda like installing something with the windows
installer: you can check to install the "minimal" or
standard version, the "full" version (which would be
perl + the "standard library"), or you could even
check "custom" and choose what you'd like to
install. And, of course, everything else
would just be on the CPAN.
- Joe
Jonadab The Unsightly One
> Well, that's what all of the ruckus is about. There is a strong
> leaning towards including *no* builtin modules with the core.
Surely, at bare minimum, there must be something included in core to
allow things that are not in core to be easily installed, the
equivalent of what CPAN.pm is for Perl5 (hopefully even better, and
hopefully without dependencies on external non-Perl things like gcc).
I could live with very little else in core, but the means to install
whatever is not in core is essential; that's the thing that allows
everything else to be made optional -- because it can be installed
easily enough.
Oh, and here's me resisting the urge to suggest that use ought to
automatically install from the CPAN anything that isn't present, as a
core behavior right out of the box.
--
$;=sub{$/};@;=map{my($a,$b)=($_,$;);$;=sub{$a.$b->()}}
split//,"ten.thgirb\@badanoj$/ --";$\=$ ;-> ();print$/
Brent 'Dax' Royal-Gordon
> Surely, at bare minimum, there must be something included in core to
> allow things that are not in core to be easily installed, the
> equivalent of what CPAN.pm is for Perl5 (hopefully even better, and
I believe that's the current plan--the core will include CPAN, LWP, and
not much else.
> hopefully without dependencies on external non-Perl things like gcc).
Don't think it'll be possible for modules that have C components,
although Parrot's Native Call Interface ought to make a lot of XS uses
obsolete. (With NCI, Parrot provides opcodes to load a shared library,
then retrieve functions from it and treat them like normal Parrot
subs--all without writing any C. That includes converting arguments and
return values from Parrot types to C types.)
> Oh, and here's me resisting the urge to suggest that use ought to
> automatically install from the CPAN anything that isn't present, as a
> core behavior right out of the box.
Security nightmare.
--
Brent "Dax" Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker
Oceania has always been at war with Eastasia.
David Storrs
> Jonadab the Unsightly One wrote:
>
> >Oh, and here's me resisting the urge to suggest that use ought to
> >automatically install from the CPAN anything that isn't present, as a
> >core behavior right out of the box.
>
> Security nightmare.
Definitely. On the other hand...I find myself wondering if we could
offer a pragma so that people can have the option if they want it.
For example:
#!/usr/bin/perl6
#use warnings; # Note that I am NOT explicitly using these
#use strict;
{ no 'warnings'; no 'strict'; # These must be explicitly turned off...
no installation_security; # or this would throw warning & error
use SomeModule; #
use OtherModule; #
use Foo; # If these are not installed,
use Bar; # they will be auto-installed.
use Baz; #
use Jaz; #
}
use FrobNitz; # If this is not installed, the script fails.
<script goes here>
__END__
Voila, this is now a completely portable-with-zero-effort(*) script.
Anywhere that I install it, it will ensure that all of its prereqs are
available before starting...and, if they aren't, it will install them
instead of failing. That's pretty darn cool.
And I had to make a concerted effort to enable this behavior. Really,
if the user wants to shoot himself in the foot, who are we to
interfere?
Note that there are still plenty of issues remaining--how to handle
varying module versions springs to mind--but those are implementation
details.
--Dks
(*) Note that here, "zero-effort" means "assuming you have a net
connection, plus all necessary installation tools, plus the rights to
install, plus your system is configured correctly to use the
installation diretory (if not in the standard paths), etc".
Brent 'Dax' Royal-Gordon
> #!/usr/bin/perl6
>
> #use warnings; # Note that I am NOT explicitly using these
> #use strict;
>
> { no 'warnings'; no 'strict'; # These must be explicitly turned off...
> no installation_security; # or this would throw warning & error
>
> use SomeModule; #
> use OtherModule; #
> use Foo; # If these are not installed,
> use Bar; # they will be auto-installed.
> use Baz; #
> use Jaz; #
> }
>
> use FrobNitz; # If this is not installed, the script fails.
>
> <script goes here>
>
> __END__
This is really something you want to enable or disable per-run, IMHO (so
e.g. the first time you run the script, you run it with "perl -Minstall"
or something). Or you could just make it part of the 'cpan' program:
brent:~$ wget www.somesite.com/somescript.cgi.gz
brent:~$ gunzip somescript.cgi.gz
brent:~$ cpan --prereqs=somescript.cgi
brent:~$ cp somescript.cgi /www/cgi-bin
brent:~$ chmod 755 /www/cgi-bin/somescript.cgi
# somescript.cgi is now ready for use
This would require 'cpan' to parse the script with a modified grammar
that noted all the 'use's (and 'require's, I guess), then install each
module. Or something like that.
Hmm...maybe this could be done for Perl 5...
Larry Wall
: This would require 'cpan' to parse the script with a modified grammar
: module. Or something like that.
:
: Hmm...maybe this could be done for Perl 5...
If the main impediment is security, you have to know what the
checksums of the modules you're looking for are, and that means you
have to know exactly what version you're looking for (or have info
on all versions). Which implies that you might need something like
Perl 6's ability to (attempt to) use two different versions of the
same module simultaneously.
But don't let me discourage you from playing with it in Perl 5... :-)
Larry
Dan Hursh
> Do any C++ programmers consider the STL to be anything other than "in
> core" even though it's not part of the compiler, and with at least GCC,
> it's distributed as a separate component?
>
> Do any C programmers consider strlen or sprintf to be outside of the
> core? It's part of the ANSI C *STANDARD LIBRARY*, not the ANSI C spec.
Does anyone really ship compilers w/o a standard library. I don't care
if they are in separate tar/rpm/etc files. As long as they go together.
Does anyone other than a spec writer document the language minus the
library?
> Define "outside of the core", please.
I'm sticking to my definition involving the camel book and packing them
together.
> If I was going to make a recommendation (which I guess I am), I would
> suggest having 4 layers of library:
>
> 1. Really-really core (the stuff you can't write the other stuff
> without), and is probably all built-ins in Parrot.
The toy for the researcher. Not much of a tool yet, but fun.
> 2. Really core. This is the sort of "standard library". Just the most
> essential bits that are required for general Perl usability. You'd
> probably include most of these, even in a "trimmed down" release, such
> as an OS installer
What most venders will ship.
> 3. The "extended core". The modules from some CPAN-alike that are
> considered to be "supported Perl extensions" In Perl 5, I think of
> libwww-perl and libnet this way, but making an official distinction lets
> users decide what modules to rely on in code that will have to be
> "updatable" for 5 years.
What my site IT people might get around to installing in the site image
for perl 6 during the perl 8.2 time frame.
> 4. The CPAN-like grab-bag of goodies.
The pre-pre-alpha modules that work on two-ish platforms. Many of which
were in the same state a year ago.
OK, maybe I'm over stating that. But I do think that if perl 6.X.Y is
not accompanied by a packaged perl library version I.J.K that is as
documented, tested and portable as the language, then most people won't
see any modules outside of those shipped in the perl6 source archive. I
hate that.
To put it another, if a library version is not done in concert with a
given language version, then either users can't rely on the availability
or exact behavior of any function beyond the core package. Each vendor
with have to make their own library, and test it.
Overlooking the duplication of effort for each vender, given that
vendors seem to have enough trouble packaging remotely recent versions
of perl 5, and given that different venders might not choose the same
selection of modules or that they might not pick compatible versions of
the same modules, this just looks like a black eye waiting to happen.
This all assumes that venders put the effort into it at all.
If not the venders, do you expect each perl programmer to have root
-like access? I could have missed the chapter, but I didn't see a good
way for a perl programmer to install from CPAN to his home directory and
use it seemlessly with an OS shipped perl install. So far my experience
do it with CPAN haven't gone well anyone. Maybe each user could compile
his own interpreter from source. That's how I had my best luck. I
still get lots of errors on CPAN installs. It's kind of a pain.
Besides, things should be run out of a user's home directory in a
production environment.
I know this isn't a new problem, but it is why I didn't use Net::FTP at
work until about a year ago. They just didn't install the bugger and I
didn't have control of the site image. The same goes for perl5.6 for
that matter, so I might not see perl6 at work before I retire in some 20
years from now and this will all be moot.
I just think there ought to be a decent library. It worked well for
python and java (even inspite of how java's library is designed!) and it
would put perl ahead of the likes of C and C++ (which I believe have far
too skinny standard libraries to be useful).
I also think that library needs to be tested as well as perl6. It needs
to be equally portable as well. (Yes I know there are limits there. If
TK won't run on my watch, I'll understand. I'm sure the watch probably
won't support all the core features either.)
And a given version of the library needs to 'go together' with a given
version of the language. The gcc folks may not tar the C++ standard lib
up with the compiler, but there tends to be a given version of one to
use with a given version of the other.
Of course, maybe there are other solutions. Maybe I'm wrong for
thinking it ought to be a requirement. Maybe I just need to work at a
shop with more perl biggots on staff.
Dan
Larry Wall
: >2. Really core. This is the sort of "standard library". Just the most
: >essential bits that are required for general Perl usability. You'd
: >probably include most of these, even in a "trimmed down" release, such
: >as an OS installer
:
: What most venders will ship.
Which is basically why we are planning not to produce one of these.
I think we should concentrate on 1 and 3.
Larry
Dan Hursh
I know. In that case, I think venders are liable to be lazy or peeved
and ship option 1 or worse, nothing. Is the answer really supposed to
be CPAN or no library? Is there a plan to at least have stable
snapshots of CPAN? Sorry to be a pest on this, but the implications
actually scare me about things I didn't worry about before because I had
confidence that they would be addressed in 'the library'. Maybe I just
need to sit and meditate some more.
Dan
Larry Wall
Sounds like you're confusing #3 with #4.
Larry
David Green
>Not necessarily. Glop, on which I'm doing a presentation at OSCON
>(have to plug it sometime ;-),
"Game Language on Perl, you say? Goodness, what's that??"
>Sorry. Got tired of English.
=)
>There's a lot of stuff like that. Way too much to include in a
>distribution. That's why we're going to distribute close to nothing
>with Perl 6 in order to force vendors to install a decent module suite.
>It's likely that CPAN will have a Bundle::EYEWTIBWATA. [1]
Well, it certainly makes sense to me to have perl itself as
minimal as possible, as long as there's an "official" bundle so that
you can make reasonable assumptions about what "everyone" has.
>I think with a sensible auto-inclusion system (really, they're quite
>nice--if it can be done at compile time), we'll be able to keep things
>out of core and still make them feel like they're in core. There's
>something to be said for modules like that.
Yup, something very good.
>And there's our second system syndrome. Fortunately, Perl is
>embracing its second system syndrome.
Works for me. Anyway, Perl already got a complete rewrite for P5,
right? So this would actually be Third System Syndrome, and thus
we've nothing to worry about. (I'm not worried, anyway.)
-David "impatient, maybe, but not worried" Green
Dan Hursh
> Sounds like you're confusing #3 with #4.
>
> Larry
Could be. Does 3 mean at install time, you down load the latest of the
'supported' packages from the CPAN alike or is it more like versioned
snapshots? (Possibly yet to be decided?) It's the idea of a standard
library being open to daily change that scaring me most at the moment.
If that's not likely, then me other concerns, well, aren't.
Dan
Jonadab The Unsightly One
>> hopefully without dependencies on external non-Perl things like gcc).
>
> Don't think it'll be possible for modules that have C components,
I'm really hoping Perl6 will be sufficiently powerful that C
components won't be needed or wanted.
>> Oh, and here's me resisting the urge to suggest that use ought to
>> automatically install from the CPAN anything that isn't present, as
>> a core behavior right out of the box.
>
> Security nightmare.
*shrug*
Running untrusted code (Perl or otherwise) on your system that you
haven't examined first is inherently a security nightmare. If it
wants to download things from wherever and install them it can
(assuming that you're running under an account that has privileges to
install or at least has its own install location just for that
account), and that's not all it can do, not by a longshot.
Or did you mean that someone might compromise the CPAN in order to
compromise all the systems downloading stuff from it? Again, that's
already true now, in theory. (More likely, a single mirror would be
compromised. One would hope it would be discovered quickly.)
However, I wasn't serious about making use automatically install.
Well, not altogether serious. CPAN.pm or something like it is really
almost good enough, and what it lacks can be added in without any
change to the behavior of use. Installing once before you use is
really not a very large burden in the scheme of things.
I was, however, very pleased to hear that we won't have to reinstall
half the contents of the CPAN every time we upgrade Perl or Parrot.
Definitely looking forward to that improvement.
Michele Dondi
> Hmm...maybe this could be done for Perl 5...
I guess it could be relatively easily done by means of the code-in-@INC
feature present in recent perls...
Michele
--
> A mathematically-inclined glove-fetishist [...]
What a wonderful introduction to a puzzle!
- Richard Heathfield in sci.math, "Re: Gloves in the dark"
Jim Cromie
> It's likely that CPAN will have a Bundle::EYEWTIBWATA. [1]
>
>[1] Everything You Ever Wanted To Install But Were Afraid To Ask
>
>
>
EYEWTIBWATL eye-witty-bwattle
.. But Were Always Too Lazy
Nicholas Clark
> Race condition: what if something deletes the file between the moment
> that perl closes the file and the moment that it re-opens it? Is
> there a cross-platform way to do an atomic reopen?
No.
To the best of my knowledge there isn't a way on Unix to reopen the same
file descriptor read-write using an existing read-only file descriptor.
Therefore there can't be a cross-platform way to do it, as it's not even
possible everywhere.
You can't store the filename of the file you originally opened, as someone
is free to move it and replace it with a symlink to a critical system file,
to try 0wn your system.
Opening needs to be done once at open time.
Nicholas Clark
Nicholas Clark
> Definitely. On the other hand...I find myself wondering if we could
> offer a pragma so that people can have the option if they want it.
> For example:
>
>
> #!/usr/bin/perl6
#!/usr/bin/perl
> #use warnings; # Note that I am NOT explicitly using these
> #use strict;
>
> { no 'warnings'; no 'strict'; # These must be explicitly turned off...
> no installation_security; # or this would throw warning & error
use Acme::Intraweb;
> use SomeModule; #
> use OtherModule; #
> use Foo; # If these are not installed,
> use Bar; # they will be auto-installed.
> use Baz; #
> use Jaz; #
> }
However, Acme::Intraweb hasn't been updated for a while, whereas CPANPLUS
has, so I'm not sure if it still works. Both are by Jos Boumans.
Nicholas Clark