LISP style question - (with-open-file)

[ccc31807]

I've read that a C programmer will write LISP as C. as i recall that
is attributed to Paul Graham. I seem to be in that category (except
I'm using Perl). I can't figure out what a proper style for LISP is
(or maybe there isn't and I'm just thinking that there is).

This involves the very common Perl task of munging data -- opening
files as input to read data, grabbing the data and manipulating it,
and opening files as output to write data. Here is a skeleton of some
typical Perl template that opens two files for input and opens tow
files for output:

my (%data_hash_in, %data_hash_out);
open IN1, '<', 'firstinput.txt';
while (<IN1>)
{
first_stuff_data_into_hash(); #using %data_hash_in
}
close IN1;

open IN2, '<', 'secondinput.txt';
while (<IN2>)
{
second_stuff_data_into_hash(); #using %data_hash_in
}
close IN2;

%data_hash_out = munge_data(%data_hash_in);

open OUT1, '>', 'csvoutput.csv';
open OUT2, '>', 'pdfoutput.pdf';
foreach my $key (sort keys %data_hash_out)
{
print_to_csv($data_hash_out{$key});
print_to_pdf($data_hash_out{$key});
}
close OUT1;
close OUT2;

exit(0);

The LISP template that I'm attempting to use looks something like
this:

(with-open-file
(open-and-read-file-1)
(with-open-file
(open-and-read-file-2)
(with-open-file
(open-and-print-to-file-3)
(with-open-file
(open-and-print-to-file-4)))))

My mental image of Perl is a sequential set of steps, called one after
another, processing lines iteratively. My mental image of LISP is an
outer function calling inner functions processing lines recursively.
Either this is an incorrect mental image of LISP or I'm not yet
equipped to write in proper LISP style (most likely).

What would be an appropriate style for the case of opening two files
(say, a students file and a classes file), mashing the data together
(with the students info and the classes info and the class), and
writing to two output files (say, a report cards file as a PDF and a
statistical analysis file in CSV)?

Thanks, CC.

[Norbert_Paul]

The with-open-file macro can be considered a convenience macro for
stuff like

(let ((in1 (open "firstinput.txt" :direction :input)))
(unwind-protect
(first-stuff-data-into-hash in1)
(close in1)))

which then would read

(with-open-file (in1 "firstinput.txt" :direction :input)
(first-stuff-data-intohash in1))

What you have writte is no recursion but a statically (lexically)
nested expression, but you are right: It is possible to make recursive
function calls in Lisp.

I'd consider the nested expression appropriate style if you have a fixed
number of files to open and have to keep them open all together.
In your case I'd rather write it similar to your Perls code.


(with-open-file (in1 "firstinput.txt" :direction :input)
;; Not doing your homework here ..
)
;; .. nor here.

Norbert

[Barry Margolin]

In article
<fcce762a-70a6-47db...@a1g2000yqd.googlegroups.com>,

In this case, I think the Lisp style would be pretty similar to the Perl
style. There's no reason to nest the calls for the input files if you
don't need to access both streams concurrently. So it would be
something like:

(let ((hash-in (make-hash-table))
hash-out)
(with-open-file (in1 "firstinput.txt")
(first-stuff-data-into-hash in1 hash-in))
(with-open-file (in2 "secondinput.txt")
(second-stuff-data-into-hash in2 hash-in))
(setq hash-out (munge-data hash-in)
(with-open-file (out1 "csvoutput.csv")
(with-open-file (out2 "pdfout.pdf")
(maphash #'(lambda (key value)
(print-to-csv value out1)
(print-to-pdf value out2))
hash-out)))

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Sam Steingold]

> * ccc31807 <pneg...@tznvy.pbz> [2012-05-29 07:59:55 -0700]:

since you need files sequentially, there is no need to keep them all
open at the same time.

(let ((hash1 (make-hash-table))
(hash2 (make-hash-table)))
(with-open-file (in "firstinput")
(loop :for line = (read-line in nil nil) :while line
:do (first_stuff_data_into_hash hash1 line)))
(with-open-file (in "secondinput")
(loop :for line = (read-line in nil nil) :while line
:do (second_stuff_data_into_hash hash1 line)))
(munge_data hash1 hash2)
(with-open-file (csv "csvoutput" :direction :output)
(with-open-file (pdf "pdfoutput" :direction :output)
(loop :for value :being :each :hash-value :of hash2 :do
(print_to_csv csv value)
(print_to_pdf pdf value)))))

--
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://memri.org http://palestinefacts.org
http://openvotingconsortium.org http://mideasttruth.com http://ffii.org
Kleptomania: the ability to find stuff even before its owner loses it.

[ccc31807]

On May 29, 11:46 am, Norbert_Paul <norbertpauls_spam...@yahoo.com>
wrote:

> What you have writte is no recursion but a statically (lexically)
> nested expression, but you are right: It is possible to make recursive
> function calls in Lisp.

Which actually responds to a question I didn't ask.

With Perl, you can manipulate a file line by line, which would process
a file that contains one record per line in a row oriented framework
(like a line in a spreadsheet or a row in a database table. Or, you
can use slurp mode to manipulate the entire file as one long string,
if (for example) you wanted to manipulate a text file.

In order to recurse on a CSV file, you would need to read the entire
file into memory as a list, with each record becoming a list element.

(defun munge-data (file)
(cond
((null file) nil)
(t (manipulate-record (car file))
(munge-data (cdr file)))))

To me, it's more intuitive, less space intensive, and maybe less time
intensive, to manipulate a file line by line iteratively that
recursively.

Comments?

CC.

[Madhu]

* ccc31807 <fcce762a-70a6-47db...@a1g2000yqd.googlegroups.com> :
Wrote on Tue, 29 May 2012 07:59:55 -0700 (PDT):

| open OUT1, '>', 'csvoutput.csv';
| open OUT2, '>', 'pdfoutput.pdf';
| foreach my $key (sort keys %data_hash_out)
| {
| print_to_csv($data_hash_out{$key});
| print_to_pdf($data_hash_out{$key});
| }
| close OUT1;
| close OUT2;

If you want to use multiple files without nesting calls to
with-open-file, you could use OPEN as in this sketch:

(let (out1 out2)
(unwind-protect
(progn
(setq out1 (OPEN "csvoutput.csv" :direction :output :if-exists
:supersede))
(setq out2 (OPEN "pdfoutput.pdf" :direction :output :if-exists
:supersede))

(loop for key being each hash-key of $data_hash_out
using (hash-value val)
do (print-csv val out1)
(print-pdf val out2)))
;; unwind forms
(unless (null out1) (close out1))
(unless (null out2) (close out2))))

Othertimes when you want to decouple OPEN from unwind-close pattern, you
can use the WITH-OPEN-STREAM macro, giving it the open stream; and the
macro which will automatically CLOSE the file, when the forms are
done. This just lets you avoid spelling out the UNWIND-PROTECT form
explicitly.

| My mental image of Perl is a sequential set of steps, called one after
| another, processing lines iteratively. My mental image of LISP is an
| outer function calling inner functions processing lines recursively.
| Either this is an incorrect mental image of LISP or I'm not yet
| equipped to write in proper LISP style (most likely).
|

file operations are the same in any language. The specific problem you
are trying to solve (beyond figuring out the file system syntax), may be
amenable to mapping solutions (i.e. map this function across all
records). However for the tasks you have exhibited here over the years,
what you could do with lisp (in employing this style), you can also do
in perl.

--- Madhu

[Pascal J. Bourguignon]

ccc31807 <cart...@gmail.com> writes:

> On May 29, 11:46 am, Norbert_Paul <norbertpauls_spam...@yahoo.com>
> wrote:
>> What you have writte is no recursion but a statically (lexically)
>> nested expression, but you are right: It is possible to make recursive
>> function calls in Lisp.
>
> Which actually responds to a question I didn't ask.
>
> With Perl, you can manipulate a file line by line, which would process
> a file that contains one record per line in a row oriented framework
> (like a line in a spreadsheet or a row in a database table. Or, you
> can use slurp mode to manipulate the entire file as one long string,
> if (for example) you wanted to manipulate a text file.
>
> In order to recurse on a CSV file, you would need to read the entire
> file into memory as a list, with each record becoming a list element.

There are several libraries to read or write CSV files.

(ql-dist:system-apropos "csv")

> (defun munge-data (file)
> (cond
> ((null file) nil)
> (t (manipulate-record (car file))
> (munge-data (cdr file)))))
>
> To me, it's more intuitive, less space intensive, and maybe less time
> intensive, to manipulate a file line by line iteratively that
> recursively.
>
> Comments?

Indeed. But how often do you have to process files bigger than the RAM
nowadays? (eg. I have 24GB of RAM in my workstation, and 4GB in my game
computer).

But yes, if you have to process multi-terabyte log files, better do it
in chunks.

clhs read-line


--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.

[Alex Mizrahi]

> In order to recurse on a CSV file, you would need to read the entire
> file into memory as a list, with each record becoming a list element.

No, you won't.

(with-open-file (frobla "a.txt")
(labels ((next ()
(let ((line (read-line frobla nil nil)))
(when line
(process-line line)
(next))))))
(next)))

I think it would be better for you to grab some tutorial and learn
something instead of asking stupid questions in group. I'm not judging,
but you're wasting your own and everybody else's time.

Please start with assumption that Lisp can do absolutely anything a
general purpose language can do (because it is a general purpose
language) and go though a couple of tutorials/books until you feel
comfortable.

Before that your "comments" and "conclusions" have absolutely no value
because you don't know anything about programming in Lisp. Is it hard to
understand?

[Barry Margolin]

In article
<c3646bd6-889d-4e32...@v9g2000yqm.googlegroups.com>,

I think most people process files iteratively. But if you wanted to
process a file recursively, you could:

(defun munge-data (stream)
(let ((line (read-line stream nil nil)))
(when line
(manipulate-record line)
(munge-data stream))))

If the implementation does tail recursion optimization, this should
compile into roughly the same code as the iterative version.

[WJ]

Using pipelining (in Racket):

(->> (make-hash)
(first-stuff-data-into-hash "firstinput.txt")
(second-stuff-data-into-hash "secondinput.txt")
(munge-data)
(print-to-csv "csvoutput.csv")
(print-to-pdf "pdfout.pdf"))


"->>" is borrowed from Clojure and is defined as:

(define-syntax ->>
(syntax-rules ()
[(_ x)
x]
[(_ x (y ...) z ...)
(->> (y ... x) z ...)]
[(_ x y z ...)
(->> (y x) z ...)]))

[ccc31807]

On May 29, 12:03 pm, Sam Steingold <s...@gnu.org> wrote:


> since you need files sequentially, there is no need to keep them all
> open at the same time.
>
> (let ((hash1 (make-hash-table))
>       (hash2 (make-hash-table)))
>   (with-open-file (in "firstinput")
>     (loop :for line = (read-line in nil nil) :while line
>       :do (first_stuff_data_into_hash hash1 line)))
>   (with-open-file (in "secondinput")
>     (loop :for line = (read-line in nil nil) :while line
>       :do (second_stuff_data_into_hash hash1 line)))
>   (munge_data hash1 hash2)
>   (with-open-file (csv "csvoutput" :direction :output)
>     (with-open-file (pdf "pdfoutput" :direction :output)
>       (loop :for value :being :each :hash-value :of hash2 :do
>         (print_to_csv csv value)
>         (print_to_pdf pdf value)))))

Thanks for this, and the others who replied. Yes, I can see how this
works now. Conceptually, it's not much different from what I would
expect to do that uses a main() function to run everything.

CC.

[Zach Beane]

ccc31807 <cart...@gmail.com> writes:

> Here is a skeleton of some typical Perl template that opens two files
> for input and opens tow files for output:
>
> my (%data_hash_in, %data_hash_out);
> open IN1, '<', 'firstinput.txt';
> while (<IN1>)
> {
> first_stuff_data_into_hash(); #using %data_hash_in
> }
> close IN1;

Perl programmers typically use IO::Handle objects instead of bare
filehandles like this.

If you're going to write LISP instead of Lisp, please be fair and write
PERL instead of Perl.

Zach

[ccc31807]

On May 30, 9:45 am, Zach Beane <x...@xach.com> wrote:
> Perl programmers typically use IO::Handle objects instead of bare
> filehandles like this.

Well ... yes. However, TIMTOWTDI, and after having used both styles, I
like the bareword style because it stands out better on the page. I
earn my daily bread by munging data with Perl, and the bareword style
works for me.

> If you're going to write LISP instead of Lisp, please be fair and write
> PERL instead of Perl.

Is there a right way and a wrong way? It makes no difference to me one
way or the other. I have some old books (predating CL) that always
write LISP, but I certainly don't mind writing Common Lisp if that's
the way to do it.

CC.

[Pascal J. Bourguignon]

LISP is LISP (1) or LISP 1.5.
Common Lisp or CL is Common Lisp.
COMMON-LISP or CL is the COMMON-LISP package.
Lisp or lisp is the generic name for the lisp family of languages.
Emacs lisp or elisp is Emacs lisp.
LISP, elisp or Common Lisp are lisp languages.

[ccc31807]

On May 30, 3:39 pm, "Pascal J. Bourguignon" <p...@informatimago.com>
wrote:

> LISP is LISP (1) or LISP 1.5.
> Common Lisp or CL is Common Lisp.
> COMMON-LISP or CL is the COMMON-LISP package.
> Lisp or lisp is the generic name for the lisp family of languages.
> Emacs lisp or elisp is Emacs lisp.
> LISP, elisp or Common Lisp are lisp languages.

perl is the name of the executable.
Perl is the name of the programming language.
PERL is either the Practical Extraction and Reporting Language or the
Pathologically Eclectic Rubbish Collector.

All in all, I'd rather see Common Lisp than LISP, and it suits me fine
to adopt that convention.

CC.