OK, I want to create a nested list in Lisp (always of only integers) from a text file, such that each line in the text file would be represented as a sublist in the 'imported' list.
> OK, I want to create a nested list in Lisp (always of only integers) from a > text file, such that each line in the text file would be represented as a > sublist in the 'imported' list.
> OK, I want to create a nested list in Lisp (always of only integers) from a > text file, such that each line in the text file would be represented as a > sublist in the 'imported' list.
(defun make-list-from-file (filename) (with-open-file (in filename) (loop for line = (read-line in nil nil) while line collect (read-from-string (concatenate 'string "(" line ")"))))) CL-USER> (make-list-from-file "tmp") ((3 10 2) (4 1) (11 18))
Drew Krause <drkra...@mindspring.com> writes: > OK, I want to create a nested list in Lisp (always of only integers) from a > text file, such that each line in the text file would be represented as a > sublist in the 'imported' list.
(defun split (delim line) "Return a list of subsequences separated by a one character delimiter, delim itself is not returned" (loop :for mark = 0 :then (1+ point) :for point = (position delim line :start mark) :collect (subseq line mark point) :while point))
(defun make-list-from-text (fn) "assemble list of lists of numbers parsed from file, posed by drkrause on cll 2009-01-16" (with-open-file (stream fn) (loop :for line = (read-line stream nil nil) :while line :collect (mapcar #'parse-integer (split #\Space line)))))
> Drew Krause <drkra...@mindspring.com> writes: > > OK, I want to create a nested list in Lisp (always of only integers) from a > > text file, such that each line in the text file would be represented as a > > sublist in the 'imported' list.
> On Jan 16, 2:29 pm, Drew Krause <drkra...@mindspring.com> wrote:
> > OK, I want to create a nested list in Lisp (always of only integers) from a > > text file, such that each line in the text file would be represented as a > > sublist in the 'imported' list.
w_a_x_...@yahoo.com wrote: > On Jan 16, 2:29 pm, Drew Krause <drkra...@mindspring.com> wrote: > > OK, I want to create a nested list in Lisp (always of only > > integers) from a text file, such that each line in the text file > > would be represented as a sublist in the 'imported' list.
I wonder why everyone suggest one's own split-sequence function instead of pointing OP to the existing library. There are more interesting tasks for lisp coding that reinventing a wheel.
> I wonder why everyone suggest one's own split-sequence function > instead of pointing OP to the existing library.
My thinking was that if the OP can't manage writing this particular bit of code, they probably haven't worked out ASDF yet. I expected that someone would point it out.
> There are more interesting tasks for lisp coding that reinventing a > wheel.
Compared to SPLIT-SEQUENCE, a caster.
And answering on c.l.l. is already a form of recreation (pun intended) ;)
Here's a interesting toy problem posted by Drew Krause to comp.lang.lisp:
------------------------ On Jan 16, 2:29 pm, Drew Krause wrote [paraphrased a bit]:
OK, I want to create a nested list in Lisp (always of only integers) from a text file, such that each line in the text file would be represented as a sublist in the 'imported' list.
example of a file's content
3 10 2 4 1 11 18
example of programing behavior (make-list-from-text "blob.txt") => ((3 10 2) (4 1) (11 18))
----------------- Here's a emacs lisp version:
(defun read-lines (file) "Return a list of lines in FILE." (with-temp-buffer (insert-file-contents file) (split-string (buffer-substring-no-properties 1 (point-max)) "\n" t) ) )
(defvar mylist '() "result list" ) (setq mylist '()) ; init in case eval'd again
The above coding style is a typical maintainable elisp.
In a show-off context, it can be reduced to by about 50%, but still far verbose than ruby or say perl (which is 1 or 2 lines. (python would be 3 or 5)).
Note that the result element is string, not numbers. There's no easy way to convert them on the fly. 3 or so more lines will be needed to do that.
The “read-lines” function really should be a built-in part of elisp. It is not so mostly because emacs people have drawn themselves into a elitist corner, closing off to the outside world. (i am sending a suggestion to the official emacs dev list on adding read-line now.)
This problem and ruby code illustrates 2 fundamental problems of lisp, namely, the cons problem, and the nested syntax pain. Both of which are practically unfixable.
The lisp's cons fundamentally makes nested list a pain to work with. Lisp's nested syntax makes functional sequencing cumbersome.
In the ruby code, its post-fix sequential notation (as a side effect of its OOP notation) brings out the beauty of functional sequencing paradigm (sometimes known as functional chain, sequencing, filtering, unix piping).
its list, like all modern high level langs such as perl, php, python, javascript, don't have the lisp's cons problem. The cons destroys the usability of lists up-front, untill you have some at least 2 full-time years of coding lisp to utilize cons properly. (and even after that, it is still a pain to work with, and all you gain is a bit of speed optimization in rare cases that requires largish data, most of which has better solutions such as a database.)
Both of these problems i've published articles on.
For more detail on the cons problem, see the section “The Cons Business” at
> Here's a interesting toy problem posted by Drew Krause to > comp.lang.lisp:
> ------------------------ > On Jan 16, 2:29 pm, Drew Krause wrote [paraphrased a bit]:
> OK, I want to create a nested list in Lisp (always of only integers) > from a text file, such that each line in the text file would be > represented as a sublist in the 'imported' list.
> example of a file's content
> 3 10 2 > 4 1 > 11 18
> example of programing behavior > (make-list-from-text "blob.txt") => ((3 10 2) (4 1) (11 18))
> This problem and ruby code illustrates 2 fundamental problems of lisp, > namely, the cons problem, and the nested syntax pain. Both of which > are practically unfixable.
*Of course* Xah is wrong, as always. Who would expect anything else? In the Lisp style Clojure for example one does exactly the same as Jillian James (JJ) did in Ruby: (map #(map (fn [s] (Integer/parseInt s)) (.split % "\\s")) (line-seq (reader "blob.txt")))
This is slightly longer, simply because the functions have longer names. Integer/parseInt vs to_i
Also the function split takes a regexp, so I have to add the "\\s" here. I don’t know though if Jillians version also handles any kind of whitespac per line. And the last thing is, that the function reader returns a BufferedReader. So in general this is more valuable in real programs as opposed to one- line scripts. If we combine line-seq and reader into readline and apply the two other changes we would say: (map #(map (fn [s] (to_i s)) (.split %)) (readlines "blob.txt")) vs IO.readlines("blob.txt").map{|line| line.split.map{|s| s.to_i }}
So, obviously Xah is far away from any reality. It took him hours of his life to write up all his ideas about Lisp. Sad to see that all that time was wasted. All of it was wrong. Poor Xah :(
And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store data for his programs in the format 3 10 2 4 1 11 18
but instead as (3 10 2) (4 1) (11 18)
And then read it back into the program with: (map read-string (line-seq (reader "blob.txt")))
world.lisp.de> wrote: > Why not try something new? No need to allocate lines, split the and > then parse integer from the parts.
> (defun read-lines (stream &aux result list (rt (copy-readtable))) > (flet ((newline-reader (stream char) > (declare (ignore stream char)) > (when list > (push (nreverse list) result) > (setf list nil)) > nil)) > (set-macro-character #\newline #'newline-reader nil rt) > (let ((*readtable* rt)) > (loop for item = (read stream nil stream) > until (eq item stream) > when item do (push item list) > finally (when list (push (nreverse list) result))) > (nreverse result))))
I agree, no need to read and re-read everything, like all the split people want. The lisp reader can be made to take care of this so easily. But why does everyone want to traverse the data at least 2 times. (in the above case 3, but the last push/nreverse could be trivially changed to a collect to avoid that)
CommonLisp give us READ-DELIMITED-LIST - we don't need to define it, or bring in a package to do it. Lisp readers are highly optimized, let's let it do the work for us.
Now unfortunately, read-delimited-list reads right though whitespace so we can't trivially say (read-delimited-list #\newline stream) as it won't see the newline like we want it to.
But that's easy enough to fix, let's just change the readtable temporarily to think that newline is a terminating, non-constituent character. What else is one of those? right-parenthesis will do, so let's just make it behave like that temporarily. And since we don't want to do this over and over and over again, let just store it in a variable (it could be in a closure around the function too if you wanted to be fancier, and if you didn't ever need this readtable for anything else, but... that's for another day).
Ok now we have a readtable, lets make a function to read a single line...
Inside the function, we'll temporarily switch to our new readtable, so that we can stop when we see a newline. Also, since read-delimited- list doesn't fail gracefully, we will have to catch the EOF condition it generates. (There might be an easier/shorter way to do that, but this works, and just returns the stream when EOF is encountered. Why can't it return nil? OP didn't say what should happen with an empty line, e.g. just a newline character, so assuming that is valid, and since we have to look for something anyway, it may as well be the stream.)
(with-open-file (s *data-file*) (loop for item = (read-list-insides s) until (eq item s) collect item))
There it is - one pass, and barely any code.
Questions for the reader/OP:
1. if you don't really want to collect all the values/lines but just operate on them one at a time, how could you change the loop call, or what else could you do.
2. if you do need to collect them all, is it worth moving the loop inside of the let for the readtable and the handler-bind. Does that buy you anything?
3. what happens if there isn't an newline at the end of the file / at the end of the last sequence of numbers? does this code expect there to be one?
On Jan 17, 12:30 pm, André Thieme <address.good.until.
2009.may...@justmail.de> wrote: > In the Lisp style Clojure for example one does exactly the same as > Jillian James (JJ) did in Ruby: > (map #(map (fn [s] (Integer/parseInt s)) (.split % "\\s")) (line-seq > (reader "blob.txt")))
Note that you have nested map. This is a problem of function chaning as i detailed.
The other problem of cons, is commonly seen. Just about every week, we see some perhaps beginning lisper asking how to do various trivial list manipulation problem, which you don't see such question or frequency in any of modern high level lang forms.
The frequently asked list manipulation question we see include how to append, prepend, or anything involving nested list such as partitioning, reordering sublists, getting leaves, interpretation of leaves, etc. This is caused by the cons.
associated lisp problem compound the issue. Namely, the not exactly regular syntax, and the eval model of sometimes symbol sometimes uneval'd symbol, e.g. “'(...)”, “`(...,@ ...)” things.
The clojure example you gave above, apparently inherited the irregular syntax problem. (you see the #, [], % things, which is breaks the lisp's sexp idea) Also, all modern lisp basically all get fucked up by inheriting the cons (e.g. clojure, NewLisp, Arc lisp, Liskell). Often, they change the semantic a bit apparently as a mesaure towards solving the cons problem. (and Qi Lisp creates a huge bag of ad hoc, extremely ugly, syntax soup)
Advice: if you are creating a lispy lang, two things:
• stick to a _pure_ nested syntax, no exception whatsoever. (e.g. no more ` ' # % shits) If you want, add a transparent layer on top to support arbitrary algeraic notation. (e.g. look at Mathematica, CGOL)
• get rid of the cons. (you can still implement the so-called linked list, but no where it should be shown to the programer)
On Jan 17, 2:30 pm, André Thieme <address.good.until.
2009.may...@justmail.de> wrote: > And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store > data for his programs in the format > 3 10 2 > 4 1 > 11 18
> but instead as > (3 10 2) > (4 1) > (11 18)
Maybe he doesn't get to designate the format he has to read?
But really, why wouldn't a Lisper consider that format? If it really is just sequences of numbers you're just adding a lot of funny parenthesis in your data file, making it larger for no good reason, when you already have delimiters.
Assuming he does get to pick the format and it is just numbers, why not just store them as sequences of bytes/integers etc. instead of text?
All that said, with usual combined goals of laziness (for the effort of the programmer) and the desire to have something (mostly) human readable/editable, I usually go with the format you are advocating.
But, I think people are selling lisp short / forgetting the tools they have, when they assume or promote that there needs to be parenthesis in a data file for lisp code to read it easily or efficiently.
> On Jan 17, 2:30 pm, André Thieme <address.good.until. > 2009.may...@justmail.de> wrote: >> And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store >> data for his programs in the format >> 3 10 2 >> 4 1 >> 11 18
>> but instead as >> (3 10 2) >> (4 1) >> (11 18)
> Maybe he doesn't get to designate the format he has to read?
> But really, why wouldn't a Lisper consider that format? If it really > is just sequences of numbers you're just adding a lot of funny > parenthesis in your data file, making it larger for no good reason, > when you already have delimiters.
If it were really like that, if those parenthesis would be there for no good reason, then it would make no sense, I agree. But they are there for a reason. Their presence makes it easier to read them back. No splitting needed, no conversion to integers, not two mappings. And writing out the data is also trivial. For small amounts of data that’s all we need. If we want bigger datasets there is always a DB system waiting.
> On Jan 17, 2:30 pm, André Thieme <address.good.until.
> 2009.may...@justmail.de> wrote: > > And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store > > data for his programs in the format > > 3 10 2 > > 4 1 > > 11 18
> > but instead as > > (3 10 2) > > (4 1) > > (11 18)
> Maybe he doesn't get to designate the format he has to read?
> But really, why wouldn't a Lisper consider that format? If it really > is just sequences of numbers you're just adding a lot of funny > parenthesis in your data file, making it larger for no good reason, > when you already have delimiters.
> Assuming he does get to pick the format and it is just numbers, why > not just store them as sequences of bytes/integers etc. instead of > text?
> All that said, with usual combined goals of laziness (for the effort > of the programmer) and the desire to have something (mostly) human > readable/editable, I usually go with the format you are advocating.
> But, I think people are selling lisp short / forgetting the tools they > have, when they assume or promote that there needs to be parenthesis > in a data file for lisp code to read it easily or efficiently.
> at least, that's my two cents, > Kevin
If I would make it easy for Lisp, then I would make it just one list:
> Note that you have nested map. This is a problem of function chaning > as i detailed.
Yes, Jillian also has nested maps: IO.readlines("blob.txt").map{|line| line.split.map{|s| s.to_i }}
> The other problem of cons, is commonly seen. Just about every week, we > see some perhaps beginning lisper asking how to do various trivial > list > manipulation problem, which you don't see such question or frequency > in any of modern high level lang forms.
You see questions about trivial problems every week in all forums about programming languages. It has nothing to do with conses. Wrong road.
> The frequently asked list manipulation question we see include how to > append, prepend, or anything involving nested list such as > partitioning, reordering sublists, getting leaves, interpretation of > leaves, etc. This is caused by the cons.
Yes, same with all containers in all programming languages.
> The clojure example you gave above, apparently inherited the irregular > syntax problem. (you see the #, [], % things, which is breaks the > lisp's sexp idea)
My code used 8 “mysterious symbols”: ( ) # [ ] . " %
The Ruby version had these 7: ( ) | { } . "
And of course, neither in Ruby nor Clojure they break any idea.
> Also, all modern lisp basically all get fucked up by > inheriting the cons (e.g. clojure, NewLisp, Arc lisp, Liskell). Often, > they change the semantic a bit apparently as a mesaure towards solving > the cons problem. (and Qi Lisp creates a huge bag of ad hoc, extremely > ugly, syntax soup)
Funny. What you write is an english text with words that most people can understand. You trick them into thinking that it actually makes sense what you say. It is so technical, that a random would believe it. But what you really say is about as meaningful as saying that it is impossible to write good programs in Python, because it adds whitespace to its syntax.
a idiot wrote: > Yes, Jillian also has nested maps:
the issue here, is not about whether Ruby has nested map or not. It is about illustrating a lisp problem. In particular, nested syntax impedes the functional programing paradigm of function chaining.
a idiot wrote: > My code used 8 “mysterious symbols”: > ( ) # [ ] . " %
> The Ruby version had these 7: > ( ) | { } . "
The issue here is not about which lang employs more special chars. The issue is about the irregularities in lisp's syntax damaged its power of its otherwise believed superior regular syntax.
2009.may...@justmail.de> wrote: > K Livingston schrieb:
> > On Jan 17, 2:30 pm, André Thieme <address.good.until. > > 2009.may...@justmail.de> wrote: > >> And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store > >> data for his programs in the format > >> 3 10 2 > >> 4 1 > >> 11 18
> >> but instead as > >> (3 10 2) > >> (4 1) > >> (11 18)
> > Maybe he doesn't get to designate the format he has to read?
> > But really, why wouldn't a Lisper consider that format? If it really > > is just sequences of numbers you're just adding a lot of funny > > parenthesis in your data file, making it larger for no good reason, > > when you already have delimiters.
> If it were really like that, if those parenthesis would be there for > no good reason, then it would make no sense, I agree. > But they are there for a reason. > Their presence makes it easier to read them back. > No splitting needed, no conversion to integers, not two mappings.
The read-delimited-list solution above works in a single pass and is not hindered in the slightest by the absence of parentheses. (and it's only a few lines of code)
> And writing out the data is also trivial.
for the data format above mapping (format stream "~{~S~^ ~}~%" list-of-numbers) seems pretty trivial.
or if you really want it in one line (format stream "~:{~@{~S~^ ~}~%~}" list-of-list-of-numbers)
Anyway, all I've been trying to point out is that the absence or presence of parentheses has pretty much no bearing on ones ability to read, manipulate, or print data. (The belief that it does comes up often.) For the most part any straightforward data format is as trivial as any other to do whatever you want with. The lisp reader by default reads s-expressions, but that doesn't mean that's all you can input in lisp. A few tweaks here or there to the reader covers most of the cases I've ever seen. CommonLisp gives you a lot of power by exposing all these tools to you, and I, for one, would rather go back to my toolbox and pick up my screwdriver, than sit there hammer in hand and say "why are all these screws in my way."
On Jan 17, 10:25 am, Tino Wildenhain <t...@wildenhain.de> wrote:
> > [[int(x) for x in line.split()] for line in open("blob.txt")]
Nice (python code).
Few comments:
• the above code is borderline of atypical. e.g. it is not a average python code would produce or one'd seen in corporate python code.
• voodoo like the above makes me dislike python. To me, the one advantage of python is its clarity enforced by its syntax. Specifically, the forced indendation and quite simple semantics. However, the way i've seen Guido's propensities and how python 3 is moving to, it is becoming more mumbo jumbo of computer sciency OOP jargons with syntax soup. (with iterators, enumerators, list comprehension... shits forced upon the users)
The above line illustrate well the ad hoc syntax soup nature python is moving into.
> On Jan 16, 9:36 pm, "jos...@corporate-world.lisp.de" <jos...@corporate- > world.lisp.de> wrote: > > Why not try something new? No need to allocate lines, split the and > > then parse integer from the parts.
> > (defun read-lines (stream &aux result list (rt (copy-readtable))) > > (flet ((newline-reader (stream char) > > (declare (ignore stream char)) > > (when list > > (push (nreverse list) result) > > (setf list nil)) > > nil)) > > (set-macro-character #\newline #'newline-reader nil rt) > > (let ((*readtable* rt)) > > (loop for item = (read stream nil stream) > > until (eq item stream) > > when item do (push item list) > > finally (when list (push (nreverse list) result))) > > (nreverse result))))
...
> CommonLisp give us READ-DELIMITED-LIST - we don't need to define it, > or bring in a package to do it. Lisp readers are highly optimized, > let's let it do the work for us.
...
> 3. what happens if there isn't an newline at the end of the file / at > the end of the last sequence of numbers? does this code expect there > to be one?
I thought about the read-delimited-list version, too. But it does not return a list when it sees an eof during read. So a line with numbers with no newline at the end can't be read with it.
It might (?) be useful to have another version of read-delimited list, where the end of the list could be determined by other factors: amount of items read, sequence as end marker, eof, eol, predicate on reading an item ... It might even have a filter than does include only those elements that pass a filter.
André Thieme wrote: > K Livingston schrieb: > > On Jan 17, 2:30 pm, André Thieme <address.good.until. > >2009.may...@justmail.de> wrote: > > > And btw, Drew Krause is just a Lisp newbie. No Lisper would ever > > > store data for his programs in the format > > > 3 10 2 > > > 4 1 > > > 11 18
> > Maybe he doesn't get to designate the format he has to read?
> > But really, why wouldn't a Lisper consider that format? If it > > really is just sequences of numbers you're just adding a lot of > > funny parenthesis in your data file, making it larger for no good > > reason, when you already have delimiters.
> If it were really like that, if those parenthesis would be there for > no good reason, then it would make no sense, I agree. > But they are there for a reason. > Their presence makes it easier to read them back. > No splitting needed, no conversion to integers, not two mappings. > And writing out the data is also trivial.
In other words, don't try to do anything in Lisp that isn't trivial. The data has to be changed to suit Lisp; the Lisp program can't be changed to suit the data.
> For small amounts of data that’s all we need. If we want bigger > datasets there is always a DB system waiting.
For non-trivial tasks, don't attempt to use Lisp. It's too CLumsy. Use a database.
André Thieme wrote: > Xah Lee schrieb: > > comp.lang.lisp,comp.lang.scheme,comp.lang.functional,comp.lang.pytho > > n,comp.lang.ruby
> > Here's a interesting toy problem posted by Drew Krause to > > comp.lang.lisp:
> > ------------------------ > > On Jan 16, 2:29 pm, Drew Krause wrote [paraphrased a bit]:
> > OK, I want to create a nested list in Lisp (always of only integers) > > from a text file, such that each line in the text file would be > > represented as a sublist in the 'imported' list.
> > example of a file's content
> > 3 10 2 > > 4 1 > > 11 18
> > example of programing behavior > >(make-list-from-text "blob.txt") => ((3 10 2) (4 1) (11 18))
> > This problem and ruby code illustrates 2 fundamental problems of > > lisp, namely, the cons problem, and the nested syntax pain. Both of > > which are practically unfixable.
> *Of course* Xah is wrong, as always. > Who would expect anything else? > In the Lisp style Clojure for example one does exactly the same as > Jillian James (JJ) did in Ruby: > (map #(map (fn [s] (Integer/parseInt s)) (.split % "\\s")) (line-seq > (reader "blob.txt")))
That fails when numbers are separated by more than one space. And what about leading or trailing space?
> This is slightly longer, simply because the functions have longer > names. Integer/parseInt vs to_i
> Also the function split takes a regexp, so I have to add the "\\s" > here. I don’t know though if Jillians version also handles any > kind of whitespac per line.
irb(main):003:0> puts " foo \t bar " foo bar => nil irb(main):004:0> " foo \t bar ".split => ["foo", "bar"] irb(main):005:0> "foo...bar?-!hoo".split /\W+/ => ["foo", "bar", "hoo"]
> And the last thing is, that the function reader returns a > BufferedReader. So in general this is more valuable in real programs > as opposed to one- line scripts. If we combine line-seq and reader > into readline and apply the two other changes we would say: > (map #(map (fn [s] (to_i s)) (.split %)) (readlines "blob.txt"))
That is not a complete program.
> vs > IO.readlines("blob.txt").map{|line| line.split.map{|s| s.to_i }}
> So, obviously Xah is far away from any reality. > It took him hours of his life to write up all his ideas about Lisp. > Sad to see that all that time was wasted. All of it was wrong. > Poor Xah :(
> And btw, Drew Krause is just a Lisp newbie. No Lisper would ever store > data for his programs in the format > 3 10 2 > 4 1 > 11 18
> but instead as > (3 10 2) > (4 1) > (11 18)
Perhaps the data was stored by someone else. Understand?
> And then read it back into the program with: > (map read-string (line-seq (reader "blob.txt")))
You make a very strong case that Lisp is very feeble at processing data. I'm almost convinced.
Ruby isn't feeble, so data like this is fine:
shall we begin? or lotus135? 1984 times! The 3 stooges: COBOL,LISP,FORTRAN. 3.14, si11y L00KING