Beginner code - splitting lines on whitespace
Steve Allan
time. As an exercise, I'm trying to port some Perl code to lisp, and
the first bit of functionality I wrote was to read a file that looks
like this:
#------------------------------------------
# unique-label hostname home-dir
#------------------------------------------
mach1 host1 /home/me
mach2 host2 /export/home/me
mach3 host3 c:\\home\\me
And split each line into a list, returning a list of lists:
(("mach1" "host1" "/home/me")
("mach2" "host2" "/export/home/me")
("mach3" "host3" "c:\\home\\me"))
The code below does this, but I'm pretty sure there's much that could
be improved. To speed up my learning, I'd love feedback in either/both
of these areas.
1) Critique on the implementation - how to improve the code as
written.
2) Suggestions for a better approach altogether.
I realize the task I'm coding is pretty mundane, but I think I could
learn a lot of the basics by really getting this right, so your
critique would be most welcome.
Thanks!
Code is below.
--
-- Steve
;;;;================================================================
(defun get-platforms (file)
(with-open-file (platforms file)
(loop for line = (read-line platforms nil)
while line
unless (position #\# line)
collect (split-on-space line))))
(defun split-on-space (string)
"Splits string on whitespace, meaning spaces and tabs"
(unless (null string)
(let ((space (or (position #\space string) (position #\tab string))))
(cond
(space (cons
(subseq string 0 space)
(split-on-space
(string-trim '(#\Space #\Tab) (subseq string space)))))
(t (list string))))))
(get-platforms "c:/projects/lisp/platforms.lst")
vanekl
check this out,
http://www.cliki.net/SPLIT-SEQUENCE
i think this would reduce your code by at least half
Ken Tilton
Not bad at all. (unless (null x)...) should be (when x...) I think you
might agree. You might avoid searching the string twice, one for tab and
once for space, by using position-if. And a lot of us have little macros
akin to Paul Graham's AIF. I have (untested):
(when string
(bif (delim-pos (position-if (lambda (c)
(or (char= c #\space)(char= c #\tab)))
string))
(cons (subseq string 0 delim-pos) ...etc...)
(list string))
BIF left as an exercise. :)
If you want to go nuts you can string-right-trim once at the outset and
then just string-left-trim thereafter.
hth,kt
--
http://smuglispweeny.blogspot.com/
http://www.theoryyalgebra.com/
ECLM rant:
http://video.google.com/videoplay?docid=-1331906677993764413&hl=en
ECLM talk:
http://video.google.com/videoplay?docid=-9173722505157942928&q=&hl=en
Thomas A. Russ
> I've been lurking here for a bit, and trying to learn lisp in my spare
> time. As an exercise, I'm trying to port some Perl code to lisp, and
> the first bit of functionality I wrote was to read a file that looks
> like this:
>
> #------------------------------------------
> # unique-label hostname home-dir
> #------------------------------------------
> mach1 host1 /home/me
> mach2 host2 /export/home/me
> mach3 host3 c:\\home\\me
>
> And split each line into a list, returning a list of lists:
>
> (("mach1" "host1" "/home/me")
> ("mach2" "host2" "/export/home/me")
> ("mach3" "host3" "c:\\home\\me"))
>
> The code below does this, but I'm pretty sure there's much that could
> be improved. To speed up my learning, I'd love feedback in either/both
> of these areas.
Actually, this looks generally pretty good.
> 1) Critique on the implementation - how to improve the code as
> written.
You could you the suggested SPLIT-SEQUENCE code or for a more full-blown
approach (especially if you are planning on porting more Perl code)
taking a look at the CL-PPCRE regular expression package
<http://www.weitz.de/cl-ppcre/>
> 2) Suggestions for a better approach altogether.
> ;;;;================================================================
> (defun get-platforms (file)
> (with-open-file (platforms file)
> (loop for line = (read-line platforms nil)
> while line
> unless (position #\# line)
> collect (split-on-space line))))
Good use of the WITH-OPEN-FILE macro. I will note that the code you
have here will throw away unprocessed all lines that contain "#"
anywhere in them. That means you can't use # to put in line comments
after data -- but perhaps that isn't a legal format. But it still seems
that it would be an issue.
> (defun split-on-space (string)
> "Splits string on whitespace, meaning spaces and tabs"
> (unless (null string)
> (let ((space (or (position #\space string) (position #\tab string))))
> (cond
> (space (cons
> (subseq string 0 space)
> (split-on-space
> (string-trim '(#\Space #\Tab) (subseq string space)))))
> (t (list string))))))
I would think you want to either define a WHITE-SPACE-P function or pass
in the delimiters to this function.
Note that the use of STRING-TRIM doesn't allow you to skip values in
your data format. So if there isn't any value for a column you can't
detect that because you remove all consecutive occurences of your
whitespace. Also, your current implementation doesn't do the trim until
the first recursive call. So that will remove multiple leading and
trailing spaces then, it won't remove an initial leading space. Try it
with
(split-on-space " This is fun! ")
Consider the following slightly modfied version, which optionally
collapses multiple delimiters.
(defun split-on-delimiter (string delimiters
&key (allow-multiple-delimiters t))
"Splits STRING on any character in the sequence DELIMITERS"
(unless allow-multiple-delimiters
(setq string (string-trim delimiters string)))
(flet ((delimiterp (char)
(find char delimiters :test #'char=)))
(unless (null string)
(let ((delimiter-position (position-if #'delimiterp string)))
(if delimiter-position
(cons (subseq string 0 delimiter-position)
(split-on-delimiter (subseq string (1+ delimiter-position))
delimiters
:allow-multiple-delimiters
allow-multiple-delimiters))
(list string))))))
Also, a lot of these type of operations tend to use iterative rather
than recursive solutions, exploiting the :START keyword to the POSITION
(or POSITION-IF) function. If you search back in the archives for
things like split-sequence, or look at the library implementation, you
can see how that would be done.
> (get-platforms "c:/projects/lisp/platforms.lst")
--
Thomas A. Russ, USC/Information Sciences Institute
Steve Allan
Kenny, thanks for pointing me to PG's aif. 'On Lisp' is a bit down on
my reading list so it would have taken me a long while to find that on
my own.
Combining your suggestions with some of Thomas', I came up with this:
(defmacro bif (test-form-binding then-form &optional else-form)
`(let ((,(car test-form-binding) ,(cadr test-form-binding)))
(if ,(car test-form-binding) ,then-form ,else-form)))
(defun white-space-p (char)
(or (char= char #\Space) (char= char #\Tab)))
;; trim both ends of string at the start, then just trim from
;; the left in recursive function
(defun split-on-space (string)
(labels ((split (string)
(when string
(bif (delim-pos (position-if #'white-space-p string))
(cons (subseq string 0 delim-pos)
(split (string-left-trim
'(#\Space #\Tab) (subseq string delim-pos))))
(list string)))))
(split (string-trim '(#\Space #\Tab) string))))
CL-USER> (split-on-space " This is fun! ")
("This" "is" "fun!")
It is indeed!
--
-- Steve
Ken Tilton
One of the nice things about defmacro is the destructuring fer free:
(defmacro bif ((bindvar boundform) yup &optional nope)
`(let ((,bindvar ,boundform))
(if ,bindvar
,yup
,nope)))
I get paid by the LOC. You also have &key and &optional in there, and
you can have as many destructuring forms as you like.
>
> (defun white-space-p (char)
> (or (char= char #\Space) (char= char #\Tab)))
>
> ;; trim both ends of string at the start, then just trim from
> ;; the left in recursive function
> (defun split-on-space (string)
> (labels ((split (string)
> (when string
> (bif (delim-pos (position-if #'white-space-p string))
> (cons (subseq string 0 delim-pos)
> (split (string-left-trim
> '(#\Space #\Tab) (subseq string delim-pos))))
> (list string)))))
> (split (string-trim '(#\Space #\Tab) string))))
>
> CL-USER> (split-on-space " This is fun! ")
> ("This" "is" "fun!")
>
> It is indeed!
>
It gets better. :)
Steve Allan
I finally got a chance to come back to this. Probably a boring topic
for experienced lispers, but this exercise has really taught me a lot.
t...@sevak.isi.edu (Thomas A. Russ) writes:
> Steve Allan <takez...@yahoo.com> writes:
>
<snip>
>> ;;;;================================================================
>> (defun get-platforms (file)
>> (with-open-file (platforms file)
>> (loop for line = (read-line platforms nil)
>> while line
>> unless (position #\# line)
>> collect (split-on-space line))))
>
> Good use of the WITH-OPEN-FILE macro. I will note that the code you
> have here will throw away unprocessed all lines that contain "#"
> anywhere in them. That means you can't use # to put in line comments
> after data -- but perhaps that isn't a legal format. But it still seems
> that it would be an issue.
Yeah, I kind of punted on that issue for now - at some point I'll have
to deal with it though.
<snip>
> Also, a lot of these type of operations tend to use iterative rather
> than recursive solutions, exploiting the :START keyword to the POSITION
> (or POSITION-IF) function. If you search back in the archives for
> things like split-sequence, or look at the library implementation, you
> can see how that would be done.
I wrote a white-space-p function as you suggested, and came up with
this iterative solution to split-on-space.
(defun white-space-p (c)
(or (char= c #\Space) (char= c #\Tab)))
(defun split-on-space (string)
(loop
for b = (position-if-not #'white-space-p string)
then (position-if-not #'white-space-p string :start e)
for e = (when b (position-if #'white-space-p string :start b))
while b
collect (subseq string b e)
while e))
I wrote a (do ...) version as well, but the loop version seems more
elegant, so I posted that one.
Thanks again for the feedback!
--
-- Steve
Wade Humeniuk
The stream paradigm can also be your friend. Just as you are reading
lines from the file, you can read "tokens" from a string.
(defun get-platforms (file)
(with-open-file (platforms file)
(loop for line = (read-line platforms nil)
while line
unless (position #\# line)
collect (with-input-from-string (s line)
(loop for token = (read-token s)
while token collect token)))))
(defun read-token (stream &optional (whitespace '(#\space)))
(with-output-to-string (token)
(loop with found-token = nil
for c = (read-char stream nil nil) do
(cond
((null c)
(if found-token
(loop-finish)
(return-from read-token nil)))
((member c whitespace)
(if found-token (loop-finish)))
(t
(setf found-token t)
(write-char c token))))))
CL-USER> (get-platforms "/home/wade/Lisp/platforms.txt")
(("mach1" "host1" "/home/me") ("mach2" "host2" "/export/home/me")
("mach3" "host3" "c:\\\\home\\\\me"))
CL-USER>
Wade
danb
> I'm trying to port some Perl code to lisp
Use cl-ppcre.
(defpackage :split (:use :cl :cl-ppcre))
(in-package :split)
(defun scan-file ()
(with-open-file (in "split-lines.in")
(loop for line = (read-line in nil nil)
while line
unless (scan "^\\s*(#.*)?$" line)
collect (split " +" line))))
SPLIT> (scan-file)
(("mach1" "host1" "/home/me")
("mach2" "host2" "/export/home/me")
("mach3" "host3" "c:\\\\home\\\\me"))
--Dan
------------------------------------------------
Dan Bensen http://www.prairienet.org/~dsb/
cl-match: expressive pattern matching in Lisp
http://common-lisp.net/project/cl-match/
Thomas A. Russ
> Thanks Thomas,
>
> I finally got a chance to come back to this. Probably a boring topic
> for experienced lispers, but this exercise has really taught me a lot.
Not really.
It was refreshing to see nice code from a new user.
> > Also, a lot of these type of operations tend to use iterative rather
> > than recursive solutions, exploiting the :START keyword to the POSITION
> > (or POSITION-IF) function. If you search back in the archives for
> > things like split-sequence, or look at the library implementation, you
> > can see how that would be done.
>
> I wrote a white-space-p function as you suggested, and came up with
> this iterative solution to split-on-space.
>
> (defun white-space-p (c)
> (or (char= c #\Space) (char= c #\Tab)))
>
> (defun split-on-space (string)
> (loop
> for b = (position-if-not #'white-space-p string)
> then (position-if-not #'white-space-p string :start e)
> for e = (when b (position-if #'white-space-p string :start b))
> while b
> collect (subseq string b e)
> while e))
This is a good implementation (I like loop).
I prefer not to intersperse WHILE clauses in the body of the Loop, but
the use inside Loop is conformant with the standard and doing both tests
does provide a solution to making sure you get the last item without
having to do surgery outside the loop construct.
WJ
It would have to get immeasurably better to stop stinking.
Arc:
arc> (tokens " This is fun! ")
Mark Tarver
>
> - Show quoted text -
That's not bad; but it doesn't completely answer the OPs request.
Needs to split at new lines.
Qi-Yacc + Qi
(define parse-file
File -> (compile <lines> (read-file-as-charlist File)))
(defcc <lines>
<line> #\Newline <lines> := [(parse-line <line>) | <lines>];
<line> := [(parse-line <line>)];)
(define parse-line
Line -> (read-chars-as-stringlist Line (= #\Space)))
(defcc <line>
<char> <line> := [<char> | <line>];
<e> := [];)
(defcc <char>
-*- := (if (= #\Newline -*-) #\Escape -*-);)
Create Richard III.txt
Now is the summer of our discontent,
made glorious summer by this house of York.
(12-) (time (parse-file "Richard III.txt"))
Real time: 0.0030002 sec.
Run time: 0.0 sec.
Space: 20136 Bytes
[["Now" "is" "the" "summer" "of" "our" "discontent,"]
["made" "glorious" "summer" "by" "this" "house" "of" "York."]]
Mark
Mark Tarver
>
> - Show quoted text -
beg pardon first 'summer' should be 'winter' - its the heat spell over
here in the UK - goes to the head in the nicest way.
WJ
Ruby:
" This is fun! ".split
==>["This", "is", "fun!"]
ccc31807
> I've been lurking here for a bit, and trying to learn lisp in my spare
> time. As an exercise, I'm trying to port some Perl code to lisp, and
> the first bit of functionality I wrote was to read a file that looks
> like this:
As one who uses Perl daily and is learning Lisp, I find this thread particularly interesting, so much so that I printed it out and took it home last night to study it in detail. Three things have occurred to me, call them meta-issues if you will, in inverse order:
3. I understand why one would want to learn a language by writing it like a language one already knows. I learned Java coming from a C background, and for a long time write Java like I wrote C. This certainly eases you into a new language, but I'm not sure that it's the best way to learn the new language.
2. I agree that something like this should use a regex, or more likely, a package designed to tokenize a string. I have found a couple of Lisp packages that parse CSV files, and it seems to be easier to use code that's already been developed and tested.
1. The only reason that you would read a file would be to use the data, and most likely you will use the discrete datums by transforming and manipulating them. Transforming the data in a file to a list doesn't seem to me to be useful at all. What would really be nice, for your little script, is a function that would place the data into some structure (in Perl, probably a multi-level hash, and in Java, probably objects) so you could munge the data.
Question: I am a little ashamed to admit that I haven't gotten this far, but I haven't quite figured this out. Given a string, you can tokenize the string and using destructuring-bind assign the values to specific variables. Using these variables, you can push them to an array, or a hash, or a series of objects for later use. Would this logic be the way you would write your Lisp code?
- open file for reading
- read each line
- while the file contains lines
- tokenize the line
- assign the tokens to distinct variables
- perform operations on variables as needed
- add the transformed variables to some data structure
- open the data structure for reading
- read each object
- while the DS contains objects
- assign the elements of the objects to distinct variables
- perform operations on variables as needed
- write the transformed variables to some IO device
Thanks, CC.
Thomas A. Russ
> Question: I am a little ashamed to admit that I haven't gotten this
> far, but I haven't quite figured this out. Given a string, you can
> tokenize the string and using destructuring-bind assign the values to
> specific variables. Using these variables, you can push them to an
> array, or a hash, or a series of objects for later use. Would this
> logic be the way you would write your Lisp code?
Well, if you were designing things from the start as a lisp project, you
wouldn't really do any string parsing yourself. Instead, you would
write your data files as lisp s-expressions (lisp data) directly and
then just read in the resulting data.
This allows you to leverage the fact that lisp already has a
serialization for a lot of its data structures and you can take
advantage of that to avoid having to write your own parser. [This is
very different from the perl development model where you are really
driven to use regular expressions to always parse your input. In lisp
you can get a lot of that by using the standard lisp reader.]
So, in essence you replace:
> - open file for reading
> - read each line
> - while the file contains lines
> - tokenize the line
> - assign the tokens to distinct variables
> - perform operations on variables as needed
> - add the transformed variables to some data structure
with this code, that essentially just calls the READ function.
(with-open-file (in filename :direction :input)
(read in))
If there are multiple forms in the data file, then you would write a
simple loop to collect them:
(with-open-file (in filename :direction :input)
(loop with eof-marker = (gensym)
for form = (read in nil eof-marker)
until (eql form eof-marker)
collect form))
> - open the data structure for reading
> - read each object
> - while the DS contains objects
> - assign the elements of the objects to distinct variables
> - perform operations on variables as needed
> - write the transformed variables to some IO device
And then you could perform whatever you need to do. This could involve
transforming the data in some fashion, or doing something else where the
input was read to setup and configure the system to do something else,
perhaps in conjunction with user interaction.
ccc31807
> Well, if you were designing things from the start as a lisp project, you
> wouldn't really do any string parsing yourself. Instead, you would
> write your data files as lisp s-expressions (lisp data) directly and
> then just read in the resulting data.
You don't really start from scratch in most cases, you have requirements and specifications. In my case, I almost always have a file (typically the result of a select statement from a database) that consists of rows that are delimited in some kind of way. What do you do, as a Lisp developer, when you must munge a data file that consists of several hundred rows, each row containing several dozen fields, delimited by an arbitrary character?
> This allows you to leverage the fact that lisp already has a
> serialization for a lot of its data structures and you can take
> advantage of that to avoid having to write your own parser. [This is
> very different from the perl development model where you are really
> driven to use regular expressions to always parse your input. In lisp
> you can get a lot of that by using the standard lisp reader.]
This sort of misses the point. Say that you had to read a number of text files in a directory (be they source listings or data files or just plain text) and modify them in some way, perhaps by changing several words, paginating the files or the reverse, and so on. Using Perl, I would slurp the entire file into memory and transform the entire file in one go with regexs. How would you handle this task in Lisp?
> So, in essence you replace:
>
> > - open file for reading
> > - read each line
> > - while the file contains lines
> > - tokenize the line
> > - assign the tokens to distinct variables
> > - perform operations on variables as needed
> > - add the transformed variables to some data structure
>
> with this code, that essentially just calls the READ function.
>
> (with-open-file (in filename :direction :input)
> (read in))
>
> If there are multiple forms in the data file, then you would write a
> simple loop to collect them:
>
> (with-open-file (in filename :direction :input)
> (loop with eof-marker = (gensym)
> for form = (read in nil eof-marker)
> until (eql form eof-marker)
> collect form))
What if your file consisted of lines that looked like thie?
Barack,H.,Obama,1600 Pennsylvania Ave.,"Washington, D.C.",12345,...
This line doesn't consist of Lisp forms, but a string of ASCII text that you've got to deal with.
I'm not studying Lisp in a vacuum, but in a context where I have a job to do. I write several scripts a week that follow the same idiom: open a file, read each line, tokenize the line and stuff the values into several different structures, close the file, and then transform the data. This is essentially what you would use an RDBMS for using SQL, but I have no need to store the data, update it, or perform queries against it -- all I do is read one file, munge the data, and write to another file.
Perhaps this is too trivial for Lisp?
> > - open the data structure for reading
> > - read each object
> > - while the DS contains objects
> > - assign the elements of the objects to distinct variables
> > - perform operations on variables as needed
> > - write the transformed variables to some IO device
>
> And then you could perform whatever you need to do. This could involve
> transforming the data in some fashion, or doing something else where the
> input was read to setup and configure the system to do something else,
> perhaps in conjunction with user interaction.
There's no doubt in my mind that Lisp can do everything that Perl can do, and more. I'm just having a very difficult time seeing it. Thank you very much for your response -- I'll continue to plug away.
CC.
Jason Earl
> On Thursday, May 12, 2011 11:40:40 AM UTC-4, Thomas A. Russ wrote:
>> Well, if you were designing things from the start as a lisp project, you
>> wouldn't really do any string parsing yourself. Instead, you would
>> write your data files as lisp s-expressions (lisp data) directly and
>> then just read in the resulting data.
>
> You don't really start from scratch in most cases, you have
> requirements and specifications. In my case, I almost always have a
> file (typically the result of a select statement from a database) that
> consists of rows that are delimited in some kind of way. What do you
> do, as a Lisp developer, when you must munge a data file that consists
> of several hundred rows, each row containing several dozen fields,
> delimited by an arbitrary character?
I am also a newbie to Lisp, but Common Lisp has libraries for regular
expressions and CSV files. The advantage of using Common Lisp is that
you have other tricks as well. For example, you can turn your data into
a domain specific language.
Here's an example, of taking a written specification and lispifying it
into runnable code.
http://lispm.dyndns.org/mov/dsl-in-lisp.mov
>> This allows you to leverage the fact that lisp already has a
>> serialization for a lot of its data structures and you can take
>> advantage of that to avoid having to write your own parser. [This is
>> very different from the perl development model where you are really
>> driven to use regular expressions to always parse your input. In
>> lisp you can get a lot of that by using the standard lisp reader.]
>
> This sort of misses the point. Say that you had to read a number of
> text files in a directory (be they source listings or data files or
> just plain text) and modify them in some way, perhaps by changing
> several words, paginating the files or the reverse, and so on. Using
> Perl, I would slurp the entire file into memory and transform the
> entire file in one go with regexs. How would you handle this task in
> Lisp?
You can do the same thing in lisp, using very similar methods, and you
even have the advantage of having a built in REPL. This allows you to
experiment on your data set in easy steps. Heck, I came to Common Lisp
from Emacs Lisp, where I because accustomed to doing this sort of thing
in my text editor.
Not to speak for Thomas, but I am sure that his point is that Lisp
basically has a built-in serialization format. Programmers deal with
CSV files and such because other languages don't have that luxury, and
so we end up with a multitude of ad-hoc formats or formats like XML that
are basically s-expressions without parentheses.
This does not mean that Lisp *can't* deal with CSV files, or XML files,
or whatever random text that you might throw at it. Just that if you
aren't dealing with a legacy serialization format you might want to
consider using leveraging Lisp's built in reader.
> Perhaps this is too trivial for Lisp?
>
>> > - open the data structure for reading
>> > - read each object
>> > - while the DS contains objects
>> > - assign the elements of the objects to distinct variables
>> > - perform operations on variables as needed
>> > - write the transformed variables to some IO device
>>
>> And then you could perform whatever you need to do. This could
>> involve transforming the data in some fashion, or doing something
>> else where the input was read to setup and configure the system to do
>> something else, perhaps in conjunction with user interaction.
>
> There's no doubt in my mind that Lisp can do everything that Perl can
> do, and more. I'm just having a very difficult time seeing it. Thank
> you very much for your response -- I'll continue to plug away.
Watch the video. It did an excellent job of showcasing Lisp's unique
features. You also might want to take a look at Chapter 25 of
/Practical Common Lisp/. That chapter deals with parsing binary files,
but it shows how Lisp allows you to change the language to fit the
problem at hand.
Jason