Q: on hashes and counting
The Glauber
the first 5 characters of each line in this file are a vendor code.
Then i want to ouptut all vendor codes and how many times
they show up in the input.
Here's what i did:
(let ((curv) (vhash (make-hash-table :test 'equal)))
(with-open-file (s "invdat.txt" :direction :input)
; read until end of file
(do ((l (read-line s) (read-line s nil 'eof)))
((eq l 'eof)
(maphash #'(lambda (k v) (format t "~A,~A~%" k v)) vhash))
; get the first 5 characters of each line
(setq curv (subseq l 0 5))
; if already in hash, increment count, else, create hash entry
(if (gethash curv vhash)
(incf (gethash curv vhash))
(setf (gethash curv vhash) 1)) )))
This is all straightforward, except for these lines:
(if (gethash curv vhash)
(incf (gethash curv vhash))
(setf (gethash curv vhash) 1))
What they do is, if there is already an entry in the hash, increment
the count. If there isn't, create one (with count set to 1).
Is this the best way to use a hash? In other words, do i really have to
do the (gethash key hash) twice? It feels wasteful.
In Perl, you do $hash{key}++ and the entry will get "magically" created if
it isn't there. (Of course, the Perl compiler may be doing the
double lookup behind my back).
The Perl version of this program runs a lot faster than the Lisp version
(6 times faster) (I'm using Clisp and AIX), but i'm assuming that
the main problem with the Lisp version is the fact that each line of text
is consed and thrown away to the garbage collector. I'm pretty sure
Perl would reuse the memory in this case.
Anyway, comparing Perl and Lisp is not the point here, i just want
to understand if i'm using the hashes right.
Thanks,
glauber
P.S.: the Perl version is a "one liner":
$c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print "$x,$c{$x}\n"}}
(very obvious, no? :-))
--
Glauber Ribeiro
thegl...@my-deja.com http://www.myvehiclehistoryreport.com
"Opinions stated are my own and not representative of Experian"
Sent via Deja.com http://www.deja.com/
Before you buy.
The Glauber
Here's another try:
Lisp:
(let ((curv) (vhash (make-hash-table :test 'equal)))
(with-open-file (s "invdat.txt" :direction :input)
; read until end of file
(do ((l (read-line s) (read-line s nil 'eof)))
((eq l 'eof)
(maphash #'(lambda (k v) (format t "~A,~A~%" k v)) vhash))
; get the first 5 characters of each line
(setq curv (subseq l 0 5))
; if already in hash, increment count, else, create hash entry
(if (gethash curv vhash)
(incf (gethash curv vhash))
(setf (gethash curv vhash) 1)) )))
Perl:
$c{substr($_,0,5)}++;
END { foreach $x (sort keys %c) {print "$x,$c{$x}\n"}}
--
Barry Margolin
The Glauber <thegl...@my-deja.com> wrote:
>Is this the best way to use a hash? In other words, do i really have to
>do the (gethash key hash) twice? It feels wasteful.
GETHASH allows you to specify a default value, so you can do:
(incf (gethash curv vhash 0))
>In Perl, you do $hash{key}++ and the entry will get "magically" created if
>it isn't there. (Of course, the Perl compiler may be doing the
>double lookup behind my back).
Actually, it's doing something more like the above code. If an entry in a
hash table doesn't exist, $hash{key} returns Perl's "undef" value, which
coerces to 0 in a numeric context.
>The Perl version of this program runs a lot faster than the Lisp version
>(6 times faster) (I'm using Clisp and AIX), but i'm assuming that
>the main problem with the Lisp version is the fact that each line of text
>is consed and thrown away to the garbage collector. I'm pretty sure
>Perl would reuse the memory in this case.
Perl is heavily optimized for string processing like this, since that's
primarily what it's used for.
--
Barry Margolin, bar...@genuity.net
Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Raymond Wiker
> This is all straightforward, except for these lines:
> (if (gethash curv vhash)
> (incf (gethash curv vhash))
> (setf (gethash curv vhash) 1))
(incf (gethash curv vhash 0))
--
Raymond Wiker
Raymon...@fast.no
Pierre R. Mai
> Suppose i have a text file and i want to count strings in it. Specifically,
> the first 5 characters of each line in this file are a vendor code.
> Then i want to ouptut all vendor codes and how many times
> they show up in the input.
(with-open-file (stream "invdat.txt")
(loop with count-hash = (make-hash-table :test #'equal)
for line being the lines of stream
for product-code = (subseq line 0 5)
do
(incf (gethash product-code count-hash 0))
finally
(maphash #'(lambda (k v) (format t "~A,~A~%" k v))
count-hash)))
Of course this depends on an new LOOP clause. If you have a LOOP
macro that is based on the MIT LOOP code (e.g. CMU CL, ACL), you can
use the definition at the end of the article...
But to answer your original question: You can use the default value
of gethash to make incf just work on non-existant keys...
> (let ((curv) (vhash (make-hash-table :test 'equal)))
> (with-open-file (s "invdat.txt" :direction :input)
> ; read until end of file
> (do ((l (read-line s) (read-line s nil 'eof)))
> ((eq l 'eof)
> (maphash #'(lambda (k v) (format t "~A,~A~%" k v)) vhash))
> ; get the first 5 characters of each line
> (setq curv (subseq l 0 5))
> ; if already in hash, increment count, else, create hash entry
> (if (gethash curv vhash)
> (incf (gethash curv vhash))
> (setf (gethash curv vhash) 1)) )))
I'd write that as
(with-open-file (s "invdat.txt")
(do ((count-hash (make-hash-table :test #'equal))
(line (read-line s nil nil) (read-line s nil nil)))
((null line)
(maphash #'(lambda (k v) (format t "~A,~A~%" k v)) count-hash))
(let ((product-code (subseq line 0 5)))
(incf (gethash product-code count-hash 0)))))
There are two IMHO important differences between my version and yours:
- Don't ever use setq to set a variable when you can just as well use
let to bind it (setq of curv vs. let of product-code, where the last
one could be folded into the gethash).
- Don't use magic EOF values in places where a simple nil will do just
as well. Magic EOF values are only really required in read&co., not
in read-line.
- Use longer and more precise variable names... Yes, I'm often guilty
of using cryptic names myself... ;)
> The Perl version of this program runs a lot faster than the Lisp version
> (6 times faster) (I'm using Clisp and AIX), but i'm assuming that
> the main problem with the Lisp version is the fact that each line of text
> is consed and thrown away to the garbage collector. I'm pretty sure
> Perl would reuse the memory in this case.
Should this really turn out to be a performance problem (which I
doubt), you could write your own read-line which re-used an internal
buffer, and only returned the first 5 characters.
> P.S.: the Perl version is a "one liner":
> $c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print "$x,$c{$x}\n"}}
> (very obvious, no? :-))
Hmmm, my Perl might be quite rusty (which is a good thing), but you
seem to be missing the open, the while(<STREAM>) and close here, which
would add at least one line ;)
Regs, Pierre.
;;;; This defines a loop path for the lines of a stream.
(in-package :ANSI-LOOP)
;;; Syntax: loop for var being the lines of stream [split-by char]
;;; Syntax: loop for var being each line in stream [split-on char]
(defun loop-lines-iteration-path (variable data-type prep-phrases)
(cond
((cddr prep-phrases)
(loop-error "Too many prepositions!"))
((null prep-phrases)
(loop-error "Missing OF or IN in ~S iteration path."))
((null (cdr prep-phrases))
(unless (member (caar prep-phrases) '(:of :in))
(loop-error "Missing OF or IN in ~S iteration path."))
(unless (symbolp variable)
(loop-error "Destructuring is not valid for line iteration."))
(let ((stream-var (loop-gentemp 'loop-line-)))
`(((,variable nil ,data-type) (,stream-var ,(cadar prep-phrases)))
()
()
()
(not (setq ,variable (read-line ,stream-var nil nil)))
())))
;; We are splitting
((consp variable)
(let ((stream-var (loop-gentemp 'loop-line-))
(by-var (loop-gentemp 'loop-line-by-))
(line-var (loop-gentemp 'loop-line-line-))
(step-var (loop-gentemp 'loop-line-step-)))
`(((,variable nil ,data-type) (,stream-var ,(cadar prep-phrases))
(,by-var ,(cadadr prep-phrases))
(,line-var nil)
(,step-var nil))
()
()
()
(not (when (setq ,line-var (read-line ,stream-var nil nil))
(setq ,step-var (pmlib:partition ,by-var ,line-var
:remove-empty-subseqs t))))
(,variable ,step-var))))
(t
(let ((stream-var (loop-gentemp 'loop-line-))
(by-var (loop-gentemp 'loop-line-by-))
(line-var (loop-gentemp 'loop-line-line-)))
`(((,variable nil ,data-type) (,stream-var ,(cadar prep-phrases))
(,by-var ,(cadadr prep-phrases))
(,line-var nil))
()
()
()
(not (when (setq ,line-var (read-line ,stream-var nil nil))
(setq ,variable (pmlib:partition ,by-var ,line-var
:remove-empty-subseqs t))))
())))))
(add-loop-path '(line lines) 'loop-lines-iteration-path *loop-ansi-universe*
:preposition-groups '((:of :in) (:split-by :split-on))
:inclusive-permitted nil)
;;; This depends on the following partition function in package pmlib:
(in-package :PMLIB)
;;; The following functions are based on the versions by Arthur
;;; Lemmens of the original code by Bernard Pfahringer posted to
;;; comp.lang.lisp. I only renamed and diddled them a bit.
(defun partition (delimiter seq
&key (maximum nil)
(remove-empty-subseqs nil)
(from-end nil)
(start 0)
(end nil)
(test nil test-supplied)
(test-not nil test-not-supplied)
(key nil key-supplied))
"Return a list of subsequences in seq delimited by delimiter.
If :remove-empty-subseqs is true, empty subsequences will be discarded
from the result; otherwise they will be included.
If :maximum is supplied, the result will contain no more than :maximum
possibly empty subsequences. The second result value contains the
unsplit rest of the sequence.
All other keywords work analogously to those for CL:POSITION."
;; DO: Make keep-delimiters include the delimiters in the result(?).
(let ((len (length seq)))
(unless end (setq end len))
;; DO: Find a more efficient way to take care of :from-end T.
(when from-end
(setf seq (reverse seq))
(psetf start (- len end)
end (- len start)))
(loop with other-keys = (nconc (when test-supplied
(list :test test))
(when test-not-supplied
(list :test-not test-not))
(when key-supplied
(list :key key)))
for left = start then (+ right 1)
for right = (min (or (apply #'position delimiter seq
:start left
other-keys)
len)
end)
unless (and (= right left) ; empty-subsequence
remove-empty-subseqs)
if (and maximum (>= nr-elts maximum))
;; We can't take any more. Return now.
return (values subseqs (subseq seq left end))
else
collect (subseq seq left right) into subseqs
and sum 1 into nr-elts
until (= right end)
finally (return (values subseqs (subseq seq right end))))))
--
Pierre R. Mai <pm...@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
Erik Naggum
| Suppose i have a text file and i want to count strings in it.
| Specifically, the first 5 characters of each line in this file are a
| vendor code. Then i want to ouptut all vendor codes and how many
| times they show up in the input.
Here's my cut at that task:
(let ((hash (make-hash-table :test #'equal)))
(with-open-file (stream ...file-args...)
(handler-case
(loop
(let ((vendor (read-sequence stream (make-string 5))))
;; skip to end of line and consume
(peek-char #\newline stream)
(read-char stream)
;; real body of loop
(incf (gethash vendor hash 0))))
;; use conditions to simplify error handling
(end-of-file ())))
(dolist (key (sort (hash-table-keys hash) #'string<))
(format t "~A,~A~%" key (gethash key hash))))
| This is all straightforward, except for these lines:
| (if (gethash curv vhash)
| (incf (gethash curv vhash))
| (setf (gethash curv vhash) 1))
|
| What they do is, if there is already an entry in the hash, increment
| the count. If there isn't, create one (with count set to 1).
You have already been informed about the default value to gethash.
| Is this the best way to use a hash? In other words, do i really have to
| do the (gethash key hash) twice? It feels wasteful.
incf does one gethash to get the value and gethash's setf method to
set the value. If this is noticeably slow, you _may_ do better with
something like this:
(let ((entry (gethash vendor hash nil)))
(if entry
(incf (car entry))
(setf (gethash vendor hash) (list 1))))
| In Perl, you do $hash{key}++ and the entry will get "magically"
| created if it isn't there. (Of course, the Perl compiler may be
| doing the double lookup behind my back).
It is not unlikely that Perl, being a joke of a language around
over-powerful regexps and hash tables, has optimized this to death.
| The Perl version of this program runs a lot faster than the Lisp
| version (6 times faster) (I'm using Clisp and AIX), but i'm assuming
| that the main problem with the Lisp version is the fact that each
| line of text is consed and thrown away to the garbage collector.
| I'm pretty sure Perl would reuse the memory in this case.
Did you compile the Lisp code? Perl _always_ runs slower than Lisp
for me. Of course, I don't use a (good) toy Lisp like CLISP, but
Allegro CL, and I generally compile with high optimization settings.
| Anyway, comparing Perl and Lisp is not the point here, i just want
| to understand if i'm using the hashes right.
Why don't you ask whether hashes are right for the problem? If you
have decided to use Perl, regexps and hashes are the right solution
because you don't have anything else, but regexps are the wrong
solution to almost every problem, and hashes may be similar special
optimizations of particular cases.
I have a situation not dissimilar to yours, where I have ticker
codes for various financial instruments, and I found that it paid
off handsomely to use symbols. Some "modern" Lisp books tell us not
to use symbols, for the same reasons you should not use regexps and
hash tables in Perl, apparently: they are so easy to use that misuse
will be hard to combat, but nobody who uses Perl warns about Perl's
insane focus on bad tools -- those who take the warnings seriously
don't use Perl. So although symbold are expensive structures, they
do have enormous benefits language-wise in Lisp, and they should be
used if the alternative is more cumbersome. With symbols, the above
suggested speed-up becomes
(let ((symbol (intern string package)))
(if (boundp symbol)
(incf (symbol-value symbol))
(setf (symbol-value symbol) 1)))
| P.S.: the Perl version is a "one liner":
| $c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print "$x,$c{$x}\n"}}
| (very obvious, no? :-))
Perl looks the way it does because 110-baud Teletypes were once upon
the time the best people had on Unix, and with lousy typers who had
to correct their mistakes all the time, it made sense to abbreviate
everything down to one or two characters. Lisp has a different
heritage, to put it mildly: Better typers, better use of brainpower
than to remember thousands of subtly similar abbreviations, better
terminals. So we aren't as interested in one-liners with compact
syntax, but for people who still live their lives as if all they can
hope for is a 300-baud Teletype, the value of one-liners cannot be
underestimated.
On the other hand, loop constructs that walk through a file line by
line have been posted, as this seems a reasonable thing to do to
some people. People have regexp libraries for string hacking in
Lisp, too, even though the much better solution is to build a parser
that actually returns the stuctured information in the string. It
turns out that it doesn't take longer, in programmer-time, to write
and test a real parser than it does to write and test regexp-based
string hacking at the same quality level, it may in fact take a lot
shorter time, because the quality level in a parser is so readily
determinable, while people stop worrying about the regexp hacks
until it breaks at sufficiently infrequent random times because they
data never was designed to be hacked by regexps.
Of all the languages I have worked with over the past 20 years,
there's only one that stands out as the Wrong Solution to everything
it does, especially what it does well, and that's Perl. Quotably:
If Perl is the solution, you're solving the wrong problem.
#:Erik
--
I agree with everything you say, but I would
attack to death your right to say it.
-- Tom Stoppard
Peter Vaneynde
> * The Glauber <thegl...@my-deja.com>
> Here's my cut at that task:
>
> (let ((hash (make-hash-table :test #'equal)))
> (with-open-file (stream ...file-args...)
> (handler-case
> (loop
> (let ((vendor (read-sequence stream (make-string 5))))
(make-string 5) stream
not?
> ;; skip to end of line and consume
...
> (dolist (key (sort (hash-table-keys hash) #'string<))
hash-table-keys is not in the ANSI-spec, nor in ACL5. I guess it's a
new feature of ACL6?
I suppose it's quite a bit faster then:
(loop for key being the hash-keys in hash collect key)
?
> Did you compile the Lisp code? Perl _always_ runs slower than Lisp
> for me. Of course, I don't use a (good) toy Lisp like CLISP, but
> Allegro CL, and I generally compile with high optimization
> settings.
Your solution outruns the Lisp one on this old SPARC here...
> I have a situation not dissimilar to yours, where I have ticker
> codes for various financial instruments, and I found that it paid
> off handsomely to use symbols. Some "modern" Lisp books tell us not
Symbols in hash-tables or using the plists of the symbols? (I'm supposing
you want to store more then one property per symbol of course)
> I agree with everything you say, but I would
> attack to death your right to say it.
> -- Tom Stoppard
I've been dying to ask in what context this was said...
Groetjes, Peter
--
LANT nv/sa, Research Park Haasrode, Interleuvenlaan 15H, B-3001 Leuven
mailto:Peter.V...@lant.be Phone: ++32 16 405140
http://www.lant.be/ Fax: ++32 16 404961
The Glauber
Thanks to the folks who pointed to the default value of gethash, that's
exactly what i was looking for (i knew it had to be there!).
Thanks to Erik for an easy solution to the problem of consing the line each
time.
I don't want to get into a LispxPerl discussion. Perl is a nice thing to have
around when you need a quick answer; it processes text very fast, and it's a
good replacement for awk and sed. So i guess if Perl is a Bad Thing, at least
it allows me to avoid other Unix tools, right? :-) Lisp is a much more mature
language, of course, and more maintainable.
Someone asked, and yes, i compiled the CLISP with maximum optimization, and
it still ran 6 times slower than the Perl one-liner. I'm sure (well, almost
sure) this is because of the consing, and that if i implemented E.N.'s
version it would run at least as fast as Perl.
And here's the code as it stands now:
(let ((vhash (make-hash-table :test 'equal)))
(do ((l (read-line *standard-input*)
(read-line *standard-input* nil nil)))
((null l)
(maphash #'(lambda (k v) (format t "~A,~A~%" k v)) vhash))
(incf (gethash (subseq l 0 5) vhash 0)) ) )
By using the default value for gethash i can easily get rid of one variable.
This version is also reading from stdin and writing to stdout. If i run this
again, i'll put together a better way to print the results in sorted order.
I understand this code is still not 100% perfect, but it's as good as i need
it to be now, so don't waste more of your time improving on it (unless you
really want to! :-)).
Thanks again,
glauber
Erik Naggum
| >
| > (let ((hash (make-hash-table :test #'equal)))
| > (with-open-file (stream ...file-args...)
| > (handler-case
| > (loop
| > (let ((vendor (read-sequence stream (make-string 5))))
| not?
Huh? Oh, the argument order. Please be a little more verbose. I
have no idea how that got in there.
| hash-table-keys is not in the ANSI-spec, nor in ACL5. I guess it's a
| new feature of ACL6?
Oops, sorry, it's my own, an internal function that knows how the
hash-table is laid out and does a quick scan through the keys vector
to collect keys.
| I suppose it's quite a bit faster then:
| (loop for key being the hash-keys in hash collect key)
| ?
That's exactly what I would suggest for the portable definition.
The internal function is twice as fast as this loop.
| > Did you compile the Lisp code? Perl _always_ runs slower than Lisp
| > for me. Of course, I don't use a (good) toy Lisp like CLISP, but
| > Allegro CL, and I generally compile with high optimization
| > settings.
|
| Your solution outruns the Lisp one on this old SPARC here...
I hope you meant "Perl", but couldn't bring yourself to type it. :)
| > I have a situation not dissimilar to yours, where I have ticker
| > codes for various financial instruments, and I found that it paid
| > off handsomely to use symbols. Some "modern" Lisp books tell us not
|
| Symbols in hash-tables or using the plists of the symbols? (I'm
| supposing you want to store more then one property per symbol of
| course)
I let the value slot of the symbols hold one of a variety of
objects, according to the type of the financial instrument.
I don't use symbols as the keys in hash-tables, I use strings, but
that's just an accident of history rather than a conscious decision.
| > I agree with everything you say, but I would
| > attack to death your right to say it.
| > -- Tom Stoppard
|
| I've been dying to ask in what context this was said...
:) Actually, I have no idea. Mark Morford picked this one for the
SF Gate Morning Fix one day. I found it hilarious. (He also uses
"If this is not what you expected, please alter your expectations"
in the disclaimer of the Morning Fix, but someone objected so
strongly to my using that as a .signature, I found it worthwhile to
pick another one.)
The Glauber
Erik Naggum <er...@naggum.net> wrote:
> * Peter Vaneynde <Peter.V...@lant.be>
[...]
> | > I agree with everything you say, but I would
> | > attack to death your right to say it.
> | > -- Tom Stoppard
> |
>
> :) Actually, I have no idea. Mark Morford picked this one for the
> SF Gate Morning Fix one day. I found it hilarious. (He also uses
> "If this is not what you expected, please alter your expectations"
> in the disclaimer of the Morning Fix, but someone objected so
> strongly to my using that as a .signature, I found it worthwhile to
> pick another one.)
Bummer. I liked the "expectations" quote better.
The Hitch-hiker's Guide To The Galaxy has a disclaimer that says something
like "in case of discrepancy between what you see here and reality, reality
has got it wrong."
The Glauber
Bruce Hoult <br...@hoult.org> wrote:
[...]
> No. You need to actually know and *understand* the fully-spelled-out
> Perl version before the abbreviated version is anything other than a
> magic incantation that you have no choice but to remember by rote. You
> still need the complete thinking process. Perl just lets you save a bit
> on the typing. And not even on that, quite often -- where is the magic
> shortcut incantation to iterate through the sorted keys of a hash,
> instead of having to spell that part out in full?
I agree with everything you said, and won't even attack you for saying it. :-)
(Sorry, i've been reading Erik Naggum.)
(Actually, the part about the Perl code being obvious to the eye was really a
joke.)
The Glauber
> > $c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print
> > "$x,$c{$x}\n"}}
>
> command line flags neded to run it, namely "-n", which means that your
> program is really:
It just occurred to me that if you're going to nit-pick things like what
happen when you use -n in the command line, you probably should
also expand all macros in the Lisp code too. Practical languages have
stuff in them to make it easier for you to get the job done. That's why
we have things like with-open-file. Of course any good programmer
with 6 months experience could code that, but why should we have to?
Also, you can do that same program in Lisp as a "one-liner" ... if
you happen to have the right initialization files!
These are just a couple of reasons why this kind of language competition
is not likely to lead to much enlightment. However, i still think that to be
a Good Programmer (whatever this means) one has to be comfortable
with more than one programming language (and i mean comfortable to
the point of writing "idiomatic" code, that is, Lisp code that looks like
Lisp and C code that looks like C). Incidentally, both Perl and TCL
(and probably Python, i haven't used it for a while) borrow heavily
from Lisp.
When i'm looking for programmers to hire, i always try to find the ones
who are comfortable in more than one language, but they are
surprisingly hard to find! There's a lot of "one language wonders"
out there.
I've never seen it put better than it was done by Peter Norvig:
http://www.norvig.com/21-days.html
Wayne Rogers
I believe he is doing something like
#!/bin/perl -n -e '$c{substr($_,0,5)}++ ...' input_file
processes each line of the file using the execution statements in the -e
directive
Wayne
"Pierre R. Mai" <pm...@acm.org> wrote in message
news:878zrlp...@orion.bln.pmsf.de...
Bruce Hoult
> P.S.: the Perl version is a "one liner":
> $c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print
> "$x,$c{$x}\n"}}
No, that's not the complete Perl source code. You forgot to mention the
command line flags neded to run it, namely "-n", which means that your
program is really:
LINE:
while (<>){
$c{substr($_,0,5)}++; END { foreach $x (sort keys %c) {print
"$x,$c{$x}\n"}}
}
Which would be better written as:
while (<>){
$c{substr($_,0,5)}++;
}
foreach $x (sort keys %c) {
print "$x,$c{$x}\n"
}
... which actually means ...
while ($line = <STDIN>){
$c{substr($line,0,5)}++;
}
foreach $x (sort keys %c) {
print "$x,$c{$x}\n"
}
... and in a program you intend to keep and maintain should be ...
#!/usr/bin/perl -w
use strict;
my %c = {};
while (my $line = <STDIN>){
$c{substr($line,0,5)}++;
}
foreach my $x (sort keys %c) {
print "$x,$c{$x}\n"
}
And now we're up to the same size program as it takes to do the same
thing in C++ or Lisp or Dylan or just about anything else with a decent
library of data structures.
Perl isn't really more compact than other languages, it just has more
special-case hack shortcuts that are *sometimes* useful (often even, for
the things Perl is usually used for), but that you'll have to spell out
by hand as soon as you start doing more complex things for which the
defaults aren't *quite* right.
> (very obvious, no? :-))
No. You need to actually know and *understand* the fully-spelled-out
Perl version before the abbreviated version is anything other than a
magic incantation that you have no choice but to remember by rote. You
still need the complete thinking process. Perl just lets you save a bit
on the typing. And not even on that, quite often -- where is the magic
shortcut incantation to iterate through the sorted keys of a hash,
instead of having to spell that part out in full?
-- Bruce
Bruce Hoult
<thegl...@my-deja.com> wrote:
> It just occurred to me that if you're going to nit-pick things like what
> happen when you use -n in the command line, you probably should
> also expand all macros in the Lisp code too. Practical languages have
> stuff in them to make it easier for you to get the job done. That's why
> we have things like with-open-file. Of course any good programmer
> with 6 months experience could code that, but why should we have to?
>
> Also, you can do that same program in Lisp as a "one-liner" ... if
> you happen to have the right initialization files!
I agree entirely.
> Incidentally, both Perl and TCL (and probably Python, i haven't used
> it for a while) borrow heavily from Lisp.
You think?
sub cons ($$) {
my ($car, $cdr) = @_;
sub ($) {$_ = shift;
/car/ && return $car;
/cdr/ && return $cdr;
}
}
sub car ($){&{$_[0]}(car);}
sub cdr ($){&{$_[0]}(cdr);}
my $x = cons(12, cons(3.1415, cons("hello world!", 78)));
print "the third element is '", car(cdr(cdr($x))), "'\n";
--------------------------------------------------------
the third element is 'hello world!'
> When i'm looking for programmers to hire, i always try to find the ones
> who are comfortable in more than one language, but they are
> surprisingly hard to find! There's a lot of "one language wonders"
> out there.
I usually seem to get told that I "can't be focussed enough" and that
"over here in the US there are so many good people that if you don't
specialize then you can't be a top performer at anything".
-- Bruce
Pierre R. Mai
> Pierre;
> I believe he is doing something like
>
> #!/bin/perl -n -e '$c{substr($_,0,5)}++ ...' input_file
>
> processes each line of the file using the execution statements in the -e
> directive
Right, I forgot about the nice command-line options that affect
program semantics in Perl. As if it's syntax weren't enough to
confuse people...
OTOH, this will use stdin, whereas the CL code was opening it's own
stream for a named file, so it seems hardly fair comparing the two.
Regs, Pierre.
The Glauber
Bruce Hoult <br...@hoult.org> wrote:
> In article <8snsfg$pre$1...@nnrp1.deja.com>, The Glauber
> <thegl...@my-deja.com> wrote:
> > When i'm looking for programmers to hire, i always try to find the ones
> > who are comfortable in more than one language, but they are
> > surprisingly hard to find! There's a lot of "one language wonders"
> > out there.
>
> I usually seem to get told that I "can't be focussed enough" and that
> "over here in the US there are so many good people that if you don't
> specialize then you can't be a top performer at anything".
In a sense this is true, like it's true of those people who get Olympic
medals. So, i'll never run 100 m in under 10 seconds, big deal, i can get
there much faster by car! :-)
I think software creation is a multi-discipline activity, and not only you
gain by knowing other programming languages, but you gain by knowing other
fields as well. Programming is solving problems, and solving problems
sometimes requires being able to think outside the box.
Erik Naggum
| I think software creation is a multi-discipline activity, and not
| only you gain by knowing other programming languages, but you gain
| by knowing other fields as well. Programming is solving problems,
| and solving problems sometimes requires being able to think outside
| the box.
How does this require multiple language skills? Must an author be
able to write great literature in multiple languages?
Put it this way: I hate it when people do reasonably smart stuff in
Perl, because it is just as incomprehensible to me as Umberto Eco's
great stuff in Italian. I'm annoyed by the fact that I have to work
with a translation, especially since I have no idea how good it is,
but I'm not going to call him a "one-language wonder".
The idea that one language cannot be all things to all people is
probably wrong, but the kinds of languages that have tried have been
all things to some people in abject disregard of most other people,
which may lead to the wrong conclusions. Designing a language that
is enabling and not restricting is very, very hard and requires so
much forethought and probably dissociation from one's own goals that
it will most probably grow into being rather than be designed from
the start. However, things that grow fall into two main categories:
weeds and produce. We have two particularly interesting examples of
weeds: Perl and C++. I consider Common Lisp a grown language, too,
as opposed to designed-to-be-everything-for-a-few-people like Scheme.
I fully agree that problem solving is not something you can do
without knowing the problems and their interrelationships with the
rest of the world. However, when most business schools churn out
generic managers, I suppose it will be harder to convince such
people that programmers actually need to understand the issues and
the problems they are facing. The era of designers and coders is
behind us, and all we know is it didn't work very well, but not why.
Perhaps if more of the people who call themselves programmers or
even software engineers these days were labeled coders and had
somebody else think for them, we would need far less of them simply
because they _can't_ think and therefore write so much miserably
buggy code, while the real thinking that goes into modern "apps" is
very sparse, indeed, compared to all the work required in coding, so
it could probably be handled by a relatively tiny group of people.
#:Erik
--
The Glauber
Erik Naggum <er...@naggum.net> wrote:
> * Peter Vaneynde <Peter.V...@lant.be>
> | >
> | > (let ((hash (make-hash-table :test #'equal)))
> | > (with-open-file (stream ...file-args...)
> | > (handler-case
> | > (loop
> | > (let ((vendor (read-sequence stream (make-string 5))))
> | not?
>
> Huh? Oh, the argument order. Please be a little more verbose. I
> have no idea how that got in there.
I still don't understand how this works. read-sequence returns a position,
not a string; "vendor" would be bound to the number 5, right?
> | hash-table-keys is not in the ANSI-spec, nor in ACL5. I guess it's a
> | new feature of ACL6?
>
> Oops, sorry, it's my own, an internal function that knows how the
> hash-table is laid out and does a quick scan through the keys vector
> to collect keys.
Here's my version (probably not very fast, but using only what i know of CL):
;;; Return the hash's keys, in no particular order
;;; NOTE: Don't modify the keys!
(defmacro hash-table-keys (hash)
(let ((key (gensym)) (val (gensym)) (keylist (gensym)))
`(let ((,keylist '()))
(maphash #'(lambda (,key ,val) (push ,key ,keylist)) ,hash)
,keylist
)
)
)
The Glauber
Erik Naggum <er...@naggum.net> wrote:
> * The Glauber <thegl...@my-deja.com>
> | I think software creation is a multi-discipline activity, and not
> | only you gain by knowing other programming languages, but you gain
> | by knowing other fields as well. Programming is solving problems,
> | and solving problems sometimes requires being able to think outside
> | the box.
>
> How does this require multiple language skills? Must an author be
> able to write great literature in multiple languages?
Great example. There's very little empyrical evidence, of course, but i
believe that someone that can write great literature, say, in Russian, would
also write great literature in Portuguese, if he had to.
It helps not in the sense that knowing C will help you write better Lisp, but
in the sense that it opens up the mind to thinking in different ways. Someone
can probably be a great programmer and only know one language, but i prefer
to work with people that are more curious than this.
Whatever it is that makes someone a good programmer is not tied to the choice
of programming language. Most people can learn another computer language
easily. It's the problem solving thinking, the ability to analyse and
integrate, that's so hard.
I'm sure you'd be a great Perl programmer! :-)
[...]
glauber
Boris Schaefer
| How does this require multiple language skills? Must an author be
| able to write great literature in multiple languages?
I don't think one has to know multiple languages, but I consider
learning multiple languages rewarding because one usually learns new
concepts together with the new languages. I also believe that
learning new concepts makes better programmers.
The languages are not terribly important, but they are a good source
of inspiration.
--
bo...@uncommon-sense.net - <http://www.uncommon-sense.net/>
Because we don't think about future generations, they will never forget us.
-- Henrik Tikkanen
Barry Margolin
The Glauber <thegl...@my-deja.com> wrote:
>It helps not in the sense that knowing C will help you write better Lisp, but
>in the sense that it opens up the mind to thinking in different ways. Someone
>can probably be a great programmer and only know one language, but i prefer
>to work with people that are more curious than this.
Conversely, if someone only knows one language, and has trouble learning
other languages, they're probably not a "great programmer". The difficulty
they have in learning those other languages suggests that their thought
processes are wedded to the paradigms embodied in that language, a sort of
tunnel vision.
I probably wouldn't consider using BASIC, APL, or Prolog for any project I
was involved in, but I consider myself enriched as a programmer because I
know a little bit of them (there was a time when I was an expert BASIC
programmer, but the language has evolved quite a bit in the two decades
since then, and it would take me a few days to get up to speed on modern
BASIC).
Dirk Zoller
> I don't think one has to know multiple languages, but I consider
> learning multiple languages rewarding because one usually learns new
> concepts together with the new languages. I also believe that
> learning new concepts makes better programmers.
There has been a direction in philosophy which strongly emphasized
and carefully scrutinized the connection between language and thinking.
These people were so caught by the fact that the only way of expressing
their thoughts was language, that they finally didn't seem to see any
difference between philosophic thought and analysis of how language works.
AFAIR Wittgenstein was one of the first in that school of thought and
he was very influential throughout the first half of the last century.
Seems like many programmers feel alike with respect to their programming
languages.
--
Dirk Zoller Phone: +49-69-50959861
Sol-3 GmbH&Co KG Fax: +49-69-50959859
Niddastrape 98-102 e-mail: d...@sol-3.de
60329 Frankfurt/M
Germany
Tom Breton
>
> Also, you can do that same program in Lisp as a "one-liner" ... if
> you happen to have the right initialization files!
Yes, exactly. All of the "how succintly can language X say *this*?"
challenges come down to library issues. (Except for some very poor
languages that can't even use libraries easily)
--
Tom Breton, http://world.std.com/~tob
Not using "gh" 1997-2000. http://world.std.com/~tob/ugh-free.html
Some vocal people in cll make frequent, hasty personal attacks, but if
you killfile them cll becomes usable.
Erik Naggum
| I still don't understand how this works. read-sequence returns a position,
| not a string; "vendor" would be bound to the number 5, right?
It doesn't work because I blew it. I have this nifty little Emacs
hack that lets me edit Lisp code in an indirect buffer, but I must
have posted the code I jotted down first and discarded the code that
worked after improvements and testing.
The real code goes like this:
(let ((vendor (make-string 5)))
(read-sequence vendor stream)
...
The return value from read-sequence is discarded because it may be
less than 5 only when the end of the file has been reached, and then
we're going to hit the real end of file in the next read operations,
so testing for zero would have been necessary, but only a waste.
I have long been annoyed that read-sequence returns 0 on end of
file. That kind of braindamaged conflation between a "no can do"
whine (useless return value) and "read past the end of the file"
condition doesn't belong in a programming language, but rather in
some underdesigned little toy operating system like early 1970's
Unix. *sigh*
Thomas A. Russ
> ;;; Return the hash's keys, in no particular order
> ;;; NOTE: Don't modify the keys!
> (defmacro hash-table-keys (hash)
> (let ((key (gensym)) (val (gensym)) (keylist (gensym)))
> `(let ((,keylist '()))
> (maphash #'(lambda (,key ,val) (push ,key ,keylist)) ,hash)
> ,keylist
> )
> )
> )
Any particular reason why this is implemented as a macro? It would seem
that a function would do just as well, and be simpler. Since you are
already doing a maphash, the function call overhead can't be worth
worrying about. [Hmmm, maybe this will spawn another premature
optimization thread??]
--
Thomas A. Russ, USC/Information Sciences Institute t...@isi.edu
Lieven Marchand
> Here's my version (probably not very fast, but using only what i know of CL):
>
> ;;; Return the hash's keys, in no particular order
> ;;; NOTE: Don't modify the keys!
> (defmacro hash-table-keys (hash)
> (let ((key (gensym)) (val (gensym)) (keylist (gensym)))
> `(let ((,keylist '()))
> (maphash #'(lambda (,key ,val) (push ,key ,keylist)) ,hash)
> ,keylist
> )
> )
> )
Why did you make this a macro? It would be a lot simpler as a function
and there is no reason to make it a macro.
I'd do something like
;;; untested
(defun hash-table-keys (hash)
(loop for key being each hash-key of hash
collect key))
--
Lieven Marchand <m...@bewoner.dma.be>
Lambda calculus - Call us a mad club
Bruce Hoult
wrote:
> Boris Schaefer wrote:
>
> > I don't think one has to know multiple languages, but I consider
> > learning multiple languages rewarding because one usually learns new
> > concepts together with the new languages. I also believe that
> > learning new concepts makes better programmers.
>
> There has been a direction in philosophy which strongly emphasized
> and carefully scrutinized the connection between language and thinking.
> These people were so caught by the fact that the only way of expressing
> their thoughts was language, that they finally didn't seem to see any
> difference between philosophic thought and analysis of how language works.
> AFAIR Wittgenstein was one of the first in that school of thought and
> he was very influential throughout the first half of the last century.
>
> Seems like many programmers feel alike with respect to their programming
> languages.
Yeah, I've heard of those guys. Some sort of hypothesis or other,
wasn't there. And isn't that the same school that says things such as
"Specifications are for the weak and timid!" and "What is this talk of
'release'? We do not make software 'releases'. Our software escapes,
leaving a bloody trail of designers and quality assurance people in its
wake!".
Starts with "W", I think :-)
-- Bruce
Conflations R Us
Erik Naggum
| Conversely, if someone only knows one language, and has trouble
| learning other languages, they're probably not a "great programmer".
| The difficulty they have in learning those other languages suggests
| that their thought processes are wedded to the paradigms embodied in
| that language, a sort of tunnel vision.
Well, this is obviously true, but it cuts both ways. Suppose you
are used to several _great_ languages and are asked to work in some
braindamaged language designed by someone whose design concepts
never got out of the 50's, how well could you do it? I have very
serious problems working with C++ for this simple reason: I have
this _overwhelming_ urge to redesign the language first. It is such
a phenomenally moronic _non-design_ anyone with exposure to a real
object-oriented language (such as Simula, on which Bjarne Stroustrup
based his desire to still use C while every other Simula programmer
in the world would have understood that C would be a good language
to implement the machinery needed to build a good Simula system at
best, not a good language in which constantly to reimplement said
machinery by hand) would have to shut off the alarms that go "it's a
trap! run! run!" (add scary movie soundtrack for best effects),
while trying not to think how it would be done in a real language.
Matter of fact, it was a project in C++ (and some class design that
stood the test of time) that caused me to look real hard at doing
something other than programming for a living for the rest of my
life, but then I finally wound up wanting to work with Common Lisp
after having studied several other interesting languages, including
Ada and Smalltalk.
I can't hack Perl, either. My brain starts to yell "This is wrong!
This is all wrong! Don't _do_ this to me!", I lose my concentration
_real_ quick and I start to need caffeine, rest, fresh air, water,
you name it, just about anything other than having to deal with that
Perl shit. I have pissed away _days_ when I was reimplementing some
200-line Perl insanity that used a freaking _database_ to store some
really simple values that would be perfectly happy just sitting in
the value slot of a symbol. What _really_ pisses me _off_ with Perl
is that people work so hard doing so terribl little while they think
they have worked very little doing something really nifty. Fools!
Using such inferior languages is like asking a chef who could have
done wonders with any kind of raw materials, to use a dirty kitchen,
a broken refrigerator with food that is about to die a second time,
broken tools, and brownish tap water that tasted of swamp land. His
first task would be to clean up the place. Creating food in there
would be the furthest from his mind. That's how I feel about Perl
and C++. I prefer to call it "good taste", not "tunnel vision".
I don't like rap, either. Call me intolerant.
Erik Naggum
| How does this require multiple language skills? Must an author be
| able to write great literature in multiple languages?
* Boris Schaefer <bo...@uncommon-sense.net>
| I don't think one has to know multiple languages, but I consider
| learning multiple languages rewarding because one usually learns new
| concepts together with the new languages. I also believe that
| learning new concepts makes better programmers.
|
| The languages are not terribly important, but they are a good source
| of inspiration.
Let me repeat the question: How does this require multiple language
skills?
Yes, multiple languages have benefits. Are those benefits tied to
multiple languages? I don't think they are. I think they are tied
to curiosity, creativity, never being fully satisfied that you have
found the best solution, and enough humility to listen to others and
their ideas too It would _not_ be a good thing for a programmer to
_have_ to learn multiple languages to be able to use new concepts,
yet that's exactly how I see most new language come into existence:
Some (relative) idiot can't bring himself to implement his concepts
(which may well be brilliant and truly new) in an existing language,
so he goes off discarding everything others spent years building so
he can have his pet concept easily expressed. That's nuts, really.
There are people who have to design their own alphabets or spellings
in order to feel able to express themselves, but I think we label
them "insane" rather than applaud them as "language designers".
Dirk Zoller
> Yes, multiple languages have benefits. Are those benefits tied to
> multiple languages? I don't think they are. I think they are tied
> to curiosity, creativity, never being fully satisfied that you have
> found the best solution, and enough humility to listen to others and
> their ideas too It would _not_ be a good thing for a programmer to
> _have_ to learn multiple languages to be able to use new concepts,
> yet that's exactly how I see most new language come into existence:
> Some (relative) idiot can't bring himself to implement his concepts
> (which may well be brilliant and truly new) in an existing language,
> so he goes off discarding everything others spent years building so
> he can have his pet concept easily expressed. That's nuts, really.
Following your argumentation would mean that it is only due to "(relative)
idiot"s that we are not still programming in Assembler or FortranIV.
(If you think you're on the safe side as a Lisp user here, you are not.
Since where is the difference between extending and changing a language
and inventing a new one? You must ask yourself why you're not programming
in some ancient Lisp? Due to idiots which were unable to express themselves
in those dialects, obviously.)
It takes some repeated reconsideration and starting over again and again
from scratch while trying to keep the good stuff, get rid of the bad and
invent a few new bits.
Where what's "good" or "bad" or "desirable new stuff" is to a great deal a
matter of taste and therefore in the big picture -- random. Some languages
survive, most die as experiments. Looks like sheer darwinism. Nature has
achieved a lot by such mechanisms.
> There are people who have to design their own alphabets or spellings
> in order to feel able to express themselves, but I think we label
> them "insane" rather than applaud them as "language designers".
People are different. People express themselves in various ways. Some
discuss in newsgroups, others paint, some design programming languages.
People experiment. This does not mean they're insane, nor stupid, nor
incompetent idiots, nor does any other of Naggum's most frequently used
attributes apply.
--
Dirk Zoller
e-mail: d...@onlinehome.de
http://www.dirk-zoller.de
Christopher Browne
the artist formerly known as The Glauber said:
>In article <31810498...@naggum.net>,
> Erik Naggum <er...@naggum.net> wrote:
>> | I think software creation is a multi-discipline activity, and not
>> | only you gain by knowing other programming languages, but you gain
>> | by knowing other fields as well. Programming is solving problems,
>> | and solving problems sometimes requires being able to think outside
>> | the box.
>>
>> able to write great literature in multiple languages?
>
>believe that someone that can write great literature, say, in Russian, would
>also write great literature in Portuguese, if he had to.
Perhaps; perhaps not. The ability to write "great literature"
involves two major things:
1. Having a great story to tell. This part could _potentially_ be
usable in any language.
2. Having the ability to _express_ that story in some particular
language. The fact that I communicate well in English does not
establish that I can do the same in Russian.
"Great literature" requires both components, in substantial measure.
>It helps not in the sense that knowing C will help you write better
>Lisp, but in the sense that it opens up the mind to thinking in
>different ways. Someone can probably be a great programmer and only
>know one language, but i prefer to work with people that are more
>curious than this.
It is useful to know some different "styles" of programming as this
influences the ability to express yourself in whatever language you
use.
An obvious "retort" to this is that if a particular language (oh, say,
Common Lisp?) happens to be sufficiently expressive, there may be
little merit in "opening" your mind by learning some other _minimally_
expressive language.
>Whatever it is that makes someone a good programmer is not tied to
>the choice of programming language. Most people can learn another
>computer language easily. It's the problem solving thinking, the
>ability to analyse and integrate, that's so hard.
As with "Point #1" up above, there's a need to have a Good Story To
Tell. Knowing arcania of whatever language, whether Lisp, Latin, or
Perl, will do NO good if you can't grapple with the problem and don't
have some sort of "mental picture" of how to solve it.
>I'm sure you'd be a great Perl programmer! :-)
Heads will be exploding all over Scandinavia, no doubt...
--
(concatenate 'string "cbbrowne" "@" "