Accidentally Retaining Head?

[aria42]

Hi,
I was experimenting with some code and I had an largish sequence I
did a doseq over. The memory hit the ceiling, which you expect since
even though the head isn't retained GC doesn't happen until you hit
your memory limit (is there a way to change that). Once it hit the
memory limit, 1.2 gigs in this case, the doseq slows to a halt.

On the other hand if rather than make a long lazy seq, I do it
implicitly in the doseq itself (see FAST snippet below), the
performance is great. Is there anyway to get good performance using a
single lazy seq? Relevant snippets below.

Thanks, Aria

(defn lines
"get lines from gz file"
[#^String path]
(-> path
java.io.FileInputStream.
java.util.zip.GZIPInputStream.
java.io.InputStreamReader.
java.io.BufferedReader.
line-seq))

; the tree-from-str does some processing, but doesn't have state
(def trees (for [l (lines "/usr/local/corpora//NANC/003.gz")
:when (not (empty? l))
:let [[t _] (tree/tree-from-str l)]]
t))

; SLOW: hits memory limit and becomes slow b/c of constant
; GC hits
(doseq [t trees]
(println (str t)))

; FAST: low memory
(doseq [l (lines "/usr/local/corpora//NANC/003.gz")
:when (not (empty? l))
:let [[t _] (tree/tree-from-str l)]]
(println (str t)))

[Per Vognsen]

It doesn't seem very accidental: the namespace binding for 'trees' is
retaining the head. You probably want to wrap it in a function:

(defn trees []
...)

-Per

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>
> To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
>

[aria42]

Whoops duh. That was silly, far to early on the west coast.

[Per Vognsen]

By the way, you seem to misunderstand some of the workings of GC. In
the kind of generational scheme used in virtually all modern
algorithms, a collection of the nursery (where allocations are first
hatched) will only be forced when the memory assigned to the nursery
is exhausted--not the memory of any higher generations.

-Per