exception in 'map' mutates result to nil -- bug?

[Hank]

I'm mapping a function that throws an exception over a collection:

=> (def mapped (map (fn [_] (throw (Exception.))) [1 2 3]))

#'user/mapped

'map' is lazy so we're not expecting to see the exception until we're trying to access the result:

=> mapped

Exception   user/fn--709 (NO_SOURCE_FILE:1)

All good, let's do that again:

=> mapped

()

Whoops! Is this by design? Why? Where does that empty sequence come from?

'map' is really just calling the lazy-seq macro but doing this with lazy seq works just fine:

=> (def lazy (lazy-seq (throw (Exception.))))

#'user/lazy

=> lazy

Exception   user/fn--733 (NO_SOURCE_FILE:1)

=> lazy

Exception   user/fn--733 (NO_SOURCE_FILE:1)

Same exception over and over again as it should be. I stared at the implementations of 'map' and lazy-seq/LazySeq for some hours but I really can't see it.

Cheers

-- hank

[julianrz]

Hmm.. I think you are raising both a technical and a philosophical issue - what exactly should a higher-order function return when some application of the supplied function throws exception... The behaviors could be:

1) throw

2) return null

3) return an empty collection (do not continue after 1st failure)

4) same, but continue through all failures

5) return collection with valid applications only

What is good behavior? We should have as little uncertainly as possible... I have to admit my first thought is 'just don't allow it to throw, return null instead'. This would allow to still collect results (where it does not throw), and they will be at the matching index locations...

Now the technicality. The map code I see for 1.4 is trying to append result of all invocations to a buffer upfront. And each time it will fail, and the buffer will be empty, hence empty result Source is a chunked collection and there is only one chunk, it will be done immediately, and not on subsequent calls - laziness has not started yet  

Please contrast it with mapv, which results in a vector, not sequence

user=> (def mapped (mapv (fn [_] (throw (Exception.))) [1 2 3]))

Exception   user/fn--1 (NO_SOURCE_FILE:1)

user=> mapped

#<Unbound Unbound: #'user/mapped>

Now the lazy-seq example - maybe the difference in behavior can be explained by the fact that lazy-seq caches the result of body evaluation and will keep returning it. Since there is an exception during each evaluation, the caching does not quite happen, and it is re-evaluated?

Should the behavior be the same in all 3 cases? I think so, at least for consistency (unsure it can be achieved, though). But throwing in function passed to map  really is something you should not do, and I would recommend to change map function to return a null or throw -- to let you know that your code can cause odd behavior. BTW I am not aware of any clojure book that alerts you to that. 

Also, maybe the implementation of map function should catch your exception internally and produce a null, and return a sequence of nulls?

-Julian

[Hank]

I've got this narrowed down to what seems an interaction issue between macros and closures in lazy-seq. Run this code:

(defn gen-lazy []
  (let [coll [1 2 3]]
    (lazy-seq
      (when-let [s (seq coll)]
        (throw (Exception.))
        )
      )
    )
  )

(def lazy (gen-lazy))

(try
  (println "lazy:" lazy)
  (catch Exception ex
    (println ex))
  )

(try
  (println "lazy, again:" lazy)
  (catch Exception ex
    (println ex))
  )

... and explain to me what you see there. The first time, exception, the second time, empty list.

Now if you either a) remove the let statement and put the sequence [1 2 3] directly into where it says coll then it "works" = same exception in both cases.

(defn gen-lazy []
  (lazy-seq
    (when-let [s (seq [1 2 3])]
      (throw (Exception.))
      )
    )
  )

Or if you take out the when-let macro it works, too:

(defn gen-lazy []
  (let [coll [1 2 3]]
    (lazy-seq
      (seq coll)
      (throw (Exception.))
      )
    )
  )

Only the combination of closure + when-let macro breaks thigs. I know clojure does some funky things to closures in lazy-seq (see "... perform closed-over local clearing on the tail call of their body" on this page: http://clojure.org/lazy), this is related to the undocumented :once keyword on function calls. Maybe that interferes with macros? Or maybe I'm barking up the wrong tree.

-- hank

[Hank]

Julian, see my other post, it has nothing to do with map per se.

Apart from philosophical questions, the reason I need this to consistently raise an exception, is that I want to interrupt the map evaluation when it is slow, throwing an InteruptedException. I then want to recommence evaluation later. rinse, repeat. I can't recommence it if the sequence has been 'closed' with a final nil. Sounds like a fair use case?

[Hank]

I opened a bug report, let's see what the pros have to say on this: http://dev.clojure.org/jira/browse/CLJ-1119

[Christophe Grand]

This behavior has the same source as CLJ-457. Applying my patch for CL-457, I get:

user=>  (def mapped (map (fn [_] (throw (Exception.))) [1 2 3]))

#'user/mapped

user=> mapped

Exception   user/fn--1 (NO_SOURCE_FILE:1)

(user=> mapped

RuntimeException Recursive seq realization  clojure.lang.LazySeq.sval (LazySeq.java:64)

(user=> mapped

RuntimeException Recursive seq realization  clojure.lang.LazySeq.sval (LazySeq.java:64)

and

lazy: (#<Exception java.lang.Exception>

lazy, again: (#<RuntimeException java.lang.RuntimeException: Recursive seq realization>

for the other example.

So the error message may be more generic.

Christophe

On Mon, Dec 3, 2012 at 10:37 AM, Hank <ha...@123mail.org> wrote:

I opened a bug report, let's see what the pros have to say on this: http://dev.clojure.org/jira/browse/CLJ-1119

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--
On Clojure http://clj-me.cgrand.net/
Clojure Programming http://clojurebook.com
Training, Consulting & Contracting http://lambdanext.eu/

[Hank]

Ah thanks many times. I saw 457 when I searched the issue list before opening a new one but it wasn't clear to me they were related. I shall use your patch in my private fork.