Now I thought maybe this had something to do with transients having a
limit to the number of ops that can be performed on them (?), so I
tried an alternative definition:
And what's getting me is that the exception is -still- triggering here
for sufficiently large input maps.
It won't trigger if transients aren't being used.
Does this make sense? Even if I'm missing something and using the
transients incorrectly, why would the second example still be
exhibiting the same problem assuming the initial map-context1 is still
persistent?
Would really appreciate any kind of input- thanks!
What does zipmap do if the key seq contains duplications?
-- Protege: What is this seething mass of parentheses?! Master: Your father's Lisp REPL. This is the language of a true hacker. Not as clumsy or random as C++; a language for a more civilized age.
On Aug 23, 11:38 pm, Ken Wesson <kwess...@gmail.com> wrote:
> What does zipmap do if the key seq contains duplications?
That was my instinct too, but (a) a few thousand numbers won't collide
very often at all given the problem space, and (b) some experimenting
indicates that the key-seq is always 100k elements large - no
duplicates.
HOWEVER, I did find the problem: the keys themselves don't collide,
but their hashes do. The number of elements that get messed up is
related to the number of hash collisions, but I still don't quite
understand how they interact.
Here's a modified code snippet that demonstrates the issue:
user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
(println (count (distinct (map hash (keys m)))))
((juxt count identity) (persistent!
(reduce dissoc! (transient m) (keys m)))))
100000 ;; no collisions
[0 {}] ;; map is empty at the end
user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
(println (count (distinct (map hash (keys m)))))
((juxt count identity) (persistent!
(reduce dissoc! (transient m) (keys m)))))
99996 ;; four collisions
[8 {0.30426231137219917 0.8531183785687654, 0.8893047006425385
0.4788315896128895, 0.47854633997540674 0.45133768991797785,
0.5265638224227486 0.7724779126227945}] ;; a four-element map that
reports its count as eight!!!
That last comment seems to indicate a very serious error somewhere:
not only is the transient map broken, but it creates a broken
persistent object. I'll file a JIRA issue for this, and see if I can
find out any more about the cause.
> On Aug 23, 11:38 pm, Ken Wesson <kwess...@gmail.com> wrote:
> > What does zipmap do if the key seq contains duplications?
> That was my instinct too, but (a) a few thousand numbers won't collide
> very often at all given the problem space, and (b) some experimenting
> indicates that the key-seq is always 100k elements large - no
> duplicates.
> HOWEVER, I did find the problem: the keys themselves don't collide,
> but their hashes do. The number of elements that get messed up is
> related to the number of hash collisions, but I still don't quite
> understand how they interact.
> Here's a modified code snippet that demonstrates the issue:
> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
> (println (count (distinct (map hash (keys m)))))
> ((juxt count identity) (persistent!
> (reduce dissoc! (transient m) (keys m)))))
> 100000 ;; no collisions
> [0 {}] ;; map is empty at the end
> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
> (println (count (distinct (map hash (keys m)))))
> ((juxt count identity) (persistent!
> (reduce dissoc! (transient m) (keys m)))))
> 99996 ;; four collisions
> [8 {0.30426231137219917 0.8531183785687654, 0.8893047006425385
> 0.4788315896128895, 0.47854633997540674 0.45133768991797785,
> 0.5265638224227486 0.7724779126227945}] ;; a four-element map that
> reports its count as eight!!!
> That last comment seems to indicate a very serious error somewhere:
> not only is the transient map broken, but it creates a broken
> persistent object. I'll file a JIRA issue for this, and see if I can
> find out any more about the cause.
Please search through previous messages/tickets before posting new issues. This issue has been fixed as of commit da412909d36551a526ed and will be included in the -beta2 release. It was originally ticketed here:
> On Aug 24, 12:27 am, Alan Malloy<a...@malloys.org> wrote: >> On Aug 23, 11:38 pm, Ken Wesson<kwess...@gmail.com> wrote:
>>> What does zipmap do if the key seq contains duplications? >> That was my instinct too, but (a) a few thousand numbers won't collide >> very often at all given the problem space, and (b) some experimenting >> indicates that the key-seq is always 100k elements large - no >> duplicates.
>> HOWEVER, I did find the problem: the keys themselves don't collide, >> but their hashes do. The number of elements that get messed up is >> related to the number of hash collisions, but I still don't quite >> understand how they interact.
>> Here's a modified code snippet that demonstrates the issue:
>> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))] >> (println (count (distinct (map hash (keys m))))) >> ((juxt count identity) (persistent! >> (reduce dissoc! (transient m) (keys m))))) >> 100000 ;; no collisions >> [0 {}] ;; map is empty at the end
>> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))] >> (println (count (distinct (map hash (keys m))))) >> ((juxt count identity) (persistent! >> (reduce dissoc! (transient m) (keys m))))) >> 99996 ;; four collisions >> [8 {0.30426231137219917 0.8531183785687654, 0.8893047006425385 >> 0.4788315896128895, 0.47854633997540674 0.45133768991797785, >> 0.5265638224227486 0.7724779126227945}] ;; a four-element map that >> reports its count as eight!!!
>> That last comment seems to indicate a very serious error somewhere: >> not only is the transient map broken, but it creates a broken >> persistent object. I'll file a JIRA issue for this, and see if I can >> find out any more about the cause.
I did, of course. I searched for "transient", and didn't find any
describing this issue. And looking at the issue you link to, I still
don't see how it's related: it's a patch specifically for vectors, and
this code doesn't touch vectors; and it involves changes to a
transient "leaking" back into the original persistent, while this code
is about getting the wrong persistent object back after some changes.
I don't have a lot of experience with building clojure, but I'll try
compiling latest master and trying it out. In the meantime, did you
try this code more than once? Because it includes randomness,
sometimes it works without problems.
On Aug 25, 6:03 am, Aaron Bedra <aaron.be...@gmail.com> wrote:
> Please search through previous messages/tickets before posting new
> issues. This issue has been fixed as of commit da412909d36551a526ed and
> will be included in the -beta2 release. It was originally ticketed here:
> > On Aug 24, 12:27 am, Alan Malloy<a...@malloys.org> wrote:
> >> On Aug 23, 11:38 pm, Ken Wesson<kwess...@gmail.com> wrote:
> >>> What does zipmap do if the key seq contains duplications?
> >> That was my instinct too, but (a) a few thousand numbers won't collide
> >> very often at all given the problem space, and (b) some experimenting
> >> indicates that the key-seq is always 100k elements large - no
> >> duplicates.
> >> HOWEVER, I did find the problem: the keys themselves don't collide,
> >> but their hashes do. The number of elements that get messed up is
> >> related to the number of hash collisions, but I still don't quite
> >> understand how they interact.
> >> Here's a modified code snippet that demonstrates the issue:
> >> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
> >> (println (count (distinct (map hash (keys m)))))
> >> ((juxt count identity) (persistent!
> >> (reduce dissoc! (transient m) (keys m)))))
> >> 100000 ;; no collisions
> >> [0 {}] ;; map is empty at the end
> >> That last comment seems to indicate a very serious error somewhere:
> >> not only is the transient map broken, but it creates a broken
> >> persistent object. I'll file a JIRA issue for this, and see if I can
> >> find out any more about the cause.
Update: just built master, and issue still exists. If you want to be
"sure" you get it, just add another zero to the input range. I'll
mention that in the ticket as well.
On Aug 25, 10:14 am, Alan Malloy <a...@malloys.org> wrote:
> I did, of course. I searched for "transient", and didn't find any
> describing this issue. And looking at the issue you link to, I still
> don't see how it's related: it's a patch specifically for vectors, and
> this code doesn't touch vectors; and it involves changes to a
> transient "leaking" back into the original persistent, while this code
> is about getting the wrong persistent object back after some changes.
> I don't have a lot of experience with building clojure, but I'll try
> compiling latest master and trying it out. In the meantime, did you
> try this code more than once? Because it includes randomness,
> sometimes it works without problems.
> On Aug 25, 6:03 am, Aaron Bedra <aaron.be...@gmail.com> wrote:
> > Please search through previous messages/tickets before posting new
> > issues. This issue has been fixed as of commit da412909d36551a526ed and
> > will be included in the -beta2 release. It was originally ticketed here:
> > > On Aug 24, 12:27 am, Alan Malloy<a...@malloys.org> wrote:
> > >> On Aug 23, 11:38 pm, Ken Wesson<kwess...@gmail.com> wrote:
> > >>> What does zipmap do if the key seq contains duplications?
> > >> That was my instinct too, but (a) a few thousand numbers won't collide
> > >> very often at all given the problem space, and (b) some experimenting
> > >> indicates that the key-seq is always 100k elements large - no
> > >> duplicates.
> > >> HOWEVER, I did find the problem: the keys themselves don't collide,
> > >> but their hashes do. The number of elements that get messed up is
> > >> related to the number of hash collisions, but I still don't quite
> > >> understand how they interact.
> > >> Here's a modified code snippet that demonstrates the issue:
> > >> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))]
> > >> (println (count (distinct (map hash (keys m)))))
> > >> ((juxt count identity) (persistent!
> > >> (reduce dissoc! (transient m) (keys m)))))
> > >> 100000 ;; no collisions
> > >> [0 {}] ;; map is empty at the end
> > >> That last comment seems to indicate a very serious error somewhere:
> > >> not only is the transient map broken, but it creates a broken
> > >> persistent object. I'll file a JIRA issue for this, and see if I can
> > >> find out any more about the cause.
> Update: just built master, and issue still exists. If you want to be > "sure" you get it, just add another zero to the input range. I'll > mention that in the ticket as well.
> On Aug 25, 10:14 am, Alan Malloy<a...@malloys.org> wrote: >> I did, of course. I searched for "transient", and didn't find any >> describing this issue. And looking at the issue you link to, I still >> don't see how it's related: it's a patch specifically for vectors, and >> this code doesn't touch vectors; and it involves changes to a >> transient "leaking" back into the original persistent, while this code >> is about getting the wrong persistent object back after some changes.
>> I don't have a lot of experience with building clojure, but I'll try >> compiling latest master and trying it out. In the meantime, did you >> try this code more than once? Because it includes randomness, >> sometimes it works without problems.
>> On Aug 25, 6:03 am, Aaron Bedra<aaron.be...@gmail.com> wrote:
>>> Please search through previous messages/tickets before posting new >>> issues. This issue has been fixed as of commit da412909d36551a526ed and >>> will be included in the -beta2 release. It was originally ticketed here: >>> http://dev.clojure.org/jira/browse/CLJ-816 >>> (let [m (into {} (for [x (range 100000)] [(rand) (rand)]))] >>> (println (count (distinct (map hash (keys m))))) >>> ((juxt count identity) (persistent! >>> (reduce dissoc! (transient m) (keys m))))) >>> 100000 >>> [0 {}] >>> Cheers, >>> Aaron Bedra >>> -- >>> Clojure/corehttp://clojure.com >>> On 08/24/2011 04:32 AM, Alan Malloy wrote: >>>> On Aug 24, 12:27 am, Alan Malloy<a...@malloys.org> wrote: >>>>> On Aug 23, 11:38 pm, Ken Wesson<kwess...@gmail.com> wrote: >>>>>> What does zipmap do if the key seq contains duplications? >>>>> That was my instinct too, but (a) a few thousand numbers won't collide >>>>> very often at all given the problem space, and (b) some experimenting >>>>> indicates that the key-seq is always 100k elements large - no >>>>> duplicates. >>>>> HOWEVER, I did find the problem: the keys themselves don't collide, >>>>> but their hashes do. The number of elements that get messed up is >>>>> related to the number of hash collisions, but I still don't quite >>>>> understand how they interact. >>>>> Here's a modified code snippet that demonstrates the issue: >>>>> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))] >>>>> (println (count (distinct (map hash (keys m))))) >>>>> ((juxt count identity) (persistent! >>>>> (reduce dissoc! (transient m) (keys m))))) >>>>> 100000 ;; no collisions >>>>> [0 {}] ;; map is empty at the end >>>>> user> (let [m (apply zipmap (repeatedly 2 #(repeatedly 100000 rand)))] >>>>> (println (count (distinct (map hash (keys m))))) >>>>> ((juxt count identity) (persistent! >>>>> (reduce dissoc! (transient m) (keys m))))) >>>>> 99996 ;; four collisions >>>>> [8 {0.30426231137219917 0.8531183785687654, 0.8893047006425385 >>>>> 0.4788315896128895, 0.47854633997540674 0.45133768991797785, >>>>> 0.5265638224227486 0.7724779126227945}] ;; a four-element map that >>>>> reports its count as eight!!! >>>>> That last comment seems to indicate a very serious error somewhere: >>>>> not only is the transient map broken, but it creates a broken >>>>> persistent object. I'll file a JIRA issue for this, and see if I can >>>>> find out any more about the cause. >>>>> FWIW, I'm using 1.2.1 for the above output. >>>> Ticket is athttp://dev.clojure.org/jira/browse/CLJ-829
