Scouring the Clojure-verse for collections that need Clojure 1.6 hash updates

176 views
Skip to first unread message

Andy Fingerhut

unread,
Feb 14, 2014, 2:36:53 PM2/14/14
to cloju...@googlegroups.com
It would be great if all Clojure contrib libraries were updated (if
needed) for Clojure 1.6's new hash function, before Clojure 1.6 is
released.

It would also be nice to update to 3rd party Clojure libraries that
implement their own custom collections, that would otherwise still
calculate old hash values.  Having a custom set or map that implements
= correctly when comparing against other set and map implementations,
but still calculates the old hash function, can lead to bugs.  For
example, not being able to find a set element/map key if those
collections are themselves used as elements in PersistentHashSet or
keys in PersistentHashMap collections.

Summary of current hash update status:

+ 1 test in Clojure has not been updated for new hash function

+ at least 4 Clojure contrib libraries still need updates

+ 1 3rd party library definitely needs updates, and I'm looking for
  others.

More details below, if you are interested.  Please do reply if you can
think of any other Clojure contrib or 3rd party libraries not already
mentioned below that implement collections that might need updating.

Andy



+ 1 test in Clojure has not been updated for new hash function

Rich disabled a few failing test in Clojure itself when making the
hash changes in this commit:
https://github.com/clojure/clojure/commit/dff9600387b962f16fc78e6477e10e34651fd366

I believe all but one of those disabled tests have since been updated
to be independent of Clojure's hash function with this commit:
https://github.com/clojure/clojure/commit/91dd867b4229a31d4d915aece97f41b3811cf4d4

The only exception I know of is a still-commented-out test in file
test/clojure/test_clojure/java_interop.clj at or near line 311:

    ;;(test-to-passed-array-for hash-set)


+ at least 4 Clojure contrib libraries still need updates

Several Clojure contrib projects had test failures when Clojure
changed its hash functions, but have since been updated to correct
this:

    tools.namespace - http://dev.clojure.org/jira/browse/TNS-16
    tools.nrepl - https://github.com/clojure/tools.nrepl/commit/93de0e0e9b60a4ee3e64918456055bc9f3760bcb

There were probably one or two more such projects in the past, but I
have forgotten the details.



Projects that have test failures, and patches written that correct the
test failures, but not committed yet:

    core.cache - http://dev.clojure.org/jira/browse/CCACHE-33
    core.match - http://dev.clojure.org/jira/browse/MATCH-94


Projects that have no test failures, but only because their Maven-run
test suites currently contain no tests that depend upon Clojure's hash
function -- if they did, they would have test failures.  There are
tests to catch this in Zach Tellman's collection-check library, but
they are currently only run via Leiningen's 'lein test', not Maven.
The tickets mentioned have patches adding such tests and updating the
hash functions for Clojure 1.6, but the patches could use updating to
take advantage of the Clojure APIs Alex Miller added to calculate
hashes of ordered and unordered collections:

    core.rrb-vector - http://dev.clojure.org/jira/browse/CRRBV-3
    data.avl - http://dev.clojure.org/jira/browse/DAVL-3


+ 1 3rd party library definitely needs updates, and I'm looking for
  others.

There are other 3rd party libraries that could use updating for
Clojure 1.6's new hash functions.  Off the top of my head, this one
definitely needs updating:

    immutable-bitset - https://github.com/ztellman/immutable-bitset

There are others that I don't know whether updates are needed:

     core.matrix - https://github.com/mikera/core.matrix
     flatland/useful - https://github.com/flatland/useful
     flatland/ordered - https://github.com/flatland/ordered

Does anyone know of any other collection implementations to check?

Chouser

unread,
Feb 14, 2014, 3:07:24 PM2/14/14
to cloju...@googlegroups.com
data.finger-tree attempts to participate in equality and hashing with
the core collections. Not sure how many combinations are covered in
the tests.

Andy Fingerhut

unread,
Feb 15, 2014, 5:19:10 PM2/15/14
to cloju...@googlegroups.com
Proposed patch to update data.finger-tree here:

    http://dev.clojure.org/jira/browse/DFINGER-2

It appears like there were no tests for the hash functions yet.  I added a few.

Andy



--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/groups/opt_out.

Mikera

unread,
Feb 16, 2014, 3:13:57 AM2/16/14
to cloju...@googlegroups.com
I can confirm that core.matrix is fine with the latest 1.6 beta.

(It doesn't make any assumptions about hash behaviour, so no big surprise there)

core.matrix doesn't make any assumptions about hashing, so I think it is fine. 

Mark Engelberg

unread,
Feb 16, 2014, 5:15:40 PM2/16/14
to clojure-dev
Instaparse will need some rethinking, since it relies on incremental hashing.  I believe that it will be possible to make it work with the 1.6 hashing scheme, since Rich took my request for supporting incremental hashing into account, but it will not be as trivial as some of the other collection updates.  Not sure yet whether it will be possible to accommodate the new hashing scheme and keep it backwards compatible with Clojure 1.5.

icamts

unread,
Feb 17, 2014, 8:50:01 AM2/17/14
to cloju...@googlegroups.com
Hi all,
may these projects: https://github.com/reiddraper/simple-check and https://github.com/ztellman/collection-check be useful to anyone testing their own collections?

Cheers,
Luca

Alex Miller

unread,
Feb 17, 2014, 9:49:13 AM2/17/14
to cloju...@googlegroups.com
Yes.

Zach Tellman

unread,
Feb 17, 2014, 5:35:23 PM2/17/14
to cloju...@googlegroups.com
Potemkin has a map implementation that is currently breaking for 1.6.  I will fix it this week.

Andy Fingerhut

unread,
Feb 18, 2014, 2:55:41 PM2/18/14
to cloju...@googlegroups.com
I have created a table at the bottom of the "Better Hashing" page to help track libraries that need updating for Clojure 1.6 hashing, and their current status regarding those updates.

    http://dev.clojure.org/display/design/Better+hashing

Thanks for all those who have replied so far -- I believe I have included their feedback in that table.  I will attempt to follow up with those libraries periodically to see how things are going.   

Andy

Mark Engelberg

unread,
Feb 19, 2014, 2:13:19 AM2/19/14
to clojure-dev
I have finished the work needed to migrate instaparse's incremental hashing to Clojure 1.6.  I have also successfully made it backwards-compatible, using the compile-if macro you suggested.

I'm curious, what kind of speed difference are the rest of you seeing between 1.5.1 and 1.6?  On my code, 1.6 seems around 10-20% slower.  (Even using criterium to benchmark I get quite a bit of variation on 1.6 running times, so it's hard to pinpoint the speed difference better than that).
Reply all
Reply to author
Forward
0 new messages