[ANN] Clojure 1.7.0-alpha5 now available

405 views
Skip to first unread message

Alex Miller

unread,
Jan 10, 2015, 10:36:51 AM1/10/15
to clo...@googlegroups.com, cloju...@googlegroups.com
Clojure 1.7.0-alpha5 is now available.

Try it via
- Leiningen: [org.clojure/clojure "1.7.0-alpha5"]

A few of the highlights in alpha5:

1) New transducer arities for map-indexed, distinct, and interpose.
The transducer versions are generally faster than the sequence versions.
See CLJ-1601 for more details.

2) The set and vec functions should now be faster in many cases. Some of the
vec improvements will not kick in until CLJ-1499 is committed in the future.
See CLJ-1546 and CLJ-1618 for more details.

3) Two particularly troublesome AOT classloading bugs have been addressed - 
CLJ-979 and CLJ-1544. Many other tickets were found to be duplicates of one of
these as well. Big thanks to Nicola Mometto for doing the hard work on 
debugging and providing fixes for these.

For all changes new in alpha5, see the issues marked "(alpha5)" in the
full changes below.

------------------------------------------------------------
Clojure 1.7.0-alpha5 has the changes below from 1.6.0:

## 1 New and Improved Features

### 1.1 Transducers

Transducers is a new way to decouple algorithmic transformations from their
application in different contexts. Transducers are functions that transform
reducing functions to build up a "recipe" for transformation.


Many existing sequence functions now have a new arity (one fewer argument
than before). This arity will return a transducer that represents the same
logic but is independent of lazy sequence processing. Functions included are:

* conj (conjs to [])
* map
* mapcat
* filter
* remove
* take
* take-while
* drop
* drop-while
* cycle
* take-nth
* replace
* partition-by
* partition-all
* keep
* keep-indexed
* map-indexed (alpha5)
* distinct (alpha5)
* interpose (alpha5)

Additionally some new transducer functions have been added:

* cat - concatenates the contents of each input
* de-dupe - removes consecutive duplicated values
* random-sample - returns items from coll with random probability

And this function can be used to make completing transforms:

* completing

There are also several new or modified functions that can be used to apply
transducers in different ways:

* sequence - takes a transformation and a coll and produces a lazy seq
* transduce - reduce with a transformation (eager)
* eduction - returns a reducible/seqable/iterable seq of applications of the transducer to items in coll. Applications are re-performed with every reduce/seq/iterator.
* run! - run the transformation for side effects on the collection

There have been a number of internal changes to support transducers:

* volatiles - there are a new set of functions (volatile!, vswap!, vreset!, volatile?) to create and use volatile "boxes" to hold state in stateful transducers. Volatiles are faster than atoms but give up atomicity guarantees so should only be used with thread isolation.
* array iterators - added support for iterators over arrays

Some related issues addressed during development:

### 1.2 Keyword and Symbol Construction

In response to issues raised in [CLJ-1439](http://dev.clojure.org/jira/browse/CLJ-1439),
several changes have been made in symbol and keyword construction:

1) The main bottleneck in construction of symbols (which also occurs inside keywords) was
interning of the name and namespace strings. This interning has been removed, resulting
in a performance increase.

2) Keywords are cached and keyword construction includes a cache check. A change was made
to only clear the cache reference queue when there is a cache miss.

### 1.3 Warn on Boxed Math

One source of performance issues is the (unintended) use of arithmetic operations on
boxed numbers. To make detecting the presence of boxed math easier, a warning will now
be emitted about boxed math if \*unchecked-math* is set to :warn-on-boxed (any truthy
value will enable unchecked-math, only this specific value enables the warning).

Example use:

    user> (defn plus-2 [x] (+ x 2))  ;; no warning, but boxed
#'user/plus-2
    user> (set! *unchecked-math* :warn-on-boxed)
true
    user> (defn plus-2 [x] (+ x 2)) ;; now we see a warning
    Boxed math warning, NO_SOURCE_PATH:10:18 - call: public static java.lang.Number
clojure.lang.Numbers.unchecked_add(java.lang.Object,long).
    #'user/plus-2
user> (defn plus-2 [^long x] (+ x 2)) ;; use a hint to avoid boxing
#'user/plus-2


### 1.4 update - like update-in for first level

`update` is a new function that is like update-in specifically for first-level keys:

    (update m k f args...)

Example use:

    user> (update {:a 1} :a inc)
{:a 2}
user> (update {:a 1} :a + 2)
{:a 3}
user> (update {} :a identity)  ;; missing returns nil
{:a nil}


## 2 Enhancements

### 2.1 Error messages

  Invalid defrecord results in exception attributed to consuming ns instead of defrecord ns
  Report line,column, and source in defmacro errors
  Give more specific hint if namespace with "-" not found to check file uses "_"

### 2.2 Documentation strings

  clojure.java.io/input-stream has incorrect docstring
  Fix typo in gen-class doc-string
  Fix typo in filterv example
  Fix typo in defmulti docstring
  Fix typo in deftype docstring
  Fix typo in clojure.main usage

### 2.3 Performance

  Improve performance of partial with more unrolling
  clojure.core/set should use transients for better performance
  Cache unknown multimethod value default dispatch
  Reduce compile times by avoiding unnecessary calls to Class.forName()
  vec is now faster on almost all inputs
  set is now faster on almost all inputs

### 2.4 Other enhancements

  Improve apropos to show some indication of namespace of symbols found
  Hints don't work with #() form of function
  Removes owner-thread check from transients - this check was preventing some valid usage of transients in core.async where a transient is created on one thread and then used again in another pooled thread (while still maintaining thread isolation).
  Extracted IAtom interface implemented by Atom.
  Don't initialize classes when importing them
  Class name clash between top-level functions and defn'ed ones
  Update to latest test.generative and add dependency on test.check
  vec now works with things that only implement Iterable or IReduceInit
  set now works with things that only implement Iterable or IReduceInit
  PersistentList/creator doesn't handle ArraySeqs correctly

## 3 Bug Fixes

  Reduce broken on some primitive vectors
  Equality bug on records created with nested calls to map->record
  Unable to set compiler options via system properties except for AOT compilation
  NPE when AOTing overrided clojure.core functions
  reductions does not check for reduced value
  Using def with metadata {:type :anything} throws ClassCastException during printing
  Error when calling primitive functions with destructuring in the arg vector
  Piping seque into seque can deadlock
  <= is incorrect when args include Double/NaN
  Make cached string value of Keyword and Symbol transient
  clojure.core/bean should implement Iterable
  Make refer of Clojure core function not throw exception on reload
  LazySeq equals() should not use equiv() logic
  into (and other fns that rely on reduce) require only IReduceInit
  PersistentVector now directly implements reduce without init
  Transient collections should guarantee thread visibility
  Some IReduce/IReduceInit implementors don't respect reduced
  Clojure resolves to wrong deftype classes when AOT compiling or reloading
  AOT bug involving namespaces loaded before AOT compilation started

tcrayford

unread,
Jan 12, 2015, 1:37:06 PM1/12/15
to clo...@googlegroups.com, cloju...@googlegroups.com
I said this already on twitter, but I see heavy (5-10x slower) perf regressions on my benchmark suite from 1.7-alpha5 vs 1.6. I am trying the other alphas now - I have a comprehensive benchmark suite that takes 4 hours or so to run, and it's a bit backed up right now (I only have one box to run it on). I'm also going to point YourKit at 1.7-alpha5 soon on the worst regressing benchmarks (not all of them have such heavy regressions, and a few benchmarks are dramatically faster) and see what happens - will report back here once I have something more to say.

Alex Miller

unread,
Jan 16, 2015, 4:07:27 PM1/16/15
to clo...@googlegroups.com, cloju...@googlegroups.com
Just to follow up on this, Tom reported to me on Twitter that the box giving slow results was bad and the regression was not repeatable on another box.

Alex

Sean Corfield

unread,
Jan 20, 2015, 3:57:38 PM1/20/15
to cloju...@googlegroups.com, clo...@googlegroups.com
Just FYI, we took 1.7.0-alpha5 into production today. We’ll let you know if we run into any issues. Other than tripping over the AOT/JIT thing with core.typed, it’s been smooth sailing on dev/QA…

…and we actually have a transducer in production now!

Sean

Jozef Wagner

unread,
Jan 26, 2015, 1:49:37 PM1/26/15
to clo...@googlegroups.com, cloju...@googlegroups.com
With CLJ-979 applied, lein may crash when doing :aot :all. This is caused when lein forces recompilation of already compiled classes. Not sure whether this should be considered a lein of clojure bug though.

See https://github.com/wagjo/aotbug for contrived example.

Jozef

Nicola Mometto

unread,
Jan 29, 2015, 1:24:50 PM1/29/15
to cloju...@googlegroups.com, clo...@googlegroups.com

Jozef, I looked into this and opened a ticket with a proposed patch to
fix this issue: http://dev.clojure.org/jira/browse/CLJ-1650
--
Reply all
Reply to author
Forward
0 new messages