[ANN] Clojure 1.7.0-alpha3 now available

265 views
Skip to first unread message

Alex Miller

unread,
Oct 26, 2014, 11:43:15 PM10/26/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Clojure 1.7.0-alpha3 is now available. 
 
Try it via 
- Leiningen: [org.clojure/clojure "1.7.0-alpha3"]

For users of Clojure 1.7.0-alpha2, there have been a few important changes in features under development:

Transducers:
- "iteration" function renamed to "eduction"
- Fixed several issues related to reduced values

*unchecked-math* boxed math warnings:
- warnings are now only emitted when (set! *unchecked-math* :warn-on-boxed). for old behavior (no warnings), use (set! *unchecked-math* true).

For other changes new in alpha3, see the issues marked "(alpha3)" in the changes below.

------------------------------------------------------------
Clojure 1.7.0-alpha3 has the changes below from 1.6.0:

## 1 New and Improved Features

### 1.1 Transducers

Transducers is a new way to decouple algorithmic transformations from their
application in different contexts. Transducers are functions that transform
reducing functions to build up a "recipe" for transformation.


Many existing sequence functions now have a new arity (one fewer argument
than before). This arity will return a transducer that represents the same
logic but is independent of lazy sequence processing. Functions included are:

* conj (conjs to [])
* map
* mapcat
* filter
* remove
* take
* take-while
* drop
* drop-while
* cycle
* take-nth
* replace
* partition-by
* partition-all
* keep
* keep-indexed

Additionally some new transducer functions have been added:

* cat - concatenates the contents of each input
* de-dupe - removes consecutive duplicated values
* random-sample - returns items from coll with random probability

And this function can be used to make completing transforms:

* completing

There are also several new or modified functions that can be used to apply
transducers in different ways:

* sequence - takes a transformation and a coll and produces a lazy seq
* transduce - reduce with a transformation (eager)
* eduction - returns a reducible/seqable/iterable seq of applications of the transducer to items in coll. Applications are re-performed with every reduce/seq/iterator.
* run! - run the transformation for side effects on the collection

There have been a number of internal changes to support transducers:

* volatiles - there are a new set of functions (volatile!, vswap!, vreset!, volatile?) to create and use volatile "boxes" to hold state in stateful transducers. Volatiles are faster than atoms but give up atomicity guarantees so should only be used with thread isolation.
* array iterators - added support for iterators over arrays

Some related issues addressed during development:

### 1.2 Keyword and Symbol Construction

In response to issues raised in [CLJ-1439](http://dev.clojure.org/jira/browse/CLJ-1439), several changes have been made in symbol and keyword construction:

1) The main bottleneck in construction of symbols (which also occurs inside keywords) was interning of the name and namespace strings. This interning has been removed, resulting in a performance increase.

2) Keywords are cached and keyword construction includes a cache check. A change was made to only clear the cache reference queue when there is a cache miss.

### 1.3 Warn on Boxed Math

One source of performance issues is the (unintended) use of arithmetic operations on boxed numbers. To make detecting the presence of boxed math easier, a warning will now be emitted about boxed math if \*unchecked-math* is set to :warn-on-boxed (any truthy value will enable unchecked-math, only this specific value enables the warning).

Example use:

    user> (defn plus-2 [x] (+ x 2))  ;; no warning, but boxed
#'user/plus-2
    user> (set! *unchecked-math* :warn-on-boxed)
true
    user> (defn plus-2 [x] (+ x 2)) ;; now we see a warning
    Boxed math warning, NO_SOURCE_PATH:10:18 - call: public static java.lang.Number
clojure.lang.Numbers.unchecked_add(java.lang.Object,long).
    #'user/plus-2
user> (defn plus-2 [^long x] (+ x 2)) ;; use a hint to avoid boxing
#'user/plus-2


### 1.4 update - like update-in for first level

`update` is a new function that is like update-in specifically for first-level keys:

    (update m k f args...)

Example use:

    user> (update {:a 1} :a inc)
{:a 2}
user> (update {:a 1} :a + 2)
{:a 3}
user> (update {} :a identity)  ;; missing returns nil
{:a nil}


## 2 Enhancements

### 2.1 Error messages

  Invalid defrecord results in exception attributed to consuming ns instead of defrecord ns
  Report line,column, and source in defmacro errors
  Give more specific hint if namespace with "-" not found to check file uses "_"

### 2.2 Documentation strings

  clojure.java.io/input-stream has incorrect docstring
  Fix typo in gen-class doc-string
  Fix typo in filterv example
  Fix typo in defmulti docstring
  Fix typo in deftype docstring
  Fix typo in clojure.main usage

### 2.3 Performance

  Improve performance of partial with more unrolling
  clojure.core/set should use transients for better performance
  Cache unknown multimethod value default dispatch

### 2.4 Other enhancements

  Improve apropos to show some indication of namespace of symbols found
  Hints don't work with #() form of function
  Removes owner-thread check from transients - this check was preventing some valid usage of transients in core.async where a transient is created on one thread and then used again in another pooled thread (while still maintaining thread isolation).
  Extracted IAtom interface implemented by Atom.
  Don't initialize classes when importing them
  Class name clash between top-level functions and defn'ed ones

## 3 Bug Fixes

  Reduce broken on some primitive vectors
  Equality bug on records created with nested calls to map->record
  Unable to set compiler options via system properties except for AOT compilation
  NPE when AOTing overrided clojure.core functions
  reductions does not check for reduced value
  Using def with metadata {:type :anything} throws ClassCastException during printing
  Error when calling primitive functions with destructuring in the arg vector
  Piping seque into seque can deadlock
  <= is incorrect when args include Double/NaN
  Make cached string value of Keyword and Symbol transient
  clojure.core/bean should implement Iterable



Stefan Kamphausen

unread,
Oct 28, 2014, 10:34:29 AM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Hi,

have there been any changes how fns with a name and recursion are compiled?  One of my projects has a function which does not compile with 1.7.0-alpha3 anymore, but did fine with 1.6.0.

I tried to create a minimal example at https://github.com/ska2342/nested-fn-breaks-clojure-17

(I know the function itself is probably stupid, I just wanted to demonstrate the case.  I don't know if it even runs.)

Compilation breaks with a java.io.IOException: File name too long


The problem seems to be the combination of

* using a long function name (not a bad thing per se),
* using a rather long name for a local binding (not common in Clojure-land, used in my case for documentation of the intent of the anon fn),
* and using a name for the anonymous function (needed for recursion and usually a good idea because it improves stacktraces, but maybe you added the local binding to the name for exactly that reason).

Regarding the second (long var binding name), my original function uses shorter names, but has some nested constructs (for, cond, ...) which seem to make the name larger, too.  There is really nothing unusually long there.

Of course, I can work around this by using different names, factoring an inner anon function out to a defn and probably in other ways.  I just wanted to make sure, that you are aware that problems like this may show up and made the change on purpose.


Thanks,
stefan

Andy Fingerhut

unread,
Oct 28, 2014, 10:42:17 AM10/28/14
to cloju...@googlegroups.com
That possibility was foreseen with a patch made for ticket CLJ-1330:


There was an alternate patch that did not include the name of the nested constructs that was also considered.

Andy

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

Alex Miller

unread,
Oct 28, 2014, 10:54:31 AM10/28/14
to cloju...@googlegroups.com
Yes, that patch is certainly the culprit. That project actually works fine on my machine (mac osx). I do not believe that Java itself actually has a limit on this (other than max string length), you're instead hitting a limitation on your OS - what is your specific environment (OS, JDK version, etc)? 

Alex Miller

unread,
Oct 28, 2014, 10:56:53 AM10/28/14
to cloju...@googlegroups.com
Also, if you could file a ticket with this info, that would be very helpful. If not, let me know and I'll file it. We will treat this as a regression and consider alternatives.

Stefan Kamphausen

unread,
Oct 28, 2014, 10:59:33 AM10/28/14
to cloju...@googlegroups.com
Hi,


On Tuesday, October 28, 2014 3:54:31 PM UTC+1, Alex Miller wrote:
Yes, that patch is certainly the culprit. That project actually works fine on my machine (mac osx). I do not believe that Java itself actually has a limit on this (other than max string length), you're instead hitting a limitation on your OS - what is your specific environment (OS, JDK version, etc)? 


Ubuntu 12.04 LTS 64bit
Java8

Problem occurs sometimes with Java classes, because I use the crypto container which limits the file name max significantly:

shell> getconf NAME_MAX `pwd`
143

 
On Tue, Oct 28, 2014 at 9:42 AM, Andy Fingerhut <andy.fi...@gmail.com> wrote:
That possibility was foreseen with a patch made for ticket CLJ-1330:


There was an alternate patch that did not include the name of the nested constructs that was also considered.


Thanks for pointing this out. 

I am fine with this.


Best,
Stefan

Alex Miller

unread,
Oct 28, 2014, 1:12:36 PM10/28/14
to cloju...@googlegroups.com
I've reopened CLJ-1330 for this and we'll definitely do something with it for the next alpha or beta. Thanks for the report!!

Alex Miller

unread,
Oct 28, 2014, 4:53:05 PM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
We will definitely be rolling this particular aspect of CLJ-1330 back in the next release.

Alex

On Tuesday, October 28, 2014 10:44:51 AM UTC-5, Sam Ritchie wrote:
Yeah, I'm getting this too with a file named "wparankings.clj". I haven't investigated which function is causing this, but this is definitely troubling.

Stefan Kamphausen

unread,
Oct 29, 2014, 7:57:22 AM10/29/14
to cloju...@googlegroups.com
Oh dear,


just with 1.6 I jump out of the box rather late and postpone a release.  Sorry!  Will try to test things earlier for 1.8 :-)

If you need anything else from me, just let me know.   Feel free to call our off-list anytime.

In case, it's important:  The problems in my real code happen in macro-expanded for-expressions which contains more anonymous function beyond my control.


Best regards,
stefan

Alex Miller

unread,
Oct 29, 2014, 8:08:24 AM10/29/14
to cloju...@googlegroups.com
Not postponed at all - this is why we release alphas! The replacement patch does not have this issue, so it's easy to address.

--

Max Penet

unread,
Oct 29, 2014, 8:25:12 AM10/29/14
to cloju...@googlegroups.com, clo...@googlegroups.com
Just curious, why http://dev.clojure.org/jira/browse/CLJ-1517 didn't make it. It seemed complete at first glance.

Alex Miller

unread,
Oct 29, 2014, 9:49:39 AM10/29/14
to clo...@googlegroups.com, cloju...@googlegroups.com
It's a big change and will require a fair amount of screening and analysis, none of which has been done yet.

Michael Blume

unread,
Oct 31, 2014, 4:34:23 PM10/31/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Picking up the variadic arguments discussion, it seems that in a simple definition like

(fn [& args] (apply f arg1 args))

One could conceivably put some sort of preprocessing smarts into the fn macro that notices that

a) this is a variadic arglist
b) the variadic arg (args) is a symbol, not some complicated destructuring expression
c) args is only ever used as the last argument to apply (maybe check this after thoroughly macroexpanding the function body)

and then autogenerates the various unrolled cases.

Even if we did only the simplest version of this, the new definition of partial would be

(defn partial
  "Takes a function f and fewer than the normal arguments to f, and
  returns a fn that takes a variable number of additional args. When
  called, the returned function calls f with args + additional args."
  {:added "1.0"
   :static true}
  ([f] f)
  ([f arg1]
   (fn [& args]
     (apply f arg1 args)))
  ([f arg1 arg2]
   (fn [& args]
     (apply f arg1 arg2 args)))
  ([f arg1 arg2 arg3]
   (fn [& args]
     (apply f arg1 arg2 arg3 args)))
  ([f arg1 arg2 arg3 & more]
   (fn [& args] (apply f arg1 arg2 arg3 (concat more args)))))

which is shorter (and would be unrolled in more cases than it currently is)

The thing that concerns me is that such a preprocessing step would be treating 'clojure.core/apply as a magic token, when apply is supposed to be a function like any other. And sufficient smarts to make 'partial single-clause would probably require treating 'clojure.core/concat as a magic token as well.
Reply all
Reply to author
Forward
0 new messages