[ANN] Clojure 1.7.0-alpha3 now available

825 views
Skip to first unread message

Alex Miller

unread,
Oct 26, 2014, 11:43:23 PM10/26/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Clojure 1.7.0-alpha3 is now available. 
 
Try it via 
- Leiningen: [org.clojure/clojure "1.7.0-alpha3"]

For users of Clojure 1.7.0-alpha2, there have been a few important changes in features under development:

Transducers:
- "iteration" function renamed to "eduction"
- Fixed several issues related to reduced values

*unchecked-math* boxed math warnings:
- warnings are now only emitted when (set! *unchecked-math* :warn-on-boxed). for old behavior (no warnings), use (set! *unchecked-math* true).

For other changes new in alpha3, see the issues marked "(alpha3)" in the changes below.

------------------------------------------------------------
Clojure 1.7.0-alpha3 has the changes below from 1.6.0:

## 1 New and Improved Features

### 1.1 Transducers

Transducers is a new way to decouple algorithmic transformations from their
application in different contexts. Transducers are functions that transform
reducing functions to build up a "recipe" for transformation.


Many existing sequence functions now have a new arity (one fewer argument
than before). This arity will return a transducer that represents the same
logic but is independent of lazy sequence processing. Functions included are:

* conj (conjs to [])
* map
* mapcat
* filter
* remove
* take
* take-while
* drop
* drop-while
* cycle
* take-nth
* replace
* partition-by
* partition-all
* keep
* keep-indexed

Additionally some new transducer functions have been added:

* cat - concatenates the contents of each input
* de-dupe - removes consecutive duplicated values
* random-sample - returns items from coll with random probability

And this function can be used to make completing transforms:

* completing

There are also several new or modified functions that can be used to apply
transducers in different ways:

* sequence - takes a transformation and a coll and produces a lazy seq
* transduce - reduce with a transformation (eager)
* eduction - returns a reducible/seqable/iterable seq of applications of the transducer to items in coll. Applications are re-performed with every reduce/seq/iterator.
* run! - run the transformation for side effects on the collection

There have been a number of internal changes to support transducers:

* volatiles - there are a new set of functions (volatile!, vswap!, vreset!, volatile?) to create and use volatile "boxes" to hold state in stateful transducers. Volatiles are faster than atoms but give up atomicity guarantees so should only be used with thread isolation.
* array iterators - added support for iterators over arrays

Some related issues addressed during development:

### 1.2 Keyword and Symbol Construction

In response to issues raised in [CLJ-1439](http://dev.clojure.org/jira/browse/CLJ-1439), several changes have been made in symbol and keyword construction:

1) The main bottleneck in construction of symbols (which also occurs inside keywords) was interning of the name and namespace strings. This interning has been removed, resulting in a performance increase.

2) Keywords are cached and keyword construction includes a cache check. A change was made to only clear the cache reference queue when there is a cache miss.

### 1.3 Warn on Boxed Math

One source of performance issues is the (unintended) use of arithmetic operations on boxed numbers. To make detecting the presence of boxed math easier, a warning will now be emitted about boxed math if \*unchecked-math* is set to :warn-on-boxed (any truthy value will enable unchecked-math, only this specific value enables the warning).

Example use:

    user> (defn plus-2 [x] (+ x 2))  ;; no warning, but boxed
#'user/plus-2
    user> (set! *unchecked-math* :warn-on-boxed)
true
    user> (defn plus-2 [x] (+ x 2)) ;; now we see a warning
    Boxed math warning, NO_SOURCE_PATH:10:18 - call: public static java.lang.Number
clojure.lang.Numbers.unchecked_add(java.lang.Object,long).
    #'user/plus-2
user> (defn plus-2 [^long x] (+ x 2)) ;; use a hint to avoid boxing
#'user/plus-2


### 1.4 update - like update-in for first level

`update` is a new function that is like update-in specifically for first-level keys:

    (update m k f args...)

Example use:

    user> (update {:a 1} :a inc)
{:a 2}
user> (update {:a 1} :a + 2)
{:a 3}
user> (update {} :a identity)  ;; missing returns nil
{:a nil}


## 2 Enhancements

### 2.1 Error messages

  Invalid defrecord results in exception attributed to consuming ns instead of defrecord ns
  Report line,column, and source in defmacro errors
  Give more specific hint if namespace with "-" not found to check file uses "_"

### 2.2 Documentation strings

  clojure.java.io/input-stream has incorrect docstring
  Fix typo in gen-class doc-string
  Fix typo in filterv example
  Fix typo in defmulti docstring
  Fix typo in deftype docstring
  Fix typo in clojure.main usage

### 2.3 Performance

  Improve performance of partial with more unrolling
  clojure.core/set should use transients for better performance
  Cache unknown multimethod value default dispatch

### 2.4 Other enhancements

  Improve apropos to show some indication of namespace of symbols found
  Hints don't work with #() form of function
  Removes owner-thread check from transients - this check was preventing some valid usage of transients in core.async where a transient is created on one thread and then used again in another pooled thread (while still maintaining thread isolation).
  Extracted IAtom interface implemented by Atom.
  Don't initialize classes when importing them
  Class name clash between top-level functions and defn'ed ones

## 3 Bug Fixes

  Reduce broken on some primitive vectors
  Equality bug on records created with nested calls to map->record
  Unable to set compiler options via system properties except for AOT compilation
  NPE when AOTing overrided clojure.core functions
  reductions does not check for reduced value
  Using def with metadata {:type :anything} throws ClassCastException during printing
  Error when calling primitive functions with destructuring in the arg vector
  Piping seque into seque can deadlock
  <= is incorrect when args include Double/NaN
  Make cached string value of Keyword and Symbol transient
  clojure.core/bean should implement Iterable



Phillip Lord

unread,
Oct 27, 2014, 3:23:15 AM10/27/14
to clo...@googlegroups.com

I am curious, with path CLJ-1430 why stop at 3 arguments? Why not unroll to 20?

I ask for a practical reason. In my own library, I have unrolled several functions which get called
a lot. Moving from all variadic to non-variadic up to 5 args has produced a given a 2x increase in
my code under extreme circumstances. I was thinking of macro'ing it out to an arity of 20, but
haven't yet, just because it seems a step too far.

Phil


________________________________________
From: clo...@googlegroups.com [clo...@googlegroups.com] on behalf of Alex Miller [al...@puredanger.com]
Sent: 27 October 2014 03:42
To: clo...@googlegroups.com; cloju...@googlegroups.com
Subject: [ANN] Clojure 1.7.0-alpha3 now available
clojure.java.io/input-stream<http://clojure.java.io/input-stream> has incorrect docstring
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com<mailto:clojure+u...@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.

Alex Miller

unread,
Oct 27, 2014, 10:02:42 AM10/27/14
to clo...@googlegroups.com
Most Clojure core functions unroll to 3 or 4, really just depends how much of the tail you think is worth picking up. I don't think we have any hard and fast rule on it. 
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com<mailto:clojure+unsu...@googlegroups.com>.

Phillip Lord

unread,
Oct 27, 2014, 10:36:02 AM10/27/14
to clo...@googlegroups.com


And the performance boost is just to avoid the list creation in the
variadic argument?

Alex Miller <al...@puredanger.com> writes:

> Most Clojure core functions unroll to 3 or 4, really just depends how much
> of the tail you think is worth picking up. I don't think we have any hard
> and fast rule on it.
>
> On Monday, October 27, 2014 2:23:15 AM UTC-5, Phillip Lord wrote:
>>
>>
>> I am curious, with path CLJ-1430 why stop at 3 arguments? Why not unroll
>> to 20?
>>
>> I ask for a practical reason. In my own library, I have unrolled several
>> functions which get called
>> a lot. Moving from all variadic to non-variadic up to 5 args has produced
>> a given a 2x increase in
>> my code under extreme circumstances. I was thinking of macro'ing it out to
>> an arity of 20, but
>> haven't yet, just because it seems a step too far.
>>
>> Phil
>>
>>
>> ________________________________________
>> From: clo...@googlegroups.com <javascript:> [clo...@googlegroups.com
>> <javascript:>] on behalf of Alex Miller [al...@puredanger.com
>> <javascript:>]
>> Sent: 27 October 2014 03:42
>> To: clo...@googlegroups.com <javascript:>; cloju...@googlegroups.com
>> <javascript:>
>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1430&sa=D&sntz=1&usg=AFQjCNHchsdAYlVCq1vKTSyZYoGRbdBn6g>)
>> To post to this group, send email to clo...@googlegroups.com <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:><mailto:
>> clojure+u...@googlegroups.com <javascript:>>.
>> For more options, visit https://groups.google.com/d/optout.
>>

--
Phillip Lord, Phone: +44 (0) 191 222 7827
Lecturer in Bioinformatics, Email: philli...@newcastle.ac.uk
School of Computing Science, http://homepages.cs.ncl.ac.uk/phillip.lord
Room 914 Claremont Tower, skype: russet_apples
Newcastle University, twitter: phillord
NE1 7RU

Alex Miller

unread,
Oct 27, 2014, 1:50:55 PM10/27/14
to clo...@googlegroups.com
On Monday, October 27, 2014 9:36:02 AM UTC-5, Phillip Lord wrote:

And the performance boost is just to avoid the list creation in the
variadic argument?

Yes. There's some perf timings in the ticket.

Timothy Baldridge

unread,
Oct 27, 2014, 2:02:44 PM10/27/14
to clo...@googlegroups.com
Both apply and variadic functions have some overhead. This helps address both. 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

Reid McKenzie

unread,
Oct 27, 2014, 4:14:02 PM10/27/14
to clo...@googlegroups.com

For grins I slapped together this benchmark, which tries to compare calls with callsite static arity to application both for fully unrolled and partially unrolled functions. The results I see indicate that in both cases the cost of apply is almost equal for all arities. For invocations/call sites with statically known arities, there seems to be a factor of ~5x speedup compared to function application. The fully unrolled function seems to do ~2x better than the “base” function for all call site arities. This suggests that apply is immensely expensive in general, and that such arity unrolling even for trivial functions would be a good thing. Albeit hard to build.

Reid

Reid McKenzie

unread,
Oct 27, 2014, 4:15:54 PM10/27/14
to clo...@googlegroups.com
Oops. Linked an old benchmark version. https://www.refheap.com/92391

Reid

Phillip Lord

unread,
Oct 28, 2014, 5:39:27 AM10/28/14
to clo...@googlegroups.com
Reid McKenzie <rmcke...@gmail.com> writes:
> This suggests that |apply| is immensely expensive in general,
> and that such arity unrolling even for trivial functions would be a good
> thing. Albeit hard to build.


Wonder whether it is macroable. Something like


(def new-function
(with-arities [20]
[args]
(blah args)
[args & rest]
(apply args rest)))


which gets replaced with

(fn
([]
(blah))
([a]
(blah a))
;;...etc
([a...t]
(blah a...t))
([a...t & rest]
(apply blah a...t rest)))


Where a...t are the symbols a to t. Would this help? The function blah,
of course, would know what it's arity is, but this is true for most
higher-order functions (if functions returned their arities, I guess you
could work around this, but they don't).

So, partial would become something like...

(def partial
(with-arities [:all]
[args]
(fn [args] (args))
[args & rest]
(fn [& rst] (apply (list* args rst)))))


Just thinking aloud!

Phil

Andy Fingerhut

unread,
Oct 28, 2014, 10:29:16 AM10/28/14
to clo...@googlegroups.com

Stefan Kamphausen

unread,
Oct 28, 2014, 10:34:28 AM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Hi,

have there been any changes how fns with a name and recursion are compiled?  One of my projects has a function which does not compile with 1.7.0-alpha3 anymore, but did fine with 1.6.0.

I tried to create a minimal example at https://github.com/ska2342/nested-fn-breaks-clojure-17

(I know the function itself is probably stupid, I just wanted to demonstrate the case.  I don't know if it even runs.)

Compilation breaks with a java.io.IOException: File name too long


The problem seems to be the combination of

* using a long function name (not a bad thing per se),
* using a rather long name for a local binding (not common in Clojure-land, used in my case for documentation of the intent of the anon fn),
* and using a name for the anonymous function (needed for recursion and usually a good idea because it improves stacktraces, but maybe you added the local binding to the name for exactly that reason).

Regarding the second (long var binding name), my original function uses shorter names, but has some nested constructs (for, cond, ...) which seem to make the name larger, too.  There is really nothing unusually long there.

Of course, I can work around this by using different names, factoring an inner anon function out to a defn and probably in other ways.  I just wanted to make sure, that you are aware that problems like this may show up and made the change on purpose.


Thanks,
stefan

Fluid Dynamics

unread,
Oct 28, 2014, 10:34:32 AM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
On Sunday, October 26, 2014 11:43:23 PM UTC-4, Alex Miller wrote:
### 1.4 update - like update-in for first level

`update` is a new function that is like update-in specifically for first-level keys:

    (update m k f args...)

Example use:

    user> (update {:a 1} :a inc)
{:a 2}
user> (update {:a 1} :a + 2)
{:a 3}
user> (update {} :a identity)  ;; missing returns nil
{:a nil}


Should there not also be an update! for transients, then, just as the other single-level map writer that has an -in variant, assoc, has assoc! for transients?

Sam Ritchie

unread,
Oct 28, 2014, 11:44:51 AM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Yeah, I'm getting this too with a file named "wparankings.clj". I haven't investigated which function is causing this, but this is definitely troubling.

October 28, 2014 at 8:34 AM
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
October 26, 2014 at 9:42 PM
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Sam Ritchie (@sritchie)

Phillip Lord

unread,
Oct 28, 2014, 12:03:19 PM10/28/14
to clo...@googlegroups.com


Ah, if only some one had implemented one.

I guess there is no reason NOT to implement this? It's still only a
single class, and single object per function? The class is going to be
longer, because method bodies like...

public Object invoke(Object arg1, Object arg2) {
return throwArity(2);
}

will get overridden with implementation, so I guess it *might* slow
start up time, and an AOT jar will get bigger.

So, the question is, did it just get forgotten or is it harder than it
looks?

Phil

Reid McKenzie

unread,
Oct 28, 2014, 12:06:59 PM10/28/14
to clo...@googlegroups.com
Phillip,

Alan was talking about this on IRC yesterday with me an Alex... I think in general you can't "just" macro this, because hand unrolling is generating the nth duff's device by partially applying the "base" case over N elements from the "head" of varargs. This is pretty straightforward for cases where the varargs case is just a reduce as above but in the general case I think you'll need a full blown partial evaluator.

For the special case of a "reduction" over varargs as in #'clojure.core/+ or #'clojure.core// where you're just generating Duff's I think the macro should be pretty straightforward but it won't be general at all. However I suspect that there are enough "trivial reduction" varargs functions as such that this may well be worth exploring. If I have time I'll kick this around some more today.

Reid

Phillip Lord

unread,
Oct 28, 2014, 1:03:48 PM10/28/14
to clo...@googlegroups.com
Reid McKenzie <rmcke...@gmail.com> writes:
> For the special case of a "reduction" over varargs as in #'clojure.core/+
> or #'clojure.core// where you're just generating Duff's I think the macro
> should be pretty straightforward but it won't be general at all.

I tend to agree with the lack of generality -- from a little thinking,
that was the conclusion I've come to as well. Though, in my case, this
could easily be just because I didn't think about it enough.

> However I suspect that there are enough "trivial reduction" varargs
> functions as such that this may well be worth exploring. If I have
> time I'll kick this around some more today.


For my use case, I have four very similar and ideally very high arity.
I've already partially macro'ed them to save typing, but it's possible I
should go the rest of the way.

If you get anything working, do let me know. I'd be happy to steal
ideas:-)

Phil

Alex Miller

unread,
Oct 28, 2014, 4:53:04 PM10/28/14
to clo...@googlegroups.com, cloju...@googlegroups.com
We will definitely be rolling this particular aspect of CLJ-1330 back in the next release.

Alex

Max Penet

unread,
Oct 29, 2014, 8:25:20 AM10/29/14
to cloju...@googlegroups.com, clo...@googlegroups.com
Just curious, why http://dev.clojure.org/jira/browse/CLJ-1517 didn't make it. It seemed complete at first glance.

Alex Miller

unread,
Oct 29, 2014, 9:49:38 AM10/29/14
to clo...@googlegroups.com, cloju...@googlegroups.com
It's a big change and will require a fair amount of screening and analysis, none of which has been done yet.

Michael Blume

unread,
Oct 31, 2014, 4:34:20 PM10/31/14
to clo...@googlegroups.com, cloju...@googlegroups.com
Picking up the variadic arguments discussion, it seems that in a simple definition like

(fn [& args] (apply f arg1 args))

One could conceivably put some sort of preprocessing smarts into the fn macro that notices that

a) this is a variadic arglist
b) the variadic arg (args) is a symbol, not some complicated destructuring expression
c) args is only ever used as the last argument to apply (maybe check this after thoroughly macroexpanding the function body)

and then autogenerates the various unrolled cases.

Even if we did only the simplest version of this, the new definition of partial would be

(defn partial
  "Takes a function f and fewer than the normal arguments to f, and
  returns a fn that takes a variable number of additional args. When
  called, the returned function calls f with args + additional args."
  {:added "1.0"
   :static true}
  ([f] f)
  ([f arg1]
   (fn [& args]
     (apply f arg1 args)))
  ([f arg1 arg2]
   (fn [& args]
     (apply f arg1 arg2 args)))
  ([f arg1 arg2 arg3]
   (fn [& args]
     (apply f arg1 arg2 arg3 args)))
  ([f arg1 arg2 arg3 & more]
   (fn [& args] (apply f arg1 arg2 arg3 (concat more args)))))

which is shorter (and would be unrolled in more cases than it currently is)

The thing that concerns me is that such a preprocessing step would be treating 'clojure.core/apply as a magic token, when apply is supposed to be a function like any other. And sufficient smarts to make 'partial single-clause would probably require treating 'clojure.core/concat as a magic token as well.

craig worrall

unread,
Nov 2, 2014, 4:29:18 PM11/2/14
to clo...@googlegroups.com, cloju...@googlegroups.com
I hit this error when moving to a new box that had an encrypted FS. Might be related to your case as well. Good luck.

Andy Fingerhut

unread,
Nov 2, 2014, 4:38:52 PM11/2/14
to clo...@googlegroups.com
As Alex Miller mentioned earlier in the discussion (it can be confusing with multiple subjects being discussed), the change that caused this behavior was due to ticket http://dev.clojure.org/jira/browse/CLJ-1330 which has been reopened, and they plan to do something to improve the situation in the next Clojure 1.7.0 alpha or beta release.

Andy

--
You received this message because you are subscribed to the Google

Luc Préfontaine

unread,
Nov 2, 2014, 9:13:07 PM11/2/14
to clo...@googlegroups.com
Encrypted fs have a file path limit around 140 bytes vs 255 w/o encryption.
I hit this problem a couple of months ago with a library and AOT.
The library had pretty long fn signatures, some generating class file names
around 110 bytes long.

The author tweaked these a bit to shorten the class file name. If I recall most of
these where anonymous fns.

With 140 (143 in my case) bytes you have to shorten the folder path as much as you can
if you expect compilation to succeed and avoid fns with complex signatures.

Luc P.
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
Luc Préfontaine<lprefo...@softaddicts.ca> sent by ibisMail!
Reply all
Reply to author
Forward
0 new messages