[ANN] Clojure 1.7.0-RC1 now available

[Alex Miller]

Clojure 1.7.0-RC1 is now available.

Try it via

- Download: https://repo1.maven.org/maven2/org/clojure/clojure/1.7.0-RC1/

- Leiningen: [org.clojure/clojure "1.7.0-RC1"]

The only change since 1.7.0-beta3 is CLJ-1706, which makes reader conditional splicing an error at the top level (previously it would silently drop all but the first spliced element).

For a full list of changes since 1.6.0, see:

https://github.com/clojure/clojure/blob/master/changes.md

Please give it a try and let us know if things are working (or not). The more and quicker feedback we get, the sooner we can release 1.7.0 final!

- Alex

[Andy Fingerhut]

Alex, doing a quick sweep of new public Vars in 1.7.0-RC1 that were not in Clojure 1.6.0 (complete list of all 21 below), I noticed that the following two do not have any doc strings.  Intentional, or oversight?

*suppress-read*

Throwable->map

Andy

*suppress-read*

->Eduction

Throwable->map

cat

completing

dedupe

eduction

ensure-reduced

random-sample

reader-conditional

reader-conditional?

run!

tagged-literal

tagged-literal?

transduce

unreduced

update

volatile!

volatile?

vreset!

vswap!

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to the Google Groups "ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojurescrip...@googlegroups.com.
To post to this group, send email to clojur...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojurescript.

[Jason Wolfe]

We haven't shipped it to production yet, but I just verified that our full test suite at Prismatic passes on RC1 after fixing a few tests that were erroneously depending on hash ordering.   Thanks everyone for all the hard work on this release!  

[Jason Wolfe]

Just deployed to our staging environment and ran into an unexpected stack overflow error.  Here's a reduced test case:

 

user> (->> (range 10000) (mapcat (fn [_] (java.util.ArrayList. (range 10)))) (reduce (constantly nil)))

java.lang.StackOverflowError: 

           RT.java:509 clojure.lang.RT.seq

          core.clj:135 clojure.core/seq

          core.clj:698 clojure.core/concat[fn]

       LazySeq.java:40 clojure.lang.LazySeq.sval

       LazySeq.java:49 clojure.lang.LazySeq.seq

   ChunkedCons.java:59 clojure.lang.ChunkedCons.chunkedNext

          core.clj:671 clojure.core/chunk-next

     protocols.clj:119 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:152 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:122 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:152 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:122 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:152 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:122 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:152 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

      protocols.clj:31 clojure.core.protocols/seq-reduce

      protocols.clj:75 clojure.core.protocols/fn

      protocols.clj:13 clojure.core.protocols/fn[fn]

     protocols.clj:122 clojure.core.protocols/fn

      protocols.clj:19 clojure.core.protocols/fn[fn]

....

On Thursday, May 21, 2015 at 9:31:08 AM UTC-7, Alex Miller wrote:

[Ghadi Shayban]

This commit [1] for CLJ-1589 changed dispatch for InternalReduce, causing a bug similar to CLJ-1237 [2].

[1] https://github.com/clojure/clojure/commit/1242c4878f3d4ef30535a22f6c74144637504fbe
[2] http://dev.clojure.org/jira/browse/CLJ-1237

[Daniel Compton]

One of the most significant features of 1.7 are Reader Conditionals. I'm pretty confident after all the discussion that has gone on that we have a good design. However I haven't seen many or any libraries which have gone through the porting process to use Reader Conditionals.

I've worked on porting a few libraries and everything has gone mostly smoothly. However I'd feel more confident that we've got the right design if we had more people trying to port their libraries/projects to cljc and reporting their experiences.

Is this a reasonable concern, or am I missing something and this isn't necessary?

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

[Jason Wolfe]

Thanks for the context.  

It looks like this is actually the same as CLJ-1237, the scope of the issue is just broader now with that commit.  I left a comment there rather than creating a new ticket.  

[Colin Taylor]

We're seeing breakage because (I believe) of http://dev.clojure.org/jira/browse/CLJ-1499. Specifically ad-hoc maps not implementing .iterator() and post 1499 cant rely on SeqIterator.
If I've got this right, I'm surprised more people aren't seeing this?

Obviously not a bug in this release per se but perhaps something to look out for / publicize..

Some examples:

http://dev.clojure.org/jira/browse/CCACHE-41 - https://github.com/clojure/core.cache/blob/master/src/main/clojure/clojure/core/cache.clj#L57
clj-http - https://github.com/dakrone/clj-http/blob/master/src/clj_http/headers.clj#L105

Simple test-case:

user=> (use 'clojure.core.cache)
nil
user=> (def C (fifo-cache-factory {:a 1, :b 2}))
#'user/C
user=> (.iterator (keys C))

AbstractMethodError clojure.core.cache.FIFOCache.iterator()Ljava/util/Iterator; clojure.lang.APersistentMap$KeySeq.iterator (APersistentMap.java:184)

[Alex Miller]

On Tuesday, May 26, 2015 at 12:26:48 AM UTC-5, Colin Taylor wrote:

We're seeing breakage because (I believe) of http://dev.clojure.org/jira/browse/CLJ-1499. Specifically ad-hoc maps not implementing .iterator() and post 1499 cant rely on SeqIterator.

I think "ad-hoc maps" needs way more clarification here - see below.

 

If I've got this right, I'm surprised more people aren't seeing this?

Obviously not a bug in this release per se but perhaps something to look out for / publicize..

Some examples:

http://dev.clojure.org/jira/browse/CCACHE-41 - https://github.com/clojure/core.cache/blob/master/src/main/clojure/clojure/core/cache.clj#L57
clj-http - https://github.com/dakrone/clj-http/blob/master/src/clj_http/headers.clj#L105

Simple test-case:

user=> (use 'clojure.core.cache)
nil
user=> (def C (fifo-cache-factory {:a 1, :b 2}))
#'user/C
user=> (.iterator (keys C))

AbstractMethodError clojure.core.cache.FIFOCache.iterator()Ljava/util/Iterator;  clojure.lang.APersistentMap$KeySeq.iterator (APersistentMap.java:184)

This is actually not CLJ-1499 (which just changed the iterators on existing concrete map types) but is actually more related to CLJ-1602 which was a change to `keys`. 

fifo-cache-factory will construct an instance of a class via defcache via deftype that implements ILookup, IPersistentMap, Associative, Counted, IPersistentCollection, and Seqable. keys routes to RT.keys(), which switches behavior based on whether the map implements IPersistentMap. If not (and this was formerly always the case), it would create an iterator from a seq of the map.

If *not* (new behavior), then an iterator from the map will be used instead, not a seq-based iterator. If the map is *persistent*, then we expect the data in it to be immutable and thus it is safe to use a direct iterator (without needing the caching capability of a seq). Because of CLJ-1499, there are now direct iterators on instances like PHM, PAM, and records that can directly walk the data of the data structure without creating (needless) seq node allocations.

So I think perhaps parsing all this out, the constraint being leveraged here is that IPersistentMap extends Iterable and assumes it can call it. From a Java-based inheritance perspective, that was always true. 

Prior to 1602, this fact was never leveraged but it now is. So I think there is an argument that the "maps" in both of these cases implemented incomplete IPersistentMap contracts. That said, I'm sympathetic to not breaking stuff so happy to discuss more.

I am on vacation this week (despite appearances) so I will just be checking in every day or two in case this spirals into discussion.

Alex

 

 

[Marshall Bockrath-Vandegrift]

Some of my code is broken by commit c47e1bbcfa227723df28d1c9e0a6df2bcb0fecc1, which landed in 1.7.0-alpha6 (I lasted tested with -alpha5 and have been unfortunately busy since).  The culprit is the switch to producing seqs over iterators as chunked iterators.  This would appear to break seq-based traversal of any iterator implementing the not-uncommon Java pattern of mutating and re-yielding the same object on each `next()` invocation.

I'm unable to find an existing ticket for this apparent-regression.  Should I create one, or did I miss the existing ticket, or is there some mitigating issue which makes this a non-problem?

Thanks.

-Marshall

--

[Alex Miller]

In what way is it broken? Both before and after wrapped a mutable iterator into a caching seq. The new one is different in that it chunks so reads 32 at a time instead of 1. However combining either with other chunking sequence operations would have the same effect which is to say that using that mutable iterator with anything else, or having expectations about its rate of consumption was as dubious before as it is now.

Unless of course I misunderstand your intent, which possible because I am on a phone without easy access to look further at the commit and am going by memory.

[Marshall Bockrath-Vandegrift]

The difference is that the original behavior allowed room to transform the mutated object into an object which *could* be safely cached in a "downstream" seq, while the new behavior pumps the iterator through 32 mutations before user-level code has a chance to see it.  Contrived example using the Java standard libary:

Clojure 1.6.0:

(->> (map vector (java.util.EnumSet/allOf java.util.concurrent.TimeUnit) (range)) (into {}) (java.util.EnumMap.) (.entrySet) (map str) (into []))

#=> ["NANOSECONDS=0" "MICROSECONDS=1" "MILLISECONDS=2" "SECONDS=3" "MINUTES=4" "HOURS=5" "DAYS=6"]

Clojure 1.7.0-RC1:

(->> (map vector (java.util.EnumSet/allOf java.util.concurrent.TimeUnit) (range)) (into {}) (java.util.EnumMap.) (.entrySet) (map str) (into []))

#=> ["DAYS=6" "DAYS=6" "DAYS=6" "DAYS=6" "DAYS=6" "DAYS=6" "DAYS=6"]

IMHO the latter behavior demonstrates a mismatch where chunked seqs and iterators are simple incompatible.

[Sean Corfield]

I don’t see that behavior on RC1:

user> (clojure-version)
"1.7.0-RC1"
user> (->> (map vector (java.util.EnumSet/allOf java.util.concurrent.TimeUnit) (range)) (into {}) (java.util.EnumMap.) (.entrySet) (map str) (into []))

["NANOSECONDS=0" "MICROSECONDS=1" "MILLISECONDS=2" "SECONDS=3" "MINUTES=4" "HOURS=5" "DAYS=6"]

Sean

[Alex Miller]

That's not what I see with 1.7.0-RC1 (or any of the betas). I tried with both Java 1.7.0_25 and 1.8.0-b132.

user=> *clojure-version*

{:major 1, :minor 7, :incremental 0, :qualifier "RC1"}

user=> (->> (map vector (java.util.EnumSet/allOf java.util.concurrent.TimeUnit) (range)) (into {}) (java.util.EnumMap.) (.entrySet) (map str) (into []))

["NANOSECONDS=0" "MICROSECONDS=1" "MILLISECONDS=2" "SECONDS=3" "MINUTES=4" "HOURS=5" "DAYS=6"]

Re "implementing the not-uncommon Java pattern of mutating and re-yielding the same object on each `next()` invocation". I'm assuming that you're somehow expecting to traverse one seq node, then having an opportunity to mutate something (the source, the iterator, the return object) in between each new advancement of the seq node? That seems a) not common at all, b) a bad idea even in Java and c) dangerous even before this change. In either case you end up with a seq that points to a succession of the same repeated (mutable and mutating) object - this violates most expectations we as Clojure users have of sequences. Any sort of chunking (map, filter, etc) over the top of that seq would force realization up to 32 elements beyond the head causing the same issue. 

The original one-at-a-time IteratorSeq is still there (for now) and you can still make one if you want via (clojure.lang.IteratorSeq/create iter) but I would consider it deprecated. I think a custom lazy-seq or a loop-recur would be a better way to handle this case, which in my opinion is highly unusual. That said, my ears are open if this is an issue for a large number of people.

-Marshall

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev+unsubscribe@googlegroups.com.

[Julien]

I just ported an app with significant CLJX usage (55 files) to reader conditionals and it works perfectly!

It looks like there is a small issue related to map literal containing comments but I am not sure if it has been reported yet.

Julien

[Marshall Bockrath-Vandegrift]

Ugh -- looks like the iterator value re-use behavior for EnumMap entrySet was "fixed" post Java 1.6 (my example was under Java 1.6, which I believe is still a Clojure-supported version of Java?).  I can throw together a synthetic example, but I think the people following this thread get what's happening.  The point isn't whether this pattern is a "good idea" or not (it certainly isn't) but whether existing Java APIs people want to interop with use it (they certainly do).

I presently depend on no less than 3 separate Java library APIs I currently know for a fact depend on this behavior:

- Hadoop ReduceContextImpl$ValueIterator

- Mahout DenseVector$AllIterator/NonDefaultIterator

- LensKit FastIterators

It is an option to explicitly construct `IteratorSeq` instances (I actually had verified that approach this afternoon for the Hadoop API in Parkour), but I'm not happy about it. That approach places a definite burden on integration library maintainers to implement the change, and on end-users to realize they need to upgrade for Clojure 1.7 compatibility. The `Iterator` interface just fundamentally in no way guarantees that the `next()` yielded values are functional-safe in the sense necessary to support chunking. I understand the desire to increase performance, but I don't think it's worth the potential silent and bewildering breakage in interop.

-Marshall

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

To post to this group, send email to cloju...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojure-dev.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.

[Marshall Bockrath-Vandegrift]

And to further explicate "bewildering" -- I mean that I only figured out what was happening because I was explicitly testing a library against 1.7.0-RC1 and even then had to `git bisect` *Clojure* to find the offending commit.  Otherwise the resulting behavior is just that values generated via the deep guts of a complex Java API suddenly became nonsensical.

[Alex Miller]

No, please report if so.

[Alex Miller]

On Tuesday, May 26, 2015 at 8:45:25 PM UTC-5, Marshall Bockrath-Vandegrift wrote:

Ugh -- looks like the iterator value re-use behavior for EnumMap entrySet was "fixed" post Java 1.6 (my example was under Java 1.6, which I believe is still a Clojure-supported version of Java?). 

Java 1.6 is still supported in Clojure 1.7 (but will likely be removed in the next release or two).

 

I can throw together a synthetic example, but I think the people following this thread get what's happening.  The point isn't whether this pattern is a "good idea" or not (it certainly isn't) but whether existing Java APIs people want to interop with use it (they certainly do).

The point I was getting at is really whether you should consider this to be broken with the old behavior too. 

 

I presently depend on no less than 3 separate Java library APIs I currently know for a fact depend on this behavior:

- Hadoop ReduceContextImpl$ValueIterator

- Mahout DenseVector$AllIterator/NonDefaultIterator

- LensKit FastIterators

Can you point to code for "the original behavior allowed room to transform the mutated object into an object which *could* be safely cached in a 'downstream' seq"? By what means does this transformation occur? It sounds to me like you are starting with an Iterator, creating a seq, then walking the seq exactly once, one element at a time, and producing a new transformed seq or other output. 

If you did reuse that IteratorSeq, all of the elements of the sequence would point to the same object which would be in the "last" state like the java 1.6 example you gave. Thus, the "caching" capability of the seq can't possibly be something you're using. And if that's true, then why are you paying the allocation and synchronization costs of making the seq at all? Why not just use the iterator directly, thus skipping all the extra allocation that these object-reusing high-performance iterators are working so hard to avoid in the first place? In 1.7, transducers would give you exactly the capability to walk the source iterator, apply a transducer version of your transformation, and output to a collection (via into), a value (via transduce), or a lazy sequence (via sequence). I think you would find this faster as well due to reduced allocation (possibly greatly reduced depending on the transformation).

 

It is an option to explicitly construct `IteratorSeq` instances (I actually had verified that approach this afternoon for the Hadoop API in Parkour), but I'm not happy about it. That approach places a definite burden on integration library maintainers to implement the change, and on end-users to realize they need to upgrade for Clojure 1.7 compatibility. The `Iterator` interface just fundamentally in no way guarantees that the `next()` yielded values are functional-safe in the sense necessary to support chunking. I understand the desire to increase performance, but I don't think it's worth the potential silent and bewildering breakage in interop.

It would be possible to restore the non-chunking IteratorSeq behavior but retain it for eduction and the other places where we first made the swap. I'm not quite ready yet to concede that that's necessary. It seems like the only constraints under which this usage makes sense is exactly the case where it's unnecessary and even harmful.

[Julien]

My bad I was using an older beta. With latest RC1 the issue goes away.

Julien

[Marshall Bockrath-Vandegrift]

On Tue, May 26, 2015 at 11:29 PM Alex Miller <al...@puredanger.com> wrote:

 

The point I was getting at is really whether you should consider this to be broken with the old behavior too.  

Such APIs are tricky to use correctly from Clojure via seqs, but it is possible to do so with the "normal" automatic clojure.lang.RT seq adapters in Clojure <=1.6. My point is that existing Clojure code can and does depend on non-chunked realization of iterator seqs, and that for such code this is a breaking change.

 

Can you point to code for "the original behavior allowed room to transform the mutated object into an object which *could* be safely cached in a 'downstream' seq"? By what means does this transformation occur? It sounds to me like you are starting with an Iterator, creating a seq, then walking the seq exactly once, one element at a time, and producing a new transformed seq or other output. 

Exactly -- the unfortunately Java <=1.6-only snippet I posted earlier is just such an example of this.

 

If you did reuse that IteratorSeq, all of the elements of the sequence would point to the same object which would be in the "last" state like the java 1.6 example you gave. Thus, the "caching" capability of the seq can't possibly be something you're using. And if that's true, then why are you paying the allocation and synchronization costs of making the seq at all? Why not just use the iterator directly, thus skipping all the extra allocation that these object-reusing high-performance iterators are working so hard to avoid in the first place? In 1.7, transducers would give you exactly the capability to walk the source iterator, apply a transducer version of your transformation, and output to a collection (via into), a value (via transduce), or a lazy sequence (via sequence). I think you would find this faster as well due to reduced allocation (possibly greatly reduced depending on the transformation).

I've personally used reducers wherever possible since they were introduced, and for the Hadoop case Parkour's primary recommended API is in terms of reducers [1]. For new code, the transducer-based facilities in Clojure 1.7 will certainly provide more options for functional-safe handling of the iterators at question.  But to repeat my main point, those don't help with existing code which depends on the one-at-a-time realization semantics of Iterators being reflected in one-at-a-time realization in iterator-backed seqs.

[1] https://github.com/damballa/parkour/blob/master/doc/reducers-vs-seqs.md

Thanks,

-Marshall

[Alex Miller]

I've logged http://dev.clojure.org/jira/browse/CLJ-1738 for this and I'll discuss with Rich.

[Tassilo Horn]

Hi Alex,

I've spotted another bug which I've already reported
http://dev.clojure.org/jira/browse/CLJ-1739. The problem is that sets
of different kinds are equal which is of course expected, e.g.,

(= #{1 2 3} (flatland.ordered.set/ordered-set 1 2 3)) ;=> true
(= #{1 2 3} (java.util.HashSet. [1 2 3])) ;=> true

but sets of equal sets aren't equal anymore, e.g.,

(= #{#{1 2 3}} #{(flatland.ordered.set/ordered-set 1 2 3)}) ;=> false
(= #{#{1 2 3}} #{(java.util.HashSet. [1 2 3])}) ;=> false

That's no recent regression in 1.7.0-RC1, though. I can reproduce the
bug also with 1.6.0.

Bye,
Tassilo

[Andy Fingerhut]

Tassilo, the issue with flatland.ordered.set/ordered-set was that library had never been updated to match the change in hash function that occurred when Clojure 1.6.0 was released.

Alan Malloy released a new version of the ordered library today that fixes that.

That has no effect on the other CLJ tickets.  It is only a fix for the ordered library.

Andy

[Tassilo Horn]

Andy Fingerhut <andy.fi...@gmail.com> writes:

Hi Andy,

> Tassilo, the issue with flatland.ordered.set/ordered-set was that
> library had never been updated to match the change in hash function
> that occurred when Clojure 1.6.0 was released.
>
> Alan Malloy released a new version of the ordered library today that
> fixes that.

Indeed, I can confirm that it works as expected with version 1.5.3.

> That has no effect on the other CLJ tickets. It is only a fix for the
> ordered library.

Yes, of course.

Bye,
Tassilo

[Sean Corfield]

We took RC1 to production yesterday afternoon (after being on beta1 since mid-April). So far so good.

I think the only 1.7 new feature we’re using so far is transducers (we wrote a transducer that takes a result set of user-to-user messages and returns a paginated, nested result set organized by conversion thread — a big simplification over how we’d done it before).

Sean