Java 8 interop in 2.11/2.12: SAMs

[Adriaan Moors]

Hi,

Wanted to summarize the current thinking on the roadmap to supporting Java 8 lambdas.

The JSR 335 spec seems to allow default methods in interfaces, so, for 2.12, we should plan to compile FunctionN to a "functional interface" (an interface with one abstract method, and default methods for the concrete ones). 

What are the steps we need to take in 2.11 to make this possible?

Also, what can we do in 2.11 (assuming Java 6) to allow our higher-order methods to be called from Java 8 using their closure syntax?

There are various proposals:

1. The most conservative approach: for each higher-order method (a method that takes a FunctionN argument), synthesize an overload that's a Java 8-style higher-order method. It boxes its argument of "functional interface type" to the Scala-style FunctionN. By introducing that functional interface as a superclass to FunctionN, we don't have to do anything for returning closures.
   
1. Disadvantages: the cost of boxing, additional overloads result in more bytecode, potential confusion (from Java; should be able to mark them synthetic so they are invisible to Scala)
2. Advantage: works on Java 6, and is source compatible, so that we can do it in 2.11
2. I like James Iry's suggestion of making FunctionN a value class that wraps a functional interface, but he also pointed out that that's not a source compatible change (FunctionN is not final). It would not require anything beyond Java 6, but would require breaking our compatibility policy (in a minor way) if we want to do this in 2.11.
   
3. Compile FunctionN to a real functional interface with default methods (as defined by JSR 335). I think this is the ideal outcome for Scala 2.12 on Java 8. We should deprecate as necessary in 2.11 to make this transition possible.

cheers,

adriaan

--

See you at Scala Days (June 10–12, NYC)!

[Rex Kerr]

On Wed, Mar 27, 2013 at 1:35 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

I like James Iry's suggestion of making FunctionN a value class that wraps a functional interface, but he also pointed out that that's not a source compatible change (FunctionN is not final). It would not require anything beyond Java 6, but would require breaking our compatibility policy (in a minor way) if we want to do this in 2.11.

Let's see what inherits from Function1, if it might become final?
 

Known Subclasses

::, <:<, =:=, AbstractFunction1, AbstractPartialFunction, Appended, Appended, ArrayBuffer, ArraySeq, ArrayStack, Atom, BasicTransformer, BitSet, BitSet, BitSet, BitSet1, BitSet2, BitSetLike, BitSetN, Buffer, BufferProxy, BufferWrapper, Comment, ConcurrentMap, Cons, Content, DefaultKeySet, DefaultKeySet, DefaultKeySet, DefaultKeySortedSet, DefaultKeySortedSet, DefaultMap, DefaultMap, DefaultMapModel, Document, DoubleLinkedList, DroppedWhile, DroppedWhile, DroppedWhile, Elem, ElementValidator, Empty, EmptyView, EmptyView, EntityRef, Exclusive, Filtered, Filtered, Filtered, FilteredKeys, FilteredKeys, FlatMapped, FlatMapped, Forced, Forced, GenSet, GenSetLike, Group, HashMap, HashMap, HashMap1, HashSet, HashSet, HashSet1, HashTrieMap, HashTrieSet, ImmutableDefaultKeySet, ImmutableMapAdaptor, ImmutableSetAdaptor, Impl, Impl, Inclusive, Inclusive, Index, IndexedSeq, IndexedSeq, IndexedSeq, IndexedSeqView, Indices, IntMap, LinearSeq, LinearSeq, LinearSeq, LinkedHashMap, LinkedHashSet, LinkedList, List, ListBuffer, ListMap, ListMap, ListSet, LongMap, Map, Map, Map, Map, Map1, Map2, Map3, Map4, MapLike, MapLike, MapLike, MapProxy, MapProxy, MapProxy, MapProxyLike, Mapped, Mapped, MappedValues, MappedValues, MultiMap, MurmurHash, MutableList, Nil, Node, Node, Node, NodeBuffer, NodeSeq, NumericRange, ObservableBuffer, ObservableMap, ObservableSet, OnceParser, OpenHashMap, PCData, PackratParser, PagedSeq, ParHashSet, ParHashSet, ParSet, ParSet, ParSet, ParSetLike, ParSetLike, Parser, PartialFunction, Patched, Patched, PicklerEnv, Prepended, Prepended, ProcInstr, Queue, Queue, QueueProxy, Range, Reactions, RefBuffer, ResizableArray, Reversed, Reversed, Reversed, RewriteRule, RuleTransformer, Script, SelectionSet, Seq, Seq, Seq, SeqForwarder, SeqProxy, SeqView, SeqViewLike, Set, Set, Set, Set1, Set2, Set3, Set4, SetLike, SetLike, SetProxy, SetProxy, SetProxy, SetProxyLike, Sliced, Sliced, Sliced, SortedMap, SortedMap, SortedMapLike, SortedSet, SortedSet, SortedSet, SortedSetLike, SpecialNode, Stack, Stack, StackProxy, Stream, StreamView, StreamViewLike, StringBuilder, SynchronizedBuffer, SynchronizedMap, SynchronizedQueue, SynchronizedSet, SynchronizedStack, SystemProperties, TakenWhile, TakenWhile, TakenWhile, Text, Transformed, Transformed, Transformed, TreeMap, TreeSet, TreeSet, TrieMap, UnPicklerEnv, Unparsed, UnrolledBuffer, ValueSet, Vector, WeakHashMap, WithDefault, WithDefault, WithDefault, WrappedArray, WrappedString, Wrapper, Zipped, Zipped, ZippedAll, ZippedAll, columns, indices, items, ofBoolean, ofByte, ofChar, ofDouble, ofFloat, ofInt, ofLong, ofRef, ofShort, ofUnit, pages, rows

I think there's a part of the proposal that I'm missing?

  --Rex

[Adriaan Moors]

First, "in a minor way" were my words.

Second, I do think it's minor as we own these classes and can refactor them. Say, they could still inherit the functional interface and an implementation trait for the missing methods.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[James Iry]

Function1 couldn't be final, it would be a SAM interface.

What would change is the location of compose and andThe - they would be moved to an implicit conversion (using a value class to avoid boxing)

The source incompatibility would break anything overriding compose or andThen.

[√iktor Ҡlang]

On Wed, Mar 27, 2013 at 7:10 PM, James Iry <jame...@typesafe.com> wrote:

Function1 couldn't be final, it would be a SAM interface.

What would change is the location of compose and andThe - they would be moved to an implicit conversion (using a value class to avoid boxing)

The source incompatibility would break anything overriding compose or andThen.

Specialization?

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Rex Kerr]

Sorry, I see I was being too oblique.

My point is that if the standard library has dozens of classes that need to be fixed, it's quite likely that others' code will also have many classes that need to be fixed, which is not likely to be interpreted as "minor" unless there is literally nothing to do in most cases (though the bytecode would be different, of course).

So it's very difficult to evaluate this option without having a bit better of an idea of the migration strategy and how much work it requires by hand.  Since 2.11 is going to be source incompatible anyway, it's really the how-much-work-by-hand metric that is key in evaluating these options, and I think I'm missing a key part of what is envisioned to be necessary.

  --Rex

[Paul Phillips]

On Wed, Mar 27, 2013 at 11:10 AM, James Iry <jame...@typesafe.com> wrote:

What would change is the location of compose and andThe - they would be moved to an implicit conversion (using a value class to avoid boxing)

The source incompatibility would break anything overriding compose or andThen.

In case anyone has forgotten, I already did this (and reverted it.)

 commit 1c3f66b6f2

 Author: Paul Phillips <pa...@improving.org>

 Date:   11 months ago

 

     Revert "Moved ancillary methods off specialized traits."

     

     This reverts commit 1d0372f84f9a7325a47beb55169cc454895ef74b.

     

     I forgot about polymorphic dispatch.  Have to seek another way.

 

 commit 1d0372f84f

 Author: Paul Phillips <pa...@improving.org>

 Date:   11 months ago

 

     Moved ancillary methods off specialized traits.

     

     Moved compose/andThen off Function1, curried/tupled off Function2.

     Pushed Tuple2 zipped into auxiliary class.  Created implicits to

     plaster over the changes.  This drops several hundred classfiles and

     takes (unoptimized) scala-library.jar from 7.8 Mb to 7.4 Mb.

[Rex Kerr]

Okay, that sounds like it's in the tolerable realm--overriding those is problematic anyway due to it breaking associativity.  So #2 sounds good to me, assuming other issues can be worked around (e.g. what Paul ran into).  Thanks for clarifying.

  --Rex

[Adriaan Moors]

Specialization makes everything a bit more complicated/convoluted (as usual), but I don't see any truly new challenges arise from it.

In terms of the first proposal, specialization simply produces more methods to overload with a Java variant.

Am I missing something?

To clarify, proposal #1 would generate an overloaded method that replaces occurrences of FunctionN[T1, ..., TN] with FunNSAM<T1,...., TN> and boxes the FunNSAM (provisional name ) as a FunctionN. As FunctionN is a subtype of FunNSAM in the proposal, no changes necessary for returning functions from combinators.

[Rex Kerr]

On Wed, Mar 27, 2013 at 1:35 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Hi,

Wanted to summarize the current thinking on the roadmap to supporting Java 8 lambdas.

The JSR 335 spec seems to allow default methods in interfaces, so, for 2.12, we should plan to compile FunctionN to a "functional interface" (an interface with one abstract method, and default methods for the concrete ones). 

What about the other way around?  If any interface that meets the appropriate criteria is a functional interface to Java, does it follow that
  trait Single { def foo(b: Bar): Baz }
is in fact a functional interface, and therefore if we see
  def bippy(s: Single) = s.foo(myBar)
we can call it with
  bippy(_.baz)
where .baz is a method on Bar that returns a Baz?  What about if we already have a Bar => Baz; is there any way to pass it to bippy (maybe with some compiler help, possibly by allowing _ to repackage as necessary into the correct type of interface)?

If Scala 2.12 (or whatever) does *not* allow this, how is it going to interoperate safely with Java which *does*?  On the other hand, if it *does* allow this, what do we do about the awkward tension between SAMs and function types?  (Most notably that not everything that looks like x => x.y() and obeys X => Y is a subclass of Function1[X,Y]?)

Upon reflection, without knowing how we intend to harmonize or at least not fall flat on our faces in this regard, I don't know enough to make an educated guess about what we should do in 2.11 (or 2.12).  When in doubt, I tend to opt for the most conservative approach: option zero.

Option zero would be to do _nothing with Scala at all_ but add twenty-two generator methods written in Java that take a SAM and produce a Scala function.  If you use Java, it's your job to wrap your higher-order method call in that generator.  The java wrapper could be packaged in scala-interop.jar or somesuch and shipped with the distribution.

  --Rex
 

What are the steps we need to take in 2.11 to make this possible?

Also, what can we do in 2.11 (assuming Java 6) to allow our higher-order methods to be called from Java 8 using their closure syntax?

There are various proposals:

1. The most conservative approach: for each higher-order method (a method that takes a FunctionN argument), synthesize an overload that's a Java 8-style higher-order method. It boxes its argument of "functional interface type" to the Scala-style FunctionN. By introducing that functional interface as a superclass to FunctionN, we don't have to do anything for returning closures.
   
1. Disadvantages: the cost of boxing, additional overloads result in more bytecode, potential confusion (from Java; should be able to mark them synthetic so they are invisible to Scala)
2. Advantage: works on Java 6, and is source compatible, so that we can do it in 2.11
2. I like James Iry's suggestion of making FunctionN a value class that wraps a functional interface, but he also pointed out that that's not a source compatible change (FunctionN is not final). It would not require anything beyond Java 6, but would require breaking our compatibility policy (in a minor way) if we want to do this in 2.11.
   
3. Compile FunctionN to a real functional interface with default methods (as defined by JSR 335). I think this is the ideal outcome for Scala 2.12 on Java 8. We should deprecate as necessary in 2.11 to make this transition possible.

cheers,

adriaan

--

See you at Scala Days (June 10–12, NYC)!

--

[√iktor Ҡlang]

On Wed, Mar 27, 2013 at 9:17 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Specialization makes everything a bit more complicated/convoluted (as usual), but I don't see any truly new challenges arise from it.

In terms of the first proposal, specialization simply produces more methods to overload with a Java variant.

Am I missing something?

Ah, so the FI will only have the apply?

trait ƒ[-T, +R] {

 def apply(param: T): R

}

But how will you go from ƒ[T,R] => (T => R) while retaining specialization (i.e. avoiding boxing)?

Am I missing something?

Cheers,

√

--

[Jason Zaugg]

On Wed, Mar 27, 2013 at 10:37 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Wed, Mar 27, 2013 at 9:17 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Specialization makes everything a bit more complicated/convoluted (as usual), but I don't see any truly new challenges arise from it.

In terms of the first proposal, specialization simply produces more methods to overload with a Java variant.

Am I missing something?

Ah, so the FI will only have the apply?

trait ƒ[-T, +R] {

 def apply(param: T): R

}

But how will you go from ƒ[T,R] => (T => R) while retaining specialization (i.e. avoiding boxing)?

Am I missing something?

I was imagining a setup where we would have two interfaces generated for a specialized trait:

  trait Function1[@spec A, @spec B] { def apply(a: A) : B }

  // interface Function1<A, B> { B apply(A a)  }

  // interface Function1$sp<A, B> { B apply(A a); int apply(int a); ....  }

Upside: Function1 can be used as a Java SAM (of course we'll still need to deal with compose/andThen)

Downside: Specialized clients need to type case on a provided Function1 to be able to call specialized methods.

(I think that) Vlad told me that an earlier incarnation of specialization used this encoding, but it was more trouble that it was worth. But maybe now it is worth more?

I haven't thought this through much, so don't take this as a concrete proposal, but rather as an idea we can draw on as we find the right design.

-jason

 

[Adriaan Moors]

Ah, so the FI will only have the apply?

yes, we're restricted to Java 6 (at least for 2.11) so can't use default methods, thus a SAM is an interface with exactly one (abstract) method (modulo override-matching, so that inherited methods with a matching signature do not contribute new abstract methods -- see the JSR 335 spec) 

trait ƒ[-T, +R] {

 def apply(param: T): R

}

But how will you go from ƒ[T,R] => (T => R) while retaining specialization (i.e. avoiding boxing)?

You'd have to box the actual closure when the Scala code is expecting a Function[T,R] whereas you received a FunSAM[T,R], but the arguments could be specialized as explained by Jason.

When your higher-order method takes an Int => Int, the Java stub's argument type would be either

  - FunSAM$spII  where interface FunSAM$spII { int apply(int x) }

  - if overloading is allowed in SAMs, exactly what Jason said

 

[Grzegorz Kossakowski]

On 27 March 2013 14:01, Rex Kerr <ich...@gmail.com> wrote:

Option zero would be to do _nothing with Scala at all_ but add twenty-two generator methods written in Java that take a SAM and produce a Scala function.  If you use Java, it's your job to wrap your higher-order method call in that generator.  The java wrapper could be packaged in scala-interop.jar or somesuch and shipped with the distribution.

I like this idea for it's simplicity.

Rex, just to make sure I understand your proposal. We would have

interface <T,U> Function1SAM {

  public U apply(x T)

}

and then:

class ScalaFunctionConverters {

  static <T,U> Function1<T,U> asScala(Function1SAM<T,U> f) {

    return new AbstractFunction1<T,U>() {
      public U apply(T x) {
        return f.apply(x);
      }
    };
  }

}

Then, from Java you would use it this way:

scala.collection.immutable.List<Int> xs = ...;

xs.map(asScala((Integer x) -> x+1)));

Is this correct?

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Simon Ochsenreither]

Upside: Function1 can be used as a Java SAM (of course we'll still need to deal with compose/andThen)

Just as a general note regarding compose: Keep in mind that Java's compose is not equivalent to rest of the world's definition of compose.

public default <V > Function<T, V> compose(Function<? super R, ? extends V> after) {

    return (T t) -> after.apply(apply(t));

}

def compose[A](g: A => T1): A => R = { x => apply(g(x)) }

[Rex Kerr]

Precisely.

[Simon Ochsenreither]

Wow, seems as if it has been fixed minutes ago: http://mail.openjdk.java.net/pipermail/lambda-dev/2013-March/008901.html

[Jason Zaugg]

On Thu, Mar 28, 2013 at 12:18 AM, Grzegorz Kossakowski <grzegorz.k...@gmail.com> wrote:

Then, from Java you would use it this way:

scala.collection.immutable.List<Int> xs = ...;

xs.map(asScala((Integer x) -> x+1)));

It's worth mentioning that the CanBuildFrom's are a much bigger barrier to calling our collection HOF's from Java than this little boilerplate. Of course there are a many other places that don't suffer from the same signature complexity and would be nice to be Java-callable.

I do like this idea for Scala 2.11.

-jason

[Adriaan Moors]

Mulling it over for a couple of days, I agree this is the way to go. Great suggestion, Rex!

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Grzegorz Kossakowski]

Ah, yes. I forgot about CanBuildFrom but I agree that many other HOFs (like in Future API) would benefit from this.

[Adriaan Moors]

Another concern is that Java's type inference will have to work harder because argument lists will have been flattened by the time they see the method signature. As you know, we rely on curried argument lists to "chunk" the challenges we present to the inferencer.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Simon Ochsenreither]

Another concern is that Java's type inference will have to work harder ...

Sadly, it looks like they decided to make the users do the work instead:

Map<String, List<String>> map = new LinkedHashMap<>();

map.put("UK", asList("Bermingham","Bradford","Liverpool"));

map.put("USA", asList("NYC","New Jersey","Boston","Buffalo"));

 

FlatMapper<Entry<String, List<String>>,String> flattener;

flattener = (entry,consumer) -> { entry.getValue().forEach(consumer); };

 

List<String> cities = map.entrySet()

             .stream()

             .flatMap( flattener )

             .collect(toList());

Ideally the lines 5, 6 should be defined as inline arguments of flatMap, but the compiler cannot properly infer the types of the expression precisely for the overloading problems that I described above. Given the amount of type information that must be provided, I felt that it was best to define it in another line as I did in #5 and #6. It looks awful,  I know. Clearly, we are in desperate need of an API that offers operations to deal with maps or tuples.

http://blog.informatech.cr/2013/03/24/java-streams-preview-vs-net-linq/ (Challenge 4).