Julia 0.5 Highlights

[Stefan Karpinski]

Since the 0.5 release affects everyone here, I wrote a longish blog post about what the major changes are: http://julialang.org/blog/2016/10/julia-0.5-highlights.

One other change that I left out of the post because it was getting pretty long and it seems a bit esoteric is that array comprehensions are now type-inference-independent. That means that the type of the resulting array only depends on the actual types of values produced, not what the compiler can prove about the expression in advance. In particular, this means that comprehensions behave the same way in global scope as in local scope now, which is a fairly major relief to anyone who's struggled with that.

[Jared Crean]

Very nice summary, thanks for posting.  One question I had was what should the signature of a function be to receive a generator?  For example, if the only method of extrema is extrema(A::AbstractArray), is that too restrictive?

  Jared Crean

[Jussi Piitulainen]

Does that mean that an empty array comprehension is always Array{Any}?

[Christoph Ortner]

f(n) = [ i^2 for i = 1:n ]

julia> f(0)

0-element Array{Int64,1}

[Christoph Ortner]

However, 

g(n) = sum( i^2 for i = 1:n )

julia> g(0)

ERROR: MethodError: no method matching zero(::Type{Any})

Closest candidates are:

  zero(::Type{Base.LibGit2.Oid}) at libgit2/oid.jl:88

  zero(::Type{Base.Pkg.Resolve.VersionWeights.VWPreBuildItem}) at pkg/resolve/versionweight.jl:80

  zero(::Type{Base.Pkg.Resolve.VersionWeights.VWPreBuild}) at pkg/resolve/versionweight.jl:120

  ...

 in mr_empty(::Base.#identity, ::Base.#+, ::Type{T}) at ./reduce.jl:130

 in mr_empty(::Base.#identity, ::Base.#+, ::Type{T}) at /Users/ortner/gits/julia/usr/lib/julia/sys.dylib:?

 in mapfoldl(::Base.#identity, ::Function, ::Base.Generator{UnitRange{Int64},##3#4}) at ./reduce.jl:60

 in g(::Int64) at ./REPL[17]:1

though this seems to have been fixed with JuliaLang/julia#18873 ????   (I haven't tested it yet)

[Evan Fields]

Thanks for writing this up; it's helpful to see certain things highlighted and explained in more detail than news.md gives!

[harven]

Le mercredi 12 octobre 2016 01:45:25 UTC+2, Jared Crean a écrit :

Very nice summary, thanks for posting.  One question I had was what should the signature of a function be to receive a generator?  For example, if the only method of extrema is extrema(A::AbstractArray), is that too restrictive?

  Jared Crean

Any functions working with iterables will work with generators.

julia> methods(extrema)
# 4 methods for generic function "extrema":
extrema(r::Range) at reduce.jl:345
extrema(x::Real) at reduce.jl:346
extrema(A::AbstractArray, dims) at reduce.jl:388
extrema(itr) at reduce.jl:362

The last line tells you that extrema will work. An object is iterable if it implements the methods start, next and done. There are in fact a few other objects that also work on generators.

 julia> methodswith(Base.Generator)
8-element Array{Method,1}:
 collect(itr::Base.Generator) at array.jl:298
 done(g::Base.Generator, s) at generator.jl:22
 indices(g::Base.Generator) at generator.jl:91
 length(g::Base.Generator) at generator.jl:89
 ndims(g::Base.Generator) at generator.jl:92 
 next(g::Base.Generator, s) at generator.jl:24
 size(g::Base.Generator) at generator.jl:90  
 start(g::Base.Generator) at generator.jl:21 

There are a few functions that work on arrays but not on iterables. You should not expect these to work on generators.

julia> show(reverse([1:10;]))
[10,9,8,7,6,5,4,3,2,1]
julia> show(reverse(i for i = 1:10))
ERROR: MethodError: no method matching reverse(::Base.Generator{UnitRange{Int64},##9#10})
Closest candidates are:
  reverse(!Matched::String) at strings/string.jl:209
  reverse(!Matched::BitArray{1}) at bitarray.jl:1416
  reverse(!Matched::Tuple) at tuple.jl:199
  ...

[Jared Crean]

  Perfect, thanks.

  Jared Crean

[Cedric St-Jean]

Very nice summary!

I assume that there's a mile-long issue discussing this somewhere, but why doesn't the return type also assert that convert returns a value of the correct type?

type A end

Base.convert(::Type{Int}, ::A) = "hey"

foo()::Int = A()

foo()  # returns "hey"

[Stefan Karpinski]

That's a fair point. It seems like it could/should be handled by the same (not-yet-implemented) mechanism that ensures that `convert(T,x)::T` is true. Of course, we could choose to enforce this fact via lowering in this case, independent of enforcing it for convert.

[Steven G. Johnson]

On Wednesday, October 12, 2016 at 9:26:54 PM UTC-4, Stefan Karpinski wrote:

That's a fair point. It seems like it could/should be handled by the same (not-yet-implemented) mechanism that ensures that `convert(T,x)::T` is true. Of course, we could choose to enforce this fact via lowering in this case, independent of enforcing it for convert.

I think we should add a typeassert in the lowering for this syntax.   I'm confused because Jeff's PR actually claimed it was using convert(T, val)::T --- see https://github.com/JuliaLang/julia/pull/16432

[Brian Rogoff]

Great summary, thanks so much!

Being a fan of typeful functional programming, I really like the return type annotations and FP performance improvements. Is there a way to describe a precise return type for a higher order function? The examples of Function I've seen have neither the arguments type/arity or return types.

[Steven G. Johnson]

On Wednesday, October 12, 2016 at 9:40:27 PM UTC-4, Steven G. Johnson wrote:

On Wednesday, October 12, 2016 at 9:26:54 PM UTC-4, Stefan Karpinski wrote:

That's a fair point. It seems like it could/should be handled by the same (not-yet-implemented) mechanism that ensures that `convert(T,x)::T` is true. Of course, we could choose to enforce this fact via lowering in this case, independent of enforcing it for convert.

Update: this was a bug that occurred for small, inlined functions.   Now fixed, and will be fixed in the next 0.5.x release: https://github.com/JuliaLang/julia/pull/18899 

[Stefan Karpinski]

On Thu, Oct 13, 2016 at 12:26 PM, Brian Rogoff <bro...@gmail.com> wrote:

Great summary, thanks so much!

Being a fan of typeful functional programming, I really like the return type annotations and FP performance improvements. Is there a way to describe a precise return type for a higher order function? The examples of Function I've seen have neither the arguments type/arity or return types.

No, Function doesn't have signatures, arity or return type as part of its type. The signature of a function is the union of its method signatures, which is potentially very complicated. Type parameters are not contravariant, so they can't be described without massively complicated Julia's (already complicated) type system. Worse still, adding any form of contravariance would almost certainly make important predicates like subtype and type intersection undecidable. There are still things that could be done to get some of the features that you probably want from function types, but dispatching on the return type is unlikely to ever be allowed. Two things that may happen are:

1. Constraining the type signature of a generic function, raising an error if any method returns something that doesn't match:

convert{T} :: (T, Any)-->T

or whatever syntax makes sense. This would implicitly mean that any call to convert(T,x) would be translated to convert(T,x)::T so that we know convert always returns the type one would expect for it. This is what I was alluding to above.

2. Intersecting a function signature on an argument with a generic function to extract a "sub-function" that will either behave the way we expect it to or raise an error:

function mysort!{T}(lt::(T,T)-->Bool, Vector{T})

    ...

end

This would mean that any use like lt(a, b) in the function body would implicitly be wrapped as lt(a::T, b::T)::Bool or something like that. This extra type information could potentially allow the compiler to reason better about the function's behavior even in cases where it otherwise can't figure out that much. Of course, in the case that's already fast, we don't need that information since the type of function calls can already be completely inferred.

Note that neither of these allow you to dispatch on the type of lt.

[Brian Rogoff]

On Thursday, October 13, 2016 at 1:00:21 PM UTC-7, Stefan Karpinski wrote:

No, Function doesn't have signatures, arity or return type as part of its type. The signature of a function is the union of its method signatures, which is potentially very complicated. Type parameters are not contravariant, so they can't be described without massively complicated Julia's (already complicated) type system. Worse still, adding any form of contravariance would almost certainly make important predicates like subtype and type intersection undecidable. There are still things that could be done to get some of the features that you probably want from function types, but dispatching on the return type is unlikely to ever be allowed. Two things that may happen are:

Got it, thanks! I want method signatures for documentation, debugging, and as constraints or hints for the compiler, which are exactly what your two things provide.

Are these are what you call 'interfaces' in your JuliaCon 2016 keynote, discussed here https://github.com/JuliaLang/julia/issues/6975? 

[Stefan Karpinski]

Yes, that's essentially it – except that since we haven't converged on a particular design, it's hard to say exactly what interfaces are at this point. But yes, it's something that provides a first class representation of some protocol/interface.