Extending functions in Base (or another module)

[Bill Hart]

We have hit an issue that we can't seem to find a workaround for. Our only working workaround is no longer supported by Julia 0.5.

The issue

========

We implement determinant for various kinds of matrix types in Nemo and Hecke (two computer algebra/number theory packages). To do this, we extend Base.det in Nemo.

Hecke depends on Nemo and tries to extend Nemo.det (or Base.det).

Hecke wants to define det for one of its own types, let's call it SpecialMat. Now Hecke's SpecialMat belongs to Nemo's abstract type MatElem, i.e. SpecialMat <: MatElem.

Nothing unusual so far.

The problem is: Hecke would like to call the det function provided by Nemo in case the algorithm they provide is going to be slower (a decision that is made at runtime).

What we try

=========

So what we naturally try in Hecke is something like the following (obviously this code doesn't actually work):

module Hecke

using Nemo

type SpecialMat <: MatElem  ## Nemo has a MatElem type class and a "generic" det algorithm for any type that belongs to it
   # some data

end

function det(m::SpecialMat)

    # do different things depending on properties of the matrix m

   if check_some_properties_of_a_matrix(m)

       # implementation of special determinant algorithm that only works for Hecke SpecialMat's

   else

       Nemo.det(m) # fallback to the Nemo implementation of det (which is extensive)

   end

end

export det

end # module

Here are some potential solutions we tried which didn't work:

1) Only import the functions from Nemo that Hecke doesn't need to overload, i.e. don't import det from Nemo (or Base)

    This causes Julia to tell the user that det could refer to Base.det or Hecke.det, even though Base.det doesn't provide a function for the specified type. We certainly can't expect the user to have to look up all the documentation for Base, Nemo and Hecke every time they call a function so they know how to qualify it. So this isn't a workable solution, obviously. It's also far too verbose.

2) Try to qualify the function name with the name of the module, i.e. call Nemo.det in the Hecke definition of the function, as above. 

   This doesn't work, since it is Nemo.det that currently has being overloaded. So the Hecke det function just segfaults when called.

3) Look up the method table for the function we require and call the specific method. 

   This works in Julia 0.4, but the ability to call methods has been removed in 0.5.

This is an exceedingly frustrating problem. In fact it also occurs within Nemo itself, since every time we want to implement a specialised version of a generic function for a specific type, and have it fall back to the generic version in certain cases determined at runtime, we can't do it, without first renaming the generic implementation to something else with a different name.

This sort of thing is making it very difficult to build large systems. It's not fair to the developers of Hecke to ask them to duplicate all the code in Nemo just so they can make this work, or alternatively force Nemo to define every function twice so that there is a callable version with a different name.

Does anyone know a workaround (any hack that works will do) for this issue, that works in Julia 0.4 and Julia 0.5?

And is there a plan to fix this sort of issue in Julia in the future? The module system currently makes it quite hard to work with multiple modules. We are often encouraged to split our large systems into smaller modules/packages to get around certain issues, and then when we do that, the module system actually gets in the way.

Bill.

[Bill Hart]

By the way, here are some potential things that could solve this for us:

1) Allow individual methods to be callable, as in Julia 0.4 (so long as we can look the methods up at "compile time" in a module and assign them to a const in the module so that they can be called). As far as we can see, this has been removed in Julia 0.5.

2) Have a syntax for calling a method with a given signature, e.g. apply(det, (MatElem,), m) or something like that. Something like this used to exist in Julia but has been removed.

3) Have Julia not print an ambiguity warning about Base.det/Nemo.det/Hecke.det when calling det on a matrix where there is a specific implementation for that type in Hecke, but only something more general in the other modules. Currently Julia complains that you might mean Base.det or Hecke.det even though the former does not implement a function for matrices of type SpecialMat. But why? What's ambiguous here?

4) Remove the requirement for a module to extend the definition from Base or whatever other packages it uses, if it is only defining the function for one of its own types and not a type that is present in Base or that other module.

5) Introduce a way of calling an implementation provided by a specific module, e.g. specifying Nemo.det would only call the version actually defined in Nemo. This might feel like the current behaviour, but it doesn't actually work, since it is Base.det or Nemo.det that Hecke needs to be overloaded in the first place. So calling Nemo.det from within Hecke.det just causes it to call Hecke.det for that specific type, causing a stack overflow. 

6) Allow us to define function Hecke.det (and still export it as det) in Hecke, and call Nemo.det from within that implementation. Again, this might feel like the current behaviour, but it doesn't work because it is Nemo.det or Base.det that needs to be defined in Hecke, currently.

Bill.

[Bill Hart]

Another possibility: introduce a special "extends" syntax to Julia, e.g.

module Hecke

type SpecialMat <: Nemo.MatElem

end

function det(a::SpecialMat)

   if blah

       # do whatever

   else

      Nemo.det

   end

end

extends det Nemo.det # tells Julia to somehow treat the det implementation above specially so it can call the Nemo version of the function without calling itself.

end # module

The behaviour would be: if extends is missing, it gives an ambiguity warning about Nemo.det vs Hecke.det if someone calls det after including Hecke and Nemo.

Bill.

[Rafael Fourquet]

I'm far from expert on those questions, but would the "invoke" function work?
I think it's considered to be a tool of the last resort, but seems to
be the situation you are in!

invoke(f, (types...), args...)

Invoke a method for the given generic function matching the
specified types (as a tuple), on the specified arguments. The
arguments must be compatible with
the specified types. This allows invoking a method other than the
most specific matching method, which is useful when the behavior of a
more general
definition is explicitly needed (often as part of the implementation
of a more specific method of the same function).

[Bill Hart]

You are a life saver. This is precisely what we need. Thank you for solving a very difficult problem for us. We were really pulling our hair out after searching for a solution.

julia> module Nemo

          import Base: det

          abstract MatElem

          function det(a::MatElem)

             return 1

          end

          type nemomat <: MatElem

          end

          export nemomat

          export det

       end

Nemo

julia> module Hecke

          using Nemo

          import Nemo.det

          type SpecialMat <: Nemo.MatElem

             data::Int

          end

          function det(a::SpecialMat)

             if a.data == 4

                return 3

             else

                return invoke(det, (Nemo.MatElem,), a)

             end

          end

          export SpecialMat

          export det

       end

Hecke

julia> using Hecke

julia> s = SpecialMat(3)

Hecke.SpecialMat(3)

julia> t = SpecialMat(4)

Hecke.SpecialMat(4)

julia> det(s)

1

julia> det(t)

3

Let's help search engines with this, since we were unable to find anything, and it is such an important issue:

trouble extending a Base function in Julia

How do I call a specific version of a function in Julia for specific types

How do I call a more general version of a function in Julia

How do I call a less specific version of a function in Julia

How do I call a specific method in Julia

method to apply a function in Julia

invoking a given version of a function in Julia

To the Julia devs: please, please don't remove this functionality!

Bill.

[Mauro]

There is some discussion on invoke on github:
https://github.com/JuliaLang/julia/pull/13123
maybe you want to weight in.

On Tue, 2016-06-28 at 19:19, 'Bill Hart' via julia-users <julia...@googlegroups.com> wrote:
> You are a life saver. This is *precisely* what we need. Thank you for

[Pablo Zubieta]

You might also want to express your perspective on why a function such as invoke is needed here https://github.com/JuliaLang/julia/pull/13123.