Equivalent of MATLAB's nargout in Julia

[Pooya]

I have a function with two outputs. The second one is computationally expensive, so I want to avoid the computation unless the user needs it. I read on another post in the group that the solution in this case is usually to define two functions, but in this case I basically need to do all the computations for the first output to compute the second. Is there any nicer way to do this in julia? In MATLAB, I would just say:

If nargout > 1

    # do all the computations for second output

end

[Avik Sengupta]

Yes, Julia doesnt have "nargout". The idiomatic way to do this would be to use optional arguments:

http://julia.readthedocs.org/en/latest/manual/functions/#optional-arguments

So:

function f(a, b=0)
  if b > 0
     #do the computations for second argument
   end
   #rest of computation
end

Use a default value that makes sense in your domain.

[Pooya]

Thanks. I thought of using a boolean variable as an input indicating if I need the second output or not, which is exactly like what you mentioned, but was wondering if this is idiomatic or not!

[Steven G. Johnson]

On Wednesday, March 4, 2015 at 11:48:28 AM UTC-5, Pooya wrote:

I have a function with two outputs. The second one is computationally expensive, so I want to avoid the computation unless the user needs it. I read on another post in the group that the solution in this case is usually to define two functions, but in this case I basically need to do all the computations for the first output to compute the second. Is there any nicer way to do this in julia? In MATLAB, I would just say:

Refactorize the code: write a lower-level subroutine that does all the computations for the first output and returns the data needed to compute the second output.   Then write two functions, one which computes and returns only the first output, and one which computes both the first and second outputs.

As others have mentioned, Julia doesn't have nargout.   You could use an optional input argument or a keyword or something to decide whether to compute the second output, but this is usually a bad idea: making the number of output arguments depend on the values of the input arguments makes the function type-unstable, and could screw up performance of anything that calls your function.

[Pooya]

Thanks for your response. I am not sure what you mean by a lower-level subroutine. Is that a function inside another one? If yes, How does the scope of variables work for that? 

Also, if I have what you called the lower level subroutine for output 1 in addition to passing the necessary data for output 2, why do I need to write two other functions? Isn't it enough to just write one more to do the computations for output 2 ?

[Steven G. Johnson]

On Wednesday, March 4, 2015 at 6:38:28 PM UTC-5, Pooya wrote:

Thanks for your response. I am not sure what you mean by a lower-level subroutine. Is that a function inside another one? If yes, How does the scope of variables work for that? 

From your description, right now you have:

function compute_two_outputs(...)

       ...do some stuff, get x, y, and z....

       ....use x, y, and z to compute output1....

       ....use output1, x, y, and z to compute output2....

      return output1, output2

end

Instead, if you don't always want to compute both outputs, but still want to write the shared computations only once, you can refactor the code to pull out the shared computations into another function (that is "lower level" in the sense that users won't normally call it directly):

function some_stuff(...)

       ...do some stuff, get x, y, and z....

       ....use x, y, and z to compute output1....

       return output1,x,y,z

end

function compute_output1(...)

      return some_stuff(...)[1]

end

function compute_two_outputs(...)

       output1,x,y,z = some_stuff(...)
       ....use output1, x, y, and z to compute output2....

      return output1, output2

end

[Pooya]

Thanks for this clear explanation. If I do the following, is my function type still unstable? How do you compare the following solution to yours in terms of efficiency, style, etc? 

function compute_outputs(..., output2Flag)

  # do some stuff, get x, y, and z

  # compute output 1

  output1 = ...

  if output2Flag

    # compute output 2

    output2 = ...

  else

    output2 = SparseMatrixCSC[]  # the same type as output2 when it is computed

  end

  return output1, output2

end

[Milan Bouchet-Valat]

Le jeudi 05 mars 2015 à 11:59 -0800, Pooya a écrit :
> Thanks for this clear explanation. If I do the following, is my
> function type still unstable? How do you compare the following
> solution to yours in terms of efficiency, style, etc?
>
> function compute_outputs(..., output2Flag)
> # do some stuff, get x, y, and z
> # compute output 1
> output1 = ...
> if output2Flag
>
> # compute output 2
> output2 = ...
> else
> output2 = SparseMatrixCSC[] # the same type as output2 when it is
> computed
> end
> return output1, output2
> end

This version indeed appears to be type-stable, so it should be quite
efficient. Users will also be able to write
a, b = compute_outputs(..., false)
or
a, = compute_outputs(..., false)
when they don't care about the second output. So it's not a bad design,
but returning a second output even when not needed isn't super
satisfying.

Thus, maybe Steven's suggestion to create two separate functions is
better. Do you have any reason to think it's not practical for your
case?

We should probably decide what's the most idiomatic solution, and
document it to ensure consistency. What do other people think?


Regards

[J Luis]

We should probably decide what's the most idiomatic solution, and
document it to ensure consistency. What do other people think?

Is there any fundamental reason why the nargout mechanism cannot (or is very hard) to implement?
Because if not I really think it would be very very handy to have it. While we can workaround the nargin concept with the multiple dispatch, the same does help for the nargout.

 

[Tim Holy]

I'll take it in the opposite direction of uniformity, and point out that
another useful approach is to pass the optional output in as an argument:

function myfunction!(output2, input1, input2, ...)
# do some calculations
if !isa(output2, Nothing)
for i = 1:n
output2[i] = ...
end
end
output1
end

output1 = myfunction!(nothing, x, params)
g = Array(T, sz)
output1 = myfunction!(g, x, params)

This is used extensively in optimization, where output2 might be storage for
the gradient.

Best,
--Tim

[Pierre-Yves Gérardy]

nargout can be emulated by (ab)using the iteration protocol on a
custom return type that will be accessed through tuple
assignment/destructuring.

Take this example:

tpl = (1,2)
a,b = tpl

the second line will call start(), done() and next() on tpl to extract
the two numbers.

So, for your use case:

type Result; output1; otherstate; end

function myfunc(x, y)
return Result(x+y, (x, y))
end

my_expensive_func(r::Result)
#compute output2
end

start(r::Result) = 1
done(::Result, i) = i > 2
function next(r::Result, i)
if i == 1
(r.output1, 2)
elseif i == 2
(my_expensive_func(r), 3)
end
end

to get the first result, use this (note the coma):

a, = myfunc(4,5)

Hopefully this helps :-)
—Pierre-Yves

[Kevin Squire]

On Fri, Mar 6, 2015 at 11:43 AM, J Luis <jmf...@gmail.com> wrote:

Is there any fundamental reason why the nargout mechanism cannot (or is very hard) to implement?
Because if not I really think it would be very very handy to have it. While we can workaround the nargin concept with the multiple dispatch, the same does help for the nargout.

I'm not an expert on the innards of the Julia compiler, but I believe the most important reason is that it interferes with type stability of a function, which in turn interferes with the ability of the compiler to generate optimized code.

Additionally, functions can currently be used as callbacks from C and Fortran code (and there's generally a desire to keep/enhance C interactivity as much as possible).  If we wanted functions to know the number of output parameters, there would need to be a mechanism to provide this information to the function at call time (e.g., the number of args would perhaps need to be passed in as a hidden parameter).  As there is no mechanism to do this in C, we would either lose compatibility with C, or need to provide a new kind of function, solely for this purpose.

So while adding such functionality might add convenience (mostly for people coming from Matlab), it 1) would produce slower, less optimized code, and 2) would require adding a new kind of function to the language and compiler (which is already pretty complicated).

I think that would be a pretty hard sell to the primary developers.

Cheers!

   Kevin

[Jameson Nash]

I'd put together one version of a gist of how this could work in the past, and Jeff's mentioned a similar draft idea he had in an early design document. In the end, however, I think this just ends up falling in the category of "features that just aren't actually desirable to have". It adds complexity to the compiler, true. More importantly though, it adds additional complexity to the user's concept of a function, and without sufficient benefit to justify itself. This feature is often useful in matlab for avoiding the expense of some extra computations (which are often very expensive in matlab in the first place), but it's also often abused to have a function do something different. Additionally, it makes composability much more difficult (since the function call starts to depend upon the syntax of the call), such that it can be harder to write fully generic code.

Julia code is already much more expressive than C, and requires explicit ccall / cfunction conversions to translate between the languages, so direct compatibility isn't actually much of a concern.

[Kevin Squire]

Thanks, Jameson--it's nice to have a response from someone who actually works on the compiler.

Cheers,

   Kevin

[J Luis]

Thanks me too for explanations.
Note that I was careful enough to have anticipated that there would likely exist deep reasons why a feature like this would have a strong impact on other aspects of the language performance.
But while these performance worries are absolutely valid reasons, I have to disagree on arguments of the kind "adds additional complexity to the user's concept of a function, and without sufficient benefit to justify itself". About these, please, let the users decide. I remember endless discussions on the Mathworks forum about GOTOs where a large number of people considered that a GOTO was a too powerful command to be let used by an average user who would undoubtedly use it to write spaghetti code.

Kevin, curious that you refer that "there is no mechanism to do this in C" because, while it's a particular C environment, the MEX env has this 'nargout' concept and it would really easy up my life if I could use it in Julia when porting the GMT MEX wrapper.

[Kevin Squire]

Kevin, curious that you refer that "there is no mechanism to do this in C" because, while it's a particular C environment, the MEX env has this 'nargout' concept and it would really easy up my life if I could use it in Julia when porting the GMT MEX wrapper.

Well, I think Jameson kind of contradicted that concern, and I think I see his point.  My concern was that, generally, calling C functions (which is quite common in Julia) involves pushing arguments onto the stack and then calling the routine.  So for the routine to know the number of output arguments, you would need to pass that in a similar manner.

Mex files have a specific mechanism for doing this, and require (?) that you write a wrapper ("gateway routine") for any C function that you want to write.  Python is similar.  In contrast, in Julia it's not necessary to write a wrapper (although it can be good to do so to provide a nicer programming interface).  But either way, the call to C code is much more direct--you're not required to write a wrapper.  But then, you have to follow C calling conventions.

My concern was actually the other way around--it's possible to pass pointers to Julia functions to C code as a callback.  My impression was that the same rules then apply--the Julia function would be required to adhere to C calling conventions. 

Jameson suggested, though, that in either case, there's a bit of compiler translation going on behind the scenes.  For the case that matters (passing Julia functions to C), this probably means that the compiler is making sure the function is compiled (or automatically wrapped) in such a way that it can be called using C conventions, and which would be different from how the function would be called in Julia.  In theory, then, you could do something similar for Matlab.

All that is to say... I was wrong. ;-)

At any rate, this has been discussed before (probably multiple times), and right now it doesn't look like Julia will evolve in that direction.  For one, while Julia looks a lot like Matlab (on purpose), the goal hasn't been to mimic all of Matlab's design, especially when that conflicts with Julia's own design goals.  The more important reason at this point, though, is probably momentum: if someone had made a decision to include this feature a while ago, it probably would have been fine.  At this point, the core language itself has generally been pretty stable, so adding this would be a pretty disruptive change, and the people who are actually working on the compiler (like Jameson) are focused on other concerns.

Cheers,

   Kevin

[Pooya]

No, I don't have any particular reason. Steven's suggestion includes three functions which need different names, but are written basically for one goal, whereas mine needed only one function. I thought it might be more elegant. However, Steven and many others who have commented here are experts on these things, so I may be wrong.

[Pooya]

Thanks for this interesting solution! Coming from MATLAB, where data is passed by value, this did not make much sense to me, however, when passing inputs to functions by reference, this works very well. Interesting that you mentioned about the case in optimization, as this is in fact what I'm doing. This function computes some mismatch function (a vector g(x)), the second output is the Jacobian of that (dg/dx), and the objective is to minimize 1/2 * g(x)' * g(x). By the way, the jacobian is computationally expensive either in MATLAB or Julia.