Optional return values

[Taylan Ulrich Bayırlı/Kammer]

Another part of my musings, after the zero-values idea. This time it
is, as far as I'm aware, a novel mechanism, i.e. I've never heard of
this idea being considered to be or actually implemented in any language
(as opposed to zero values which are possible in Scheme and just need to
be specified for certain forms), and the couple times I mentioned it to
someone it's been met with "meh" reactions, but here goes anyway just
for the sake of sharing.


The usefulness of a procedure with optional parameters is widely
acknowledged and has many use-cases. But what about the counterpart,
that is providing arguments or values which are optional to handle by a
procedure or continuation?

Perhaps the most obvious use-case:

(table-lookup table key) => value, [found?]

It returns the value associated with 'key' in 'table' or #f when there
is none, and it returns the optional second value that is a Boolean
indicating whether a value was found. So if you care, you can use it to
distinguish between an #f that was found in the table, and an #f
signaling "not found." If you don't care, you can call the procedure in
a context expecting one value without getting an "excess values" error.

Perhaps the reason this has never caught on is that the above is, in
fact, the only use case I could think of so far. :-)

***

When a procedure with optional parameters is called also with optional
values, the behavior would be as follows:

The mandatory and optional parameters of the procedure are conceptually
appended to create a full parameter list. The mandatory and optional
values passed to the procedure are appended as well, creating a full
value list. Each value in the value list fills the corresponding slot
in the parameter list until either list is exhausted. It is an error if
there remain unfilled mandatory parameter slots, or unused mandatory
values.

(That's a mouthful to word, but pretty clean and intuitive once you've
grokked it, IMO.)

***

On a conceptual level, the combination of optional parameters and
optional arguments should be about as efficient to implement as an
efficient implementation of optional parameters only. When you have
optional parameters only, you need to compare the number of provided
values to the minimum (non-optional) and total (non-optional + optional)
number of arguments accepted by your procedure.

nargs >= nparams - nparams_opt && nargs <= nparams

When you also have optional arguments, you need to compare the minimum
number of arguments accepted to the total number of values provided, and
the total number of arguments accepted to the minimum number of values
that need to be handled.

nargs >= nparams - nparams_opt && nargs - nargs_opt <= nparams

That is conceptual only of course, I have no "real" implementation of
this, only <http://taylanub.github.io/doc/scheme-values.scm.txt> which
is a bit incomplete and too high-level. Someone more knowledgeable than
me might say "nope, way too much overhead on every procedure call/return
once you implement that close to the metal."


*** *** ***


The above idea stands on it own, and I could have made an additional
post on the following, but it probably doesn't deserve it since it
starts getting a bit crazy from here on:

The idea can be carried over to keyword parameters/arguments as well.
While keyword parameters are intrinsically optional, keyword arguments
will normally make a procedure that didn't expect them to spew an error
(unless the implementation is so lenient it doesn't do the consistency
check at all, but I'd say that's bad). Providing keyword arguments that
are optional to handle might sound like an obscure use-case, but guess
in which code-base I've found a use-case for them. Once more, it's in
GNU Guix. :-)

The build mechanism is fairly sophisticated (necessarily so), and every
build phase procedure receives a ton of keyword arguments from the
framework, no matter which ones that precise phase cares about. Most
build phase procedures thus use the #:allow-other-keys token in their
parameter list, which disables the check that the procedure handles all
keyword arguments it's given. This is a bit annoying to type every
time, and it means that the framework can't pass any keyword values
which it deems mandatory to handle. Both of these are weak points, but
it's still something.

***

I've had some additional thoughts, especially about the interaction of
keyword parameters/arguments and "remaining arguments" parameters, over
here: <http://taylanub.github.io/doc/scheme-values.html>.

(It's originally a plain-text file of "sketch" nature; I was able to
convert it to this format via the 'pandoc' MarkDown/HTML converter, and
10 lines of minimalist CSS I copied from somewhere. Don't expect much
comprehensibility from the content just because the CSS looks stylish.)

I won't touch on that much further here because the ideas get more and
more unusual, but the gist of it is 1. allow a rest-keyword-arguments
parameter like in Python, 2. allow a super-special "give me a values
object" parameter which instead of allocating you a list of the
positional arguments (like in '(lambda foo ...)'), allocates you an
object of a new type, "values," which encapsulates all information on
the values passed to the procedure: positional/mandatory,
positional/optional, keyword/mandatory, and keyword/optional. Objects
of this type would be accepted by 'apply' in addition to lists. Thus
you can continue to wrap arbitrary procedures, remaining fully agnostic
to their parameters, just passing on the values given to your wrapper to
the wrapped procedure.

I believe that yields a *comprehensive* semantics for value passing: by
position, by name, mandatory to provide, mandatory to accept, optional
to provide, optional to accept, gathering remaining positional arguments
in a list, gathering remaining keyword arguments in a table, or reifying
the whole shebang in an object of a specialized type.

The question would be whether it's useful.

Taylan

[jeffrey.ma...@gmail.com]

Before Scheme had multiple return values, Common Lisp, and Lisp Machine Lisp dialects before that, (and perhaps MacLisp before that; I don't know) had multiple return values with optional multiple values. This goes back at least as far as the 1970s.

[Taylan Ulrich Bayırlı/Kammer]

Don't they just ignore *all* excess values?

I think ignoring data silently by default is a very evil thing to do in
general.

Taylan

[Barry Margolin]

In article <87si62f...@T420.taylan>,

taylan...@gmail.com (Taylan Ulrich Bay?rl?/Kammer) wrote:

> jeffrey.ma...@gmail.com writes:
>
> > Before Scheme had multiple return values, Common Lisp, and Lisp
> > Machine Lisp dialects before that, (and perhaps MacLisp before that; I
> > don't know) had multiple return values with optional multiple

MacLisp didn't have multiple values. I think LM Lisp introduced it, and
CL inherited it.

> > values. This goes back at least as far as the 1970s.
>
> Don't they just ignore *all* excess values?

Yes. And if you expect more values than are returned, the extra
variables get NIL. If you want to know the precise number of returned
values, you have to use MULTIPLE-VALUE-LIST to call it and convert the
returned values into a list.

>
> I think ignoring data silently by default is a very evil thing to do in
> general.

It was a necessary evil for upward compatibility, to avoid having two
versions of all these functions, or forcing applications to recode to
use M-V forms to call them.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Rob Warnock]

Taylan Ulrich Bayırlı /Kammer <taylan...@gmail.com> wrote:
+---------------
| But what about the counterpart, that is providing ... values

| which are optional to handle by a procedure or continuation?
|
| Perhaps the most obvious use-case:
|
| (table-lookup table key) => value, [found?]
|
| It returns the value associated with 'key' in 'table' or #f when there
| is none, and it returns the optional second value that is a Boolean
| indicating whether a value was found. So if you care, you can use it to
| distinguish between an #f that was found in the table, and an #f
| signaling "not found." If you don't care, you can call the procedure in
| a context expecting one value without getting an "excess values" error.

+---------------

You have just decribed how GETHASH (hash table lookup) works in
ANSI Common Lisp:

http://www.lispworks.com/documentation/HyperSpec/Body/f_gethas.htm
Accessor GETHASH
Syntax:
(gethash key hash-table &optional default) => value, present-p
...
Description:
VALUE is the object in hash-table whose key is the same as KEY
under the hash-table's equivalence test. If there is no such entry,
VALUE is the DEFAULT.

PRESENT-P is true if an entry is found; otherwise, it is false.
...
Notes:
The secondary value, PRESENT-P, can be used to distinguish the
absence of an entry from the presence of an entry that has a
value of DEFAULT.

+---------------

| Perhaps the reason this has never caught on is that the above is,
| in fact, the only use case I could think of so far. :-)

+---------------

Au contraire, it *has* caught on... in Common Lisp. CL has *many*
functions which return secondary value(s) which may be ignored if
you don't care about them, e.g.:

(floor number &optional divisor) => quotient, remainder
[...and the like: CEILING, ROUND, TRUNCATE, and so on.]

(find-symbol string &optional package) => symbol, status

(compile name &optional definition) => function, warnings-p, failure-p

(subtypep type-1 type-2 &optional environment) => subtype-p, valid-p

just to name a very few.

For a more on how CL handles multiple values, see:

http://www.lispworks.com/documentation/HyperSpec/Body/03_ag.htm
3.1.7 Return Values
...
In order to receive other than exactly one value from a form,
one of several special forms or macros must be used to request
those values. If a form produces multiple values which were not
requested in this way, then the first value is given to the caller
and all others are discarded; if the form produces zero values,
then the caller receives NIL as a value.
...

Examples:

> (list (values) (values 12) (values 34 56 78))

(NIL 12 34)
> (multiple-value-bind (a b c d)
(values 12 34)
(list a b c d))

(12 34 NIL NIL)
>

The main objection to CL's convenient liberal approach to mismatches
in multiple values it that it sometimes masks programmer errors.
The main objection to Scheme's helpful strict matching of multiple
values it that it is often less convenient than one might want.
Viva la difference!! ;-}


-Rob

-----
Rob Warnock <rp...@rpw3.org>
627 26th Avenue <http://rpw3.org/>
San Mateo, CA 94403

[Taylan Ulrich Bayırlı/Kammer]

rp...@rpw3.org (Rob Warnock) writes:

> You have just decribed how GETHASH (hash table lookup) works in
> ANSI Common Lisp:
>

> [ ... snip ... ]

>
> The main objection to CL's convenient liberal approach to mismatches
> in multiple values it that it sometimes masks programmer errors.
> The main objection to Scheme's helpful strict matching of multiple
> values it that it is often less convenient than one might want.
> Viva la difference!! ;-}

It seems my opening post wasn't clear enough, since you're the second
person to say that.

What I describe in fact involves explicitly marking the returned values
that are optional to handle, separate from the mandatory ones, just like
a procedure with optional arguments also has a number of arguments which
are mandatory to pass to it when called.

This way the strict checking is retained, with controlled tolerance of
certain return values getting ignored. The API designer makes the
choice.

Best of both worlds? :-)


Also note that procedure-call semantics and value-returning semantics
would be one and the same under the envisioned scheme. If one could
pass optional-to-take arguments arg3 and arg4 to a user-provided
procedure like

(proc arg1 arg2 #!optional arg3 arg4)

then returning optional-to-handle values to a caller is just

(call/cc (lambda (k) (k val1 val2 #!optional val3 val4)))

though of course there would be a more convenient form to do that.


Taylan

[Alan Bawden]

taylan...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
> What I describe in fact involves explicitly marking the returned values
> that are optional to handle, separate from the mandatory ones, just like
> a procedure with optional arguments also has a number of arguments which
> are mandatory to pass to it when called.
>
> This way the strict checking is retained, with controlled tolerance of
> certain return values getting ignored. The API designer makes the
> choice.
>
> Best of both worlds? :-)
>
> Also note that procedure-call semantics and value-returning semantics

> would be one and the same under the envisioned scheme....

I think I made this exact proposal for Scheme on multiple occasions
during the 1980s. The fact that it makes the call and return semantics
parallel seemed very attractive to me. If your compiler works via
CPS-conversion, this kind of multiple values almost comes for free.

But not everybody saw things that way, so the standard eventually
settled on a lowest-common-denominator semantics that (in my opinion) is
almost completely worthless in portable code.

--
Alan Bawden

[Taylan Ulrich Bayırlı/Kammer]

I believe standard call/cc allows one to pass multiple values that way,
i.e.

(define (values . vals)
(call/cc (lambda (k) (apply k vals))))

should be a valid definition for 'values'. I'm not sure how explicit
the R5RS is about that (reasonably explicit if I'm reading it right),
but I think it will work in most serious Scheme implementations. Your
efforts have not been in vain, and thank you very much for that, it's a
great semantics indeed!

Taylan