[racket] Splicing `values' in-place

[Laurent]

In some postfix languages, if a procedure returns multiple values, these values can be used directly as multiple arguments to another procedure call, i.e., they are "spliced" in the latter call.

In an extended Racket, this would look like this:

(+ (values 1 2) (values 3 4))

would be equivalent to
(+ 1 2 3 4)

(map values '(0 1 2) '(a b c))

would return

'(0 a 1 b 2 c)

(call-with-values (lambda()(my-proc ....)) list)
would simply be

(list (my-proc ....))

(values (values 1 2) (values 'a 'b))

would be equivalent to

(values 1 2 'a 'b)

Correct me if I'm wrong, but I think all the cases where this feature should be useful currently throws an error, so it would probably break only very little.

Such a missing feature tickles me from time to time, and I often find that Racket `values' system is too cumbersome to be used more often, i.e., you need to go through stages of `call-with-values', 'let/define-values', `(apply values ....)', etc. and I often find myself not wanting to go down this road.

IMO, `values' is *meant* to be the way I describe above: `values' is exactly like `list', except than instead of encapsulating the values in a container, it splices them in-place.

Do you see some disadvantages of using values this way?

For example, in some occasions, for things like

(define (foo x) (values x x))

(map + (foo '(1 2 3)))

it may be more difficult to infer that there are actually 2 lists in the map, but to me it's just a matter of style/taste/comments/documentation, not a matter of feature.

Laurent

[Matthias Felleisen]

Your uses of values are covered in apply/map/append/list trickeries. Using values might be more elegant, but yes, it's currently not possible.

> ____________________
> Racket Users list:
> http://lists.racket-lang.org/users

[Laurent]

On Thu, Jul 11, 2013 at 4:42 PM, Matthias Felleisen <matt...@ccs.neu.edu> wrote:

Your uses of values are covered in apply/map/append/list trickeries.

Yes, for example

(+ (values 1 2) (values 3 4))

would need to be written

(apply + (append (call-with-values (lambda()(values 1 2)) list) (call-with-values (lambda()(values 3 4)) list)))

which is really cumbersome.

However, actually, if `values' were exactly `list' (e.g., with `define-list' instead of `define-values'), I think it would be more usable than what it is now.

At least there would be no need to convert values to lists, use list operations, then convert lists back to values.

The above example would be written:

(apply + (append (list 1 2) (list 3 4)))

which is already much better.

But then, compared to splicing values, every call would need to be like this one, using `apply' and `append' everywhere, which is also quite cumbersome (and not elegant).

And `(list 1)' is not equivalent to `1'.

On the contrary, with splicing values, `(values 1)' is equivalent to `1', as it should be.

I guess Scribble would be happy to use such splicing values.

 

Using values might be more elegant, but yes, it's currently not possible.

And I think it's the kind of elegance that makes the distinction between usable and not.

(Since most languages are Turing-complete, features are not a matter of whether something can be done but of how much effort it takes to do something, but you are well aware of that, and Racket is very good at that, most of the time. I just wish some of the core designs would allow for a bit more flexibility.)

Laurent

[Matthew Flatt]

To elaborate on "currently not possible" (because this idea shows up
from time to time), allowing splicing of results in function-call
subexpressions would break equivalences that are currently exploited by
macros and the compiler.

For example, many macros assume that

(rator rand1 rand2 ... randn)

can be rewritten as

(let ([r rator]
[a1 rand1]
[a2 rand2]
...
[an randn])
(r a1 a2 ... an))

That would not be the case if the `rand's can produce multiple values.
(I assume that you don't want to allow binding multiple values to a
variable in `let'.) I think that disallowing this kind of
transformation would make many macros more difficult to implement ---
and maybe impossible, in some cases.

The Racket compiler takes advantage of transformations like the one
above to speed up your code. Although the compiler could still perform
transformations when the relevant subexpressions are known to be
single-valued, I think the transformations would apply much less often
than now.

Along similar lines, various tools can tell statically you that

(cons e1 e2 e3)

will be an arity error (assuming that `cons' is the usual binding).
That kind of support would become much weaker, since `e2' might return
zero values while `e1' and `e3' return a single value.

In short, the kind of splicing that you suggest is a significant sense
more "dynamic" than Racket. You could always embed such a dynamic
language in Racket. Due to macros, however, I don't think it would work
to re-interpret our existing code as being written in that language.
And due to the extra constraints on the compiler and run-time system,
I'm certain that it would perform worse than Racket. Overall, my sense
is that the potential extra convenience of splicing values is not worth
the costs.

> ------------------------------------------------------------------------------
> [application/pkcs7-signature "smime.p7s"] [~/Desktop & open] [~/Temp & open]

[Neil Van Dyke]

FWIW, several years ago, I implemented a wrapper for "#%app" that supported this, as an exercise and without actually using it, but I kept it in mind.

I use multiple-values a lot, including in named-"let" recursion, and, since that multiple-value-splicing "#%app" exercise, I have noticed times when it would have been a win.  But I have also noticed at least as many times when I had a coding error that multiple-value-splicing would've obscured.  I intuitively think it would be bad for readability, but I don't have even anecdotal data for that.

Of course, people with other coding styles or different kinds of programs would put different weights on the pros and cons.  Do you think people would be happy with an "app" macro that they'd use only when then wanted to so an apply with multiple-values splicing?  It's easy to implement in an inefficient way (below is such an inefficient implementation I made for a blog post a long time ago):

(define-syntax app
 (syntax-rules ()
   ((_ PROC)         (PROC))
   ((_ PROC ARG ...) (%app:1 (ARG ...) () (PROC)))))

(define-syntax %app:1
 (syntax-rules ()
   ((_ () (BIND ...) (PROC VAR ...))
    (let (BIND ...)
      (apply PROC (append VAR ...))))
   ((_ (ARG0 ARG1 ...) (BIND ...) (PROC VAR ...))
    (%app:1 (ARG1 ...)
            (BIND ... (actual (call-with-values (lambda () ARG0) list)))
            (PROC VAR ... actual)))))

(app vector 1 (values 2 3) 4 (list 5) (values 6 7))
;;==> #(1 2 3 4 (5) 6 7)

I haven't really needed such a macro myself so far.  Before I'd want that macro, I would want things like multiple-value support in "let".

Neil V.

____________________
  Racket Users list:
  http://lists.racket-lang.org/users
  

[Hendrik Boom]

All this might be possible in typed racket, if the specific type(s) of
a multiple returned value were known.

>
> In short, the kind of splicing that you suggest is a significant sense
> more "dynamic" than Racket.

And suitable typing would make it less dynamic again.

-- hendrik

[Jens Axel Søgaard]

Hi Laurent,

It is interesting to see the historical evolution of multiple values in Scheme.

In June 1988 this proposal was posted:

http://groups.csail.mit.edu/mac/ftpdir/scheme-mail/HTML/rrrs-1988/msg00230.html

At the Snowbird meeting in July 1988 no concensus was found.

3.7. Add LAMBDA*. Not adopted. The main objection seemed to be that

this proposal requires a new kind of stored value (multiple values) that

either is not an expressed value or does not behave like other expressed

values.

 

3.8. Multiple return values. Not adopted. Debate centered on whether

returning a single "multiple value" should be equivalent to returning

that value normally. A secondary issue concerned whether extra return

values should be ignored or an error. We approached consensus on this.

After much debate it was decided to go on to other issues and let

Dybvig and Hanson discuss the matter in private. Their discussion

later that night led to a conclusion that we don't understand the

issues well enough to charge ahead with either proposal 3.7 or 3.8

or a variation.

 

3.9. Optional arguments. Not considered, as it was felt that this

would be a more controversial issue than proposal 3.8 on which we had

just spent a great deal of time.

The discussion went on on this thread:

http://groups.csail.mit.edu/mac/ftpdir/scheme-mail/HTML/rrrs-1989/msg00148.html

Scroll down on this page to see all entries: http://groups.csail.mit.edu/mac/ftpdir/scheme-mail/HTML/rrrs-1989/threads.html

The discussion on multiple values pops up now and again. See for example this

thread on comp.lang.scheme.

https://groups.google.com/d/msg/comp.lang.scheme/ty2YeqZibNI/VR21X6XP9-IJ

As for using multiple values in Racket, I use these lines

    (define-syntax defv (make-rename-transformer #'define-values))

    (define-syntax defm (make-rename-transformer #'match-define))

    (define-syntax def  (make-rename-transformer #'define))

to make local definitions short. 

A little function to make a rotation matrix:

    (define (rotmat/rad θ)

      (defv (cosθ sinθ) (cos&sin θ))

      (mat cosθ (- sinθ) sinθ cosθ))

-- 

Jens Axel Søgaard

[Laurent]

Ah that unfortunately explains a lot. Thank you Matthew for this explanation.

It's too bad that one needs to throw away nice semantics for speed...

Neil's macro idea may then well be one of the best intermediate solutions then.

Jens, thanks for all the links, they are quite informative.

Indeed, quite a number of people seem unhappy with `values'.

However, since I did not find an answer for the following, I will risk this anyway:
Then why not return lists instead of values? Performance problem again?

There would not be such a thing as "multiple return values" though, and one should not see them this way.

Several values returned in a list is just a list, i.e., a single return value. Then there are helpers to bind several values from a list, etc.
No need for `apply-values' or `call-with-values', just use `apply'.
Therefore returning `(list 1)' would of course be different from returning `1', but if you don't see them as multiple return values, it's not inconsistent.

The type of the procedure  would tell what is returned anyway.

Some languages like Python, PHP (and Matlab?) do that, and I find this more convenient than `values'.

A "problem" would then be quieter errors for multiple/single value mismatch, but I don't really see this one as important.

I also don't see the need for a different data type like `sequences' as they call it in Jen's thread.

When I use `values', either I (have to) bind them directly to identifiers, or I turn them into a list of values to manipulate them otherwise.
The above would make this much easier and simpler I suspect.

Laurent

[Matthias Felleisen]

On Jul 11, 2013, at 2:23 PM, Laurent wrote:

Then why not return lists instead of values? 

Error checking. See Dybvig's paper, which extensively explores multiple value implementations: 

J. Michael Ashley and R. Kent Dybvig. An efficient implementation of multiple return values in Scheme. In Proceedings of the 1994 ACM Conference on Lisp and Functional Programming, 140-149, June 1994. Describes Chez Scheme's implementation of the Scheme multiple return values interface.

On: http://www.cs.indiana.edu/chezscheme/pubs/

[Hendrik Boom]

On Thu, Jul 11, 2013 at 08:23:01PM +0200, Laurent wrote:
> Ah that unfortunately explains a lot. Thank you Matthew for this
> explanation.
> It's too bad that one needs to throw away nice semantics for speed...
>
> Neil's macro idea may then well be one of the best intermediate solutions
> then.
>
> Jens, thanks for all the links, they are quite informative.
> Indeed, quite a number of people seem unhappy with `values'.
>
> However, since I did not find an answer for the following, I will risk this
> anyway:
> Then why not return lists instead of values? Performance problem again?
> There would not be such a thing as "multiple return values" though, and one
> should not see them this way.

I think the multiple return values are a remnant of ancient LISP
systems, where they happened to have a few registers left over to
return values in. The caller could just ignore the extra register
contents unless special code was used to retrieve them.

But the feature has unfortunate implications for the language
design, static checking, code comprehensibility, and so forth, as
described in a previous post.

I'd avoid putting it in the language with a ten-foot pole.

Well, maybe specific syntax for splicing in extra arguments could be
the tten-foot pole all these tools use to avoid touching it, but I've
always thought it stinks.

> Several values returned in a list is just a list, i.e., a single return
> value. Then there are helpers to bind several values from a list, etc.
> No need for `apply-values' or `call-with-values', just use `apply'.
> Therefore returning `(list 1)' would of course be different from returning
> `1', but if you don't see them as multiple return values, it's not
> inconsistent.

Lists I understand.

> The type of the procedure would tell what is returned anyway.
> Some languages like Python, PHP (and Matlab?) do that, and I find this more
> convenient than `values'.

Types could work, But types are not static in Scheme. As a language
designer, I prefer static typing, so maybe I'm showing a prejudice
here.

> A "problem" would then be quieter errors for multiple/single value
> mismatch, but I don't really see this one as important.
> I also don't see the need for a different data type like `sequences' as
> they call it in Jen's thread.
>
> When I use `values', either I (have to) bind them directly to identifiers,
> or I turn them into a list of values to manipulate them otherwise.
> The above would make this much easier and simpler I suspect.
>
> Laurent

[Jay McCarthy]

I prefer to think of values as being justified by not restricting the
arity of continuations.

(let/cc k
(k 1 2 3))

(let/cc k
(k))

etc.

define-values & let-values create N-arity continuations whereas
call-with-values creates an any arity context

Jay

Jay McCarthy <j...@cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93

[Stephan Houben]

Hi Jay,

So what about keyword return values?
Both because of symmetry and because getting three return values straight is just as hard as getting three arguments straight.

Stephan Houben

Op 11 jul. 2013 21:39 schreef "Jay McCarthy" <jay.mc...@gmail.com> het volgende:

[Jay McCarthy]

On Thu, Jul 11, 2013 at 1:52 PM, Stephan Houben <steph...@gmail.com> wrote:
> Hi Jay,
>
> So what about keyword return values?
> Both because of symmetry and because getting three return values straight is
> just as hard as getting three arguments straight.

That's old news:

https://github.com/jeapostrophe/exp/blob/master/values.ss

FWIW, keywords aren't built-in in Racket for function calls, so this
approach to them in function returns *is* symmetric.

Jay

[Stephan Houben]

I am deeply impressed.

Python's Guido van Rossum is often suspected of owning a time machine,

but apparently you have one too...

2013/7/11 Jay McCarthy <jay.mc...@gmail.com>

[Greg Hendershott]

>> Then why not return lists instead of values?
> Error checking.

Although nowadays we have match-define and friends.

(define-values (a b c) (return-3-values))
(match-define (list a b c) (return-3-element-list))

Both would error appropriately.

[Jay McCarthy]

I just wrote a blog post about this code, in case it is unclear how it works

http://jeapostrophe.github.io/2013-07-15-values-post.html

[Eli Barzilay]

Two days ago, Jay McCarthy wrote:
> On Thu, Jul 11, 2013 at 1:52 PM, Stephan Houben <steph...@gmail.com> wrote:
> > Hi Jay,
> >
> > So what about keyword return values? Both because of symmetry and
> > because getting three return values straight is just as hard as
> > getting three arguments straight.
>
> That's old news:
>
> https://github.com/jeapostrophe/exp/blob/master/values.ss
>
> FWIW, keywords aren't built-in in Racket for function calls, so this
> approach to them in function returns *is* symmetric.

IIRC, when the then-new keywords where added Matthew had a concrete
way to wrap continuations so that things would be symmetric again, but
I think that practicalities wo and he preferred to avoid the cost.
(Either a run-time cost, or making things complicated.)

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

[Eli Barzilay]

Two days ago, Matthias Felleisen wrote:
>
> Your uses of values are covered in apply/map/append/list
> trickeries. Using values might be more elegant, but yes, it's
> currently not possible.

Not possible, but as usual, easy to play with (if you're willing to
pay the runtime overhead). Since it's a semi-popular subject that
comes up frequently enough, at some point I made it into a "language
one-liner" thing -- it's a cute exercise but unfortunately multiple
values are already an "exotic" enough feature to make it into a good
enough first-language thing...

You start with a quick macro as a first step (I haven't looked, but
it's probably close to what Neil posted):

#lang racket
(define-syntax-rule (vals expr ...)
(append (call-with-values (lambda () expr) list) ...))
(define-syntax-rule (app f expr ...)
(let ([l (vals f expr ...)]) (apply (car l) (cdr l))))

The next step is to rename things so you can use it implicitly in the
rest of the code for all function applications

#lang racket
(require (only-in racket [#%app r:app]))
(define-syntax-rule (vals expr ...)
(r:app append (r:app call-with-values (lambda () expr) list) ...))
(define-syntax-rule (#%app f expr ...)
(let ([l (vals f expr ...)]) (r:app apply (r:app car l) (r:app cdr l))))
;;
(define (foo) (values list-ref '(x y z)))
((foo) 1) ; => 'y

but then you take it in the other direction, and make it into a
language instead, first do it with sub-modules:

#lang racket
(module values racket
(define-syntax-rule (vals expr ...)
(append (call-with-values (lambda () expr) list) ...))
(define-syntax-rule (app f expr ...)
(let ([l (vals f expr ...)]) (apply (car l) (cdr l))))
(provide (rename-out [app #%app]) (except-out (all-from-out racket) #%app)))
(require 'values)
(define (foo) (values list-ref '(x y z)))
((foo) 1) ; => 'y

And finally just remove the sub-module wrapper and make it into a
"proper" language.

(BTW, for people on IRC: one feature of rudybot is that you can
initialize it from a URL holding some code -- so it's probably still
possible to use "init http://tmp.barzilay.org/values.rkt" to play with
this. (Yeah, just confirmed that it still works.))

Two days ago, Matthew Flatt wrote:
> To elaborate on "currently not possible" (because this idea shows up
> from time to time), allowing splicing of results in function-call
> subexpressions would break equivalences that are currently exploited
> by macros and the compiler.

IMO, it's a bad idea for a more basic problem: you get functions to
interact with each other based on where they appear in the code. The
above is an extreme example of an application with three different
types, so if you have:

(define (foo) (values list-ref '(x y z)))
(define (bar) 1)
((foo) (bar))

and you decide to return just the function from `foo', then you need
to adjust `bar' too. This means that changes in return arity are very
expensive since you need to track down all caller sites and revise
them. (You need to do that now too -- but if you forget something
you'd almost always get an error instead of random-and-possibly-buggy-
behavior.)

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

[Eli Barzilay]

Two days ago, Laurent wrote:
>
> I guess Scribble would be happy to use such splicing values.

It already does that with nested list structures. (And IMO it's
cleaner to use lists as containers for random pacakging of values
rather than multiple values which is more natural for distinct roles
of the different values.)

--
((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
http://barzilay.org/ Maze is Life!

[Laurent]

Matthias,

On Fri, Jul 12, 2013 at 7:36 PM, Greg Hendershott <greghen...@gmail.com> wrote:

>> Then why not return lists instead of values?
> Error checking.

Well, expecting a number and receiving a list (instead of several values) is not very different from expecting a number and receiving a string.
In both cases the user didn't read the specs correctly.

But sure, the more checks we can have, the better.

I suspect it has more to do with optimization (once again), than error checking.

> See Dybvig's paper,

Thanks for the reference, this is very informative and well written.

The performance checks seem to be quite tailored toward `values', and there are cases where what you really need to do is to handle the returned values as a list.

I read it quickly (for that first time) but the "error handling" part does not seem to be the main reason.

Greg,

 

Although nowadays we have match-define and friends.

(define-values      (a b c) (return-3-values))
(match-define  (list a b c) (return-3-element-list))

Both would error appropriately.

Probably those cases are not the ones with problems.

I guess one of the worst cases would be if you call:

(foo (bar 3))

where (bar 3) returns a list, but at some point its implementation changes and returns both a list and a string, encapsulated as multiple values in a list.
Then foo will not error correctly and it may be non-trivial to find the bug.

But of course this is a backward-incompatible change so that's the kind of problem you can expect from such a change.

A related problem is if you misread the docs and consider that `bar' returns `(list 3 3 3)' whereas it actually returns multiple values `(list "something" (list 3 3 3))'.

An intermediate solution would be that `values' returns indeed a `list', but (somehow) tagged with 'values, so that one can now it's suppose to mean a multiple return value but can still be handled with all the tools from `list'.

Thanks to all other repliers for their interesting comments.

Laurent