How does "convert" work?

[George]

Gidday

I’m trying to figure out how “convert” works.

I see lots of examples of convert in use. For instance in “~/extempore/libs/core/” 

there are several varieties of it where there is

(convert followed by one argument:

(bind-func vmean

  (lambda (buf:i32* len:i64)

    (/ (i32tod (vsum buf len)) (convert len)))) ;; from ~/extempore/libs/core/math.xtm

I guess convert can figure what type to convert into from the context.

In lots of others where there is a single argument only:

(convert (tref face 0))

(if (> z (convert 50))  …  )

(adrange (convert 0) mx (convert 1)).  ;; From  algebraic_data_types.xtm

(convert null)

(convert 0)

(convert 1)

In other examples, e.g. from ~/extempore/libs/core/math.xtm , there is a third argument.

Presumably  that’s the type you want to convert into.

(convert (pref buf i) double) ;; from ~/extempore/libs/core/math.xtm

Also:

(convert (tref-ptr str 1) i8*)

(convert (/ x 2.0) i64)

Then of course there are plenty of examples of specific conversion: i32tod, i32tof, i32toi64, i64toi32, dtof, etc.

I guess convert can handle varieties of other types defined by bind-type such as:

(bind-type Maybe <i1,!a>)  ;; from ~/extempore/tests/core/adt.xtm

I could not find any place where a macro “convert” is defined. But (using Michele Pasin’s “https://extempore.michelepasin.org/index.html”) I did find this in runtime:

Defined in:  https://github.com/digego/extempore/tree/v0.8.9/runtime/llvmti.xtm

But compiler stuff is a bit beyond me at this stage so it would be nice to have sone guidelines on how “convert” works.

Anideas welcome.

Regards

George

[Toby Gifford]

I remember wondering about this at one stage before having a small epiphany about type-inferencing, polymorphism and generic programming.  There is something similar going on with the (alloc n) command, which seems to magically know to allocate enough space for n elements of whatever type is being allocated.

Keep in mind that the extempore compiler is strongly typed: every compiled xtlang function has fully defined types for its input arguments and its return argument. However:

1. xtlang functions are polymorphic in the sense that a symbol myfunc may be bound to multiple compiled functions with different arguments (as in different argument types or different number of arguments).  Some languages like C++ call this overloading. But xtlang overloading is different to C++ in that you can overload just on the return type.  So a single symbol can be bound to two functions with the same input argument types, but different return types. This is the key to understanding what alloc and convert are doing.  When convert is being called with a single argument (the value to convert), one of the various overloads is being called depending on the return type.

2. xtlang performs type inferencing. This means that when you compile a function with bind-func you don't necessarily have to spell out explicitly the type of every argument/variable.  Keep in mind that functions are first-class citizens in xtlang (they are actually closures) that have a specific type, something like [returntype,inputtype1,inputtype2]*.  But like any variable, the type of a closure may be possible to infer from context. If xtlang can infer the type, it will.  So this again is what is happening with (convert someval). When this appears in a bind-func expression the type-inferencer will look at the context, for example is the return value of convert being used to initialize a variable with a known type?  if so then it knows which overload of convert is intended.

3. Type inferencing is not just about saving a few keystrokes.  It facilitates generic programming.  The point being that by absolving the source code from the responsibility of explicitly specifying types for everything -- one can write source code that services types not yet defined at the time of writing. Now, extempore also has a generic programming facility more like C++ templates (with templated type arguments) which is another approach to generics.  So that can be a bit confusing.

4. Type inferencing is difficult, sometimes impossible - as in there may not be sufficient information to determine the types at compile time.  Or even if there is theoretically enough information, the type inferencer may not be able to figure it out (I have come across this a few times).  This is presumably because the problem in general is very difficult. I suspect there is some sort of Turing Halting problem where the question is fundamentally intractable in some cases.  Anyway, in practice sometimes you need to give the inferencer some more clues here and there, hence the occasional use of a second argument to convert which is an explicit description of the return type required.

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/extemporelang/4f85bd58-6c5a-42ff-a8fc-5c257364daeen%40googlegroups.com.

[George]

Thanks Toby

I'm impressed by your reply. Although it will take some time me to absorb. But I think I'm getting the idea.

I suggest your outline could be included in the docs as It is something puzzling me for quite a while and I couldn't find any statements on it.

You mentioned "overloading just on return type". I wonder if the following is an example:

(sys:load "libs/core/math.xtm")

;; line 145

(bind-func vmean

(lambda (buf:i32* len:i64)

(/ (i32tod (vsum buf len)) (convert len)))). ;; Compiled:  vmean >>> [double,i32*,i64]*. RETURNS DOUBLE

(bind-func vmean

(lambda (buf:i64* len:i64)

(/ (i64tod (vsum buf len)) (convert len)))). ;; Compiled:  vmean >>> [double,i64*,i64]*

(bind-func vmean

(lambda (buf:float* len:i64)

(/ (vsum buf len) (convert len)))). ;; Compiled:  vmean >>> [float,float*,i64]*. RETURNS FLOAT

(bind-func vmean

(lambda (buf:double* len:i64)

(/ (vsum buf len) (convert len)))) ;; Compiled:  vmean >>> [double,double*,i64]*

Is it that three versions of the closure return double and the other returns a float.

Or is it that all four closures are different - returned by the same (bind-func vmean ?

Regards

George

[Toby Gifford]

Hi George, the example you give is an example of overloading, but not an example of overloading only on return type.

Each of the four vmean functions you list has a different type for the first argument (i32*, i64*, float*, double*).

To view this discussion on the web visit https://groups.google.com/d/msgid/extemporelang/05535fc6-cd7a-42d2-a9d1-3f6531d5ce03n%40googlegroups.com.

[George]

OK

So I looked widely for examples of same-named "(bind-func ..."   where the closures have the same input types but different return types.

Couldn't find anything.

Could you point me to an example please.

Regards

George

[Toby Gifford]

Hmm, well i'm not sure about bind-func examples of this in the extempore source or examples.  But as I said, both convert and alloc are examples of the concept, though I think they are defined somewhere deeper in the xtlang source (not using bind-func). However, you can readily create an example.  For example, if we ignore for the moment that alloc has these overloads, and just use it as a byte buffer allocator, we could make our own return-type-overloaded allocator like this:

(bind-func allocator:[double*,i64]*
  (λ (n)
    (let ((buffer:i8* (alloc (* 8 n))))
      (printf "allocating %ld bytes to hold %ld doubles\n" (* 8 n) n)
      (cast buffer double*))))


(bind-func allocator:[float*,i64]*
  (λ (n)
    (let ((buffer:i8* (alloc (* 4 n))))
      (printf "allocating %ld bytes to hold %ld floats\n" (* 4 n) n)
      (cast buffer float*))))

(bind-func test:[void]*
  (λ ()
    (let ((floatBuf:float* (allocator 10))
          (doubleBuf:double* (allocator 10)))
      void)))

(test)

;; expected output

Compiled:  allocator >>> [double*,i64]*
Compiled:  allocator >>> [float*,i64]*
Compiled:  test >>> [void]*
allocating 40 bytes to hold 10 floats
allocating 80 bytes to hold 10 doubles

To view this discussion on the web visit https://groups.google.com/d/msgid/extemporelang/78fe7921-00dd-42ae-8e2d-1423103c2a30n%40googlegroups.com.

[George]

OK Toby very nice and I get the same result.

Part of my confusion I think comes from having an idea that if you redefine, the new definition overrides the old.

But that seems to be incorrect. 

;; What about this from: extempore_lang.xtm

https://github.com/digego/extempore/blob/0368489bbacdb4cdd8d4da83f4be13e0516ee8da/examples/core/extempore_lang.xtm#L73

;; line 21 ;; integer literals default to 64 bit integers

(bind-func my-test-1

(lambda (a)

(* a 5))) ;; Compiled: my-test-1 >>> [i64,i64]*

;; you are free to recompile an existing closure

;; so we can change my-test-1 to

(bind-func my-test-1

(lambda (a)

(/ a 5))) ;; Compiled: my-test-1 >>> [i64,i64]*

($ (my-test-1 33))

;; note that the closures signature is still the same

;; as it was before. This is important because we are

;; NOT allowed to change an existing compiled closures

;; type signature.

;;

;; So we CANNOT do this

(bind-func my-test-1 ;; But we can do this!

(lambda (a)

(/ a 5.0))) ;; Compiled: my-test-1 >>> [double,double]*

($ (my-test-1 33.4))

;; Just remember that you are not currently allowed to redefine an

;; existing function to a new definition that requres a different type signature.

;; This is to protect against the situation where you have allready compiled

;; code which requires the current signature.

Following all that I expected your redefinition of allocator to raise an error.

Maybe that document is out of date?

Or am I missing something?

Many thanks again for your help Toby.

Regards

George

[Toby Gifford]

That comment is out of date.  At one point (when those docs were written) you couldn't overload functions at all (neither on return type nor input type) and instead used bind-poly to create a family of related functions with different argument types. But now you can overload and bind-poly is deprecated

To view this discussion on the web visit https://groups.google.com/d/msgid/extemporelang/1f413d15-65fe-4f2e-8273-6460e61c4cf6n%40googlegroups.com.

[George]

Ahaah! That explains a lot. I've seen bind-poly stuff 

and oodles of (bind func something) repeated many times with the same name but different bunches of arguments.

But I've never understood the magic behind it all. 

I feel I now have a better idea of what polymorphism is about. 💡

No doubt that extempore_lang.xtm file needs to be updated. It sure was a bit of a hurdle for this learner.

Thanks Toby

[Ben Swift]

> No doubt that extempore_lang.xtm file needs to be updated. It sure was a bit
> of a hurdle for this learner.

Yep, that's true. If you wanted to submit a cheeky pull request that at
least removed the misleading comment I'd happily accept it :)

Thanks George, and thanks Toby for the explanation.

Cheers
Ben

[George]

OK Ben & Toby, I'll have a go at it.

Something that might be a help for me is an answer to this: Is there a command that can list all the closures of the same name that are "active" at any moment?

What I'm thinking of is a list of the varieties of (bind func something) that are "live".

Hope that makes sense.

Regards

George

[George]

Ben, Toby

I haven't figured out how to do the pull request thing just yet so I am posting my suggestions for change here.

I'm sure you will be able to improve.

I've used some of Toby's explanations.

;; https://github.com/digego/extempore/blob/0368489bbacdb4cdd8d4da83f4be13e0516ee8da/examples/core/extempore_lang.xtm#L60

;; replacing lines 60 to 84 in extempore_lang.xtm
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; xtlang performs type inferencing. This means that when you compile a function with bind-func you don't necessarily have to spell out explicitly the type of every argument/variable.  Keep in mind that functions are first-class citizens in xtlang. They are actually closures that have a specific type, something like [returntype,inputtype1,inputtype2]*.

;; In the case of the next example "int-or-double-literal", the compiler has enough clues to figure out that it takes an integer input and returns an integer. So the compiler presents the closure as type [i64,i64]*.

(bind-func int-or-double-literal
  (lambda (a)
    (/ a 5)))  ;; Compiled:  int-or-double-literal >>> [i64,i64]*

(println ($ (int-or-double-literal 30)))  ;; 6

;; If another version that takes a double input is needed, you are free to recompile an existing closure with different type signature. This time the compiler presents the closure as type [double,double]*. It will take an double and return an double.

(bind-func int-or-double-literal
  (lambda (a)
    (/ a 5.000)))  ;; Compiled:  int-or-double-literal >>> [double,double]*

(println ($ (int-or-double-literal 30)))  ;; 6. The first version still works
(println ($ (int-or-double-literal 30.000)))  ;; 6.000000 The new version works too!

;; xtlang functions are polymorphic in the sense that a symbol myfunc may be bound to multiple compiled functions with different arguments - as in different argument types or different number of arguments. Also xtlang polymorphism allows single symbol to be bound to multiple functions with the same input argument types, but different return types.

;; There are now TWO versions of our "int-or-double-literal" closure.
;; The appropriate version is called depending on what input type is supplied.
;; This is an example of xtlang polymorphism. Many functions provided in xtlang have
;; several versions with the same name. Different versions might take different inputs and
;; return different outputs.

;; We can check the two versions of our "int-or-double-literal" as follows:
;; For the first:
(println (car (impc:ti:get-polyfunc-candidate-list "int-or-double-literal")))
;; #("int-or-double-literal_adhoc_W2RvdWJsZSxkb3VibGVd" (213 0 0))

;; For the second:
(println (cadr (impc:ti:get-polyfunc-candidate-list "int-or-double-literal")))
;; #("int-or-double-literal_adhoc_W2k2NCxpNjRd" (213 2 2))

;; Notice there are different addresses attached to the closure names.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Regards George

[Ben Swift]

Hey George, thanks. I can update that for you.

Re: the "pull request" thing; if you've got a GitHub account you can actually
just use the "Edit this file" button in the web interface and it will package it
up as a pull request for you. But sending it through as email is ok as well.

[George]

Ben

I found a nicer way to pretty print the poly function list using (impc:ti:polyfunc-pretty-print "int-or-double-literal")

So I made a change to the end of my suggested changes;

https://gist.github.com/georgejwright/b5b7c4b14367b6d0856fac3cac037b36

;; https://github.com/digego/extempore/blob/0368489bbacdb4cdd8d4da83f4be13e0516ee8da/examples/core/extempore_lang.xtm#L60

;; replacing lines 60 to 84 in extempore_lang.xtm

;; xtlang performs type inferencing. This means that when you compile a function with bind-func you don't necessarily have to spell out explicitly the type of every argument/variable.  Keep in mind that functions are first-class citizens in xtlang. They are actually closures that have a specific type, something like [returntype,inputtype1,inputtype2]*.

;; In the case of the next example "int-or-double-literal", the compiler has enough clues to figure out that it takes an integer input and returns an integer. So the compiler presents the closure as type [i64,i64]*.

(bind-func int-or-double-literal
  (lambda (a)
    (/ a 5)))  ;; Compiled:  int-or-double-literal >>> [i64,i64]*

(println ($ (int-or-double-literal 30)))  ;; 6

;; If another version that takes a double input is needed, you are free to recompile an existing closure with different type signature. This time the compiler presents the closure as type [double,double]*. It will take a double and return a double.

(bind-func int-or-double-literal
  (lambda (a)
    (/ a 5.000)))  ;; Compiled:  int-or-double-literal >>> [double,double]*

(println ($ (int-or-double-literal 30)))  ;; 6. The first version still works
(println ($ (int-or-double-literal 30.000)))  ;; 6.000000 The new version works too!

;; xtlang functions are polymorphic in the sense that a symbol myfunc may be bound to multiple compiled functions with different arguments - as in different argument types or different number of arguments. Also xtlang polymorphism allows single symbol to be bound to multiple functions with the same input argument types, but different return types.

;; There are now TWO versions of our "int-or-double-literal" closure.
;; The appropriate version is called depending on what input type is supplied.
;; This is an example of xtlang polymorphism. Many functions provided in xtlang have
;; several versions with the same name. Different versions might take different inputs and
;; return different outputs.

;; We can check the two versions of our "int-or-double-literal" as follows:

(impc:ti:polyfunc-pretty-print "int-or-double-literal")

;; This gets printed out to log:
;; Polymorphic options for int-or-double-literal
;;   int-or-double-literal_adhoc_W2RvdWJsZSxkb3VibGVd:[double,double]*
;;   int-or-double-literal_adhoc_W2k2NCxpNjRd:[i64,i64]*

;; Notice there are different addresses attached to the closure names.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Regards

George

[Ben Swift]

Hi George

Great, thanks - will update (sorry I haven't applied your earlier
suggestion yet, I'll get on that today).

Re: the polyfunc-pretty-print thing, there's no nice wrapper for that
currently, but we could well add one.

Cheers,
Ben