Porting project from C++

[Jason Levine]

Hi XtmLand,

I've finally started to port my t-SNE sampler from C++ to Extempore.  I've got a minimal version working and wow it sounds tight, what an improvement! 

However, I keep running into  Scheme <-> xtlang interop problems and I'm realizing there 3 different ways to port my project:

1) Scheme for the high level scheduling, xtlang for the number crunching. This is the ideal, and what I am currently attempting but I keep running into Scheme/xtlang interop problems.  I've found some work arounds but I keep hitting obstacles.  For instance, right now I'm trying to get an xtlang function to return a list of sample indices and a list of sample velocities.  I had high hopes that the xtlang list type from adt.xtm would transfer to a Scheme list, but it doesn't seem to work. 

This made me consider "going native"

2) Port everything to xtlang. This gets rid of all interop problems and makes syncing graphics and sound easy. But!  In the comments of scheduler.xtm it states that ";; This code is here as a jump off point. ;; it is NOT tested, nor ready for prime time.".  So I'm worried about relying on it for my two upcoming shows in January.  Can anyone comment on why the native scheduler is not ready for prime time? Or maybe that comment is obsolete?

Because this made me consider...

3) Porting everything to Scheme.  The worry here is that Scheme won't be able to run through 1000 points and find the N nearest neighbours in time to play the notes at their scheduled time.  That's the whole point of having the xtlang, right?  Either way, I'll need xtlang for visualizing what's going on the the interop will still happen somewhere.

Any advice would be greatly appreciated,

Happy Holidays!

--

Jason Levine

new media performer + creative coder

https://www.behance.net/jasonlevine

[Andrew Sorensen]

Hi Jason,

As you say, you have many options.  Although scheme doesn't *understand* xtlang data it can (and does) reference it as an opaque C pointer if the type returned from an xtlang function is a pointer type. So an xtlang list does get returned to scheme and can be bound to a scheme variable - although only as an opaque C pointer that scheme doesn't really understand (i.e. void*).

There are a number of ways to handle this, but below I show one possible solution for your particular case.  Here I assume you have a list of i64 index values but this basic idea can be extended to xtlang lists of any type (including lists of composite types (i.e. lists of lists) with additional conversion as required). 

The *trick* is that you need to provide xtlang functions which scheme can call with (potentially) opaque C pointer variables.  Scheme doesn't know what the opaque C pointers contain but your xtlang functions obviously do (i.e. they have type signatures!). Significantly any opaque (i.e. void*) pointers in a scheme->xtlang call get cast to the required xtlang pointer type.  So clearly there is no type safety here, in the sense that when you pass an opaque C pointer (a void*) from scheme to xtlang, the xtlang compiler has no choice but to trust that you are passing it the correct type.   There are various ways to mitigate this type safety risk, but I'll leave that for another time.  For the moment the example below should get you going.

Also important to remember is that scheme can only call monomorphic xtlang functions at the moment - not polymorphic functions.  Any scheme calls to polymorphic xtlang functions will result in the error message "Ambiguous or unavailable xtlang wrapper: "AA""

---------------------------------

(sys:load "libs/core/adt.xtm")

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;

;; XTLang world

;;

;; we will need explicit monomorphic calls

;; for scheme to call.

;;

(bind-func list-i64-ref

  (lambda (lst:List{i64}* idx:i64)

    (nth lst idx)))

(bind-func list-i64-length

  (lambda (lst:List{i64}*)

    (length lst)))

;; generate a List{i64}* for testing

(bind-func test-list-builder

  (lambda (x:i64)

    (range x)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;

;; Scheme world

;;

;; convert xtlang list to scheme list

(define list-i64-convert

  (lambda (lst)

    (map (lambda (x)

            (list-i64-ref lst x))

         (range (list-i64-length lst)))))

 

;; bind xtlang list to scheme variable

(define AVar (test-list-builder 10))

;; print avar (shows opaque pointer)

(println AVar)

;; convert xtlang list to scheme list

(define BVar (list-i64-convert AVar))

;; print bvar (normal scheme list)

(println BVar)

(println (reverse BVar))

(println (map (lambda (x) (* x x)) BVar))

;; etc..

;; or as a single expression

(println (list-i64-convert (test-list-builder 10)))

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelang+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Sorensen]

p.s. there shouldn't be any problem with scheduler.xtm - it works, although I think I'm the only one using it, so not heavily tested.

[Jason Levine]

The one problem I've seen with native scheduling is that it takes a while to compile so your changes don't take effect immediately, this is especially problematic when you want to stop a loop suddenly. 

[Jason Levine]

Hi Andrew,

I got your code working and I think I get the general idea: Scheme can understand basic types returned from xtlang functions, for compound types you need an scheme wrapped xtlang helper function that gives you each component of the compound type as a basic type.  Is that right?

[Andrew Sorensen]

Correct, we usually refer to these as "primitive types" (rather than basic).

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Jason Levine]

Hey Andrew,

Do you really use the native scheduler in live performances?  Nearly everything I try to do results in a type error, and it takes 3 or 4 seconds after I evaluate before the temporal recursion recompiles and the changes take effect.

For reference, here is my native temporal recursion:

```(bind-func beat_loop

  (lambda (beat:Rational)

     (if (= 1/4 (% beat 2/1))

      (play_tSNE (dtof (cosr 0.1 0.1 1/4)) 0.3 5 300. 1/1))

      

     (play_tSNE (dtof (cosr 0.9 0.1 (random (list 1/2 1/8 1/3)))) 0.4 5 (dtof (cosr 200. 100. (random (list 3/2)))) 1/4)

    void))````

[Toby Gifford]

I use the native scheduler (without issue so far), though i'm not doing typical live coding. I usually expose some control variables (for example stop/start) from the xtlang closure and manipulate them from scheme using dot syntax.

[Jason Levine]

Interesting! That solves the compilation delay for starting and stopping, and is actually a nicer interface than the classic (define loop) method which doesn't work in xtlang. However, you'll still have the compilation delay whenever you make changes to the code in the body of the temporal recursion. I guess since you are not livecoding, the compilation delay is irrelevant, for me it is a dealbreaker.  I will add that one advantage of livecoding in xtlang is that Scheme will stop a temporal recursion if you make a typo and then eval, while xtlang will throw a compiler error and keep the loop going.

To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Extempore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jason Levine

new media performer + creative coder

https://www.behance.net/jasonlevine

--

Jason Levine

new media performer + creative coder

https://www.behance.net/jasonlevine

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--

Jason Levine

new media performer + creative coder

https://www.behance.net/jasonlevine

--
You received this message because you are subscribed to the Google Groups "Extempore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.

[Andrew Sorensen]

Yep, the xtlang compiler is known to be too slow to be practicable at the moment – for either live-coding or offline coding (building and loading libraries for example!).   More specifically, type inference is too slow, which has become increasingly problematic as xtlang’s type system has become increasingly complex.  In short, this is the primary reason why bootstrapping the xtlang compiler is considered such a high priority.