Dealing with structs (not pointer-to-struct) in C-libraries

[Toby Gifford]

OK, here's my potted understanding of stack vs heap allocated variables

---------------------

C 
// one ThingaMeBob's worth of memory is allocated on the stack,
// x is (under the hood) bound to the address of this memory in the stack
ThingaMeBob x;

x.a = 1; x.b = 3; x.c = 5; x.d = 7;

// one ThingaMeBob's worth of memory is allocated on the heap, and one address's worth of memory is allocated on the stack
// px is (under the hood) bound to the address of the memory in the stack (the value of which is an address in the heap)
ThingaMeBob *px = (ThingaMeBob *) malloc (sizeof(ThingaMeBob));

px->a = 1; px->b = 3; px->c = 5; px->d = 7;

C++
// one ThingaMeBob's worth of memory is allocated on the stack,
// x is (under the hood) bound to the address of this memory in the stack
ThingaMeBob x = ThingaMeBob(1,3,5,7);         

// one ThingaMeBob's worth of memory is allocated on the heap, and one address's worth of memory is allocated on the stack
// px is (under the hood) bound to the address of the memory in the stack (the value of which is an address in the heap)
ThingaMeBob *px = new ThingaMeBob(1,3,5,7);

xtlang

// one ThingaMeBob's worth of memory is allocated on the current zone, and one address's worth of memory is allocated on the stack
// px is (under the hood) bound to the address of the memory in the stack (the value of which is an address in the current zone)
(let ((px:ThingaMeBob* (ThingaMeBob 1 3 5 7))))

-------------------

In xtlang one doesn't allocate tuples on the stack. Is it possible though? something like:

(let ((x:ThingaMeBob (convert 0))))

The reason I ask is because some C-libraries use stack allocated structs, and i'm not sure how to deal with them. In particular, I want to use Hue/Saturation/Brightness colour space in nanovg.  The nanovg.xtm wrapper has the following prototype:
(bind-lib libnanovg nvgHSLA [NVGcolor,float,float,float,i8]*)

so I can create a NVGcolour from HSLA coordinates, but because it is a tuple (rather than a pointer to a tuple) I can't pass it into
(bind-lib libnanovg _nvgFillColor [void,NVGcontext*,NVGcolor*]*)   // error

nor can I extract its contents with

(tref my-non-pointer-colour 0)   // error

Because there is no address-of operator in xtlang, I can't figure out how to do anything at all with the return value from nvgHSLA. Can I cast it somehow? Something like:

(cast my-non-pointer-colour float*).

I did try that but it didn't work. What is the internal layout of tuples anyway?

[Ben Swift]

Hi Toblerone

This is a bit tricky, in fact it's the reason that the _nvg* functions
exists---as complementary to the plain nvg* equivalents which take the
structs by value.

The short answer is you can't, as far as I know. The actual struct
passing calling convention is not part of the C standard, so different
compilers/abis do it differently (on amd64 a lot of the things are
passed in registers for smallish structs). So the problem isn't passing
tuples by value per. se., it's passing tuples by value to functions
which are actually expecting C structs from code generated by another
compiler.

The best way around it, in this case, is probably to refactor the nanovg
library to only pass colours around by reference. The _nvg* versions
are a bit of a hack, but since I'm maintaining my own fork of nanovg
anyway we have the opportunity to re-write the api a bit. Perhaps
that's worth having a look at? Repo is here:

https://github.com/extemporelang/nanovg

Cheers,
Ben

Toby Gifford <toby.g...@gmail.com> writes:

> OK, here's my potted understanding of stack vs heap allocated variables
>
> ---------------------

> *C*

> // one ThingaMeBob's worth of memory is allocated on the stack,
> // x is (under the hood) bound to the address of this memory in the stack
> ThingaMeBob x;
> x.a = 1; x.b = 3; x.c = 5; x.d = 7;
>
> // one ThingaMeBob's worth of memory is allocated on the heap, and one
> address's worth of memory is allocated on the stack
> // px is (under the hood) bound to the address of the memory in the stack
> (the value of which is an address in the heap)
> ThingaMeBob *px = (ThingaMeBob *) malloc (sizeof(ThingaMeBob));
> px->a = 1; px->b = 3; px->c = 5; px->d = 7;
>
>

> *C*++

> // one ThingaMeBob's worth of memory is allocated on the stack,
> // x is (under the hood) bound to the address of this memory in the stack
> ThingaMeBob x = ThingaMeBob(1,3,5,7);
>
> // one ThingaMeBob's worth of memory is allocated on the heap, and one
> address's worth of memory is allocated on the stack
> // px is (under the hood) bound to the address of the memory in the stack
> (the value of which is an address in the heap)
> ThingaMeBob *px = new ThingaMeBob(1,3,5,7);
>
>

> *xtlang*

[Toby Gifford]

Aha!   Hmm.  OK, will fork your nanovg fork.

--
You received this message because you are subscribed to the Google Groups "Extempore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to extemporelan...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Toby Gifford]

So if I fiddle around with the nanovg C code, what's the best way to build it / integrate it with the current extempore build system?

On Mon, Apr 25, 2016 at 11:24 AM, Ben Swift <b...@benswift.me> wrote:

[Ben Swift]

Yep, PRs welcome there. Just remember that we might need to update the
current examples as well.

Cheers

[Andrew Sorensen]

Hey Toby,

Sorry, this email got away from me a little ;)

I think Ben answered your question, but I just wanted to clarify a few things about value tuples.

For every ThingaMeBob constructor that returns a ref, there is also a ThingaMeBob_val constructor that returns a value (you might remember a list email about vals and refs a few weeks back).  Note in the example below that copying a 'value tuple' into a 'reference tuple', is fine.  So I'm not sure why you had trouble with (tref my-non-pointer-colour 0), which should work fine?  What you can not do is tset! into a 'value' tuple - all value tuples (named or otherwise) are immutable.  Of course you can easily just create a *new* value tuple instead of trying to tset! into an existing one.

(bind-type ThingaMeBob <float,float,float,float>)

;; Have a look at the return type

(bind-func test_bobs

  (lambda ()

    (let ((a (ThingaMeBob 1.0 2.0 3.0 4.0))

          (b (ThingaMeBob_val 1.0 2.0 3.0 4.0)))

      (tuple a b))))

(bind-func use_test_bobs

  (lambda ()

    (let ((ret (test_bobs))

          (a (tref ret 0))

          (b (tref ret 1))

          ;; make c from b

          (c (ThingaMeBob (tref b 0) (tref b 1) (tref b 2) (tref b 3))))

      ;; tref works fine for all of the above

      (println (tref a 0) (tref b 1) (tref c 2))

      void)))

;; should print 1.0 2.0 3.0

($ (use_test_bobs))

Just on the matter of C struct value passing, there should be nothing stopping your code from compiling properly against the C lib bindings that you discussed.   If you were having trouble compiling then there was something else going on.

The problem with the struct 'value' calling conventions that Ben correctly raised will not stop xtlang compilation - unfortunately you'll only know there is a problem when either (a) your code crashes or (b) you notice that you are getting crud data (yes we should force no struct values in bind-lib - someone should raise a github issue ;-) .  A good way to think about the problem is that it is basically the same issue as using a library compiled with CPU extensions that you don't have.  The original compiler used Jessica's fancy new CPU to pass values using 256bit SIMD registers - that you don't have.  You successfully link against Jessica's library only to have the linked function call crash (or otherwise screw up your stack etc.) at runtime.  

There is a bunch of lowering that xtlang *should* do to try to better support struct value passing from C.  But the calling conventions are different between Unix/Windows, and between 32bit/64bit and between x86/ARM.  And even then, there are still no guarantees that your compilers are going to play nice.  In short struct value passing is a complete pain, and so to be honest we have been ignoring it for the most part (i.e. passing value structs between xtlang and C is off limits).  Instead, as Ben mentioned you should pass your structs by ref - and most sane C libraries do this, for exactly this reason.   Unfortunately occasionally this means refactoring C libraries a little - certainly a pain :(

Now, having said all this, you *are* allowed to pass struct values freely between xtlang calls - no problem. This is only an issue when crossing the C divide - in either direction.  And just to reiterate C struct references are no problem - in either direction.

Just one more xtlang code example that you might not have considered previously.  Remembering that named types are ThingaMeBob (returns ref) and ThingaMeBob (returns value), and that tuple (returns value) and tuple_ref (returns ref) - I know they are opposites, but they represent the *common* case.

(bind-type MyStuff <<float,float,float>,<i64,float,double>>)

(bind-func print:[void,MyStuff*]*

  (lambda (obj)

    (let ((o1 (tref obj 0))

          (o2 (tref obj 1)))

      (printout "<MyStuff:"

                (tref o1 0) " " (tref o1 1) " " (tref o1 2) ","

                (tref o2 0) " " (tref o2 1) " " (tref o2 2) ">"))))

(bind-func test_my_stuff

  (lambda ()

    (let ((a (MyStuff (tuple 1.0 2.0 3.0) (tuple 1 2.0 3.0))))

      (println 'before: a)

      (tset! a 0 (tuple 3.0 2.0 1.0))

      (tset! a 1 (tuple 3 2.0 1.0))      

      (println 'after: a)

      void)))

($ (test_my_stuff))

Just as a general note while I'm here, using constructors (i.e. (MyStuff ...) ) is definitely the preferred practice these days (also note how it helps out with type inference). As a general rule of thumb, if you find yourself using alloc/halloc/salloc directly, then there is probably a better way to do what you're doing.  Unfortunately most of the example and library code does not abide by this, but we will hopefully find time to clean things up bit by bit.

Cheers,

Andrew.