How do I represent a convenient two-step allocation using ffi/unsafe?

[Sage Gerard]

Consider the following C++ that calls a function once to learn how much it should allocate for a vector, and again to populate said vector.

uint32_t count;

vkEnumerateInstanceLayerProperties(&count, NULL);

std::vector<VkLayerProperties> properties(count);

vkEnumerateInstanceLayerProperties(&count, properties.data());

Below is my generated signature so far. The docs for _list shows similar examples, but they assume either the length of the vector is known in advance.

How would I make a custom function type to manage both o1 and o0 values with the garbage collector, call the function twice as described, and return the value for o1 in a single call?

(define-vulkan vkEnumerateInstanceLayerProperties

  (_fun (o0 : (_ptr o _uint32_t))

        (o1 : (_ptr o _VkLayerProperties))

        -> (r : _VkResult)

        -> (begin (check-vkResult r (quote vkEnumerateInstanceLayerProperties))

                  (values o0 o1))))

~slg

[Matthew Flatt]

I don't think `ffi/unsafe` can generate that kind of wrapper for you.
I'd write something like this:

(define-vulkan vkEnumerateInstanceLayerProperties/private
(_fun (o0 : (_ptr io _uint32_t))
_pointer

-> (r : _VkResult)
-> (begin

(check-vkResult r 'vkEnumerateInstanceLayerProperties)
o0))
#:c-id vkEnumerateInstanceLayerProperties)

(define (vkEnumerateInstanceLayerProperties)
(define len (vkEnumerateInstanceLayerProperties/private 0 #f))
(define props (malloc _VkLayerProperties len))
(define new-len (vkEnumerateInstanceLayerProperties/private len props))
(values new-len props))

At Sun, 20 Oct 2019 03:47:52 +0000, Sage Gerard wrote:
> Consider the following C++ that calls a function once to learn how much it
> should allocate for a vector, and again to populate said vector.
>
> uint32_t count;
>
> vkEnumerateInstanceLayerProperties(&count, NULL);
>
> std::vector<VkLayerProperties> properties(count);
>
> vkEnumerateInstanceLayerProperties(&count, properties.data());
>
> Below is my generated signature so far. The docs for

> [_list](https://docs.racket-lang.org/foreign/foreign_procedures.html?q=_fun#%28
> form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29) shows similar examples,

> but they assume either the length of the vector is known in advance.
>
> How would I make a custom function type to manage both o1 and o0 values with
> the garbage collector, call the function twice as described, and return the
> value for o1 in a single call?
>
> (define-vulkan vkEnumerateInstanceLayerProperties
> (_fun (o0 : (_ptr o _uint32_t))
> (o1 : (_ptr o _VkLayerProperties))
> -> (r : _VkResult)
> -> (begin (check-vkResult r (quote vkEnumerateInstanceLayerProperties))
> (values o0 o1))))
>
> ~slg
>

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/Vo2OHZYjsSlsnJojlfmw5yolaYJWQzql
> unk7S24hrnFb4Lk1jz0kqe2RCGM_aPFXxeVQfRqDPM2p7GKeN_drxRoi2Q9ZGu2srPhAme7LCAk%3D%
> 40sagegerard.com.

[Sage Gerard]

Hi Matthew,

That makes sense, thanks. I have a follow-up question, and forgive me for the 180-degree turn.

I realized I was wrong to convey that I want the wrapper functions to help with allocations. My situation is that I have 346 generated procedure bindings for Vulkan, and 1342 bindings total. The more "helpful" the bindings I generate, the more differences I have to document, the more corner cases I have to support in the code generator, and the higher the learning curve for my users. I think it would be wiser to keep the bindings entirely consistent with their C equivalents and hand-write helpers on top to capture common patterns.

So if I take the `vkEnumerateInstanceLayerProperties/private` you showed me and change the (_ptr io _uint32_t) to (_cpointer _uint32_t), will it have any other noticeable usability differences for someone thinking like a C programmer? Do I have to use _ptr at all to do what I intend to do?

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, October 20, 2019 9:10 AM, Matthew Flatt mfl...@cs.utah.edu wrote:

> I don't think `ffi/unsafe` can generate that kind of wrapper for you.
> I'd write something like this:
> (define-vulkan vkEnumerateInstanceLayerProperties/private
> (_fun (o0 : (_ptr io _uint32_t))
> _pointer
> -> (r : _VkResult)
>
> -> (begin
>
> (check-vkResult r 'vkEnumerateInstanceLayerProperties)
> o0))
>
>
> #:c-id vkEnumerateInstanceLayerProperties)
> (define (vkEnumerateInstanceLayerProperties)
> (define len (vkEnumerateInstanceLayerProperties/private 0 #f))
> (define props (malloc _VkLayerProperties len))
> (define new-len (vkEnumerateInstanceLayerProperties/private len props))
> (values new-len props))
> At Sun, 20 Oct 2019 03:47:52 +0000, Sage Gerard wrote:
>
> > Consider the following C++ that calls a function once to learn how much it
> > should allocate for a vector, and again to populate said vector.
> > uint32_t count;
> > vkEnumerateInstanceLayerProperties(&count, NULL);
> > std::vector<VkLayerProperties> properties(count);
> > vkEnumerateInstanceLayerProperties(&count, properties.data());
> > Below is my generated signature so far. The docs for

> > [_list](https://docs.racket-lang.org/foreign/foreign_procedures.html?q=fun#(
> > form.%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29) shows similar examples,but they assume either the length of the vector is known in advance.

> > How would I make a custom function type to manage both o1 and o0 values with
> > the garbage collector, call the function twice as described, and return the
> > value for o1 in a single call?
> > (define-vulkan vkEnumerateInstanceLayerProperties
> > (_fun (o0 : (_ptr o _uint32_t))
> > (o1 : (_ptr o _VkLayerProperties))
> > -> (r : _VkResult)
> > -> (begin (check-vkResult r (quote vkEnumerateInstanceLayerProperties))
> > (values o0 o1))))
> > ~slg
> >

> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[Matthew Flatt]

At Sun, 20 Oct 2019 23:07:58 +0000, Sage Gerard wrote:
> So if I take the `vkEnumerateInstanceLayerProperties/private` you showed me
> and change the (_ptr io _uint32_t) to (_cpointer _uint32_t), will it have any
> other noticeable usability differences for someone thinking like a C
> programmer? Do I have to use _ptr at all to do what I intend to do?

You don't have to use `_ptr`, but `(_cpointer _uint32_t)` is awkward,
because it requires that the allocated pointer is tagged using
`_uint32_t`. Then, instead of just

(malloc _uin32_t)

a client would have to write

(let ([p (malloc _uint32_t)])
(cpointer-push-tag! p _uint32_t)
p)

To expose the raw API without `_ptr`, I'd just use `_pointer`.

If possible, using `_ptr` is the right way to expose this function
without trying to provide a higher-level interface:

(define-vulkan vkEnumerateInstanceLayerProperties

(_fun (o0 : (_ptr io _uint32_t))
_pointer
-> (r : _VkResult)
-> (begin
(check-vkResult r 'vkEnumerateInstanceLayerProperties)

o0)))

I don't know if the information you're working with lends itself to
generating that kind of view, though, which would mean describing the
first argument is an in--out `uint32_t`.

[Sage Gerard]

Thank you for taking the time. I am following you well enough, but I'm having

trouble applying your advice. I'll use a different function to hopefully simplify the discussion.

typedef VkInstance_T* VkInstance;

VkInstance instance;

vkCreateInstance(&createInfo, NULL, &instance) // assume createInfo is defined

VkInstance_T is private. It's allocated in the implementation of vkCreateInstance

as a side-effect. In Racket, I want the code to look and behave as much like that as possible,

even if it means the instance is not allocated for me, or returned from the wrapper

function.

As per your advice I changed the wrapper procedure to use (_ptr io ___). However, I did not bind it to an identifier.

I'm hoping to maintain the same arity as the C function, declare the instance in client code, and have it populated

by side-effect as in the C.

(define _VkInstance (_cpointer 'VkInstance_T))

(define-vulkan vkCreateInstance

   (_fun _VkInstanceCreateInfo-pointer/null

         _VkAllocationCallbacks-pointer/null

         (_ptr io _VkInstance)

         -> (r : _VkResult)

         -> (begin (check-vkResult r (quote vkCreateInstance)))))

And, based on your email, I tried allocating the pointer and assuming that the wrapper procedure

would use the address of the pointer for me (That's what I guessed was happening when you passed `0` across (_ptr io _uint32_t) earlier).

(define instance (malloc _VkInstance))

(vkCreateInstance instcreateinfo #f instance)

But this produces "VkInstance_T->C: argument is not non-null `VkInstance_T' pointer".

I reproduce that exact error even if I replace (_ptr io _VkInstance) with just _pointer, and use cpointer-push-tag!

What do I have to do to make this example work? If I can just understand that, I can bring that back to the prior example.

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Sunday, October 20, 2019 7:24 PM, Matthew Flatt mfl...@cs.utah.edu wrote:

At Sun, 20 Oct 2019 23:07:58 +0000, Sage Gerard wrote:

So if I take the vkEnumerateInstanceLayerProperties/private you showed me

and change the (_ptr io _uint32_t) to (_cpointer _uint32_t), will it have any

other noticeable usability differences for someone thinking like a C

programmer? Do I have to use _ptr at all to do what I intend to do?

You don't have to use_ptr, but (_cpointer _uint32_t) is awkward,

because it requires that the allocated pointer is tagged using

_uint32_t. Then, instead of just

(malloc _uin32_t)

a client would have to write

(let ([p (malloc _uint32_t)])

(cpointer-push-tag! p _uint32_t)

p)

To expose the raw API without _ptr, I'd just use _pointer.

If possible, using _ptr is the right way to expose this function

without trying to provide a higher-level interface:

(define-vulkan vkEnumerateInstanceLayerProperties

(_fun (o0 : (_ptr io _uint32_t))

_pointer

-> (r : _VkResult)

     -> (begin

          (check-vkResult r 'vkEnumerateInstanceLayerProperties)
          o0)))

I don't know if the information you're working with lends itself to

generating that kind of view, though, which would mean describing the

first argument is an in--out uint32_t.

[Matthew Flatt]

The short answer is: `(_ptr io _VkInstance)` does not mean that a
pointer to a `_VkInstance` happens on the Racket side. It means that a
`_VkInstance` (not a pointer to a `_VkInstance`) happens on the Racket
side, and a pointer to a `_VkInstance` happens on the C side.

Taking it from the top:

At Mon, 21 Oct 2019 19:07:40 +0000, Sage Gerard wrote:
> (define _VkInstance (_cpointer 'VkInstance_T))

So far, so good. `VkInstance` is defined in Vulkan to be a pointer to
unspecified memory, so this makes sense.

To be clear,

(malloc _VkInstance)

does not create a `_VkInstance`. It creates a pointer to a
`_VkInstance` (i.e., a pointer to a pointer). That could be what you
want, depending on how `vkCreateInstance` is defined. Specifically, to
make this work:

(define instance-ptr (malloc _VkInstance))

(vkCreateInstance instcreateinfo #f instance-ptr)

(define created-instance (ptr-ref instance-ptr _VkInstance))

Define `vkCreateInstance` like this:

(define-vulkan vkCreateInstance
(_fun _VkInstanceCreateInfo-pointer/null
_VkAllocationCallbacks-pointer/null

_pointer ; to a _VkInstance
-> (r : _VkResult)
-> (check-vkResult r 'vkCreateInstance)))

I've written the third type as just `_pointer` because there's not a
simple way to say "pointer to a `_VkInstance`". There's no way to write
that because it would offer no more checking than just `_pointer`. It
might be a little better to write the comment as a `define`:

(define _VkInstance-pointer _pointer)

(define-vulkan vkCreateInstance
(_fun _VkInstanceCreateInfo-pointer/null
_VkAllocationCallbacks-pointer/null

_VkInstance-pointer
-> (r : _VkResult)
-> (check-vkResult r 'vkCreateInstance)))

You could go further and write

;; pointer to a `_VkInstance`:
(define _VkInstance-pointer (_cpointer 'VkInstance_T*))

(define-vulkan vkCreateInstance
(_fun _VkInstanceCreateInfo-pointer/null
_VkAllocationCallbacks-pointer/null

_VkInstance-pointer
-> (r : _VkResult)
-> (check-vkResult r 'vkCreateInstance)))

(define instance-ptr (malloc _VkInstance))
(cpointer-push-tag! instance-ptr 'VkInstance_T*)
(vkCreateInstance instcreateinfo #f instance-ptr)
(define created-instance (ptr-ref instance-ptr _VkInstance))

That's probably more trouble than it's worth, and the checking benefits
depend on `malloc` and `cpointer-push-tag!` begin used consistently
together. Declaring `_VkInstance-pointer` could make sense if the
library provides a `make-VkInstance-pointer` function to combine the
`malloc` and `cpointer-push-tag!` steps, helping to make sure they're
always consistent.


Instead of `_pointer` or `_VkInstance-pointer`, you've written

(_ptr io _VkInstance)

As an argument type, that means:

* The caller supplies a `_VkInstance` (not a pointer to a `VkInstance`).

* Fresh memory for a `_VkInstance` is allocated.

* The given `_VkInstance` is copied into the fresh memory.

* The address of the fresh memory is passed to the foreign function as
a pointer to a `_VkInstance`.

So, you're seeing an error because `(malloc _VkInstance)` does not
create a `_VkInstance`. Anyway, this is not what you want, because
`vkCreateInstance` would set the pointer at the fresh memory, which is
then discarded.

Generally, a `(_ptr io ...)` argument without a name binding doesn't
make any sense --- unless you really want to ignore the result (which
you don't, in this case).


It's pretty typical for "middleware" layers like the FFI to take into
account the in--out behavior of function arguments, and that's why many
C prototype descriptions are augmented with that information. The
`ffi/unsafe` layer is meant to be used that way, too, so the
translation that works best for `ffi/unsafe` is

(define-vulkan vkCreateInstance
(_fun _VkInstanceCreateInfo-pointer/null
_VkAllocationCallbacks-pointer/null

(o : (_ptr o _VkInstance))

-> (r : _VkResult)
-> (begin

(check-vkResult r 'vkCreateInstance)
o)))

because the third argument is an out argument.

If you don't make to make these in--out distinctions, then the first or
second definition of `vkCreateInstance` above is the most practical
choice.

[Sage Gerard]

Hi Matthew,

Wow! Thank you for the detailed response, and the saintly patience behind it.

In following your email, this is the working result after using '_pointer' in the function signature:

(define instance-ptr (malloc _VkInstance))
(vkCreateInstance instcreateinfo #f instance-ptr)

(define instance (ptr-ref instance-ptr _VkInstance))
(vkDestroyInstance instance #f)


My confusion did indeed stem from not understanding how pointer indirection was modeled in Racket. I was distracted by questions like "What's the address-of operator equivalent?" while staring at the arguments to the wrapper procedures.

I still prefer this form for the unsafe layer because of its faithful representation of Vulkan. This is not at all to discount your advice for using _ptr as the best working translation. The spec supports a view where I can generate a second layer of helpful wrapper procedures that use the first, as opposed to generating the most tailored wrapper procedure at the first oppurtunity.

Thank you again. I know you didn't have to take the time, so it means a lot to me that you did. Hopefully I won't end up confused again. :)

~slg

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 22, 2019 9:28 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

&gt; The short answer is: `(_ptr io _VkInstance)` does not mean that a
&gt; pointer to a `_VkInstance` happens on the Racket side. It means that a
&gt; `_VkInstance` (not a pointer to a `_VkInstance`) happens on the Racket
&gt; side, and a pointer to a `_VkInstance` happens on the C side.
&gt;
&gt; Taking it from the top:
&gt;
&gt; At Mon, 21 Oct 2019 19:07:40 +0000, Sage Gerard wrote:
&gt;
&gt; &gt; (define _VkInstance (_cpointer 'VkInstance_T))
&gt;
&gt; So far, so good.`VkInstance` is defined in Vulkan to be a pointer to
&gt; unspecified memory, so this makes sense.
&gt;
&gt; To be clear,
&gt;
&gt; (malloc _VkInstance)
&gt;
&gt; does not create a `_VkInstance`. It creates a pointer to a
&gt; `_VkInstance` (i.e., a pointer to a pointer). That could be what you
&gt; want, depending on how `vkCreateInstance` is defined. Specifically, to
&gt; make this work:
&gt;
&gt; (define instance-ptr (malloc _VkInstance))
&gt;
&gt; (vkCreateInstance instcreateinfo #f instance-ptr)
&gt;
&gt; (define created-instance (ptr-ref instance-ptr _VkInstance))
&gt;
&gt; Define `vkCreateInstance` like this:
&gt;
&gt; (define-vulkan vkCreateInstance
&gt; (_fun _VkInstanceCreateInfo-pointer/null
&gt; _VkAllocationCallbacks-pointer/null
&gt; _pointer ; to a _VkInstance
&gt; -&gt; (r : _VkResult)
&gt;
&gt; -&gt; (check-vkResult r 'vkCreateInstance)))
&gt;
&gt;
&gt; I've written the third type as just`_pointer` because there's not a
&gt; simple way to say "pointer to a `_VkInstance`". There's no way to write
&gt; that because it would offer no more checking than just `_pointer`. It
&gt; might be a little better to write the comment as a `define`:
&gt;
&gt; (define _VkInstance-pointer _pointer)
&gt;
&gt; (define-vulkan vkCreateInstance
&gt; (_fun _VkInstanceCreateInfo-pointer/null
&gt; _VkAllocationCallbacks-pointer/null
&gt; _VkInstance-pointer
&gt; -&gt; (r : _VkResult)
&gt;
&gt; -&gt; (check-vkResult r 'vkCreateInstance)))
&gt;
&gt;
&gt; You could go further and write
&gt;
&gt; ;; pointer to a `_VkInstance`:
&gt; (define _VkInstance-pointer (_cpointer 'VkInstance_T*))
&gt;
&gt; (define-vulkan vkCreateInstance
&gt; (_fun _VkInstanceCreateInfo-pointer/null
&gt; _VkAllocationCallbacks-pointer/null
&gt; _VkInstance-pointer
&gt; -&gt; (r : _VkResult)
&gt;
&gt; -&gt; (check-vkResult r 'vkCreateInstance)))
&gt;
&gt;
&gt; (define instance-ptr (malloc _VkInstance))
&gt; (cpointer-push-tag! instance-ptr 'VkInstance_T*)
&gt; (vkCreateInstance instcreateinfo #f instance-ptr)
&gt; (define created-instance (ptr-ref instance-ptr _VkInstance))
&gt;
&gt; That's probably more trouble than it's worth, and the checking benefits
&gt; depend on `malloc` and `cpointer-push-tag!` begin used consistently
&gt; together. Declaring `_VkInstance-pointer` could make sense if the
&gt; library provides a `make-VkInstance-pointer` function to combine the
&gt; `malloc` and `cpointer-push-tag!` steps, helping to make sure they're
&gt; always consistent.
&gt;
&gt; Instead of `_pointer` or `_VkInstance-pointer`, you've written
&gt;
&gt; (_ptr io _VkInstance)
&gt;
&gt; As an argument type, that means:
&gt;
&gt; - The caller supplies a `_VkInstance` (not a pointer to a `VkInstance`).
&gt;
&gt; - Fresh memory for a `_VkInstance` is allocated.
&gt;
&gt; - The given `_VkInstance` is copied into the fresh memory.
&gt;
&gt; - The address of the fresh memory is passed to the foreign function as
&gt; a pointer to a `_VkInstance`.
&gt;
&gt; So, you're seeing an error because `(malloc _VkInstance)` does not
&gt; create a `_VkInstance`. Anyway, this is not what you want, because
&gt; `vkCreateInstance` would set the pointer at the fresh memory, which is
&gt; then discarded.
&gt;
&gt; Generally, a `(_ptr io ...)` argument without a name binding doesn't
&gt; make any sense --- unless you really want to ignore the result (which
&gt; you don't, in this case).
&gt;
&gt; It's pretty typical for "middleware" layers like the FFI to take into
&gt; account the in--out behavior of function arguments, and that's why many
&gt; C prototype descriptions are augmented with that information. The
&gt; `ffi/unsafe` layer is meant to be used that way, too, so the
&gt; translation that works best for `ffi/unsafe` is
&gt;
&gt; (define-vulkan vkCreateInstance
&gt; (_fun _VkInstanceCreateInfo-pointer/null
&gt; _VkAllocationCallbacks-pointer/null
&gt; (o : (_ptr o _VkInstance))
&gt; -&gt; (r : _VkResult)
&gt;
&gt; -&gt; (begin
&gt;
&gt; (check-vkResult r 'vkCreateInstance)
&gt; o)))
&gt;
&gt;
&gt;
&gt; because the third argument is an out argument.
&gt;
&gt; If you don't make to make these in--out distinctions, then the first or
&gt; second definition of `vkCreateInstance` above is the most practical
&gt; choice.

</mfl...@cs.utah.edu>