Working with untyped data blobs in Swift

[Jens Alfke]

I’m curious how to model a binding to a C API that uses void*s to represent uninterpreted data blobs. Specifically, I’m working with a key-value store (similar to Berkeley DB) that stores keys and values as blobs. Here’s one function:

fdb_status fdb_get_kv(fdb_kvs_handle *handle,

                      const void *key, size_t keylen,

                      void **value_out, size_t *valuelen_out);

This function allocates the value blob on the heap. value_out is an ‘out’ parameter that on return will point to the value, and valuelen_out will contain the length of the value. The caller is responsible for freeing the value.

In Swift 2, the corresponding wrapper method would return the value, but as what type? NSData is a fairly obvious choice, especially since I can initialize it with bytesNoCopy: and avoid having to copy the value. But NSData doesn’t seem like a very Swift-y type, and I’m also thinking ahead to cross-platform code that probably won’t have Foundation types available. 

So maybe an Array[UInt8]? But how do I create one of those from an UnsafeMutablePointer<Void>, which is what value_out comes out as in Swift? (And is it possible to create one that will adopt the heap block without copying?)

(An obvious question is “what will the caller interpret the value as?” At this level the API is agnostic. Some callers might store a UTF-8 string, some might store JPEG data, some might store a structured binary encoding like a protocol buffer.)

—Jens

[Jim Dovey]

What you're probably going to end up with is UnsafeMutablePointer<RawByte>. This can be used to initialize a pointer to any specific type, and can be treated as something incrementable in similar fashion to a uint8* in C. You can also use it to initialize an UnsafeMutableBufferPointer<> which will then provide a CollectionType interface to the underlying storage.

If you'd like something a little higher-level, take a look at ManagedBuffer<Value, Element>; this is a class, so it manages allocation/deallocation for you. It's common to use the Value to store the number of elements allocated, and you can get at a mutable pointer to the elements via a withMutablePointerToElements() function similar to those on String and Array. I'm not familiar with the type-casting applicability of that one though.

As an aside: when looking at temporary storage for these sorts of things, it appears that Array is flexible enough that you don't need to go nuts with UnsafePointer types. At least, I've seen the folks who write Swift using that, so I assume it's not *too* bad 😉

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Swift Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swift-languag...@googlegroups.com.
To post to this group, send email to swift-l...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/swift-language/8E22BFEA-10C9-47FC-B2AD-C490B263BED5%40mooseyard.com.
For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Jun 14, 2015, at 2:41 PM, Jim Dovey <jimd...@gmail.com> wrote:

What you're probably going to end up with is UnsafeMutablePointer<RawByte>. This can be used to initialize a pointer to any specific type, and can be treated as something incrementable in similar fashion to a uint8* in C. You can also use it to initialize an UnsafeMutableBufferPointer<> which will then provide a CollectionType interface to the underlying storage.

Thanks! So far I’ve got a simple class that holds onto an UnsafeBufferPointer and frees the memory when it’s done:

class ValueBuffer {

    init(block: UnsafeMutablePointer<Void>, length: Int) {

        let start = UnsafePointer<UInt8>(block);

        buf = UnsafeBufferPointer<UInt8>(start: start, count: length);

    }

    deinit {

        let block = UnsafeMutablePointer<Void>(buf.baseAddress)

        free(block);

    }

    private var buf: UnsafeBufferPointer<UInt8>;

}

But it looks like, to make ValueBuffer a Collection, I have to look up all the primitive methods of that protocol and its ancestors and implement them all in ValueBuffer, by delegating to .buf. Which seems like annoying boilerplate work :-p

If you'd like something a little higher-level, take a look at ManagedBuffer<Value, Element>; this is a class, so it manages allocation/deallocation for you.

Yeah, I don’t want to use that because it’ll require copying the data, so I might as well create an Array.

As an aside: when looking at temporary storage for these sorts of things, it appears that Array is flexible enough that you don't need to go nuts with UnsafePointer types. At least, I've seen the folks who write Swift using that, so I assume it's not *too* bad 😉

I’m concerned about performance, because this is super low-level code that ends up on the critical path of higher level tasks. The code this would be replacing is some tuned/optimized C++.

—Jens