How do I create an ExternalByteArray in Dart ?

[Nelson Silva]

Hi,

I'd like to pass some ByteArrays to a native extension and I'm using Uint32List but apparently these are created in the head and are not external.

Is there any public class that uses external byte arrays ? If not what is the recommended approach to pass a type array to a native extension ? Should I just copy the data to another location ?

Thanks!

[Mads Ager]

Hi Nelson,

external byte arrays are allocated through the Dart VM API.

http://code.google.com/p/dart/source/browse/branches/bleeding_edge/dart/runtime/include/dart_api.h#1601

So, in your extension code, you can have a native method that goes into C++ to allocate an external byte array through the API and returns that as its return value.

In dart:io we use external byte arrays for data received from the system. The data that you get when reading file and socket data are in external byte arrays. All of them allocated in C++ via the API above:

http://code.google.com/p/dart/source/browse/branches/bleeding_edge/dart/runtime/bin/io_buffer.cc#12

Hope this helps.

Cheers,    -- Mads

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

[Nelson Silva]

I'm experimenting with WebCL bindings for Dart ( https://github.com/nelsonsilva/dart_webcl).

I'm using typed lists in dart : var vector1 = new Uint32List(vectorLength);
and I need to get the data from these in the native code (for instance in: cmdQueue.enqueueWriteBuffer (bufIn1, false, 0, vector1.lengthInBytes(), vector1, []))

What you're saying is I that I either come up with some new classes that rely on external byte arrays or I'll have to copy all the data to a new mem location.

Are there plans to expose typed arrays in the VM (Uint32Array, etc..) ? And if so will these use external byte arrays ?

If not I could probably mimic the typed arrays from Dartium which would be closer to the WebCL stuff but I'd like to know if this is in the roadmap already.

Thanks.

[Mads Ager]

There is support for both internal and external typed lists in the VM. These should be used to implement the typed arrays in dartium going forward. In fact they should be the same. In any case, the external ones are going to be allocated through the VM API. The ones you will get from Dartium will be allocated in the WebKit C++ bindings as external typed arrays. That is definitely on the roadmap, but I can't provide a concrete timeline for it.

Using the typed lists in dart:scalarlist is the right thing to do. In order for them to have external backing store you need to go into C++ to allocate them but you can still use the Uint8List type. In your extension you can have:

Uint8List allocateExternalList(int size) native "AllocateExternalList";

Would that work for you?

Cheers,    -- Mads

[Nelson Silva]

So, dart_api.h only allows allocating Uint8 external lists for now, right ?

Any workarounds for having the other types as well ?

Would it be feasible to create simple typed arrays for the other types

[Anton Muhin]

Nelson, you can create Uint8 external list and reuse it external
storage to create other types of arrays.

However, chances are high we'll change dart:scalarlist slightly soon,
roughly this quater, so be prepared.

yours,
anton.

[Mads Ager]

Here is how you would treat the backing store for a Uint8List as some other list type using views:

import 'dart:scalarlist';

main() {

  var list8 = new Uint8List(16);

  var bytes = list8.asByteArray(0, 16);

  var list32 = new Int32List.view(bytes, 0, 4);

  print('list8 length: ${list8.length}');

  print('list32 length: ${list32.length}');

  list8[0] = 1;

  list8[1] = 1;

  print('byte value at offset 0: ${list8[0]}');

  print('int32 value at offset 0: ${list32[0]}');

}

Output:

list8 length: 16

list32 length: 4

byte value at offset 0: 1

int32 value at offset 0: 257

Cheers,    -- Mads

[Nelson Silva]

Thanks!

I got it working using:

Uint8List allocateExternalList(int size) native "AllocateExternalList";

class Uint32Array implements ByteArrayViewable {
  Uint8List _list8;
  Int32List _list32;
 
  Uint32Array(int size){
    _list8 = allocateExternalList(size * 4);
    _list32 = new Int32List.view(this.asByteArray(), 0, size);
  }
 
  int bytesPerElement() => _list32.bytesPerElement();
  int lengthInBytes() => _list32.lengthInBytes();
  ByteArray asByteArray([int start, int length]) => _list8.asByteArray();
 
  Int32List get array32 => _list32;
  Uint8List get array8 => _list8;
  operator[]= (int index, val) => _list32[index] = val;
  operator[] (int index) => _list32[index];
}

The only "ugly" stuff is that I need to use the array8 getter to pass the UInt8List to the native methods but I can get around that by checking for a Uint32Array as argument and extracting the Uint8List from the object in the native code.