Bytes and BytesBuffer should use Uint8Array

[Daniel Kuschny]

Hi there.

The current JavaScript implementation of BytesData is Array<Int>. This produces an extreme overhead when working with binary data. Most Browsers (http://caniuse.com/#feat=typedarrays) already support typed arrays and therefore the Uint8Array.  To support the IE a fallback like this could be included: https://gist.github.com/1057924  

I already thought about creating an issue on Google Code directly, but after writing several denied change-requests to Haxe I wanted to discuss here first. Maybe there are reasons why this feature is not implemented yet.

Here a draft how the JavaScript version of the binary classes could look like. I deleted the conditional compilation parts of the other targets.

std/haxe/io/BytesData.hx

// https://github.com/xirsys/stdjs/blob/master/haxelib/js/webgl/TypedArray.hx
typedef BytesData = Uint8Array;

std/haxe/io/Bytes.hx

class Bytes {

public var length(default,null) : Int;

var b : BytesData;

function new(length,b) {

this.length = length;

this.b = b;

}

public inline function get( pos : Int ) : Int {

return b[pos];

}

public inline function set( pos : Int, v : Int ) : Void {

b[pos] = v;

}

public function blit( pos : Int, src : Bytes, srcpos : Int, len : Int ) : Void {

var subSrc = src.subarray(srcpos, srcpos + len);

b.set(srcSub, pos);

}

public function sub( pos : Int, len : Int ) : Bytes {

return new Bytes(len, b.subarray(pos, pos + len);

}

public function compare( other : Bytes ) : Int {

// no changes 

}

public function readString( pos : Int, len : Int ) : String {

// http://updates.html5rocks.com/2012/06/How-to-convert-ArrayBuffer-to-and-from-String

return String.fromCharCode.apply(null, new Uint16Array(b.buffer, pos, len));

}

public function toString() : String {

return readString(0,length);

}

public static function alloc( length : Int ) : Bytes {

return new Bytes(length, new Uint8Array(length));

}

public static function ofString( s : String ) : Bytes {

var bufView = new Uint16Array(new ArrayBuffer(s.length * 2));

var strlen = s.length;

for(i in 0 ... strlen) {

bufView[i] = str.charCodeAt(i);

}

var b = new Uint8Array(bufView.buffer);

return new Bytes(b.length, b);

}

public static function ofData( b : BytesData ) {

return new Bytes(b.length, b);

}

}

std/haxe/io/BytesBuffer.hx

A bit more complex implementation is needed by the BytesBuffer class. It would need a dynamic resizing solution for those fixed arrays just like ArrayList/List in Java/C#. (Recreate + Blit)

class BytesBuffer {

var b : BytesData;

var size : Int;

public function new() {

b = new BytesData(0);

}

public inline function addByte( byte : Int ) {

grow(1);

b[size++] = byte;

}

public inline function add( src : Bytes ) {

addBytes(src, 0, src.length);

}

public inline function addBytes( src : Bytes, pos : Int, len : Int ) {

grow(len);

b.blit(size, src, pos, len);

size += src.length;

}

public function getBytes() : Bytes untyped {

var sub = b.subarray(0, size);

var bytes = new Bytes(size, sub);

b = null;

return bytes;

}

private function grow( inc : Int ) : Void {

var minCapacity = size + inc;

if(minCapacity >= b.length) {

var capacity = Std.int(Math.max(Math.max(b.length * 2, 4), minCapacity));

var oldB = b;

b = new BytesData(capacity);

b.blit(0, oldB, 0, oldB.length);

}

}

}

With the current standard library it's really hard working with binary data in case of JavaScript. You have to create wrappers and stuff for all IO handlings. 

[Simon Krajewski]

There are plans to change something in that regard, see: http://code.google.com/p/haxe/issues/detail?id=1045

We could change Bytes to an abstract type similar to the new Vector class: http://code.google.com/p/haxe/source/browse/trunk/std/haxe/Vector.hx
This way we can provide a consistent Api and use different underlying types depending on the platform. If you want to help out with that, feel free to give it a go. You can contact me for any questions regarding abstract types.

Simon

[Nicolas Cannasse]

Le 21/01/2013 19:57, Daniel Kuschny a �crit :

> Hi there.
>
> The current JavaScript implementation of BytesData is Array<Int>. This
> produces an extreme overhead when working with binary data. Most
> Browsers (http://caniuse.com/#feat=typedarrays) already support typed
> arrays and therefore the Uint8Array. To support the IE a fallback like
> this could be included: https://gist.github.com/1057924

Seems like the IE wrapper is small enough to get a very low overhead in
terms of JS output size. Please open an issue on Google Code.

If you want to submit an implementation to speedup inclusion, please use
the SVN js.html.* HTML5 API.

Best,
Nicolas