ByteString / ByteArray duality

[Wes Garland]

> (Side note: Do we really need the ByteString/ByteArray duality? At
> least we should introduce a toBinary() that can either return a
> ByteArray or ByteString without requiring a copy...)

I've introduced a syntax in GPSEE (originally intended for the FFI library) which uses the constructor function without "new" as a casting operator, which uses the underlying C buffer of the other type.

So, code like this does not involve copying:

var ba = new ByteArray([1,2,3,4,5]);
var bs = ByteString(ba);

I also have Memory and MutableStruct classes implemented which can be casted to/from.  Memory is basically a wrapper for malloc and MutableStruct maps the backing store onto a C struct (decoded as a JS object so that struct members become properties).

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102

[Kris Kowal]

On Fri, Sep 4, 2009 at 10:18 AM, Wes Garland<w...@page.ca> wrote:
>> (Side note: Do we really need the ByteString/ByteArray duality? At
>> least we should introduce a toBinary() that can either return a
>> ByteArray or ByteString without requiring a copy...)
>
> I've introduced a syntax in GPSEE (originally intended for the FFI library)
> which uses the constructor function without "new" as a casting operator,
> which uses the underlying C buffer of the other type.
>
> So, code like this does not involve copying:
>
> var ba = new ByteArray([1,2,3,4,5]);
> var bs = ByteString(ba);

If that's the case, modifications to "ba" would cause "bs" to change,
if I'm not mistaken. I very much approve of the optimization, but I
do not think that "new" and "newless" constructors should differ in
behavior. Perhaps the original ByteArray can set a "copy-on-write"
flag, forfeiting the underlying storage to the ByteString whenever it
needs to make a change.

Kris Kowal

[Wes Garland]

Kris:

Good eye:  one the problems I had when I was working on that syntax was the mutable/immutable barrier. For the time being I've simply disallowed casts where that's an issue, which means the example I provided is terrible. :)

What should /really/ happen is COW, but that's actually more complicated that it seems at first blush, for memory allocation and GC reasons: when a write happens on the mutable object, we have to not only copy, but create a reference for the old memory with the correct lifespan for the immutable objects' references.

As for newed vs. newless constructor functions: the syntax was chosen because it was easy,  resembles C++ casts, and unlikely to get overloaded by the CommonJS-wg. :)   

It also takes different arguments than the constructor, does not require the ability to construct a half-initialized object, and does not require native objects to be "friendly" across the module boundary.  Having a .toWhatever() method, unfortunately, has at least one of those problems.

Here's is some real-world test code using that syntax, rather than the random ramblings I normally post ;) :

#! /usr/bin/gsr -ddzz

const ffi = require("gffi");
const opendir = new ffi.CFunction(ffi.pointer, "opendir", ffi.pointer);
const readdir = new ffi.CFunction(ffi.pointer, "readdir", ffi.pointer);
const stat = new ffi.CFunction(ffi.int, "stat", ffi.pointer, ffi.pointer);

var dirp;
var dent;
var sb = new ffi.MutableStruct("struct stat");
var filename;

dirp = opendir.call("/etc");
if (!dirp)
  throw("can't open dir");

while (dent = readdir.call(dirp))
{
  dent = ffi.MutableStruct(dent, "struct dirent");
  filename = "/etc/" + dent.d_name.asString(-1);

  if (stat.call(filename, sb) != 0)
    print("can't stat " + filename + ", errno=" + ffi.errno);
  else
    print(filename + " is " + sb.st_size + " bytes long");
}

In this case, readdir.call() returns a Memory() instance (the C function returns a pointer).

Memory instances are ByteThings (ByteThing is what I call things like ByteString which are cast-compatible).  In the first line of the while loop, we cast that Memory instance into a MutableStruct with the struct layout and JS object properties of a struct dirent.

I'm not tied to the syntax, but the capability is really useful -- and actually absolutely necessary in this example.

Wes