Binary data in embedded V8 again

[ry]

Hi
I've seen the "Binary data in embedded V8" thread but it is from
several months ago.
Is any work being done now to make importation/exportation of binary
strings possible?

[Stephan Beal]

If i'm not mistaken you can stuff binary data into a String by using
the two-arg ctor:

String::New("....",length_of_data);

e.g. over the weekend i wrote an an I/O library (http://
code.google.com/p/v8-juice/wiki/PluginWhio) and i use vector<char> as
my read buffer. Once i've populated it i stuff it into a String with:

String::New( &vec[0], static_cast<int>( vec.size() ) ); // i still
can't believe the second arg is a signed type!!!

that seemed to do the job, and i was reading data with embedded nulls,
and all of them seemed to make it into JS space.

[Erik Corry]

2009/3/9 Stephan Beal <sgb...@googlemail.com>:

>
> On Mar 9, 1:28 pm, ry <coldredle...@gmail.com> wrote:
>> Hi
>> I've seen the "Binary data in embedded V8" thread but it is from
>> several months ago.
>> Is any work being done now to make importation/exportation of binary
>> strings possible?
>
> If i'm not mistaken you can stuff binary data into a String by using
> the two-arg ctor:
>
> String::New("....",length_of_data);

If you make an ASCII string then the top bits must be zero so it's not clean.

If you make it a UC16 string then it has to have an even byte length.

So the status is that there isn't any good way to store binary data in
JS at the moment. Of course it is possible to put the data in an
external object instead.

>
> e.g. over the weekend i wrote an an I/O library (http://
> code.google.com/p/v8-juice/wiki/PluginWhio) and i use vector<char> as
> my read buffer. Once i've populated it i stuff it into a String with:
>
> String::New( &vec[0], static_cast<int>( vec.size() ) ); // i still
> can't believe the second arg is a signed type!!!
>
> that seemed to do the job, and i was reading data with embedded nulls,
> and all of them seemed to make it into JS space.
>
> >
>

--
Erik Corry, Software Engineer
Google Denmark ApS. CVR nr. 28 86 69 84
c/o Philip & Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018
Copenhagen K, Denmark.

[Stephan Beal]

On Mon, Mar 9, 2009 at 7:20 PM, Erik Corry <erik....@gmail.com> wrote:
>> String::New("....",length_of_data);
>
> If you make an ASCII string then the top bits must be zero so it's not clean.

Why must it be 8-bit clean? Is that a limitation/feature of the
String() class? Will the String treat it as UTF automatically if it
sees a high bit? (Some API docs would be nice! hint, hint!)

In my case i'm working on an i/o library which of course treats the
data as opaque (void*). If i understand you correctly, if it happens
to read something with a high bit set then the data it passes back to
the caller (via a String insance) is effectively undefined (or, at
least not guaranteed to be the same bits that the input device read)?
Do i need to document that handling data with non-ASCII chars
essentially leads to undefined results? (Not a huge deal, IMO, for JS
code, as i can't imagine people doing much binary i/o with it, but i'd
like to document it if it's not going to work as expected.)

> If you make it a UC16 string then it has to have an even byte length.

Are there any docs on handling 2-byte strings in v8, or is this a
"must be done by implementations using ExternalStringRepresentation"
feature? Could/Should i potentially use ExternalStringRepresentation
as an internal buffer for the data, rather than an External-to-void*
(which can't be dereferenced by the caller)?

> So the status is that there isn't any good way to store binary data in
> JS at the moment.  Of course it is possible to put the data in an
> external object instead.

That's an idea. Didn't think of that. It'd mean (in my case) buffering
arbitrarily large read buffers, and since v8 doesn't guaranty GC will
ever be called, i don't want to risk it causing an arbitrarily-sized
leak.

:)

--
----- stephan beal
http://wanderinghorse.net/home/stephan/

[Erik Corry]

2009/3/9 Stephan Beal <sgb...@googlemail.com>:

>
> On Mon, Mar 9, 2009 at 7:20 PM, Erik Corry <erik....@gmail.com> wrote:
>>> String::New("....",length_of_data);
>>
>> If you make an ASCII string then the top bits must be zero so it's not clean.
>
> Why must it be 8-bit clean? Is that a limitation/feature of the
> String() class? Will the String treat it as UTF automatically if it
> sees a high bit? (Some API docs would be nice! hint, hint!)

Here's the text from v8.h:

* Allocates a new string from either utf-8 encoded or ascii data.
* The second parameter 'length' gives the buffer length.
* If the data is utf-8 encoded, the caller must
* be careful to supply the length parameter.
* If it is not given, the function calls
* 'strlen' to determine the buffer length, it might be
* wrong if 'data' contains a null character.
*/
static Local<String> New(const char* data, int length = -1);

So it will assume that it is UTF-8 if it is not ASCII. Not all binary
sequences are valid UTF-8 so you can't use this for binary data.
Internally, V8 does not use UTF-8 so this data will be converted to
UC16.

/** Allocates a new string from utf16 data.*/
static Local<String> New(const uint16_t* data, int length = -1);

This one takes 16 bit characters and can represent binary data with no
corruption, but the length is in characters, so you can's use it for
an odd number of bytes.

>
> In my case i'm working on an i/o library which of course treats the
> data as opaque (void*). If i understand you correctly, if it happens
> to read something with a high bit set then the data it passes back to
> the caller (via a String insance) is effectively undefined (or, at
> least not guaranteed to be the same bits that the input device read)?
> Do i need to document that handling data with non-ASCII chars
> essentially leads to undefined results? (Not a huge deal, IMO, for JS
> code, as i can't imagine people doing much binary i/o with it, but i'd
> like to document it if it's not going to work as expected.)

Giving binary data to the above New method will result in undefined behaviour.

>
>> If you make it a UC16 string then it has to have an even byte length.
>
> Are there any docs on handling 2-byte strings in v8, or is this a
> "must be done by implementations using ExternalStringRepresentation"
> feature? Could/Should i potentially use ExternalStringRepresentation
> as an internal buffer for the data, rather than an External-to-void*
> (which can't be dereferenced by the caller)?

The external strings must have their data either in ASCII or in UC16.
There's no Latin1 and undefined stuff will result if you try. In the
case of an external string the actual string data is not on the V8
heap. It is assumed to be immutable too of course since all JS
strings are immutable.

>
>> So the status is that there isn't any good way to store binary data in
>> JS at the moment.  Of course it is possible to put the data in an
>> external object instead.
>
> That's an idea. Didn't think of that. It'd mean (in my case) buffering
> arbitrarily large read buffers, and since v8 doesn't guaranty GC will
> ever be called, i don't want to risk it causing an arbitrarily-sized
> leak.

If the data is on the V8 heap then it won't be collected without a GC either. :)

>
> :)
>
> --
> ----- stephan beal
> http://wanderinghorse.net/home/stephan/
>
> >
>

[Stephan Beal]

On Mon, Mar 9, 2009 at 8:41 PM, Erik Corry <erik....@gmail.com> wrote:
> 2009/3/9 Stephan Beal <sgb...@googlemail.com>:

> Here's the text from v8.h:
>
>   * Allocates a new string from either utf-8 encoded or ascii data.
>   * The second parameter 'length' gives the buffer length.
>   * If the data is utf-8 encoded, the caller must
>   * be careful to supply the length parameter.
>   * If it is not given, the function calls
>   * 'strlen' to determine the buffer length, it might be
>   * wrong if 'data' contains a null character.
>   */

Aha, okay i wasn't clear on the automatic assumption to utf8. Fair enough.

> So it will assume that it is UTF-8 if it is not ASCII.  Not all binary
> sequences are valid UTF-8 so you can't use this for binary data.
> Internally, V8 does not use UTF-8 so this data will be converted to
> UC16.

Doh, and here all along i assumed utf8 was what WAS used, as the API
has Utf8Value but no Utf16Value.

>  /** Allocates a new string from utf16 data.*/
>  static Local<String> New(const uint16_t* data, int length = -1);
>
> This one takes 16 bit characters and can represent binary data with no
> corruption, but the length is in characters, so you can's use it for
> an odd number of bytes.

What's the byte order?

>> In my case i'm working on an i/o library which of course treats the
>> data as opaque (void*). If i understand you correctly, if it happens

...

> Giving binary data to the above New method will result in undefined behaviour.

Fair enough.

> The external strings must have their data either in ASCII or in UC16.
> There's no Latin1 and undefined stuff will result if you try.  In the
> case of an external string the actual string data is not on the V8
> heap.  It is assumed to be immutable too of course since all JS
> strings are immutable.

That wouldn't solve my case, which is effectively latin1. i'll need to
think about that (but don't mind living with the limitation of ascii
read/write).

>> That's an idea. Didn't think of that. It'd mean (in my case) buffering
>> arbitrarily large read buffers, and since v8 doesn't guaranty GC will
>> ever be called, i don't want to risk it causing an arbitrarily-sized
>> leak.
>
> If the data is on the V8 heap then it won't be collected without a GC either. :)

But even if i registered it for gc via a weak pointer callback, it's
not guaranteed to be freed, so i'm forced to add external gc to it in
*any* case and have the client call the cleanup routine when their
context dies (this is currently handled via a sentry object in the
client app which cleans up when it goes out of scope).

[Erik Corry]

2009/3/9 Stephan Beal <sgb...@googlemail.com>:

>
> On Mon, Mar 9, 2009 at 8:41 PM, Erik Corry <erik....@gmail.com> wrote:
>> 2009/3/9 Stephan Beal <sgb...@googlemail.com>:
>> Here's the text from v8.h:
>>
>>   * Allocates a new string from either utf-8 encoded or ascii data.
>>   * The second parameter 'length' gives the buffer length.
>>   * If the data is utf-8 encoded, the caller must
>>   * be careful to supply the length parameter.
>>   * If it is not given, the function calls
>>   * 'strlen' to determine the buffer length, it might be
>>   * wrong if 'data' contains a null character.
>>   */
>
> Aha, okay i wasn't clear on the automatic assumption to utf8. Fair enough.
>
>> So it will assume that it is UTF-8 if it is not ASCII.  Not all binary
>> sequences are valid UTF-8 so you can't use this for binary data.
>> Internally, V8 does not use UTF-8 so this data will be converted to
>> UC16.
>
> Doh, and here all along i assumed utf8 was what WAS used, as the API
> has Utf8Value but no Utf16Value.
>
>>  /** Allocates a new string from utf16 data.*/
>>  static Local<String> New(const uint16_t* data, int length = -1);
>>
>> This one takes 16 bit characters and can represent binary data with no
>> corruption, but the length is in characters, so you can's use it for
>> an odd number of bytes.
>
> What's the byte order?

Native.

>
>>> In my case i'm working on an i/o library which of course treats the
>>> data as opaque (void*). If i understand you correctly, if it happens
> ...
>> Giving binary data to the above New method will result in undefined behaviour.
>
> Fair enough.
>
>> The external strings must have their data either in ASCII or in UC16.
>> There's no Latin1 and undefined stuff will result if you try.  In the
>> case of an external string the actual string data is not on the V8
>> heap.  It is assumed to be immutable too of course since all JS
>> strings are immutable.
>
> That wouldn't solve my case, which is effectively latin1. i'll need to
> think about that (but don't mind living with the limitation of ascii
> read/write).
>
>>> That's an idea. Didn't think of that. It'd mean (in my case) buffering
>>> arbitrarily large read buffers, and since v8 doesn't guaranty GC will
>>> ever be called, i don't want to risk it causing an arbitrarily-sized
>>> leak.
>>
>> If the data is on the V8 heap then it won't be collected without a GC either. :)
>
> But even if i registered it for gc via a weak pointer callback, it's
> not guaranteed to be freed, so i'm forced to add external gc to it in
> *any* case and have the client call the cleanup routine when their
> context dies (this is currently handled via a sentry object in the
> client app which cleans up when it goes out of scope).

If there is a global GC then the weak pointer callbacks will be
called. If there is no global GC then strings on the heap may not be
collected either.

Did you notice that recent V8s have had logic added to do a GC on
context creation if there was a context that died since the last
context creation (the slightly convoluted logic is to avoid a wave of
expensive global GCs when a series of contexts are destroyed at around
the same time).

>
>
> --
> ----- stephan beal
> http://wanderinghorse.net/home/stephan/
>
> >
>

[Alex Iskander]

It will be freed, you just don't have any control over when unless you use external garbage collection.

Objects in v8 exist outside of contexts, so their weak handle callback will get called whenever v8 garbage collects, even if "their" context dies (as, since the object is not tied to a single context, it never has a context that can die). So as long as v8 is still being used for any scripts, you won't have any leaks, because garbage collection will keep occurring.

You don't have to worry about memory leaks, because when the application itself dies, all of its memory will be freed by the operating system.

The only times you need external garbage collection, from what I can tell, are when destructors must be called, which is only necessary when doing file i/o, etc, with resources that must be explicitly closed.

Alex

On Mar 9, 2009, at 2:52 PM, Stephan Beal wrote:

But even if i registered it for gc via a weak pointer callback, it's
not guaranteed to be freed, so i'm forced to add external gc to it in
*any* case and have the client call the cleanup routine when their
context dies (this is currently handled via a sentry object in the
client app which cleans up when it goes out of scope).

Alex Iskander, TPSi

[Stephan Beal]

On Mar 9, 9:12 pm, Alex Iskander <a...@tpsitulsa.com> wrote:
> It will be freed, you just don't have any control over when unless you  
> use external garbage collection.

If i don't use external collection at all, it might not be freed (been
there, done that). In my tests, my Weak callbacks were never getting
called until i created thousands of the objects, and if i didn't
create thousands of them, the context would eventually exit without
calling the callbacks. So i've been forced to add "just in case" GC
which gets "removed" *if* v8 gc calls the weak pointer callback,
otherwise it cleans up all remaining objects at app exit.

> Objects in v8 exist outside of contexts, so their weak handle callback  
> will get called whenever v8 garbage collects, even if "their" context  
> dies (as, since the object is not tied to a single context, it never  
> has a context that can die). So as long as v8 is still being used for  
> any scripts, you won't have any leaks, because garbage collection will  
> keep occurring.

In my tests, i my callbacks would normally never be called, so "will
get called" is a tad bit misleading (unless that's been fixed in the
past week or so). i tested this by adding debug output to my callback
routines and class dtors. The only time i could get ANY weak callback
was when memory use jumped (which it didn't when the script is only "if
(1){var x = new MyType()}", which "should" call gc when the script (or
context) exits).

> You don't have to worry about memory leaks, because when the  
> application itself dies, all of its memory will be freed by the  
> operating system.

That's not the way C++ does destruction. A C++ destructor may free
resources, and dtors are not called at app exit if the objects are on
the heap (which nearly all JS/Native wrapped objects are). That leads
to db handles, file handles, etc. which don't get closed in a well-
defined fashion (i.e. could lead to corruption). In the cases of the
ncurses bindings, such behaviour (destruction via non-dtor) will cause
the app to stay in curses screen mode when it exits (which is ugly and
requires one to type "reset" to get the screen back in order). That
type of problem is unfortunately only solved via extra client-side
garbage collection. IMO v8 should either guaranty that gc is called at
context shutdown or give us a Context configuration option which
enables that guaranty for those of us who (dearly) care about it. It
would save tons of client-side GC efforts which we have to add only
for the sake of Chrome's performance (as that is reportedly the reason
that v8 GC is considered so time-critical).

> The only times you need external garbage collection, from what I can  
> tell, are when destructors must be called, which is only necessary  
> when doing file i/o, etc, with resources that must be explicitly closed.

And almost certainly necessary when we're wrapping arbitrary classes
(with arbitrary dtors). In my experience, most classes worth wrapping
aren't coypable, which means we cannot wrap an arbitrary number of
them to JS, as we don't have a place to store them which could
guaranty their proper destruction (e.g. a std:: container where they
would be destroyed when the container goes). e.g. sqlite3 db, window
handles of any sort, any opaque handle of any type, etc., all have to
be created on the heap. And those are also the types of objects for
which dtors *need* to be called to ensure proper clean-up.

Nondeterministism is the nature of GC, and i'm fine with that. But a
GC which says "your object *will* be collected [at some point]" and
then doesn't do it "for performance reasons", i just can't accept as
proper/correct behaviour. v8 developers: at least give us a Context
configuration option to tell it force GC at context exit (or during v8/
static destruction, if cross-context objects are a problem)!

[Stephan Beal]

On Mar 9, 11:57 pm, Stephan Beal <sgb...@googlemail.com> wrote:
> That's not the way C++ does destruction. A C++ destructor may free
> resources, and dtors are not called at app exit if the objects are on
> the heap (which nearly all JS/Native wrapped objects are). That leads
> to db handles, file handles, etc. which don't get closed in a well-
> defined fashion (i.e. could lead to corruption).

Here's a silly example, but an example nonetheless:

Let's say i have a TempFile class which extends some File type but has
the feature that it deletes itself at destruction. In v8 that class
will sometimes be leaving temp files laying around. On Unix platforms
this can be worked around by deleting the file directly after opening
it (that's legal), in which case the handle is valid until the app
exits (and the filesystem may then free the space since it has no FAT
entry). On Windows we can't do that (we can't even rename an opened
file), so this isn't reliable from v8 on those platforms.

But the point applies more generally: arbitrary classes may allocate
arbitrary resources (even via a third party, such as some type of
object/Corba server), and not destructing may leave those resources
dangling in places outside the immediate application.

[Stephan Beal]

On Mar 9, 8:41 pm, Erik Corry <erik.co...@gmail.com> wrote:
> So it will assume that it is UTF-8 if it is not ASCII.  Not all binary
> sequences are valid UTF-8 so you can't use this for binary data.
> Internally, V8 does not use UTF-8 so this data will be converted to
> UC16.

Sorry, i'm confused again. If v8 sees a high bit set, it assumes the
data is utf8, changes modes, and then then internally stores the
string as uc16? Is that correct?

Maybe it's time for me to write a BinaryData class which proxies all
bytes as integers. That might be a useful workaround (inefficient as
hell, creating Arrays of Integers, but JS isn't about efficiency, is
it?).

[Alex Iskander]

Absolutely. Otherwise I would not have been required to create my
autoDestroy class to handle file closing.

I was referring to memory, not resources. Memory will be freed on
application exit.

Conceptually, however, it is not v8's place to garbage collect the
objects. The only time running a final garbage collection would make
sense is at application exit (not context exit), and the only things
that are, in theory, v8's to free are it's own objects and values.

v8 does not own the native C++ objects; C++ owns them. V8 simply owns
handles to pointers (externals) to C++ objects. The MakeWeak fubction
is meant specifically to tell you when JavaScript is no longer using
the object; it is not meant to destroy the object. Why should it waste
valuable CPU time at application exit, causing it to take as long to
close as Adobe InDesign, Testing each and every object to see if it
MIGHT be referenced by any weak handles, so that it can tell you that
the object is no longer being used by JavaScript -- something you
should already know, since the application is exiting.

When NOT at application exit, none of this is a concern, because, as
long as you are still using v8, the garbage will be collected
eventually; and, if necessary, you can make it occur more often so the
scripting uses less memory.

Alex

Sent from Alex's iPhone

[Stephan Beal]

On Mar 10, 2:13 am, Alex Iskander <a...@tpsitulsa.com> wrote:
> v8 does not own the native C++ objects; C++ owns them. V8 simply owns  
> handles to pointers (externals) to C++ objects. The MakeWeak fubction  
> is meant specifically to tell you when JavaScript is no longer using  
> the object; it is not meant to destroy the object. Why should it waste  
> valuable CPU time at application exit, causing it to take as long to  
> close as Adobe InDesign, Testing each and every object to see if it  
> MIGHT be referenced by any weak handles, so that it can tell you that  
> the object is no longer being used by JavaScript -- something you  
> should already know, since the application is exiting.

If that's not what they're for, they're being misrepresented. It's
advertised as "you'll get notified when we're done with." Well, when a
context closes, "the context is done with it," so it should tell us.
Otherwise we get a half-ass solution - we never know if our callbacks
will be called.

> When NOT at application exit, none of this is a concern, because, as  
> long as you are still using v8, the garbage will be collected  
> eventually; and, if necessary, you can make it occur more often so the  
> scripting uses less memory.

i don't want to fudge the memory size just to force gc. That's way too
vague and kludgy, and there's no guaranty the user hasn't set some
arbitrarily small limit which i might, though my abuse of the gc
engine, violate.

The WeakJSClassGenerator i just posted will save me a lot of grief on
this front, though -i've not got guaranteed dtos as long as main()
exits normally (as opposed to abort() or a signal).

[Stephan Beal]

On Tue, Mar 10, 2009 at 3:58 AM, Stephan Beal <sgb...@googlemail.com> wrote:
> The WeakJSClassGenerator i just posted will save me a lot of grief on
> this front, though -i've not got guaranteed dtos as long as main()
> exits normally (as opposed to abort() or a signal).

Big correction: i've GOT guaranteed dtors...

[Alex Iskander]

On Mar 9, 2009, at 9:58 PM, Stephan Beal wrote:

Well, when a context closes, "the context is done with it," so it should tell us.

Here is the crucial thing: the CONTEXT cannot be done with it. v8 as a whole can be done with it. The garbage collection does not occur per context, but through the entire v8 instance. Your callbacks will be called when garbage collects; they do not belong to one context or another.

This is because one object can be used from more than one context in v8.

Again, the only catch is application exit. But in that situation, you know v8 is no longer using the resource anyway, so can handle it with your automatic destruction sentry.

i don't want to fudge the memory size just to force gc. That's way too
vague and kludgy, and there's no guaranty the user hasn't set some
arbitrarily small limit which i might, though my abuse of the gc
engine, violate.

There is also a way to specify memory limits for v8, which would have a similar effect. The documentation is rather spotty, but it is used in some of v8's tests. v8, since it stops JavaScript each time it garbage collects, tries to do so only when it needs to. So, you have to tell it how to decide it needs to.

Alex Iskander, TPSi

[Ryan Dahl]

> So the status is that there isn't any good way to store binary data in
> JS at the moment.  Of course it is possible to put the data in an
> external object instead.

A v8 byte array type would be nice (which had, say, some toString()
methods for various encodings). I know google gears has binary Blobs,
the XMLHttpRequest level 2 has a ByteArray (but doesn't specify what
it would do), I think AIR and various other systems (?) also have it.
It seems like a reasonable thing to have in v8.

[Stephan Beal]

On Mar 10, 1:47 pm, Ryan Dahl <coldredle...@gmail.com> wrote:
> A v8 byte array type would be nice (which had, say, some toString()
> methods for various encodings). I know google gears has binary Blobs,
> the XMLHttpRequest level 2 has a ByteArray (but doesn't specify what
> it would do), I think AIR and various other systems (?) also have it.
> It seems like a reasonable thing to have in v8.

How does this interface sound to you:

ByteArray([int len]);
int operator[int index];
void close(); // forces freeing of the underlying array
string toString(); // what encoding?
string toAscii(); // convert up to first entry >127.
int length(); // in bytes

Conversions to misc. string representations could/should be done via
add-on functions, i think.

Internally we could could use (unsigned char *) instead of JS Array-of-
Integer (which would be extremely costly).

:-?

[Christian 'Grytpype' Plesner]

I think an easier way to store binary data in v8 would be a good
addition to the api. I have filed a feature request for this,
http://code.google.com/p/v8/issues/detail?id=270. This is not
something we in the v8 team need so it's not something we're likely to
implement any time soon but if someone wanted to contribute an
implementation we'd certainly be open to that (see
http://code.google.com/p/v8/wiki/Contributing).