Accessing Canvas Pixel Data

[Cyril MONGIS]

Dear all !

I'm curerntly creating some kind of Image Anaysis Application with App JS and Node OpenCV. Problem comes when it comes to load pixels in the Canvas from C++. Loading it through a javascript loop takes way to much time for some unknow reasons. So I decided to go the C++ way. The canvas element seems to use a Uint8ClampedArray to store the pixels but the v8 header files or NodeJS header files don't mention this type.

Does anyone know a way to cast or access this Uint8ClampedArray ?

On the "Issues page" on GitHub, it has been said that ArrayBuffer are present int the "Window object" but I don't have any clue of how to access or use it in C++.

[Cyril MONGIS]

To make myself a little clearer, here is what I would like to do :

<script>
//getting the canvas context
canvas = getElementById("#canvas");
context = canvas.getCanvasContext("2d");
imageData = context.createImageData(100,100);

//now I call the function put in my C++ NodeJS addon

myFunctionThatFillCanvasWidthPixel(imageData.data); // NB : imageData.data is shown by the debugger as a Uint8ClampedArray

context.putImageData(imageData,0,0);

</script>

And in the C++ side

<Handle>Value
myFunctionThatFillCanvasWithPixel(const Arguments & args) {
   
    Local<Object> obj = args[0]->ToObject();

    unit * pixels  =  obj->SomeMagick() // I don't know what to do
              // Nothing seems to work
   
    for(...) {
        pixels[i] = 150; // or whatever value
    }

    return Undefined() //or whatever that could help me to draw in the canvas    
}

Here is a couple of things that what I found :
   - Uint8ClampedArray are handled by NodeJS since 0.7, that's cool I use 0.8.9
   - Those structures are variants of ArrayBuffer which shows the follow structure :
    ArrayBuffer
        int length
        Buffer * buffer (node buffer)
        ...
   - normally the length and the pointer to the data can be accessed through the method "GetIndexedPropertiesToExternalArrayData". Those method is part of the v8::Object so it should normally give me access to the gold mine but it doesn't.
   - If I ask for the length of the ArrayBuffer from C++, -1 is returned, althrough the chromium debugger shows something different


So now, I have been struggling for days, and I tried many possibilities but I feel that I'm missing something. Does someone has any idea of the solution ?

[Simon Horton]

Can this be caused by mixing the use of two separate contexts? So the browser (chromium) has its V8 context for the browser, and in that there is the canvas and the byte array. You are then passing that into a different v8 context (the nodejs application which is running your C++ module), so your code is running in a different context and so cannot read the memory from the other context?

I would guess the solution is then that you have to "copy" the data between the two contexts which will of course then kill the performance reason for coding in C++.

I think in general it is going to be really difficult to have a nodejs C++ module that plays around with objects directly from the browser, however you can look at the "bridge" appjs code that manipulates the browser and allows javascript to access it.

Alternative solutions would be I guess to use webgl (not supported in the existing version of appjs) or even something exotic like LLVM compilation of c code to js, although if you are doing tight loops then that might not help.

Unfortunately my C coding skills are very old now so I don't think I would be of much practical use in helping you get this mixing of contexts to work.

/Simon

--
You received this message because you are subscribed to the Google Groups "appjs-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appjs-dev+...@googlegroups.com.
To post to this group, send email to appj...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Cyril MONGIS]

Hi !

Thanks for this answer, it helped me a lot going forward.
Actually, your theory of separate contexts seems really plausible. I switched to node-webkit which says to run webkit and node in the same thread. My code actually worked. I was happy with this solution until I really discovered what it also implied. If I do intensive tasks with javascript, the browser rendering is blocked. The ideal solution would be, like for any other language, to launch a separate thread to handle the calculation while the view refresh itself. The problem is that I didn't find any way to launch a separate thread with javascript. There is thread-a-gogo but the newly created thread runs in a totally different context and I didn't find a way to pass objects from a context to the other before the thread starts. If you know a way, please tell me. I tested my code back with appjs and I got the same problem (rendering blocked during calculations).

So now, I trying to find a way to create a separate thread that would handle the calculations but the problem is the communication between this thread and the main thread which must be efficient and permissive since this communication will mostly involve tons of images 11Mb images.

[Simon Horton]

Hi Cyril,

Maybe you could look into native client for straight chrome. I know it is possible to write games in C++ and have it compiled into the native client. I think it is extremely locked down so no direct access to the users disk but maybe it will give you what you want which seems to be C++ access to a webpage.

I know there is webgl and I have seen various demos of quite cool stuff with javascript and canvas but it might involve a lot of optimisation work in order to get it to run well -- that is a trade off against researching something like native client and seeing if that works well, since that is also a time investment. It might be possible to change your algorithm to reduce the number of loops or to access canvas in a lighter way, but that can be a lot of effort. I know there is something like OpenCL that can also run in the browser . 

It might be possible to create your own plugin (so something like flash) that is displayed in a particular part of the webpage, if you are using old internet explorer you can even embed an ActiveX control directly in the page! However these methods are not popular and everyone is trying to move away from them towards html5 instead.

/Simon