Writing bitmap data to a canvas with WebAssembly

[Scott Pakin]

I'm new to WebAssembly and am trying to get Go to generate a bitmapped image to display in the Web browser.  Here's what I've come up with so far:

func DrawPicture(this js.Value, args []js.Value) interface{} {
        // Create a canvas on the Web page.
        const width = 256
        const height = 256
        canvas := js.Global().Get("document").Call("getElementById", "my-canvas")
        canvas.Set("width", width)
        canvas.Set("height", height)
        canvas.Set("style", "border: thin solid black")
        ctx := canvas.Call("getContext", "2d")
        ctx.Call("clearRect", 0, 0, width, height)


        // Render a picture.
        imgData := ctx.Call("createImageData", width, height)
        data := imgData.Get("data")
        for y := 0; y < height; y++ {
                for x := 0; x < width; x++ {
                        ofs := 4 * (y*width + x)
                        data.SetIndex(ofs+2, x+y)
                        data.SetIndex(ofs+3, 255)
                }
        }
        ctx.Call("putImageData", imgData, 0, 0)
        return nil
}

My question is: Is SetIndex the most efficient way to write the image's pixels?  I see that syscall/js provides a CopyBytesToJS function, but I was unable to get that to work because the image data is a Uint8ClampedArray and CopyBytesToJS expects a Uint8Array.  I kept trying to populate a Go []byte and transfer that to the image but never succeeded in making that work.

Thanks,

— Scott

[howar...@gmail.com]

I've no relevant experience, but I can recommend a couple of projects to look at in the absence of anyone chiming in with actual experience:

https://agniva.me/wasm/2018/06/18/shimmer-wasm.html

https://github.com/agnivade/shimmer

This is a play-project that involves loading images. It appears to be using a side-load method of converting the image to Base64 and assigning it to the image's src attribute - so it is not using the canvas directly, but if a call to the canvas can draw an image from a URL, this might work - have a look at

// updateImage writes the image to a byte buffer and then converts it to base64.

// Then it sets the value to the src attribute of the target image.

func (s *Shimmer) updateImage(img *image.RGBA, start time.Time) 

in https://github.com/agnivade/shimmer/blob/master/shimmer.go

Here is another project that also went with the Base64 method of passing the array, look for the section labeled "Pixels are Pixels": 

https://blog.jeremylikness.com/blog/2019-03-03_gopher-meet-plasma-a-webassembly-experiment/

His final verdict was that the pure Javascript version performed better than the Go-WASM+JS version.

[Scott Pakin]

Thanks for the links and summaries.  I had actually seen a few of those pages before.  The plasma demonstration at least provides some empirical evidence that as hacky as it is to convert binary→base-64 ASCII→binary for every frame update, the actual performance isn't too terrible.

On Wednesday, March 11, 2020 at 7:22:15 AM UTC-6, howar...@gmail.com wrote:

His final verdict was that the pure Javascript version performed better than the Go-WASM+JS version.

…which is not too surprising given the excessive data-format conversions and what makes me wonder if there isn't a more direct way—especially in newer versions of Go—to pass image data directly between Go and JavaScript.

Regards,

— Scott

[Agniva De Sarker]

On Wednesday, 11 March 2020 18:52:15 UTC+5:30, howar...@gmail.com wrote:

I've no relevant experience, but I can recommend a couple of projects to look at in the absence of anyone chiming in with actual experience:

https://agniva.me/wasm/2018/06/18/shimmer-wasm.html

https://github.com/agnivade/shimmer

This is a play-project that involves loading images. It appears to be using a side-load method of converting the image to Base64 and assigning it to the image's src attribute - so it is not using the canvas directly, but if a call to the canvas can draw an image from a URL, this might work - have a look at

// updateImage writes the image to a byte buffer and then converts it to base64.

// Then it sets the value to the src attribute of the target image.

func (s *Shimmer) updateImage(img *image.RGBA, start time.Time) 

Woops, that comment is outdated and needs to be fixed. The blog post is also outdated now :D.

There is no base64 conversion now anywhere. If you read the code below, you'll see I use 

dst := js.Global().Get("Uint8Array").New(len(s.outBuf.Bytes()))

n := js.CopyBytesToJS(dst, s.outBuf.Bytes())

s.console.Call("log", "bytes copied:", strconv.Itoa(n))

js.Global().Call("displayImage", dst)

Essentially, I copy over the bytes to the array and pass the array over to JS land.

And then displayImage does this:

function displayImage(buf) {
  let blob = new Blob([buf], {'type': imageType});
  document.getElementById('targetImg').src = URL.createObjectURL(blob);
} 

Before the CopyBytesToJS API, I used unsafe to get the slice header and then pass it to JS, and populate the slice with the image contents. Definitely hacky, but that did not require passing the entire image from JS to wasm. With the new API, we are back to passing data, but it's a lot safer.

Coming to the original problem, yes there is no Uint8ClampedArray support. People have already raised it here: https://github.com/golang/go/issues/32402. I think you can use a similar hack for your purposes.

Feel free to hop in to #webassembly slack channel if you have more questions.

-Agniva

[Scott Pakin]

On Thursday, March 12, 2020 at 11:35:30 AM UTC-6, Agniva De Sarker wrote:

There is no base64 conversion now anywhere. If you read the code below, you'll see I use 

dst := js.Global().Get("Uint8Array").New(len(s.outBuf.Bytes()))

n := js.CopyBytesToJS(dst, s.outBuf.Bytes())

s.console.Call("log", "bytes copied:", strconv.Itoa(n))

js.Global().Call("displayImage", dst)

Essentially, I copy over the bytes to the array and pass the array over to JS land.

And then displayImage does this:

function displayImage(buf) {
  let blob = new Blob([buf], {'type': imageType});
  document.getElementById('targetImg').src = URL.createObjectURL(blob);
} 

I see.  In your case, targetImg is an <img>, right?  Do you know how I could use this same technique with a <canvas>? 

Thanks for the code.  It's good to see that there's a way to skip base64 conversions during the Go-to-JavaScript data transfer.

— Scott

[howar...@gmail.com]

  let blob = new Blob([buf], {'type': imageType});
  document.getElementById('targetImg').src = URL.createObjectURL(blob);
} 

I see.  In your case, targetImg is an <img>, right?  Do you know how I could use this same technique with a <canvas>? 

If the <img> was off-screen or hidden, couldn't you still reference it to get the data into the canvas like:

  var img = document.getElementById('targetImg');

  ctx.drawImage(img, 10, 10);

Might not be the most efficient way, but it seems like it should work.

[Scott Pakin]

On Thursday, March 12, 2020 at 1:14:43 PM UTC-6, howar...@gmail.com wrote:

If the <img> was off-screen or hidden, couldn't you still reference it to get the data into the canvas like:

  var img = document.getElementById('targetImg');

  ctx.drawImage(img, 10, 10);

Might not be the most efficient way, but it seems like it should work.

 Yes, that definitely sounds like it's worth a shot.

Thanks,

— Scott

[Scott Pakin]

I figure I ought to follow up with some results.  First, I got the suggested approach of local render + js.CopyBytesToJS + update canvas from image to work, so thanks, Agniva and Howard!  Second, for the benefit of future readers of this thread, one thing that wasn't obvious to me is that one needs to render the image data in a browser-recognizable image format—I used PNG—not raw {red, green, blue, alpha} bytes as is needed when writing directly to a canvas's image data.  Third, I used JavaScript code like the following to update an invisible img then copy the image data from there to a visible canvas:

function copyBytesToCanvas(data) {
    let blob = new Blob([data], {"type": "image/png"});
    let img = document.getElementById("myImage");
    img.onload = function() {
        let canvas = document.getElementById("myCanvas");
        let ctx = canvas.getContext("2d");
        ctx.drawImage(this, 0, 0);
    };
    img.src = URL.createObjectURL(blob);
}

Fourth, the performance is indeed substantially faster than my previous approach based on using SetIndex to write directly to the canvas, even though the new approach requires the extra steps of encoding the image in PNG format and copying the image data from an img to a canvas.  The following performance data, measured with Go 1.14 and Chromium 80.0.3987.132 on an Ubuntu Linux system, is averaged over 10 runs:

Old: 686.9 ± 7.6 ms

New: 290.4 ± 4.1 ms (284.3 ± 4.2 on the WebAssembly side plus 6.0 ± 2.3 on the JavaScript side)

This is the time to render a simple 800×800 gradient pattern.

I hope others find this useful.

— Scott

[Mark Farnan]

Little old, but this might also help. 

 A while back I built a helper package to deal with these issues for canvas rendering from Go.  

https://github.com/markfarnan/go-canvas  

I'm currently working on it to add  WebGL support & get it working in TinyGo (some issues still).

Regards

Mark.

[Kai O'Reilly]

For future reference for anyone who comes across this thread, you can directly pass RGBA image data to a canvas, which is around 200 times faster than encoding it to a PNG and rendering it to an offscreen image. Also, passing the unsafe pointer to the slice to JavaScript instead of copying the bytes improves performance by another factor of 5. These two changes combined reduced the render time for me from roughly 300 milliseconds to 300 microseconds (a 1000x improvement). This performance is enough to run an interactive pure Go GUI through WASM without any noticeable lag, just by writing images to a canvas.

This is the relevant Go code:

```go

 sz := dw.image.Bounds().Size()
ptr := uintptr(unsafe.Pointer(&dw.image.Pix[0]))
js.Global().Call("displayImage", ptr, len(dw.image.Pix), sz.X, sz.Y)

```

And this is the relevant JS code:

```js

const appCanvas = document.getElementById('app');
const appCanvasCtx = appCanvas.getContext('2d');

let wasm;
let memoryBytes;

// displayImage takes the pointer to the target image in the wasm linear memory
// and its length. Then, it gets the resulting byte slice and creates an image data
// with the given width and height.
function displayImage(pointer, length, w, h) {
  // if it doesn't exist or is detached, we have to make it
  if (!memoryBytes || memoryBytes.byteLength === 0) {
    memoryBytes = new Uint8ClampedArray(wasm.instance.exports.mem.buffer);
  }

  // using subarray instead of slice gives a 5x performance improvement due to no copying
  let bytes = memoryBytes.subarray(pointer, pointer + length);
  let data = new ImageData(bytes, w, h);
  appCanvasCtx.putImageData(data, 0, 0);
}

```

In the JS code, `wasm` is initialized as the result of `WebAssembly.instantiateStreaming`.

[robert engels]

Your numbers do not make sense.

At 300 us a frame, it would be more than 3k frames a second - which would only be possible for an extremely tiny area.

You need to include the time to produce the data when considering “render time” - otherwise you are talking apples and oranges.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/267bc2d7-1e7a-4704-98cd-3c471fa0d19dn%40googlegroups.com.

[Kai O'Reilly]

Apologies for any confusion I created. I was referring to the time spent transferring the image from Go to JS and placing it on the canvas, not the actual time it takes the browser to render the canvas, as the transfer and placing time is the only thing that can be easily controlled.